# coder autoimprovement loop

## Idea

The agent that writes code can also drive a review of its own transcript. After a turn finishes, the same model (or a second instance) reads the recorded interaction — user prompts, tool calls, file edits, test output, final diff — and scores how well the work was done. The score is not abstract; it is grounded in what the model can now see about the repository, the task, and the sequence of choices it made. From that review it proposes one or two concrete changes to the process: a better prompt pattern, a missing verification step, a different order of operations, a more accurate tool call, a smaller commit. The next turn starts with those changes applied.

This is not a new idea, but the context here is special. The coder has access to the full transcript, the exact code it wrote, and the test results. It can compare what it did against what it could have done. It can ask itself whether it explored enough before editing, whether it ran the right tests, whether it made the smallest change, whether the commit message matched the diff. The feedback is factual, not rhetorical.

## What a review looks like

A review is a short chat, initiated by the agent, with a structured prompt. The prompt includes:

- the task as given by the user,

- the complete transcript of the work,

- the final diff,

- the test or lint output,

- the commit message and any push result,

- a short list of known best practices for this repository and toolchain.

The model answers with a scored assessment and one or more proposed improvements. The assessment names the strongest and weakest parts of the turn, with specific evidence. Each proposal has a justification, a risk, and a way to verify it.

The known best practices are not fixed. They live in a file the agent can read and update: `docs/coder/best-practices.md`. After each review, the agent can add a newly discovered principle or remove one that no longer holds. The best practices themselves are subject to the same loop.

## Example questions the review asks

- Did the agent read the relevant files before editing? Did it use `grep` and `find` to locate dependencies, or did it guess paths?

- Did it run `cargo check` before `cargo test`? Did it narrow the test scope to the crate that changed?

- Did it make the smallest change that satisfies the request, or did it refactor unrelated code?

- Did it preserve existing style, naming, and error handling patterns?

- Did it update tests when the behavior changed?

- Did it write a commit message that explains why, not just what?

- Did it push a working state, or did it leave the repository with a failing check?

- Did it ask for clarification when the request was ambiguous?

- Did it document non-obvious decisions?

The review does not have to be kind. It can say that a turn wasted time, made a wrong assumption, or introduced a regression. The agent is the reader and the writer, so there is no social cost to candor.

## Iterative loop

1. **Plan.** Given the user request, the agent produces a short plan: the files to touch, the tests to run, the risks to watch.

2. **Work.** The agent executes the plan, calling tools and writing edits, recording the transcript.

3. **Review.** The agent starts a review chat. The review prompt is a new conversation, separated from the work, so the model does not confuse the two contexts. It returns a score and proposals.

4. **Adopt.** The agent applies the highest-impact, lowest-risk proposals. Some proposals become new instructions for the next turn; others are rejected with a note.

5. **Update knowledge.** The agent updates `docs/coder/best-practices.md` or `docs/coder/autoimprove.md` with what it learned.

6. **Next turn.** The next user request starts with the updated instructions, the updated best practices, and the accumulated history of reviews.

The loop is not expected to converge on perfection. It is expected to stop the same mistake from happening twice and to surface patterns that a single turn cannot see.

## Why this might work

- **Grounded feedback.** The model is not evaluating itself in a vacuum. It has the actual transcript and diff.

- **Self-critique without defensiveness.** There is no user or manager to appease; the model can be direct.

- **Accumulated memory.** The best-practices file and the review history give the agent a long-term context that a single chat cannot hold.

- **Targeted improvement.** The loop does not ask the model to be smarter; it asks it to follow a better process.

## Why it might not work

- **Overfitting to the review prompt.** The model might learn to game the scoring rather than improve the work.

- **False confidence.** A model can produce a convincing review without actually understanding the code.

- **Extra cost and latency.** Each review is another model call. The value has to exceed the overhead.

- **Stagnation.** The best-practices file can accumulate contradictions if no one removes obsolete entries.

- **Self-reinforcing errors.** If the review model shares the same blind spots as the work model, it will not catch them.

## First implementation

The smallest version is a manual one. After a session, the agent or the user can copy the transcript into a prompt and ask for a review. The review output is pasted into `docs/coder/autoimprove.md` or `docs/coder/best-practices.md`. A more automated version can call a second agent with the session id from `trace ingest` and ask it to produce the same review.

A fully automated loop would require:

- a way to extract the session transcript in a structured form,

- a review agent prompt that returns machine-readable proposals,

- a claim on whether to apply each proposal,

- a way to surface the review to the user for the first few turns,

- a regression test that fails if a known best practice is violated.

## Suggested review prompt

```

You have just completed a coding task. The user asked:

<request>

{user_request}

</request>

The transcript of your work follows:

<transcript>

{transcript}

</transcript>

The final diff is:

<diff>

{diff}

</diff>

The verification output is:

<verification>

{verification}

</verification>

The best practices known to the project are:

<practices>

{practices}

</practices>

Score the work on a scale from 0 to 10, with specific evidence for each point gained or lost. Propose one to three concrete changes to the process that would have improved the outcome. Each proposal must include: the problem it solves, the evidence from the transcript, the risk, and how to verify it. Finally, list any best practices that should be added, removed, or changed.

```

The output can be appended to a session log. Over time, the log becomes a dataset of which process changes actually helped.

# Coder autoimprovement: the plan

Date: 2026-08-26. Supersedes the 2026-08-25 proposal that this file used to

hold. Companion documents: `docs/coder/runbook.md` (the operating procedure an

agent executes), `docs/coder/best-practices.md` (the ledger the loop reads and

writes). The measurement substrate is the Gym: the Harbor plan in the

openagents.com repo (`docs/2026-08-24-harbor-terminal-bench-plan.md`), the

harness in `bench/`, the scoring package in `packages/coder-effectiveness/`,

and the receipted store in `bench-results/`.

## 1. The claim

The coder can improve itself, and the improvement can be real rather than

narrated, because every piece the loop needs already exists in this

repository:

- **A graded arena.** Harbor runs Terminal-Bench 2.0 and 79 other datasets

  against `openagents coder` through the installed-agent adapter

  (`bench/adapters/openagents_coder.py`). A verifier, not the agent, decides

  whether a task passed.

- **A pinned recipe.** Suites pin tasks by content digest

  (`bench/suites/*.suite.json`), thresholds set floors

  (`packages/coder-effectiveness/thresholds/`), and a run records CLI

  version, model, lane, and rate catalog. Two rows are comparable only when

  their suite keys match.

- **A tamper-evident record.** `bench-results/*.jsonl` is a hash chain. A

  smoke run cannot be recorded as a score, an edited row breaks its receipt,

  and a removed row breaks the chain. The store's refusals

  (`unclassified_run`, `smoke_run`, exit 3) are the enforcement, not advice.

- **A transcript the reviewer can read.** Every trial leaves an ATIF

  trajectory with token metrics and `tool.ran` steps, the Harbor

  `result.json` with the verifier's decision, and `coder.txt` with the

  thread announcement.

- **An extension surface.** WASM plugins under `plugins/` load from

  digest-pinned manifests, and their `tool.ran` steps carry provenance, so a

  plugin's effect on a score is attributable to an exact artifact.

The loop is therefore: **change one thing, run the same suite, read the

delta, keep what helped, write down why.** Everything else in this document

is discipline around that sentence.

## 2. The improvement axes

Four kinds of change move the numbers, and each has its own evidence base

already on record.

### 2.1 Process levers: how the coder spends tokens and rounds

The first graded runs (openagents.com repo,

`docs/terminalbench/2026-08-24-fix-git-run-analysis.md`) measured the cost

structure directly. Five `fix-git` trials, all passing, split on cost shape:

- One lane solved the task in 6 model calls and 43,894 prompt tokens;

  another took 15 calls and 124,941, because every round re-sends the

  growing transcript. **Round count is the dominant metered cost**, and the

  highest-leverage habit is batching independent commands into one tool

  call.

- Two `git log -p` dumps rode along in a transcript re-sent a dozen times.

  `--stat` first, `-p` only for the file in question.

- The local lane inverts the economics: 17,160 output tokens cost nothing in

  dollars and ~18.5 minutes in wall clock. Verbosity guidance must be

  lane-aware.

These levers live in the system prompt, the tool descriptions, and the tool

budget — all cheap to change, all measurable on the same suite.

### 2.2 Plugins: capabilities with attributable deltas

Twelve plugins are in-tree with digest-pinned artifacts (`plugins/`):

`word-stats`, `file-stats`, `dir-stats`, `foreign-sessions`,

`read-conversation`, `knowledge-base`, `code-search`, `git-facts`,

`git-lost-work`, `patch-check`, `repo-map`, `repo-tree`, `session-search`.

The harvest backlog, ordered by expected Gym delta per unit of work, is the

openagents.com repo's `docs/2026-08-25-plugin-harvest-targets.md`.

The standing rule from the Harbor plan applies: **a plugin lands with a

before-and-after suite score on the same recipe.** The A/B is cheap — same

suite, plugin present versus absent — and the ATIF provenance stamps make

the delta attributable to the exact digest. A plugin that moves no score is

questioned; a graded task class the coder cannot pass is a plugin (or core

capability) backlog item. `tb2-cross-section` was built with plugin oracles

in mind: `git-leak-recovery` and `sanitize-git-repo` for the git-forensics

plugins, `password-recovery` for file forensics.

### 2.3 Harness and runtime: what the coder is missing structurally

The Claude Code teardown series (local working set, `docs/teardowns/cc/`,

verifiable directly against `packages/openagents-cli/src/`) names the

structural gaps in cost order:

- **No compaction.** The coder has per-result output caps and nothing else;

  long tasks pay quadratic transcript replay. `schemelike-metacircular-eval`

  in the cross-section suite is the designated oracle for this cost.

- **No client-side prompt history**; resume replays server events.

- **Shell safety is a static regex refusal table**, not a parser.

Each gap is a candidate change with a designated suite oracle. Structural

work is the most expensive axis; it enters the loop only when a cheaper

lever has stopped paying, and it lands under the repository completion gate

(`pnpm run check`) like any other code.

### 2.4 Lanes and routing: which model gets which task class

The same suite per catalog model, per lane, is the comparative matrix the

compute mix wants. Scores per lane are not just a leaderboard: they are the

evidence that turns "which lane should this task class route to" into

policy. One caveat is on record and blocking honest cost comparison: the

proxy's usage records do not surface cached-token splits, so metered-lane

costs are overstated on exactly the transcript-replay workloads that matter

(tracked as OpenAgentsInc/openagents.com#220). Until it lands, compare

lanes on success rate and rounds, and treat dollar figures as ceilings.

## 3. The review loop

After a graded run, the same model (or a second instance) reviews the work

in a **separate conversation** so the review context cannot contaminate the

working context. The review inputs are artifacts, not memories:

- the task instruction and the verifier's decision (`result.json`),

- the ATIF trajectory (steps, tool calls, token metrics),

- the coder transcript (`coder.txt`),

- the diff, where the task produced one,

- the current `docs/coder/best-practices.md`.

The review returns a scored assessment with evidence, and one to three

proposals. A proposal is typed: **the lever** (which axis in §2), **the

evidence** (specific steps in the trajectory), **the risk**, and **the

verification** (which suite, and the delta direction that would confirm

it). Reviews append to `docs/coder/reviews/` as dated files; over time the

directory is a dataset of which process changes actually helped.

The review is allowed to be harsh. There is no audience to appease, and a

review that praises a wasteful run is itself a defect the next review

should catch.

## 4. The verification law

The Rust CLI port failure (openagents.com repo,

`docs/2026-08-26-rust-cli-port-parity-failure-postmortem.md`) is this

loop's founding negative example. Autonomous sessions reported full parity

and closed seven issues; the delivered TUI had no input widget, three

commands were missing from the argument parser, and ~2,700 lines stood in

for ~35,400. Three mechanisms produced the false report, and each yields a

standing rule:

1. **"Compiles and green tests" is not done.** The tests asserted struct

   constructors. Rule: a completion claim names the oracle that observed

   the behavior the user asked for, and a unit test of a stub observes

   nothing.

2. **The verifier shared the worker's blind spot.** Agent shells are

   non-TTY, so every check hit the headless branch and never rendered the

   TUI at all. Rule: verify on the surface the user touches. For

   interactive terminal claims that means a PTY-driven harness; until one

   exists in this repository, no agent may close an interactive-TUI issue

   on headless evidence.

3. **Scope truncation under goal pressure.** Facades matched signatures and

   omitted the machines behind them. Rule: parity claims quantify — line

   counts, command inventory, feature checklist against the source — and

   the reviewer checks the quantities, not the adjectives.

The same law covers the measurement side, where the store already enforces

it: a run that skipped pinned tasks is a smoke run, a crashed verifier is

`abandoned` not a grade, unpriced is never zero, and a regression stays in

the chain. **A deliberate regression row is already in

`bench-results/tb2-quick.jsonl` to prove the floor fires; do not clean it

up.**

## 5. The knowledge ledger

`docs/coder/best-practices.md` is the loop's memory. Its governance:

- Every entry is **falsifiable** and carries provenance: the run, review,

  or postmortem that produced it.

- Entries have a status: `adopted`, `proposed`, or `refuted`. Refuted

  entries stay, struck through in place, so the loop does not rediscover

  them.

- An entry states how a violation is detected — an automated gate where one

  exists (the store's refusals, the repository check), otherwise the review

  question that catches it. An entry nothing can detect is an aspiration,

  not a practice; it does not get `adopted`.

- The ledger is subject to the loop: a review may propose adding, demoting,

  or refuting an entry, with the same evidence requirements as any other

  proposal.

## 6. Known failure modes, and their controls

| Failure mode | Control |

| --- | --- |

| Gaming the score instead of the work | The verifier grades outcomes, not transcripts; suites are content-pinned; threshold edits are a separate change from any run they would flatter |

| Confident review without understanding | Proposals must cite trajectory steps; a proposal with no evidence pointer is rejected in the adopt step |

| Cost of the loop exceeding its value | `tb2-quick` (2 tasks, floored for its size) is the iteration suite; the 12-task cross-section runs only when quick results justify it |

| Ledger accumulating contradictions | Refutation is a first-class status; the runbook's adopt step requires checking new entries against existing ones |

| Reviewer sharing the worker's blind spots | Verify on the user-facing surface (§4); prefer a different model instance for review when the finding is load-bearing |

| Noise mistaken for signal | Two tasks give rates of 0, .5, 1 only; a quick-suite delta motivates a cross-section run, it does not conclude anything |

## 7. Sequencing

1. **Seeded** — this document, the runbook, and the best-practices ledger

   (this change).

2. **Manual loop** — an agent follows the runbook end to end: baseline,

   one lever, re-run, review, ledger update. Each cycle is a normal commit.

3. **Plugin A/B cadence** — every plugin from the harvest backlog lands

   with its delta; existing plugins get retroactive A/B rows as suite time

   allows.

4. **The missing gate** — a PTY-driven interactive harness for the TUI, so

   the class of failure in §4 has an automated detector rather than a rule.

5. **Automated review** — a second agent invoked with the trial artifacts

   produces the review without a human copying transcripts; proposals

   arrive machine-readable and the adopt step becomes a diff.

6. **Routing feedback** — per-lane suite scores feed lane selection, once

   cached-token accounting (#220) makes the cost axis honest.

The loop is not expected to converge on perfection. It is expected to stop

the same mistake from happening twice, and to make "the coder got better"

a sentence with a receipt behind it.

# Coder best practices

The knowledge ledger of the autoimprovement loop (`docs/coder/autoimprove.md`

§5). Every entry is falsifiable, carries provenance, and says how a violation

is detected. Statuses: `adopted` (in force), `proposed` (awaiting a measured

delta or an oracle), `refuted` (kept, struck through, so it is not

rediscovered). Reviews propose changes here; the adopt step of the runbook

applies them.

Provenance abbreviations:

- **fix-git** — openagents.com repo,

  `docs/terminalbench/2026-08-24-fix-git-run-analysis.md`

- **postmortem** — openagents.com repo,

  `docs/2026-08-26-rust-cli-port-parity-failure-postmortem.md`

- **bench** — `bench/README.md` and `bench-results/README.md`

- **harvest** — openagents.com repo,

  `docs/2026-08-25-plugin-harvest-targets.md`

## Verification

### V1. A completion claim names its oracle — `adopted`

State what observed the behavior: which verifier, which test, which surface.

"It compiles and the tests pass" is a claim about stubs unless the tests

reach the behavior the user asked for.

**Provenance:** postmortem (tests asserted struct constructors; seven issues

closed falsely). **Detection:** review question; for graded work, the

Harbor verifier is the oracle of record.

### V2. Verify on the surface the user touches — `adopted`

Headless output does not prove an interactive TUI works. Agent shells are

non-TTY and will take the non-interactive branch every time.

**Provenance:** postmortem (every check bypassed ratatui via the headless

fallback). **Detection:** review question. An automated PTY harness is the

planned gate (autoimprove §7.4); until it exists, no interactive-TUI issue

closes on headless evidence.

### V3. Parity claims quantify — `adopted`

Port and parity work reports counts against the source: commands in the

parser, lines per subsystem, features per checklist. Adjectives

("complete", "full parity") are not evidence.

**Provenance:** postmortem (~2,700 lines reported as parity with ~35,400).

**Detection:** review compares the claimed inventory to the source

inventory.

### V4. A crashed grader is not a grade — `adopted`

A run whose verifier never ran is `abandoned`, not scored, and a partial

run of a pinned suite is a smoke run.

**Provenance:** bench. **Detection:** automated — `run-suite.sh` patches to

`abandoned`; the results store refuses `smoke_run` and `unclassified_run`

with exit 3.

## Tool habits

### T1. Batch independent commands into one tool call — `adopted`

Round count drives metered cost: prompt tokens scale with rounds times

transcript size, not with work done. The same task has been done in 6 model

calls and in 15; the 15 cost three times the input tokens.

**Provenance:** fix-git. **Detection:** review reads rounds and prompt

tokens from the ATIF trajectory; regression shows as rising

rounds-per-accepted-outcome on an unchanged suite.

### T2. `--stat` before `-p` — `adopted`

Never dump a full patch into the transcript to find out which file changed.

Survey with `--stat`, then read the one file in question. A patch dumped

early is re-sent every round after.

**Provenance:** fix-git (two `git log -p` dumps re-sent ~12 times).

**Detection:** review flags full-patch dumps followed by further rounds.

### T3. Verbosity guidance is lane-aware — `adopted`

On metered lanes, output tokens are money; on the local lane they are

minutes (17k output tokens ≈ 18.5 wall-clock minutes at local generation

speed). Fewer, larger, quieter rounds on local; terse output everywhere.

**Provenance:** fix-git. **Detection:** review compares output tokens and

wall clock against lane norms in the results store.

### T4. Prove repository topology before acting on it — `proposed`

The winning `fix-git` runs read the recovered commit's content before

resolving the conflict, and the strongest run proved ancestry with

`merge-base --is-ancestor` before choosing a merge. Assuming the

relationship is how a plausible wrong resolution passes local inspection

and fails the verifier.

**Provenance:** fix-git. **Detection:** review question on version-control

tasks. Promote to `adopted` when a cross-section run shows the habit

correlating with acceptance on tasks 1–3.

## Measurement

### M1. One lever per cycle — `adopted`

A comparison is between two rows that differ in one thing. A run that

changes the model and the prompt and a plugin attributes its delta to

nothing.

**Provenance:** bench (the compare view names `model also varies` as a

confounder by design). **Detection:** automated in part — `suiteKey`

excludes the axes a comparison varies, and compare names confounders; the

runbook's cycle enforces the rest.

### M2. Unknown cost stays unknown — `adopted`

`null` with a disposition (`unmetered_local_lane`, `no_accepted_outcomes`,

`cost_unknown`), never zero. A lane that gets cheaper per attempt while

accepting less is a regression, and cost-per-accepted-outcome is the shape

that catches it.

**Provenance:** bench. **Detection:** automated — the store and compare

refuse to launder unpriced into free.

### M3. Regressions stay in the chain — `adopted`

A graded run of a real configuration is recorded even when — especially

when — it embarrasses the trend. The store's hash chain exists to make the

alternative detectable.

**Provenance:** bench (the deliberate `qwen3:0.6b` failure row in

`tb2-quick.jsonl`). **Detection:** automated — `receipt_mismatch` and

`chain_broken` on verify.

### M4. Threshold edits are their own change — `adopted`

Never adjust a floor in the same change as a run it would flatter. Raising

a floor as a lane proves itself out is the intended direction; lowering one

to pass is a contract change and says so in its commit.

**Provenance:** bench thresholds `$comment` blocks. **Detection:** review

of the diff that touches `packages/coder-effectiveness/thresholds/`.

### M5. Quick suites iterate, cross-sections conclude — `adopted`

`tb2-quick`'s two tasks admit success rates of 0, .5, and 1 only. A quick

delta selects what to run next; only a full pinned-suite run is a score,

and the two suites share no suite key so no tool will compare them.

**Provenance:** bench. **Detection:** automated (suite keys) plus the

runbook's cycle.

## Plugins

### P1. A plugin lands with its Gym delta — `adopted`

Before-and-after on the same recipe, plugin present versus absent, delta

attributed through the ATIF `tool.ran` provenance stamps to the exact

digest. A plugin that moves no score is questioned in the review, not

waved through.

**Provenance:** harvest; Harbor plan. **Detection:** review of the landing

change for the A/B rows.

### P2. Plugins are read-only, one-shot, bounded, and honest — `adopted`

The sandbox grants read-only mounts and nothing else; one packet in, one

packet out, under time and memory bounds; typed schemas in the manifest;

output that names its own truncation (`tail_only`,

`dropped_leading_turns`). Edits, writes, and process execution stay in the

coder's core tools under the permission profile.

**Provenance:** harvest; `plugins/README.md`. **Detection:** the host

refuses undeclared imports and stale digests; review checks the manifest

against the shape rules.

### P3. A catalog line is contested space — `adopted`

The capability tool carries each installed plugin's name and first sentence,

capped at twelve. A plugin competes for that slot with everything installed;

its first sentence must earn a model's attention on the turns where the

plugin applies.

**Provenance:** harvest. **Detection:** review question when the A/B shows

a plugin installed but never invoked on tasks it should have served.

## Repository

### R1. Pack with `pnpm pack`, never `npm pack` — `adopted`

The manifest carries pnpm `catalog:` versions that only `pnpm pack`

rewrites into versions npm can install. An `npm pack` tarball looks healthy

and fails for every consumer.

**Provenance:** bench; `CLAUDE.md` deploy section (shipped broken once as

`@openagentsinc/cli@0.2.0`). **Detection:** the CLI's `verify:package`

refuses a manifest carrying `catalog:`.

### R2. Push the forge, not GitHub — `adopted`

The `openagents` remote records the push in the WAL; GitHub is a mirror

production force-pushes. A direct GitHub push is overwritten, not merged.

**Provenance:** openagents.com `AGENTS.md`. **Detection:** automated —

`push-remote-check.sh` refuses a non-forge push where installed.

### R3. Fresh worktree per unit of work — `adopted`

Implement off current `origin/main` in a new worktree; never edit through

another agent's dirty checkout, and never move its uncommitted work aside.

**Provenance:** repository contract (owner mandate 2026-07-20).

**Detection:** review of the working-tree state in the transcript.

# Coder autoimprovement runbook

The operating procedure for an agent running the loop in

`docs/coder/autoimprove.md`. Follow it top to bottom the first time; after

that, each cycle is §4 (a process or code lever) or §5 (a plugin), always

followed by §6 (review) and §7 (record and land).

Read first, in order:

1. `docs/coder/autoimprove.md` — the plan and the verification law.

2. `docs/coder/best-practices.md` — the ledger. You will update it.

3. `bench/README.md` — harness mechanics, environment pitfalls.

4. `bench-results/README.md` — what the store accepts and refuses.

## 0. Authority and boundaries

You may change: the coder system prompt and tool descriptions

(`crates/coder-lite`, `packages/openagents-cli`), tool budgets, plugins

(`plugins/`), suites and thresholds (as their own change — best practice

M4), this directory's docs, and CLI code under the repository completion

gate.

You may not: edit or reorder `bench-results/*.jsonl` rows (append only,

through the tooling), record a smoke or partial run as a score, delete the

deliberate regression row in `tb2-quick.jsonl`, close an interactive-TUI

issue on headless evidence (best practice V2), or change a threshold in the

same change as a run it would flatter.

Work in a fresh worktree per unit (`git fetch openagents main && git

worktree add --detach <path> openagents/main`), and land by pushing to the

forge remote, never GitHub (best practices R2, R3). For CLI code changes

the completion gate is `pnpm run check`; docs-only changes push with

`--no-verify` after the docs checks.

## 1. Prerequisites

- **Docker with amd64 emulation that works.** Terminal-Bench images are

  amd64. On Apple Silicon, enable Rosetta in Docker Desktop (Settings →

  General → "Use Rosetta for x86_64/amd64 emulation"); under plain qemu the

  verifier's `uv`/`pytest` segfaults after the agent phase and every trial

  grades `ungraded`.

- **Harbor** installed (the clone is `../projects/repos/harbor`;

  `pip install harbor` or run from the clone).

- **A dev forge**, for proxy-lane runs: in the openagents.com repo,

  `PHX_LISTEN_ALL=true mix phx.server`, and an `OPENAGENTS_TOKEN` with

  `chat:account` scope. If another session holds the port loopback-only,

  bridge `0.0.0.0:4001 → 127.0.0.1:4000` and set

  `OPENAGENTS_CODER_API_URL=http://host.docker.internal:4001` instead of

  fighting for the port.

- **A packed CLI tarball** from the working tree — this is what the adapter

  installs into containers, so it is how your change reaches the arena:

  ```sh

  cd packages/openagents-cli

  pnpm build

  pnpm pack --pack-destination ../../bench

  ```

  `pnpm pack`, never `npm pack` (best practice R1).

- **Ollama** with the local-lane model pulled, for local-lane runs. Local

  runs need no token; keep `--n-concurrent 1` because the model owns the

  cores.

## 2. Baseline

Establish the number you are trying to move, on the lane you are testing.

```sh

bench/run-suite.sh bench/suites/tb2-quick.suite.json \

  --model openai/gpt-5.6-luna --lane proxy \

  --jobs-dir /tmp/gym-jobs/baseline

```

Then score and record it:

```sh

pnpm run effectiveness:report -- /tmp/gym-jobs/baseline/<job-dir> \

  --suite tb2-quick --lane proxy \

  --suite-manifest bench/suites/tb2-quick.suite.json \

  --thresholds packages/coder-effectiveness/thresholds/tb2-quick.json \

  --append bench-results/tb2-quick.jsonl

```

Exit codes: `0` gate passed, `1` a floor breached, `2` unverifiable

(not a pass), `3` scored but refused by the store — read the refusal, it is

telling you the run does not qualify as a score. Verify the chain and read

the trend:

```sh

pnpm run effectiveness:compare -- bench-results/tb2-quick.jsonl

```

With `OPENAGENTS_TOKEN` set, the runner registers the run at

`POST /api/v1/gym/runs/start` and `/gym` shows it live; finalization to

`graded` (or `abandoned` when no verifier ran) happens through

`bench/post_gym_run.py --run-id` automatically.

If no baseline exists yet for the lane and suite you are working, the

baseline run **is** the first cycle's deliverable. Record it and stop

there; a lever with no baseline produces a delta against nothing.

## 3. Choose one lever

One per cycle (best practice M1). Sources, in order of cost:

1. **The ledger's `proposed` entries** (`docs/coder/best-practices.md`) —

   each names its promotion oracle.

2. **The latest reviews** (`docs/coder/reviews/`) — unadopted proposals

   with evidence already attached.

3. **The harvest backlog** (openagents.com repo,

   `docs/2026-08-25-plugin-harvest-targets.md`) — plugins ordered by

   expected delta; go to §5.

4. **Structural gaps** (autoimprove §2.3: compaction, history, shell

   parsing) — only when cheaper levers have stopped paying, and with the

   designated suite oracle named before you start.

Write down, before implementing: the lever, the suite that will measure it,

and the delta direction that would confirm it. If you cannot name the

measuring suite, the lever is not ready for a cycle.

## 4. The cycle: process and code levers

1. **Implement** in a fresh worktree. Prompt/tool-description levers land

   in the coder's system prompt and tool declarations; budget levers in the

   tool budget; code levers under `pnpm run check`.

2. **Repack** the CLI tarball (§1) so the arena runs your change.

3. **Re-run the same suite, same model, same lane** as the baseline.

   Nothing else varies. The runner pins CLI version into the row, so the

   two rows differ in exactly the axis you changed.

4. **Compare.**

   ```sh

   pnpm run effectiveness:compare -- bench-results/tb2-quick.jsonl

   ```

   Read the trend line and the confounder note. On `tb2-quick`, treat the

   delta as a selector, not a conclusion (best practice M5): an

   encouraging quick delta earns a cross-section run —

   ```sh

   bench/run-suite.sh bench/suites/tb2-cross-section.suite.json \

     --model openai/gpt-5.6-luna --lane proxy --n-concurrent 2 \

     --jobs-dir /tmp/gym-jobs/xsec-<lever>

   ```

   — scored against `thresholds/tb2-cross-section.json` and appended to

   `bench-results/tb2-cross-section.jsonl`. Success rate is not the only

   axis: rounds, prompt tokens, and wall clock per accepted outcome are

   where process levers show first (the fix-git analysis is the worked

   example).

5. **Decide.** Improved or neutral-but-simpler: keep. Worse: revert the

   lever, keep the recorded row (best practice M3), and write the refutation

   into the review. Timeout-shaped failures are not efficiency signal —

   rerun with `--timeout-multiplier 2.0` before concluding anything.

## 5. The cycle: plugins

1. **Pick the target** from the harvest backlog and confirm it fits the

   admitted shape (best practice P2): read-only or pure, one-shot, bounded,

   typed, truncation-honest. If it needs writes, execution, or a lifetime

   beyond one call, it is core-tool work, not a plugin.

2. **Build** in `plugins/<name>/` on `openagents-pdk`: one

   `fn handle(input) -> Result<Output, Refusal>` plus `plugin_entry!`.

   Declare only the imports the manifest's mounts grant — the host refuses

   undeclared imports.

3. **Pin the artifact.** Build for `wasm32-unknown-unknown --release`, copy

   the `.wasm` beside the source, and update the manifest's

   `artifact.digest` (`plugins/README.md` has the exact loop). The host

   refuses a stale digest — that refusal is the supply chain, keep it.

4. **A/B on the oracle tasks.** Same suite, same recipe, plugin absent then

   present. The cross-section's designated oracles: tasks 1–2

   (`git-leak-recovery`, `sanitize-git-repo`) for git forensics, task 9

   (`password-recovery`) for file forensics; for a new capability class,

   name the oracle tasks in the landing change. Confirm in the ATIF that

   the with-plugin run actually invoked it (`tool.ran` steps carry the

   digest); a plugin installed but never called on its own oracle tasks is

   a catalog-description problem (best practice P3) before it is a

   capability problem.

5. **Land with the delta** (best practice P1): both rows recorded, the

   attribution stated in the change. No delta and no convincing rationale:

   the plugin waits.

## 6. Review

After every cycle, run the review as a **separate conversation** — a fresh

agent context whose only inputs are the artifacts:

- the trial directories (`result.json`, `trajectory.json` ATIF,

  `coder.txt`) from the run,

- the diff of the lever,

- the two store rows (before and after),

- the current `docs/coder/best-practices.md`.

Review prompt skeleton:

```

You are reviewing one autoimprovement cycle of `openagents coder`.

<lever>{what changed, and the predicted delta}</lever>

<baseline-row>{jsonl row}</baseline-row>

<result-row>{jsonl row}</result-row>

<trials>{per-trial: instruction, verifier decision, ATIF metrics,

notable transcript spans}</trials>

<diff>{the lever's diff}</diff>

<practices>{docs/coder/best-practices.md}</practices>

Score the cycle 0–10 with specific evidence for each point gained or

lost. Answer: did the lever cause the delta, or does a confounder

explain it? Were any ledger practices violated (cite the entry and the

transcript step)? Propose one to three changes, each typed as

{lever, evidence: trajectory steps, risk, verification: suite and

expected delta direction}. Finally, list ledger entries to add, promote,

demote, or refute, with provenance.

```

The review must cite trajectory steps for every claim; a proposal without

an evidence pointer is rejected at the adopt step. Save the output to

`docs/coder/reviews/YYYY-MM-DD-<lever-slug>.md`.

## 7. Record, adopt, land

1. **Adopt** the accepted proposals: apply ledger changes to

   `docs/coder/best-practices.md` (checking new entries against existing

   ones for contradiction), carry rejected proposals into the review file

   with a one-line reason.

2. **Commit** the cycle as one unit: the lever, the review file, the ledger

   change, and the appended store rows. The commit message states the lever

   and the measured delta with its suite — a number, not an adjective

   (best practice V3).

3. **Push to the forge** (`git push openagents HEAD:main`), reconcile the

   canonical checkout, remove the worktree.

## 8. Stop rules

- **Three consecutive failures of the same gate** — the suite will not

  grade, the store refuses, the environment will not come up — stop

  attempting it. Preserve the last failure, write the smallest

  reproducible blocker and the next falsifiable hypothesis, and hand off.

  A fourth attempt needs new evidence.

- **An ungraded epidemic** (`ungraded` trials, `uv`/`pytest` segfaults)

  is the emulation problem, not your lever. Fix Rosetta or move to a cloud

  environment (`--env daytona` and peers) before reading any number.

- **A quick-suite success rate of 0** on a lane that previously scored:

  suspect the lane (auth, proxy URL, model availability) before the lever;

  the third row of `tb2-quick.jsonl` is what a lane-degradation signature

  looks like.

- **Budget**: a cross-section run is ~12 tasks × minutes-to-hours under

  emulation. Do not start one you cannot let finish; an abandoned run is

  recorded as `abandoned`, and that is the honest state, but it bought

  nothing.

- When blocked on an owner action, write the `NEEDS-OWNER:` note and pull

  the next non-blocked lever; the loop does not idle.

## 9. Current state (update as it changes)

- Baselines on record: `tb2-quick`, local lane, `qwen3.8:27b-mtp-q8_0` at

  0.5 success (two runs), plus the deliberate `qwen3:0.6b` regression row.

  No proxy-lane rows recorded yet; no cross-section rows recorded yet.

- `owned-closed-issues` is a smoke suite until its environments exist

  (`bench/tasks/owned/README.md`).

- Cached-token splits are not surfaced end to end

  (OpenAgentsInc/openagents.com#220); until then, metered-lane dollar

  figures are ceilings, and lane comparisons lean on success rate and

  rounds.

- The PTY-driven interactive harness (autoimprove §7.4) does not exist

  yet; best practice V2 is enforced by rule, not by gate.