Build the effectiveness suite: graded coder runs with cost per accepted outcome #34

Re-based on Harbor after a deep read of projects/repos/harbor (docs/2026-08-24-harbor-terminal-bench-plan.md in openagents.com): the suite runs ON Harbor rather than a hand-rolled runner. Harbor is the Terminal-Bench 2.0 harness with an 80-dataset registry (SWE-bench and Aider Polyglot included), 48 existing agent adapters giving us the competitive baseline for free, ATIF as its native trajectory format (the same v1.7 the coder already exports), and pluggable container environments. What stays ours is exactly this issue's contract: thresholds files, digest-pinned recipes, receipted results in bench-results, and cost per accepted outcome as the headline — computed from each trial's ATIF token metrics joined to the model catalog's pricing. First dataset: terminal-bench@2.0 (agent-agnostic terminal work, closest to what the coder is for), then SWE-bench/Aider Polyglot from the same registry with zero new harness code. Now blocked by #35, the openagents-coder installed-agent adapter.

Same ingest note as #33: post each graded suite run to POST /api/v3/gym/runs (contract in that comment) so the /gym scoreboard carries the effectiveness trend without a second reporting path.

Unblocked now that #35 is closed, and every piece this needs is in place:

• Adapter — bench/adapters/openagents_coder.py, a Harbor installed-agent adapter that runs the real openagents coder and copies its ATIF trajectory into the job's logs.
• Runner — bench/run-suite.sh takes a suite file and a model, packs the working-tree CLI, runs the whole suite through Harbor in one invocation, and posts the graded job.
• Suite — bench/suites/tb2-cross-section.txt, twelve curated Terminal-Bench 2.0 tasks across git forensics, builds, C extensions, coverage, security fixes, log parsing, tokenisation, certificates, web-server config, and an interpreter task.
• Scoreboard — POST/GET /api/v3/gym/runs and the operator-gated /gym page, with per-run duration and lane.
• Lanes — proxy (house providers) and local (Ollama) both proven end to end on fix-git; cost per run is already recorded from grant usage, and cached-token splits now land beside the totals (openagents.com#220).

What is left is the run itself: twelve tasks across three lanes is thirty-six graded container executions, each a real model spend and roughly a Docker-hour in total. That is a deliberate cost, so I am not starting it unattended — it wants an explicit go, and a choice of which lanes to include.

One caveat to fold into the plan when it runs: gemini-3.7-flash is currently dead upstream (openagents.com#238), so a run today should either exclude that lane or expect it to fail wholesale rather than reading the failure as an agent result.

Shipped the grading, pricing, aggregation, and gate layer as packages/coder-effectiveness in 8703101286. Leaving this open: the acceptance clause needs live scheduled runs, which fixture runs cannot stand in for.

What landed

bench/run-suite.sh already packs the CLI, runs a suite through Harbor against the #35 adapter, and leaves a job directory. The new package reads that directory and answers what the run was for — no new artifact from the coder, just the trial results, the ATIF export, and coder.txt already on disk.

pnpm run effectiveness:report -- <job-dir> --suite tb2-cross-section --lane proxy \
  --thresholds packages/coder-effectiveness/thresholds/tb2-cross-section.json

Cost per accepted outcome is total run cost over accepted outcomes, failures included — the same definition benchmark/report.ts already uses for costPerAcceptedOutcomeMsat. Dividing accepted-run cost by accepted runs would hide exactly the regression the gate exists to catch. Reported alongside it: success rate, ungraded count, tokens, cached reads, tool calls, wall clock, and a run digest over suite/lane/tasks/CLI version/model/catalog version.

The lane we run most cannot be priced, and the suite says so

gpt-5.6-luna has no pricing key in the forge model catalog, deliberately — "so a grant pinned to it records no estimated cost rather than a made-up zero." The worker table omits it for the same stated reason. gemini-3.7-flash is also absent from the worker table; the forge catalog prices it and ox-alpha, and marks both "Placeholder: the operator must set real provider rates before accepting any spend."

So a cost number computed today is partly fictional, and the suite is built around that:

• Unpriced → unknown, never $0. Four dispositions carry usd: null: unpriced_model (the catalog knows the id and declines to price it), unknown_model (never heard of it — a weaker finding), unmetered_local_lane (an ollama: lane bills no metered tokens; unmetered, not free), unknown_usage.
• No borrowed fallback. The worker's UNKNOWN_MODEL_COST exists so an un-tabled model is not under-charged at the till, and its own source says it is not a measured rate. Charging conservatively and measuring honestly are opposite jobs, so this package does not import it.
• Partial coverage withholds the number. When some trials price and others do not, the run reports cost_partial and prints coverage instead. Summing the priced trials over every accepted outcome yields a real-looking figure that is too low by whatever the rest cost — the case a careless aggregate gets wrong.
• Placeholder rates are labelled. A number built from provisional rates carries rateBasis: "operator_placeholder" everywhere it travels.

The gate has a third verdict

passed / failed / unverifiable, exiting 0 / 1 / 2. A cost floor that could not be measured is unverifiable, not passed — otherwise the gate goes quietest exactly when the lane is least accountable, and a CI step checking only for a zero exit reads silence as green. Scoring a dollar ceiling against placeholder rates is likewise unverifiable unless the thresholds file opts in with acceptPlaceholderRates: true. A measured breach still outranks an unmeasurable criterion.

A trial whose verifier never ran is ungraded: neither pass nor failure, out of the success-rate denominator, counted, and capped by maxUngradedRatio. Terminal-Bench images are amd64 and their verifier segfaults under qemu on Apple Silicon, so a crashed grader is routine — folding it into either bucket would move the headline number for a reason unrelated to the coder.

Proof

57 tests over five fixture Harbor jobs — fully priced, luna, mixed, crashed verifier, and a deliberate regression that raises cost per accepted outcome and trips the gate. No model called, no Docker image, no clock read, so CI and a laptop agree. check:fast green.

Remaining, and why it is not closed

• Acceptance is unmet. Two consecutive scheduled runs producing comparable rows, and one deliberately introduced regression caught by the floor, need real Harbor runs on amd64 hardware. The fixtures prove the grading and the arithmetic, not the schedule. The regression fixture proves the floor catches a regression in the aggregation; it is not the live proof this clause asks for.
• bench-results append with receipts is not wired. The report is --json-shaped and ready for it.
• Suite scope. The floors point at bench/suites/tb2-cross-section.txt (12 tasks). The 20-30 task target and the owned set drawn from this tracker's closed issues are not built.
• Per-lane comparison falls out of the digest for free once two lanes have real runs, but has not been run.
• #36 dependency. The ATIF exporter writes token totals and no cost, and cached reads survive only per step as metrics.extra.cache_read_input_tokens (this package sums them). The token-economy delta #36 measures is readable from these rows without a change here.
• Pre-existing: packages/openagents-cli/package.json fails root fmt:check on current main. Left alone deliberately — #36 is active in that package and a format fix there would collide.

bench-results and per-lane comparison shipped in df073b9b46. Acceptance is still unmet, so this stays open.

What landed

The store. --append <store> on the report writes one JSON line per graded run under bench-results/. Each row carries a receipt:<sha256> over its own fields and the receipt of the row before it, so editing a figure in place breaks that row's receipt (receipt_mismatch, naming the row) and inserting or removing a row breaks the next row's link (chain_broken, naming the row). That is what makes the file append-only in practice rather than by convention — a benchmark history nobody can quietly rewrite is worth more than one that merely has not been rewritten yet.

It is a hash chain and not a signature on purpose. A signature answers "who wrote this", which needs a key this tooling has no business holding; the chain answers "has this been rewritten since it was written", which is the question a trend line is actually asking. If a signing seam arrives, it signs the head receipt and the chain still holds underneath.

Two refusals, both reported rather than thrown:

• duplicate_job — the store already holds this Harbor job. Re-scoring a run against different floors is a useful thing to do and a second row is not what it produces; two rows for one execution would double-count it in every trend after.
• chain_broken — appending to a store that does not verify would extend a rewritten history and bury the break a row deeper.

A refused append exits 3, its own code, so a scheduled run whose result never reached the store is not reported as green. It only ever replaces a 0: a breached floor (1) or an unverifiable gate (2) is the more important finding.

The comparison. pnpm run effectiveness:compare -- <store> verifies the chain before comparing anything and refuses a store that does not verify, because a trend over a rewritten history reads exactly like a trend over an honest one. Then two views off the same rows: Trend (one suite, one lane, over time — the shape this issue's acceptance clause asks for, and the shape a regression appears in) and Lanes (one suite across lanes at their latest runs, each against a baseline).

Three rules keep both honest:

1. Only rows sharing a suiteKey are compared — suite, sorted task list, rate catalog. Lane, model, and CLI version are deliberately excluded, because those are the axes a comparison varies. Rows in other groups are reported as their own groups rather than folded into one table with a footnote.
2. A cost delta needs both sides priced. Against an unpriced side the delta is unpriced — not zero, and not "improved". Same refusal the report already makes about a single run, applied to the difference between two.
3. A comparison that varies two things says so. Differing CLI versions, a model that also changed, or a row priced from placeholder rates each land as a named confounder. Nothing is suppressed: a confounded comparison is often the only one available, and it is readable once it is labelled.

The unknowns survive the round trip. A row's cost is null with its disposition and coverage beside it, never 0, for the same reason the report withholds it — gpt-5.6-luna is deliberately unpriced upstream, and a zero in the file would launder an unmeasured lane into a free one at exactly the point where the figure stops being read next to its reason.

Proof. 44 new tests over the same five fixture Harbor jobs, 101 in the package. No model called, no Docker image, clock injected. check:fast green.

bench-results/ ships with its README and no rows. Every row must come from a real Harbor run, and a seeded example would be a fabricated measurement sitting in the file the trend reads.

Still open, and why

• Acceptance. Two consecutive scheduled runs producing comparable rows, and one deliberately introduced regression caught by the floor, need real Harbor runs on amd64 hardware. The store now holds the rows those runs will produce and the fixture cases prove the trend reads a regression as a rise in cost per accepted outcome — but a fixture is not a schedule, and the fixture regression is not the live proof this clause asks for. Per the earlier comment, thirty-six graded container executions is a deliberate spend that wants an explicit go and a choice of lanes, and gemini-3.7-flash is dead upstream (openagents.com#238), so a run today should exclude that lane rather than read its failure as an agent result.
• Suite scope. The floors still point at bench/suites/tb2-cross-section.txt, twelve tasks. The 20-30 task target and the owned set drawn from this tracker's closed issues are not built.
• /api/v3/gym/runs ingest. The report is --json-shaped and the store rows are flat; posting them is not wired.

Acceptance met, and the scale it was met at matters more than the fact that it was, so that comes first.

Be clear about the scale

6 graded trials ran live, all on local Ollama models, --n-concurrent 1, one harbor process at a time. Zero API spend — no budget was approved for the proxy lane, so it was not used.

Not run live: the 20-task coder-effectiveness-v1 suite, all 8 swebench-verified instances, 10 of the 12 terminal-bench cross-section tasks, and all 6 owned tasks. Everything else is exercised by checked-in fixture Harbor jobs.

No cost-per-accepted-outcome number has ever been produced in this repository. Every recorded row is unmetered_local_lane, which is not the same as free — it is unmetered. The machinery, the pins, the smoke rule, and the floor are proven. The headline number this issue exists to produce is not yet measured.

Acceptance, as the issue states it

Three runs of the 2-task tb2-quick suite, live:

Identical suite digest, run digest, model, CLI version, and per-task outcome; trend Δ success rate +0.000 unchanged.

The deliberate regression — lane degraded to qwen3:0.6b, nothing else changed — was caught by the floor: [FAIL] success_rate>=0.500 success rate 0.000 over 2 graded trials, exit 1, trend -100.0% worse with model also varies named as the confounder. The degraded lane is about ten times faster and accepts nothing, so cost per accepted outcome goes to no_accepted_outcomes — the case a per-attempt average would report as an improvement.

Chain verified against the real store: editing the regression row to look like a pass gives receipt_mismatch naming the row, exit 2, nothing compared. Re-scoring a recorded job gives duplicate_job, exit 3, no row written.

What landed

A suite is now a manifest that pins each task by content — dataset, git url, commit, and path for a registry task; issue and closing commit for an owned one — with a task digest over that identity and a suite digest over the sorted task digests plus the tier. A bare list of names was never a digest of anything.

"A fast run is never a published score" is structural rather than documented. classifyRun compares the trials on disk against the pinned list, so a partial run is a smoke run whatever the invocation called itself, and two independent consequences follow that no flag reaches: the store refuses the row, and the gate carries a run_tier=score criterion with no passing branch. Naming no manifest is refused too, because naming the suite is exactly what exposes whether you ran it.

The 20-task suite is the 12-task terminal-bench cross-section plus 8 swebench-verified@1.0 instances, one per repository, chosen by rule — lexicographically first id in each of the eight largest — rather than by hand, so nobody has to wonder whether the picks flatter the coder. swebench-verified stands in for the lite candidate: it is the human-validated subset, which is what belongs under a floor, and it arrives through the same --dataset contract with no new harness code.

The owned set is real and deliberately not runnable yet. Six closed issues (#21, #24, #31, #36, #40, #41) carry a closing_reference commit in their forge evidence, and none has a container that can grade it. Each is marked environmentAvailable: false and a score suite refuses them, because a trial nobody can run reads as the coder failing rather than as a missing environment.

Three faults the live runs exposed, each fixed with a test

The model pin vanished from a trial the timeout killed. Two model ids were reported for one model, because lane detection sniffed the ollama: prefix out of the model id — so a trial that spelled it bare priced as unknown_model, reading as a gap in the rate catalog when it was a lane with no rates to have; priceUsage now takes the lane off the run. And the comparison ended a report with "1 run shape was left out" directly beneath the trend over exactly those rows.

Fixing the second changed two lane-comparison tests that had asserted a priced cost delta between a proxy row and a local row — only reachable because a fixture pretended a local lane could be priced. The proxy-versus-local comparison this issue asks for has a permanently unstatable cost delta in one direction, and now says so.

142 tests over 8 files, single-worker.

Remaining

The owned environments; a priced-lane run to make cost real; /api/v3/gym/runs ingest; and the per-release and weekly cadence.