bench: record the tb2-cross-section proxy-lane baseline, gate unverifiable

8909d2eb5286 · AtlantisPleb · 2d ago · parent e753d0b60f46

bench: record the tb2-cross-section proxy-lane baseline, gate unverifiable

Opens bench-results/tb2-cross-section.jsonl. Suite tb2-cross-section on lane
proxy with model gpt-5.6-luna: 9 of 12 accepted, success rate 0.750 over 12
graded trials, 0 ungraded, 12 of 12 pinned tasks covered, wall clock 3151.4s,
6,006,329 prompt and 88,163 output tokens over 250 tool calls. Rejected:
count-dataset-tokens, fix-code-vulnerability, sanitize-git-repo.

The gate is UNVERIFIABLE and the report exits 2. Four of five criteria pass:
graded_trials 12>=8, success_rate 0.750>=0.250, ungraded_ratio 0.000<=0.200,
run_tier=score. The fifth, maxCostPerAcceptedOutcomeUsd 2.0, cannot be
measured because gpt-5.6-luna is unpriced in the forge catalog, and
unverifiable beats passed when the gate folds. A fully covered, fully graded
run at 75% success therefore cannot exit 0.

That is #125, filed before this run as a prediction from the code and now
confirmed by a completed run. The threshold was NOT edited to make this run
pass (best practice M4); the exit code is recorded as it fell.

Cost is null (cost_unknown): 0 of 12 trials could be priced.

Harbor job 0d0fc552-ea21-4934-9f25-0d30fe11f3f3, Gym run
e3b9503d-95bc-4200-9a65-c580d6bce9c9 (graded, 9/12), CLI 0.4.0.

An earlier attempt the same day reached 1 of 12 and was recorded abandoned
when Docker Desktop failed to start mid-suite; it is on the Gym only and was
never scored.

Refs #118, #125

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K7q2vA5LJroLTR6ZFbRq6j

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K7q2vA5LJroLTR6ZFbRq6j

Deploy story

What this commit did to the running system — joined from the forge receipt chain, the part a commit page elsewhere cannot show.

pushed: by user · WAL seq 265 · 2026-08-26T15:03:18.579961Z

Changed files

added bench-results/tb2-cross-section.jsonl
modified docs/coder/runbook.md

Diff

2 files changed, +18 -9

{"schema":"openagents.bench_result.v2","recordedAt":"2026-08-26T15:01:37.786Z","suite":"tb2-cross-section","lane":"proxy","runDigest":"effectiveness:84f0f813b86dbc96c27e268d43c59dddc470a7ae7c06397415249b9ca2522a15","suiteKey":"suite:f1083887d9b3fb18c0b5b95a0eb535d3b5e47ce1b5d2e6e42c3587877e86a3f9","jobId":"0d0fc552-ea21-4934-9f25-0d30fe11f3f3","suiteId":"tb2-cross-section","suiteDigest":"suite-manifest:eeb336e0eebbc94ff18ba59d5a373522c1b30f69e1c84dd255a14c26776f0bc9","tier":"score","models":["gpt-5.6-luna"],"agentVersions":["0.4.0"],"rateCatalogVersion":"openagents.coder-rate-catalog.2026-08-25","tasks":["build-cython-ext","count-dataset-tokens","fix-code-vulnerability","git-leak-recovery","merge-diff-arc-agi-task","nginx-request-logging","openssl-selfsigned-cert","password-recovery","regex-log","sanitize-git-repo","schemelike-metacircular-eval","sqlite-with-gcov"],"trialsTotal":12,"accepted":9,"rejected":3,"ungraded":0,"graded":12,"successRate":0.75,"ungradedRatio":0,"costPerAcceptedOutcomeUsd":null,"costDisposition":"cost_unknown","totalCostUsd":null,"costCoverage":"unknown","rateBasis":null,"promptTokens":6006329,"completionTokens":88163,"cachedInputTokens":0,"toolCalls":250,"wallClockSeconds":3151.3969999999995,"gateStatus":"unverifiable","thresholdsId":"tb2-cross-section","previousReceipt":null,"receipt":"receipt:b54e863eb7abde8fd7ef2178711218c79b6af8ee01b669de54e46a806113a780"}

  trials, 221.9s, gate passed (2026-08-26, #118). The proxy row is the
  iteration baseline; cost is `null` because `gpt-5.6-luna` is unpriced.
  No cross-section rows recorded on any lane yet.
- `tb2-cross-section` has not been recorded. The 2026-08-26 proxy attempt
  reached 1 of 12 tasks and was recorded `abandoned` when Docker Desktop
  failed to start mid-suite (the host volume was full); the local-lane
  cross-section run was never started. Both remain open on #118.
- The `tb2-cross-section` thresholds declare
  `maxCostPerAcceptedOutcomeUsd: 2.0`, but `gpt-5.6-luna` is unpriced, so
  that criterion is unverifiable and the whole gate exits 2 on every proxy
  run. Filed as #125 rather than edited alongside a run it would flatter
  (best practice M4).
- `tb2-cross-section`, proxy lane, `gpt-5.6-luna`: 9 of 12 accepted, 0.75
  success over 12 graded trials, 0 ungraded, 12 of 12 pinned tasks covered,
  3151.4s, 6,006,329 prompt and 88,163 output tokens over 250 tool calls
  (2026-08-26, #118). Rejected: `count-dataset-tokens`,
  `fix-code-vulnerability`, `sanitize-git-repo`. An earlier attempt the same
  day reached 1 of 12 and was recorded `abandoned` when Docker Desktop failed
  to start mid-suite (the host volume was full); that row is not in the store,
  only on the Gym.
- **That run's gate is `unverifiable`, not passed, and the report exits 2.**
  Four of five criteria pass. `maxCostPerAcceptedOutcomeUsd: 2.0` cannot be
  measured because `gpt-5.6-luna` is unpriced, and `failed` beats
  `unverifiable` beats `passed`, so a fully covered, fully graded, healthy
  run still cannot exit 0. This is #125, now confirmed by a completed run
  rather than predicted from the code. The threshold was not edited to make
  the run pass (best practice M4).
- No `tb2-cross-section` row on the local lane yet, so the store holds one
  run shape and `effectiveness:compare` has nothing to compare it against.
- Rosetta note, measured 2026-08-26: on this rig the verifier's `uv`/
  `pytest` does **not** segfault under emulation. `uv 0.9.5` plus
  `uvx pytest` returned `pytest 9.1.1` in an amd64 container with

270	270		`trials, 221.9s, gate passed (2026-08-26, #118). The proxy row is the`
271	271		iteration baseline; cost is `null` because `gpt-5.6-luna` is unpriced.
272	272		`No cross-section rows recorded on any lane yet.`
273		-	- `tb2-cross-section` has not been recorded. The 2026-08-26 proxy attempt
274		-	reached 1 of 12 tasks and was recorded `abandoned` when Docker Desktop
275		-	`failed to start mid-suite (the host volume was full); the local-lane`
276		-	`cross-section run was never started. Both remain open on #118.`
277		-	- The `tb2-cross-section` thresholds declare
278		-	`maxCostPerAcceptedOutcomeUsd: 2.0`, but `gpt-5.6-luna` is unpriced, so
279		-	`that criterion is unverifiable and the whole gate exits 2 on every proxy`
280		-	`run. Filed as #125 rather than edited alongside a run it would flatter`
281		-	`(best practice M4).`
	273	+	- `tb2-cross-section`, proxy lane, `gpt-5.6-luna`: 9 of 12 accepted, 0.75
	274	+	`success over 12 graded trials, 0 ungraded, 12 of 12 pinned tasks covered,`
	275	+	`3151.4s, 6,006,329 prompt and 88,163 output tokens over 250 tool calls`
	276	+	(2026-08-26, #118). Rejected: `count-dataset-tokens`,
	277	+	`fix-code-vulnerability`, `sanitize-git-repo`. An earlier attempt the same
	278	+	day reached 1 of 12 and was recorded `abandoned` when Docker Desktop failed
	279	+	`to start mid-suite (the host volume was full); that row is not in the store,`
	280	+	`only on the Gym.`
	281	+	- That run's gate is `unverifiable`, not passed, and the report exits 2.
	282	+	Four of five criteria pass. `maxCostPerAcceptedOutcomeUsd: 2.0` cannot be
	283	+	measured because `gpt-5.6-luna` is unpriced, and `failed` beats
	284	+	`unverifiable` beats `passed`, so a fully covered, fully graded, healthy
	285	+	`run still cannot exit 0. This is #125, now confirmed by a completed run`
	286	+	`rather than predicted from the code. The threshold was not edited to make`
	287	+	`the run pass (best practice M4).`
	288	+	- No `tb2-cross-section` row on the local lane yet, so the store holds one
	289	+	run shape and `effectiveness:compare` has nothing to compare it against.
282	290		- Rosetta note, measured 2026-08-26: on this rig the verifier's `uv`/
283	291		`pytest` does not segfault under emulation. `uv 0.9.5` plus
284	292		`uvx pytest` returned `pytest 9.1.1` in an amd64 container with