Point the runner and its docs at the pinned manifests

Point the runner and its docs at the pinned manifests

`bench/run-suite.sh` now takes a `*.suite.json` and reads its task ids, so the
file that pins the suite is the file that runs it — the alternative is a task
list and a manifest that agree only by hand. The plain `.txt` lists still work
and the runner says what they cost you: no pin, so no recordable score.

After a suite run it prints the report command with `--suite-manifest` already
filled in, rather than running a gate the operator did not ask for. A suite run
that exited non-zero for a floor nobody set would be a surprising thing for a
runner to do.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SoZMfWRSGnf6FZX2Ar9rQ2

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SoZMfWRSGnf6FZX2Ar9rQ2

Deploy story

What this commit did to the running system — joined from the forge receipt chain, the part a commit page elsewhere cannot show.

Not deployed through the forge lane

No push, promotion, build, or deploy receipt references this commit (receipts are scanned over a bounded recent window). Changes shipped by full node replacement carry their proof in the release gate receipt instead.

Changed files

modified bench/README.md
modified bench/run-suite.sh
modified packages/coder-effectiveness/README.md
modified packages/coder-effectiveness/src/cli.ts

Diff

4 files changed, +28 -7

Examples:

```sh
bench/run-suite.sh bench/suites/tb2-cross-section.txt \
bench/run-suite.sh bench/suites/tb2-cross-section.suite.json \
  --model openai/gpt-5.6-luna --lane proxy --n-concurrent 2

bench/run-suite.sh bench/suites/tb2-cross-section.txt \
bench/run-suite.sh bench/suites/tb2-cross-section.suite.json \
  --model ollama/qwen3.8:27b-mtp-q8_0 --lane local --dry-run
```

## Suites

`bench/suites/*.suite.json` are the pinned suites, regenerated from Harbor's
registry and this tracker's closed issues by:

```sh
pnpm run effectiveness:suites -- \
  --registry ../projects/repos/harbor/registry.json \
  --issues <(openagents issue list -R OpenAgentsInc/openagents --state closed --limit 200 --json)
```

Add `--check` to rebuild and diff without writing, which is how a manifest is
kept from drifting away from the registry it claims to pin.

A manifest pins each task by content — dataset, git url, commit, path — rather
than by name, and `coder-effectiveness report --suite-manifest` scores a run
against it. A run that did not cover every pinned task is a smoke run and
cannot be recorded, whatever it was invoked as; see
`packages/coder-effectiveness/README.md`.

The plain `.txt` lists still run. They carry no pin, so a run of one cannot be
recorded as a score.

Options include `--lane`, `--api-url`, `--jobs-dir`, `--n-concurrent`,
`--timeout-multiplier`, and `--dry-run`. Set `OPENAGENTS_TOKEN` unless the
model starts with `ollama/`.

--run-id. A suite that fails before grading patches the run to abandoned.

Examples:
  bench/run-suite.sh bench/suites/tb2-cross-section.txt \
  bench/run-suite.sh bench/suites/tb2-cross-section.suite.json \
    --model openai/gpt-5.6-luna --lane proxy --n-concurrent 2

  bench/run-suite.sh bench/suites/local-llm.txt \

  comparison now hold the rows those runs will produce, and the fixture cases
  prove the trend reads a regression as a rise in cost per accepted outcome —
  but a fixture is not a schedule, and no real run has been recorded.
- **The 20-30 task suite.** The floors point at
  `bench/suites/tb2-cross-section.txt`, twelve tasks. The wider suite and the
  owned set drawn from this tracker's closed issues are not built.

 * directory. This command reads that directory and answers the question the
 * run was for.
 *
 *     bench/run-suite.sh bench/suites/tb2-cross-section.txt \
 *     bench/run-suite.sh bench/suites/tb2-cross-section.suite.json \
 *       --model openai/gpt-5.6-luna --jobs-dir /tmp/gym-jobs-run
 *
 *     pnpm run effectiveness:report -- /tmp/gym-jobs-run/<job> \
 *       --suite tb2-cross-section --lane proxy \
 *       --suite-manifest bench/suites/tb2-cross-section.suite.json \
 *       --thresholds packages/coder-effectiveness/thresholds/tb2-cross-section.json
 *
 * EXIT CODES. 0 the gate passed, 1 a floor was breached, 2 the gate could not

56	56		`Examples:`
57	57
58	58		```sh
59		-	`bench/run-suite.sh bench/suites/tb2-cross-section.txt \`
	59	+	`bench/run-suite.sh bench/suites/tb2-cross-section.suite.json \`
60	60		`--model openai/gpt-5.6-luna --lane proxy --n-concurrent 2`
61	61
62		-	`bench/run-suite.sh bench/suites/tb2-cross-section.txt \`
	62	+	`bench/run-suite.sh bench/suites/tb2-cross-section.suite.json \`
63	63		`--model ollama/qwen3.8:27b-mtp-q8_0 --lane local --dry-run`
64	64		```
65	65
	66	+	`## Suites`
	67	+
	68	+	`bench/suites/*.suite.json` are the pinned suites, regenerated from Harbor's
	69	+	`registry and this tracker's closed issues by:`
	70	+
	71	+	```sh
	72	+	`pnpm run effectiveness:suites -- \`
	73	+	`--registry ../projects/repos/harbor/registry.json \`
	74	+	`--issues <(openagents issue list -R OpenAgentsInc/openagents --state closed --limit 200 --json)`
	75	+	```
	76	+
	77	+	Add `--check` to rebuild and diff without writing, which is how a manifest is
	78	+	`kept from drifting away from the registry it claims to pin.`
	79	+
	80	+	`A manifest pins each task by content — dataset, git url, commit, path — rather`
	81	+	than by name, and `coder-effectiveness report --suite-manifest` scores a run
	82	+	`against it. A run that did not cover every pinned task is a smoke run and`
	83	+	`cannot be recorded, whatever it was invoked as; see`
	84	+	`packages/coder-effectiveness/README.md`.
	85	+
	86	+	The plain `.txt` lists still run. They carry no pin, so a run of one cannot be
	87	+	`recorded as a score.`
	88	+
66	89		Options include `--lane`, `--api-url`, `--jobs-dir`, `--n-concurrent`,
67	90		`--timeout-multiplier`, and `--dry-run`. Set `OPENAGENTS_TOKEN` unless the
68	91		model starts with `ollama/`.

60	60		`--run-id. A suite that fails before grading patches the run to abandoned.`
61	61
62	62		`Examples:`
63		-	`bench/run-suite.sh bench/suites/tb2-cross-section.txt \`
	63	+	`bench/run-suite.sh bench/suites/tb2-cross-section.suite.json \`
64	64		`--model openai/gpt-5.6-luna --lane proxy --n-concurrent 2`
65	65
66	66		`bench/run-suite.sh bench/suites/local-llm.txt \`

319	319		`comparison now hold the rows those runs will produce, and the fixture cases`
320	320		`prove the trend reads a regression as a rise in cost per accepted outcome —`
321	321		`but a fixture is not a schedule, and no real run has been recorded.`
322		-	`- The 20-30 task suite. The floors point at`
323		-	`bench/suites/tb2-cross-section.txt`, twelve tasks. The wider suite and the
324		-	`owned set drawn from this tracker's closed issues are not built.`

6	6		`* directory. This command reads that directory and answers the question the`
7	7		`* run was for.`
8	8		`*`
9		-	`* bench/run-suite.sh bench/suites/tb2-cross-section.txt \`
	9	+	`* bench/run-suite.sh bench/suites/tb2-cross-section.suite.json \`
10	10		`* --model openai/gpt-5.6-luna --jobs-dir /tmp/gym-jobs-run`
11	11		`*`
12	12		`* pnpm run effectiveness:report -- /tmp/gym-jobs-run/<job> \`
13	13		`* --suite tb2-cross-section --lane proxy \`
	14	+	`* --suite-manifest bench/suites/tb2-cross-section.suite.json \`
14	15		`* --thresholds packages/coder-effectiveness/thresholds/tb2-cross-section.json`
15	16		`*`
16	17		`* EXIT CODES. 0 the gate passed, 1 a floor was breached, 2 the gate could not`