Delete the task list the manifest replaced

db4a30252be5 · Claude Fable 5 · · parent f86647b4d376

Delete the task list the manifest replaced

`bench/suites/tb2-cross-section.txt` and its manifest named the same twelve
tasks, and nothing kept them agreeing. One of them pins each task to a dataset
commit and one is a list of names, so the list can only ever be the copy that
goes stale — and a suite that drifted would still run, still digest, and still
classify a run over the wrong tasks as full coverage.

The `.md` beside it keeps the selection rationale; its loop-form example now
reads task ids out of the manifest, and the runner's help points its local-lane
example at a suite that exists.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SoZMfWRSGnf6FZX2Ar9rQ2
Co-Authored-By
Claude Fable 5 <noreply@anthropic.com>

Deploy story

What this commit did to the running system — joined from the forge receipt chain, the part a commit page elsewhere cannot show.

Not deployed through the forge lane

No push, promotion, build, or deploy receipt references this commit (receipts are scanned over a bounded recent window). Changes shipped by full node replacement carry their proof in the release gate receipt instead.

Changed files

  • modified bench/run-suite.sh
  • modified bench/suites/tb2-cross-section.md
  • deleted bench/suites/tb2-cross-section.txt

Diff

3 files changed, +3 -23

bench/run-suite.sh modified +1 -1

@@ -63,7 +63,7 @@ Examples:

63 63
  bench/run-suite.sh bench/suites/tb2-cross-section.suite.json \
64 64
    --model openai/gpt-5.6-luna --lane proxy --n-concurrent 2
65 65
66
  bench/run-suite.sh bench/suites/local-llm.txt \
66
  bench/run-suite.sh bench/suites/tb2-quick.suite.json \
67 67
    --model ollama/qwen3.8:27b-mtp-q8_0 --lane local --dry-run
68 68
EOF
69 69
}
bench/suites/tb2-cross-section.md modified +2 -1

@@ -143,7 +143,8 @@ machine-readable list; also the fallback if a future Harbor version

143 143
changes `-i` semantics):
144 144
145 145
```sh
146
grep -v '^#' bench/suites/tb2-cross-section.txt | while read -r task; do
146
python3 -c 'import json;[print(t["id"]) for t in json.load(open("bench/suites/tb2-cross-section.suite.json"))["tasks"]]' |
147
  while read -r task; do
147 148
  [ -z "$task" ] && continue
148 149
  PYTHONPATH=bench OPENAGENTS_TOKEN=... harbor run \
149 150
    --dataset terminal-bench@2.0 \
bench/suites/tb2-cross-section.txt deleted -21

@@ -1,21 +0,0 @@

1
# Terminal-Bench 2.0 cross-section suite (see tb2-cross-section.md)
2
# One task name per line; lines starting with # are comments.
3
# version control / forensics
4
git-leak-recovery
5
sanitize-git-repo
6
merge-diff-arc-agi-task
7
# build / compile
8
build-cython-ext
9
sqlite-with-gcov
10
# debugging / security code-fix
11
fix-code-vulnerability
12
# data processing
13
regex-log
14
count-dataset-tokens
15
# security / forensics / sysadmin
16
password-recovery
17
openssl-selfsigned-cert
18
# networking / service configuration
19
nginx-request-logging
20
# long-horizon
21
schemelike-metacircular-eval

This page updates live while a promote is in flight · changelog