Teach the coder's tools token economy, per model family #36

Closed AtlantisPleb opened this 2d ago 4 comments

Outcome

The coder's tool declarations carry efficiency guidance the way Gemini CLI's do, resolved per model family, and the Gym proves the delta.

Evidence

The first graded fix-git runs (OpenAgentsInc/openagents.com docs/terminalbench/2026-08-24-fix-git-run-analysis.md): gemini-3.7-flash did the same job as gpt-5.6-luna in 15 tool rounds instead of 6 and spent 3x the input tokens, because each round re-sends the growing transcript and it favored one tiny command per round plus full git log -p dumps. Batching guidance in the system prompt was not enough for that family. Gemini's own harness fixes this inside the declarations: efficiency guidelines in the shell tool (quiet flags, --no-pager), "surgical reads are mandatory / truncation is token-inefficient" in read-range descriptions, and a ~30-line base-plus-overrides resolver keyed by model (projects/repos/gemini-cli packages/core/src/tools/definitions/{resolver.ts,model-family-sets/}).

Contract

  • Shell tool description gains efficiency language: batch independent commands into one call, quiet/--no-pager flags, --stat before -p for git. Applies to every family.
  • A small declaration resolver: base description plus per-family overrides (the coder knows its model from the grant). First overrides: batching emphasis for gemini-family; brevity emphasis for the local lane, where generation latency, not tokens, is the cost.
  • Lane-aware system-prompt economics (LOCAL_LANE vs THREAD_LANE already exist as seams).
  • Adapt the pattern; if any Gemini CLI code is taken it is Apache-2.0 with attribution.
  • Acceptance: the same fix-git (or #34 suite) run per family shows reduced rounds/input tokens for gemini-family with no pass-rate regression, recorded in the Gym.
  1. AtlantisPleb opened this issue 2d ago
  2. A AtlantisPleb Author 2d ago

    First slice landed on main as 8ff2060e7c (WAL receipt seq 101) and the Gym oracle already answered. Shipped: efficiency guidance in the shell tool's base description (batch with &&, quiet flags, git --no-pager, --stat before -p — applies to every family), coder-tool-families.ts resolving a family from the model or lane with data-shaped description overrides (batching emphasis for gemini, latency-awareness for the local lane), both lanes declaring through the resolver, ATIF recording what was actually declared, and lane sentences carrying their economics.

    The before/after, same task (terminal-bench@2.0 fix-git), same model (gemini-3.7-flash), same lane:

    Before: 15 tool rounds, 124,941 input / 640 output tokens, pass.
    After: 9 tool rounds, 40,973 input / 464 output tokens, pass — a 67 percent input-token reduction.

    The mechanism visibly matches the intervention, not luck: the after-run's calls are composite (git status && branch && reflog && stash in one; diff && log -p -n 1 && show file in one), it reached for --stat before any full patch, and it resolved the conflict with git checkout --theirs in the correct direction. One trial per condition — an existence proof of the lever, not a rate claim; #34's suite is where this becomes statistics. Remaining on this issue: overrides for further families as the suite measures need, and read-range/grep-style economy language if and when those become native tools.

  3. A AtlantisPleb Author 2d ago

    The local-family oracle answered too, and its delta is larger than gemini's. Same task (fix-git), same local model (qwen3.8:27b via Ollama), before and after the latency-awareness emphasis and lane economics:

    Before: 19 steps, ~10 tool calls, 110,061 in / 17,160 out, about 18.5 minutes of agent wall time.
    After: 16 steps, 7 tool calls, 37,295 in / 1,792 out, 69 seconds — pass.

    That is a 16x wall-time reduction on the lane where wall time IS the cost, a 90 percent output-token cut (the verbosity emphasis landing), and a 66 percent input cut from fewer, tighter rounds over a smaller transcript. The after-run's behavior matches the teaching: recon in two composite calls including git fsck --lost-found, --stat before the full show, one verification pass instead of several, and the same correct conflict resolution. Same caveat as the gemini oracle: one trial per condition, existence proof not statistics; #34's suite is the rate-maker.

  4. A AtlantisPleb Author 2d ago

    Code half shipped: packages/openagents-cli/src/coder-tool-families.ts carries the base-plus-per-family declaration resolver (867d7101d5 and the slice before it), with batching emphasis for the gemini family and brevity emphasis for the local lane, plus general ranged-read guidance. Lane-aware system-prompt economics landed alongside it. Unit tests cover the resolver, and an integration test pins that ThreadReplySource.toolDefinitions() really carries the family emphasis.

    Measured before/after on the fix-git oracle when the first slice landed: gemini-3.7-flash went from 124,941 input tokens over 15 rounds to 40,973 over 9 (−67%); the local qwen lane went from 18.5 minutes to 69 seconds with output tokens 17,160 → 1,792; gpt-5.6-luna 43,894 → 24,950 (−43%). All still passed.

    What remains for this issue's stated acceptance is the suite-level proof rather than the single-task oracle — the same fix-git or #34 suite run per family with no pass-rate regression, recorded in the Gym. That waits on #34, which is blocked. Leaving open for it.

    Contamination note, since the Gym measures these declarations: the shipped guidance is general efficiency language only (batch independent commands, quiet flags and pagers, ranged reads over whole-file dumps, lane latency). It names no benchmark, task, git subcommand, or file that any verifier reads — audited and recorded in docs/terminalbench/2026-08-24-fix-git-run-analysis.md section 7.

  5. AtlantisPleb closed this as completed in 2a631cf 1d ago
  6. A AtlantisPleb Author 1d ago

    Landed in 2a631cf631.

    Tool results are now budgeted per model family on the way to the model, and every cut says so.

    Before: one constant, TOOL_RESULT_KEPT = 4_000 characters, written twice — once in coder-thread.ts for the hosted lane, once in coder-ollama.ts for the local one — applied identically to a 32k local model and a million-token hosted one, with nothing said about what those characters cost the family reading them. Three tool-level caps above it ended an output without saying how much went: the shell tool's [truncated; …], the CLI tool's [The output was cut off here…], and delegate's …[truncated], which said nothing at all. The shell and CLI collectors discarded the overflow unrecorded, so the amount was not even knowable.

    After: packages/openagents-cli/src/coder-tool-budget.ts holds one budget per ToolFamily, stated in tokens against that family's context window and converted to characters through that family's own characters-per-token figure. Nothing tokenizes — a tokenizer per family is a download for a decision that ends in a character count either way — so each density is documented as an approximation, held low so an error underspends the window, and each row carries why its allowance is what it is. Both non-default budgets are tighter than the 4,000 they replace: gemini (2,800) because its measured cost is round count and re-sent whole-file dumps, local (1,750) because its window is a fraction of a hosted one and every re-sent character is paid in wall clock. default (3,960) holds the shipped figure. The table is exhaustive over the family union, so adding a family without deciding what it may spend does not compile; a family arriving from data with no row falls back to the smallest budget and the notice says it was substituted.

    Visibility: a cut result now names how much went, out of what, and which family's budget cut it, plus that it must not be summarized as the whole answer. The shell and CLI tools count what their collectors refuse instead of dropping it, so their notices carry real totals, and a clipped child answer names its own missing half. This is the fail-closed limit discipline in INVARIANTS.md rather than a new rule.

    Tests: test/coder-tool-budget.test.ts proves the same 20,000-character result is budgeted differently per family and that each reports its cut, plus the substituted-budget path; coder-thread.test.ts proves it end to end through a real session on gpt-5.6-luna versus gemini-3.7-flash; coder-resume.test.ts proves a replayed transcript is budgeted by the resumed session's family; coder-shell.test.ts and coder-tools.test.ts cover the two tool-level notices. 866 tests pass in the package; check:fast and the assure-repo generate/audit pair are green.

    The declaration resolver half of this issue (coder-tool-families.ts, shell efficiency language, gemini/local overrides) was already in place; this adds the token economy the resolver was missing. The Gym delta the contract asks for is #34's suite.

Sign in with GitHub to comment on this issue.