Budget a tool result by the model family reading it

2a631cf63107 · AtlantisPleb · · parent cff5e4b62f0f

Budget a tool result by the model family reading it

One number bounded every tool result on the way to a model: 4,000
characters, written twice — once in the thread lane, once in the local
lane — and applied to a 32k local model and a million-token hosted one
alike. It could not be right for both, and it said nothing about what
those characters cost the family that had to read them.

The budget is now per family and stated in tokens, converted to
characters through that family's own approximate density. Nothing
tokenizes: a tokenizer per family is a download for a decision that ends
in a character count either way, so each figure is documented as an
approximation, held low so an error underspends the window, and carries
the reason it is what it is. Both non-default budgets are tighter than
the number they replace — gemini because its measured cost is the round
count and the whole-file dumps it re-sends, local because its window is
a fraction of a hosted one and every re-sent character is paid in wall
clock.

A cut result now says it was cut, by how much, out of what, and against
which family's budget. So do the three tool-level caps that used to end
an output mid-line: the shell tool and the CLI tool count what the
collector refused instead of dropping it unrecorded, and a child's
clipped answer names its own missing half rather than trailing off in
"…[truncated]". A model handed half a `git log` with nothing said reads
it as the whole log, which is what INVARIANTS.md forbids a bounded
mechanism from allowing.

An unrecognized family falls back to the smallest budget on the table
and says it was substituted; the table is exhaustive over the family
union, so adding a family without deciding what it may spend does not
compile.

Closes #36.
Closes
#36

Deploy story

What this commit did to the running system — joined from the forge receipt chain, the part a commit page elsewhere cannot show.

Not deployed through the forge lane

No push, promotion, build, or deploy receipt references this commit (receipts are scanned over a bounded recent window). Changes shipped by full node replacement carry their proof in the release gate receipt instead.

Changed files

  • modified docs/assure-repo/false-green-candidates.v1.json
  • modified docs/assure-repo/surface-inventory.v1.json
  • modified packages/openagents-cli/src/cli.ts
  • modified packages/openagents-cli/src/coder-ollama.ts
  • modified packages/openagents-cli/src/coder-resume.ts
  • modified packages/openagents-cli/src/coder-self-harness.ts
  • modified packages/openagents-cli/src/coder-shell.ts
  • modified packages/openagents-cli/src/coder-thread.ts
  • added packages/openagents-cli/src/coder-tool-budget.ts
  • modified packages/openagents-cli/src/coder-tools.ts
  • modified packages/openagents-cli/src/coder-zen.ts
  • modified packages/openagents-cli/test/coder-resume.test.ts
  • modified packages/openagents-cli/test/coder-shell.test.ts
  • modified packages/openagents-cli/test/coder-thread.test.ts
  • added packages/openagents-cli/test/coder-tool-budget.test.ts
  • modified packages/openagents-cli/test/coder-tools.test.ts

Diff

16 files changed, +490 -90

docs/assure-repo/false-green-candidates.v1.json modified +1 -1

@@ -4,7 +4,7 @@

4 4
  "note": "Heuristic false-green LEADS, not findings. A finding requires a demonstrated reproduction (surviving mutation via mutation-runner). Do not treat a candidate as a confirmed false green. Coverage-theater leads may include tests that delegate their assertion to a custom helper the classifier does not recognise; verify before acting.",
5 5
  "sourceDigest": "sha256:dd810dd48c5bdbc9becd7fcc01dd41a4ca2abf0b2d6f6a545907247f6e3e8361",
6 6
  "summary": {
7
    "filesScanned": 2473,
7
    "filesScanned": 2474,
8 8
    "candidateCount": 16,
9 9
    "byMode": {
10 10
      "false_green_coverage_theater": 15,
docs/assure-repo/surface-inventory.v1.json modified +2 -2

@@ -1,7 +1,7 @@

1 1
{
2 2
  "schemaVersion": "1",
3 3
  "repository": "OpenAgentsInc/openagents",
4
  "sourceDigest": "sha256:d9d93b65d7614d382ef47b625cf1b305e5f4a907d8103d09dd0951d49b3d9ff0",
4
  "sourceDigest": "sha256:9c762fe37db50019a875a89c30c5bcb2a21ebc1d5e3d8ab94ea8eaa538bc7b0f",
5 5
  "surfaces": [
6 6
    {
7 7
      "id": "app:@openagentsinc/acceptance-runner",

@@ -1876,7 +1876,7 @@

1876 1876
      "oracles": [
1877 1877
        {
1878 1878
          "type": "test",
1879
          "ref": "packages/openagents-cli (68 tracked test files)"
1879
          "ref": "packages/openagents-cli (69 tracked test files)"
1880 1880
        },
1881 1881
        {
1882 1882
          "type": "behavior-contract",
packages/openagents-cli/src/cli.ts modified +2 -1

@@ -87,6 +87,7 @@ import {

87 87
  resumableThreads,
88 88
  type ThreadSummary,
89 89
} from "./coder-resume.js";
90
import { toolFamilyOf } from "./coder-tool-families.js";
90 91
import { openLocalThread, threadAnnouncement, threadSyncWanted } from "./coder-local-thread.js";
91 92
import { ThreadTranscriptWriter } from "./coder-transcript.js";
92 93
import { delegateTool, openagentsTool, shellTool, skillTool } from "./coder-tools.js";

@@ -2156,7 +2157,7 @@ const coderCommand = Command.make(

2156 2157
              // The replayed history reaches the model transcript and the
2157 2158
              // interface, never the transcript writer: the server already
2158 2159
              // holds these events, and a resume must not post them twice.
2159
              source.preload(replayWire(events));
2160
              source.preload(replayWire(events, toolFamilyOf(source.model)));
2160 2161
              return { source, entries: replayEntries(events) };
2161 2162
            },
2162 2163
            catch: (cause) => coderRefusal(endpoint.origin, cause, "coder.thread.resume"),
packages/openagents-cli/src/coder-ollama.ts modified +15 -20

@@ -18,6 +18,7 @@ import type {

18 18
  ToolCall as OllamaToolCall,
19 19
} from "ollama";
20 20
21
import { budgetedResult } from "./coder-tool-budget.js";
21 22
import { declaredDescription } from "./coder-tool-families.js";
22 23
import { merge } from "./coder-merge.js";
23 24
import type { ReplyChunk, ReplySource } from "./coder-session.js";

@@ -39,35 +40,23 @@ const DEFAULT_HOST = "http://127.0.0.1:11434";

39 40
 */
40 41
const MAX_TOOL_STEPS = 100;
41 42
42
/**
43
 * How much of one tool's output is kept on the transcript.
44
 *
45
 * The reader sees all of it; this is only what goes back to the model on every
46
 * round after. A session that read the issue boards accumulated 82 KB of tool
47
 * output and re-sent it 25 times, which is where its wall clock went: 91% of
48
 * everything sent each round was output the model had already read.
49
 *
50
 * Generous enough that a normal command survives whole, and the head and the
51
 * tail are what a long one is read for anyway.
52
 */
53
const TOOL_RESULT_KEPT = 4_000;
54
55 43
/**
56 44
 * How much of one tool's output reaches the durable `tool.ran` event.
57 45
 *
58
 * The same figure the thread lane uses, for the same reason: the 4,000 above
59
 * is a context-budget decision re-spent on every round, and this bounds a
60
 * record written once, so it is set where every result a real session has
61
 * produced fits whole.
46
 * What the model is fed is bounded elsewhere and differently: this lane's
47
 * results are budgeted for the `local` family in `coder-tool-budget.ts`,
48
 * because they are re-sent on every round and a local model's window is a
49
 * fraction of a hosted one. This figure bounds a record written once, so it is
50
 * set where every result a real session has produced fits whole.
62 51
 */
63 52
const EVENT_RESULT_KEPT = 64_000;
64 53
65 54
/** A long tool result, kept at both ends. */
66
const bounded = (output: string, keep = TOOL_RESULT_KEPT): string => {
55
const bounded = (output: string, keep: number): string => {
67 56
  if (output.length <= keep) return output;
68 57
  const half = Math.floor(keep / 2);
69 58
  const cut = output.length - keep;
70
  return `${output.slice(0, half)}\n\n[${String(cut)} characters omitted from the middle; run it again more narrowly if you need them]\n\n${output.slice(-half)}`;
59
  return `${output.slice(0, half)}\n\n[${String(cut)} of ${String(output.length)} characters omitted from the middle; run it again more narrowly if you need them]\n\n${output.slice(-half)}`;
71 60
};
72 61
73 62
/**

@@ -643,6 +632,12 @@ export class OllamaReplySource implements ReplySource {

643 632
        : { error: bounded(failure, EVENT_RESULT_KEPT) }),
644 633
    });
645 634
646
    this.transcript.push({ role: "tool", content: bounded(output), tool_name: name });
635
    // Budgeted for the local family: a small window and slow generation make
636
    // every re-sent character expensive in wall clock (coder-tool-budget.ts).
637
    this.transcript.push({
638
      role: "tool",
639
      content: budgetedResult(output, "local"),
640
      tool_name: name,
641
    });
647 642
  }
648 643
}
packages/openagents-cli/src/coder-resume.ts modified +13 -7

@@ -38,7 +38,9 @@

38 38
import { createInterface } from "node:readline";
39 39
40 40
import type { CoderEntry } from "./coder-session.js";
41
import { boundedResult, ThreadUnavailable, type WireMessage } from "./coder-thread.js";
41
import { budgetedResult } from "./coder-tool-budget.js";
42
import type { ToolFamily } from "./coder-tool-families.js";
43
import { ThreadUnavailable, type WireMessage } from "./coder-thread.js";
42 44
import { THREADS_PATH } from "./constants.js";
43 45
44 46
/** The server's listing cap. Pages are read at exactly this size. */

@@ -268,10 +270,11 @@ export function replayEntries(events: ReadonlyArray<ThreadEvent>): ReadonlyArray

268 270
 * what reached the wire. `tool.ran` becomes the standard chat exchange: an
269 271
 * assistant message carrying the call in `tool_calls`, with the arguments as
270 272
 * the raw JSON string the record kept, then a `tool` message named by
271
 * `tool_call_id` with the result bounded by the same figure the live loop
272
 * uses, because this transcript is re-sent on every round and the bound is a
273
 * context-budget decision, not a property of the record. `turn.assistant` is
274
 * the turn's whole answer. `turn.reasoning` is deliberately absent: the live
273
 * `tool_call_id` with the result budgeted by the same allowance the live loop
274
 * uses, which is why the resumed session's model family has to be passed in:
275
 * this transcript is re-sent on every round and the bound is a context-budget
276
 * decision against that family's window, not a property of the record.
277
 * `turn.assistant` is the turn's whole answer. `turn.reasoning` is deliberately absent: the live
275 278
 * loop never puts a thought on the wire.
276 279
 *
277 280
 * One call per assistant message, not one per round: the record does not

@@ -282,7 +285,10 @@ export function replayEntries(events: ReadonlyArray<ThreadEvent>): ReadonlyArray

282 285
 * `tool_calls` message is answered before the next assistant message, and
283 286
 * this shape keeps that invariant per call.
284 287
 */
285
export function replayWire(events: ReadonlyArray<ThreadEvent>): ReadonlyArray<WireMessage> {
288
export function replayWire(
289
  events: ReadonlyArray<ThreadEvent>,
290
  family: ToolFamily,
291
): ReadonlyArray<WireMessage> {
286 292
  const messages: WireMessage[] = [];
287 293
288 294
  for (const event of events) {

@@ -315,7 +321,7 @@ export function replayWire(events: ReadonlyArray<ThreadEvent>): ReadonlyArray<Wi

315 321
      messages.push({
316 322
        role: "tool",
317 323
        tool_call_id: callId,
318
        content: boundedResult(outcome),
324
        content: budgetedResult(outcome, family),
319 325
      });
320 326
    } else if (event.eventType === "turn.assistant") {
321 327
      const said = text(payload["text"]);
packages/openagents-cli/src/coder-self-harness.ts modified +3 -2

@@ -3,7 +3,8 @@ import { appendFileSync } from "node:fs";

3 3
import { accumulate, frames, parse, parseArguments } from "./coder-thread.js";
4 4
import type { ChildGrant } from "./coder-child-gateway.js";
5 5
import type { DelegateEvent, DelegateHarness } from "./coder-delegate.js";
6
import { boundedResult } from "./coder-thread.js";
6
import { budgetedResult } from "./coder-tool-budget.js";
7
import { toolFamilyOf } from "./coder-tool-families.js";
7 8
import type { CoderTool } from "./coder-tools.js";
8 9
import { shellTool } from "./coder-tools.js";
9 10
import { Redacted } from "effect";

@@ -234,7 +235,7 @@ export class SelfHarness implements DelegateHarness {

234 235
        transcript.push({
235 236
          role: "tool",
236 237
          tool_call_id: call.id,
237
          content: boundedResult(output),
238
          content: budgetedResult(output, toolFamilyOf(this.model)),
238 239
        });
239 240
        record({ type: "tool_result", callId: call.id, output });
240 241
      }
packages/openagents-cli/src/coder-shell.ts modified +23 -6

@@ -69,6 +69,14 @@ export interface ShellResult {

69 69
  readonly output: string;
70 70
  readonly code: number | undefined;
71 71
  readonly timedOut: boolean;
72
  /**
73
   * Characters the collector refused to hold, once the output passed the cap.
74
   *
75
   * Counted rather than discarded unrecorded: the reader of this result is a
76
   * model, and a `git log` it was handed half of reads exactly like a whole
77
   * one. The count is what lets the notice say how much is missing.
78
   */
79
  readonly dropped: number;
72 80
}
73 81
74 82
/**

@@ -96,6 +104,7 @@ export async function runShell(

96 104
    });
97 105
98 106
    let output = "";
107
    let dropped = 0;
99 108
    let settled = false;
100 109
    const finish = (result: ShellResult) => {
101 110
      if (settled) return;

@@ -107,17 +116,22 @@ export async function runShell(

107 116
108 117
    const timer = setTimeout(() => {
109 118
      child.kill("SIGKILL");
110
      finish({ output, code: undefined, timedOut: true });
119
      finish({ output, code: undefined, timedOut: true, dropped });
111 120
    }, timeoutMs);
112 121
113 122
    const onAbort = () => {
114 123
      child.kill("SIGKILL");
115
      finish({ output, code: undefined, timedOut: false });
124
      finish({ output, code: undefined, timedOut: false, dropped });
116 125
    };
117 126
    options.signal.addEventListener("abort", onAbort, { once: true });
118 127
119 128
    const collect = (chunk: Buffer) => {
120
      if (output.length < OUTPUT_LIMIT) output += chunk.toString("utf8");
129
      const text = chunk.toString("utf8");
130
      // Past the cap the text is counted rather than kept. Counting it is the
131
      // whole difference between a result that says what is missing and one
132
      // that quietly ends mid-file.
133
      if (output.length < OUTPUT_LIMIT) output += text;
134
      else dropped += text.length;
121 135
    };
122 136
    child.stdout.on("data", collect);
123 137
    child.stderr.on("data", collect);

@@ -127,19 +141,22 @@ export async function runShell(

127 141
        output: `The command could not be started: ${cause.message}`,
128 142
        code: undefined,
129 143
        timedOut: false,
144
        dropped: 0,
130 145
      });
131 146
    });
132 147
    child.on("close", (code) => {
133
      finish({ output, code: code ?? undefined, timedOut: false });
148
      finish({ output, code: code ?? undefined, timedOut: false, dropped });
134 149
    });
135 150
  });
136 151
}
137 152
138 153
/** What the model is shown for one run. */
139 154
export const renderShell = (result: ShellResult, timeoutMs: number): string => {
155
  const kept = Math.min(result.output.length, OUTPUT_LIMIT);
156
  const cut = result.output.length - kept + result.dropped;
140 157
  const bounded =
141
    result.output.length > OUTPUT_LIMIT
142
      ? `${result.output.slice(0, OUTPUT_LIMIT)}\n\n[truncated; narrow the command or write to a file and read part of it]`
158
    cut > 0
159
      ? `${result.output.slice(0, OUTPUT_LIMIT)}\n\n[The command printed ${String(kept + cut)} characters and this tool holds ${String(OUTPUT_LIMIT)}, so ${String(cut)} were cut from the end. What you have stops mid-output and must not be read as the whole of it: narrow the command, or write it to a file and read the part you need.]`
143 160
      : result.output;
144 161
  const body = bounded.trim();
145 162
packages/openagents-cli/src/coder-thread.ts modified +15 -20

@@ -61,6 +61,7 @@

61 61
import { Redacted } from "effect";
62 62
63 63
import type { ChildGrant } from "./coder-child-gateway.js";
64
import { budgetedResult, describeBudget, toolResultBudget } from "./coder-tool-budget.js";
64 65
import { declaredDescription, toolFamilyOf } from "./coder-tool-families.js";
65 66
import { merge } from "./coder-merge.js";
66 67
import type { ReplyChunk, ReplySource } from "./coder-session.js";

@@ -80,18 +81,16 @@ import { THREADS_PATH } from "./constants.js";

80 81
 */
81 82
const MAX_TOOL_STEPS = 100;
82 83
83
/** How much of one tool's output is kept on the transcript. */
84
const TOOL_RESULT_KEPT = 4_000;
85
86 84
/**
87 85
 * How much of one tool's output reaches the durable `tool.ran` event.
88 86
 *
89
 * A separate figure from `TOOL_RESULT_KEPT`, because they answer different
90
 * questions. The 4,000 above is a context-budget decision made against a
91
 * model's window: it is re-sent on every round of the turn. This one bounds a
92
 * record written once, so it is set where every result a real session has
93
 * produced fits whole — the largest measured was 8.4 KB — and only a
94
 * pathological dump is cut, kept at both ends the same way.
87
 * A separate figure from the model-facing budget in `coder-tool-budget.ts`,
88
 * because they answer different questions. That one is a context-budget
89
 * decision made against a model's window, per family: it is re-sent on every
90
 * round of the turn. This one bounds a record written once, so it is set where
91
 * every result a real session has produced fits whole — the largest measured
92
 * was 8.4 KB — and only a pathological dump is cut, kept at both ends the same
93
 * way.
95 94
 */
96 95
const EVENT_RESULT_KEPT = 64_000;
97 96

@@ -100,16 +99,9 @@ const bounded = (output: string, keep: number): string => {

100 99
  if (output.length <= keep) return output;
101 100
  const half = Math.floor(keep / 2);
102 101
  const cut = output.length - keep;
103
  return `${output.slice(0, half)}\n\n[${String(cut)} characters omitted from the middle; run it again more narrowly if you need them]\n\n${output.slice(-half)}`;
102
  return `${output.slice(0, half)}\n\n[${String(cut)} of ${String(output.length)} characters omitted from the middle; run it again more narrowly if you need them]\n\n${output.slice(-half)}`;
104 103
};
105 104
106
/**
107
 * A tool result as the model transcript carries it. Exported for the replay in
108
 * `coder-resume.ts`, which must feed a resumed model exactly what the live
109
 * loop would have.
110
 */
111
export const boundedResult = (output: string): string => bounded(output, TOOL_RESULT_KEPT);
112
113 105
/** What the thread may still spend, as the server last reported it. */
114 106
export interface ThreadBudget {
115 107
  /** `undefined` where the server set no ceiling: there is nothing counting down. */

@@ -500,6 +492,8 @@ export class ThreadReplySource implements ReplySource {

500 492
      `System message sent with every turn:\n\n${systemPrompt(this.tools, THREAD_LANE, this.standing)}`,
501 493
      "",
502 494
      declarations,
495
      "",
496
      describeBudget(toolResultBudget(toolFamilyOf(this.model))),
503 497
    ].join("\n");
504 498
  }
505 499

@@ -752,11 +746,12 @@ export class ThreadReplySource implements ReplySource {

752 746
        : { error: bounded(failure, EVENT_RESULT_KEPT) }),
753 747
    });
754 748
755
    // Bounded on the way toward the model, not on the way to the reader or the
749
    // Budgeted on the way toward the model, not on the way to the reader or the
756 750
    // record: this is what goes back on every round after, and a session that
757 751
    // re-sends everything it has already read spends its wall clock on reading
758
    // it again. The `tool.ran` event above kept the fuller copy.
759
    results.set(call.id, boundedResult(output));
752
    // it again. The allowance is this model family's, and a cut result says so.
753
    // The `tool.ran` event above kept the fuller copy.
754
    results.set(call.id, budgetedResult(output, toolFamilyOf(this.model)));
760 755
  }
761 756
762 757
  /**
packages/openagents-cli/src/coder-tool-budget.ts added +190

@@ -0,0 +1,190 @@

1
/**
2
 * What one tool result may cost, per model family.
3
 *
4
 * A tool result is not paid for once. Every later round of the turn re-sends
5
 * the whole transcript, so the result a command printed on round two is bought
6
 * again on rounds three through fifteen. The measured bill for that is in the
7
 * first graded Gym runs: a session that read the issue boards accumulated 82 KB
8
 * of tool output and re-sent it 25 times, and 91% of everything it sent each
9
 * round was output the model had already read
10
 * (`openagents.com` docs/terminalbench/2026-08-24-fix-git-run-analysis.md).
11
 *
12
 * The bound on that used to be one number — 4,000 characters — written twice,
13
 * once in the thread lane and once in the local lane, and applied to a 32k
14
 * local model and to a million-token hosted one alike. A single number cannot
15
 * be right for both: it is either most of a small model's window or a rounding
16
 * error in a large one, and in neither case does it reflect what the family's
17
 * tokenizer charges for those characters. So the budget is per family, it is
18
 * stated in tokens, and it is converted to characters through that family's own
19
 * density.
20
 *
21
 * ## The density figures are approximations, and say so
22
 *
23
 * Nothing here tokenizes. A real tokenizer per family is a dependency and a
24
 * download for a decision that is a ceiling, not an accounting entry — the
25
 * result is cut at a character count either way. What each family's figure is
26
 * for is that tokenizers differ enough to matter: byte-pair vocabularies
27
 * trained mostly on English prose land near four characters a token on prose
28
 * and lower on the shell output, paths, and diffs a coding session actually
29
 * reads. The numbers below are documented approximations, held low rather than
30
 * high, so an error spends less of the window than the budget says rather than
31
 * more. Replace one when a family is measured; do not read it as a measurement.
32
 *
33
 * ## Cutting is reported, never silent
34
 *
35
 * The caller of a tool is a model, and a model handed a quietly shortened
36
 * `git log` will summarize it as though it were the whole log. So a cut result
37
 * says it was cut, by how much, out of what, and against which family's budget
38
 * — the fail-closed limit discipline in `INVARIANTS.md`: a cap may drop
39
 * coverage, but it may never let an incomplete result read as complete.
40
 */
41
42
import type { ToolFamily } from "./coder-tool-families.js";
43
44
/** The figures a family's budget is derived from. */
45
interface FamilyBudget {
46
  /** The family's context window, in tokens, as the lane advertises it. */
47
  readonly contextWindowTokens: number;
48
  /** What one tool result may take of that window, in tokens. */
49
  readonly resultTokens: number;
50
  /** Approximate characters per token for this family's tokenizer. */
51
  readonly charactersPerToken: number;
52
  /** Why this family's result allowance is what it is. */
53
  readonly because: string;
54
}
55
56
/**
57
 * The budgets, as data.
58
 *
59
 * Exhaustive over `ToolFamily` on purpose: adding a family to that union
60
 * without deciding what it may spend does not compile. Each row carries the
61
 * reason it holds the figure it does, because a number nobody can argue with
62
 * is a number nobody will ever correct.
63
 */
64
const BUDGETS: Record<ToolFamily, FamilyBudget> = {
65
  default: {
66
    contextWindowTokens: 200_000,
67
    resultTokens: 1_100,
68
    // Byte-pair vocabularies of the o200k shape run near four characters a
69
    // token on prose and lower on the command output a session reads. Held at
70
    // 3.6 so the estimate errs toward spending less.
71
    charactersPerToken: 3.6,
72
    because:
73
      "the hosted general lanes carry a large window, and 1,100 tokens is the " +
74
      "measured allowance the shipped 4,000-character bound already amounted to",
75
  },
76
  gemini: {
77
    contextWindowTokens: 1_000_000,
78
    resultTokens: 700,
79
    // Google documents roughly four characters a token for English on the
80
    // SentencePiece vocabulary these lanes use.
81
    charactersPerToken: 4,
82
    because:
83
      "the window is the largest of any lane and the round count is what costs: " +
84
      "this family issued fifteen tool rounds where another issued six and spent " +
85
      "three times the input tokens replaying whole-file dumps, so its results " +
86
      "are cut sooner to make a narrower second read the cheaper move",
87
  },
88
  local: {
89
    contextWindowTokens: 32_768,
90
    resultTokens: 500,
91
    // Qwen-shaped byte-pair vocabularies, again held low for code and paths.
92
    charactersPerToken: 3.5,
93
    because:
94
      "the window is a fraction of a hosted one and generation is slow on one " +
95
      "machine, so every re-sent character is paid in wall clock rather than in " +
96
      "money",
97
  },
98
};
99
100
/** The smallest budget on the table: what an unrecognized family is given. */
101
const mostConservative = (): ToolFamily => {
102
  const families = Object.keys(BUDGETS) as ReadonlyArray<ToolFamily>;
103
  let smallest: ToolFamily = "local";
104
  for (const family of families) {
105
    if (charactersOf(BUDGETS[family]) < charactersOf(BUDGETS[smallest])) smallest = family;
106
  }
107
  return smallest;
108
};
109
110
const charactersOf = (budget: FamilyBudget): number =>
111
  Math.floor(budget.resultTokens * budget.charactersPerToken);
112
113
/** What one tool result may spend, resolved for one family. */
114
export interface ToolResultBudget {
115
  /** The family the budget was asked for. */
116
  readonly family: ToolFamily;
117
  /** The ceiling the result is cut to. */
118
  readonly characters: number;
119
  /** What those characters are believed to cost. */
120
  readonly tokens: number;
121
  /** The approximation the two are related by. */
122
  readonly charactersPerToken: number;
123
  readonly contextWindowTokens: number;
124
  /**
125
   * True when the family had no row and the smallest budget was substituted.
126
   *
127
   * Carried rather than hidden: the notice on a cut result says so, because a
128
   * budget that is a guess and a budget that is a decision are not the same
129
   * claim.
130
   */
131
  readonly substituted: boolean;
132
}
133
134
/**
135
 * The budget for a family.
136
 *
137
 * A family with no row falls back to the smallest budget on the table rather
138
 * than to a generous default. A family name reaches this from data — a lane
139
 * name derived from a server catalog, a resumed session's record — so the case
140
 * is reachable at runtime even though the union is exhaustive at compile time,
141
 * and guessing high is the failure that is expensive.
142
 */
143
export const toolResultBudget = (family: ToolFamily): ToolResultBudget => {
144
  const held = BUDGETS[family] as FamilyBudget | undefined;
145
  const substituted = held === undefined;
146
  const budget = held ?? BUDGETS[mostConservative()];
147
  return {
148
    family,
149
    characters: charactersOf(budget),
150
    tokens: budget.resultTokens,
151
    charactersPerToken: budget.charactersPerToken,
152
    contextWindowTokens: budget.contextWindowTokens,
153
    substituted,
154
  };
155
};
156
157
/** One line naming a family's allowance, for a reader asking what it is. */
158
export const describeBudget = (budget: ToolResultBudget): string =>
159
  `Tool results are cut to ${String(budget.characters)} characters for the ` +
160
  `${budget.family} model family: about ${String(budget.tokens)} tokens at an ` +
161
  `approximate ${String(budget.charactersPerToken)} characters per token, against a ` +
162
  `${String(budget.contextWindowTokens)}-token window` +
163
  (budget.substituted
164
    ? " — the smallest budget on the table, substituted because that family has none of its own."
165
    : ".");
166
167
/**
168
 * One tool result as the model transcript carries it.
169
 *
170
 * Kept at both ends, which is what a long output is read for: a command's
171
 * first lines say what it did and its last lines say how it ended, and the
172
 * middle is the part a second, narrower run can recover. The notice in place of
173
 * the middle is written for the model that has to decide what to do next, so it
174
 * carries the arithmetic rather than the word "truncated".
175
 */
176
export const budgetedResult = (output: string, family: ToolFamily): string => {
177
  const budget = toolResultBudget(family);
178
  if (output.length <= budget.characters) return output;
179
180
  const half = Math.floor(budget.characters / 2);
181
  const omitted = output.length - budget.characters;
182
  const notice =
183
    `[${String(omitted)} of ${String(output.length)} characters omitted from the middle. ` +
184
    `${describeBudget(budget)} ` +
185
    "Every later round of this turn re-sends what you are reading, which is why the " +
186
    "budget exists. What you have is incomplete and must not be summarized as if it " +
187
    "were the whole answer: run it again more narrowly — a range, a filter, a count — " +
188
    "if you need what is missing.]";
189
  return `${output.slice(0, half)}\n\n${notice}\n\n${output.slice(-half)}`;
190
};
packages/openagents-cli/src/coder-tools.ts modified +24 -4

@@ -227,8 +227,21 @@ function report(

227 227
  return [header, "", ...lines].join("\n");
228 228
}
229 229
230
/**
231
 * A child's answer, cut to what the parent is shown.
232
 *
233
 * The cut says its own size. A child that reported ten findings and was shown
234
 * as three reads, to the model holding the report, exactly like a child that
235
 * found three.
236
 */
230 237
function clip(text: string, limit: number): string {
231
  return text.length <= limit ? text : `${text.slice(0, limit)}\n…[truncated]`;
238
  if (text.length <= limit) return text;
239
  const cut = text.length - limit;
240
  return (
241
    `${text.slice(0, limit)}\n` +
242
    `…[${String(cut)} of ${String(text.length)} characters cut from the end of this child's ` +
243
    "answer; ask the child again for the part you need, or give it a narrower task]"
244
  );
232 245
}
233 246
234 247
/**

@@ -437,6 +450,7 @@ export function openagentsTool(): CoderTool {

437 450
          stdio: ["ignore", "pipe", "pipe"],
438 451
        });
439 452
        let output = "";
453
        let dropped = 0;
440 454
        let done = false;
441 455
442 456
        const finish = (text: string) => {

@@ -461,7 +475,11 @@ export function openagentsTool(): CoderTool {

461 475
        signal.addEventListener("abort", onAbort, { once: true });
462 476
463 477
        const collect = (chunk: Buffer) => {
464
          if (output.length < CLI_OUTPUT_LIMIT) output += chunk.toString("utf8");
478
          const text = chunk.toString("utf8");
479
          // Past the cap the text is counted, not dropped unrecorded: a listing
480
          // the model was handed half of reads like a whole listing.
481
          if (output.length < CLI_OUTPUT_LIMIT) output += text;
482
          else dropped += text.length;
465 483
        };
466 484
        child.stdout.on("data", collect);
467 485
        child.stderr.on("data", collect);

@@ -478,9 +496,11 @@ export function openagentsTool(): CoderTool {

478 496
          const advice = args.includes("--json")
479 497
            ? "drop --json and read the plain output, or ask for one record"
480 498
            : "narrow it with a flag such as --limit, --label, or --state";
499
          const kept = Math.min(output.length, CLI_OUTPUT_LIMIT);
500
          const cut = output.length - kept + dropped;
481 501
          const bounded =
482
            output.length > CLI_OUTPUT_LIMIT
483
              ? `${output.slice(0, CLI_OUTPUT_LIMIT)}\n\n[The output was cut off here. Run it again and ${advice}; what you have above is incomplete and must not be summarized as if it were the whole answer.]`
502
            cut > 0
503
              ? `${output.slice(0, CLI_OUTPUT_LIMIT)}\n\n[The command printed ${String(kept + cut)} characters and this tool holds ${String(CLI_OUTPUT_LIMIT)}, so ${String(cut)} were cut off here. Run it again and ${advice}; what you have above is incomplete and must not be summarized as if it were the whole answer.]`
484 504
              : output;
485 505
          const body = bounded.trim();
486 506
          // The exit code is reported on failure because it is what the CLI
packages/openagents-cli/src/coder-zen.ts modified +4 -2

@@ -4,7 +4,9 @@ import { join } from "node:path";

4 4
5 5
import type { ReplyChunk, ReplySource } from "./coder-session.js";
6 6
import { systemPrompt } from "./coder-system.js";
7
import { accumulate, boundedResult, frames, parse, parseArguments } from "./coder-thread.js";
7
import { budgetedResult } from "./coder-tool-budget.js";
8
import { toolFamilyOf } from "./coder-tool-families.js";
9
import { accumulate, frames, parse, parseArguments } from "./coder-thread.js";
8 10
import type { CoderTool } from "./coder-tools.js";
9 11
10 12
/**

@@ -254,7 +256,7 @@ export class ZenReplySource implements ReplySource {

254 256
        this.transcript.push({
255 257
          role: "tool",
256 258
          tool_call_id: call.id,
257
          content: boundedResult(output),
259
          content: budgetedResult(output, toolFamilyOf(this.model)),
258 260
        });
259 261
        yield { type: "tool_result", callId: call.id, output, error: undefined };
260 262
      }
packages/openagents-cli/test/coder-resume.test.ts modified +42 -19

@@ -378,7 +378,7 @@ describe("replayEntries", () => {

378 378
379 379
describe("replayWire", () => {
380 380
  it("rebuilds the messages in the shape the live loop holds", () => {
381
    expect(replayWire(FIXTURE)).toEqual([
381
    expect(replayWire(FIXTURE, "default")).toEqual([
382 382
      { role: "user", content: "standing context\n\n---\n\nwhat is in mix.exs?" },
383 383
      {
384 384
        role: "assistant",

@@ -411,7 +411,7 @@ describe("replayWire", () => {

411 411
  });
412 412
413 413
  it("keeps the recorded arguments as the raw JSON string", () => {
414
    const wire = replayWire(FIXTURE);
414
    const wire = replayWire(FIXTURE, "default");
415 415
    const call = wire[1];
416 416
    expect(call?.role === "assistant" && call.tool_calls?.[0]?.function.arguments).toBe(
417 417
      `{"command":"cat mix.exs"}`,

@@ -419,29 +419,52 @@ describe("replayWire", () => {

419 419
  });
420 420
421 421
  it("keeps reasoning off the wire, as the live loop does", () => {
422
    const wire = replayWire(FIXTURE);
422
    const wire = replayWire(FIXTURE, "default");
423 423
    expect(wire.some((message) => message.content.includes("before answering"))).toBe(false);
424 424
  });
425 425
426
  it("bounds a stored tool result to the live loop's wire figure", () => {
427
    const wire = replayWire([
428
      {
429
        id: 1,
430
        eventType: "tool.ran",
431
        payload: {
432
          call_id: "call-1",
433
          tool: "shell",
434
          arguments: "{}",
435
          status: "succeeded",
436
          output: "x".repeat(10_000),
426
  it("budgets a stored tool result by the resumed session's model family", () => {
427
    const wire = replayWire(
428
      [
429
        {
430
          id: 1,
431
          eventType: "tool.ran",
432
          payload: {
433
            call_id: "call-1",
434
            tool: "shell",
435
            arguments: "{}",
436
            status: "succeeded",
437
            output: "x".repeat(10_000),
438
          },
439
          emittedAt: undefined,
437 440
        },
438
        emittedAt: undefined,
439
      },
440
    ]);
441
      ],
442
      "gemini",
443
    );
441 444
    const result = wire[1];
442 445
    expect(result?.role).toBe("tool");
443 446
    expect(result?.content).toContain("characters omitted from the middle");
444
    expect(result?.content.length ?? 0).toBeLessThan(5_000);
447
    expect(result?.content).toContain("for the gemini model family");
448
    // The same event replayed for a hosted lane keeps more of it, because the
449
    // budget is the family's and not the record's.
450
    const hosted = replayWire(
451
      [
452
        {
453
          id: 1,
454
          eventType: "tool.ran",
455
          payload: {
456
            call_id: "call-1",
457
            tool: "shell",
458
            arguments: "{}",
459
            status: "succeeded",
460
            output: "x".repeat(10_000),
461
          },
462
          emittedAt: undefined,
463
        },
464
      ],
465
      "default",
466
    );
467
    expect((hosted[1]?.content.length ?? 0) > (result?.content.length ?? 0)).toBe(true);
445 468
  });
446 469
});
447 470

@@ -533,7 +556,7 @@ describe("remintThread", () => {

533 556
    };
534 557
535 558
    const source = await remintThread({ origin: ORIGIN, token: TOKEN, threadId: THREAD_ID });
536
    const replayed = replayWire(FIXTURE);
559
    const replayed = replayWire(FIXTURE, "default");
537 560
    source.preload(replayed);
538 561
    source.useTranscript(sink);
539 562
packages/openagents-cli/test/coder-shell.test.ts modified +12

@@ -105,4 +105,16 @@ describe("running a command", () => {

105 105
  it("asks for a command rather than running nothing", async () => {
106 106
    await expect(run("   ")).resolves.toContain("`command` is required");
107 107
  });
108
109
  it("says how much of an oversized output was cut, rather than ending mid-line", async () => {
110
    // 100,000 characters against a 30,000-character hold. What the collector
111
    // refuses is counted, so the notice can name the whole size: a result that
112
    // stops without saying so is read as the whole of it.
113
    const output = await run("yes wwwwwwwwwwwwwwwwwww | head -n 5000", 30);
114
115
    expect(output).toContain("The command printed 100000 characters");
116
    expect(output).toContain("this tool holds 30000");
117
    expect(output).toContain("70000 were cut from the end");
118
    expect(output).toContain("must not be read as the whole of it");
119
  });
108 120
});
packages/openagents-cli/test/coder-thread.test.ts modified +32 -6

@@ -8,6 +8,7 @@ import {

8 8
  ThreadUnavailable,
9 9
} from "../src/coder-thread.js";
10 10
import { Redacted } from "effect";
11
import { toolResultBudget } from "../src/coder-tool-budget.js";
11 12
import { shellTool } from "../src/coder-tools.js";
12 13
import { ThreadTranscriptWriter } from "../src/coder-transcript.js";
13 14

@@ -560,14 +561,18 @@ describe("ThreadReplySource", () => {

560 561
    expect(messages[1]?.["tool_calls"]).toHaveLength(1);
561 562
  });
562 563
563
  it("bounds a tool result on the wire while the record keeps the fuller copy", async () => {
564
  /** One turn that calls a tool returning 10,000 characters, on one model. */
565
  const dumped = async (model: string) => {
564 566
    const round = [
565 567
      `data: {"choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"id":"call-1","type":"function","function":{"name":"shell","arguments":"{}"}}]}}]}`,
566 568
      `data: {"choices":[{"index":0,"delta":{},"finish_reason":"tool_calls"}]}`,
567 569
      `data: [DONE]`,
568 570
      "",
569 571
    ].join("\n\n");
570
    const calls = stub({ proxy: [sse([round]), sse([LIVE_SSE])] });
572
    const calls = stub({
573
      create: json(201, { ...CREATED, grant: { ...CREATED.grant, model } }),
574
      proxy: [sse([round]), sse([LIVE_SSE])],
575
    });
571 576
    const source = await open();
572 577
    const sink = recorder();
573 578
    source.useTranscript(sink);

@@ -578,15 +583,36 @@ describe("ThreadReplySource", () => {

578 583
    const spends = calls.filter((call) => call.url.endsWith("/api/inference/proxy"));
579 584
    const messages = spends[1]?.body["messages"] as Array<Record<string, unknown>>;
580 585
    const result = messages.find((message) => message["role"] === "tool");
581
    const wire = result?.["content"] as string;
582
    // The 4,000-char context budget still holds on the way to the model...
583
    expect(wire.length).toBeLessThan(4_200);
584
    expect(wire).toContain("characters omitted");
586
    return { wire: result?.["content"] as string, sink };
587
  };
588
589
  it("budgets a tool result on the wire while the record keeps the fuller copy", async () => {
590
    const { wire, sink } = await dumped("gpt-5.6-luna");
591
592
    // The context budget for this model's family holds on the way to the
593
    // model, and the notice for what it cut is part of what is sent...
594
    const budget = toolResultBudget("default");
595
    expect(wire.length).toBeLessThan(budget.characters + 800);
596
    expect(wire).toContain(`${String(10_000 - budget.characters)} of 10000 characters omitted`);
585 597
    // ...while the durable event keeps the result whole, as before.
586 598
    const ran = sink.events.find((event) => event.eventType === "tool.ran");
587 599
    expect(ran?.payload["output"]).toBe("x".repeat(10_000));
588 600
  });
589 601
602
  it("budgets the same result differently on a model of another family", async () => {
603
    const hosted = await dumped("gpt-5.6-luna");
604
    const gemini = await dumped("gemini-3.7-flash");
605
606
    // Same tool, same output, two allowances — and each result says which
607
    // family's budget cut it and by how much.
608
    expect(gemini.wire.length).toBeLessThan(hosted.wire.length);
609
    expect(hosted.wire).toContain("for the default model family");
610
    expect(gemini.wire).toContain("for the gemini model family");
611
    expect(gemini.wire).toContain(
612
      `${String(10_000 - toolResultBudget("gemini").characters)} of 10000 characters omitted`,
613
    );
614
  });
615
590 616
  it("drops the calls past the step limit and asks for an answer without tools", async () => {
591 617
    const round = [
592 618
      `data: {"choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"id":"call-1","type":"function","function":{"name":"shell","arguments":"{}"}}]}}]}`,
packages/openagents-cli/test/coder-tool-budget.test.ts added +99

@@ -0,0 +1,99 @@

1
import { describe, expect, it } from "vitest";
2
3
import { budgetedResult, describeBudget, toolResultBudget } from "../src/coder-tool-budget.js";
4
import type { ToolFamily } from "../src/coder-tool-families.js";
5
import { toolFamilyOf } from "../src/coder-tool-families.js";
6
7
/** One oversized result, the same one for every family. */
8
const oversized = "x".repeat(20_000);
9
10
/** The characters a family keeps of that result, notice excluded. */
11
const kept = (family: ToolFamily): number =>
12
  budgetedResult(oversized, family).replaceAll(/\n\n\[[\s\S]*?\]\n\n/g, "").length;
13
14
describe("toolResultBudget", () => {
15
  it("gives each family its own allowance, in tokens and in characters", () => {
16
    const hosted = toolResultBudget("default");
17
    const gemini = toolResultBudget("gemini");
18
    const local = toolResultBudget("local");
19
20
    expect(hosted.characters).not.toBe(gemini.characters);
21
    expect(gemini.characters).not.toBe(local.characters);
22
    // Every budget is a token allowance converted through that family's own
23
    // approximate density, not a character count someone picked.
24
    for (const budget of [hosted, gemini, local]) {
25
      expect(budget.characters).toBe(Math.floor(budget.tokens * budget.charactersPerToken));
26
      expect(budget.substituted).toBe(false);
27
    }
28
    expect(gemini.contextWindowTokens).toBeGreaterThan(local.contextWindowTokens);
29
  });
30
31
  it("substitutes the smallest budget for a family it does not know, and says so", () => {
32
    const unknown = toolResultBudget("mystery-lane" as ToolFamily);
33
34
    expect(unknown.substituted).toBe(true);
35
    expect(unknown.characters).toBe(
36
      Math.min(
37
        toolResultBudget("default").characters,
38
        toolResultBudget("gemini").characters,
39
        toolResultBudget("local").characters,
40
      ),
41
    );
42
    expect(describeBudget(unknown)).toContain("substituted");
43
  });
44
45
  it("states the approximation rather than implying a token count was measured", () => {
46
    expect(describeBudget(toolResultBudget("gemini"))).toContain("approximate");
47
  });
48
});
49
50
describe("budgetedResult", () => {
51
  it("budgets one oversized result differently for two model families", () => {
52
    const forGemini = kept("gemini");
53
    const forHosted = kept("default");
54
55
    expect(forGemini).not.toBe(forHosted);
56
    expect(forGemini).toBeLessThan(oversized.length);
57
    expect(forHosted).toBeLessThan(oversized.length);
58
    expect(forGemini).toBeLessThanOrEqual(toolResultBudget("gemini").characters);
59
    expect(forHosted).toBeLessThanOrEqual(toolResultBudget("default").characters);
60
  });
61
62
  it("reports the cut to both families, with how much went and out of what", () => {
63
    for (const family of ["gemini", "default", "local"] as const) {
64
      const budget = toolResultBudget(family);
65
      const result = budgetedResult(oversized, family);
66
      const omitted = oversized.length - budget.characters;
67
68
      expect(result).toContain(`${String(omitted)} of ${String(oversized.length)} characters`);
69
      expect(result).toContain(`${String(budget.characters)} characters for the ${family}`);
70
      expect(result).toContain("must not be summarized as if it were the whole answer");
71
    }
72
  });
73
74
  it("keeps both ends of what it does show", () => {
75
    const output = `${"head".padEnd(4_000, "h")}${"tail".padStart(4_000, "t")}`;
76
    const result = budgetedResult(output, "default");
77
78
    expect(result.startsWith("head")).toBe(true);
79
    expect(result.endsWith("tail")).toBe(true);
80
  });
81
82
  it("leaves a result inside the budget exactly as the tool produced it", () => {
83
    const small = "the command succeeded";
84
85
    expect(budgetedResult(small, "local")).toBe(small);
86
    expect(budgetedResult(small, "gemini")).toBe(small);
87
  });
88
89
  it("budgets the lanes the catalog actually runs, through their families", () => {
90
    // The catalog names lanes, not families, so the two steps compose: a lane
91
    // resolves to a family and the family carries the allowance.
92
    const gemini = budgetedResult(oversized, toolFamilyOf("gemini-3.7-flash"));
93
    const luna = budgetedResult(oversized, toolFamilyOf("gpt-5.6-luna"));
94
    const ox = budgetedResult(oversized, toolFamilyOf("ox-alpha"));
95
96
    expect(gemini.length).not.toBe(luna.length);
97
    expect(luna.length).toBe(ox.length);
98
  });
99
});
packages/openagents-cli/test/coder-tools.test.ts modified +13

@@ -64,6 +64,19 @@ describe("delegateTool", () => {

64 64
    expect(delegation.registry.list()).toHaveLength(3);
65 65
  });
66 66
67
  it("says how much of a long child answer was cut, rather than ending it silently", async () => {
68
    const answer = "finding. ".repeat(1_000);
69
    const output = await delegateTool(delegationOf(harness(answer))).run(
70
      { prompt: "report everything" },
71
      new AbortController().signal,
72
    );
73
74
    // A child shown as three findings, having reported ten, reads to the model
75
    // holding the report exactly like a child that found three.
76
    expect(output).toContain(`of ${String(answer.trim().length)} characters cut from the end`);
77
    expect(output).toContain("ask the child again for the part you need");
78
  });
79
67 80
  it("tells each child of a fan-out which one it is", async () => {
68 81
    const prompts: string[] = [];
69 82
    const delegation = delegationOf({

This page updates live while a promote is in flight · changelog