Teach the tools token economy, per model family

44ada8baa306 · AtlantisPleb · · parent 5509cbec12f8

Teach the tools token economy, per model family

The Gym's first graded runs measured a family habit, not a capability
gap: gemini-3.7-flash did the same task as gpt-5.6-luna in fifteen
single-command tool rounds instead of six composite ones and spent
three times the input tokens replaying the transcript
(docs/terminalbench/2026-08-24-fix-git-run-analysis.md). The system
prompt's batching advice was not enough for every family, and Gemini
CLI's own harness answers this inside the declarations.

That pattern, adopted as data: the shell tool's base description gains
efficiency guidance for everyone (batch with &&, quiet flags,
git --no-pager, --stat before -p), and coder-tool-families resolves a
family from the model or lane and appends the emphasis that family has
measurably needed — batching for gemini, latency-awareness for the
local lane, nothing for the default. Both lanes declare through the
resolver, ATIF records what was actually declared, and the lane
sentences carry their economics (metered: rounds replay the
conversation; local: tokens free, generation slow). An override earns
its place with a Gym delta, per the analysis. Also removes an unused
import that landed on main failing the strict typecheck.

Closes OpenAgentsInc/openagents#36's first slice; the Gym rerun is the
oracle.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GoYpb8FEmdxVErsv7ABCYi
Co-Authored-By
Claude Fable 5 <noreply@anthropic.com>
Closes
OpenAgentsInc/openagents#36 (another repository — recorded, not closed)

Deploy story

What this commit did to the running system — joined from the forge receipt chain, the part a commit page elsewhere cannot show.

Not deployed through the forge lane

No push, promotion, build, or deploy receipt references this commit (receipts are scanned over a bounded recent window). Changes shipped by full node replacement carry their proof in the release gate receipt instead.

Changed files

  • modified packages/openagents-cli/src/coder-ollama.ts
  • modified packages/openagents-cli/src/coder-system.ts
  • modified packages/openagents-cli/src/coder-thread.ts
  • added packages/openagents-cli/src/coder-tool-families.ts
  • modified packages/openagents-cli/src/coder-tools.ts
  • modified packages/openagents-cli/test/coder-delegate-lanes.test.ts
  • added packages/openagents-cli/test/coder-tool-families.test.ts

Diff

7 files changed, +142 -8

packages/openagents-cli/src/coder-ollama.ts modified +9 -2

@@ -14,6 +14,7 @@

14 14
import { Ollama } from "ollama";
15 15
import type { Message as OllamaMessage, Tool as OllamaTool, ToolCall as OllamaToolCall } from "ollama";
16 16
17
import { declaredDescription } from "./coder-tool-families.js";
17 18
import { merge } from "./coder-merge.js";
18 19
import type { ReplyChunk, ReplySource } from "./coder-session.js";
19 20
import { LOCAL_LANE, systemPrompt } from "./coder-system.js";

@@ -297,7 +298,13 @@ export class OllamaReplySource implements ReplySource {

297 298
  toolDefinitions(): ReadonlyArray<Record<string, unknown>> {
298 299
    return this.tools.map((tool) => ({
299 300
      type: "function",
300
      function: { name: tool.name, description: tool.description, parameters: tool.parameters },
301
      function: {
302
        name: tool.name,
303
        // The local family's emphasis is the lane's economics — free tokens,
304
        // slow generation — so it applies whatever weights answer.
305
        description: declaredDescription(tool, "local"),
306
        parameters: tool.parameters,
307
      },
301 308
    }));
302 309
  }
303 310

@@ -416,7 +423,7 @@ export class OllamaReplySource implements ReplySource {

416 423
                type: "function",
417 424
                function: {
418 425
                  name: tool.name,
419
                  description: tool.description,
426
                  description: declaredDescription(tool, "local"),
420 427
                  // The client's type for a schema is narrower than JSON
421 428
                  // Schema. The server takes the schema as written.
422 429
                  parameters: tool.parameters as NonNullable<OllamaTool["function"]["parameters"]>,
packages/openagents-cli/src/coder-system.ts modified +6 -2

@@ -55,8 +55,12 @@ export const systemPrompt = (

55 55
56 56
/** The lane sentence for a session answering from a model on this machine. */
57 57
export const LOCAL_LANE =
58
  "You answer from a model running locally on this machine through Ollama.";
58
  "You answer from a model running locally on this machine through Ollama. Tokens here cost " +
59
  "nothing, but generation is slow: prefer a few composite tool calls over many small ones, " +
60
  "keep narration brief, and verify in one final pass rather than several.";
59 61
60 62
/** The lane sentence for a session answering through the account's thread. */
61 63
export const THREAD_LANE =
62
  "You answer through the OpenAgents inference proxy, on a thread opened for this session.";
64
  "You answer through the OpenAgents inference proxy, on a thread opened for this session. " +
65
  "Every round of tool calls re-sends the whole conversation to a metered model, so batch " +
66
  "independent commands into one call and keep large dumps out of the transcript.";
packages/openagents-cli/src/coder-thread.ts modified +11 -2

@@ -61,6 +61,7 @@

61 61
import { Redacted } from "effect";
62 62
63 63
import type { ChildGrant } from "./coder-child-gateway.js";
64
import { declaredDescription, toolFamilyOf } from "./coder-tool-families.js";
64 65
import { merge } from "./coder-merge.js";
65 66
import type { ReplyChunk, ReplySource } from "./coder-session.js";
66 67
import type { CoderTool } from "./coder-tools.js";

@@ -468,9 +469,14 @@ export class ThreadReplySource implements ReplySource {

468 469
   */
469 470
  /** The tools as declared, in the shape ATIF records them. */
470 471
  toolDefinitions(): ReadonlyArray<Record<string, unknown>> {
472
    const family = toolFamilyOf(this.model);
471 473
    return this.tools.map((tool) => ({
472 474
      type: "function",
473
      function: { name: tool.name, description: tool.description, parameters: tool.parameters },
475
      function: {
476
        name: tool.name,
477
        description: declaredDescription(tool, family),
478
        parameters: tool.parameters,
479
      },
474 480
    }));
475 481
  }
476 482

@@ -801,11 +807,14 @@ export class ThreadReplySource implements ReplySource {

801 807
          ...(this.tools.length === 0 || this.mustAnswer
802 808
            ? {}
803 809
            : {
810
                // Declarations resolve per model family: the base says what
811
                // a tool is, and a family override adds the emphasis that
812
                // family has measurably needed (coder-tool-families.ts).
804 813
                tools: this.tools.map((tool) => ({
805 814
                  type: "function",
806 815
                  function: {
807 816
                    name: tool.name,
808
                    description: tool.description,
817
                    description: declaredDescription(tool, toolFamilyOf(this.model)),
809 818
                    parameters: tool.parameters,
810 819
                  },
811 820
                })),
packages/openagents-cli/src/coder-tool-families.ts added +67

@@ -0,0 +1,67 @@

1
/**
2
 * Per-model-family tool declaration overrides.
3
 *
4
 * The first graded Gym runs measured a model-family habit, not a capability
5
 * gap: on the same task, gemini-3.7-flash issued fifteen single-command tool
6
 * rounds where gpt-5.6-luna issued six composite ones, and spent three times
7
 * the input tokens replaying the growing transcript
8
 * (`openagents.com` docs/terminalbench/2026-08-24-fix-git-run-analysis.md).
9
 * Saying "batch" once in the system prompt was not enough for every family.
10
 *
11
 * Gemini CLI's own harness answers the same problem inside the declarations:
12
 * a base declaration per tool plus per-model-family description overrides,
13
 * resolved at request time (`gemini-cli`
14
 * packages/core/src/tools/definitions/resolver.ts). This is that pattern,
15
 * kept as data: a family is a name derived from the model or lane, and an
16
 * override is an extra sentence appended to a tool's base description. The
17
 * base descriptions stay the single source of what a tool is; a family
18
 * override only adds the emphasis that family has measurably needed.
19
 */
20
21
import type { CoderTool } from "./coder-tools.js";
22
23
/** The families with distinct declared emphasis. `default` adds nothing. */
24
export type ToolFamily = "default" | "gemini" | "local";
25
26
/**
27
 * The family for a model name, by prefix.
28
 *
29
 * The local lane passes `"local"` explicitly rather than relying on model
30
 * names, because what distinguishes it is the lane's economics — free
31
 * tokens, slow generation — not which weights answer.
32
 */
33
export const toolFamilyOf = (model: string | undefined): ToolFamily => {
34
  if (model === undefined) return "default";
35
  const normalized = model.toLowerCase();
36
  if (normalized.startsWith("gemini")) return "gemini";
37
  if (normalized.startsWith("ollama:")) return "local";
38
  return "default";
39
};
40
41
/**
42
 * Extra emphasis per family and tool, appended to the base description.
43
 *
44
 * Measured, not speculative: an override earns its place with a Gym delta on
45
 * the same suite, and the analysis document above records why each exists.
46
 */
47
const emphasis: Partial<Record<ToolFamily, Partial<Record<string, string>>>> = {
48
  gemini: {
49
    shell:
50
      " IMPORTANT: batch independent commands into ONE call joined with && — " +
51
      "each separate call replays the whole conversation to the model, so ten " +
52
      "one-line calls cost several times what one composite call costs. Never " +
53
      "run one small inspection per call.",
54
  },
55
  local: {
56
    shell:
57
      " This session's model generates slowly on this machine: prefer a few " +
58
      "composite calls over many small ones, and keep verification to one " +
59
      "final pass.",
60
  },
61
};
62
63
/** A tool's description as declared to this family's model. */
64
export const declaredDescription = (tool: CoderTool, family: ToolFamily): string => {
65
  const extra = emphasis[family]?.[tool.name];
66
  return extra === undefined ? tool.description : tool.description + extra;
67
};
packages/openagents-cli/src/coder-tools.ts modified +4 -1

@@ -508,7 +508,10 @@ export function shellTool(cwd: string): CoderTool {

508 508
      "output. Both output streams come back together with the exit code. There is no terminal, " +
509 509
      "so a command that would prompt gets end-of-file instead of waiting; pass a flag that " +
510 510
      "answers the prompt. A few commands that cannot be undone are refused, such as erasing a " +
511
      "root or a home directory, reformatting a disk, or halting the machine.",
511
      "root or a home directory, reformatting a disk, or halting the machine. Work economically: " +
512
      "batch independent commands into one call with && instead of one call each — every call " +
513
      "replays the conversation so far. Use quiet flags and `git --no-pager`; prefer `--stat` " +
514
      "before `-p`, and dump a full patch only for the file you are actually deciding about.",
512 515
    parameters: {
513 516
      type: "object",
514 517
      properties: {
packages/openagents-cli/test/coder-delegate-lanes.test.ts modified -1

@@ -1,7 +1,6 @@

1 1
import { describe, expect, it } from "vitest";
2 2
3 3
import {
4
  CHILD_LANE_ALIASES,
5 4
  CHILD_MODELS,
6 5
  childLaneName,
7 6
  resolveChildLane,
packages/openagents-cli/test/coder-tool-families.test.ts added +45

@@ -0,0 +1,45 @@

1
import { describe, expect, it } from "vitest";
2
3
import { declaredDescription, toolFamilyOf } from "../src/coder-tool-families.js";
4
import type { CoderTool } from "../src/coder-tools.js";
5
6
const tool = (name: string): CoderTool => ({
7
  name,
8
  description: "Base description.",
9
  parameters: { type: "object", properties: {} },
10
  run: async () => "",
11
});
12
13
describe("toolFamilyOf", () => {
14
  it("names the gemini family by model prefix", () => {
15
    expect(toolFamilyOf("gemini-3.7-flash")).toBe("gemini");
16
    expect(toolFamilyOf("gemini-3.5-flash")).toBe("gemini");
17
  });
18
19
  it("names the local family by the ollama model shape", () => {
20
    expect(toolFamilyOf("ollama:qwen3.8:27b-mtp-q8_0")).toBe("local");
21
  });
22
23
  it("defaults everything else, including absence", () => {
24
    expect(toolFamilyOf("gpt-5.6-luna")).toBe("default");
25
    expect(toolFamilyOf("ox-alpha")).toBe("default");
26
    expect(toolFamilyOf(undefined)).toBe("default");
27
  });
28
});
29
30
describe("declaredDescription", () => {
31
  it("adds batching emphasis to shell for the gemini family", () => {
32
    const declared = declaredDescription(tool("shell"), "gemini");
33
    expect(declared.startsWith("Base description.")).toBe(true);
34
    expect(declared).toContain("batch independent commands into ONE call");
35
  });
36
37
  it("adds the latency emphasis to shell for the local family", () => {
38
    expect(declaredDescription(tool("shell"), "local")).toContain("generates slowly");
39
  });
40
41
  it("leaves the default family and unlisted tools at the base", () => {
42
    expect(declaredDescription(tool("shell"), "default")).toBe("Base description.");
43
    expect(declaredDescription(tool("skill"), "gemini")).toBe("Base description.");
44
  });
45
});

This page updates live while a promote is in flight · changelog