OpenAgents

Commit

Stop re-sending what the model already read

da174dc06e7a · AtlantisPleb · · parent c0e5957660db 

Stop re-sending what the model already read

A session that worked the issue boards took 59 minutes over 51 steps, and the
gaps grew with it: 13s early, 192s in the middle, 705s at the end. The shape is
unmistakable — latency tracked the transcript, not the work.

Measured from the trajectory, what went back to the model on every round was
90,778 bytes, and 82,546 of them were tool output. Ninety-one per cent of every
request was output the model had already read, re-read twenty-five times.

Tool results are now bounded on the transcript at 4,000 characters, head and
tail kept, with the omission counted. The reader still sees all of it; this is
only what is re-sent. Both lanes do it, because both re-send.

The reader's other guess — that the model's own reasoning was going back too —
is worth recording as false. Reasoning is display-only and was never on the
transcript; the 150,322 characters of it in that session were generated, not
re-read. That is a second and separate cost, and it is what `shift+tab` is for.

Usage is now reported however a turn ends. It was yielded only where a turn
finished cleanly, so the expensive ones — interrupted, or having lost the
server — recorded nothing, and a three-turn export showed one turn's figures
for the whole session. That is also what would have settled the question above
without arithmetic, so it is fixed first.

And `/help` is a command. It was going to the model, which answered with
nothing. The keys and the commands are the interface's own facts and it should
not have to ask anything to state them.

412 tests pass.

Deploy story

What this commit did to the running system — joined from the forge receipt chain, the part a commit page elsewhere cannot show.

Not deployed through the forge lane

No push, promotion, build, or deploy receipt references this commit (receipts are scanned over a bounded recent window). Changes shipped by full node replacement carry their proof in the release gate receipt instead.

Changed files

  • modified packages/openagents-cli/src/coder-ollama.ts
  • modified packages/openagents-cli/src/coder-session.ts
  • modified packages/openagents-cli/src/coder-thread.ts
  • modified packages/openagents-cli/test/coder-ollama.test.ts

Diff

4 files changed, +161 -13

packages/openagents-cli/src/coder-ollama.ts modified +46 -12

@@ -31,6 +31,27 @@ const DEFAULT_HOST = "http://127.0.0.1:11434";

31 31 
 */
32 32 
const MAX_TOOL_STEPS = 100;
33 33
34 
/**
35 
 * How much of one tool's output is kept on the transcript.
36 
 *
37 
 * The reader sees all of it; this is only what goes back to the model on every
38 
 * round after. A session that read the issue boards accumulated 82 KB of tool
39 
 * output and re-sent it 25 times, which is where its wall clock went: 91% of
40 
 * everything sent each round was output the model had already read.
41 
 *
42 
 * Generous enough that a normal command survives whole, and the head and the
43 
 * tail are what a long one is read for anyway.
44 
 */
45 
const TOOL_RESULT_KEPT = 4_000;
46
47 
/** A long tool result, kept at both ends. */
48 
const bounded = (output: string): string => {
49 
  if (output.length <= TOOL_RESULT_KEPT) return output;
50 
  const half = Math.floor(TOOL_RESULT_KEPT / 2);
51 
  const cut = output.length - TOOL_RESULT_KEPT;
52 
  return `${output.slice(0, half)}\n\n[${String(cut)} characters omitted from the middle; run it again more narrowly if you need them]\n\n${output.slice(-half)}`;
53 
};
54
34 55 
/**
35 56 
 * What a reasoning level means to Ollama.
36 57 
 *

@@ -233,6 +254,10 @@ export class OllamaReplySource implements ReplySource {

233 254 
   */
234 255 
  private steered: string[] = [];
235 256 
  private reasoningLevel: string;
257 
  /** What the turn in flight has spent, so it is reported however it ends. */
258 
  private spentIn = 0;
259 
  private spentOut = 0;
260 
  private calls = 0;
236 261 
  private callCount = 0;
237 262
238 263 
  get model(): string {

@@ -339,6 +364,18 @@ export class OllamaReplySource implements ReplySource {

339 364 
  }
340 365
341 366 
  private async *turn(prompt: string, signal: AbortSignal): AsyncIterable<ReplyChunk> {
367 
    // Reported on the way out however the turn ends. Yielding it only where a
368 
    // turn finished cleanly lost it on exactly the expensive ones: a long turn
369 
    // that was interrupted, or that lost the server, recorded nothing, and the
370 
    // export showed one turn's figures for a whole session.
371 
    try {
372 
      yield* this.rounds(prompt, signal);
373 
    } finally {
374 
      yield { type: "usage", promptTokens: this.spentIn, completionTokens: this.spentOut, calls: this.calls };
375 
    }
376 
  }
377
378 
  private async *rounds(prompt: string, signal: AbortSignal): AsyncIterable<ReplyChunk> {
342 379 
    // Built on the first turn rather than in the constructor: the tools are
343 380 
    // declared after construction, and the prompt is derived from them.
344 381 
    if (this.transcript.length === 0) {

@@ -350,9 +387,9 @@ export class OllamaReplySource implements ReplySource {

350 387 
    // A turn is a loop, not a single call: the model may answer, or it may ask
351 388 
    // for tools and then answer once it has seen what they returned. The
352 389 
    // ceiling is what stops a model that only ever delegates.
353 
    let promptTokens = 0;
354 
    let completionTokens = 0;
355 
    let llmCalls = 0;
390 
    this.spentIn = 0;
391 
    this.spentOut = 0;
392 
    this.calls = 0;
356 393
357 394 
    for (let step = 0; step < MAX_TOOL_STEPS; step += 1) {
358 395 
      if (signal.aborted) return;

@@ -432,9 +469,9 @@ export class OllamaReplySource implements ReplySource {

432 469 
            // The counts ride on the final chunk of each round, so they are
433 470 
            // summed across the rounds a turn took rather than reported from
434 471 
            // the last one.
435 
            promptTokens += chunk.prompt_eval_count ?? 0;
436 
            completionTokens += chunk.eval_count ?? 0;
437 
            llmCalls += 1;
472 
            this.spentIn += chunk.prompt_eval_count ?? 0;
473 
            this.spentOut += chunk.eval_count ?? 0;
474 
            this.calls += 1;
438 475 
            break;
439 476 
          }
440 477 
        }

@@ -452,10 +489,7 @@ export class OllamaReplySource implements ReplySource {

452 489 
        ...(calls.length === 0 ? {} : { tool_calls: calls }),
453 490 
      });
454 491
455 
      if (calls.length === 0) {
456 
        yield { type: "usage", promptTokens, completionTokens, calls: llmCalls };
457 
        return;
458 
      }
492 
      if (calls.length === 0) return;
459 493
460 494 
      for (const call of calls) {
461 495 
        if (signal.aborted) return;

@@ -463,7 +497,7 @@ export class OllamaReplySource implements ReplySource {

463 497 
      }
464 498 
    }
465 499
466 
    yield { type: "usage", promptTokens, completionTokens, calls: llmCalls };
500
467 501 
  }
468 502
469 503 
  /**

@@ -503,6 +537,6 @@ export class OllamaReplySource implements ReplySource {

503 537
504 538 
    yield { type: "tool_result", callId, output, error: failure };
505 539
506 
    this.transcript.push({ role: "tool", content: output, tool_name: name });
540 
    this.transcript.push({ role: "tool", content: bounded(output), tool_name: name });
507 541 
  }
508 542 
}
packages/openagents-cli/src/coder-session.ts modified +28

@@ -542,6 +542,34 @@ export class CoderSession {

542 542 
      return;
543 543 
    }
544 544
545 
    // `/help` was going to the model, which answered with nothing. The keys and
546 
    // the commands are the interface's own facts and it should not have to ask
547 
    // anything to state them.
548 
    if (/^\/(help|\?)\s*$/.test(prompt.trim())) {
549 
      this.entries.push({ role: "you", text: prompt, settled: true, at: Date.now() });
550 
      this.notice(
551 
        [
552 
          "Commands:",
553 
          "  /help     this list",
554 
          "  /system   what the model is told, including tools and skills",
555 
          "  /skills   choose which skills the model is offered",
556 
          "  /export   write this conversation as an ATIF trajectory",
557 
          "  /reload   rebuild and restart on the current source",
558 
          "  /delegate [<n>x] <prompt>   run child agents on a prompt",
559 
          "",
560 
          "Keys:",
561 
          "  enter        send, or steer a running turn",
562 
          "  shift+enter  queue for when the turn ends",
563 
          "  esc          interrupt the reply",
564 
          "  tab          switch model · shift+tab  change thinking",
565 
          "  ctrl+o       expand a tool call · ctrl+x  stop children",
566 
          "  pgup/pgdn    scroll · ctrl+d  quit",
567 
        ].join("\n"),
568 
      );
569 
      this.emit();
570 
      return;
571 
    }
572
545 573 
    const delegate = parseDelegateCommand(prompt);
546 574 
    if (delegate !== undefined) {
547 575 
      this.entries.push({ role: "you", text: prompt, settled: true, at: Date.now() });
packages/openagents-cli/src/coder-thread.ts modified +19 -1

@@ -79,6 +79,17 @@ const THREADS_PATH = "/api/v3/threads";

79 79 
 */
80 80 
const MAX_TOOL_STEPS = 100;
81 81
82 
/** How much of one tool's output is kept on the transcript. */
83 
const TOOL_RESULT_KEPT = 4_000;
84
85 
/** A long tool result, kept at both ends, which is what it is read for. */
86 
const boundedResult = (output: string): string => {
87 
  if (output.length <= TOOL_RESULT_KEPT) return output;
88 
  const half = Math.floor(TOOL_RESULT_KEPT / 2);
89 
  const cut = output.length - TOOL_RESULT_KEPT;
90 
  return `${output.slice(0, half)}\n\n[${String(cut)} characters omitted from the middle; run it again more narrowly if you need them]\n\n${output.slice(-half)}`;
91 
};
92
82 93 
/** What the thread may still spend, as the server last reported it. */
83 94 
export interface ThreadBudget {
84 95 
  readonly calls: number;

@@ -422,7 +433,14 @@ export class ThreadReplySource implements ReplySource {

422 433 
      role: "assistant",
423 434 
      content: `[tool call]\n${call.name}(${call.args})`,
424 435 
    });
425 
    this.transcript.push({ role: "user", content: `[tool result ${call.name}]\n${output}` });
436 
    // Bounded on the way onto the transcript, not on the way to the reader:
437 
    // this is what goes back to the model on every round after, and a session
438 
    // that re-sends everything it has already read spends its wall clock on
439 
    // reading it again.
440 
    this.transcript.push({
441 
      role: "user",
442 
      content: `[tool result ${call.name}]\n${boundedResult(output)}`,
443 
    });
426 444 
  }
427 445
428 446 
  /**
packages/openagents-cli/test/coder-ollama.test.ts modified +68

@@ -442,3 +442,71 @@ describe("how hard the model is asked to think", () => {

442 442 
    expect(new OllamaReplySource({ model: "m", reasoning: "low" }).reasoning.level).toBe("low");
443 443 
  });
444 444 
});
445
446 
describe("what goes back to the model each round", () => {
447 
  it("keeps a long tool result at both ends rather than whole", async () => {
448 
    const long = "A".repeat(3_000) + "MIDDLE" + "B".repeat(3_000);
449 
    const { source, stub } = sourceWith([
450 
      [
451 
        chunk({ content: "", tool_calls: [{ function: { name: "t", arguments: {} } }] }),
452 
        chunk({}, true),
453 
      ],
454 
      [chunk({ content: "done" }, true)],
455 
    ]);
456 
    source.useTools([
457 
      { name: "t", description: "d", parameters: {}, run: () => Promise.resolve(long) },
458 
    ]);
459
460 
    await collect(source, "go");
461
462 
    const messages = stub.requests[1]?.["messages"] as ReadonlyArray<Record<string, unknown>>;
463 
    const result = messages.find((message) => message["role"] === "tool");
464 
    const content = String(result?.["content"]);
465
466 
    // The reader saw all of it; this is what is re-sent on every round after,
467 
    // and a session that re-sends what it has already read spends its wall
468 
    // clock reading it again.
469 
    expect(content.length).toBeLessThan(long.length);
470 
    expect(content).toContain("characters omitted from the middle");
471 
    // Both ends survive, which is what a long output is read for.
472 
    expect(content.startsWith("A")).toBe(true);
473 
    expect(content.endsWith("B")).toBe(true);
474 
  });
475
476 
  it("leaves an ordinary result alone", async () => {
477 
    const { source, stub } = sourceWith([
478 
      [
479 
        chunk({ content: "", tool_calls: [{ function: { name: "t", arguments: {} } }] }),
480 
        chunk({}, true),
481 
      ],
482 
      [chunk({ content: "done" }, true)],
483 
    ]);
484 
    source.useTools([
485 
      { name: "t", description: "d", parameters: {}, run: () => Promise.resolve("short output") },
486 
    ]);
487
488 
    await collect(source, "go");
489
490 
    const messages = stub.requests[1]?.["messages"] as ReadonlyArray<Record<string, unknown>>;
491 
    expect(messages.find((message) => message["role"] === "tool")?.["content"]).toBe(
492 
      "short output",
493 
    );
494 
  });
495
496 
  it("reports what a turn spent even when it is cut short", async () => {
497 
    const { source } = sourceWith([
498 
      [{ message: {}, done: true, prompt_eval_count: 500, eval_count: 50 } as never],
499 
    ]);
500 
    const controller = new AbortController();
501 
    const chunks: unknown[] = [];
502
503 
    for await (const piece of source.reply("go", controller.signal)) {
504 
      chunks.push(piece);
505 
      controller.abort();
506 
    }
507
508 
    // The expensive turns are the ones that get interrupted, and those were
509 
    // recording nothing at all.
510 
    expect(chunks.at(-1)).toMatchObject({ type: "usage" });
511 
  });
512 
});

This page updates live while a promote is in flight · changelog