Run a turn's tool calls at once, and say how long a child has been running

0a86e618d385 · AtlantisPleb · · parent e66c1d21d7b8

Run a turn's tool calls at once, and say how long a child has been running

Two things from one session that looked stuck for two minutes and was not.

**A turn's tool calls ran in order.** A model asking for two tools in one turn
is saying they do not depend on each other, and both lanes ran anyway one after
the other: three minutes thirty-eight, then eleven seconds. The session then
reported that they "ran in parallel", which was the reasonable thing to believe
and not true. They now run together and the turn costs the slower of the two —
the same two-lane fan-out finishes in twenty-one seconds.

Order within one call is preserved, because a tool call and its result are a
sequence. A tool that throws no longer strands the others mid-flight: the first
failure is raised once every stream has stopped, so a caller never sees half a
turn's events and no error.

**A running child said only `Initializing…`.** It says that until its first tool
call arrives, and a recon child that thinks for ninety seconds before touching
anything showed it for all ninety. That is indistinguishable from a hang, which
is exactly how it was read. Every running row now carries its elapsed time.

Neither was a deadlock. The child in that session completed in about a hundred
seconds when run again on its own; what was missing was any way to tell slow
from stuck.

Still missing, and worth its own change: a delegated child has no deadline at
all. The only timer in the fleet is the kill grace after someone asks a child to
stop, so a child that genuinely hangs hangs the turn until the reader presses
ctrl+x.

438 tests pass, five of them on running streams together.

Deploy story

What this commit did to the running system — joined from the forge receipt chain, the part a commit page elsewhere cannot show.

Not deployed through the forge lane

No push, promotion, build, or deploy receipt references this commit (receipts are scanned over a bounded recent window). Changes shipped by full node replacement carry their proof in the release gate receipt instead.

Changed files

  • modified docs/assure-repo/false-green-candidates.v1.json
  • modified docs/assure-repo/surface-inventory.v1.json
  • modified packages/openagents-cli/src/coder-fleet.ts
  • added packages/openagents-cli/src/coder-merge.ts
  • modified packages/openagents-cli/src/coder-ollama.ts
  • modified packages/openagents-cli/src/coder-thread.ts
  • modified packages/openagents-cli/test/coder-delegate.test.ts
  • added packages/openagents-cli/test/coder-merge.test.ts

Diff

8 files changed, +161 -18

docs/assure-repo/false-green-candidates.v1.json modified +1 -1

@@ -4,7 +4,7 @@

4 4
  "note": "Heuristic false-green LEADS, not findings. A finding requires a demonstrated reproduction (surviving mutation via mutation-runner). Do not treat a candidate as a confirmed false green. Coverage-theater leads may include tests that delegate their assertion to a custom helper the classifier does not recognise; verify before acting.",
5 5
  "sourceDigest": "sha256:dd810dd48c5bdbc9becd7fcc01dd41a4ca2abf0b2d6f6a545907247f6e3e8361",
6 6
  "summary": {
7
    "filesScanned": 2439,
7
    "filesScanned": 2440,
8 8
    "candidateCount": 16,
9 9
    "byMode": {
10 10
      "false_green_coverage_theater": 15,
docs/assure-repo/surface-inventory.v1.json modified +2 -2

@@ -1,7 +1,7 @@

1 1
{
2 2
  "schemaVersion": "1",
3 3
  "repository": "OpenAgentsInc/openagents",
4
  "sourceDigest": "sha256:b66f940c59b2d89100cc77723ed92cd43247cc8b3b868330a5c2d39cdb6b33d9",
4
  "sourceDigest": "sha256:534cff090b222b33201687df5e82e45eb96b388e9dfe03bacd0b360e0e737184",
5 5
  "surfaces": [
6 6
    {
7 7
      "id": "app:@openagentsinc/acceptance-runner",

@@ -1876,7 +1876,7 @@

1876 1876
      "oracles": [
1877 1877
        {
1878 1878
          "type": "test",
1879
          "ref": "packages/openagents-cli (39 tracked test files)"
1879
          "ref": "packages/openagents-cli (40 tracked test files)"
1880 1880
        },
1881 1881
        {
1882 1882
          "type": "behavior-contract",
packages/openagents-cli/src/coder-fleet.ts modified +14 -5

@@ -73,7 +73,7 @@ export function fleetPhrase(tasks: ReadonlyArray<CoderTask>): string | undefined

73 73
 * wrong. A running child with no activity yet says `Initializing…` rather than
74 74
 * nothing, because an empty cell reads as a stalled child.
75 75
 */
76
export function taskActivity(task: CoderTask): string {
76
export function taskActivity(task: CoderTask, now: number = Date.now()): string {
77 77
  if (task.status === "pending") return "Queued";
78 78
  if (task.status === "stopped") return "Stopped";
79 79
  if (task.status === "failed") return `Failed: ${collapse(task.error ?? "unknown error")}`;

@@ -86,10 +86,15 @@ export function taskActivity(task: CoderTask): string {

86 86
    return `Done (${cost})`;
87 87
  }
88 88
89
  // A running child says how long it has been running, because the reader's
90
  // question is never "what is it doing" alone — it is "is this slow or is it
91
  // stuck". A recon child that thinks for ninety seconds before its first tool
92
  // call showed `Initializing…` for all ninety, which reads as stuck.
93
  const running = formatDuration(now - task.startedAt);
89 94
  const activity = task.progress.lastActivity;
90
  if (activity === undefined) return "Initializing…";
95
  if (activity === undefined) return `Initializing… (${running})`;
91 96
  const target = activity.target === undefined ? "" : `(${collapse(activity.target)})`;
92
  return `${activity.toolName}${target}`;
97
  return `${activity.toolName}${target} (${running})`;
93 98
}
94 99
95 100
/**

@@ -119,7 +124,11 @@ export function taskCounters(task: CoderTask): string {

119 124
 * lines up, which is what lets a reader scan fifteen children for the one that
120 125
 * failed.
121 126
 */
122
export function fleetRows(tasks: ReadonlyArray<CoderTask>, width: number): ReadonlyArray<FleetRow> {
127
export function fleetRows(
128
  tasks: ReadonlyArray<CoderTask>,
129
  width: number,
130
  now: number = Date.now(),
131
): ReadonlyArray<FleetRow> {
123 132
  const room = Math.max(20, width);
124 133
  const descriptionRoom = Math.min(28, Math.max(12, Math.floor(room * 0.35)));
125 134
  const longest = tasks.reduce((most, task) => Math.max(most, task.description.length), 0);

@@ -129,7 +138,7 @@ export function fleetRows(tasks: ReadonlyArray<CoderTask>, width: number): Reado

129 138
    const branch = index === tasks.length - 1 ? "└─" : "├─";
130 139
    const description = pad(cut(task.description, column), column);
131 140
    const counters = taskCounters(task);
132
    const activity = taskActivity(task);
141
    const activity = taskActivity(task, now);
133 142
    const tail = counters.length > 0 ? `${activity} · ${counters}` : activity;
134 143
    return {
135 144
      status: task.status,
packages/openagents-cli/src/coder-merge.ts added +64

@@ -0,0 +1,64 @@

1
/**
2
 * Run several event streams at once and yield what they produce as it arrives.
3
 *
4
 * A model that asks for two tools in one turn is saying they do not depend on
5
 * each other. Running them in order anyway made a session that fanned out to two
6
 * models take the sum of both — three minutes and thirty-eight seconds, then
7
 * eleven — and report that they had run in parallel, which was the reasonable
8
 * thing to believe and not true.
9
 *
10
 * Order within one stream is preserved, because a tool call and its result are
11
 * a sequence. Order between streams is arrival order, which is the point.
12
 */
13
export async function* merge<T>(
14
  streams: ReadonlyArray<AsyncIterable<T>>,
15
): AsyncIterable<T> {
16
  if (streams.length === 0) return;
17
  if (streams.length === 1) {
18
    yield* streams[0] as AsyncIterable<T>;
19
    return;
20
  }
21
22
  const queue: T[] = [];
23
  let live = streams.length;
24
  let failure: unknown;
25
  let wake: (() => void) | undefined;
26
27
  const nudge = () => {
28
    wake?.();
29
    wake = undefined;
30
  };
31
32
  for (const stream of streams) {
33
    void (async () => {
34
      try {
35
        for await (const item of stream) {
36
          queue.push(item);
37
          nudge();
38
        }
39
      } catch (cause) {
40
        // The first failure is kept and raised once every stream has stopped:
41
        // a tool that threw must not strand the others mid-flight, and a caller
42
        // that saw half a turn's events and no error would report a turn that
43
        // did not happen.
44
        failure ??= cause;
45
      } finally {
46
        live -= 1;
47
        nudge();
48
      }
49
    })();
50
  }
51
52
  while (live > 0 || queue.length > 0) {
53
    while (queue.length > 0) {
54
      const next = queue.shift();
55
      if (next !== undefined) yield next;
56
    }
57
    if (live === 0) break;
58
    await new Promise<void>((resolve) => {
59
      wake = resolve;
60
    });
61
  }
62
63
  if (failure !== undefined) throw failure;
64
}
packages/openagents-cli/src/coder-ollama.ts modified +6 -4

@@ -14,6 +14,7 @@

14 14
import { Ollama } from "ollama";
15 15
import type { Message as OllamaMessage, Tool as OllamaTool, ToolCall as OllamaToolCall } from "ollama";
16 16
17
import { merge } from "./coder-merge.js";
17 18
import type { ReplyChunk, ReplySource } from "./coder-session.js";
18 19
import type { CoderTool } from "./coder-tools.js";
19 20

@@ -498,10 +499,11 @@ export class OllamaReplySource implements ReplySource {

498 499
499 500
      if (calls.length === 0) return;
500 501
501
      for (const call of calls) {
502
        if (signal.aborted) return;
503
        yield* this.invoke(call, signal);
504
      }
502
      // Concurrently. A model asking for two tools in one turn is saying they do
503
      // not depend on each other, and running them in order anyway makes a fan-out
504
      // to two models cost the sum of both.
505
      if (signal.aborted) return;
506
      yield* merge(calls.map((call) => this.invoke(call, signal)));
505 507
    }
506 508
507 509
packages/openagents-cli/src/coder-thread.ts modified +6 -4

@@ -63,6 +63,7 @@

63 63
import { Redacted } from "effect";
64 64
65 65
import type { ChildGrant } from "./coder-child-gateway.js";
66
import { merge } from "./coder-merge.js";
66 67
import type { ReplyChunk, ReplySource } from "./coder-session.js";
67 68
import type { CoderTool } from "./coder-tools.js";
68 69

@@ -380,10 +381,11 @@ export class ThreadReplySource implements ReplySource {

380 381
          continue;
381 382
        }
382 383
383
        for (const call of calls) {
384
          if (signal.aborted) return;
385
          yield* this.invoke(call, signal);
386
        }
384
        // Concurrently. A model asking for two tools in one turn is saying they do
385
        // not depend on each other, and running them in order anyway makes a fan-out
386
        // to two models cost the sum of both.
387
        if (signal.aborted) return;
388
        yield* merge(calls.map((call) => this.invoke(call, signal)));
387 389
      }
388 390
    } finally {
389 391
      // Read the budget on the way out of every turn, including an interrupted
packages/openagents-cli/test/coder-delegate.test.ts modified +5 -2

@@ -319,9 +319,12 @@ describe("fleet rendering", () => {

319 319
    const running = registry.list()[0];
320 320
    expect(running).toBeDefined();
321 321
    if (running === undefined) return;
322
    expect(taskActivity(running)).toBe("bash(pnpm test)");
322
    // A running child says how long it has been running, so the reader can
323
    // tell a slow child from a stuck one.
324
    const now = running.startedAt + 95_000;
325
    expect(taskActivity(running, now)).toBe("bash(pnpm test) (1m 35s)");
323 326
    expect(fleetPhrase([running])).toBe("1 agent");
324
    const rows = fleetRows([running], 80);
327
    const rows = fleetRows([running], 80, now);
325 328
    expect(rows[0]?.branch).toBe("└─");
326 329
    expect(rows[0]?.mark).toBe("◐");
327 330
    expect(rows[0]?.text).toContain("bash(pnpm test)");
packages/openagents-cli/test/coder-merge.test.ts added +63

@@ -0,0 +1,63 @@

1
import { describe, expect, it } from "vitest";
2
3
import { merge } from "../src/coder-merge.js";
4
5
const after = (ms: number, ...items: string[]): AsyncIterable<string> => ({
6
  async *[Symbol.asyncIterator]() {
7
    for (const item of items) {
8
      await new Promise((resolve) => setTimeout(resolve, ms));
9
      yield item;
10
    }
11
  },
12
});
13
14
describe("running streams together", () => {
15
  it("yields in arrival order, not in the order the streams were given", async () => {
16
    const seen: string[] = [];
17
    for await (const item of merge([after(40, "slow"), after(5, "fast")])) seen.push(item);
18
19
    // The whole point: a fan-out to two models should cost the slower of the
20
    // two rather than the sum.
21
    expect(seen).toEqual(["fast", "slow"]);
22
  });
23
24
  it("takes the time of the slowest, not the total", async () => {
25
    const started = Date.now();
26
    for await (const _ of merge([after(60, "a"), after(60, "b"), after(60, "c")])) void _;
27
28
    expect(Date.now() - started).toBeLessThan(150);
29
  });
30
31
  it("keeps each stream's own order", async () => {
32
    const seen: string[] = [];
33
    for await (const item of merge([after(10, "1", "2", "3")])) seen.push(item);
34
35
    // A tool call and its result are a sequence, whatever else is running.
36
    expect(seen).toEqual(["1", "2", "3"]);
37
  });
38
39
  it("lets the others finish before it raises a failure", async () => {
40
    const angry: AsyncIterable<string> = {
41
      // eslint-disable-next-line require-yield -- it only throws
42
      async *[Symbol.asyncIterator]() {
43
        throw new Error("tool exploded");
44
      },
45
    };
46
    const seen: string[] = [];
47
48
    await expect(
49
      (async () => {
50
        for await (const item of merge([angry, after(20, "survivor")])) seen.push(item);
51
      })(),
52
    ).rejects.toThrow("tool exploded");
53
54
    // A tool that threw must not strand the others mid-flight.
55
    expect(seen).toEqual(["survivor"]);
56
  });
57
58
  it("passes a single stream straight through", async () => {
59
    const seen: string[] = [];
60
    for await (const item of merge([after(1, "only")])) seen.push(item);
61
    expect(seen).toEqual(["only"]);
62
  });
63
});

This page updates live while a promote is in flight · changelog