Run Devin children over ACP, so a running one can be seen

d9a54393a29d · AtlantisPleb · · parent 867d7101d5e7

Run Devin children over ACP, so a running one can be seen

Two Devin children showed `Initializing…` for four and a half minutes and
looked hung. They were not hung — they had no way to say anything.

`devin -p` is a black box. Measured: fourteen seconds of silence for a one-word
answer, then the whole thing at once; `--export` is written at close too. So a
child doing real work reported no tool calls, no tokens, and wrote no
transcript at all. The fleet said `Initializing…` because that is what a
running child with no recorded activity says, and opening one showed an empty
screen. Nothing distinguished four minutes of work from four minutes of nothing.

`devin acp` is the same agent as an Agent Client Protocol server over stdio,
and it streams what the fleet already draws. Live, against the real binary:

    +0.1s   session thundering-timbale
    +1.8s   tokens in=16023 out=75
    +1.8s   tool  execute — Ran ls
    +14.6s  tokens in=16272 out=214
    +14.6s  tool  execute — Ran date
    +19.3s  text  **`ls /tmp | head -3`** …

Every one of those arrived at 19.3s before, as one blob.

The protocol is recorded to the child's transcript as it arrives, so opening a
running Devin child now shows what it has run and what came back. The reader
takes the three update kinds worth reading — the tool call, its output, the
answer — out of the hundreds a child writes; the handshake and the thinking,
which arrives a token at a time, are not shown.

`--permission-mode dangerous` becomes a `bypass` session, which is ACP's word
for the same posture, so a caller that wrote the old name does not have to
learn the new one.

Three things found while building it, each of which would have read as
something else:

A permission request has no one to answer it. A delegated child is launched
unattended, so an unanswered request would hang it for as long as the reader
left it. It answers with the option that lets the work continue.

The transcript was a write stream, and `end()` does not flush before the next
line runs — a child read the instant it finished showed a transcript missing
everything it had just done. Appended synchronously now.

A missing binary raises `error` and then `close`, and the close rejects
everything still pending, so the later failure won: `devin-does-not-exist`
reported "the agent exited before it answered" instead of "not on PATH". The
first failure is kept. And a stopped fleet is not a failed child — killing the
agent closes its stdio, which would have made every `ctrl+x` look like a crash.

723 tests pass, where 719 did.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012TRDRrfL1khQhQtNr3SRrA
Co-Authored-By
Claude Opus 5 (1M context) <noreply@anthropic.com>

Deploy story

What this commit did to the running system — joined from the forge receipt chain, the part a commit page elsewhere cannot show.

Not deployed through the forge lane

No push, promotion, build, or deploy receipt references this commit (receipts are scanned over a bounded recent window). Changes shipped by full node replacement carry their proof in the release gate receipt instead.

Changed files

  • modified packages/openagents-cli/src/coder-child-transcript.ts
  • modified packages/openagents-cli/src/coder-delegate.ts
  • added packages/openagents-cli/src/coder-devin-acp.ts
  • modified packages/openagents-cli/test/coder-child-transcript.test.ts
  • modified packages/openagents-cli/test/coder-delegate-devin.test.ts

Diff

5 files changed, +644 -129

packages/openagents-cli/src/coder-child-transcript.ts modified +53

@@ -108,6 +108,9 @@ const parseLine = (line: string): ChildEntry | undefined => {

108 108
  const own = fromSelfHarness(record);
109 109
  if (own !== undefined) return own;
110 110
111
  const acp = fromAcp(record);
112
  if (acp !== undefined) return acp;
113
111 114
  // Anything else is a harness with its own event stream, and `opencode`'s is
112 115
  // the one the fleet already reads live.
113 116
  const event = parseOpencodeEvent(trimmed);

@@ -125,6 +128,56 @@ const parseLine = (line: string): ChildEntry | undefined => {

125 128
  }
126 129
};
127 130
131
/**
132
 * A Devin child, which writes the ACP protocol it spoke.
133
 *
134
 * The harness records every JSON-RPC message so a reader can open a running
135
 * child and watch it work. Most of those messages are not worth showing — the
136
 * handshake, the mode, dozens of thought fragments a token at a time — so this
137
 * takes the three that are: what it ran, what came back, and what it said.
138
 */
139
const fromAcp = (message: Record<string, unknown>): ChildEntry | undefined => {
140
  if (message["method"] !== "session/update") return undefined;
141
142
  const update = nested(nested(message["params"])["update"]);
143
  const kind = update["sessionUpdate"];
144
145
  if (kind === "tool_call") {
146
    // Devin's `title` is already the phrase a person would read — "Ran ls",
147
    // "Read src/a.ts" — so it is the target rather than a name to look up.
148
    const name = text(update["kind"]) ?? "tool";
149
    return { kind: "tool", name, target: text(update["title"]) };
150
  }
151
152
  if (kind === "tool_call_update") {
153
    const said = firstContentText(update["content"]);
154
    return said === undefined ? undefined : { kind: "output", text: said };
155
  }
156
157
  if (kind === "agent_message_chunk") {
158
    const said = text(nested(update["content"])["text"]);
159
    return said === undefined ? undefined : { kind: "text", text: said };
160
  }
161
162
  return undefined;
163
};
164
165
/** The first piece of text inside an ACP content array, if there is one. */
166
const firstContentText = (content: unknown): string | undefined => {
167
  if (!Array.isArray(content)) return undefined;
168
  for (const part of content) {
169
    const inner = nested(nested(part)["content"]);
170
    const said = text(inner["text"]);
171
    if (said !== undefined) return said;
172
  }
173
  return undefined;
174
};
175
176
const nested = (value: unknown): Record<string, unknown> =>
177
  typeof value === "object" && value !== null && !Array.isArray(value)
178
    ? (value as Record<string, unknown>)
179
    : {};
180
128 181
const fromSelfHarness = (record: Record<string, unknown>): ChildEntry | undefined => {
129 182
  const type = record["type"];
130 183
packages/openagents-cli/src/coder-delegate.ts modified +55 -90

@@ -31,7 +31,9 @@

31 31
32 32
import type { ChildProcess } from "node:child_process";
33 33
import { execFileSync, spawn } from "node:child_process";
34
import { createWriteStream, mkdirSync } from "node:fs";
34
import { appendFileSync, createWriteStream, mkdirSync } from "node:fs";
35
36
import { runDevinAcp } from "./coder-devin-acp.js";
35 37
import { tmpdir } from "node:os";
36 38
import { join } from "node:path";
37 39
import type { CoderTaskId, CoderTaskRegistry, CoderToolActivity } from "./coder-tasks.js";

@@ -403,11 +405,21 @@ export class DevinHarness implements DelegateHarness {

403 405
  readonly model: string;
404 406
405 407
  constructor(private readonly options: DevinHarnessOptions = {}) {
406
    // Devin picks its own model from its own configuration, and print mode does
407
    // not report which. Naming one here would be inventing it.
408
    // Devin picks its own model from its own configuration, and neither print
409
    // mode nor ACP reports which. Naming one here would be inventing it.
408 410
    this.model = options.permissionMode ?? "dangerous";
409 411
  }
410 412
413
  /**
414
   * Run the child over ACP rather than print mode.
415
   *
416
   * `devin -p` writes nothing until it finishes — measured: fourteen seconds of
417
   * silence for a one-word answer, and `--export` is written at close too. A
418
   * child doing four minutes of real work reported no tool calls, no tokens,
419
   * and no transcript, so the fleet showed `Initializing…` for the whole run
420
   * and a reader could not tell it from a hang. `devin acp` is the same agent
421
   * streaming the events the fleet already draws.
422
   */
411 423
  async *run(
412 424
    input: {
413 425
      readonly prompt: string;

@@ -417,99 +429,52 @@ export class DevinHarness implements DelegateHarness {

417 429
    },
418 430
    signal: AbortSignal,
419 431
  ): AsyncIterable<DelegateEvent> {
420
    const command = this.options.command ?? "devin";
421
    const mode = this.options.permissionMode ?? "dangerous";
422
423
    const child = spawn(
424
      command,
425
      [
426
        "-p",
427
        input.prompt,
428
        "--permission-mode",
429
        mode,
430
        // Print mode cannot show the trust prompt, so without this a child in a
431
        // directory nobody has opened Devin in exits before doing anything.
432
        "--respect-workspace-trust",
433
        "false",
434
      ],
435
      {
436
        cwd: input.cwd,
437
        env: { ...process.env, ...this.options.env },
438
        // No terminal: a child that would prompt gets end-of-file and stops
439
        // rather than waiting where the fleet shows it as still working.
440
        stdio: ["ignore", "pipe", "pipe"],
441
      },
442
    );
443
444
    const events: DelegateEvent[] = [];
445
    let resolveNext: (() => void) | undefined;
446
    const wake = () => {
447
      resolveNext?.();
448
      resolveNext = undefined;
449
    };
450
451
    let answer = "";
452
    let failure = "";
453
    let done = false;
454
    let startFailure: Error | undefined;
455
456
    child.stdout.setEncoding("utf8");
457
    child.stdout.on("data", (chunk: string) => {
458
      answer += chunk;
459
    });
460
    child.stderr.setEncoding("utf8");
461
    child.stderr.on("data", (chunk: string) => {
462
      failure += chunk;
463
    });
464
465
    const onAbort = () => child.kill("SIGKILL");
466
    signal.addEventListener("abort", onAbort, { once: true });
467
468
    child.on("error", (cause: Error) => {
469
      startFailure =
470
        (cause as NodeJS.ErrnoException).code === "ENOENT"
471
          ? new Error(`The \`${command}\` command is not on PATH.`)
472
          : cause;
473
      done = true;
474
      wake();
475
    });
476
    child.on("close", (code: number | null) => {
477
      const text = answer.trim();
478
      if (code === 0) {
479
        if (text.length > 0) events.push({ type: "text", value: text });
480
      } else {
481
        const said = `${failure.trim()}\n${text}`.trim();
482
        events.push({
483
          type: "error",
484
          message:
485
            said.length > 0
486
              ? said
487
              : `The \`${command}\` child exited with code ${String(code ?? -1)}.`,
488
        });
489
      }
490
      done = true;
491
      wake();
492
    });
493
494 432
    try {
495
      for (;;) {
496
        while (events.length > 0) {
497
          const next = events.shift();
498
          if (next !== undefined) yield next;
499
        }
500
        if (done) break;
501
        await new Promise<void>((resolve) => {
502
          resolveNext = resolve;
503
        });
504
      }
433
      yield* runDevinAcp(
434
        { prompt: input.prompt, cwd: input.cwd },
435
        {
436
          ...(this.options.command === undefined ? {} : { command: this.options.command }),
437
          ...(this.options.env === undefined ? {} : { env: this.options.env }),
438
          // Devin's own word for what this harness has always called a
439
          // permission mode. `bypass` is its `dangerous`.
440
          mode: DEVIN_MODES[this.options.permissionMode ?? "dangerous"] ?? "bypass",
441
          // Appended synchronously as each message arrives, not through a
442
          // stream: a stream's `end()` does not flush before the next line of
443
          // this process runs, so a child read the instant it finished showed
444
          // a transcript missing everything it had just done.
445
          record: (entry) => {
446
            try {
447
              appendFileSync(input.transcriptPath, `${JSON.stringify(entry)}\n`);
448
            } catch {
449
              // A transcript that cannot be written must not end the child.
450
            }
451
          },
452
        },
453
        signal,
454
      );
505 455
    } finally {
506
      signal.removeEventListener("abort", onAbort);
456
      // Nothing to close: every line was already on disk.
507 457
    }
508
509
    if (startFailure !== undefined) throw startFailure;
510 458
  }
511 459
}
512 460
461
/**
462
 * The old permission-mode names, mapped to the session modes ACP offers.
463
 *
464
 * `devin -p --permission-mode dangerous` and a `bypass` ACP session are the
465
 * same posture, and a caller that wrote the old name should not have to learn
466
 * the new one.
467
 */
468
const DEVIN_MODES: Readonly<Record<string, string>> = {
469
  dangerous: "bypass",
470
  bypass: "bypass",
471
  auto: "accept-edits",
472
  "accept-edits": "accept-edits",
473
  ask: "ask",
474
  plan: "plan",
475
  smart: "smart",
476
};
477
513 478
/**
514 479
 * The models a child is given, in the order they are preferred.
515 480
 *
packages/openagents-cli/src/coder-devin-acp.ts added +308

@@ -0,0 +1,308 @@

1
import { spawn } from "node:child_process";
2
3
import type { DelegateEvent } from "./coder-delegate.js";
4
5
/**
6
 * A Devin child over ACP, so the fleet can see it working.
7
 *
8
 * Devin's print mode (`devin -p`) is a black box. It writes nothing to stdout
9
 * until the very end — measured: fourteen seconds of silence, then the whole
10
 * answer at once — and `--export` is written at close too. A child doing four
11
 * minutes of real work therefore reported no tool calls, no tokens, and no
12
 * transcript, so the fleet showed `Initializing…` for the whole run and a
13
 * reader could not tell it from a hang.
14
 *
15
 * `devin acp` is the same agent as an Agent Client Protocol server over stdio,
16
 * and it streams: `tool_call` with a title, `tool_call_update` with a status,
17
 * `usage_update` with real token counts, and `agent_message_chunk` for the
18
 * answer. That is every event the fleet already knows how to draw.
19
 *
20
 * Newline-delimited JSON-RPC, and one server per child. A shared server would
21
 * save a process per child and cost a lifecycle nobody asked for: a crash
22
 * would take every child with it, and a child that hangs would hold the
23
 * server's queue.
24
 *
25
 * Devin logs heavily to stderr and none of it is protocol. It is drained and
26
 * dropped rather than parsed.
27
 */
28
29
/** What a running Devin child reports, normalized. */
30
export interface DevinAcpOptions {
31
  readonly command?: string | undefined;
32
  /**
33
   * The session mode, which is Devin's own word for what the old harness passed
34
   * as `--permission-mode`. `bypass` is the equivalent of `dangerous`.
35
   */
36
  readonly mode?: string | undefined;
37
  readonly env?: Record<string, string> | undefined;
38
  /** Every protocol message, for the transcript. */
39
  readonly record?: ((entry: Record<string, unknown>) => void) | undefined;
40
}
41
42
interface Pending {
43
  readonly resolve: (result: Record<string, unknown>) => void;
44
  readonly reject: (cause: Error) => void;
45
}
46
47
export async function* runDevinAcp(
48
  input: { readonly prompt: string; readonly cwd: string },
49
  options: DevinAcpOptions,
50
  signal: AbortSignal,
51
): AsyncIterable<DelegateEvent> {
52
  const command = options.command ?? "devin";
53
  const child = spawn(command, ["acp"], {
54
    cwd: input.cwd,
55
    env: { ...process.env, ...options.env },
56
    stdio: ["pipe", "pipe", "pipe"],
57
    // Its own process group, so stopping a child stops what the child started.
58
    detached: true,
59
  });
60
61
  // Devin's own logging. Not protocol, and it is a lot of it.
62
  child.stderr.resume();
63
64
  const events: DelegateEvent[] = [];
65
  const pending = new Map<number, Pending>();
66
  let wake: (() => void) | undefined;
67
  let finished = false;
68
  let failure: Error | undefined;
69
  let sequence = 0;
70
  let buffer = "";
71
  let answer = "";
72
73
  const nudge = () => {
74
    wake?.();
75
    wake = undefined;
76
  };
77
78
  const send = (method: string, params?: Record<string, unknown>) =>
79
    new Promise<Record<string, unknown>>((resolve, reject) => {
80
      const id = (sequence += 1);
81
      pending.set(id, { resolve, reject });
82
      child.stdin.write(`${JSON.stringify({ jsonrpc: "2.0", id, method, params })}\n`);
83
    });
84
85
  const reply = (id: number, result: Record<string, unknown>) => {
86
    child.stdin.write(`${JSON.stringify({ jsonrpc: "2.0", id, result })}\n`);
87
  };
88
89
  child.stdout.setEncoding("utf8");
90
  child.stdout.on("data", (chunk: string) => {
91
    buffer += chunk;
92
    for (;;) {
93
      const at = buffer.indexOf("\n");
94
      if (at === -1) break;
95
      const line = buffer.slice(0, at).trim();
96
      buffer = buffer.slice(at + 1);
97
      if (line.length === 0) continue;
98
99
      let message: Record<string, unknown>;
100
      try {
101
        message = JSON.parse(line) as Record<string, unknown>;
102
      } catch {
103
        // Not protocol. Devin writes plain lines too.
104
        continue;
105
      }
106
107
      options.record?.(message);
108
      handle(message);
109
    }
110
    nudge();
111
  });
112
113
  const handle = (message: Record<string, unknown>) => {
114
    const id = message["id"];
115
116
    // A reply to something asked for.
117
    if (typeof id === "number" && message["method"] === undefined) {
118
      const waiting = pending.get(id);
119
      pending.delete(id);
120
      if (waiting === undefined) return;
121
      const error = message["error"];
122
      if (error !== undefined) {
123
        waiting.reject(new Error(`Devin refused ${JSON.stringify(error).slice(0, 200)}`));
124
      } else {
125
        waiting.resolve((message["result"] ?? {}) as Record<string, unknown>);
126
      }
127
      return;
128
    }
129
130
    const method = message["method"];
131
132
    // A request from the agent. The only one that matters is permission, and a
133
    // delegated child has no one to ask: it was launched to run unattended, so
134
    // an unanswered request would hang it for as long as the reader left it.
135
    if (typeof id === "number" && method === "session/request_permission") {
136
      const params = record(message["params"]);
137
      const chosen = firstAllowOption(params);
138
      reply(id, {
139
        outcome:
140
          chosen === undefined
141
            ? { outcome: "cancelled" }
142
            : { outcome: "selected", optionId: chosen },
143
      });
144
      return;
145
    }
146
147
    if (method !== "session/update") return;
148
149
    const update = record(record(message["params"])["update"]);
150
    const kind = update["sessionUpdate"];
151
152
    if (kind === "tool_call") {
153
      const callId = text(update["toolCallId"]) ?? `devin_${String(events.length)}`;
154
      // Devin's `title` is already the phrase a person would read — "Ran ls",
155
      // "Read src/a.ts" — so it is the activity rather than a name to look up.
156
      events.push({
157
        type: "tool",
158
        callId,
159
        name: text(update["kind"]) ?? "tool",
160
        target: text(update["title"]),
161
      });
162
      return;
163
    }
164
165
    if (kind === "usage_update") {
166
      const meta = record(update["_meta"]);
167
      const input_tokens = number(meta["cognition.ai/inputTokens"]);
168
      const output_tokens = number(meta["cognition.ai/outputTokens"]);
169
      if (input_tokens !== undefined && output_tokens !== undefined) {
170
        events.push({ type: "tokens", input: input_tokens, output: output_tokens });
171
      }
172
      return;
173
    }
174
175
    if (kind === "agent_message_chunk") {
176
      const piece = text(record(update["content"])["text"]);
177
      if (piece !== undefined) answer += piece;
178
    }
179
  };
180
181
  child.on("error", (cause: Error) => {
182
    failure =
183
      (cause as NodeJS.ErrnoException).code === "ENOENT"
184
        ? new Error(`The \`${command}\` command is not on PATH.`)
185
        : cause;
186
    finished = true;
187
    nudge();
188
  });
189
190
  child.on("close", () => {
191
    finished = true;
192
    for (const waiting of pending.values()) {
193
      waiting.reject(new Error("The Devin agent exited before it answered."));
194
    }
195
    pending.clear();
196
    nudge();
197
  });
198
199
  const stop = () => {
200
    try {
201
      process.kill(-child.pid!, "SIGKILL");
202
    } catch {
203
      child.kill("SIGKILL");
204
    }
205
  };
206
  signal.addEventListener("abort", stop, { once: true });
207
208
  // The conversation, driven from here while the generator yields whatever the
209
  // agent has said since the last time it was asked.
210
  const turn = (async () => {
211
    await send("initialize", {
212
      protocolVersion: 1,
213
      clientCapabilities: { fs: { readTextFile: false, writeTextFile: false } },
214
    });
215
216
    const opened = await send("session/new", { cwd: input.cwd, mcpServers: [] });
217
    const sessionId = text(opened["sessionId"]);
218
    if (sessionId === undefined) throw new Error("Devin opened no session.");
219
    events.push({ type: "session", sessionId });
220
221
    const mode = options.mode;
222
    if (mode !== undefined) {
223
      // Best effort. A build of Devin without this mode should not lose the
224
      // child over the name of a permission setting.
225
      await send("session/set_mode", { sessionId, modeId: mode }).catch(() => ({}));
226
    }
227
228
    await send("session/prompt", {
229
      sessionId,
230
      prompt: [{ type: "text", text: input.prompt }],
231
    });
232
  })();
233
234
  turn.catch((cause: unknown) => {
235
    // Whichever failure came first. A missing binary raises `error` and then
236
    // `close`, and the close rejects everything still pending — so taking the
237
    // later one reports "the agent exited before it answered" for a command
238
    // that was never there.
239
    failure ??= cause instanceof Error ? cause : new Error(String(cause));
240
  });
241
242
  const settled = turn.then(
243
    () => {
244
      finished = true;
245
      nudge();
246
    },
247
    () => {
248
      finished = true;
249
      nudge();
250
    },
251
  );
252
253
  try {
254
    for (;;) {
255
      while (events.length > 0) {
256
        const event = events.shift();
257
        if (event !== undefined) yield event;
258
      }
259
      if (finished) break;
260
      if (signal.aborted) return;
261
      await new Promise<void>((resolve) => {
262
        wake = resolve;
263
      });
264
    }
265
266
    await settled;
267
268
    // A stopped fleet is not a failed child. Killing the agent closes its
269
    // stdio, which rejects everything still in flight, and reporting that as
270
    // an error would make every `ctrl+x` look like a crash.
271
    if (signal.aborted) return;
272
273
    if (failure !== undefined) {
274
      yield { type: "error", message: failure.message };
275
      throw failure;
276
    }
277
278
    const said = answer.trim();
279
    if (said.length > 0) yield { type: "text", value: said };
280
  } finally {
281
    signal.removeEventListener("abort", stop);
282
    stop();
283
  }
284
}
285
286
/** The option a permission request offers that lets the work continue. */
287
const firstAllowOption = (params: Record<string, unknown>): string | undefined => {
288
  const options = params["options"];
289
  if (!Array.isArray(options)) return undefined;
290
291
  const ranked = options
292
    .map((option) => record(option))
293
    .filter((option) => text(option["optionId"]) !== undefined);
294
295
  const allow = ranked.find((option) => String(option["kind"] ?? "").startsWith("allow"));
296
  return text((allow ?? ranked[0] ?? {})["optionId"]);
297
};
298
299
const record = (value: unknown): Record<string, unknown> =>
300
  typeof value === "object" && value !== null && !Array.isArray(value)
301
    ? (value as Record<string, unknown>)
302
    : {};
303
304
const text = (value: unknown): string | undefined =>
305
  typeof value === "string" && value.length > 0 ? value : undefined;
306
307
const number = (value: unknown): number | undefined =>
308
  typeof value === "number" && Number.isFinite(value) ? value : undefined;
packages/openagents-cli/test/coder-child-transcript.test.ts modified +67

@@ -94,3 +94,70 @@ describe("reading a child that has written nothing", () => {

94 94
    expect(readChildTranscript(undefined)).toEqual([]);
95 95
  });
96 96
});
97
98
describe("reading a child Devin ran", () => {
99
  it("reads the ACP protocol the Devin harness records", () => {
100
    const path = transcript([
101
      { jsonrpc: "2.0", id: 1, result: { protocolVersion: 1 } },
102
      {
103
        jsonrpc: "2.0",
104
        method: "session/update",
105
        params: {
106
          sessionId: "s",
107
          update: {
108
            sessionUpdate: "tool_call",
109
            toolCallId: "functions.exec:0",
110
            title: "Ran ls",
111
            kind: "execute",
112
          },
113
        },
114
      },
115
      {
116
        jsonrpc: "2.0",
117
        method: "session/update",
118
        params: {
119
          sessionId: "s",
120
          update: {
121
            sessionUpdate: "tool_call_update",
122
            toolCallId: "functions.exec:0",
123
            status: "in_progress",
124
            content: [{ type: "content", content: { type: "text", text: "a.txt" } }],
125
          },
126
        },
127
      },
128
      {
129
        jsonrpc: "2.0",
130
        method: "session/update",
131
        params: {
132
          sessionId: "s",
133
          update: { sessionUpdate: "agent_message_chunk", content: { type: "text", text: "done" } },
134
        },
135
      },
136
    ]);
137
138
    expect(readChildTranscript(path)).toEqual([
139
      { kind: "tool", name: "execute", target: "Ran ls" },
140
      { kind: "output", text: "a.txt" },
141
      { kind: "text", text: "done" },
142
    ]);
143
  });
144
145
  it("shows none of the handshake, and none of the thinking a token at a time", () => {
146
    // A Devin child writes hundreds of messages and three kinds are worth
147
    // reading. Showing the rest would bury what it actually did.
148
    const path = transcript([
149
      { jsonrpc: "2.0", id: 2, result: { sessionId: "ses_x" } },
150
      {
151
        jsonrpc: "2.0",
152
        method: "session/update",
153
        params: {
154
          sessionId: "s",
155
          update: { sessionUpdate: "agent_thought_chunk", content: { type: "text", text: "hmm" } },
156
        },
157
      },
158
      { jsonrpc: "2.0", method: "_cognition.ai/output", params: { message: "Connecting to MCP" } },
159
    ]);
160
161
    expect(readChildTranscript(path)).toEqual([]);
162
  });
163
});
packages/openagents-cli/test/coder-delegate-devin.test.ts modified +161 -39

@@ -1,17 +1,75 @@

1
import { chmodSync, mkdtempSync, writeFileSync } from "node:fs";
1
import { chmodSync, mkdtempSync, readFileSync, writeFileSync } from "node:fs";
2 2
import { tmpdir } from "node:os";
3 3
import { join } from "node:path";
4 4
import { describe, expect, it } from "vitest";
5 5
6 6
import { DevinHarness, type DelegateEvent } from "../src/coder-delegate.js";
7 7
8
/** A stand-in for the binary, so the tests cost nothing and never call out. */
9
const fake = (script: string): string => {
10
  const directory = mkdtempSync(join(tmpdir(), "devin-harness-"));
11
  const path = join(directory, "devin-stub");
12
  writeFileSync(path, `#!/bin/sh\n${script}\n`);
13
  chmodSync(path, 0o755);
14
  return path;
8
/**
9
 * A stand-in for `devin acp`, so the tests cost nothing and never call out.
10
 *
11
 * It speaks the protocol the harness speaks: newline-delimited JSON-RPC on
12
 * stdio, replying to `initialize`, `session/new`, and `session/prompt`, and
13
 * sending whatever `session/update` notifications the test asks for.
14
 */
15
const fakeAgent = (options: {
16
  readonly updates?: ReadonlyArray<Record<string, unknown>>;
17
  readonly answer?: string;
18
  readonly failPrompt?: string;
19
  readonly hang?: boolean;
20
  readonly recordArgsTo?: string;
21
} = {}): string => {
22
  const directory = mkdtempSync(join(tmpdir(), "devin-acp-"));
23
  const script = join(directory, "agent.mjs");
24
25
  writeFileSync(
26
    script,
27
    `
28
import { appendFileSync } from "node:fs";
29
const recordTo = ${JSON.stringify(options.recordArgsTo ?? null)};
30
if (recordTo) appendFileSync(recordTo, process.argv.slice(2).join(" ") + "\\n");
31
if (${options.hang === true}) { setInterval(() => {}, 1000); }
32
33
let buffer = "";
34
const write = (m) => process.stdout.write(JSON.stringify(m) + "\\n");
35
process.stdin.setEncoding("utf8");
36
process.stdin.on("data", (chunk) => {
37
  buffer += chunk;
38
  for (;;) {
39
    const at = buffer.indexOf("\\n");
40
    if (at === -1) break;
41
    const line = buffer.slice(0, at).trim();
42
    buffer = buffer.slice(at + 1);
43
    if (!line) continue;
44
    const m = JSON.parse(line);
45
    if (recordTo) appendFileSync(recordTo, m.method + " " + JSON.stringify(m.params ?? {}) + "\\n");
46
    if (m.method === "initialize") write({ jsonrpc: "2.0", id: m.id, result: { protocolVersion: 1 } });
47
    else if (m.method === "session/new") write({ jsonrpc: "2.0", id: m.id, result: { sessionId: "ses_test" } });
48
    else if (m.method === "session/set_mode") write({ jsonrpc: "2.0", id: m.id, result: {} });
49
    else if (m.method === "session/prompt") {
50
      for (const update of ${JSON.stringify(options.updates ?? [])})
51
        write({ jsonrpc: "2.0", method: "session/update", params: { sessionId: "ses_test", update } });
52
      ${
53
        options.answer === undefined
54
          ? ""
55
          : `write({ jsonrpc: "2.0", method: "session/update", params: { sessionId: "ses_test", update: { sessionUpdate: "agent_message_chunk", content: { type: "text", text: ${JSON.stringify(options.answer)} } } } });`
56
      }
57
      ${
58
        options.failPrompt === undefined
59
          ? `write({ jsonrpc: "2.0", id: m.id, result: { stopReason: "end_turn" } });`
60
          : `write({ jsonrpc: "2.0", id: m.id, error: { code: -32000, message: ${JSON.stringify(options.failPrompt)} } });`
61
      }
62
      if (!${options.hang === true}) setTimeout(() => process.exit(0), 10);
63
    }
64
  }
65
});
66
`,
67
  );
68
69
  const shim = join(directory, "devin-stub");
70
  writeFileSync(shim, `#!/bin/sh\nexec ${process.execPath} ${script} "$@"\n`);
71
  chmodSync(shim, 0o755);
72
  return shim;
15 73
};
16 74
17 75
const collect = async (

@@ -19,10 +77,8 @@ const collect = async (

19 77
  cwd = process.cwd(),
20 78
): Promise<ReadonlyArray<DelegateEvent>> => {
21 79
  const events: DelegateEvent[] = [];
22
  for await (const event of harness.run(
23
    { prompt: "do the thing", cwd, transcriptPath: join(tmpdir(), "unused.jsonl") },
24
    new AbortController().signal,
25
  )) {
80
  const transcriptPath = join(mkdtempSync(join(tmpdir(), "devin-t-")), "child.jsonl");
81
  for await (const event of harness.run({ prompt: "do the thing", cwd, transcriptPath }, new AbortController().signal)) {
26 82
    events.push(event);
27 83
  }
28 84
  return events;

@@ -36,44 +92,109 @@ describe("running children on the Devin CLI", () => {

36 92
  });
37 93
38 94
  it("reports the child's answer as text", async () => {
39
    const events = await collect(new DevinHarness({ command: fake('echo "PONG"') }));
95
    const events = await collect(new DevinHarness({ command: fakeAgent({ answer: "PONG" }) }));
40 96
41
    expect(events).toEqual([{ type: "text", value: "PONG" }]);
97
    expect(events.at(-1)).toEqual({ type: "text", value: "PONG" });
42 98
  });
43 99
44
  it("passes the prompt, the unattended mode, and the trust flag", async () => {
45
    // Print mode cannot show the trust prompt, so without the flag a child in a
46
    // directory nobody has opened Devin in exits before doing anything.
47
    const events = await collect(new DevinHarness({ command: fake('echo "$@"') }));
100
  it("reports what the child is doing while it does it", async () => {
101
    // The whole reason this harness speaks ACP. In print mode a child doing
102
    // four minutes of work reported nothing at all until it finished, so the
103
    // fleet showed `Initializing…` for the entire run and a reader could not
104
    // tell it from a hang.
105
    const events = await collect(
106
      new DevinHarness({
107
        command: fakeAgent({
108
          updates: [
109
            {
110
              sessionUpdate: "tool_call",
111
              toolCallId: "functions.exec:0",
112
              title: "Ran ls",
113
              kind: "execute",
114
            },
115
            {
116
              sessionUpdate: "usage_update",
117
              used: 16_086,
118
              _meta: { "cognition.ai/inputTokens": 16_020, "cognition.ai/outputTokens": 66 },
119
            },
120
          ],
121
          answer: "done",
122
        }),
123
      }),
124
    );
48 125
49
    const said = (events[0] as { value: string }).value;
50
    expect(said).toContain("-p do the thing");
51
    expect(said).toContain("--permission-mode dangerous");
52
    expect(said).toContain("--respect-workspace-trust false");
126
    expect(events).toContainEqual({
127
      type: "tool",
128
      callId: "functions.exec:0",
129
      name: "execute",
130
      target: "Ran ls",
131
    });
132
    expect(events).toContainEqual({ type: "tokens", input: 16_020, output: 66 });
133
    expect(events.at(-1)).toEqual({ type: "text", value: "done" });
53 134
  });
54 135
55
  it("takes a different permission mode when one is asked for", async () => {
56
    const harness = new DevinHarness({ command: fake('echo "$@"'), permissionMode: "auto" });
136
  it("reports the session, so a retry can resume it", async () => {
137
    const events = await collect(new DevinHarness({ command: fakeAgent({ answer: "x" }) }));
57 138
58
    const said = ((await collect(harness))[0] as { value: string }).value;
59
    expect(said).toContain("--permission-mode auto");
60
    // The mode is what the fleet shows beside the agent, because print mode
61
    // does not report which model answered.
62
    expect(harness.model).toBe("auto");
139
    expect(events[0]).toEqual({ type: "session", sessionId: "ses_test" });
63 140
  });
64 141
65
  it("reports a failing child as an error, with what it said", async () => {
66
    const events = await collect(
67
      new DevinHarness({ command: fake('echo "went wrong" >&2; exit 2') }),
68
    );
142
  it("writes the protocol to the transcript as it arrives", async () => {
143
    const transcriptPath = join(mkdtempSync(join(tmpdir(), "devin-t-")), "child.jsonl");
144
    const harness = new DevinHarness({
145
      command: fakeAgent({
146
        updates: [
147
          { sessionUpdate: "tool_call", toolCallId: "c1", title: "Read a.ts", kind: "read" },
148
        ],
149
        answer: "ok",
150
      }),
151
    });
152
153
    for await (const _event of harness.run(
154
      { prompt: "x", cwd: process.cwd(), transcriptPath },
155
      new AbortController().signal,
156
    )) {
157
      void _event;
158
    }
159
160
    // A child that is killed still leaves what it had done behind, and opening
161
    // a running child shows something rather than an empty screen.
162
    const written = readFileSync(transcriptPath, "utf8");
163
    expect(written).toContain("Read a.ts");
164
  });
165
166
  it("asks for the unattended session mode, in Devin's own word for it", async () => {
167
    const log = join(mkdtempSync(join(tmpdir(), "devin-args-")), "log");
168
    const harness = new DevinHarness({
169
      command: fakeAgent({ answer: "x", recordArgsTo: log }),
170
      permissionMode: "dangerous",
171
    });
172
173
    await collect(harness);
174
175
    const said = readFileSync(log, "utf8");
176
    expect(said).toContain("acp");
177
    // `bypass` is ACP's name for what the flag called `dangerous`. A caller
178
    // that wrote the old name should not have to learn the new one.
179
    expect(said).toContain(`"modeId":"bypass"`);
180
    // The mode is what the fleet shows beside the agent, because neither print
181
    // mode nor ACP reports which model answered.
182
    expect(harness.model).toBe("dangerous");
183
  });
184
185
  it("carries the prompt through as a content block", async () => {
186
    const log = join(mkdtempSync(join(tmpdir(), "devin-args-")), "log");
187
    await collect(new DevinHarness({ command: fakeAgent({ answer: "x", recordArgsTo: log }) }));
69 188
70
    expect(events).toEqual([{ type: "error", message: "went wrong" }]);
189
    expect(readFileSync(log, "utf8")).toContain(`"text":"do the thing"`);
71 190
  });
72 191
73
  it("says so when a failing child said nothing at all", async () => {
74
    const events = await collect(new DevinHarness({ command: fake("exit 3") }));
192
  it("reports a refused prompt as an error rather than an empty child", async () => {
193
    const harness = new DevinHarness({
194
      command: fakeAgent({ failPrompt: "the model provider is unavailable" }),
195
    });
75 196
76
    expect((events[0] as { message: string }).message).toContain("exited with code 3");
197
    await expect(collect(harness)).rejects.toThrow(/model provider is unavailable/);
77 198
  });
78 199
79 200
  it("throws when the binary is not on PATH, rather than reporting an empty child", async () => {

@@ -84,13 +205,14 @@ describe("running children on the Devin CLI", () => {

84 205
  });
85 206
86 207
  it("stops when the fleet is stopped", async () => {
87
    const harness = new DevinHarness({ command: fake("sleep 30") });
208
    const harness = new DevinHarness({ command: fakeAgent({ hang: true }) });
88 209
    const controller = new AbortController();
89 210
    const events: DelegateEvent[] = [];
211
    const transcriptPath = join(mkdtempSync(join(tmpdir(), "devin-t-")), "child.jsonl");
90 212
91 213
    const running = (async () => {
92 214
      for await (const event of harness.run(
93
        { prompt: "x", cwd: process.cwd(), transcriptPath: "/tmp/x" },
215
        { prompt: "x", cwd: process.cwd(), transcriptPath },
94 216
        controller.signal,
95 217
      )) {
96 218
        events.push(event);

This page updates live while a promote is in flight · changelog