Stage the coder's optimizable text as versioned artifacts

cf63540b5abe · AtlantisPleb · · parent 3d188862756c

Stage the coder's optimizable text as versioned artifacts

The system prompt and the tool descriptions were string literals in three
places -- `crates/coder-lite`, `crates/openagents-cli`, and
`packages/openagents-cli`. That is fine for a human editing one sentence and
wrong for everything the autoimprovement plan wants to do with it: a lever
cannot be diffed as an artifact, two cycles cannot be compared by digest, and
an optimizer has nothing to mutate.

They now live in `surfaces/coder/` as flat key-to-text maps with a schema id
and a `sha256` content digest, in the knowledge base's shape: corpus file,
build step, pinned digest. `pnpm run build:coder-surfaces` regenerates the two
modules the CLIs compile -- `crates/openagents-cli/src/surfaces.rs` and
`packages/openagents-cli/src/coder-surfaces.generated.ts` -- and re-pins
`index.json`. `pnpm run check:coder-surfaces` refuses a tree where an artifact
and its build disagree, inside `check:fast`, because a surface edited without
the rebuild is the knowledge base's failure mode: it ships nothing and says
nothing while it does.

BEHAVIOR IS UNCHANGED. This is a move, and it is proven to be one rather than
asserted: goldens were captured from the pre-change code and the composed
prompts and tool descriptions are byte-identical after it
(`crates/*/tests/coder-surfaces-golden.json`, and a TypeScript capture diffed
before and after). Where the two harnesses had already forked -- the local-lane
notice, `You have 1 tools`, the Rust `shell` description missing the
capability-first steer -- the fork is preserved key by key rather than
reconciled. Reconciling it is a hillclimb cycle with its own measured delta
(#119), not a refactor.

The catalog-lines artifact is a mirror and the docs say so. A plugin's catalog
line was never a literal: it is the top-level `description` of
`plugins/<id>/manifest.json`, discovered at runtime by `discover_catalog()`.
The manifest stays where that text is edited; the artifact gives it one
diffable object and one digest, and the check fails when the two disagree.

Run rows now name the text that produced them. A `--plain` session announces
`[oa:surfaces <id>=<digest>,...]` beside its thread line; `coder-effectiveness`
reads it out of the trial rather than off the repository at scoring time, folds
it into the run digest, and records it on an `openagents.bench_result.v3` row.
So two runs differing only in the prompt are different runs, they stay
comparable (the suite key is untouched), and `effectiveness:compare` says
"staged text also varies" instead of letting a text change read as noise. The
v2 rows already in `bench-results/` keep verifying: the store is append-only
and the receipt digests the keys a row actually carries, so a v2 row names no
staged text rather than a borrowed one.

The candidate format is documented, not redefined. `docs/coder/candidate-format.md`
explains `openagents.coder_candidate.v1` as implemented for #121 and pins the
surface-id vocabulary it draws on, so a review proposal and an optimizer
mutation stay one object.

Closes the staging half of OpenAgentsInc/openagents#122.

Deploy story

What this commit did to the running system — joined from the forge receipt chain, the part a commit page elsewhere cannot show.

Not deployed through the forge lane

No push, promotion, build, or deploy receipt references this commit (receipts are scanned over a bounded recent window). Changes shipped by full node replacement carry their proof in the release gate receipt instead.

Changed files

  • modified crates/coder-lite/src/runtime.rs
  • added crates/coder-lite/tests/coder-surfaces-golden.json
  • added crates/coder-lite/tests/coder_surfaces_golden.rs
  • modified crates/openagents-cli/src/lib.rs
  • modified crates/openagents-cli/src/plugins.rs
  • modified crates/openagents-cli/src/runtime.rs
  • added crates/openagents-cli/src/surfaces.rs
  • modified crates/openagents-cli/src/tools.rs
  • added crates/openagents-cli/tests/coder-surfaces-golden.json
  • added crates/openagents-cli/tests/coder_surfaces_golden.rs
  • modified docs/assure-repo/false-green-candidates.v1.json
  • modified docs/assure-repo/surface-inventory.v1.json
  • added docs/coder/candidate-format.md
  • modified package.json
  • modified packages/coder-effectiveness/src/compare.ts
  • modified packages/coder-effectiveness/src/effectiveness.test.ts
  • modified packages/coder-effectiveness/src/effectiveness.ts
  • modified packages/coder-effectiveness/src/harbor-job.ts
  • modified packages/coder-effectiveness/src/results-store.test.ts
  • modified packages/coder-effectiveness/src/results-store.ts
  • added packages/coder-effectiveness/src/surface-pin.test.ts
  • modified packages/coder-effectiveness/src/thresholds.test.ts
  • modified packages/openagents-cli/src/cli.ts
  • modified packages/openagents-cli/src/coder-capability.ts
  • modified packages/openagents-cli/src/coder-goals.ts
  • modified packages/openagents-cli/src/coder-remember.ts
  • added packages/openagents-cli/src/coder-surfaces.generated.ts
  • modified packages/openagents-cli/src/coder-system.ts
  • modified packages/openagents-cli/src/coder-tool-families.ts
  • modified packages/openagents-cli/src/coder-tools.ts
  • added packages/openagents-cli/test/coder-surfaces.test.ts
  • added scripts/coder-surfaces-literal-search.mjs
  • added scripts/coder-surfaces.mjs
  • added surfaces/coder/README.md
  • added surfaces/coder/catalog-lines.v1.json
  • added surfaces/coder/index.json
  • added surfaces/coder/system-prompt.v1.json
  • added surfaces/coder/tool-descriptions.v1.json

Diff

38 files changed, +1608 -205

crates/coder-lite/src/runtime.rs modified +13 -21

@@ -7,7 +7,9 @@

7 7
//!
8 8
//! What this file owns is the part that is coder-lite's:
9 9
//!
10
//! - [`SYSTEM_INSTRUCTIONS`], carried verbatim. It is the reason the session
10
//! - The system prompt, composed from the staged `system-prompt` surface
11
//!   (`surfaces/coder/system-prompt.v1.json`) and carried verbatim. It is
12
//!   the reason the session
11 13
//!   answers as a terminal rather than as an assistant, and a merge that
12 14
//!   reworded it would have changed the product.
13 15
//! - [`Control`], the one-way channel the TUI loop reads. Text, tool calls,

@@ -27,14 +29,12 @@ use std::sync::atomic::{AtomicBool, Ordering};

27 29
use std::sync::mpsc::Sender;
28 30
use std::sync::{Arc, Mutex};
29 31
32
use openagents_cli::surfaces::system_prompt as prompt;
30 33
use openagents_cli::runtime::{
31 34
    ChatMessage, CoderRuntimeSession, Lane, ToolEvent, TurnUsage,
32 35
};
33 36
use openagents_cli::tools::{DelegationGate, HarnessToolRegistry, ToolDefinition};
34 37
35
/// coder-lite's voice. Carried verbatim from the first version of this file;
36
/// see the module header for why it does not move.
37
const SYSTEM_INSTRUCTIONS: &str = "You are OpenAgents Coder. Do not say you are from Google, Anthropic, OpenAI, or any other company. Do not mention your model, training, or architecture. Respond as a neutral, terse terminal: no greetings, no \"As an AI\", no explanations of your role, and no unnecessary padding. Use short sentences and dense, factual output. Answer questions directly. Output only code and minimal context when asked for code.";
38 38
39 39
type Failure = Box<dyn std::error::Error + Send + Sync>;
40 40

@@ -82,32 +82,24 @@ pub fn send(sink: &Sink, message: Control) {

82 82
83 83
/// The system message this session opens with.
84 84
///
85
/// [`SYSTEM_INSTRUCTIONS`] first and unchanged, then the tools — because a
85
/// The staged instructions first and unchanged, then the tools — because a
86 86
/// model that is told it has no tools when it has five will not use them, and
87 87
/// one told it has tools it does not have will claim to have run them. The
88 88
/// list is generated from what was actually declared, so the two cannot
89 89
/// disagree.
90 90
pub fn system_prompt(tools: &[ToolDefinition]) -> String {
91
    let mut lines = vec![SYSTEM_INSTRUCTIONS.to_string(), String::new()];
91
    let mut lines = vec![prompt::CODER_LITE_INSTRUCTIONS.to_string(), String::new()];
92 92
    if tools.is_empty() {
93
        lines.push(prompt::CODER_LITE_NO_TOOLS.to_string());
94
    } else {
93 95
        lines.push(
94
            "You have no tools in this session: you cannot read or write files, run commands, or \
95
             reach anything outside this conversation. Say plainly when something would need a \
96
             tool you do not have."
97
                .to_string(),
96
            prompt::CODER_LITE_TOOL_LIST_HEADER.replace("{count}", &tools.len().to_string()),
98 97
        );
99
    } else {
100
        lines.push(format!("You have {} tools, and no others:", tools.len()));
101 98
        for tool in tools {
102 99
            lines.push(format!("- `{}`", tool.name));
103 100
        }
104 101
        lines.push(String::new());
105
        lines.push(
106
            "That list is complete: a capability not on it is one you do not have. Read a tool's \
107
             description before assuming what it covers. Never say you ran something you did not \
108
             run."
109
                .to_string(),
110
        );
102
        lines.push(prompt::CODER_LITE_TOOL_LIST_CLOSING.to_string());
111 103
    }
112 104
    lines.join("\n")
113 105
}

@@ -446,11 +438,11 @@ mod tests {

446 438
    fn the_system_prompt_opens_with_the_terse_instructions_unchanged() {
447 439
        let prompt = system_prompt(&[]);
448 440
        assert!(
449
            prompt.starts_with(SYSTEM_INSTRUCTIONS),
441
            prompt.starts_with(prompt::CODER_LITE_INSTRUCTIONS),
450 442
            "the instructions were not carried verbatim: {prompt}"
451 443
        );
452
        assert!(SYSTEM_INSTRUCTIONS.contains("no greetings"));
453
        assert!(SYSTEM_INSTRUCTIONS.contains("no unnecessary padding"));
444
        assert!(prompt::CODER_LITE_INSTRUCTIONS.contains("no greetings"));
445
        assert!(prompt::CODER_LITE_INSTRUCTIONS.contains("no unnecessary padding"));
454 446
    }
455 447
456 448
    /// A model told it has tools it does not have will claim to have run them.
crates/coder-lite/tests/coder-surfaces-golden.json added +5

@@ -0,0 +1,5 @@

1
{
2
  "system_prompt.no_tools": "You are OpenAgents Coder. Do not say you are from Google, Anthropic, OpenAI, or any other company. Do not mention your model, training, or architecture. Respond as a neutral, terse terminal: no greetings, no \"As an AI\", no explanations of your role, and no unnecessary padding. Use short sentences and dense, factual output. Answer questions directly. Output only code and minimal context when asked for code.\n\nYou have no tools in this session: you cannot read or write files, run commands, or reach anything outside this conversation. Say plainly when something would need a tool you do not have.",
3
  "system_prompt.one_tool": "You are OpenAgents Coder. Do not say you are from Google, Anthropic, OpenAI, or any other company. Do not mention your model, training, or architecture. Respond as a neutral, terse terminal: no greetings, no \"As an AI\", no explanations of your role, and no unnecessary padding. Use short sentences and dense, factual output. Answer questions directly. Output only code and minimal context when asked for code.\n\nYou have 1 tools, and no others:\n- `alpha`\n\nThat list is complete: a capability not on it is one you do not have. Read a tool's description before assuming what it covers. Never say you ran something you did not run.",
4
  "system_prompt.two_tools": "You are OpenAgents Coder. Do not say you are from Google, Anthropic, OpenAI, or any other company. Do not mention your model, training, or architecture. Respond as a neutral, terse terminal: no greetings, no \"As an AI\", no explanations of your role, and no unnecessary padding. Use short sentences and dense, factual output. Answer questions directly. Output only code and minimal context when asked for code.\n\nYou have 2 tools, and no others:\n- `alpha`\n- `beta`\n\nThat list is complete: a capability not on it is one you do not have. Read a tool's description before assuming what it covers. Never say you ran something you did not run."
5
}
crates/coder-lite/tests/coder_surfaces_golden.rs added +62

@@ -0,0 +1,62 @@

1
//! coder-lite's own voice, pinned byte for byte.
2
//!
3
//! The system prompt this crate composes is a staged artifact
4
//! (`surfaces/coder/system-prompt.v1.json`, OpenAgentsInc/openagents#122): a
5
//! change to it is meant to be a diff over that artifact with a measured
6
//! delta behind it, never a quiet edit in passing. This test composes the
7
//! prompt from fixed inputs and compares it with `coder-surfaces-golden.json`,
8
//! so moving the text out of `runtime.rs` and into the artifact had to be a
9
//! move and nothing else.
10
//!
11
//! Re-pin deliberately, with `UPDATE_CODER_SURFACES_GOLDEN=1 cargo test -p
12
//! coder-lite --test coder_surfaces_golden`, and only when the change to the
13
//! text is the change you meant to make.
14
15
use std::collections::BTreeMap;
16
use std::path::PathBuf;
17
18
use coder_lite::runtime::system_prompt;
19
use openagents_cli::tools::ToolDefinition;
20
21
fn declared(name: &str) -> ToolDefinition {
22
    ToolDefinition {
23
        name: name.to_string(),
24
        description: format!("A declared {name}."),
25
        parameters: serde_json::json!({"type": "object"}),
26
    }
27
}
28
29
fn captured() -> BTreeMap<String, String> {
30
    let mut out = BTreeMap::new();
31
    out.insert("system_prompt.no_tools".to_string(), system_prompt(&[]));
32
    out.insert(
33
        "system_prompt.one_tool".to_string(),
34
        system_prompt(&[declared("alpha")]),
35
    );
36
    out.insert(
37
        "system_prompt.two_tools".to_string(),
38
        system_prompt(&[declared("alpha"), declared("beta")]),
39
    );
40
    out
41
}
42
43
#[test]
44
fn the_composed_prompt_matches_the_pinned_golden() {
45
    let golden_path = PathBuf::from(env!("CARGO_MANIFEST_DIR"))
46
        .join("tests")
47
        .join("coder-surfaces-golden.json");
48
    let captured = captured();
49
50
    if std::env::var("UPDATE_CODER_SURFACES_GOLDEN").is_ok() {
51
        std::fs::write(
52
            &golden_path,
53
            format!("{}\n", serde_json::to_string_pretty(&captured).unwrap()),
54
        )
55
        .unwrap();
56
        return;
57
    }
58
59
    let raw = std::fs::read_to_string(&golden_path).expect("the golden file is checked in");
60
    let expected: BTreeMap<String, String> = serde_json::from_str(&raw).unwrap();
61
    assert_eq!(captured, expected);
62
}
crates/openagents-cli/src/lib.rs modified +1

@@ -35,6 +35,7 @@ pub mod repo;

35 35
pub mod resume;
36 36
pub mod runtime;
37 37
pub mod signals;
38
pub mod surfaces;
38 39
pub mod tools;
39 40
pub mod trace;
40 41
pub mod trace_client;
crates/openagents-cli/src/plugins.rs modified +1 -8

@@ -1229,14 +1229,7 @@ pub fn plugin_tool_definition(plugin: &LoadedPlugin) -> crate::tools::ToolDefini

1229 1229
pub fn capability_tool_definition() -> crate::tools::ToolDefinition {
1230 1230
    crate::tools::ToolDefinition {
1231 1231
        name: "capability".to_string(),
1232
        description: "Discover and load installed plugin capabilities: sandboxed, digest-pinned \
1233
             WebAssembly programs this machine already holds for common agent work. Before writing \
1234
             a script for a task, search here first — a capability that covers it is bounded, \
1235
             reviewable, and returns structured output. Call with `query` describing what you need \
1236
             to get the best matches; then call again with `name` set to the exact returned name \
1237
             to load it and make its dedicated tool available. Every later call to the loaded \
1238
             capability uses that exact name as the tool name."
1239
            .to_string(),
1232
        description: crate::surfaces::tool_descriptions::RUST_CAPABILITY.to_string(),
1240 1233
        parameters: serde_json::json!({
1241 1234
            "type": "object",
1242 1235
            "properties": {
crates/openagents-cli/src/runtime.rs modified +10 -24

@@ -119,15 +119,9 @@ pub enum ToolEvent {

119 119
/// and the observer usually outlives the call that installed it.
120 120
pub type ToolObserver = Arc<dyn Fn(ToolEvent) + Send + Sync>;
121 121
122
pub const THREAD_LANE_NOTICE: &str =
123
    "You answer through the OpenAgents inference proxy, on a thread opened for this session. \
124
    Every round of tool calls re-sends the whole conversation to a metered model, so batch \
125
    independent commands into one call and keep large dumps out of the transcript.";
122
pub const THREAD_LANE_NOTICE: &str = crate::surfaces::system_prompt::CODER_LANE_THREAD;
126 123
127
pub const LOCAL_LANE_NOTICE: &str =
128
    "You answer from a model running on this machine through Ollama. Nothing in this \
129
    conversation leaves the machine and nothing is metered, but the context window is a \
130
    fraction of a hosted model's, so keep large dumps out of the transcript.";
124
pub const LOCAL_LANE_NOTICE: &str = crate::surfaces::system_prompt::CODER_LANE_LOCAL_RUST;
131 125
132 126
/// Where an Ollama server listens unless `OPENAGENTS_OLLAMA_HOST` says otherwise.
133 127
pub const OLLAMA_HOST: &str = "http://127.0.0.1:11434";

@@ -719,34 +713,26 @@ impl CoderRuntimeSession {

719 713
        } else {
720 714
            THREAD_LANE_NOTICE
721 715
        };
716
        use crate::surfaces::system_prompt as prompt;
722 717
        let mut lines = vec![
723
            format!("You are `openagents coder`, a coding assistant in a terminal. {notice}"),
718
            prompt::CODER_OPENING.replace("{lane}", notice),
724 719
            "".to_string(),
725
            "Answer very concisely unless the reader asks for a longer response.".to_string(),
720
            prompt::CODER_CONCISION.to_string(),
726 721
            "".to_string(),
727 722
        ];
728 723
729 724
        if tool_defs.is_empty() {
725
            lines.push(prompt::CODER_NO_TOOLS.to_string());
726
        } else {
730 727
            lines.push(
731
                "You have no tools in this session: you cannot read or write files, run commands, or \
732
                reach anything outside this conversation. Answer from what the reader tells you, and \
733
                say plainly when something would need a tool you do not have.".to_string()
728
                prompt::CODER_TOOL_LIST_HEADER_RUST
729
                    .replace("{count}", &tool_defs.len().to_string()),
734 730
            );
735
        } else {
736
            lines.push(format!(
737
                "You have {} tools, and no others:",
738
                tool_defs.len()
739
            ));
740 731
            for t in tool_defs {
741 732
                lines.push(format!("- `{}`", t.name));
742 733
            }
743 734
            lines.push("".to_string());
744
            lines.push(
745
                "That list is complete: a capability not on it is one you do not have, whatever a model \
746
                like you usually has. Read a tool's description before assuming what it covers. Where \
747
                a description says what a child agent can do, that is the child's capability and not \
748
                yours. Never say you ran something you did not run.".to_string()
749
            );
735
            lines.push(prompt::CODER_TOOL_LIST_CLOSING.to_string());
750 736
        }
751 737
752 738
        // Skill injection. The `skill` tool's catalog is names and
crates/openagents-cli/src/surfaces.rs added +57

@@ -0,0 +1,57 @@

1
// @generated by scripts/coder-surfaces.mjs — do not edit.
2
//
3
// The staged coder text surfaces (`surfaces/coder/`), embedded at build
4
// time. Editing a sentence here is editing a build output: the artifact
5
// is the source, `pnpm run build:coder-surfaces` is the build, and
6
// `pnpm run check:coder-surfaces` refuses a tree where the two disagree.
7
//
8
//! Staged coder text surfaces, embedded from `surfaces/coder/`.
9
10
/// The system prompt surface: `surfaces/coder/system-prompt.v1.json`.
11
pub mod system_prompt {
12
    /// `coder_lite.instructions`
13
    pub const CODER_LITE_INSTRUCTIONS: &str = "You are OpenAgents Coder. Do not say you are from Google, Anthropic, OpenAI, or any other company. Do not mention your model, training, or architecture. Respond as a neutral, terse terminal: no greetings, no \"As an AI\", no explanations of your role, and no unnecessary padding. Use short sentences and dense, factual output. Answer questions directly. Output only code and minimal context when asked for code.";
14
    /// `coder_lite.no_tools`
15
    pub const CODER_LITE_NO_TOOLS: &str = "You have no tools in this session: you cannot read or write files, run commands, or reach anything outside this conversation. Say plainly when something would need a tool you do not have.";
16
    /// `coder_lite.tool_list_header`
17
    pub const CODER_LITE_TOOL_LIST_HEADER: &str = "You have {count} tools, and no others:";
18
    /// `coder_lite.tool_list_closing`
19
    pub const CODER_LITE_TOOL_LIST_CLOSING: &str = "That list is complete: a capability not on it is one you do not have. Read a tool's description before assuming what it covers. Never say you ran something you did not run.";
20
    /// `coder.opening`
21
    pub const CODER_OPENING: &str = "You are `openagents coder`, a coding assistant in a terminal. {lane}";
22
    /// `coder.concision`
23
    pub const CODER_CONCISION: &str = "Answer very concisely unless the reader asks for a longer response.";
24
    /// `coder.no_tools`
25
    pub const CODER_NO_TOOLS: &str = "You have no tools in this session: you cannot read or write files, run commands, or reach anything outside this conversation. Answer from what the reader tells you, and say plainly when something would need a tool you do not have.";
26
    /// `coder.tool_list_header.rust`
27
    pub const CODER_TOOL_LIST_HEADER_RUST: &str = "You have {count} tools, and no others:";
28
    /// `coder.tool_list_closing`
29
    pub const CODER_TOOL_LIST_CLOSING: &str = "That list is complete: a capability not on it is one you do not have, whatever a model like you usually has. Read a tool's description before assuming what it covers. Where a description says what a child agent can do, that is the child's capability and not yours. Never say you ran something you did not run.";
30
    /// `coder.lane.local.rust`
31
    pub const CODER_LANE_LOCAL_RUST: &str = "You answer from a model running on this machine through Ollama. Nothing in this conversation leaves the machine and nothing is metered, but the context window is a fraction of a hosted model's, so keep large dumps out of the transcript.";
32
    /// `coder.lane.thread`
33
    pub const CODER_LANE_THREAD: &str = "You answer through the OpenAgents inference proxy, on a thread opened for this session. Every round of tool calls re-sends the whole conversation to a metered model, so batch independent commands into one call and keep large dumps out of the transcript.";
34
}
35
36
/// The tool-description surface: `surfaces/coder/tool-descriptions.v1.json`.
37
pub mod tool_descriptions {
38
    /// `rust.shell`
39
    pub const RUST_SHELL: &str = "Run a shell command on this machine. The working directory is {cwd}, so paths are relative to it and you do not need to ask where you are. Returns combined stdout and stderr with the exit code. Batch independent commands into one call with && instead of one call each: every call replays the conversation so far.";
40
    /// `rust.skill`
41
    pub const RUST_SKILL: &str = "Read one of this repository skill procedures: a written procedure with conventions, commands, and rules. Call it before doing work a skill covers. Skills available:{skills}";
42
    /// `rust.openagents`
43
    pub const RUST_OPENAGENTS: &str = "Run the OpenAgents CLI commands (issue, project, repo, auth, etc.). Pass the arguments as a list without openagents itself.";
44
    /// `rust.delegate`
45
    pub const RUST_DELEGATE: &str = "Run one prompt on independent child coding agents in parallel and return what each one found or did. Use it when work splits into parts that do not depend on each other: several files to change the same way, several hypotheses to check, several tests to run down. Each child is a full coding agent with its own shell tool, working in a git worktree of its own so children cannot overwrite each other, and it starts with no context from this conversation and cannot ask questions — so the prompt has to be self-contained. Every child runs the same prompt and each is told separately which number it is, so write the prompt for whichever child reads it: say \"read the file at your own number\" rather than naming one child. Children run on {lane} and on this session's budget. Prefer one call with a count over several calls, and prefer `shell` over this for a single command — a child agent is for work worth a whole agent, not one line of output. At most {max_count} children.";
46
    /// `rust.capability`
47
    pub const RUST_CAPABILITY: &str = "Discover and load installed plugin capabilities: sandboxed, digest-pinned WebAssembly programs this machine already holds for common agent work. Before writing a script for a task, search here first — a capability that covers it is bounded, reviewable, and returns structured output. Call with `query` describing what you need to get the best matches; then call again with `name` set to the exact returned name to load it and make its dedicated tool available. Every later call to the loaded capability uses that exact name as the tool name.";
48
}
49
50
/// Every staged surface and the digest of the artifact this was built from.
51
///
52
/// A run records these so a bench row names exactly which text produced it.
53
pub const SURFACE_DIGESTS: [(&str, &str); 3] = [
54
    ("system-prompt", "sha256:8f0fbbbb38f4ce609a09fe2ba4e03ca160ff2c0d0f698dd4676c2bf940ded849"),
55
    ("tool-descriptions", "sha256:513c7518c72ba8c9959e517ea9f1262d7d3ee6b08452ddb48568bdabf6383c1b"),
56
    ("catalog-lines", "sha256:98a96bcf6acfd5847bc1bddcc575761af71be7ea144f3584b3594b24b2f80911"),
57
];
crates/openagents-cli/src/tools.rs modified +8 -25

@@ -174,6 +174,8 @@ pub struct DelegationGate {

174 174
    pub child: crate::delegate::ChildOptions,
175 175
}
176 176
177
use crate::surfaces::tool_descriptions as text;
178
177 179
pub struct HarnessToolRegistry {
178 180
    pub cwd: PathBuf,
179 181
    /// Discovered skills by name, in catalog order.

@@ -380,13 +382,7 @@ impl HarnessToolRegistry {

380 382
        let mut tools = vec![
381 383
            ToolDefinition {
382 384
                name: "shell".to_string(),
383
                description: format!(
384
                    "Run a shell command on this machine. The working directory is {}, so paths are \
385
                    relative to it and you do not need to ask where you are. Returns combined stdout \
386
                    and stderr with the exit code. Batch independent commands into one call with && \
387
                    instead of one call each: every call replays the conversation so far.",
388
                    self.cwd.display()
389
                ),
385
                description: text::RUST_SHELL.replace("{cwd}", &self.cwd.display().to_string()),
390 386
                parameters: serde_json::json!({
391 387
                    "type": "object",
392 388
                    "properties": {

@@ -398,7 +394,7 @@ impl HarnessToolRegistry {

398 394
            },
399 395
            ToolDefinition {
400 396
                name: "skill".to_string(),
401
                description: format!("Read one of this repository skill procedures: a written procedure with conventions, commands, and rules. Call it before doing work a skill covers. Skills available:{}", skill_list),
397
                description: text::RUST_SKILL.replace("{skills}", &skill_list),
402 398
                parameters: serde_json::json!({
403 399
                    "type": "object",
404 400
                    "properties": {

@@ -409,7 +405,7 @@ impl HarnessToolRegistry {

409 405
            },
410 406
            ToolDefinition {
411 407
                name: "openagents".to_string(),
412
                description: "Run the OpenAgents CLI commands (issue, project, repo, auth, etc.). Pass the arguments as a list without openagents itself.".to_string(),
408
                description: text::RUST_OPENAGENTS.to_string(),
413 409
                parameters: serde_json::json!({
414 410
                    "type": "object",
415 411
                    "properties": {

@@ -433,22 +429,9 @@ impl HarnessToolRegistry {

433 429
        if let Some(gate) = &self.delegation {
434 430
            tools.push(ToolDefinition {
435 431
                name: "delegate".to_string(),
436
                description: format!(
437
                    "Run one prompt on independent child coding agents in parallel and return what \
438
                    each one found or did. Use it when work splits into parts that do not depend on \
439
                    each other: several files to change the same way, several hypotheses to check, \
440
                    several tests to run down. Each child is a full coding agent with its own shell \
441
                    tool, working in a git worktree of its own so children cannot overwrite each \
442
                    other, and it starts with no context from this conversation and cannot ask \
443
                    questions — so the prompt has to be self-contained. Every child runs the same \
444
                    prompt and each is told separately which number it is, so write the prompt for \
445
                    whichever child reads it: say \"read the file at your own number\" rather than \
446
                    naming one child. Children run on {} and on this session's budget. Prefer one \
447
                    call with a count over several calls, and prefer `shell` over this for a single \
448
                    command — a child agent is for work worth a whole agent, not one line of \
449
                    output. At most {} children.",
450
                    gate.lane, gate.max_count
451
                ),
432
                description: text::RUST_DELEGATE
433
                    .replace("{lane}", &gate.lane)
434
                    .replace("{max_count}", &gate.max_count.to_string()),
452 435
                parameters: serde_json::json!({
453 436
                    "type": "object",
454 437
                    "properties": {
crates/openagents-cli/tests/coder-surfaces-golden.json added +10

@@ -0,0 +1,10 @@

1
{
2
  "system_prompt.local.no_tools": "You are `openagents coder`, a coding assistant in a terminal. You answer from a model running on this machine through Ollama. Nothing in this conversation leaves the machine and nothing is metered, but the context window is a fraction of a hosted model's, so keep large dumps out of the transcript.\n\nAnswer very concisely unless the reader asks for a longer response.\n\nYou have no tools in this session: you cannot read or write files, run commands, or reach anything outside this conversation. Answer from what the reader tells you, and say plainly when something would need a tool you do not have.",
3
  "system_prompt.thread.no_tools": "You are `openagents coder`, a coding assistant in a terminal. You answer through the OpenAgents inference proxy, on a thread opened for this session. Every round of tool calls re-sends the whole conversation to a metered model, so batch independent commands into one call and keep large dumps out of the transcript.\n\nAnswer very concisely unless the reader asks for a longer response.\n\nYou have no tools in this session: you cannot read or write files, run commands, or reach anything outside this conversation. Answer from what the reader tells you, and say plainly when something would need a tool you do not have.",
4
  "system_prompt.thread.two_tools": "You are `openagents coder`, a coding assistant in a terminal. You answer through the OpenAgents inference proxy, on a thread opened for this session. Every round of tool calls re-sends the whole conversation to a metered model, so batch independent commands into one call and keep large dumps out of the transcript.\n\nAnswer very concisely unless the reader asks for a longer response.\n\nYou have 2 tools, and no others:\n- `alpha`\n- `beta`\n\nThat list is complete: a capability not on it is one you do not have, whatever a model like you usually has. Read a tool's description before assuming what it covers. Where a description says what a child agent can do, that is the child's capability and not yours. Never say you ran something you did not run.",
5
  "tool.capability": "Discover and load installed plugin capabilities: sandboxed, digest-pinned WebAssembly programs this machine already holds for common agent work. Before writing a script for a task, search here first — a capability that covers it is bounded, reviewable, and returns structured output. Call with `query` describing what you need to get the best matches; then call again with `name` set to the exact returned name to load it and make its dedicated tool available. Every later call to the loaded capability uses that exact name as the tool name.",
6
  "tool.delegate": "Run one prompt on independent child coding agents in parallel and return what each one found or did. Use it when work splits into parts that do not depend on each other: several files to change the same way, several hypotheses to check, several tests to run down. Each child is a full coding agent with its own shell tool, working in a git worktree of its own so children cannot overwrite each other, and it starts with no context from this conversation and cannot ask questions — so the prompt has to be self-contained. Every child runs the same prompt and each is told separately which number it is, so write the prompt for whichever child reads it: say \"read the file at your own number\" rather than naming one child. Children run on the parent lane and on this session's budget. Prefer one call with a count over several calls, and prefer `shell` over this for a single command — a child agent is for work worth a whole agent, not one line of output. At most 8 children.",
7
  "tool.openagents": "Run the OpenAgents CLI commands (issue, project, repo, auth, etc.). Pass the arguments as a list without openagents itself.",
8
  "tool.shell": "Run a shell command on this machine. The working directory is /fixed/cwd, so paths are relative to it and you do not need to ask where you are. Returns combined stdout and stderr with the exit code. Batch independent commands into one call with && instead of one call each: every call replays the conversation so far.",
9
  "tool.skill": "Read one of this repository skill procedures: a written procedure with conventions, commands, and rules. Call it before doing work a skill covers. Skills available:"
10
}
crates/openagents-cli/tests/coder_surfaces_golden.rs added +119

@@ -0,0 +1,119 @@

1
//! The optimizable text this crate declares, pinned byte for byte.
2
//!
3
//! The system prompt, the lane notices, and the built-in tool descriptions are
4
//! staged artifacts (`surfaces/coder/`, OpenAgentsInc/openagents#122): a
5
//! change to any of them is meant to be a diff over an artifact with a
6
//! measured delta behind it, never a quiet edit in passing. This test is the
7
//! second half of that: it composes each surface from fixed inputs and
8
//! compares the result with `coder-surfaces-golden.json`, so moving the text
9
//! out of these files and into the artifact had to be a move and nothing else.
10
//!
11
//! Re-pin deliberately, with `UPDATE_CODER_SURFACES_GOLDEN=1 cargo test -p
12
//! openagents-cli --test coder_surfaces_golden`, and only when the change to
13
//! the text is the change you meant to make.
14
15
use std::collections::BTreeMap;
16
use std::path::PathBuf;
17
18
use openagents_cli::delegate::ChildOptions;
19
use openagents_cli::runtime::{CoderRuntimeSession, Lane};
20
use openagents_cli::tools::{DelegationGate, HarnessToolRegistry, ToolDefinition};
21
22
/// A directory that does not exist, so no plugin catalog is discovered and
23
/// the composed text depends on nothing outside this file.
24
const FIXED_CWD: &str = "/fixed/cwd";
25
26
/// Skills are discovered from `$HOME/.agents/skills` as well as the workspace,
27
/// so the machine running this test would otherwise decide what the `skill`
28
/// declaration says. Point `HOME` at an empty directory first: the surface
29
/// under test is the sentence around the catalog, not the catalog.
30
fn without_local_skills() -> tempfile::TempDir {
31
    let empty = tempfile::tempdir().unwrap();
32
    unsafe { std::env::set_var("HOME", empty.path()) };
33
    empty
34
}
35
36
fn registry() -> HarnessToolRegistry {
37
    HarnessToolRegistry::with_delegation(
38
        Some(PathBuf::from(FIXED_CWD)),
39
        DelegationGate {
40
            lane: "the parent lane".to_string(),
41
            user_token: None,
42
            max_count: 8,
43
            child: ChildOptions::default(),
44
        },
45
    )
46
}
47
48
fn session(lane: Lane) -> CoderRuntimeSession {
49
    CoderRuntimeSession::new(
50
        lane,
51
        Some("https://example.invalid/api/v1".to_string()),
52
        None,
53
        HarnessToolRegistry::new(Some(PathBuf::from(FIXED_CWD))),
54
    )
55
}
56
57
fn declared(name: &str) -> ToolDefinition {
58
    ToolDefinition {
59
        name: name.to_string(),
60
        description: format!("A declared {name}."),
61
        parameters: serde_json::json!({"type": "object"}),
62
    }
63
}
64
65
fn captured() -> BTreeMap<String, String> {
66
    let mut out = BTreeMap::new();
67
68
    let two = [declared("alpha"), declared("beta")];
69
    out.insert(
70
        "system_prompt.local.no_tools".to_string(),
71
        session(Lane::Local(String::new())).build_system_prompt(&[]),
72
    );
73
    out.insert(
74
        "system_prompt.thread.no_tools".to_string(),
75
        session(Lane::OxAlpha).build_system_prompt(&[]),
76
    );
77
    out.insert(
78
        "system_prompt.thread.two_tools".to_string(),
79
        session(Lane::OxAlpha).build_system_prompt(&two),
80
    );
81
82
    for tool in registry().list_tools() {
83
        out.insert(format!("tool.{}", tool.name), tool.description);
84
    }
85
    out
86
}
87
88
#[test]
89
fn the_declared_text_matches_the_pinned_golden() {
90
    let _empty_home = without_local_skills();
91
    let golden_path = PathBuf::from(env!("CARGO_MANIFEST_DIR"))
92
        .join("tests")
93
        .join("coder-surfaces-golden.json");
94
    let captured = captured();
95
96
    if std::env::var("UPDATE_CODER_SURFACES_GOLDEN").is_ok() {
97
        std::fs::write(
98
            &golden_path,
99
            format!("{}\n", serde_json::to_string_pretty(&captured).unwrap()),
100
        )
101
        .unwrap();
102
        return;
103
    }
104
105
    let raw = std::fs::read_to_string(&golden_path).expect("the golden file is checked in");
106
    let expected: BTreeMap<String, String> = serde_json::from_str(&raw).unwrap();
107
108
    for (key, want) in &expected {
109
        let got = captured
110
            .get(key)
111
            .unwrap_or_else(|| panic!("`{key}` is no longer declared"));
112
        assert_eq!(got, want, "`{key}` changed");
113
    }
114
    assert_eq!(
115
        captured.keys().collect::<Vec<_>>(),
116
        expected.keys().collect::<Vec<_>>(),
117
        "the set of declared surfaces changed"
118
    );
119
}
docs/assure-repo/false-green-candidates.v1.json modified +1 -1

@@ -4,7 +4,7 @@

4 4
  "note": "Heuristic false-green LEADS, not findings. A finding requires a demonstrated reproduction (surviving mutation via mutation-runner). Do not treat a candidate as a confirmed false green. Coverage-theater leads may include tests that delegate their assertion to a custom helper the classifier does not recognise; verify before acting.",
5 5
  "sourceDigest": "sha256:dd810dd48c5bdbc9becd7fcc01dd41a4ca2abf0b2d6f6a545907247f6e3e8361",
6 6
  "summary": {
7
    "filesScanned": 2503,
7
    "filesScanned": 2505,
8 8
    "candidateCount": 16,
9 9
    "byMode": {
10 10
      "false_green_coverage_theater": 15,
docs/assure-repo/surface-inventory.v1.json modified +3 -3

@@ -1,7 +1,7 @@

1 1
{
2 2
  "schemaVersion": "1",
3 3
  "repository": "OpenAgentsInc/openagents",
4
  "sourceDigest": "sha256:5ef645157ec7a0f6f2beca87dffd96dde98e6966272a12b95febff453a3d246a",
4
  "sourceDigest": "sha256:931a55aba01fd586c019996b1f02bf4d383179c895bc0053a5ed770bac5deb7b",
5 5
  "surfaces": [
6 6
    {
7 7
      "id": "app:@openagentsinc/acceptance-runner",

@@ -1910,7 +1910,7 @@

1910 1910
      "oracles": [
1911 1911
        {
1912 1912
          "type": "test",
1913
          "ref": "packages/openagents-cli (90 tracked test files)"
1913
          "ref": "packages/openagents-cli (91 tracked test files)"
1914 1914
        },
1915 1915
        {
1916 1916
          "type": "behavior-contract",

@@ -1969,7 +1969,7 @@

1969 1969
      "oracles": [
1970 1970
        {
1971 1971
          "type": "test",
1972
          "ref": "packages/coder-effectiveness (8 tracked test files)"
1972
          "ref": "packages/coder-effectiveness (9 tracked test files)"
1973 1973
        }
1974 1974
      ],
1975 1975
      "obligation": {
docs/coder/candidate-format.md added +197

@@ -0,0 +1,197 @@

1
# The candidate: one object for a review proposal and an optimizer mutation
2
3
Date: 2026-08-26. Status: the format of record for
4
OpenAgentsInc/openagents#122, #121, and #123. Companions:
5
`docs/coder/autoimprove.md` §3 and §7.6, `docs/coder/runbook.md`,
6
`docs/coder/best-practices.md`, and the analysis
7
`docs/coder/2026-08-26-dspy-gepa-coder-optimization.md` §4.
8
9
A candidate is a proposed change to the coder's optimizable text, carrying
10
everything needed to judge it: which surfaces it edits, where it came from,
11
which model family it was written or evolved against, the evidence behind it,
12
and how it would be confirmed or refuted.
13
14
The standing law it exists to serve: **an optimizer output is a candidate,
15
never a deployment.** A candidate becomes a change the way every other lever
16
does — a fresh worktree, a landing commit stating the measured delta, the
17
review, the ledger.
18
19
## 1. Why there is one format and not two
20
21
`autoimprove.md` §7.6 says the optimizer lane shares the review schema "so a
22
reflection and a mutation are the same object". If a review proposal and a
23
GEPA mutation were different shapes, the Pareto pool could not hold both, a
24
human cycle could not seed the optimizer, and every consumer would need two
25
readers. So the schema is defined once.
26
27
The definition lives in code, at
28
`packages/openagents-cli/src/coder-review-candidate.ts`, under the schema id
29
`openagents.coder_candidate.v1`. That file is the normative artifact; this
30
document explains it and pins the vocabulary the `surfaces` field draws on.
31
A review proposal is a candidate whose `lineage.origin` is `review`; an
32
optimizer mutation is one whose origin is `optimizer`; a hand-written cycle
33
lever is `human`. Nothing else about the object changes between them.
34
35
## 2. The shape
36
37
```
38
CoderCandidate {
39
  schema:        "openagents.coder_candidate.v1"
40
  candidateId:   "candidate:<8 hex>"      // computed, never supplied
41
  lever: {
42
    axis:        "process" | "plugin" | "harness" | "optimizer" | "routing" | "ledger"
43
    summary:     string                   // one sentence naming the change
44
  }
45
  surfaces:      [{ surface: string, diff: string }]
46
  lineage: {
47
    origin:      "review" | "optimizer" | "human"
48
    parent:      string | null            // the candidateId this came from
49
    producedBy:  string                   // e.g. "coder review-run:<reviewId>"
50
  }
51
  transferLabel: { modelFamily: string, lane: string }
52
  evidence:      [{ ref: string, note: string }]
53
  risk:          string
54
  verification:  { suite: string, metric: string, expectedDirection: "up" | "down" | "unchanged" }
55
}
56
```
57
58
Three fields carry most of the weight.
59
60
**`transferLabel`** is ledger O5 and the evolve-the-harness finding behind it:
61
code mechanisms transferred across model families and tuned prompts did not.
62
A candidate evolved against one family is not evidence for another, and the
63
label is what stops a pool from quietly pretending otherwise.
64
65
**`evidence`** is checked, not decorative.
66
`autoimprove.md` §6 lists "confident review without understanding" as a failure
67
mode. The parser refuses a proposal with no evidence, and refuses one whose
68
refs do not resolve against the artifacts the producer was actually given. The
69
ref grammar is small on purpose, because every scheme in it has to be
70
resolvable: `trial:<task>#step-<id>`, `trial:<task>#outcome`,
71
`row:<suite>#<recordedAt>`, `ledger:<id>`, `diff:<path>`.
72
73
**`candidateId`** is FNV-1a over the candidate's own facts — an identity for a
74
pool entry, not a receipt. `bench-results` owns tamper-evidence; borrowing its
75
`receipt:` vocabulary here would suggest this digest carries the same weight.
76
77
## 3. The surface vocabulary
78
79
`surfaces[].surface` names a staged artifact from `surfaces/coder/index.json`.
80
The whole current vocabulary:
81
82
| `surface` | Artifact | What it holds |
83
| --- | --- | --- |
84
| `system-prompt` | `surfaces/coder/system-prompt.v1.json` | The instructions, the concision sentence, the no-tools and tool-list sentences, and the lane notices, for all three harnesses |
85
| `tool-descriptions` | `surfaces/coder/tool-descriptions.v1.json` | The description of each declared tool, plus the per-model-family emphasis overrides |
86
| `catalog-lines` | `surfaces/coder/catalog-lines.v1.json` | Each installed plugin's catalog line, keyed by plugin id |
87
88
`diff` is a unified diff over that artifact file where one exists, and the
89
proposed text otherwise. The artifact's `text` map is flat and keyed, so a
90
one-sentence change is a one-line diff and reads as one.
91
92
A surface a candidate does not touch is simply absent from the array. A
93
candidate touching none of them is legitimate — the `plugin`, `process`, and
94
`routing` axes change code, skills, or procedure rather than staged text — and
95
its `surfaces` array is empty.
96
97
### The catalog-lines surface is a mirror, not a move
98
99
Stated plainly, because it changes how a candidate against it is applied. The
100
system prompt and the tool descriptions were string literals in
101
`crates/coder-lite`, `crates/openagents-cli`, and `packages/openagents-cli`;
102
staging them moved the text out of code and into the artifact, and the
103
artifact is now where it is edited.
104
105
A plugin's catalog line was never a literal. It is the top-level `description`
106
of `plugins/<id>/manifest.json`, discovered at runtime by `discover_catalog()`
107
(`crates/openagents-cli/src/plugins.rs`) and its TypeScript equivalent. The
108
manifest stays where that text is edited. `catalog-lines.v1.json` mirrors those
109
descriptions into one diffable object with a content digest, so an optimizer
110
has a single file to diff and a bench row has a single digest to record — and
111
`pnpm run check:coder-surfaces` fails when a manifest and the mirror disagree.
112
113
So: applying a candidate against `system-prompt` or `tool-descriptions` means
114
patching the artifact. Applying one against `catalog-lines` means patching the
115
named `plugins/<id>/manifest.json`. Both then need the rebuild.
116
117
## 4. Applying a candidate
118
119
1. Patch the artifact (or, for `catalog-lines`, the manifest).
120
2. `pnpm run build:coder-surfaces` — re-pins `surfaces/coder/index.json` and
121
   regenerates the embedded modules the two CLIs compile.
122
3. Run the suite named in `verification.suite`. The run announces its staged
123
   text as `[oa:surfaces <id>=<digest>,…]`, `bench-results` records those
124
   digests on the row, and `pnpm run effectiveness:compare` names the change as
125
   a variable when two compared rows carry different pins.
126
4. Land or discard on the measured delta, per `runbook.md`.
127
128
Skipping step 2 is the failure this staging exists to prevent, and it is a
129
named check rather than a silence: `check:coder-surfaces` runs inside
130
`check:fast` and refuses a tree where an artifact and its build disagree.
131
132
## 5. A worked example
133
134
A real observation from the staging work, kept here as a candidate rather than
135
as a change: the `shell` tool description has forked between the two harnesses.
136
The TypeScript one steers the model toward an installed capability before it
137
writes a script and carries a paragraph of token economy; the Rust one carries
138
neither. A candidate proposing to close that gap looks like this.
139
140
```json
141
{
142
  "schema": "openagents.coder_candidate.v1",
143
  "candidateId": "candidate:computed",
144
  "lever": {
145
    "axis": "harness",
146
    "summary": "Give the Rust shell description the capability-first steer the TypeScript one already carries."
147
  },
148
  "surfaces": [
149
    {
150
      "surface": "tool-descriptions",
151
      "diff": "--- a/surfaces/coder/tool-descriptions.v1.json\n+++ b/surfaces/coder/tool-descriptions.v1.json\n@@\n-    \"rust.shell\": \"Run a shell command on this machine. The working directory is {cwd}, …\",\n+    \"rust.shell\": \"Run a shell command on this machine. The working directory is {cwd}, … When an installed capability covers the task — the `capability` tool names what is installed — load and call it instead of scripting the same thing here: it is sandboxed, bounded, and returns structured output. …\",\n"
152
    }
153
  ],
154
  "lineage": {
155
    "origin": "human",
156
    "parent": null,
157
    "producedBy": "openagents#122 staging"
158
  },
159
  "transferLabel": {
160
    "modelFamily": "unmeasured",
161
    "lane": "unmeasured"
162
  },
163
  "evidence": [
164
    {
165
      "ref": "diff:surfaces/coder/tool-descriptions.v1.json",
166
      "note": "The two harnesses' `shell` descriptions, side by side in one artifact, which is what made the fork visible."
167
    },
168
    {
169
      "ref": "ledger:P3",
170
      "note": "A plugin competes for rank in a five-result search; a shell description that never mentions the capability tool is one reason a search never happens."
171
    }
172
  ],
173
  "risk": "The steer is unmeasured on the Rust harness. It may cost tokens on tasks no installed capability covers, and it lengthens a description that is currently short.",
174
  "verification": {
175
    "suite": "tb2-quick",
176
    "metric": "successRate",
177
    "expectedDirection": "up"
178
  }
179
}
180
```
181
182
Note what the object refuses to pretend. `transferLabel` says `unmeasured`
183
rather than naming a family, because this candidate came from reading two
184
files and not from a run. Under ledger O5 that is a candidate worth screening
185
and not a change worth landing — which is the whole point of the format.
186
187
## 6. What is deliberately not here
188
189
- **No optimizer.** #123 owns the GEPA lane; this document owns the object it
190
  emits.
191
- **No adoption rule.** `runbook.md` owns the acceptance gate and
192
  `best-practices.md` owns the ledger operations. A candidate that passes its
193
  verification is still a candidate until a cycle lands it.
194
- **No parameter-level tool text.** The staged `tool-descriptions` surface
195
  holds each tool's own description and the family overrides. The JSON-Schema
196
  `description` of an individual argument is still a literal in code, and
197
  staging it is a later slice rather than a silent part of this one.
package.json modified +3 -1

@@ -5,6 +5,8 @@

5 5
  "type": "module",
6 6
  "scripts": {
7 7
    "check:vp2-node-runtime": "node scripts/vp2-node-runtime-guard.mjs",
8
    "build:coder-surfaces": "node scripts/coder-surfaces.mjs --write",
9
    "check:coder-surfaces": "node scripts/coder-surfaces.mjs",
8 10
    "check:agent-client-protocol": "pnpm --dir packages/agent-client-protocol run check:generated",
9 11
    "check:agent-client-protocol-conformance": "pnpm --dir packages/agent-client-protocol-conformance run check:artifacts",
10 12
    "check:all-work-contract": "pnpm --dir packages/all-work-contract run check:generated && cargo test -p openagents-all-work-contract",

@@ -18,7 +20,7 @@

18 20
    "agent-computer:codex-auth": "node --import tsx apps/pylon/deploy/agent-computer/native-codex-auth.ts",
19 21
    "deploy:aiur": "pnpm --dir apps/aiur run deploy",
20 22
    "check": "pnpm run fmt:check && pnpm run lint && pnpm run check:fast && pnpm run typecheck && pnpm run test",
21
    "check:fast": "vp lint --quiet && pnpm run check:all-work-contract && pnpm run check:agent-client-protocol && pnpm run check:agent-client-protocol-conformance && pnpm run check:codex-app-server-protocol && pnpm run check:ste:public && pnpm run check:ste-public-semantics && node scripts/vp1-retired-money-surface-guard.mjs . && node scripts/zero-supported-bun-guard.mjs . && node scripts/dse-single-authority-guard.mjs . && node scripts/google-cloud-authority-guard.mjs && node scripts/sarah-participant-join-authority-guard.mjs . && node scripts/uncalled-production-symbol-guard.mjs . && node scripts/cli-invocability-guard.mjs . && pnpm run check:documented-route-mounts && pnpm run check:assure-repo && pnpm run check:assure-repo-audit && pnpm run check:assure-repo-drift",
23
    "check:fast": "vp lint --quiet && pnpm run check:all-work-contract && pnpm run check:agent-client-protocol && pnpm run check:agent-client-protocol-conformance && pnpm run check:codex-app-server-protocol && pnpm run check:ste:public && pnpm run check:ste-public-semantics && node scripts/vp1-retired-money-surface-guard.mjs . && node scripts/zero-supported-bun-guard.mjs . && node scripts/dse-single-authority-guard.mjs . && node scripts/google-cloud-authority-guard.mjs && node scripts/sarah-participant-join-authority-guard.mjs . && node scripts/uncalled-production-symbol-guard.mjs . && node scripts/cli-invocability-guard.mjs . && pnpm run check:coder-surfaces && pnpm run check:documented-route-mounts && pnpm run check:assure-repo && pnpm run check:assure-repo-audit && pnpm run check:assure-repo-drift",
22 24
    "check:afs-boundaries": "node --import tsx scripts/check-afs-boundaries.ts",
23 25
    "check:afs-apple-fm-version-drift": "node --import tsx scripts/check-afs-apple-fm-version-drift.ts",
24 26
    "check:documented-route-mounts": "pnpm --dir apps/openagents.com/workers/api run test src/routing/documented-route-mounts.test.ts",
packages/coder-effectiveness/src/compare.ts modified +12

@@ -30,6 +30,7 @@

30 30
 *    job, and it already has a third verdict for the cases nothing measured.
31 31
 */
32 32
33
import { surfacePinOf } from "./results-store.ts";
33 34
import type { BenchResultRow } from "./results-store.ts";
34 35
35 36
export type DeltaDirection = "better" | "worse" | "unchanged" | "unpriced" | "unknown";

@@ -279,6 +280,17 @@ const confoundersOf = (

279 280
  if (axis === "recordedAt" && models.length > 1) {
280 281
    notes.push(`model also varies (${models.join(", ")})`);
281 282
  }
283
  // The staged text is a variable like any other (OpenAgentsInc/openagents#122).
284
  // Named on both axes: a lane comparison assumes the prompt is held still
285
  // while the lane changes, and a trend step that changed the prompt is a
286
  // cycle whose delta belongs to that change and to nothing else in the step.
287
  const pins = distinct(rows.map(surfacePinOf));
288
  if (pins.length > 1) {
289
    notes.push(`staged text also varies (${pins.join(" | ")})`);
290
  }
291
  if (rows.some((row) => surfacePinOf(row) === null) && pins.length > 0) {
292
    notes.push("at least one row names no staged text, so what it measured cannot be identified");
293
  }
282 294
  if (bases.includes("operator_placeholder")) {
283 295
    notes.push(
284 296
      "at least one row is priced from operator placeholder rates, so its cost is provisional",
packages/coder-effectiveness/src/effectiveness.test.ts modified +2

@@ -171,6 +171,7 @@ describe("cost per accepted outcome", () => {

171 171
      suite: "tb2-cross-section",
172 172
      lane: "proxy",
173 173
      runDigest: "effectiveness:test",
174
        surfaceDigests: null,
174 175
      trials: [
175 176
        {
176 177
          task: "fix-git",

@@ -183,6 +184,7 @@ describe("cost per accepted outcome", () => {

183 184
          toolCalls: 1,
184 185
          wallClockSeconds: 60,
185 186
          threadId: null,
187
          surfaceDigests: null,
186 188
          exception: null,
187 189
        },
188 190
      ],
packages/coder-effectiveness/src/effectiveness.ts modified +6

@@ -86,6 +86,11 @@ export interface EffectivenessReport {

86 86
  readonly suite: string;
87 87
  readonly lane: string;
88 88
  readonly runDigest: string;
89
  /**
90
   * The staged text surfaces the run composed from, by content digest, or
91
   * `null` when the trials announced none or disagreed.
92
   */
93
  readonly surfaceDigests: Readonly<Record<string, string>> | null;
89 94
  readonly jobId: string | null;
90 95
  readonly models: ReadonlyArray<string>;
91 96
  readonly agentVersions: ReadonlyArray<string>;

@@ -172,6 +177,7 @@ export const summarizeRun = (

172 177
    suite: run.suite,
173 178
    lane: run.lane,
174 179
    runDigest: run.runDigest,
180
    surfaceDigests: run.surfaceDigests,
175 181
    jobId: run.jobId,
176 182
    models: distinct(run.trials.map((trial) => trial.modelId)),
177 183
    agentVersions: distinct(run.trials.map((trial) => trial.agentVersion)),
packages/coder-effectiveness/src/harbor-job.ts modified +77 -3

@@ -33,6 +33,17 @@ import { join } from "node:path";

33 33
/** The coder's `--plain` thread announcement, the same contract bench parses. */
34 34
const THREAD_LINE = /\[oa:thread ([0-9a-fA-F-]{36})\]/u;
35 35
36
/**
37
 * The coder's `--plain` staged-text announcement
38
 * (OpenAgentsInc/openagents#122), emitted beside the thread line.
39
 *
40
 * Read from the trial rather than from the repository, because the repository
41
 * at scoring time is not the repository the run happened on. A trial from a
42
 * CLI that predates the announcement carries no pin, and the row then records
43
 * none rather than a borrowed one.
44
 */
45
const SURFACES_LINE = /\[oa:surfaces ([^\]]+)\]/u;
46
36 47
/** What a verifier decided about one trial. */
37 48
export type TrialOutcome = "accepted" | "rejected" | "ungraded";
38 49

@@ -51,6 +62,12 @@ export interface TrialRecord {

51 62
  readonly wallClockSeconds: number | null;
52 63
  /** The forge thread the trial ran in, when the coder announced one. */
53 64
  readonly threadId: string | null;
65
  /**
66
   * The staged text surfaces this trial composed its prompt and tool
67
   * declarations from, by content digest. `null` when the trial announced
68
   * none.
69
   */
70
  readonly surfaceDigests: Readonly<Record<string, string>> | null;
54 71
  /** The typed error Harbor classified, when the trial raised one. */
55 72
  readonly exception: string | null;
56 73
}

@@ -66,6 +83,16 @@ export interface GradedRun {

66 83
   * report says so rather than letting a reader assume.
67 84
   */
68 85
  readonly runDigest: string;
86
  /**
87
   * The staged text every trial agreed on, or `null` when the trials announced
88
   * none or disagreed.
89
   *
90
   * Disagreement is recorded as absence rather than as one of the two answers:
91
   * a job whose trials ran different prompts measured no single prompt, and a
92
   * row that named either half would be naming text that produced some of the
93
   * outcomes it reports.
94
   */
95
  readonly surfaceDigests: Readonly<Record<string, string>> | null;
69 96
  readonly trials: ReadonlyArray<TrialRecord>;
70 97
}
71 98

@@ -97,11 +124,13 @@ export const readHarborJob = (jobDir: string, options: ReadHarborJobOptions): Gr

97 124
    trials.push(readTrial(entry, trialDir, trialResult));
98 125
  }
99 126
127
  const surfaceDigests = agreedSurfaceDigests(trials);
100 128
  return {
101 129
    jobId: readString(readField(jobResult, "id")),
102 130
    suite: options.suite,
103 131
    lane: options.lane,
104
    runDigest: runDigestOf(trials, options),
132
    runDigest: runDigestOf(trials, options, surfaceDigests),
133
    surfaceDigests,
105 134
    trials,
106 135
  };
107 136
};

@@ -124,6 +153,7 @@ const readTrial = (dirName: string, trialDir: string, trialResult: unknown): Tri

124 153
    toolCalls: usage.toolCalls,
125 154
    wallClockSeconds: wallClockOf(trialResult),
126 155
    threadId: threadIdOf(trialDir),
156
    surfaceDigests: surfaceDigestsOf(trialDir),
127 157
    exception: readString(readField(readField(trialResult, "exception_info"), "exception_type")),
128 158
  };
129 159
};

@@ -244,13 +274,54 @@ const threadIdOf = (trialDir: string): string | null => {

244 274
  return match?.[1] ?? null;
245 275
};
246 276
277
/** The staged text one trial announced, or `null` when it announced none. */
278
const surfaceDigestsOf = (trialDir: string): Readonly<Record<string, string>> | null => {
279
  const path = join(trialDir, "agent", "coder.txt");
280
  if (!existsSync(path)) return null;
281
  const match = SURFACES_LINE.exec(readFileSync(path, "utf8"));
282
  if (match?.[1] === undefined) return null;
283
  const digests: Record<string, string> = {};
284
  for (const pair of match[1].split(",")) {
285
    const at = pair.indexOf("=");
286
    if (at <= 0) continue;
287
    digests[pair.slice(0, at).trim()] = pair.slice(at + 1).trim();
288
  }
289
  return Object.keys(digests).length === 0 ? null : digests;
290
};
291
292
/**
293
 * The staged text the whole job ran on, when every trial agrees.
294
 *
295
 * One disagreeing trial makes the job's pin `null`. Two prompts in one job is
296
 * a job that measured neither of them, and the honest column for that is
297
 * empty.
298
 */
299
const agreedSurfaceDigests = (
300
  trials: ReadonlyArray<TrialRecord>,
301
): Readonly<Record<string, string>> | null => {
302
  const announced = trials
303
    .map((trial) => trial.surfaceDigests)
304
    .filter((digests): digests is Readonly<Record<string, string>> => digests !== null);
305
  if (announced.length === 0 || announced.length !== trials.length) return null;
306
  const first = JSON.stringify(Object.entries(announced[0]!).toSorted());
307
  const agreed = announced.every(
308
    (digests) => JSON.stringify(Object.entries(digests).toSorted()) === first,
309
  );
310
  return agreed ? announced[0]! : null;
311
};
312
247 313
/**
248
 * The recipe pin. Deliberately independent of the `harbor:` digest
314
 * The recipe pin, including the staged text surfaces. Deliberately
315
 * independent of the `harbor:` digest
249 316
 * `bench/post_gym_run.py` computes: that one hashes a Python-serialised config
250 317
 * and this one hashes an explicit list of the facts that make two runs
251 318
 * comparable, so claiming they agree would be a claim neither can keep.
252 319
 */
253
const runDigestOf = (trials: ReadonlyArray<TrialRecord>, options: ReadHarborJobOptions): string => {
320
const runDigestOf = (
321
  trials: ReadonlyArray<TrialRecord>,
322
  options: ReadHarborJobOptions,
323
  surfaceDigests: Readonly<Record<string, string>> | null,
324
): string => {
254 325
  const source = JSON.stringify({
255 326
    suite: options.suite,
256 327
    lane: options.lane,

@@ -258,6 +329,9 @@ const runDigestOf = (trials: ReadonlyArray<TrialRecord>, options: ReadHarborJobO

258 329
    tasks: trials.map((trial) => trial.task).toSorted(),
259 330
    agentVersions: distinct(trials.map((trial) => trial.agentVersion)),
260 331
    models: distinct(trials.map((trial) => trial.modelId)),
332
    // The staged text is part of the recipe, so two runs that differ only in
333
    // the prompt do not share a digest and are not read as the same run.
334
    surfaces: surfaceDigests === null ? null : Object.entries(surfaceDigests).toSorted(),
261 335
  });
262 336
  return `effectiveness:${createHash("sha256").update(source).digest("hex")}`;
263 337
};
packages/coder-effectiveness/src/results-store.test.ts modified +3 -1

@@ -186,7 +186,9 @@ describe("what the store refuses", () => {

186 186
  test("throws on a row written under another schema", () => {
187 187
    writeFileSync(store, `${JSON.stringify({ schema: "something.else.v1" })}\n`, "utf8");
188 188
189
    expect(() => readResultRows(store)).toThrow(/expected openagents\.bench_result\.v2/u);
189
    expect(() => readResultRows(store)).toThrow(
190
      /expected openagents\.bench_result\.v3 or openagents\.bench_result\.v2/u,
191
    );
190 192
  });
191 193
192 194
  test("names a v1 row for what it is rather than reading it as a v2 one", () => {
packages/coder-effectiveness/src/results-store.ts modified +68 -8

@@ -48,12 +48,26 @@ import type { RunClassification } from "./suite-manifest.ts";

48 48
import type { CriterionVerdict, ThresholdGate } from "./thresholds.ts";
49 49
50 50
/**
51
 * Bumped from v1 when the suite pin became mandatory. A v1 row carried no
52
 * `suiteId` or `suiteDigest`, so nothing in it says which pinned task list it
53
 * measured — readable as history, not comparable to a v2 row, and
54
 * {@link readResultRows} says so by name rather than by silently coercing it.
51
 * Bumped from v2 when a row had to name the staged text it measured
52
 * (OpenAgentsInc/openagents#122). A v3 row carries `surfaceDigests`; a v2 row
53
 * does not, and the difference is exactly "we did not record this" rather than
54
 * "this run used no staged text".
55
 *
56
 * v1 was the bump before it, when the suite pin became mandatory.
57
 */
58
export const BENCH_RESULT_SCHEMA = "openagents.bench_result.v3";
59
60
/**
61
 * The predecessor this store still reads.
62
 *
63
 * A v2 row is comparable to a v3 row on every column both carry, so it is not
64
 * refused the way a v1 row is: what a v2 row cannot do is say which prompt
65
 * produced it, and {@link surfacePinOf} answers `null` for it rather than
66
 * inventing one. Its receipt is unaffected — {@link receiptOf} digests the
67
 * keys a row actually has, so adding a column to the writer does not rewrite
68
 * what an already-written row asserts.
55 69
 */
56
export const BENCH_RESULT_SCHEMA = "openagents.bench_result.v2";
70
export const BENCH_RESULT_SCHEMA_V2 = "openagents.bench_result.v2";
57 71
58 72
/** The v1 rows this store used to write, kept only to name them in an error. */
59 73
const BENCH_RESULT_SCHEMA_V1 = "openagents.bench_result.v1";

@@ -67,13 +81,31 @@ const BENCH_RESULT_SCHEMA_V1 = "openagents.bench_result.v1";

67 81
 * is here is what two runs can be compared on.
68 82
 */
69 83
export interface BenchResultRow {
70
  readonly schema: typeof BENCH_RESULT_SCHEMA;
84
  readonly schema: typeof BENCH_RESULT_SCHEMA | typeof BENCH_RESULT_SCHEMA_V2;
71 85
  readonly recordedAt: string;
72 86
73 87
  readonly suite: string;
74 88
  readonly lane: string;
75
  /** The report's pin over suite, lane, tasks, CLI version, model, and rates. */
89
  /**
90
   * The report's pin over suite, lane, tasks, CLI version, model, rates, and
91
   * the staged text surfaces.
92
   */
76 93
  readonly runDigest: string;
94
  /**
95
   * The staged text this run composed from, by content digest
96
   * (`surfaces/coder/index.json`).
97
   *
98
   * Three distinguishable states, on purpose:
99
   *
100
   * - a record — the run announced this text, and the row names it;
101
   * - `null` — the writer records the column, and this run announced no text
102
   *   or its trials disagreed;
103
   * - absent — a v2 row, written before the column existed.
104
   *
105
   * Optional in the type because a v2 row read off disk genuinely lacks the
106
   * key, and giving it a default here would turn "not recorded" into a claim.
107
   */
108
  readonly surfaceDigests?: Readonly<Record<string, string>> | null | undefined;
77 109
  /**
78 110
   * The narrower pin two rows must share to be comparable at all: the suite,
79 111
   * the sorted task list, and the rate catalog. Lane, model, and CLI version

@@ -178,6 +210,7 @@ export const buildResultRow = (

178 210
    suite: report.suite,
179 211
    lane: report.lane,
180 212
    runDigest: report.runDigest,
213
    surfaceDigests: report.surfaceDigests,
181 214
    suiteKey: suiteKeyOf(report, classification),
182 215
    jobId: report.jobId,
183 216

@@ -235,6 +268,22 @@ export const receiptOf = (row: UnreceiptedRow): string => {

235 268
  return `receipt:${createHash("sha256").update(source).digest("hex")}`;
236 269
};
237 270
271
/**
272
 * A short, comparable rendering of a row's staged text pin.
273
 *
274
 * `null` for a row that names none — a v2 row, or a run whose trials did not
275
 * announce or did not agree. Two rows with equal non-null pins ran the same
276
 * text; two with different pins did not, and a comparison across them varies
277
 * the prompt as well as whatever it meant to vary.
278
 */
279
export const surfacePinOf = (row: BenchResultRow): string | null => {
280
  const digests = row.surfaceDigests;
281
  if (digests === undefined || digests === null) return null;
282
  const entries = Object.entries(digests).toSorted();
283
  if (entries.length === 0) return null;
284
  return entries.map(([id, digest]) => `${id}:${digest.replace(/^sha256:/u, "").slice(0, 8)}`).join(" ");
285
};
286
238 287
export type ChainBreak =
239 288
  | { readonly kind: "receipt_mismatch"; readonly index: number; readonly detail: string }
240 289
  | { readonly kind: "chain_broken"; readonly index: number; readonly detail: string };

@@ -289,6 +338,11 @@ export const verifyResultChain = (rows: ReadonlyArray<BenchResultRow>): ChainVer

289 338
 * corrupted store into a shorter, apparently valid one, and a trend that
290 339
 * silently drops the rows it could not parse is the worst of both worlds.
291 340
 * A store that does not exist yet reads as no rows, which is not a corruption.
341
 *
342
 * A v2 row is read as itself. The store is append-only and hash-chained, so a
343
 * schema bump cannot rewrite the rows already in it — every existing row keeps
344
 * the receipt it was written with, and {@link verifyResultChain} recomputes
345
 * that receipt over the keys the row actually carries.
292 346
 */
293 347
export const readResultRows = (storePath: string): ReadonlyArray<BenchResultRow> => {
294 348
  if (!existsSync(storePath)) return [];

@@ -306,6 +360,12 @@ export const readResultRows = (storePath: string): ReadonlyArray<BenchResultRow>

306 360
    // Deliberately widened: a row on disk can carry any schema string, and the
307 361
    // point of the next two checks is to find out which.
308 362
    const row = parsed as { schema: string } as BenchResultRow;
363
    if ((row.schema as string) === BENCH_RESULT_SCHEMA_V2) {
364
      // Readable and comparable, minus the column it predates. Returned as-is:
365
      // filling `surfaceDigests` in here would make a row assert something it
366
      // never recorded, and its receipt would then stop verifying.
367
      return row;
368
    }
309 369
    if ((row.schema as string) === BENCH_RESULT_SCHEMA_V1) {
310 370
      throw new Error(
311 371
        `${storePath} line ${String(index + 1)} is a ${BENCH_RESULT_SCHEMA_V1} row, written before a run had to name the suite manifest it covered. It carries no suite digest, so nothing in it says which pinned task list it measured and it cannot be compared to a ${BENCH_RESULT_SCHEMA} row. Move it to an archive file rather than migrating it: a digest cannot be invented for a run that never recorded one.`,

@@ -313,7 +373,7 @@ export const readResultRows = (storePath: string): ReadonlyArray<BenchResultRow>

313 373
    }
314 374
    if (row.schema !== BENCH_RESULT_SCHEMA) {
315 375
      throw new Error(
316
        `${storePath} line ${String(index + 1)} has schema ${String(row.schema)}, expected ${BENCH_RESULT_SCHEMA}`,
376
        `${storePath} line ${String(index + 1)} has schema ${String(row.schema)}, expected ${BENCH_RESULT_SCHEMA} or ${BENCH_RESULT_SCHEMA_V2}`,
317 377
      );
318 378
    }
319 379
    return row;
packages/coder-effectiveness/src/surface-pin.test.ts added +192

@@ -0,0 +1,192 @@

1
/**
2
 * The staged-text pin on a run row (OpenAgentsInc/openagents#122).
3
 *
4
 * A trend that cannot say which prompt produced a figure cannot tell a text
5
 * change from noise, so a run announces the staged surfaces it composed from
6
 * and the row records them. The cases here are the three claims that makes:
7
 *
8
 * 1. the pin is read from the trial, not from the repository at scoring time;
9
 * 2. two runs alike in every other way but the text are different runs, and
10
 *    the run digest says so;
11
 * 3. the schema bump that added the column did not invalidate the rows written
12
 *    before it — the store is append-only, and a v2 row keeps verifying.
13
 */
14
15
import { cpSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs";
16
import { tmpdir } from "node:os";
17
import { join } from "node:path";
18
import { fileURLToPath } from "node:url";
19
import { afterEach, beforeEach, describe, expect, test } from "vite-plus/test";
20
21
import { compareRuns } from "./compare.ts";
22
import { summarizeRun } from "./effectiveness.ts";
23
import { readHarborJob } from "./harbor-job.ts";
24
import { CODER_RATE_CATALOG_VERSION } from "./pricing.ts";
25
import {
26
  BENCH_RESULT_SCHEMA,
27
  BENCH_RESULT_SCHEMA_V2,
28
  buildResultRow,
29
  readResultRows,
30
  receiptOf,
31
  surfacePinOf,
32
  verifyResultChain,
33
  type BenchResultRow,
34
} from "./results-store.ts";
35
import { classifyRun, parseSuiteManifest } from "./suite-manifest.ts";
36
37
const FIXTURE = fileURLToPath(new URL("../fixtures/priced-lane", import.meta.url));
38
39
const PIN_A =
40
  "[oa:surfaces system-prompt=sha256:aaaa1111,tool-descriptions=sha256:bbbb2222,catalog-lines=sha256:cccc3333]";
41
const PIN_B =
42
  "[oa:surfaces system-prompt=sha256:dddd4444,tool-descriptions=sha256:bbbb2222,catalog-lines=sha256:cccc3333]";
43
44
let workspace: string;
45
46
beforeEach(() => {
47
  workspace = mkdtempSync(join(tmpdir(), "coder-surface-pin-"));
48
});
49
50
afterEach(() => {
51
  rmSync(workspace, { recursive: true, force: true });
52
});
53
54
/** A copy of the fixture job whose trials announce `pin`, or announce nothing. */
55
const jobAnnouncing = (name: string, pin: string | null): string => {
56
  const jobDir = join(workspace, name);
57
  cpSync(FIXTURE, jobDir, { recursive: true });
58
  for (const trial of [
59
    "build-cmake__bbbbbbbb",
60
    "fix-git__aaaaaaaa",
61
    "parse-log__cccccccc",
62
    "port-forward__dddddddd",
63
  ]) {
64
    const path = join(jobDir, trial, "agent", "coder.txt");
65
    const existing = readFileSync(path, "utf8");
66
    writeFileSync(path, pin === null ? existing : `${existing}${pin}\n`);
67
  }
68
  return jobDir;
69
};
70
71
const runOf = (jobDir: string, lane = "proxy") =>
72
  readHarborJob(jobDir, {
73
    suite: "tb2-cross-section",
74
    lane,
75
    rateCatalogVersion: CODER_RATE_CATALOG_VERSION,
76
  });
77
78
describe("the staged-text pin", () => {
79
  test("is read from the trial that announced it", () => {
80
    const run = runOf(jobAnnouncing("pinned", PIN_A));
81
    expect(run.surfaceDigests).toEqual({
82
      "system-prompt": "sha256:aaaa1111",
83
      "tool-descriptions": "sha256:bbbb2222",
84
      "catalog-lines": "sha256:cccc3333",
85
    });
86
  });
87
88
  test("is absent, not invented, when the CLI announced nothing", () => {
89
    const run = runOf(jobAnnouncing("silent", null));
90
    expect(run.surfaceDigests).toBeNull();
91
  });
92
93
  // A job whose trials ran different prompts measured no single prompt, so the
94
  // honest pin for it is none rather than whichever trial was read first.
95
  test("is absent when the trials disagree", () => {
96
    const jobDir = jobAnnouncing("split", PIN_A);
97
    const path = join(jobDir, "fix-git__aaaaaaaa", "agent", "coder.txt");
98
    writeFileSync(path, readFileSync(path, "utf8").replace(PIN_A, PIN_B));
99
    expect(runOf(jobDir).surfaceDigests).toBeNull();
100
  });
101
});
102
103
describe("two runs that differ only in staged text", () => {
104
  test("do not share a run digest", () => {
105
    const first = runOf(jobAnnouncing("text-a", PIN_A));
106
    const second = runOf(jobAnnouncing("text-b", PIN_B));
107
108
    expect(first.trials.map((trial) => trial.task).toSorted()).toEqual(
109
      second.trials.map((trial) => trial.task).toSorted(),
110
    );
111
    expect(first.lane).toBe(second.lane);
112
    expect(first.runDigest).not.toBe(second.runDigest);
113
  });
114
115
  // The suite key is what two rows must SHARE to be comparable at all, and the
116
  // prompt is an axis a comparison varies rather than a precondition for it.
117
  // A text change that moved the suite key would make the two runs invisible
118
  // to each other, which is the opposite of what recording it is for.
119
  test("are still comparable: the suite key is unchanged", () => {
120
    const rows = [
121
      rowFor(jobAnnouncing("text-a", PIN_A), null),
122
      rowFor(jobAnnouncing("text-b", PIN_B), null),
123
    ];
124
    expect(rows[0]!.suiteKey).toBe(rows[1]!.suiteKey);
125
  });
126
127
  test("are named as varying by the comparison", () => {
128
    const first = rowFor(jobAnnouncing("text-a", PIN_A), null, "proxy");
129
    const second = rowFor(jobAnnouncing("text-b", PIN_B), first.receipt, "local");
130
    const comparison = compareRuns([first, second]);
131
    const notes = comparison.laneComparisons.flatMap((lane) => lane.confounders);
132
    expect(notes.some((note) => note.startsWith("staged text also varies"))).toBe(true);
133
  });
134
});
135
136
describe("the v2 rows written before the column existed", () => {
137
  test("still verify beside a v3 row", () => {
138
    const older = rowFor(jobAnnouncing("older", null), null, "proxy");
139
    // Exactly the shape the previous writer produced: no `surfaceDigests` key
140
    // at all, and the receipt it was written with.
141
    const { surfaceDigests, receipt, ...rest } = older;
142
    void surfaceDigests;
143
    void receipt;
144
    const v2 = legacyRow(rest);
145
    const v3 = rowFor(jobAnnouncing("newer", PIN_A), v2.receipt, "local");
146
147
    const store = join(workspace, "store.jsonl");
148
    writeFileSync(store, `${JSON.stringify(v2)}\n${JSON.stringify(v3)}\n`);
149
150
    const rows = readResultRows(store);
151
    expect(rows.map((row) => row.schema)).toEqual([BENCH_RESULT_SCHEMA_V2, BENCH_RESULT_SCHEMA]);
152
    expect(verifyResultChain(rows)).toMatchObject({ ok: true, rows: 2 });
153
  });
154
155
  test("name no staged text rather than a borrowed one", () => {
156
    const older = rowFor(jobAnnouncing("older", null), null);
157
    const { surfaceDigests, receipt, ...rest } = older;
158
    void surfaceDigests;
159
    void receipt;
160
    expect(surfacePinOf(legacyRow(rest))).toBeNull();
161
    expect(surfacePinOf(rowFor(jobAnnouncing("newer", PIN_A), null))).toBe(
162
      "catalog-lines:cccc3333 system-prompt:aaaa1111 tool-descriptions:bbbb2222",
163
    );
164
  });
165
});
166
167
// ─────────────────────────────────────────────────────────────────── helpers
168
169
const rowFor = (jobDir: string, previousReceipt: string | null, lane = "proxy"): BenchResultRow => {
170
  const run = runOf(jobDir, lane);
171
  const report = summarizeRun(run);
172
  // The checked-in manifest over exactly these four tasks, so the fixture
173
  // reads as a complete score run rather than a smoke one.
174
  const manifest = parseSuiteManifest(
175
    JSON.parse(
176
      readFileSync(fileURLToPath(new URL("../fixtures/fixture-suite.suite.json", import.meta.url)), "utf8"),
177
    ) as unknown,
178
  );
179
  return buildResultRow(report, null, classifyRun(manifest, report.perTrial.map((trial) => trial.task)), previousReceipt, {
180
    recordedAt: "2026-08-26T00:00:00.000Z",
181
  });
182
};
183
184
/** A row exactly as the v2 writer produced it: the column simply is not there. */
185
const legacyRow = (rest: Omit<BenchResultRow, "surfaceDigests" | "receipt">): BenchResultRow => {
186
  const unreceipted = { ...rest, schema: BENCH_RESULT_SCHEMA_V2 } as Omit<
187
    BenchResultRow,
188
    "receipt"
189
  >;
190
  return { ...unreceipted, receipt: receiptOf(unreceipted) };
191
};
192
packages/coder-effectiveness/src/thresholds.test.ts modified +2

@@ -182,6 +182,7 @@ describe("evaluateThresholds", () => {

182 182
      suite: "tb2-cross-section",
183 183
      lane: "proxy",
184 184
      runDigest: "effectiveness:test",
185
        surfaceDigests: null,
185 186
      trials: [
186 187
        {
187 188
          task: "fix-git",

@@ -194,6 +195,7 @@ describe("evaluateThresholds", () => {

194 195
          toolCalls: 1,
195 196
          wallClockSeconds: 10,
196 197
          threadId: null,
198
          surfaceDigests: null,
197 199
          exception: "VerifierCrashedError",
198 200
        },
199 201
      ],
packages/openagents-cli/src/cli.ts modified +10

@@ -84,6 +84,7 @@ import {

84 84
} from "./coder-resume.js";
85 85
import { toolFamilyOf } from "./coder-tool-families.js";
86 86
import { openLocalThread, threadAnnouncement, threadSyncWanted } from "./coder-local-thread.js";
87
import { surfaceAnnouncement } from "./coder-system.js";
87 88
import { ThreadTranscriptWriter } from "./coder-transcript.js";
88 89
import { delegateTool, openagentsTool, shellTool, skillTool } from "./coder-tools.js";
89 90
import { InMemoryGoalStore, goalTool } from "./coder-goals.js";

@@ -2533,6 +2534,15 @@ const coderCommand = Command.make(

2533 2534
        process.stderr.write(`${threadAnnouncement(transcriptThreadId)}\n`);
2534 2535
      }
2535 2536
2537
      // The staged text this session composed from
2538
      // (OpenAgentsInc/openagents#122), announced the same way and for the
2539
      // same reason: a bench row that cannot name the prompt it measured
2540
      // cannot tell a text change from noise. Unconditional in plain mode,
2541
      // because the surfaces are known whether or not a thread opened.
2542
      if (plain) {
2543
        process.stderr.write(`${surfaceAnnouncement()}\n`);
2544
      }
2545
2536 2546
      // The model is told what it can do rather than the reader being asked to
2537 2547
      // remember a slash command. A turn that needs three agents asks for them
2538 2548
      // mid-sentence, and `/delegate` stays as the way to launch a fan-out
packages/openagents-cli/src/coder-capability.ts modified +2 -8

@@ -18,6 +18,7 @@ import { dirname, join } from "node:path";

18 18
import { fileURLToPath } from "node:url";
19 19
20 20
import type { CoderTool } from "./coder-tools.js";
21
import { TOOL_DESCRIPTION_SURFACE } from "./coder-surfaces.generated.js";
21 22
import {
22 23
  describeLoad,
23 24
  isRefusal,

@@ -213,14 +214,7 @@ export function capabilityTool(options: CapabilityOptions): CoderTool {

213 214
    // Constant-size on purpose (OpenAgentsInc/openagents#42): the catalog is
214 215
    // searched, never enumerated here, so the standing prompt does not grow
215 216
    // as capabilities are installed.
216
    description:
217
      "Discover and load installed plugin capabilities: sandboxed, sealed programs this " +
218
      "machine already holds for common agent work. Before writing a script for a task, " +
219
      "search here first — a capability that covers it is bounded, reviewable, and returns " +
220
      "structured output. Call with `query` describing what you need to get the best " +
221
      "matches; then call again with `name` set to the exact returned name to load it and " +
222
      "make its dedicated tool available. Every later call to the loaded capability uses " +
223
      "that exact name as the tool name.",
217
    description: TOOL_DESCRIPTION_SURFACE["node.capability"],
224 218
    parameters: {
225 219
      type: "object",
226 220
      properties: {
packages/openagents-cli/src/coder-goals.ts modified +2 -4

@@ -14,6 +14,7 @@

14 14
 */
15 15
16 16
import type { CoderTool } from "./coder-tools.js";
17
import { TOOL_DESCRIPTION_SURFACE } from "./coder-surfaces.generated.js";
17 18
18 19
export type GoalStatus =
19 20
  | "active"

@@ -227,10 +228,7 @@ export function goalBudgetExhaustedPrompt(goal: PersistentGoal): string {

227 228
export function goalTool(goalStore: GoalStore): CoderTool {
228 229
  return {
229 230
    name: "goal",
230
    description:
231
      "Report a state change on the active persistent task goal for this session. " +
232
      "The goal's objective, status, and budget already accompany each turn; " +
233
      "call this with action='complete' when the goal is done and verified, or 'block'/'pause'/'resume' to update its status.",
231
    description: TOOL_DESCRIPTION_SURFACE["node.goal"],
234 232
    parameters: {
235 233
      type: "object",
236 234
      properties: {
packages/openagents-cli/src/coder-remember.ts modified +2 -11

@@ -26,6 +26,7 @@

26 26
 */
27 27
28 28
import type { CoderTool } from "./coder-tools.js";
29
import { TOOL_DESCRIPTION_SURFACE } from "./coder-surfaces.generated.js";
29 30
import { MEMORIES_PATH } from "./constants.js";
30 31
import { trackerErrorDetails } from "./tracker-request.js";
31 32

@@ -73,17 +74,7 @@ export function rememberTool(options: RememberOptions): CoderTool {

73 74
74 75
  return {
75 76
    name: "remember",
76
    description:
77
      "Store one thing the reader has explicitly asked you to remember, in their account's " +
78
      "memory. Call it when they say to remember, note, or keep something — a preference, a " +
79
      "constraint, a fact about how they work.\n\n" +
80
      "Explicit requests only. Do not call this because a conversation revealed a preference, " +
81
      "because something seemed worth keeping, or to summarize a session. A memory exists " +
82
      "because somebody asked for it.\n\n" +
83
      "There is no matching read: the account's relevant memories are attached to your context " +
84
      "by the server before you see the turn, so what is remembered already reaches you without " +
85
      "a tool call. To correct a memory you were shown, call this with the corrected sentence " +
86
      "and pass the old memory's id as `supersedes`; memories are never edited in place.",
77
    description: TOOL_DESCRIPTION_SURFACE["node.remember"],
87 78
    parameters: {
88 79
      type: "object",
89 80
      properties: {
packages/openagents-cli/src/coder-surfaces.generated.ts added +48

@@ -0,0 +1,48 @@

1
// @generated by scripts/coder-surfaces.mjs — do not edit.
2
//
3
// The staged coder text surfaces (`surfaces/coder/`), embedded at build time.
4
// Editing a sentence here is editing a build output: the artifact is the
5
// source, `pnpm run build:coder-surfaces` is the build, and
6
// `pnpm run check:coder-surfaces` refuses a tree where the two disagree.
7
8
/** The system prompt surface: `surfaces/coder/system-prompt.v1.json`. */
9
export const SYSTEM_PROMPT_SURFACE = {
10
  "coder.opening": "You are `openagents coder`, a coding assistant in a terminal. {lane}",
11
  "coder.concision": "Answer very concisely unless the reader asks for a longer response.",
12
  "coder.no_tools": "You have no tools in this session: you cannot read or write files, run commands, or reach anything outside this conversation. Answer from what the reader tells you, and say plainly when something would need a tool you do not have.",
13
  "coder.tool_list_header.node": "You have {count} tool{plural}, and no others:",
14
  "coder.tool_list_closing": "That list is complete: a capability not on it is one you do not have, whatever a model like you usually has. Read a tool's description before assuming what it covers. Where a description says what a child agent can do, that is the child's capability and not yours. Never say you ran something you did not run.",
15
  "coder.lane.local.node": "You answer from a model running locally on this machine through Ollama. Tokens here cost nothing, but generation is slow: prefer a few composite tool calls over many small ones, keep narration brief, and verify in one final pass rather than several.",
16
  "coder.lane.thread": "You answer through the OpenAgents inference proxy, on a thread opened for this session. Every round of tool calls re-sends the whole conversation to a metered model, so batch independent commands into one call and keep large dumps out of the transcript.",
17
} as const;
18
19
/** The tool-description surface: `surfaces/coder/tool-descriptions.v1.json`. */
20
export const TOOL_DESCRIPTION_SURFACE = {
21
  "node.shell": "Run a shell command on this machine. The working directory is {cwd}, so paths are relative to it and you do not need to ask where you are. Returns what the command printed. Use it for anything you would type at a terminal: reading files, listing directories, searching, git, running builds and tests. For the `openagents` CLI use the `openagents` tool instead — it carries the list of commands, so running it through here costs a turn finding out what exists. When an installed capability covers the task — the `capability` tool names what is installed — load and call it instead of scripting the same thing here: it is sandboxed, bounded, and returns structured output. Prefer it over `delegate` for single commands -- a child agent is for work worth a whole agent, not for one line of output. Both output streams come back together with the exit code. There is no terminal, so a command that would prompt gets end-of-file instead of waiting; pass a flag that answers the prompt. A few commands that cannot be undone are refused, such as erasing a root or a home directory, reformatting a disk, or halting the machine. Work economically: batch independent commands into one call with && instead of one call each — every call replays the conversation so far. Disable pagers and prefer quiet flags (for example `git --no-pager`, `PAGER=cat`), and ask for summaries before full dumps: get the shape of a thing (a stat, a listing, a count) before printing all of it, and print all of it only for what you are actually deciding about.",
22
  "node.skill": "Read one of this repository's skills: a written procedure for a kind of work, with the conventions, commands, and rules it needs. Call it before doing work a skill covers, and follow what it says over your own habits. Skills available:\n{catalog}",
23
  "node.openagents.head": "Run the OpenAgents CLI: issues, projects, repositories, the forum, authentication, and any API route through `api`. Pass the arguments after `openagents` as a list, without `openagents` itself.\n\n",
24
  "node.openagents.commands": "Commands:\n{tree}\n\n",
25
  "node.openagents.body": "Run `<command> --help` when you need a flag you do not know; the commands above are the whole set, so you do not need to go looking for them.\n\nRead the plain output. It is what a person reads and it is small: a list of three issues is 442 bytes plain and 20,000 as JSON, because the JSON carries every issue's whole body. Add `--json` only when you need one field out of one record, and prefer a narrower command over a wider one you then have to read past.\n\nReads are free; a write is visible to other people at once, so say what you are about to write before the first one. Read the `openagents-cli` skill for the auth model and what works with no credential.",
26
  "node.goal": "Report a state change on the active persistent task goal for this session. The goal's objective, status, and budget already accompany each turn; call this with action='complete' when the goal is done and verified, or 'block'/'pause'/'resume' to update its status.",
27
  "node.remember": "Store one thing the reader has explicitly asked you to remember, in their account's memory. Call it when they say to remember, note, or keep something — a preference, a constraint, a fact about how they work.\n\nExplicit requests only. Do not call this because a conversation revealed a preference, because something seemed worth keeping, or to summarize a session. A memory exists because somebody asked for it.\n\nThere is no matching read: the account's relevant memories are attached to your context by the server before you see the turn, so what is remembered already reaches you without a tool call. To correct a memory you were shown, call this with the corrected sentence and pass the old memory's id as `supersedes`; memories are never edited in place.",
28
  "node.delegate.head": "Run one prompt on independent child coding agents in parallel, in this repository, and return what each one found or did. Use it whenever work splits into parts that do not depend on each other: several files to change the same way, several hypotheses to check, several tests to run down. Each child is a full coding agent with its own file and shell tools, it starts with no context from this conversation, and it cannot ask questions, so the prompt has to be self-contained. Children run on this session's budget. Every child runs the same prompt, and each is told separately which number it is, so write the prompt for whichever child reads it: say \"read the file at your own number\" rather than naming one child (\"you are child 1\"), which gives every child the same work and wastes the fan-out. Prefer one call with a count over several calls. At most {max_count} children.",
29
  "node.delegate.lane_default": " Children run on {label} unless `model` names another lane.\n\n",
30
  "node.delegate.lane_preamble": "A lane is a harness and a model together. The harness runs the child and gives it its tools; the model is what answers. Choosing a lane chooses both:\n",
31
  "node.capability": "Discover and load installed plugin capabilities: sandboxed, sealed programs this machine already holds for common agent work. Before writing a script for a task, search here first — a capability that covers it is bounded, reviewable, and returns structured output. Call with `query` describing what you need to get the best matches; then call again with `name` set to the exact returned name to load it and make its dedicated tool available. Every later call to the loaded capability uses that exact name as the tool name.",
32
  "node.family.gemini.shell": " IMPORTANT: batch independent commands into ONE call joined with && — each separate call replays the whole conversation to the model, so ten one-line calls cost several times what one composite call costs. Never run one small inspection per call. Read only the region you need; prefer offset/limit ranged reads or summaries over whole-file dumps, which are token-inefficient.",
33
  "node.family.local.shell": " This session's model generates slowly on this machine: prefer a few composite calls over many small ones, and keep verification to one final pass.",
34
} as const;
35
36
/**
37
 * Every staged surface and the digest of the artifact this was built from.
38
 *
39
 * A run records these so a bench row names exactly which text produced it.
40
 */
41
export const CODER_SURFACE_DIGESTS = {
42
  "system-prompt": "sha256:8f0fbbbb38f4ce609a09fe2ba4e03ca160ff2c0d0f698dd4676c2bf940ded849",
43
  "tool-descriptions": "sha256:513c7518c72ba8c9959e517ea9f1262d7d3ee6b08452ddb48568bdabf6383c1b",
44
  "catalog-lines": "sha256:98a96bcf6acfd5847bc1bddcc575761af71be7ea144f3584b3594b24b2f80911",
45
} as const;
46
47
/** The staged surface ids, in index order. */
48
export type CoderSurfaceId = keyof typeof CODER_SURFACE_DIGESTS;
packages/openagents-cli/src/coder-system.ts modified +31 -20

@@ -1,4 +1,5 @@

1 1
import type { CoderTool } from "./coder-tools.js";
2
import { CODER_SURFACE_DIGESTS, SYSTEM_PROMPT_SURFACE } from "./coder-surfaces.generated.js";
2 3
3 4
/**
4 5
 * What a coder session tells the model about itself, on every lane.

@@ -24,21 +25,19 @@ export const systemPrompt = (

24 25
  standing?: string,
25 26
): string => {
26 27
  const lines = [
27
    `You are \`openagents coder\`, a coding assistant in a terminal. ${lane}`,
28
    SYSTEM_PROMPT_SURFACE["coder.opening"].replace("{lane}", lane),
28 29
    "",
29
    "Answer very concisely unless the reader asks for a longer response.",
30
    SYSTEM_PROMPT_SURFACE["coder.concision"],
30 31
    "",
31 32
  ];
32 33
33 34
  if (tools.length === 0) {
34
    lines.push(
35
      "You have no tools in this session: you cannot read or write files, run commands, or " +
36
        "reach anything outside this conversation. Answer from what the reader tells you, and " +
37
        "say plainly when something would need a tool you do not have.",
38
    );
35
    lines.push(SYSTEM_PROMPT_SURFACE["coder.no_tools"]);
39 36
  } else {
40 37
    lines.push(
41
      `You have ${String(tools.length)} tool${tools.length === 1 ? "" : "s"}, and no others:`,
38
      SYSTEM_PROMPT_SURFACE["coder.tool_list_header.node"]
39
        .replace("{count}", String(tools.length))
40
        .replace("{plural}", tools.length === 1 ? "" : "s"),
42 41
      ...tools.map((tool) => `- \`${tool.name}\``),
43 42
      "",
44 43
      // Stated as a closed list rather than by naming the capabilities that are

@@ -46,10 +45,7 @@ export const systemPrompt = (

46 45
      // there was no shell, and then there was one — and a system message that
47 46
      // has to be edited when the tool list changes is one that will be wrong
48 47
      // in between.
49
      "That list is complete: a capability not on it is one you do not have, whatever a model " +
50
        "like you usually has. Read a tool's description before assuming what it covers. Where " +
51
        "a description says what a child agent can do, that is the child's capability and not " +
52
        "yours. Never say you ran something you did not run.",
48
      SYSTEM_PROMPT_SURFACE["coder.tool_list_closing"],
53 49
    );
54 50
  }
55 51

@@ -59,13 +55,28 @@ export const systemPrompt = (

59 55
};
60 56
61 57
/** The lane sentence for a session answering from a model on this machine. */
62
export const LOCAL_LANE =
63
  "You answer from a model running locally on this machine through Ollama. Tokens here cost " +
64
  "nothing, but generation is slow: prefer a few composite tool calls over many small ones, " +
65
  "keep narration brief, and verify in one final pass rather than several.";
58
export const LOCAL_LANE = SYSTEM_PROMPT_SURFACE["coder.lane.local.node"];
66 59
67 60
/** The lane sentence for a session answering through the account's thread. */
68
export const THREAD_LANE =
69
  "You answer through the OpenAgents inference proxy, on a thread opened for this session. " +
70
  "Every round of tool calls re-sends the whole conversation to a metered model, so batch " +
71
  "independent commands into one call and keep large dumps out of the transcript.";
61
export const THREAD_LANE = SYSTEM_PROMPT_SURFACE["coder.lane.thread"];
62
63
/**
64
 * The machine-readable staged-text announcement.
65
 *
66
 * A bench row records which text produced it (OpenAgentsInc/openagents#122).
67
 * The digests cannot be read off the repository at scoring time — a run scored
68
 * a week later would be pinned to whatever the tree says then — so the session
69
 * names them itself, on stderr, in the same shape and the same place as the
70
 * thread announcement (`[oa:thread <uuid>]`, #38): one line, parsed by
71
 * `packages/coder-effectiveness/src/harbor-job.ts` with
72
 * `\[oa:surfaces ([^\]]+)\]`.
73
 *
74
 * Absent from an older CLI, which is not an error: the row then records no
75
 * surface pin rather than a wrong one.
76
 */
77
export const surfaceAnnouncement = (
78
  digests: Readonly<Record<string, string>> = CODER_SURFACE_DIGESTS,
79
): string =>
80
  `[oa:surfaces ${Object.entries(digests)
81
    .map(([id, digest]) => `${id}=${digest}`)
82
    .join(",")}]`;
packages/openagents-cli/src/coder-tool-families.ts modified +3 -15

@@ -19,6 +19,7 @@

19 19
 */
20 20
21 21
import type { CoderTool } from "./coder-tools.js";
22
import { TOOL_DESCRIPTION_SURFACE } from "./coder-surfaces.generated.js";
22 23
23 24
/** The families with distinct declared emphasis. `default` adds nothing. */
24 25
export type ToolFamily = "default" | "gemini" | "local";

@@ -45,21 +46,8 @@ export const toolFamilyOf = (model: string | undefined): ToolFamily => {

45 46
 * the same suite, and the analysis document above records why each exists.
46 47
 */
47 48
const emphasis: Partial<Record<ToolFamily, Partial<Record<string, string>>>> = {
48
  gemini: {
49
    shell:
50
      " IMPORTANT: batch independent commands into ONE call joined with && — " +
51
      "each separate call replays the whole conversation to the model, so ten " +
52
      "one-line calls cost several times what one composite call costs. Never " +
53
      "run one small inspection per call. Read only the region you need; prefer " +
54
      "offset/limit ranged reads or summaries over whole-file dumps, which are " +
55
      "token-inefficient.",
56
  },
57
  local: {
58
    shell:
59
      " This session's model generates slowly on this machine: prefer a few " +
60
      "composite calls over many small ones, and keep verification to one " +
61
      "final pass.",
62
  },
49
  gemini: { shell: TOOL_DESCRIPTION_SURFACE["node.family.gemini.shell"] },
50
  local: { shell: TOOL_DESCRIPTION_SURFACE["node.family.local.shell"] },
63 51
};
64 52
65 53
/** A tool's description as declared to this family's model. */
packages/openagents-cli/src/coder-tools.ts modified +17 -52

@@ -18,6 +18,7 @@ import { spawnSync } from "node:child_process";

18 18
import { existsSync } from "node:fs";
19 19
import { fileURLToPath } from "node:url";
20 20
21
import { TOOL_DESCRIPTION_SURFACE } from "./coder-surfaces.generated.js";
21 22
import type { CoderDelegation } from "./coder-session.js";
22 23
import type { CoderTaskRegistry } from "./coder-tasks.js";
23 24
import {

@@ -66,26 +67,21 @@ export function delegateTool(delegation: CoderDelegation): CoderTool {

66 67
  return {
67 68
    name: "delegate",
68 69
    description:
69
      "Run one prompt on independent child coding agents in parallel, in this repository, and " +
70
      "return what each one found or did. Use it whenever work splits into parts that do not " +
71
      "depend on each other: several files to change the same way, several hypotheses to check, " +
72
      "several tests to run down. Each child is a full coding agent with its own file and shell " +
73
      "tools, it starts with no context from this conversation, and it cannot ask questions, so " +
74
      "the prompt has to be self-contained. Children run on this session's budget. Every child " +
75
      "runs the same prompt, and each is told separately which number it is, so write the prompt " +
76
      'for whichever child reads it: say "read the file at your own number" rather than naming ' +
77
      'one child ("you are child 1"), which gives every child the same work and wastes the ' +
78
      "fan-out. Prefer one call with a count over several calls. At most " +
79
      `${String(MAX_DELEGATE_COUNT)} children.` +
70
      TOOL_DESCRIPTION_SURFACE["node.delegate.head"].replace(
71
        "{max_count}",
72
        String(MAX_DELEGATE_COUNT),
73
      ) +
80 74
      (models.length === 0
81 75
        ? ""
82
        : ` Children run on ${delegation.label} unless \`model\` names another lane.\n\n` +
76
        : TOOL_DESCRIPTION_SURFACE["node.delegate.lane_default"].replace(
77
            "{label}",
78
            delegation.label,
79
          ) +
83 80
          // Named in full, because the enum alone reads as a list of models and
84 81
          // it is not one: two of these lanes are the same model under
85 82
          // different harnesses, and a session that could not tell described
86 83
          // one of them as a "fast preview / experimental" model it is not.
87
          "A lane is a harness and a model together. The harness runs the child and gives it " +
88
          "its tools; the model is what answers. Choosing a lane chooses both:\n" +
84
          TOOL_DESCRIPTION_SURFACE["node.delegate.lane_preamble"] +
89 85
          CHILD_LANES.filter((lane) => models.includes(lane.name))
90 86
            .map((lane) => `- ${describeChildLane(lane)}`)
91 87
            .join("\n")),

@@ -261,11 +257,7 @@ export function skillTool(skills: ReadonlyArray<CoderSkill>): CoderTool {

261 257
  const catalog = skills.map((skill) => catalogEntry(skill)).join("\n");
262 258
  return {
263 259
    name: "skill",
264
    description:
265
      "Read one of this repository's skills: a written procedure for a kind of work, with the " +
266
      "conventions, commands, and rules it needs. Call it before doing work a skill covers, and " +
267
      "follow what it says over your own habits. Skills available:\n" +
268
      catalog,
260
    description: TOOL_DESCRIPTION_SURFACE["node.skill"].replace("{catalog}", catalog),
269 261
    parameters: {
270 262
      type: "object",
271 263
      properties: {

@@ -402,19 +394,11 @@ export function openagentsTool(): CoderTool {

402 394
  return {
403 395
    name: "openagents",
404 396
    description:
405
      "Run the OpenAgents CLI: issues, projects, repositories, the forum, authentication, and " +
406
      "any API route through `api`. Pass the arguments after `openagents` as a list, without " +
407
      "`openagents` itself.\n\n" +
408
      (tree === undefined ? "" : `Commands:\n${tree}\n\n`) +
409
      "Run `<command> --help` when you need a flag you do not know; the commands above are the " +
410
      "whole set, so you do not need to go looking for them.\n\n" +
411
      "Read the plain output. It is what a person reads and it is small: a list of three issues " +
412
      "is 442 bytes plain and 20,000 as JSON, because the JSON carries every issue's whole body. " +
413
      "Add `--json` only when you need one field out of one record, and prefer a narrower " +
414
      "command over a wider one you then have to read past.\n\n" +
415
      "Reads are free; a write is visible to other people at once, so say what you are about to " +
416
      "write before the first one. Read the `openagents-cli` skill for the auth model and what " +
417
      "works with no credential.",
397
      TOOL_DESCRIPTION_SURFACE["node.openagents.head"] +
398
      (tree === undefined
399
        ? ""
400
        : TOOL_DESCRIPTION_SURFACE["node.openagents.commands"].replace("{tree}", tree)) +
401
      TOOL_DESCRIPTION_SURFACE["node.openagents.body"],
418 402
    parameters: {
419 403
      type: "object",
420 404
      properties: {

@@ -533,26 +517,7 @@ export function openagentsTool(): CoderTool {

533 517
export function shellTool(cwd: string): CoderTool {
534 518
  return {
535 519
    name: "shell",
536
    description:
537
      `Run a shell command on this machine. The working directory is ${cwd}, so paths are ` +
538
      "relative to it and you do not need to ask where you are. Returns what the command " +
539
      "printed. Use it for anything you would type at a terminal: reading files, listing " +
540
      "directories, searching, git, running builds and tests. For the `openagents` CLI use the " +
541
      "`openagents` tool instead — it carries the list of commands, so running it through here " +
542
      "costs a turn finding out what exists. When an installed capability covers the task — " +
543
      "the `capability` tool names what is installed — load and call it instead of scripting " +
544
      "the same thing here: it is sandboxed, bounded, and returns structured output. " +
545
      "Prefer it over `delegate` for " +
546
      "single commands -- a child agent is for work worth a whole agent, not for one line of " +
547
      "output. Both output streams come back together with the exit code. There is no terminal, " +
548
      "so a command that would prompt gets end-of-file instead of waiting; pass a flag that " +
549
      "answers the prompt. A few commands that cannot be undone are refused, such as erasing a " +
550
      "root or a home directory, reformatting a disk, or halting the machine. Work economically: " +
551
      "batch independent commands into one call with && instead of one call each — every call " +
552
      "replays the conversation so far. Disable pagers and prefer quiet flags (for example " +
553
      "`git --no-pager`, `PAGER=cat`), and ask for summaries before full dumps: get the shape " +
554
      "of a thing (a stat, a listing, a count) before printing all of it, and print all of it " +
555
      "only for what you are actually deciding about.",
520
    description: TOOL_DESCRIPTION_SURFACE["node.shell"].replace("{cwd}", cwd),
556 521
    parameters: {
557 522
      type: "object",
558 523
      properties: {
packages/openagents-cli/test/coder-surfaces.test.ts added +85

@@ -0,0 +1,85 @@

1
/**
2
 * The staged text surfaces, and the announcement that names them.
3
 *
4
 * `surfaces/coder/` owns the coder's optimizable text
5
 * (OpenAgentsInc/openagents#122) and `coder-surfaces.generated.ts` is the copy
6
 * this package compiles. `pnpm run check:coder-surfaces` is the gate that
7
 * refuses a stale build; this is the second net, inside the ordinary test
8
 * sweep, so a surface edited without the rebuild fails here too rather than
9
 * shipping the previous sentence in silence.
10
 */
11
12
import { createHash } from "node:crypto";
13
import { readFileSync } from "node:fs";
14
import { fileURLToPath } from "node:url";
15
import { describe, expect, test } from "vite-plus/test";
16
17
import {
18
  CODER_SURFACE_DIGESTS,
19
  SYSTEM_PROMPT_SURFACE,
20
  TOOL_DESCRIPTION_SURFACE,
21
} from "../src/coder-surfaces.generated.js";
22
import { surfaceAnnouncement, systemPrompt, THREAD_LANE } from "../src/coder-system.js";
23
24
const artifact = (name: string): string =>
25
  readFileSync(fileURLToPath(new URL(`../../../surfaces/coder/${name}`, import.meta.url)), "utf8");
26
27
const index = JSON.parse(artifact("index.json")) as {
28
  surfaces: Record<string, { file: string; digest: string }>;
29
};
30
31
describe("the embedded surfaces", () => {
32
  test("carry the digest of the artifact they were built from", () => {
33
    for (const [id, digest] of Object.entries(CODER_SURFACE_DIGESTS)) {
34
      const entry = index.surfaces[id];
35
      expect(entry, `${id} is not in surfaces/coder/index.json`).toBeDefined();
36
      const found = `sha256:${createHash("sha256").update(artifact(entry!.file)).digest("hex")}`;
37
      expect(found, `${entry!.file} does not digest to what index.json pins`).toBe(entry!.digest);
38
      expect(digest, `the embedded ${id} digest is stale`).toBe(entry!.digest);
39
    }
40
  });
41
42
  test("hold text rather than placeholders", () => {
43
    for (const [key, value] of Object.entries({
44
      ...SYSTEM_PROMPT_SURFACE,
45
      ...TOOL_DESCRIPTION_SURFACE,
46
    })) {
47
      expect(value.length, `${key} is empty`).toBeGreaterThan(0);
48
    }
49
  });
50
});
51
52
describe("the staged-text announcement", () => {
53
  // The shape `packages/coder-effectiveness/src/harbor-job.ts` parses. A change
54
  // here without a change there loses the pin on every later bench row.
55
  test("is one line naming every surface by digest", () => {
56
    const line = surfaceAnnouncement();
57
    expect(line).toMatch(/^\[oa:surfaces .+\]$/u);
58
    const inner = /\[oa:surfaces ([^\]]+)\]/u.exec(line)?.[1];
59
    expect(inner).toBeDefined();
60
    const parsed = Object.fromEntries(
61
      inner!.split(",").map((pair) => {
62
        const at = pair.indexOf("=");
63
        return [pair.slice(0, at), pair.slice(at + 1)];
64
      }),
65
    );
66
    expect(parsed).toEqual(CODER_SURFACE_DIGESTS);
67
  });
68
});
69
70
describe("the system prompt", () => {
71
  test("is composed from the staged surface", () => {
72
    const prompt = systemPrompt([], THREAD_LANE);
73
    expect(prompt).toContain(SYSTEM_PROMPT_SURFACE["coder.concision"]);
74
    expect(prompt).toContain(SYSTEM_PROMPT_SURFACE["coder.no_tools"]);
75
    expect(prompt).toContain(THREAD_LANE);
76
    expect(prompt).not.toContain("{lane}");
77
  });
78
79
  test("singularizes a one-tool session and does not leave the placeholder", () => {
80
    const one = systemPrompt([{ name: "shell" } as never], THREAD_LANE);
81
    expect(one).toContain("You have 1 tool, and no others:");
82
    expect(one).not.toContain("{count}");
83
    expect(one).not.toContain("{plural}");
84
  });
85
});
scripts/coder-surfaces-literal-search.mjs added +95

@@ -0,0 +1,95 @@

1
// Prove no staged surface sentence survives as a literal outside its artifact.
2
//
3
// The extraction in OpenAgentsInc/openagents#122 is only worth its cost if the
4
// artifact is the single home of the text. A leftover copy in a `.rs` or `.ts`
5
// file is worse than no extraction at all: an optimizer would diff the
6
// artifact, the build would embed the artifact, and the second copy would go on
7
// being the one some path actually read.
8
//
9
// So this takes a distinctive fragment of every staged string and searches the
10
// whole tree for it, allowing exactly the places a copy is meant to be:
11
//
12
//   - `surfaces/coder/` — the artifacts themselves;
13
//   - the two generated modules, which ARE the build output;
14
//   - `plugins/<id>/manifest.json` — where a catalog line is edited, the
15
//     artifact being its mirror rather than its replacement;
16
//   - the two Rust golden files, which pin the composed output on purpose;
17
//   - `docs/`, where a document quoting a sentence is prose about the text and
18
//     not a copy any code path reads.
19
//
20
// Reported rather than wired into `check:fast`: the search is a `ripgrep` per
21
// staged string over the whole repository, which is the wrong cost for a gate
22
// that runs on every push. `check:coder-surfaces` is the gate.
23
//
24
// Usage: node scripts/coder-surfaces-literal-search.mjs
25
26
import { readFileSync } from "node:fs";
27
import { dirname, join } from "node:path";
28
import { fileURLToPath } from "node:url";
29
import { spawnSync } from "node:child_process";
30
31
const ROOT = dirname(dirname(fileURLToPath(import.meta.url)));
32
33
const ALLOWED = [
34
  "surfaces/coder/",
35
  "crates/openagents-cli/src/surfaces.rs",
36
  "packages/openagents-cli/src/coder-surfaces.generated.ts",
37
  "plugins/",
38
  "crates/coder-lite/tests/coder-surfaces-golden.json",
39
  "crates/openagents-cli/tests/coder-surfaces-golden.json",
40
  "docs/",
41
];
42
43
const surfaces = ["system-prompt", "tool-descriptions", "catalog-lines"].map((id) => ({
44
  id,
45
  text: JSON.parse(readFileSync(join(ROOT, "surfaces", "coder", `${id}.v1.json`), "utf8")).text,
46
}));
47
48
/**
49
 * A fragment distinctive enough to search on: the first line of at least 40
50
 * characters, with the `{placeholder}` tokens removed so a template still
51
 * matches the prose it was cut from.
52
 */
53
const probeOf = (text) => {
54
  const cleaned = text.replaceAll(/\{[a-z_]+\}/gu, "").trim();
55
  for (const line of cleaned.split("\n")) {
56
    const trimmed = line.trim();
57
    if (trimmed.length >= 40) return trimmed.slice(0, 60);
58
  }
59
  return cleaned.length >= 20 ? cleaned.slice(0, 60) : null;
60
};
61
62
let searched = 0;
63
let skipped = 0;
64
const strays = [];
65
66
for (const surface of surfaces) {
67
  for (const [key, text] of Object.entries(surface.text)) {
68
    const needle = probeOf(text);
69
    if (needle === null) {
70
      skipped += 1;
71
      console.log(`  skipped ${surface.id}#${key}: too short to search distinctively`);
72
      continue;
73
    }
74
    searched += 1;
75
    const found = spawnSync("rg", ["--fixed-strings", "--files-with-matches", needle, "."], {
76
      cwd: ROOT,
77
      encoding: "utf8",
78
    });
79
    const files = (found.stdout ?? "")
80
      .split("\n")
81
      .map((line) => line.trim().replace(/^\.\//u, ""))
82
      .filter((line) => line !== "");
83
    const stray = files.filter((file) => !ALLOWED.some((prefix) => file.startsWith(prefix)));
84
    if (stray.length > 0) strays.push({ surface: surface.id, key, stray });
85
  }
86
}
87
88
for (const { surface, key, stray } of strays) {
89
  console.error(`STRAY LITERAL ${surface}#${key}: ${stray.join(", ")}`);
90
}
91
console.log(
92
  `\n${String(searched)} staged strings searched across the tree (${String(skipped)} too short to search).`,
93
);
94
console.log(`stray copies: ${String(strays.length)}`);
95
process.exit(strays.length > 0 ? 1 : 0);
scripts/coder-surfaces.mjs added +277

@@ -0,0 +1,277 @@

1
// The staged coder text surfaces: build the artifacts, or prove the build ran.
2
//
3
// `surfaces/coder/` holds the coder's optimizable text as data
4
// (OpenAgentsInc/openagents#122): the system prompt, the tool descriptions,
5
// and the capability catalog lines. Each artifact carries a schema id and each
6
// gets a content digest in `surfaces/coder/index.json`. Two consumers embed
7
// the text at build time — `crates/openagents-cli/src/surfaces.rs` and
8
// `packages/openagents-cli/src/coder-surfaces.generated.ts` — so neither CLI
9
// keeps a second copy of a sentence the artifact owns.
10
//
11
// THE FAILURE THIS SCRIPT EXISTS TO PREVENT is the knowledge base's: a corpus
12
// edit that is not followed by the rebuild ships nothing, and says nothing
13
// while it does. Here the same edit has two ways to go quiet:
14
//
15
//   1. The generated modules are the copies the CLIs actually compile. Edit an
16
//      artifact and skip the rebuild and the CLIs keep shipping the old
17
//      sentence.
18
//   2. The digests in `index.json` are what a bench row records as the text
19
//      that produced it. Edit an artifact and skip the re-pin and every later
20
//      row names text that was never run.
21
//
22
// So `--check` (the default, wired into `check:fast` as `check:coder-surfaces`)
23
// regenerates everything in memory and refuses when any of it differs from
24
// what is on disk, naming the artifact and the reason. `--write` does the
25
// rebuild.
26
//
27
// The catalog-lines artifact is deliberately the odd one out and the doc says
28
// so: a plugin's catalog line is already data — the top-level `description` of
29
// its `plugins/<id>/manifest.json`, discovered at runtime by
30
// `discover_catalog()` — so staging it is not a move. The artifact mirrors
31
// those descriptions into one diffable object with a digest, and the check
32
// fails when a manifest and the mirror disagree. The manifest stays the place
33
// the text is edited.
34
//
35
// Usage:
36
//   node scripts/coder-surfaces.mjs            # check
37
//   node scripts/coder-surfaces.mjs --write    # rebuild and re-pin
38
39
import { createHash } from "node:crypto";
40
import { readdirSync, readFileSync, writeFileSync } from "node:fs";
41
import { dirname, join } from "node:path";
42
import { fileURLToPath } from "node:url";
43
44
const ROOT = dirname(dirname(fileURLToPath(import.meta.url)));
45
const SURFACES_DIR = join(ROOT, "surfaces", "coder");
46
const PLUGINS_DIR = join(ROOT, "plugins");
47
const INDEX_PATH = join(SURFACES_DIR, "index.json");
48
const RUST_MODULE = join(ROOT, "crates", "openagents-cli", "src", "surfaces.rs");
49
const TS_MODULE = join(ROOT, "packages", "openagents-cli", "src", "coder-surfaces.generated.ts");
50
51
/** The artifacts, in the order the index lists them. */
52
const SURFACES = [
53
  { id: "system-prompt", file: "system-prompt.v1.json", authored: true },
54
  { id: "tool-descriptions", file: "tool-descriptions.v1.json", authored: true },
55
  { id: "catalog-lines", file: "catalog-lines.v1.json", authored: false },
56
];
57
58
const INDEX_SCHEMA = "openagents.coder_surface_index.v1";
59
60
/** Exactly how an artifact is serialised, so a digest is a fact about bytes. */
61
const canonical = (doc) => `${JSON.stringify(doc, null, 2)}\n`;
62
63
const digestOf = (text) => `sha256:${createHash("sha256").update(text).digest("hex")}`;
64
65
const readJson = (path) => JSON.parse(readFileSync(path, "utf8"));
66
67
/**
68
 * The catalog-lines artifact, mirrored from the installed plugin manifests.
69
 *
70
 * Keyed by the manifest's own `name`, which is the id the `capability` tool
71
 * loads by, so a candidate diff over this surface names the same plugin the
72
 * catalog does.
73
 */
74
const buildCatalogLines = () => {
75
  const text = {};
76
  for (const entry of readdirSync(PLUGINS_DIR, { withFileTypes: true }).sort((left, right) =>
77
    left.name.localeCompare(right.name),
78
  )) {
79
    if (!entry.isDirectory()) continue;
80
    const manifestPath = join(PLUGINS_DIR, entry.name, "manifest.json");
81
    let manifest;
82
    try {
83
      manifest = readJson(manifestPath);
84
    } catch {
85
      continue; // Not a plugin directory. Discovery skips these too.
86
    }
87
    if (typeof manifest.name !== "string" || typeof manifest.description !== "string") continue;
88
    text[manifest.name] = manifest.description;
89
  }
90
  return {
91
    schema: "openagents.coder_surface.catalog_lines.v1",
92
    surface: "catalog-lines",
93
    // Named here because it is the one surface this script does not own: the
94
    // manifest is where the text is edited, and this artifact is its mirror.
95
    source: "plugins/<id>/manifest.json#description",
96
    text,
97
  };
98
};
99
100
/**
101
 * Which embedded module a key reaches.
102
 *
103
 * The two harnesses have forked on some of this text — a `node` or `rust`
104
 * segment in the key is what says so, and a key with neither is shared. Only
105
 * the keys a harness actually reads are emitted into its module, so a constant
106
 * in either module is one that harness uses.
107
 */
108
const consumersOf = (key) => {
109
  const segments = key.split(".");
110
  if (segments.includes("node")) return ["node"];
111
  if (segments.includes("rust") || segments[0] === "coder_lite") return ["rust"];
112
  return ["node", "rust"];
113
};
114
115
const entriesFor = (doc, consumer) =>
116
  Object.entries(doc.text).filter(([key]) => consumersOf(key).includes(consumer));
117
118
const rustConst = (key) => key.replaceAll(/[.-]/gu, "_").toUpperCase();
119
120
const rustLiteral = (value) => JSON.stringify(value);
121
122
const rustModule = (docs, digests) => {
123
  const lines = [
124
    "// @generated by scripts/coder-surfaces.mjs — do not edit.",
125
    "//",
126
    "// The staged coder text surfaces (`surfaces/coder/`), embedded at build",
127
    "// time. Editing a sentence here is editing a build output: the artifact",
128
    "// is the source, `pnpm run build:coder-surfaces` is the build, and",
129
    "// `pnpm run check:coder-surfaces` refuses a tree where the two disagree.",
130
    "//",
131
    "//! Staged coder text surfaces, embedded from `surfaces/coder/`.",
132
    "",
133
    "/// The system prompt surface: `surfaces/coder/system-prompt.v1.json`.",
134
    "pub mod system_prompt {",
135
  ];
136
  for (const [key, value] of entriesFor(docs["system-prompt"], "rust")) {
137
    lines.push(`    /// \`${key}\``);
138
    lines.push(`    pub const ${rustConst(key)}: &str = ${rustLiteral(value)};`);
139
  }
140
  lines.push("}", "");
141
  lines.push("/// The tool-description surface: `surfaces/coder/tool-descriptions.v1.json`.");
142
  lines.push("pub mod tool_descriptions {");
143
  for (const [key, value] of entriesFor(docs["tool-descriptions"], "rust")) {
144
    lines.push(`    /// \`${key}\``);
145
    lines.push(`    pub const ${rustConst(key)}: &str = ${rustLiteral(value)};`);
146
  }
147
  lines.push("}", "");
148
  lines.push("/// Every staged surface and the digest of the artifact this was built from.");
149
  lines.push("///");
150
  lines.push("/// A run records these so a bench row names exactly which text produced it.");
151
  lines.push(`pub const SURFACE_DIGESTS: [(&str, &str); ${String(digests.length)}] = [`);
152
  for (const [id, digest] of digests) {
153
    lines.push(`    (${rustLiteral(id)}, ${rustLiteral(digest)}),`);
154
  }
155
  lines.push("];", "");
156
  return lines.join("\n");
157
};
158
159
const tsModule = (docs, digests) => {
160
  const record = (doc) =>
161
    entriesFor(doc, "node")
162
      .map(([key, value]) => `  ${JSON.stringify(key)}: ${JSON.stringify(value)},`)
163
      .join("\n");
164
  return `// @generated by scripts/coder-surfaces.mjs — do not edit.
165
//
166
// The staged coder text surfaces (\`surfaces/coder/\`), embedded at build time.
167
// Editing a sentence here is editing a build output: the artifact is the
168
// source, \`pnpm run build:coder-surfaces\` is the build, and
169
// \`pnpm run check:coder-surfaces\` refuses a tree where the two disagree.
170
171
/** The system prompt surface: \`surfaces/coder/system-prompt.v1.json\`. */
172
export const SYSTEM_PROMPT_SURFACE = {
173
${record(docs["system-prompt"])}
174
} as const;
175
176
/** The tool-description surface: \`surfaces/coder/tool-descriptions.v1.json\`. */
177
export const TOOL_DESCRIPTION_SURFACE = {
178
${record(docs["tool-descriptions"])}
179
} as const;
180
181
/**
182
 * Every staged surface and the digest of the artifact this was built from.
183
 *
184
 * A run records these so a bench row names exactly which text produced it.
185
 */
186
export const CODER_SURFACE_DIGESTS = {
187
${digests.map(([id, digest]) => `  ${JSON.stringify(id)}: ${JSON.stringify(digest)},`).join("\n")}
188
} as const;
189
190
/** The staged surface ids, in index order. */
191
export type CoderSurfaceId = keyof typeof CODER_SURFACE_DIGESTS;
192
`;
193
};
194
195
const build = () => {
196
  const docs = {};
197
  const files = {};
198
  for (const surface of SURFACES) {
199
    const path = join(SURFACES_DIR, surface.file);
200
    const doc = surface.authored ? readJson(path) : buildCatalogLines();
201
    docs[surface.id] = doc;
202
    files[surface.file] = canonical(doc);
203
  }
204
205
  const digests = SURFACES.map((surface) => [surface.id, digestOf(files[surface.file])]);
206
207
  const index = {
208
    schema: INDEX_SCHEMA,
209
    surfaces: Object.fromEntries(
210
      SURFACES.map((surface, at) => [
211
        surface.id,
212
        {
213
          file: surface.file,
214
          schema: docs[surface.id].schema,
215
          keys: Object.keys(docs[surface.id].text).length,
216
          digest: digests[at][1],
217
        },
218
      ]),
219
    ),
220
  };
221
222
  return {
223
    files,
224
    index: canonical(index),
225
    rust: rustModule(docs, digests),
226
    ts: tsModule(docs, digests),
227
  };
228
};
229
230
const failures = [];
231
const complain = (what, detail) => failures.push(`${what}: ${detail}`);
232
233
const compare = (path, expected, label) => {
234
  let found;
235
  try {
236
    found = readFileSync(path, "utf8");
237
  } catch {
238
    complain(label, `${path} does not exist. Run \`pnpm run build:coder-surfaces\`.`);
239
    return;
240
  }
241
  if (found !== expected) {
242
    complain(
243
      label,
244
      `${path} is stale — it does not match what the staged artifacts build to. A surface was edited without the rebuild, so what ships is not what the artifact says. Run \`pnpm run build:coder-surfaces\`.`,
245
    );
246
  }
247
};
248
249
const built = build();
250
const write = process.argv.includes("--write");
251
252
if (write) {
253
  for (const surface of SURFACES) {
254
    writeFileSync(join(SURFACES_DIR, surface.file), built.files[surface.file]);
255
  }
256
  writeFileSync(INDEX_PATH, built.index);
257
  writeFileSync(RUST_MODULE, built.rust);
258
  writeFileSync(TS_MODULE, built.ts);
259
  console.log(
260
    `coder surfaces rebuilt: ${SURFACES.map((surface) => surface.id).join(", ")} + index, Rust module, TypeScript module`,
261
  );
262
} else {
263
  for (const surface of SURFACES) {
264
    compare(join(SURFACES_DIR, surface.file), built.files[surface.file], `surface ${surface.id}`);
265
  }
266
  compare(INDEX_PATH, built.index, "surface digest index");
267
  compare(RUST_MODULE, built.rust, "embedded Rust module");
268
  compare(TS_MODULE, built.ts, "embedded TypeScript module");
269
270
  if (failures.length > 0) {
271
    console.error("coder surfaces are out of date:\n");
272
    for (const failure of failures) console.error(`  - ${failure}`);
273
    console.error("");
274
    process.exit(1);
275
  }
276
  console.log(`coder surfaces are current (${String(SURFACES.length)} artifacts, digests pinned)`);
277
}
surfaces/coder/README.md added +98

@@ -0,0 +1,98 @@

1
# The coder's staged text surfaces
2
3
The optimizable text the coder harness carries into every turn, held as data
4
rather than as string literals: the system prompt, the tool descriptions, and
5
the capability catalog lines. Staged so a change to any of it is a diff over an
6
artifact with a content digest, rather than an edit inside three source files
7
that no run can afterwards name.
8
9
The shape is the `knowledge-base` plugin's, deliberately: a corpus file, a build
10
step, and a pinned digest, with the build refusing a stale pin.
11
12
## The files
13
14
| File | What it holds |
15
| --- | --- |
16
| `system-prompt.v1.json` | The instructions, the concision sentence, the no-tools and tool-list sentences, and the lane notices |
17
| `tool-descriptions.v1.json` | Each declared tool's own description, plus the per-model-family emphasis overrides |
18
| `catalog-lines.v1.json` | Each installed plugin's catalog line, keyed by plugin id |
19
| `index.json` | Every surface's schema id, key count, and `sha256` content digest |
20
21
Each artifact is a flat `text` map of key to string, so a one-sentence change is
22
a one-line diff. A `{placeholder}` in a value is substituted by the consumer —
23
`{cwd}`, `{count}`, `{lane}`, `{catalog}`, and so on.
24
25
A key's segments say which harness reads it. A key containing `node` reaches
26
`packages/openagents-cli` only; one containing `rust`, or beginning
27
`coder_lite`, reaches the Rust crates only; a key with neither is shared. The
28
harnesses have forked on some of this text, and the key names are where that is
29
visible rather than hidden.
30
31
## Rebuilding
32
33
```sh
34
pnpm run build:coder-surfaces
35
```
36
37
That re-pins `index.json` and regenerates the two modules the CLIs compile:
38
39
- `crates/openagents-cli/src/surfaces.rs` — read by `crates/openagents-cli` and,
40
  through it, by `crates/coder-lite`
41
- `packages/openagents-cli/src/coder-surfaces.generated.ts`
42
43
Neither generated module is edited by hand. Both are build output.
44
45
```sh
46
pnpm run check:coder-surfaces
47
```
48
49
refuses a tree where an artifact and its build disagree, and runs inside
50
`check:fast`. A surface edited without the rebuild is the knowledge base's
51
failure mode — the edit ships nothing and says nothing while it does — so it is
52
a named check here rather than a silence.
53
54
`node scripts/coder-surfaces-literal-search.mjs` is the stronger, slower proof:
55
it searches the whole tree for a distinctive fragment of every staged string and
56
reports any copy living outside the artifact and its build output.
57
58
## The catalog lines are a mirror, not a move
59
60
`system-prompt` and `tool-descriptions` were string literals in
61
`crates/coder-lite`, `crates/openagents-cli`, and `packages/openagents-cli`.
62
Staging them moved the text; the artifact is now where it is edited.
63
64
A plugin's catalog line was never a literal. It is the top-level `description`
65
of `plugins/<id>/manifest.json`, discovered at runtime by `discover_catalog()`.
66
The manifest stays where that text is edited, and `catalog-lines.v1.json`
67
mirrors it into one diffable object with a digest so an optimizer has a single
68
file to diff and a bench row has a single digest to record. The check fails when
69
a manifest and the mirror disagree.
70
71
## What reads the digests
72
73
A `--plain` session announces the staged text it composed from, on stderr,
74
beside the thread announcement:
75
76
```
77
[oa:surfaces system-prompt=sha256:…,tool-descriptions=sha256:…,catalog-lines=sha256:…]
78
```
79
80
`packages/coder-effectiveness` reads that line out of a trial's `coder.txt`,
81
folds the digests into the run digest, and records them on the
82
`openagents.bench_result.v3` row. So two runs that differ only in the prompt are
83
different runs, a row names exactly which text produced it, and
84
`pnpm run effectiveness:compare` says "staged text also varies" instead of
85
letting a text change read as noise.
86
87
The pin is read from the trial rather than from this directory at scoring time,
88
because the repository a week later is not the repository the run happened on.
89
90
## Changing the text
91
92
Do not edit a sentence here because it reads better. The text is a measured
93
surface: `docs/coder/runbook.md` owns the cycle, `docs/coder/best-practices.md`
94
owns the ledger, and `docs/coder/candidate-format.md` owns the object a proposed
95
change travels as. A wording change belongs in a hillclimb cycle with its own
96
measured delta.
97
98
Issue: OpenAgentsInc/openagents#122.
surfaces/coder/catalog-lines.v1.json added +20

@@ -0,0 +1,20 @@

1
{
2
  "schema": "openagents.coder_surface.catalog_lines.v1",
3
  "surface": "catalog-lines",
4
  "source": "plugins/<id>/manifest.json#description",
5
  "text": {
6
    "code_search": "Search the code in this repository or workspace for a literal string or a regex pattern — grep for where a function, symbol, error message, config key, or route is handled, defined, used, or mentioned, and answer questions like `where is X handled?`. Matches are grouped per file with line numbers and bounded context lines. Traversal honors gitignore through the same documented subset as repo_tree (no negation; `!` lines are counted), and `.git` is always skipped. Regex support is a documented subset: `.`, character classes with ranges and negation, `\\d`/`\\w`/`\\s`, `*`/`+`/`?` on single atoms, `^`/`$` line anchors, and `|` alternation; groups and counted repetition are refused with a reason. Read-only, deterministic, and bounded: ceilings on files scanned, matches returned per file and in total, and context lines, with truncation reported honestly — files considered, scanned, and unscanned, matches dropped, and binary, oversized, and unreadable files counted, never hidden.",
7
    "dir_stats": "List one directory inside the plugin's declared read-only mount: entry names, kinds (file, dir, symlink), sizes, and modification times, bounded per listing. It cannot read file contents, write, or reach outside the mount. Use it when asked what a mounted directory contains.",
8
    "file_stats": "Compute statistics for one file inside the plugin's declared read-only mount: byte count, whether it is UTF-8 text, and its line count. It cannot write, list directories, or reach outside the mount. Use it when asked to measure a mounted file.",
9
    "foreign_sessions": "Discover recent Claude Code and Codex CLI sessions from their local state directories, mounted read-only: for each session its source, id, working directory, modification time, size, and record count. Metadata only — it never reads whole conversations back, never writes, and cannot resume anything. Use it when asked what foreign coding-agent sessions exist on this machine, optionally filtered to a working directory. To read a conversation's content back, load the read_conversation capability instead.",
10
    "git_facts": "Report the git state of this repository: current branch, recent commits and history, and which files changed, are untracked, or are missing. It reads .git plumbing files directly — HEAD, refs/heads, packed-refs, the reflog, and a version-2 index — never a git binary, the object store, or packfiles. Its honest limits: history is reflog-based (only as far back as .git/logs/HEAD reaches), changed files are size-and-mtime candidates rather than content-hashed verdicts, index versions above 2 report counts but not per-entry facts, and there is no blame. Anything skipped or degraded is named in notes.",
11
    "git_lost_work": "Scan a mounted .git directory for unreachable commits and stash entries. It reads HEAD, loose refs, packed-refs, reflogs, and loose objects directly. It does not shell out to git, write anything, or parse packfiles. Use it when asked what commits or stashes may have been lost from a local git repository.",
12
    "knowledge_base": "Answer questions about OpenAgents itself from the generated knowledge base: what is built, parked, or planned — earning bitcoin or money, payouts, tipping, threads, plugins, Coder tiers — plus summaries of every public doc. Pure lookup over an embedded corpus with no file, network, or environment access. The harness consults it automatically and attaches relevant stances and docs to the conversation; prefer its dated, reviewed positions over guesses when they conflict with intuition.",
13
    "patch_check": "Check whether a unified diff or patch still applies to a file's current content, and where it drifts. Use it before claiming a diff applies cleanly: it validates each hunk against the text you pass in, reports the line each hunk lands on, the signed drift when a hunk placed off its declared position, and why a hunk failed (context_not_found or ambiguous, quoting the first mismatching context line). Pure computation with no file, network, or environment access — both the diff and the file content arrive as input. One file per call; optionally returns a bounded preview of the post-application content.",
14
    "read_conversation": "Read a conversation back from this machine: the transcript of a Claude Code or Codex CLI session, returned as ordered turns of user and assistant text. Use it when asked to read a convo, conversation, chat, or session transcript — the latest one here, or one named by session id, or the newest under a working directory. Read-only and bounded: thinking and tool activity are counted rather than replayed, long conversations return their most recent turns and say how many were dropped, and oversized session files are read from the tail. It never resumes, continues, or writes anything.",
15
    "repo_map": "Map the structure of this repository or codebase: an outline of files and their functions, classes, and symbols, with definition lookup and reference counts. Use it when asked what a repo contains, where a function or class or symbol is defined, or how often a name is used. The extraction is a heuristic line-based outline — def/class, fn/struct/trait/impl, defmodule/defp, func/type patterns over Python, TypeScript, JavaScript, Rust, Elixir, Go, and Ruby — not a real parse. Read-only and bounded: a fixed skip list (.git, node_modules, _build, deps, target, .elixir_ls, dist, build), a 256 KB per-file bound (larger files are listed with null symbols and counted oversized), capped files and symbols per file, and honest truncation flags.",
16
    "repo_tree": "List the files and directory tree of this repository or workspace, honoring gitignore, with fuzzy file name lookup. Tree mode (the default) walks the workspace depth-first and returns entries with path, kind, and size, applying root and nested .gitignore files through a documented subset: comments and blanks skipped, trailing-slash directory patterns, leading-slash anchoring, `*` within a segment, `**` across segments, and no negation (`!` lines are counted as ignored_negations). Pass query to get fuzzy file-name matches instead of the tree: a case-insensitive subsequence match against each file path, ranked by match tightness. `.git` is always skipped. Read-only and bounded: depth, entry, and listing ceilings apply, and truncation is reported honestly.",
17
    "session_search": "Search all Claude Code and Codex conversations and sessions on this machine for a word, error, or phrase. Use it when asked where something was discussed, which session mentioned an error message, a file, a decision, or a topic, or to grep across past conversations and chat history. Case-insensitive, newest sessions first, matching only what people actually said — thinking and tool payloads are not searched. Read-only and bounded: each hit returns its role and a small window of surrounding context, oversized session files are searched from their tail and marked, and the output says when the session budget cut the search short. It never resumes, continues, or writes anything.",
18
    "word_stats": "Compute statistics for a piece of text: byte, character, word, and line counts, the longest word, and the most frequent word. Pure computation with no file, network, or environment access. Use it when asked to measure or summarize the size or composition of text."
19
  }
20
}
surfaces/coder/index.json added +23

@@ -0,0 +1,23 @@

1
{
2
  "schema": "openagents.coder_surface_index.v1",
3
  "surfaces": {
4
    "system-prompt": {
5
      "file": "system-prompt.v1.json",
6
      "schema": "openagents.coder_surface.system_prompt.v1",
7
      "keys": 13,
8
      "digest": "sha256:8f0fbbbb38f4ce609a09fe2ba4e03ca160ff2c0d0f698dd4676c2bf940ded849"
9
    },
10
    "tool-descriptions": {
11
      "file": "tool-descriptions.v1.json",
12
      "schema": "openagents.coder_surface.tool_descriptions.v1",
13
      "keys": 18,
14
      "digest": "sha256:513c7518c72ba8c9959e517ea9f1262d7d3ee6b08452ddb48568bdabf6383c1b"
15
    },
16
    "catalog-lines": {
17
      "file": "catalog-lines.v1.json",
18
      "schema": "openagents.coder_surface.catalog_lines.v1",
19
      "keys": 13,
20
      "digest": "sha256:98a96bcf6acfd5847bc1bddcc575761af71be7ea144f3584b3594b24b2f80911"
21
    }
22
  }
23
}
surfaces/coder/system-prompt.v1.json added +19

@@ -0,0 +1,19 @@

1
{
2
  "schema": "openagents.coder_surface.system_prompt.v1",
3
  "surface": "system-prompt",
4
  "text": {
5
    "coder_lite.instructions": "You are OpenAgents Coder. Do not say you are from Google, Anthropic, OpenAI, or any other company. Do not mention your model, training, or architecture. Respond as a neutral, terse terminal: no greetings, no \"As an AI\", no explanations of your role, and no unnecessary padding. Use short sentences and dense, factual output. Answer questions directly. Output only code and minimal context when asked for code.",
6
    "coder_lite.no_tools": "You have no tools in this session: you cannot read or write files, run commands, or reach anything outside this conversation. Say plainly when something would need a tool you do not have.",
7
    "coder_lite.tool_list_header": "You have {count} tools, and no others:",
8
    "coder_lite.tool_list_closing": "That list is complete: a capability not on it is one you do not have. Read a tool's description before assuming what it covers. Never say you ran something you did not run.",
9
    "coder.opening": "You are `openagents coder`, a coding assistant in a terminal. {lane}",
10
    "coder.concision": "Answer very concisely unless the reader asks for a longer response.",
11
    "coder.no_tools": "You have no tools in this session: you cannot read or write files, run commands, or reach anything outside this conversation. Answer from what the reader tells you, and say plainly when something would need a tool you do not have.",
12
    "coder.tool_list_header.node": "You have {count} tool{plural}, and no others:",
13
    "coder.tool_list_header.rust": "You have {count} tools, and no others:",
14
    "coder.tool_list_closing": "That list is complete: a capability not on it is one you do not have, whatever a model like you usually has. Read a tool's description before assuming what it covers. Where a description says what a child agent can do, that is the child's capability and not yours. Never say you ran something you did not run.",
15
    "coder.lane.local.node": "You answer from a model running locally on this machine through Ollama. Tokens here cost nothing, but generation is slow: prefer a few composite tool calls over many small ones, keep narration brief, and verify in one final pass rather than several.",
16
    "coder.lane.local.rust": "You answer from a model running on this machine through Ollama. Nothing in this conversation leaves the machine and nothing is metered, but the context window is a fraction of a hosted model's, so keep large dumps out of the transcript.",
17
    "coder.lane.thread": "You answer through the OpenAgents inference proxy, on a thread opened for this session. Every round of tool calls re-sends the whole conversation to a metered model, so batch independent commands into one call and keep large dumps out of the transcript."
18
  }
19
}
surfaces/coder/tool-descriptions.v1.json added +24

@@ -0,0 +1,24 @@

1
{
2
  "schema": "openagents.coder_surface.tool_descriptions.v1",
3
  "surface": "tool-descriptions",
4
  "text": {
5
    "rust.shell": "Run a shell command on this machine. The working directory is {cwd}, so paths are relative to it and you do not need to ask where you are. Returns combined stdout and stderr with the exit code. Batch independent commands into one call with && instead of one call each: every call replays the conversation so far.",
6
    "rust.skill": "Read one of this repository skill procedures: a written procedure with conventions, commands, and rules. Call it before doing work a skill covers. Skills available:{skills}",
7
    "rust.openagents": "Run the OpenAgents CLI commands (issue, project, repo, auth, etc.). Pass the arguments as a list without openagents itself.",
8
    "rust.delegate": "Run one prompt on independent child coding agents in parallel and return what each one found or did. Use it when work splits into parts that do not depend on each other: several files to change the same way, several hypotheses to check, several tests to run down. Each child is a full coding agent with its own shell tool, working in a git worktree of its own so children cannot overwrite each other, and it starts with no context from this conversation and cannot ask questions — so the prompt has to be self-contained. Every child runs the same prompt and each is told separately which number it is, so write the prompt for whichever child reads it: say \"read the file at your own number\" rather than naming one child. Children run on {lane} and on this session's budget. Prefer one call with a count over several calls, and prefer `shell` over this for a single command — a child agent is for work worth a whole agent, not one line of output. At most {max_count} children.",
9
    "rust.capability": "Discover and load installed plugin capabilities: sandboxed, digest-pinned WebAssembly programs this machine already holds for common agent work. Before writing a script for a task, search here first — a capability that covers it is bounded, reviewable, and returns structured output. Call with `query` describing what you need to get the best matches; then call again with `name` set to the exact returned name to load it and make its dedicated tool available. Every later call to the loaded capability uses that exact name as the tool name.",
10
    "node.shell": "Run a shell command on this machine. The working directory is {cwd}, so paths are relative to it and you do not need to ask where you are. Returns what the command printed. Use it for anything you would type at a terminal: reading files, listing directories, searching, git, running builds and tests. For the `openagents` CLI use the `openagents` tool instead — it carries the list of commands, so running it through here costs a turn finding out what exists. When an installed capability covers the task — the `capability` tool names what is installed — load and call it instead of scripting the same thing here: it is sandboxed, bounded, and returns structured output. Prefer it over `delegate` for single commands -- a child agent is for work worth a whole agent, not for one line of output. Both output streams come back together with the exit code. There is no terminal, so a command that would prompt gets end-of-file instead of waiting; pass a flag that answers the prompt. A few commands that cannot be undone are refused, such as erasing a root or a home directory, reformatting a disk, or halting the machine. Work economically: batch independent commands into one call with && instead of one call each — every call replays the conversation so far. Disable pagers and prefer quiet flags (for example `git --no-pager`, `PAGER=cat`), and ask for summaries before full dumps: get the shape of a thing (a stat, a listing, a count) before printing all of it, and print all of it only for what you are actually deciding about.",
11
    "node.skill": "Read one of this repository's skills: a written procedure for a kind of work, with the conventions, commands, and rules it needs. Call it before doing work a skill covers, and follow what it says over your own habits. Skills available:\n{catalog}",
12
    "node.openagents.head": "Run the OpenAgents CLI: issues, projects, repositories, the forum, authentication, and any API route through `api`. Pass the arguments after `openagents` as a list, without `openagents` itself.\n\n",
13
    "node.openagents.commands": "Commands:\n{tree}\n\n",
14
    "node.openagents.body": "Run `<command> --help` when you need a flag you do not know; the commands above are the whole set, so you do not need to go looking for them.\n\nRead the plain output. It is what a person reads and it is small: a list of three issues is 442 bytes plain and 20,000 as JSON, because the JSON carries every issue's whole body. Add `--json` only when you need one field out of one record, and prefer a narrower command over a wider one you then have to read past.\n\nReads are free; a write is visible to other people at once, so say what you are about to write before the first one. Read the `openagents-cli` skill for the auth model and what works with no credential.",
15
    "node.goal": "Report a state change on the active persistent task goal for this session. The goal's objective, status, and budget already accompany each turn; call this with action='complete' when the goal is done and verified, or 'block'/'pause'/'resume' to update its status.",
16
    "node.remember": "Store one thing the reader has explicitly asked you to remember, in their account's memory. Call it when they say to remember, note, or keep something — a preference, a constraint, a fact about how they work.\n\nExplicit requests only. Do not call this because a conversation revealed a preference, because something seemed worth keeping, or to summarize a session. A memory exists because somebody asked for it.\n\nThere is no matching read: the account's relevant memories are attached to your context by the server before you see the turn, so what is remembered already reaches you without a tool call. To correct a memory you were shown, call this with the corrected sentence and pass the old memory's id as `supersedes`; memories are never edited in place.",
17
    "node.delegate.head": "Run one prompt on independent child coding agents in parallel, in this repository, and return what each one found or did. Use it whenever work splits into parts that do not depend on each other: several files to change the same way, several hypotheses to check, several tests to run down. Each child is a full coding agent with its own file and shell tools, it starts with no context from this conversation, and it cannot ask questions, so the prompt has to be self-contained. Children run on this session's budget. Every child runs the same prompt, and each is told separately which number it is, so write the prompt for whichever child reads it: say \"read the file at your own number\" rather than naming one child (\"you are child 1\"), which gives every child the same work and wastes the fan-out. Prefer one call with a count over several calls. At most {max_count} children.",
18
    "node.delegate.lane_default": " Children run on {label} unless `model` names another lane.\n\n",
19
    "node.delegate.lane_preamble": "A lane is a harness and a model together. The harness runs the child and gives it its tools; the model is what answers. Choosing a lane chooses both:\n",
20
    "node.capability": "Discover and load installed plugin capabilities: sandboxed, sealed programs this machine already holds for common agent work. Before writing a script for a task, search here first — a capability that covers it is bounded, reviewable, and returns structured output. Call with `query` describing what you need to get the best matches; then call again with `name` set to the exact returned name to load it and make its dedicated tool available. Every later call to the loaded capability uses that exact name as the tool name.",
21
    "node.family.gemini.shell": " IMPORTANT: batch independent commands into ONE call joined with && — each separate call replays the whole conversation to the model, so ten one-line calls cost several times what one composite call costs. Never run one small inspection per call. Read only the region you need; prefer offset/limit ranged reads or summaries over whole-file dumps, which are token-inefficient.",
22
    "node.family.local.shell": " This session's model generates slowly on this machine: prefer a few composite calls over many small ones, and keep verification to one final pass."
23
  }
24
}

This page updates live while a promote is in flight · changelog