Fall back to this binary for the `openagents` tool, and cover the audit end to end

0a18c54e6b3c · AtlantisPleb · · parent 43c92585c635

Fall back to this binary for the `openagents` tool, and cover the audit end to end

The live-execution audit for openagents#89 found six defects by driving the
compiled agent against the deployment. Five are fixed in c48fa5b138. This is
the sixth, and the end-to-end tests for all of them.

**The `openagents` tool reached whatever was on PATH, or nothing.** It was
`Command::new("openagents")`, a bare name with nothing behind it. On the
machine this was audited on that ran `openagents v0.4.0`, the TypeScript CLI,
while the agent calling the tool was `oa 0.1.0`; on a machine carrying only
the Rust CLI there is no `openagents` on PATH at all and every call failed
with `No such file or directory`.

`resolve_openagents_cli` settles it: PATH first, because the CLI installed
under that name covers more subcommands than this binary does, then
`current_exe()` so the tool still works where only the Rust binary exists.
The result names which one answered, since the two differ in what they
support and a model reading `unknown command` should not have to guess who
said it. If neither resolves it returns an error rather than a success
carrying nothing. Verified live both ways: with the TypeScript CLI on PATH
the tool reports `openagents v0.4.0`; with it removed, the tool reports
falling back and the model reads `oa 0.1.0`.

`tests/autonomous_test.rs` holds the audit's end-to-end half. Each test now
asserts the fixed behaviour and keeps the account of the defect, because the
reason a test exists outlives its assertion. They sit one layer out from the
unit tests beside the fixes: through `execute_tool` with real subprocesses,
and through `CoderRuntimeSession` against a real socket. Where a defect is
covered at both layers the duplicate is dropped — the hosted-lane transcript
test stays in `runtime_test.rs`, and what remains here is the local lane,
which that fix also changed and nothing else covered.

Each of the six was verified to bite by reverting its fix and watching that
test alone fail, then restoring.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SoZMfWRSGnf6FZX2Ar9rQ2
Co-Authored-By
Claude Opus 5 (1M context) <noreply@anthropic.com>

Deploy story

What this commit did to the running system — joined from the forge receipt chain, the part a commit page elsewhere cannot show.

Not deployed through the forge lane

No push, promotion, build, or deploy receipt references this commit (receipts are scanned over a bounded recent window). Changes shipped by full node replacement carry their proof in the release gate receipt instead.

Changed files

  • modified crates/openagents-cli/src/tools.rs
  • added crates/openagents-cli/tests/autonomous_test.rs

Diff

2 files changed, +805 -39

crates/openagents-cli/src/tools.rs modified +240 -39

@@ -166,7 +166,8 @@ impl HarnessToolRegistry {

166 166
    }
167 167
168 168
    fn build(cwd: Option<PathBuf>, delegation: Option<DelegationGate>) -> Self {
169
        let root = cwd.unwrap_or_else(|| std::env::current_dir().unwrap_or_else(|_| PathBuf::from(".")));
169
        let root =
170
            cwd.unwrap_or_else(|| std::env::current_dir().unwrap_or_else(|_| PathBuf::from(".")));
170 171
        let catalog = plugins::discover_catalog(&root);
171 172
        let mut registry = Self {
172 173
            cwd: root,

@@ -189,7 +190,12 @@ impl HarnessToolRegistry {

189 190
            dirs.push(PathBuf::from(home).join(".agents").join("skills"));
190 191
        }
191 192
        // The skills this CLI ships, read from the package they live in.
192
        dirs.push(self.cwd.join("packages").join("openagents-cli").join("skills"));
193
        dirs.push(
194
            self.cwd
195
                .join("packages")
196
                .join("openagents-cli")
197
                .join("skills"),
198
        );
193 199
        dirs
194 200
    }
195 201

@@ -265,7 +271,10 @@ impl HarnessToolRegistry {

265 271
266 272
    /// The plugins loaded into this session so far.
267 273
    pub fn loaded_plugins(&self) -> Vec<Arc<LoadedPlugin>> {
268
        self.loaded.lock().map(|held| held.clone()).unwrap_or_default()
274
        self.loaded
275
            .lock()
276
            .map(|held| held.clone())
277
            .unwrap_or_default()
269 278
    }
270 279
271 280
    pub fn list_tools(&self) -> Vec<ToolDefinition> {

@@ -379,8 +388,14 @@ impl HarnessToolRegistry {

379 388
    pub async fn execute_tool(&self, call: &ToolCall) -> ToolOutput {
380 389
        match call.name.as_str() {
381 390
            "shell" => {
382
                let cmd = call.arguments.get("command").and_then(|v| v.as_str()).unwrap_or("");
383
                let timeout_secs = call.arguments.get("timeout_seconds")
391
                let cmd = call
392
                    .arguments
393
                    .get("command")
394
                    .and_then(|v| v.as_str())
395
                    .unwrap_or("");
396
                let timeout_secs = call
397
                    .arguments
398
                    .get("timeout_seconds")
384 399
                    .and_then(|v| v.as_u64())
385 400
                    .unwrap_or(DEFAULT_TIMEOUT_SECS)
386 401
                    .min(MAXIMUM_TIMEOUT_SECS);

@@ -401,7 +416,11 @@ impl HarnessToolRegistry {

401 416
                }
402 417
            }
403 418
            "skill" => {
404
                let name = call.arguments.get("name").and_then(|v| v.as_str()).unwrap_or("");
419
                let name = call
420
                    .arguments
421
                    .get("name")
422
                    .and_then(|v| v.as_str())
423
                    .unwrap_or("");
405 424
                if let Some(skill_info) = self.skills.get(name) {
406 425
                    ToolOutput {
407 426
                        call_id: call.id.clone(),

@@ -417,9 +436,16 @@ impl HarnessToolRegistry {

417 436
                }
418 437
            }
419 438
            "openagents" => {
420
                let args_array = call.arguments.get("args")
439
                let args_array = call
440
                    .arguments
441
                    .get("args")
421 442
                    .and_then(|v| v.as_array())
422
                    .map(|arr| arr.iter().filter_map(|v| v.as_str()).map(String::from).collect::<Vec<_>>())
443
                    .map(|arr| {
444
                        arr.iter()
445
                            .filter_map(|v| v.as_str())
446
                            .map(String::from)
447
                            .collect::<Vec<_>>()
448
                    })
423 449
                    .unwrap_or_default();
424 450
425 451
                let (output_str, failed) = run_openagents_cli(&args_array).await;

@@ -487,7 +513,11 @@ impl HarnessToolRegistry {

487 513
                // turn dies of; the only `is_error` here is the absence of a
488 514
                // plugin where one was named.
489 515
                let is_error = loaded.is_none()
490
                    && call.arguments.get("name").and_then(|v| v.as_str()).is_some();
516
                    && call
517
                        .arguments
518
                        .get("name")
519
                        .and_then(|v| v.as_str())
520
                        .is_some();
491 521
                if let Some(plugin) = loaded {
492 522
                    if let Ok(mut held) = self.loaded.lock() {
493 523
                        held.retain(|existing| existing.manifest.name != plugin.manifest.name);

@@ -639,7 +669,12 @@ pub fn render_skill(skill: &SkillInfo) -> String {

639 669
    } else {
640 670
        skill.body.clone()
641 671
    };
642
    format!("Skill `{}` ({}):\n\n{}", skill.name, skill.path.display(), body)
672
    format!(
673
        "Skill `{}` ({}):\n\n{}",
674
        skill.name,
675
        skill.path.display(),
676
        body
677
    )
643 678
}
644 679
645 680
/// What the two OpenAgents repositories are, when the session is in one.

@@ -678,7 +713,10 @@ fn openagents_workspace_note(cwd: &Path) -> Option<String> {

678 713
    // whole workspace root, which holds every read-only reference clone, and
679 714
    // it spent the tool's whole budget before being stopped.
680 715
    if let Some(sibling) = sibling_checkout(cwd) {
681
        let parent = sibling.parent().map(|p| p.display().to_string()).unwrap_or_default();
716
        let parent = sibling
717
            .parent()
718
            .map(|p| p.display().to_string())
719
            .unwrap_or_default();
682 720
        lines.push(String::new());
683 721
        lines.push(format!(
684 722
            "The other one is checked out at `{}`. To search or read it, change directory first — \

@@ -810,8 +848,7 @@ async fn run_real_shell(cmd: &str, cwd: &Path, timeout_secs: u64) -> (String, bo

810 848
        .kill_on_drop(true);
811 849
    #[cfg(unix)]
812 850
    command.process_group(0);
813
    let child = match command.spawn()
814
    {
851
    let child = match command.spawn() {
815 852
        Ok(c) => c,
816 853
        Err(e) => return (format!("Failed to spawn shell command: {}", e), true),
817 854
    };

@@ -837,21 +874,31 @@ async fn run_real_shell(cmd: &str, cwd: &Path, timeout_secs: u64) -> (String, bo

837 874
838 875
            if output.status.success() {
839 876
                if bounded.trim().is_empty() {
840
                    ("The command succeeded and printed nothing.".to_string(), false)
877
                    (
878
                        "The command succeeded and printed nothing.".to_string(),
879
                        false,
880
                    )
841 881
                } else {
842 882
                    (bounded.trim().to_string(), false)
843 883
                }
844 884
            } else {
845 885
                let code = output.status.code().unwrap_or(1);
846 886
                (
847
                    format!("The command exited with code {}.\n\n{}", code, bounded.trim()),
887
                    format!(
888
                        "The command exited with code {}.\n\n{}",
889
                        code,
890
                        bounded.trim()
891
                    ),
848 892
                    true,
849 893
                )
850 894
            }
851 895
        }
852 896
        Ok(Err(e)) => (format!("Shell execution error: {}", e), true),
853 897
        Err(_) => (
854
            format!("The command timed out after {} seconds and was stopped.", timeout_secs),
898
            format!(
899
                "The command timed out after {} seconds and was stopped.",
900
                timeout_secs
901
            ),
855 902
            true,
856 903
        ),
857 904
    }

@@ -859,8 +906,88 @@ async fn run_real_shell(cmd: &str, cwd: &Path, timeout_secs: u64) -> (String, bo

859 906
860 907
/// As [`run_real_shell`]: the text, and whether it worked. A CLI that could
861 908
/// not even be spawned was previously reported to the model as a success.
909
/// Where the program behind the `openagents` tool came from.
910
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
911
pub enum OpenAgentsCliSource {
912
    /// A program named `openagents` found on `PATH`.
913
    Path,
914
    /// Nothing on `PATH`, so this binary answers for it.
915
    ThisBinary,
916
}
917
918
/// The program the `openagents` tool runs, and where it was found.
919
///
920
/// `PATH` first, deliberately: the CLI installed under that name is the
921
/// TypeScript one, and it covers more subcommands than this binary does, so
922
/// where both exist the model should reach the fuller of the two.
923
///
924
/// This binary is the fallback rather than the first choice because it is
925
/// installed as `oa`. On a machine carrying only the Rust CLI there is nothing
926
/// named `openagents` anywhere on `PATH`, and this used to be
927
/// `Command::new("openagents")` — so every call the model made failed with
928
/// `No such file or directory` while the very binary that could have answered
929
/// was the one running the tool.
930
pub fn resolve_openagents_cli() -> Result<(PathBuf, OpenAgentsCliSource), String> {
931
    let name = format!("openagents{}", std::env::consts::EXE_SUFFIX);
932
    let on_path = std::env::var_os("PATH")
933
        .map(|path| std::env::split_paths(&path).collect::<Vec<_>>())
934
        .unwrap_or_default()
935
        .into_iter()
936
        .map(|dir| dir.join(&name))
937
        .find(|candidate| is_executable_file(candidate));
938
    if let Some(found) = on_path {
939
        return Ok((found, OpenAgentsCliSource::Path));
940
    }
941
    match std::env::current_exe() {
942
        Ok(exe) => Ok((exe, OpenAgentsCliSource::ThisBinary)),
943
        // Neither resolves. This is an error rather than an empty success: a
944
        // tool result that says nothing reads to a model as a command that ran
945
        // and printed nothing.
946
        Err(error) => Err(format!(
947
            "No `openagents` CLI is available: nothing named `openagents` is on PATH, \
948
             and this binary's own path could not be read: {error}"
949
        )),
950
    }
951
}
952
953
fn is_executable_file(path: &Path) -> bool {
954
    let Ok(meta) = std::fs::metadata(path) else {
955
        return false;
956
    };
957
    if !meta.is_file() {
958
        return false;
959
    }
960
    #[cfg(unix)]
961
    {
962
        use std::os::unix::fs::PermissionsExt;
963
        meta.permissions().mode() & 0o111 != 0
964
    }
965
    #[cfg(not(unix))]
966
    {
967
        true
968
    }
969
}
970
862 971
async fn run_openagents_cli(args: &[String]) -> (String, bool) {
863
    let mut cmd = Command::new("openagents");
972
    let (program, source) = match resolve_openagents_cli() {
973
        Ok(resolved) => resolved,
974
        Err(error) => return (error, true),
975
    };
976
977
    // Which program answered is part of the result. The two differ in what
978
    // they support, so a model reading `unknown command` needs to know which
979
    // CLI said it rather than guessing.
980
    let note = match source {
981
        OpenAgentsCliSource::Path => {
982
            format!("[ran the `openagents` CLI on PATH: {}]", program.display())
983
        }
984
        OpenAgentsCliSource::ThisBinary => format!(
985
            "[no `openagents` on PATH; ran this binary instead: {}]",
986
            program.display()
987
        ),
988
    };
989
990
    let mut cmd = Command::new(&program);
864 991
    cmd.args(args);
865 992
    cmd.stdout(Stdio::piped());
866 993
    cmd.stderr(Stdio::piped());

@@ -870,9 +997,15 @@ async fn run_openagents_cli(args: &[String]) -> (String, bool) {

870 997
            let mut combined = String::new();
871 998
            combined.push_str(&String::from_utf8_lossy(&output.stdout));
872 999
            combined.push_str(&String::from_utf8_lossy(&output.stderr));
873
            (combined.trim().to_string(), !output.status.success())
1000
            (
1001
                format!("{note}\n\n{}", combined.trim()),
1002
                !output.status.success(),
1003
            )
874 1004
        }
875
        Err(e) => (format!("Failed to run openagents CLI: {}", e), true),
1005
        Err(e) => (
1006
            format!("{note}\n\nFailed to run the openagents CLI: {e}"),
1007
            true,
1008
        ),
876 1009
    }
877 1010
}
878 1011

@@ -933,7 +1066,10 @@ mod tests {

933 1066
    #[test]
934 1067
    fn a_refusal_says_what_the_command_would_have_done() {
935 1068
        let refusal = check_shell_refusal("rm -rf ~/").expect("refused");
936
        assert!(refusal.contains("erase a root or a home directory"), "{refusal}");
1069
        assert!(
1070
            refusal.contains("erase a root or a home directory"),
1071
            "{refusal}"
1072
        );
937 1073
        assert!(refusal.contains("name the directory"), "{refusal}");
938 1074
    }
939 1075

@@ -956,7 +1092,8 @@ mod tests {

956 1092
957 1093
    #[test]
958 1094
    fn a_folded_block_description_is_one_paragraph() {
959
        let source = "---\nname: folded\ndescription: >-\n  First line\n  second line.\n---\nBody.\n";
1095
        let source =
1096
            "---\nname: folded\ndescription: >-\n  First line\n  second line.\n---\nBody.\n";
960 1097
        let (_, description, _) = parse_skill_front_matter(source).expect("parsed");
961 1098
        assert_eq!(description, "First line second line.");
962 1099
    }

@@ -972,8 +1109,17 @@ mod tests {

972 1109
    #[test]
973 1110
    fn the_nearest_skills_directory_keeps_a_contested_name() {
974 1111
        let root = tempfile::tempdir().unwrap();
975
        write_skill(root.path(), "shared", "---\nname: shared\ndescription: The repository's.\n---\nRepo body.\n");
976
        let shipped = root.path().join("packages").join("openagents-cli").join("skills").join("shared");
1112
        write_skill(
1113
            root.path(),
1114
            "shared",
1115
            "---\nname: shared\ndescription: The repository's.\n---\nRepo body.\n",
1116
        );
1117
        let shipped = root
1118
            .path()
1119
            .join("packages")
1120
            .join("openagents-cli")
1121
            .join("skills")
1122
            .join("shared");
977 1123
        std::fs::create_dir_all(&shipped).unwrap();
978 1124
        std::fs::write(
979 1125
            shipped.join("SKILL.md"),

@@ -999,7 +1145,9 @@ mod tests {

999 1145
        let tools = registry.list_tools();
1000 1146
        let skill_tool = tools.iter().find(|t| t.name == "skill").expect("declared");
1001 1147
1002
        assert!(skill_tool.description.contains("`brewing`: How to make tea."));
1148
        assert!(skill_tool
1149
            .description
1150
            .contains("`brewing`: How to make tea."));
1003 1151
        // The catalog is what a session pays for on every turn; a body in it
1004 1152
        // is 46 KB of instructions the model may never use.
1005 1153
        assert!(

@@ -1026,7 +1174,11 @@ mod tests {

1026 1174
            })
1027 1175
            .await;
1028 1176
        assert!(!out.is_error);
1029
        assert!(out.output.contains("STEEP_FOR_FOUR_MINUTES"), "{}", out.output);
1177
        assert!(
1178
            out.output.contains("STEEP_FOR_FOUR_MINUTES"),
1179
            "{}",
1180
            out.output
1181
        );
1030 1182
        assert!(out.output.contains("SKILL.md"), "{}", out.output);
1031 1183
1032 1184
        let missing = registry

@@ -1054,7 +1206,9 @@ mod tests {

1054 1206
        );
1055 1207
        let registry = HarnessToolRegistry::new(Some(root.path().to_path_buf()));
1056 1208
1057
        let context = registry.standing_context().expect("an auto skill is injected");
1209
        let context = registry
1210
            .standing_context()
1211
            .expect("an auto skill is injected");
1058 1212
        assert!(context.contains("WORK_THIS_WAY"), "{context}");
1059 1213
        assert!(
1060 1214
            !context.contains("ONLY_ON_REQUEST"),

@@ -1065,7 +1219,11 @@ mod tests {

1065 1219
    #[test]
1066 1220
    fn a_workspace_with_no_auto_skill_injects_nothing() {
1067 1221
        let root = tempfile::tempdir().unwrap();
1068
        write_skill(root.path(), "plain", "---\nname: plain\ndescription: Read on request.\n---\nBody.\n");
1222
        write_skill(
1223
            root.path(),
1224
            "plain",
1225
            "---\nname: plain\ndescription: Read on request.\n---\nBody.\n",
1226
        );
1069 1227
        // A temporary directory is not named for either OpenAgents repository,
1070 1228
        // so the workspace note does not apply either.
1071 1229
        assert!(HarnessToolRegistry::new(Some(root.path().to_path_buf()))

@@ -1097,11 +1255,17 @@ mod tests {

1097 1255
        // context this session starts with.
1098 1256
        let context = registry.standing_context().expect("something is injected");
1099 1257
        assert!(
1100
            registry.skills.get("superdelegate").is_some_and(|skill| skill.auto),
1258
            registry
1259
                .skills
1260
                .get("superdelegate")
1261
                .is_some_and(|skill| skill.auto),
1101 1262
            "superdelegate is the repository's auto skill"
1102 1263
        );
1103 1264
        let body = &registry.skills["superdelegate"].body;
1104
        assert!(context.contains(&body[..80.min(body.len())]), "the auto body was not injected");
1265
        assert!(
1266
            context.contains(&body[..80.min(body.len())]),
1267
            "the auto body was not injected"
1268
        );
1105 1269
    }
1106 1270
1107 1271
    // ───────────────────────────────────────────── the capability tool wiring

@@ -1113,10 +1277,17 @@ mod tests {

1113 1277
        let root = tempfile::tempdir().unwrap();
1114 1278
        let registry = HarnessToolRegistry::with_delegation(
1115 1279
            Some(root.path().to_path_buf()),
1116
            DelegationGate { lane: "test".to_string(), user_token: None, max_count: 2 },
1280
            DelegationGate {
1281
                lane: "test".to_string(),
1282
                user_token: None,
1283
                max_count: 2,
1284
            },
1117 1285
        );
1118 1286
        let names: Vec<String> = registry.list_tools().into_iter().map(|t| t.name).collect();
1119
        assert_eq!(names, vec!["shell", "skill", "openagents", "capability", "delegate"]);
1287
        assert_eq!(
1288
            names,
1289
            vec!["shell", "skill", "openagents", "capability", "delegate"]
1290
        );
1120 1291
1121 1292
        let runtime = tokio::runtime::Runtime::new().unwrap();
1122 1293
        for name in &names {

@@ -1138,11 +1309,19 @@ mod tests {

1138 1309
    #[tokio::test]
1139 1310
    async fn a_capability_search_names_a_plugin_and_loading_it_declares_its_tool() {
1140 1311
        let repo = Path::new(env!("CARGO_MANIFEST_DIR")).join("..").join("..");
1141
        if !repo.join("plugins").join("word-stats").join("manifest.json").is_file() {
1312
        if !repo
1313
            .join("plugins")
1314
            .join("word-stats")
1315
            .join("manifest.json")
1316
            .is_file()
1317
        {
1142 1318
            return;
1143 1319
        }
1144 1320
        let registry = HarnessToolRegistry::new(Some(repo));
1145
        assert!(!registry.catalog.is_empty(), "the checked-in catalog was not discovered");
1321
        assert!(
1322
            !registry.catalog.is_empty(),
1323
            "the checked-in catalog was not discovered"
1324
        );
1146 1325
1147 1326
        let search = registry
1148 1327
            .execute_tool(&ToolCall {

@@ -1168,7 +1347,10 @@ mod tests {

1168 1347
            .into_iter()
1169 1348
            .find(|t| t.name == "word_stats")
1170 1349
            .expect("the loaded plugin declares its tool");
1171
        assert_eq!(word_stats.parameters["properties"]["text"]["type"], "string");
1350
        assert_eq!(
1351
            word_stats.parameters["properties"]["text"]["type"],
1352
            "string"
1353
        );
1172 1354
1173 1355
        // And the tool the plugin declared runs the plugin.
1174 1356
        let ran = registry

@@ -1186,7 +1368,12 @@ mod tests {

1186 1368
    #[tokio::test]
1187 1369
    async fn a_mounted_capability_refuses_to_load_without_an_operator() {
1188 1370
        let repo = Path::new(env!("CARGO_MANIFEST_DIR")).join("..").join("..");
1189
        if !repo.join("plugins").join("file-stats").join("manifest.json").is_file() {
1371
        if !repo
1372
            .join("plugins")
1373
            .join("file-stats")
1374
            .join("manifest.json")
1375
            .is_file()
1376
        {
1190 1377
            return;
1191 1378
        }
1192 1379
        let unattended = HarnessToolRegistry::new(Some(repo.clone()));

@@ -1197,8 +1384,15 @@ mod tests {

1197 1384
                arguments: serde_json::json!({"name": "file_stats"}),
1198 1385
            })
1199 1386
            .await;
1200
        assert!(refused.output.contains("approval_unavailable"), "{}", refused.output);
1201
        assert!(unattended.list_tools().iter().all(|t| t.name != "file_stats"));
1387
        assert!(
1388
            refused.output.contains("approval_unavailable"),
1389
            "{}",
1390
            refused.output
1391
        );
1392
        assert!(unattended
1393
            .list_tools()
1394
            .iter()
1395
            .all(|t| t.name != "file_stats"));
1202 1396
1203 1397
        let attended = HarnessToolRegistry::new(Some(repo)).allowing_plugin_mounts();
1204 1398
        let loaded = attended

@@ -1208,7 +1402,11 @@ mod tests {

1208 1402
                arguments: serde_json::json!({"name": "file_stats"}),
1209 1403
            })
1210 1404
            .await;
1211
        assert!(loaded.output.contains("digest verified"), "{}", loaded.output);
1405
        assert!(
1406
            loaded.output.contains("digest verified"),
1407
            "{}",
1408
            loaded.output
1409
        );
1212 1410
        assert!(attended.list_tools().iter().any(|t| t.name == "file_stats"));
1213 1411
    }
1214 1412

@@ -1242,7 +1440,10 @@ mod defect_tests {

1242 1440
        // 29,999 ASCII bytes, then a 3-byte character straddling byte 30,000.
1243 1441
        let mut text = "a".repeat(OUTPUT_LIMIT - 1);
1244 1442
        text.push_str("€€€");
1245
        assert!(!text.is_char_boundary(OUTPUT_LIMIT), "the probe must straddle");
1443
        assert!(
1444
            !text.is_char_boundary(OUTPUT_LIMIT),
1445
            "the probe must straddle"
1446
        );
1246 1447
1247 1448
        let cut = floor_char_boundary(&text, OUTPUT_LIMIT);
1248 1449
        // The old code did `&text[..OUTPUT_LIMIT]`, which panics here.
crates/openagents-cli/tests/autonomous_test.rs added +565

@@ -0,0 +1,565 @@

1
//! The end-to-end half of the live-execution audit for
2
//! OpenAgentsInc/openagents#89.
3
//!
4
//! Six defects were found by driving the compiled binary against the live
5
//! deployment rather than by reading the source. Five are fixed in
6
//! `c48fa5b138` and the sixth in the commit this file arrives on; each test
7
//! below now asserts the **fixed** behaviour, and each keeps the account of
8
//! what the defect was, because the reason a test exists outlives the
9
//! assertion.
10
//!
11
//! These run one layer out from the unit tests that accompany the fixes:
12
//! `tools::defect_tests` calls `run_real_shell` and `floor_char_boundary`
13
//! directly, while these go through `HarnessToolRegistry::execute_tool` with
14
//! real subprocesses, and through `CoderRuntimeSession` against a real socket.
15
//! Where a defect is covered at both layers the outer one is kept here and the
16
//! duplicate dropped — the hosted-lane transcript test lives in
17
//! `runtime_test.rs` as `the_second_turn_carries_what_the_first_turn_answered`,
18
//! so what remains here is the local lane, which that fix also changed and
19
//! nothing else covers.
20
//!
21
//! What the live run confirmed working, and what is therefore not re-asserted
22
//! here: a headless turn reaches a real model and prints its answer, a turn
23
//! calls `shell` and answers from the result, a two-child fan-out returns two
24
//! real outputs from separate git worktrees, every lane opens on the model it
25
//! names, and reported token counts match what `GET /api/v1/threads` records.
26
27
use openagents_cli::runtime::{CoderRuntimeSession, Lane};
28
use openagents_cli::tools::{
29
    resolve_openagents_cli, HarnessToolRegistry, OpenAgentsCliSource, ToolCall,
30
};
31
use std::sync::{Arc, Mutex};
32
use tokio::io::{AsyncReadExt, AsyncWriteExt};
33
34
// ───────────────────────────────────────────────────────────────── the stub
35
36
/// What a request should be answered with.
37
enum Reply {
38
    Json(String),
39
    /// Server-sent events, the shape the inference proxy streams.
40
    Sse(Vec<String>),
41
    /// Newline-delimited JSON, the shape Ollama streams.
42
    Ndjson(Vec<String>),
43
}
44
45
/// A stand-in for whichever server the session is talking to, recording what
46
/// it was asked.
47
///
48
/// It answers on a real socket, so everything between the session and the wire
49
/// is the production path and what these tests assert on is the bytes that
50
/// were actually sent.
51
struct Stub {
52
    base: String,
53
    origin: String,
54
    requests: Arc<Mutex<Vec<String>>>,
55
}
56
57
impl Stub {
58
    /// Every request this stub has taken, headers and body, oldest first.
59
    fn requests(&self) -> Vec<String> {
60
        self.requests.lock().unwrap().clone()
61
    }
62
63
    /// Just the bodies of the calls that asked a model to say something.
64
    fn completions(&self) -> Vec<String> {
65
        self.requests()
66
            .into_iter()
67
            .filter(|r| r.starts_with("POST /proxy") || r.starts_with("POST /api/chat"))
68
            .collect()
69
    }
70
}
71
72
/// Start a stub whose reply is chosen by the request and by how many
73
/// completions it has already served.
74
fn start<H>(handler: H) -> Stub
75
where
76
    H: Fn(&str, usize, &str) -> Reply + Send + Sync + 'static,
77
{
78
    let requests = Arc::new(Mutex::new(Vec::new()));
79
    let recorder = Arc::clone(&requests);
80
    let listener = std::net::TcpListener::bind("127.0.0.1:0").unwrap();
81
    listener.set_nonblocking(true).unwrap();
82
    let port = listener.local_addr().unwrap().port();
83
    let origin = format!("http://127.0.0.1:{port}");
84
    let base = format!("{origin}/api/v1");
85
    let handler_origin = origin.clone();
86
87
    tokio::spawn(async move {
88
        let listener = tokio::net::TcpListener::from_std(listener).unwrap();
89
        let mut served = 0usize;
90
        loop {
91
            let Ok((mut socket, _)) = listener.accept().await else {
92
                return;
93
            };
94
            let Some(request) = read_request(&mut socket).await else {
95
                continue;
96
            };
97
            recorder.lock().unwrap().push(request.clone());
98
99
            let reply = handler(&request, served, &handler_origin);
100
            match reply {
101
                Reply::Json(body) => {
102
                    let response = format!(
103
                        "HTTP/1.1 200 OK\r\ncontent-type: application/json\r\ncontent-length: {}\r\nconnection: close\r\n\r\n{body}",
104
                        body.len()
105
                    );
106
                    let _ = socket.write_all(response.as_bytes()).await;
107
                }
108
                Reply::Sse(frames) => {
109
                    served += 1;
110
                    let _ = socket
111
                        .write_all(b"HTTP/1.1 200 OK\r\ncontent-type: text/event-stream\r\nconnection: close\r\n\r\n")
112
                        .await;
113
                    for frame in frames {
114
                        let _ = socket
115
                            .write_all(format!("data: {frame}\n\n").as_bytes())
116
                            .await;
117
                    }
118
                    let _ = socket.write_all(b"data: [DONE]\n\n").await;
119
                }
120
                Reply::Ndjson(lines) => {
121
                    served += 1;
122
                    let _ = socket
123
                        .write_all(b"HTTP/1.1 200 OK\r\ncontent-type: application/x-ndjson\r\nconnection: close\r\n\r\n")
124
                        .await;
125
                    for line in lines {
126
                        let _ = socket.write_all(format!("{line}\n").as_bytes()).await;
127
                    }
128
                }
129
            }
130
            let _ = socket.flush().await;
131
        }
132
    });
133
134
    Stub {
135
        base,
136
        origin,
137
        requests,
138
    }
139
}
140
141
/// Read one request, headers and declared body, and return it as text.
142
async fn read_request(socket: &mut tokio::net::TcpStream) -> Option<String> {
143
    let mut request = Vec::new();
144
    let mut buffer = [0u8; 4096];
145
    loop {
146
        let read = socket.read(&mut buffer).await.ok()?;
147
        if read == 0 {
148
            break;
149
        }
150
        request.extend_from_slice(&buffer[..read]);
151
        let text = String::from_utf8_lossy(&request);
152
        if let Some(headers_end) = text.find("\r\n\r\n") {
153
            let length = text
154
                .lines()
155
                .find_map(|line| {
156
                    line.strip_prefix("content-length: ")
157
                        .or_else(|| line.strip_prefix("Content-Length: "))
158
                })
159
                .and_then(|value| value.trim().parse::<usize>().ok())
160
                .unwrap_or(0);
161
            if request.len() >= headers_end + 4 + length {
162
                break;
163
            }
164
        }
165
    }
166
    Some(String::from_utf8_lossy(&request).to_string())
167
}
168
169
fn grant(origin: &str) -> String {
170
    format!(
171
        r#"{{"thread":{{"id":"th_test"}},"grant":{{"token":"tok_test","url":"{origin}/proxy","model":"ox-alpha"}}}}"#
172
    )
173
}
174
175
/// One frame asking for a tool, whole rather than fragmented.
176
fn call_tool(id: &str, name: &str) -> String {
177
    serde_json::json!({
178
        "choices": [{ "delta": { "tool_calls": [{
179
            "index": 0, "id": id,
180
            "function": { "name": name, "arguments": "{}" }
181
        }]}}]
182
    })
183
    .to_string()
184
}
185
186
fn registry() -> (tempfile::TempDir, HarnessToolRegistry) {
187
    let dir = tempfile::tempdir().unwrap();
188
    let registry = HarnessToolRegistry::new(Some(dir.path().to_path_buf()));
189
    (dir, registry)
190
}
191
192
// ──────────────────────────────────────────────────────────────── defect 1
193
194
/// **Output cut through a multi-byte character no longer kills the agent.**
195
///
196
/// `run_real_shell` bounded output with `&combined[..OUTPUT_LIMIT]` — a *byte*
197
/// index into a `String`, which panics when the index is not a character
198
/// boundary. Any command whose combined output crossed 30,000 bytes
199
/// mid-character took the whole process down in the middle of a turn: a `git
200
/// log`, a test run with UTF-8 output, any file with an accent or an emoji
201
/// past 30 kB. It aborted before the thread could be revoked, so the grant's
202
/// remaining budget was stranded too.
203
///
204
/// Reproduced live against the built binary before the fix:
205
///
206
/// ```text
207
/// $ oa coder --headless "Use the shell tool to run exactly: sh repro.sh"
208
/// thread 'main' panicked at crates/openagents-cli/src/tools.rs:424:98:
209
/// byte index 30000 is not a char boundary; it is inside '€' (bytes 29999..30002)
210
/// EXIT=101
211
/// ```
212
///
213
/// `tools::defect_tests` covers `floor_char_boundary` on its own. This runs
214
/// the real `/bin/sh` through the tool dispatch and asserts the whole path
215
/// survives: the panic showed up through a command, so a command is what
216
/// proves it gone.
217
#[tokio::test]
218
async fn shell_output_cut_through_a_multibyte_character_is_truncated_not_fatal() {
219
    let (_dir, registry) = registry();
220
221
    // 29,999 single-byte characters, then three-byte ones. Byte 30,000 lands
222
    // inside the first `€`, which occupies bytes 29,999-30,001.
223
    let command = "head -c 29999 /dev/zero | tr '\\0' 'a' && printf '€€€€€€€€€€'";
224
    let call = ToolCall {
225
        id: "call_trunc".to_string(),
226
        name: "shell".to_string(),
227
        arguments: serde_json::json!({ "command": command }),
228
    };
229
230
    // On its own task, so a panic is reported rather than unwinding the test.
231
    let output = tokio::spawn(async move { registry.execute_tool(&call).await })
232
        .await
233
        .expect("the shell tool panicked on a multi-byte truncation boundary");
234
235
    assert!(
236
        output
237
            .output
238
            .contains("[Output truncated: printed 30029 characters, limit is 30000]"),
239
        "the output should say it was cut, and by how much: {}",
240
        &output.output[output.output.len().saturating_sub(200)..]
241
    );
242
    assert!(
243
        output.output.starts_with("aaa"),
244
        "the kept head should be the start of the command's output"
245
    );
246
    assert!(
247
        !output.output.contains('\u{FFFD}'),
248
        "the cut left a broken character behind"
249
    );
250
    assert!(!output.is_error, "the command itself succeeded");
251
}
252
253
// ──────────────────────────────────────────────────────────────── defect 2
254
255
/// **The local lane records what it answered, so the next turn can see it.**
256
///
257
/// `run_tools` records an assistant turn only when that turn called a tool, so
258
/// a turn that simply answered never joined `self.messages`. The session is
259
/// reused across turns, so the model was never shown a word it had said
260
/// itself. Live, in the interactive session, it did not admit the gap — it
261
/// confabulated:
262
///
263
/// ```text
264
/// turn 1: "Invent a random six-letter nonsense codeword." -> QORVEN
265
/// turn 2: "What was the codeword you just invented?"      -> ZORBEX
266
/// ```
267
///
268
/// A test that asks the model to recall something from the **user's** prompt
269
/// passes against the defect, because user messages were always recorded. The
270
/// word has to be one only the assistant ever said.
271
///
272
/// The hosted lane is covered by `the_second_turn_carries_what_the_first_turn_answered`
273
/// in `runtime_test.rs`. The fix changed `run_local_turn` the same way, and
274
/// this is that half: the local lane keeps its own message list, in Ollama's
275
/// shape, through a separate code path.
276
#[tokio::test]
277
async fn the_local_lane_records_what_it_answered() {
278
    let stub = start(|request, served, _origin| {
279
        if request.starts_with("GET /api/tags") {
280
            return Reply::Json(
281
                r#"{"models":[{"name":"qwen3:0.6b","modified_at":"2026-08-26T00:00:00Z"}]}"#
282
                    .to_string(),
283
            );
284
        }
285
        let word = if served == 0 { "QORVEN" } else { "ZORBEX" };
286
        Reply::Ndjson(vec![
287
            serde_json::json!({
288
                "model": "qwen3:0.6b",
289
                "message": { "role": "assistant", "content": word },
290
                "done": false
291
            })
292
            .to_string(),
293
            serde_json::json!({
294
                "model": "qwen3:0.6b",
295
                "message": { "role": "assistant", "content": "" },
296
                "done": true, "done_reason": "stop",
297
                "prompt_eval_count": 10, "eval_count": 3
298
            })
299
            .to_string(),
300
        ])
301
    });
302
303
    let (_dir, tools) = registry();
304
    let mut session = CoderRuntimeSession::new(
305
        Lane::Local(String::new()),
306
        Some(stub.base.clone()),
307
        None,
308
        tools,
309
    );
310
    session.ollama_host = stub.origin.clone();
311
312
    let first = session
313
        .execute_turn("invent a six-letter codeword", |_| {})
314
        .await
315
        .unwrap();
316
    assert_eq!(first, "QORVEN");
317
318
    session
319
        .execute_turn("what was the codeword?", |_| {})
320
        .await
321
        .unwrap();
322
323
    let chats = stub.completions();
324
    assert_eq!(chats.len(), 2, "expected one chat call per turn");
325
    assert!(
326
        chats[1].contains("QORVEN"),
327
        "the second turn did not carry the first turn's answer, so the local \
328
         model cannot see what it said: {}",
329
        chats[1]
330
    );
331
    assert!(
332
        session
333
            .messages
334
            .iter()
335
            .any(|m| m.role == "assistant" && m.content.as_deref() == Some("QORVEN")),
336
        "the answer is missing from the session's own transcript"
337
    );
338
}
339
340
// ──────────────────────────────────────────────────────────────── defect 3
341
342
/// **A turn that runs out of tool steps refuses instead of reporting success.**
343
///
344
/// `run_thread_turn` loops `for _ in 0..MAX_TOOL_STEPS` and assigned
345
/// `final_answer` only on the step that came back without tool calls. A model
346
/// that asked for a tool on all thirty steps fell out of the bottom and the
347
/// function returned `Ok(String::new())` — the same `Ok` a finished turn
348
/// returns, carrying nothing. `run_headless_coder` printed `Turn result:`
349
/// followed by a blank line and exited 0, and the interactive session settled
350
/// `TurnEvent::Done("")` as an answered turn.
351
///
352
/// That is the shape the issue was reopened over, one level down: the turn did
353
/// not finish and no caller could tell.
354
#[tokio::test]
355
async fn a_turn_that_exhausts_its_tool_steps_refuses() {
356
    // Every step asks for a tool and never stops asking. The name is one no
357
    // registry has, so the loop spends no time running anything.
358
    let stub = start(|request, served, origin| {
359
        if request.starts_with("POST /api/v1/threads") {
360
            return Reply::Json(grant(origin));
361
        }
362
        Reply::Sse(vec![call_tool(
363
            &format!("call_{served}"),
364
            "a_tool_that_does_not_exist",
365
        )])
366
    });
367
368
    let (_dir, tools) = registry();
369
    let mut session = CoderRuntimeSession::new(
370
        Lane::OxAlpha,
371
        Some(stub.base.clone()),
372
        Some("tok_user".to_string()),
373
        tools,
374
    );
375
376
    let error = session
377
        .execute_turn("loop forever", |_| {})
378
        .await
379
        .expect_err("a turn that never answered was reported as a finished turn");
380
    let error = error.to_string();
381
382
    assert!(
383
        error.contains("30") && error.contains("tool steps"),
384
        "the refusal should say the step budget ran out: {error}"
385
    );
386
    assert_eq!(
387
        stub.completions().len(),
388
        30,
389
        "MAX_TOOL_STEPS is 30, so the turn should spend the whole budget before refusing"
390
    );
391
}
392
393
// ──────────────────────────────────────────────────────────────── defect 4
394
395
/// **The `openagents` tool prefers `PATH` and falls back to this binary.**
396
///
397
/// The tool is declared as "Run the OpenAgents CLI commands (issue, project,
398
/// repo, auth, etc.)" and was implemented as `Command::new("openagents")` — a
399
/// bare name resolved through `PATH`, with nothing behind it. On the machine
400
/// this was verified on that reached `openagents v0.4.0`, the TypeScript CLI,
401
/// while the agent running the tool was `oa 0.1.0`; on a machine carrying only
402
/// the Rust CLI there is no `openagents` on `PATH` at all and every call
403
/// failed with `No such file or directory`.
404
///
405
/// The contract now: prefer `PATH`, because the CLI installed under that name
406
/// covers more subcommands than this binary does; fall back to
407
/// `current_exe()`, so the tool still works where only the Rust binary exists;
408
/// name which one ran in the result, so a model reading `unknown command`
409
/// knows which CLI said it; and if neither resolves, return an error rather
410
/// than a success carrying nothing.
411
#[tokio::test]
412
async fn the_openagents_tool_prefers_path_and_falls_back_to_this_binary() {
413
    let dir = tempfile::tempdir().unwrap();
414
    let shim = dir.path().join("openagents");
415
    std::fs::write(&shim, "#!/bin/sh\necho SHIM-ON-PATH\n").unwrap();
416
    #[cfg(unix)]
417
    {
418
        use std::os::unix::fs::PermissionsExt;
419
        std::fs::set_permissions(&shim, std::fs::Permissions::from_mode(0o755)).unwrap();
420
    }
421
422
    // The only test in this file that touches the environment.
423
    let original = std::env::var("PATH").unwrap_or_default();
424
425
    // With something named `openagents` on PATH, that is what runs.
426
    std::env::set_var("PATH", format!("{}:{original}", dir.path().display()));
427
    let registry = HarnessToolRegistry::new(Some(dir.path().to_path_buf()));
428
    let output = registry
429
        .execute_tool(&ToolCall {
430
            id: "call_oa".to_string(),
431
            name: "openagents".to_string(),
432
            arguments: serde_json::json!({ "args": ["--version"] }),
433
        })
434
        .await;
435
    let on_path = resolve_openagents_cli();
436
437
    // With nothing named `openagents` anywhere, this binary answers for it.
438
    std::env::set_var("PATH", "");
439
    let fallback = resolve_openagents_cli();
440
441
    std::env::set_var("PATH", original);
442
443
    assert!(
444
        output.output.contains("SHIM-ON-PATH"),
445
        "the CLI on PATH should have run: {}",
446
        output.output
447
    );
448
    assert!(
449
        output.output.contains("[ran the `openagents` CLI on PATH:")
450
            && output.output.contains(&shim.display().to_string()),
451
        "the result should name the program that answered: {}",
452
        output.output
453
    );
454
    assert!(!output.is_error, "the shim exited zero");
455
456
    let (found, source) = on_path.expect("a shim on PATH must resolve");
457
    assert_eq!(source, OpenAgentsCliSource::Path);
458
    assert_eq!(found, shim);
459
460
    let (found, source) = fallback.expect("an empty PATH must still resolve to this binary");
461
    assert_eq!(
462
        source,
463
        OpenAgentsCliSource::ThisBinary,
464
        "with nothing on PATH the tool should fall back rather than fail"
465
    );
466
    assert_eq!(found, std::env::current_exe().unwrap());
467
}
468
469
// ──────────────────────────────────────────────────────────────── defect 5
470
471
/// **Cancelling a turn stops the command the `shell` tool started.**
472
///
473
/// `run_real_shell` spawned with neither `process_group(0)` nor
474
/// `kill_on_drop(true)`, while `delegate.rs`, `computer.rs` and `acp.rs` all
475
/// already put their children in a group. Dropping the future — which is what
476
/// `run_proxy_child`'s `tokio::select!` does when a fan-out is cancelled —
477
/// left the operating-system process running, reparented to init. On the
478
/// default `ox-alpha` lane a child is an in-process task with no pid at all,
479
/// so `signals::stop_tree` is never called for it and nothing else stopped
480
/// what it had started.
481
///
482
/// Observed live before the fix: `oa coder --delegate --count 2` told to run
483
/// `sleep 300`, interrupted with `SIGINT`, printed "Stopping the fan-out;
484
/// children are being signalled", reported both children "stopped before
485
/// finishing", exited — and left two `sleep 300` processes at `PPID 1`.
486
///
487
/// Asserted without pgrep: the abandoned command must not go on to finish its
488
/// work, so the file it was told to create must never appear.
489
#[tokio::test]
490
async fn cancelling_a_shell_tool_call_stops_the_command_it_started() {
491
    let dir = tempfile::tempdir().unwrap();
492
    let witness = dir.path().join("the-orphan-kept-going.txt");
493
    let registry = HarnessToolRegistry::new(Some(dir.path().to_path_buf()));
494
495
    let call = ToolCall {
496
        id: "call_orphan".to_string(),
497
        name: "shell".to_string(),
498
        arguments: serde_json::json!({
499
            "command": format!("sleep 2 && touch '{}'", witness.display())
500
        }),
501
    };
502
503
    let handle = tokio::spawn(async move { registry.execute_tool(&call).await });
504
    // Long enough for the shell to be spawned, far short of its `sleep`.
505
    tokio::time::sleep(std::time::Duration::from_millis(400)).await;
506
    handle.abort();
507
    assert!(handle.await.is_err(), "the call should have been cancelled");
508
    assert!(
509
        !witness.exists(),
510
        "the command finished before it could be cancelled; the test is racing"
511
    );
512
513
    // Well past the command's own sleep.
514
    tokio::time::sleep(std::time::Duration::from_millis(3500)).await;
515
516
    assert!(
517
        !witness.exists(),
518
        "the shell subprocess outlived the cancelled call and finished its work"
519
    );
520
}
521
522
// ──────────────────────────────────────────────────────────────── defect 6
523
524
/// **A failed tool is reported to the model as a failure.**
525
///
526
/// `execute_tool`'s `shell` arm returned `is_error: false` unconditionally,
527
/// whatever the command exited with, and the `openagents` arm did the same
528
/// even when the program could not be spawned. Only the refusal check and an
529
/// unknown tool name ever set the flag, so a failing build read to the model
530
/// exactly like a passing one.
531
///
532
/// `tools::defect_tests` asserts this on `run_real_shell`'s own return value.
533
/// This asserts the layer above — that the outcome survives the dispatch into
534
/// `ToolOutput`, which is the field a caller actually reads.
535
#[tokio::test]
536
async fn a_failing_shell_command_is_reported_as_a_failure_through_the_tool_result() {
537
    let (_dir, registry) = registry();
538
539
    let failed = registry
540
        .execute_tool(&ToolCall {
541
            id: "call_fail".to_string(),
542
            name: "shell".to_string(),
543
            arguments: serde_json::json!({ "command": "exit 42" }),
544
        })
545
        .await;
546
    assert!(
547
        failed.output.contains("exited with code 42"),
548
        "the exit code belongs in the text the model reads: {}",
549
        failed.output
550
    );
551
    assert!(failed.is_error, "a non-zero exit must reach the caller");
552
553
    let worked = registry
554
        .execute_tool(&ToolCall {
555
            id: "call_ok".to_string(),
556
            name: "shell".to_string(),
557
            arguments: serde_json::json!({ "command": "echo fine" }),
558
        })
559
        .await;
560
    assert_eq!(worked.output, "fine");
561
    assert!(
562
        !worked.is_error,
563
        "a successful command must not be an error"
564
    );
565
}

This page updates live while a promote is in flight · changelog