Fix three oa box defects that only a real box could show

322e5920aee7 · AtlantisPleb · · parent 830538b341d3

Fix three oa box defects that only a real box could show

A `box:control` token was obtained through the device authorization flow, so
every `oa box` subcommand ran against production for the first time: a box was
provisioned, a command executed on it, a durable run started, followed, and
cancelled, a fanout plan admitted and read back, and every box stopped again.
Three defects surfaced that no test had reached, because each one lived in a
place where the code and its stub agreed with each other rather than with the
server.

`BoxRunRecord::finished` named states the server does not have. It matched
`succeeded`, `canceled`, and `expired`, and missed `completed` — the state a
run that worked ends in — and `lost`. `OpenAgents.Box.Run` declares exactly
`completed failed cancelled timed_out lost`. So `oa box runs output --follow`
could not end a successful follow: against production it ran 41 seconds past a
run that had already finished and then died printing a gateway error page. The
integration test did not catch this because its stub also answered `succeeded`;
it now answers what the server answers, and the unit test walks every state in
`@states` and every spelling that is not one.

The follow loop slept only on passes that found no new output, so a run that
printed steadily was followed by an unthrottled loop issuing two requests per
pass. `--interval-ms` now bounds every pass. That is almost certainly what
provoked the 502 above.

`resolve_conversation_id` reported any failure as "This deployment does not
report a conversation for the account", including a 502 and a dead socket.
Observed live: a transient gateway failure printed that sentence for an account
whose conversation resolved a minute either side of it. A refusal the server
authored still gets the sentence that names `--conversation`; anything else is
surfaced as itself.

`oa box fanout --request-id` was unreachable. clap required `--count`, so the
read path could only be entered by passing a number it then ignored. `--count`
is now required only when a plan is being requested.

Verified against https://openagents.com, conversation 3dd6d813: box
bx_cfyk253d, run 82191236 followed to `completed` and exited on its own the
same second the run finished. `cargo test -p openagents-cli`: 272 passed, 0
failed.

Refs #78

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SoZMfWRSGnf6FZX2Ar9rQ2
Co-Authored-By
Claude Opus 5 (1M context) <noreply@anthropic.com>

Deploy story

What this commit did to the running system — joined from the forge receipt chain, the part a commit page elsewhere cannot show.

Not deployed through the forge lane

No push, promotion, build, or deploy receipt references this commit (receipts are scanned over a bounded recent window). Changes shipped by full node replacement carry their proof in the release gate receipt instead.

Changed files

  • modified crates/openagents-cli/src/box_client.rs
  • modified crates/openagents-cli/src/cli.rs
  • added crates/openagents-cli/tests/box_conversation_test.rs
  • modified crates/openagents-cli/tests/box_follow_test.rs

Diff

4 files changed, +253 -19

crates/openagents-cli/src/box_client.rs modified +68 -14

@@ -65,10 +65,21 @@ pub struct BoxRunRecord {

65 65
66 66
impl BoxRunRecord {
67 67
    /// True once the server will produce no further output for this run.
68
    ///
69
    /// These are the server's own terminal states, from
70
    /// `OpenAgents.Box.Run`'s `@terminal_states` — `completed failed cancelled
71
    /// timed_out lost`. Getting this list wrong is not cosmetic: the whole
72
    /// list existed to end `--follow`, and an earlier version of it matched
73
    /// `succeeded`, `canceled`, and `expired`, none of which the server ever
74
    /// sends, while missing `completed`, which is what a run that worked ends
75
    /// in. Against production that made `oa box runs output --follow` spin
76
    /// past the end of a successful run until the API refused a request. The
77
    /// spellings that never appear are deliberately not kept "just in case":
78
    /// carrying them is what made the mistake survive a passing test.
68 79
    pub fn finished(&self) -> bool {
69 80
        matches!(
70 81
            self.state.as_str(),
71
            "succeeded" | "failed" | "cancelled" | "canceled" | "timed_out" | "expired"
82
            "completed" | "failed" | "cancelled" | "timed_out" | "lost"
72 83
        )
73 84
    }
74 85
}

@@ -306,11 +317,28 @@ impl BoxClient {

306 317
            }
307 318
        }
308 319
309
        let status = match (&named, &user) {
310
            (Err(ApiError::Refused { status, .. }), _) => *status,
311
            (_, Err(ApiError::Refused { status, .. })) => *status,
312
            _ => 200,
313
        };
320
        // A route that broke is not a deployment that has no conversation.
321
        // Only a refusal the *server* authored — it read the token and said no
322
        // — means "ask for the conversation another way". A 5xx, a gateway
323
        // error page, or a dead socket means the request never got an answer,
324
        // and reporting either of those as "this deployment does not report a
325
        // conversation for the account" sends the reader to fix a
326
        // configuration that was never wrong. Observed against production: a
327
        // transient 502 on `GET /api/v1/conversation` printed exactly that
328
        // sentence, for an account whose conversation resolved fine a minute
329
        // earlier and a minute later.
330
        let mut status = 200;
331
        for outcome in [named, user] {
332
            match outcome {
333
                Ok(_) => {}
334
                Err(ApiError::Refused { status: code, .. }) if code < 500 => {
335
                    if status == 200 {
336
                        status = code;
337
                    }
338
                }
339
                Err(error) => return Err(error),
340
            }
341
        }
314 342
        Err(ApiError::Refused {
315 343
            operation: "resolve user conversation".to_string(),
316 344
            status,

@@ -608,7 +636,6 @@ impl BoxClient {

608 636
                .run_output(conversation, box_id, run_id, cursor)
609 637
                .await?;
610 638
            sink(&chunk);
611
            let advanced = Some(chunk.next_offset) != cursor;
612 639
            cursor = Some(chunk.next_offset);
613 640
614 641
            let run = self.view_run(conversation, box_id, run_id).await?;

@@ -621,9 +648,13 @@ impl BoxClient {

621 648
                }
622 649
                return Ok((run, tail.next_offset));
623 650
            }
624
            if !advanced {
625
                tokio::time::sleep(interval).await;
626
            }
651
            // Sleep on every pass, including the ones that carried output.
652
            // This used to sleep only when the offset had not advanced, so a
653
            // run that printed steadily was followed by an unthrottled loop
654
            // issuing two requests per iteration as fast as the network
655
            // allowed. `--interval-ms` is the poll rate the caller asked for;
656
            // it is not a rate that applies only when nothing is happening.
657
            tokio::time::sleep(interval).await;
627 658
        }
628 659
    }
629 660

@@ -760,9 +791,32 @@ mod tests {

760 791
            cancellation_requested_at: None,
761 792
            cancellation_effective_at: None,
762 793
        };
763
        assert!(run("succeeded").finished());
764
        assert!(run("failed").finished());
765
        assert!(!run("running").finished());
766
        assert!(!run("queued").finished());
794
        // `OpenAgents.Box.Run` declares
795
        //   @states          admitted dispatched running completed failed
796
        //                    cancelled timed_out lost
797
        //   @terminal_states completed failed cancelled timed_out lost
798
        // Nothing else is a run state, so every state is checked here and the
799
        // split is asserted both ways. The version of this test that only
800
        // asserted `succeeded` and `failed` passed while `--follow` could not
801
        // end a successful run, because `succeeded` is not a state the server
802
        // has and `completed`, the one it uses, was not in the list.
803
        for state in ["completed", "failed", "cancelled", "timed_out", "lost"] {
804
            assert!(run(state).finished(), "{state} is a terminal run state");
805
        }
806
        for state in ["admitted", "dispatched", "running"] {
807
            assert!(
808
                !run(state).finished(),
809
                "{state} is a live run state and must keep --follow polling"
810
            );
811
        }
812
        // Spellings the server never sends. Treating one as terminal would end
813
        // a follow early on a run that was still producing output; the reason
814
        // they are named is that three of them were once in the list.
815
        for state in ["succeeded", "canceled", "expired", "queued", ""] {
816
            assert!(
817
                !run(state).finished(),
818
                "{state:?} is not a state OpenAgents.Box.Run can hold"
819
            );
820
        }
767 821
    }
768 822
}
crates/openagents-cli/src/cli.rs modified +10 -2

@@ -861,8 +861,13 @@ pub enum BoxAction {

861 861
    },
862 862
    /// Request a multi-box fanout admission plan
863 863
    Fanout {
864
        #[arg(long, help = "Number of boxes to request")]
865
        count: u64,
864
        // Optional, not required, because `--request-id` reads a plan that
865
        // already exists and has a count of its own. While this was
866
        // `count: u64`, clap refused every `--request-id` invocation for a
867
        // missing `--count`, and the only way to reach the read path was to
868
        // pass a number the command then ignored.
869
        #[arg(long, help = "Number of boxes to request; required without --request-id")]
870
        count: Option<u64>,
866 871
        #[arg(long, help = "Comma-separated labels for the fanout boxes")]
867 872
        labels: Option<String>,
868 873
        #[arg(long, help = "Allow scaling up to the budgeted limit")]

@@ -2914,6 +2919,9 @@ async fn run_box(action: BoxAction, api_base: &str, token: Option<String>, json:

2914 2919
                                .collect()
2915 2920
                        })
2916 2921
                        .unwrap_or_default();
2922
                    let Some(count) = count else {
2923
                        fail("pass --count <n> to request a fanout, or --request-id <id> to read an existing plan");
2924
                    };
2917 2925
                    or_fail(client.fanout(&id, count, &parsed, budgeted).await)
2918 2926
                }
2919 2927
            };
crates/openagents-cli/tests/box_conversation_test.rs added +124

@@ -0,0 +1,124 @@

1
//! What `oa box` says when it cannot resolve the account's conversation.
2
//!
3
//! Every box subcommand starts by turning "no `--conversation`" into a
4
//! conversation id, so this is the first thing a reader sees when anything is
5
//! wrong, and the sentence it prints is the one they act on. There are two
6
//! different situations behind it and they need two different sentences:
7
//!
8
//!   * the server read the credential and answered — a `401` because the token
9
//!     carries `forge:write` and not `box:control`, say. Nothing is broken;
10
//!     the reader needs `--conversation`, or a token with the scope.
11
//!   * the request never got an answer — a `502` from the edge, a gateway
12
//!     error page, a dead socket. Nothing about the account is known, least of
13
//!     all that it has no conversation.
14
//!
15
//! Reporting the second as the first was observed against production: a
16
//! transient `502` on `GET /api/v1/conversation` printed "This deployment does
17
//! not report a conversation for the account", for an account whose
18
//! conversation resolved a minute earlier and a minute later.
19
20
use openagents_cli::box_client::BoxClient;
21
use openagents_cli::tracker::ApiError;
22
use tokio::io::{AsyncReadExt, AsyncWriteExt};
23
24
/// A server that answers every request with one canned status and body.
25
async fn start_stub(status_line: &'static str, body: &'static str) -> String {
26
    let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
27
    let port = listener.local_addr().unwrap().port();
28
29
    tokio::spawn(async move {
30
        loop {
31
            let Ok((mut socket, _)) = listener.accept().await else {
32
                return;
33
            };
34
            let mut buffer = vec![0u8; 8192];
35
            if socket.read(&mut buffer).await.unwrap_or(0) == 0 {
36
                continue;
37
            }
38
            let response = format!(
39
                "HTTP/1.1 {status_line}\r\ncontent-type: text/html\r\ncontent-length: {}\r\nconnection: close\r\n\r\n{body}",
40
                body.len()
41
            );
42
            let _ = socket.write_all(response.as_bytes()).await;
43
            let _ = socket.flush().await;
44
        }
45
    });
46
47
    format!("http://127.0.0.1:{port}/api/v1")
48
}
49
50
/// A gateway failure is reported as a gateway failure.
51
#[tokio::test]
52
async fn a_five_hundred_from_the_conversation_route_is_not_reported_as_a_missing_conversation() {
53
    let base = start_stub("502 Bad Gateway", "<html><title>502</title></html>").await;
54
    let client = BoxClient::new(&base, Some("token".to_string()));
55
56
    let error = client
57
        .resolve_conversation_id()
58
        .await
59
        .expect_err("a 502 must not resolve to a conversation id");
60
61
    match error {
62
        ApiError::Refused {
63
            status, message, ..
64
        } => {
65
            assert_eq!(
66
                status, 502,
67
                "the status the server actually sent must survive"
68
            );
69
            assert!(
70
                !message.contains("does not report a conversation"),
71
                "a 502 is not an account without a conversation; the CLI said: {message}"
72
            );
73
        }
74
        other => panic!("expected the server's own refusal, got {other:?}"),
75
    }
76
}
77
78
/// A transport failure is reported as a transport failure. Port 1 refuses
79
/// connections, so nothing about the account is ever known here.
80
#[tokio::test]
81
async fn an_unreachable_api_is_not_reported_as_a_missing_conversation() {
82
    let client = BoxClient::new("http://127.0.0.1:1/api/v1", Some("token".to_string()));
83
84
    let error = client
85
        .resolve_conversation_id()
86
        .await
87
        .expect_err("an unreachable API must not resolve to a conversation id");
88
89
    assert!(
90
        matches!(error, ApiError::Transport { .. }),
91
        "an unreachable API must surface as a transport failure, got {error:?}"
92
    );
93
}
94
95
/// The refusal the caller *can* act on keeps its sentence. A `401` on both
96
/// routes is a real answer from the server: the credential was read and turned
97
/// down, and naming `--conversation` is the way past it.
98
#[tokio::test]
99
async fn a_refused_credential_still_names_the_flag_that_unblocks_the_caller() {
100
    let base = start_stub(
101
        "401 Unauthorized",
102
        r#"{"error":{"code":"invalid_api_token"}}"#,
103
    )
104
    .await;
105
    let client = BoxClient::new(&base, Some("token".to_string()));
106
107
    let error = client
108
        .resolve_conversation_id()
109
        .await
110
        .expect_err("a 401 must not resolve to a conversation id");
111
112
    match error {
113
        ApiError::Refused {
114
            status, message, ..
115
        } => {
116
            assert_eq!(status, 401);
117
            assert!(
118
                message.contains("--conversation"),
119
                "the refusal must name the flag that unblocks the caller; it said: {message}"
120
            );
121
        }
122
        other => panic!("expected a refusal, got {other:?}"),
123
    }
124
}
crates/openagents-cli/tests/box_follow_test.rs modified +51 -3

@@ -74,9 +74,20 @@ async fn start_stub() -> Stub {

74 74
                }
75 75
            } else {
76 76
                // The run reports `running` until both of the first two windows
77
                // have been read, then `succeeded`.
77
                // have been read, then `completed`.
78
                //
79
                // `completed` is what `OpenAgents.Box.Run` actually sets, and
80
                // saying so here is the point of this line. This stub used to
81
                // answer `succeeded`, which no deployment has ever sent, and
82
                // the client agreed with the stub instead of the server: it
83
                // treated `succeeded` as terminal and `completed` as still
84
                // running. Both sides were self-consistently wrong, this test
85
                // passed, and against production `oa box runs output --follow`
86
                // ran past the end of every successful run until the API
87
                // refused a request. A stub that invents the server's
88
                // vocabulary tests the stub.
78 89
                let state = if reads.load(Ordering::SeqCst) >= 2 {
79
                    "succeeded"
90
                    "completed"
80 91
                } else {
81 92
                    "running"
82 93
                };

@@ -143,7 +154,7 @@ async fn following_a_run_reads_past_the_first_window_and_past_the_terminal_state

143 154
        "the follow must read every window, in order, including the one the box \
144 155
         wrote after the run turned terminal"
145 156
    );
146
    assert_eq!(run.state, "succeeded");
157
    assert_eq!(run.state, "completed");
147 158
    assert!(run.finished());
148 159
    assert_eq!(next_offset, 18);
149 160
}

@@ -168,3 +179,40 @@ async fn a_refused_read_ends_the_follow_rather_than_truncating_it() {

168 179
        "an unreachable box must not produce a finished run"
169 180
    );
170 181
}
182
183
/// The poll interval applies to every pass, not only to the ones that found
184
/// nothing new.
185
///
186
/// The loop used to skip its sleep whenever the offset had advanced, so a run
187
/// that printed steadily was followed by an unthrottled loop issuing two
188
/// requests per pass — an output read and a run read — as fast as the network
189
/// allowed. Against production that is what a `--follow` looked like right up
190
/// until the edge answered 502 mid-stream.
191
///
192
/// The stub advances the offset on every read and only turns terminal on the
193
/// second pass, so the fixed loop sleeps exactly once — on a pass that carried
194
/// output, which is the pass the old code skipped. Before the fix this whole
195
/// follow finished in single-digit milliseconds over loopback; after it, it
196
/// cannot finish in less than one interval. That gap is what is asserted, and
197
/// it is wide enough not to turn into a timing flake on a loaded machine.
198
#[tokio::test]
199
async fn following_sleeps_between_passes_even_while_output_is_arriving() {
200
    let stub = start_stub().await;
201
    let client = BoxClient::new(&stub.base, None);
202
    let interval = std::time::Duration::from_millis(250);
203
204
    let started = std::time::Instant::now();
205
    let (run, _next_offset) = client
206
        .follow_run_output("conv_1", "bx_1", "run_1", Some(0), interval, |_| {})
207
        .await
208
        .expect("the follow failed");
209
    let elapsed = started.elapsed();
210
211
    assert_eq!(run.state, "completed");
212
    assert!(
213
        elapsed >= interval,
214
        "the pass that carried output must still wait out the poll interval; \
215
         this follow took {elapsed:?}, under the {interval:?} it asked for, \
216
         which means the loop is spinning on the API while output arrives"
217
    );
218
}

This page updates live while a promote is in flight · changelog