Why
A Gym run exists on the server only after it is over. OpenAgents.Gym.record_run/1 records one completed row through POST /api/v1/gym/runs (idempotent by recipe digest), and /gym renders the table. While a suite is actually running — the interesting part — the server knows nothing.
Meanwhile the transcript of each trial already streams through this server: the Harbor adapter (bench/adapters/openagents_coder.py in the monorepo) runs openagents coder --plain on the thread lane, so every proxy-lane trial opens a thread here, appends typed events (turn.user, turn.reasoning, tool.ran, turn.assistant) as it works, and /threads/:id can already watch it live over PubSub. The two records are never connected: the gym run does not know which threads were its trials.
This issue is the server-side prerequisite for streaming runs on /gym with the chat transcript components (see the follow-on issue).
What
- Run lifecycle. Let the harness register a run when the suite starts (status
running, with suite, agent, model, lane, recipe digest, and planned task count), and finalize it with the graded totals the existing POST /api/v1/gym/runs shape carries today. Keep recipe-digest idempotency across both steps. Add an end state for runs that die without a grade (abandoned or similar) so the scoreboard never shows a forever-running row — a bounded staleness sweep is acceptable.
- Trial records. Within a run, record each trial: task name, state (running / passed / failed / ungraded), and — on the thread lane — the id of the thread the coder opened for it. Verify a claimed thread belongs to the bearer's account before linking; refuse the link otherwise. Local-lane (
ollama/...) trials have no thread and stay transcript-less; that is expected, not an error.
- Broadcast. Publish run and trial changes on a PubSub topic (for example
gym) so GymLive and a future run-detail LiveView can subscribe instead of polling.
- Compatibility. The current one-shot completed-run
POST /api/v1/gym/runs from bench/post_gym_run.py must keep working unchanged; the lifecycle is additive.
Auth stays exactly the Gym ingest posture: an ordinary forge:write bearer, with a live operator-standing recheck on every call.
Acceptance
- A run registered at start appears immediately with status
running; finalizing it folds in the grades and flips the status; resubmitting the final grade replays rather than duplicates.
- A trial can be attached with its thread id, the linkage is refused for a thread the bearer's account does not own, and the run's trials list the thread ids.
- Subscribing to the gym topic observes run registration, trial state changes, and finalization without polling.
mix precommit green.
Part of the Gym lane (project 14). Companion harness work: OpenAgentsInc/openagents (bench lane) must register the run and report each trial's thread.
Why
A Gym run exists on the server only after it is over.
OpenAgents.Gym.record_run/1records one completed row throughPOST /api/v1/gym/runs(idempotent by recipe digest), and/gymrenders the table. While a suite is actually running — the interesting part — the server knows nothing.Meanwhile the transcript of each trial already streams through this server: the Harbor adapter (
bench/adapters/openagents_coder.pyin the monorepo) runsopenagents coder --plainon the thread lane, so every proxy-lane trial opens a thread here, appends typed events (turn.user,turn.reasoning,tool.ran,turn.assistant) as it works, and/threads/:idcan already watch it live over PubSub. The two records are never connected: the gym run does not know which threads were its trials.This issue is the server-side prerequisite for streaming runs on
/gymwith the chat transcript components (see the follow-on issue).What
running, with suite, agent, model, lane, recipe digest, and planned task count), and finalize it with the graded totals the existingPOST /api/v1/gym/runsshape carries today. Keep recipe-digest idempotency across both steps. Add an end state for runs that die without a grade (abandonedor similar) so the scoreboard never shows a forever-running row — a bounded staleness sweep is acceptable.ollama/...) trials have no thread and stay transcript-less; that is expected, not an error.gym) soGymLiveand a future run-detail LiveView can subscribe instead of polling.POST /api/v1/gym/runsfrombench/post_gym_run.pymust keep working unchanged; the lifecycle is additive.Auth stays exactly the Gym ingest posture: an ordinary
forge:writebearer, with a live operator-standing recheck on every call.Acceptance
running; finalizing it folds in the grades and flips the status; resubmitting the final grade replays rather than duplicates.mix precommitgreen.Part of the Gym lane (project 14). Companion harness work: OpenAgentsInc/openagents (bench lane) must register the run and report each trial's thread.