Give Gym runs a live lifecycle linked to their trial threads #241

Closed AtlantisPleb opened this 1d ago 1 comment

Why

A Gym run exists on the server only after it is over. OpenAgents.Gym.record_run/1 records one completed row through POST /api/v1/gym/runs (idempotent by recipe digest), and /gym renders the table. While a suite is actually running — the interesting part — the server knows nothing.

Meanwhile the transcript of each trial already streams through this server: the Harbor adapter (bench/adapters/openagents_coder.py in the monorepo) runs openagents coder --plain on the thread lane, so every proxy-lane trial opens a thread here, appends typed events (turn.user, turn.reasoning, tool.ran, turn.assistant) as it works, and /threads/:id can already watch it live over PubSub. The two records are never connected: the gym run does not know which threads were its trials.

This issue is the server-side prerequisite for streaming runs on /gym with the chat transcript components (see the follow-on issue).

What

  1. Run lifecycle. Let the harness register a run when the suite starts (status running, with suite, agent, model, lane, recipe digest, and planned task count), and finalize it with the graded totals the existing POST /api/v1/gym/runs shape carries today. Keep recipe-digest idempotency across both steps. Add an end state for runs that die without a grade (abandoned or similar) so the scoreboard never shows a forever-running row — a bounded staleness sweep is acceptable.
  2. Trial records. Within a run, record each trial: task name, state (running / passed / failed / ungraded), and — on the thread lane — the id of the thread the coder opened for it. Verify a claimed thread belongs to the bearer's account before linking; refuse the link otherwise. Local-lane (ollama/...) trials have no thread and stay transcript-less; that is expected, not an error.
  3. Broadcast. Publish run and trial changes on a PubSub topic (for example gym) so GymLive and a future run-detail LiveView can subscribe instead of polling.
  4. Compatibility. The current one-shot completed-run POST /api/v1/gym/runs from bench/post_gym_run.py must keep working unchanged; the lifecycle is additive.

Auth stays exactly the Gym ingest posture: an ordinary forge:write bearer, with a live operator-standing recheck on every call.

Acceptance

  • A run registered at start appears immediately with status running; finalizing it folds in the grades and flips the status; resubmitting the final grade replays rather than duplicates.
  • A trial can be attached with its thread id, the linkage is refused for a thread the bearer's account does not own, and the run's trials list the thread ids.
  • Subscribing to the gym topic observes run registration, trial state changes, and finalization without polling.
  • mix precommit green.

Part of the Gym lane (project 14). Companion harness work: OpenAgentsInc/openagents (bench lane) must register the run and report each trial's thread.

  1. AtlantisPleb opened this issue 1d ago
  2. A AtlantisPleb Author 1d ago

    Shipped in 32ef279 (re-landed after a push race; originally 633dfaf), with the lineage registration repaired in a82f7f9. Routes: POST /api/v1/gym/runs/start, POST /api/v1/gym/runs/:id/trials (thread ownership verified at ingest), PATCH /api/v1/gym/runs/:id (graded / abandoned, 409 on conflicts). PubSub on "gym" and "gym:run:". Verified live against a dev server: start → trial upsert → unowned-thread refusal (422) → finalize → 409 on re-finalize, and then a real Harbor terminal-bench run (fix-git, 1/1) registered, linked its trial's thread, and folded grades on finalize. Full precommit green (4,533 at land; 4,568 on repaired main).

  3. closed this as completed 1d ago
Sign in with GitHub to comment on this issue.