Stream Gym runs live at /gym with the chat transcript components

c60a36497750 · AtlantisPleb · · parent 829615707966

Stream Gym runs live at /gym with the chat transcript components

Closes-nothing; implements issue #242 on top of #241's run lifecycle.

The index goes live: GymLive subscribes to the gym topic, renders
running runs above the graded table with their live trial tally, an
elapsed clock, and the shimmer working indicator, and flips a run into
the graded table in place on finalize — no reload anywhere. The suite
filter and the operator recheck on every event are unchanged.

The new run page at /gym/runs/:id (GymRunLive, same :operator_chat live
session, recheck on mount and on every event) shows the run header, the
trials with per-state presentation, and the selected trial's transcript
streaming through the same conversation components /chat renders with.
The event-to-component mapping lives in the shared
OpenAgentsWeb.AI.ThreadTranscript module so ThreadShowLive can adopt it
later; payloads are read defensively and a malformed one degrades to
the neutral bounded raw row. The projection protocol follows
ThreadShowLive exactly: subscribe first, then snapshot, then dedup
buffered broadcasts with a monotonic last_event_id.

The read path is deliberate: OpenAgents.Gym.fetch_trial_thread/1 reads
a thread only through a stored trial linkage that record_trial/3
ownership-verified at ingest, for the operator-gated gym surface.
THREAD-001 and ADMIN-001 name the reader, and the operator-surface and
grant-token-reach enumeration proofs carry the new route, module, and
Threads.unsubscribe/1 export.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RfZq5s3rc6zpnBR75pTQaU
Co-Authored-By
Claude Fable 5 <noreply@anthropic.com>

Deploy story

What this commit did to the running system — joined from the forge receipt chain, the part a commit page elsewhere cannot show.

pushed
by user · WAL seq 372 · 2026-08-25T12:48:22.889422Z

Changed files

  • modified INVARIANTS.md
  • modified lib/openagents/gym.ex
  • modified lib/openagents/threads.ex
  • added lib/openagents_web/components/ai/thread_transcript.ex
  • modified lib/openagents_web/live/gym_live.ex
  • added lib/openagents_web/live/gym_run_live.ex
  • modified lib/openagents_web/route_authority.ex
  • modified lib/openagents_web/router.ex
  • modified test/openagents/gym_test.exs
  • modified test/openagents/threads/grant_token_reach_test.exs
  • modified test/openagents_web/live/gym_live_test.exs
  • added test/openagents_web/live/gym_run_live_test.exs
  • modified test/openagents_web/operator_surface_test.exs

Diff

13 files changed, +1080 -19

INVARIANTS.md modified +29 -9

@@ -2317,6 +2317,20 @@ conversation, and a thread is not one.

2317 2317
  anybody else's — ADMIN-001 names the same check at the route. It resolves
2318 2318
  and links; it never writes to the thread, mints for it, or returns it.
2319 2319
2320
  Amended 2026-08-25 (issue #242): the stored linkage gained a reader.
2321
  `OpenAgents.Gym.fetch_trial_thread/1` resolves a thread only through a
2322
  stored trial row's `thread_id` — a link `record_trial/3` admitted through
2323
  `get_for_user/2` at ingest — and serves the operator-gated `/gym/runs/:id`
2324
  transcript view (`OpenAgentsWeb.GymRunLive`, recheck on mount and on every
2325
  event). The trial row is the only key, so an arbitrary thread id has no
2326
  path in; the function reads, and never writes to the thread, mints for it,
2327
  or returns a grant. This is an operator read of another account's
2328
  transcript — exactly the transcripts a bearer deliberately linked to a
2329
  benchmark trial, and nothing else — and ADMIN-001 names it beside the
2330
  ingest check. `test/openagents/gym_test.exs` proves the refusals (unknown
2331
  trial, unlinked trial, deleted thread) and
2332
  `test/openagents_web/live/gym_run_live_test.exs` proves the gate.
2333
2320 2334
Evidence: `OpenAgents.Threads`, `OpenAgents.Threads.Thread`,
2321 2335
`OpenAgents.Threads.Event`, `OpenAgents.Inference.mint/1`,
2322 2336
`OpenAgents.Inference.expire_elapsed_for_owner/1`,

@@ -2998,15 +3012,21 @@ sentence:

2998 3012
  and `PATCH /api/v1/gym/runs/:id`
2999 3013
  (`OpenAgentsWeb.GymRunController`, which rechecks the operator on every
3000 3014
  request over the bearer scope), and reading the scoreboard from `/gym`
3001
  (`OpenAgentsWeb.GymLive`, recheck on mount and on every event). A run is a
3002
  benchmark record — recipe digest, task, model, lane, reward, duration —
3003
  never account data; the surface is operator-only because it is
3004
  pre-release instrumentation, not because it reads across accounts. The one
3005
  cross-record link a trial may carry, a `thread_id`, is verified at ingest:
3006
  `OpenAgents.Gym.record_trial/3` admits a thread only when
3007
  `OpenAgents.Threads.get_for_user/2` resolves it for the bearer's account,
3008
  and an unknown thread and an unowned one refuse identically, so the Gym
3009
  cannot be used to confirm that a foreign thread id exists.
3015
  and a run's page from `/gym/runs/:id` (`OpenAgentsWeb.GymLive` and
3016
  `OpenAgentsWeb.GymRunLive`, recheck on mount and on every event). A run
3017
  is a benchmark record — recipe digest, task, model, lane, reward,
3018
  duration — never account data. The one cross-record link a trial may
3019
  carry, a `thread_id`, is verified at ingest: `OpenAgents.Gym.record_trial/3`
3020
  admits a thread only when `OpenAgents.Threads.get_for_user/2` resolves it
3021
  for the bearer's account, and an unknown thread and an unowned one refuse
3022
  identically, so the Gym cannot be used to confirm that a foreign thread
3023
  id exists. Since issue #242 that link is also read back: the run page
3024
  streams a linked trial's thread transcript to the operator through
3025
  `OpenAgents.Gym.fetch_trial_thread/1`, which resolves only through a
3026
  stored, ingest-verified trial linkage — so this surface does read another
3027
  account's data, exactly the transcripts a bearer deliberately linked to a
3028
  benchmark trial, and nothing else (THREAD-001 names the same reader from
3029
  the thread side).
3010 3030
3011 3031
Reading a private forum board and raising a repository's transparency tier to
3012 3032
`glass` are operator reads that widen with the same allowlist
lib/openagents/gym.ex modified +35 -3

@@ -23,9 +23,11 @@ defmodule OpenAgents.Gym do

23 23
  A trial's thread link is verified at ingest: the thread must exist and
24 24
  belong to the bearer's account, and an unknown thread and an unowned one
25 25
  refuse identically, so the check confirms nothing about threads the
26
  account cannot see. A run still `running` whose last update is older
27
  than six hours is swept to `abandoned` lazily on the read paths, so the
28
  scoreboard never shows a forever-running row.
26
  account cannot see. The operator-gated gym surface reads a linked
27
  transcript back through `fetch_trial_thread/1`, which resolves only
28
  through that stored, verified linkage. A run still `running` whose last
29
  update is older than six hours is swept to `abandoned` lazily on the read
30
  paths, so the scoreboard never shows a forever-running row.
29 31
30 32
  ## PubSub
31 33

@@ -215,6 +217,36 @@ defmodule OpenAgents.Gym do

215 217
    trials_query() |> where([t], t.run_id == ^run_id) |> Repo.all()
216 218
  end
217 219
220
  @doc """
221
  The thread a trial's transcript lives on, read through the trial linkage.
222
223
  This is the read path for the operator-gated gym surface. The `/gym`
224
  viewer is any operator, not necessarily the thread's owner, so it cannot
225
  read through the account-scoped `OpenAgents.Threads.fetch_readable/2`.
226
  Reading here is sound because the linkage is the authority:
227
  `record_trial/3` admitted the `thread_id` only after
228
  `OpenAgents.Threads.get_for_user/2` resolved it for the reporting
229
  bearer's account (INVARIANTS THREAD-001, ADMIN-001), so every stored
230
  linkage names a thread its reporter owned and deliberately attached to a
231
  benchmark trial. The trial row is the only key — an arbitrary thread id
232
  has no path in — and the function reads; it never writes to the thread,
233
  mints for it, or returns a grant.
234
235
  Returns `:error` for an unknown trial, a trial with no linkage, and a
236
  linked thread that no longer exists — a thread may be deleted with its
237
  account while the benchmark record stays.
238
  """
239
  @spec fetch_trial_thread(String.t()) :: {:ok, Threads.Thread.t()} | :error
240
  def fetch_trial_thread(trial_id) when is_binary(trial_id) do
241
    with {:ok, id} <- Ecto.UUID.cast(trial_id),
242
         %Trial{thread_id: thread_id} when is_binary(thread_id) <- Repo.get(Trial, id),
243
         %Threads.Thread{} = thread <- Repo.get(Threads.Thread, thread_id) do
244
      {:ok, thread}
245
    else
246
      _no_verified_linkage -> :error
247
    end
248
  end
249
218 250
  @doc """
219 251
  Runs, newest first, optionally filtered by suite. Bounded.
220 252
lib/openagents/threads.ex modified +13

@@ -554,6 +554,19 @@ defmodule OpenAgents.Threads do

554 554
    Phoenix.PubSub.subscribe(OpenAgents.PubSub, topic(thread_id))
555 555
  end
556 556
557
  @doc """
558
  Drop a `subscribe/1` subscription.
559
560
  For a viewer that moves between transcripts on one socket, so events from
561
  the thread it left stop arriving instead of being filtered forever.
562
  """
563
  @spec unsubscribe(Thread.t() | String.t()) :: :ok
564
  def unsubscribe(%Thread{id: thread_id}), do: unsubscribe(thread_id)
565
566
  def unsubscribe(thread_id) when is_binary(thread_id) do
567
    Phoenix.PubSub.unsubscribe(OpenAgents.PubSub, topic(thread_id))
568
  end
569
557 570
  defp topic(thread_id), do: "thread:" <> thread_id
558 571
559 572
  defp broadcast(%Event{} = event) do
lib/openagents_web/components/ai/thread_transcript.ex added +198

@@ -0,0 +1,198 @@

1
defmodule OpenAgentsWeb.AI.ThreadTranscript do
2
  @moduledoc """
3
  One thread transcript event, rendered through the shared AI Elements
4
  components — the same vocabulary `/chat` renders with.
5
6
  This module is the mapping from the thread event vocabulary onto the chat
7
  components: `turn.user` becomes a user `OpenAgentsWeb.AI.Conversation`
8
  message, `turn.reasoning` becomes the `OpenAgentsWeb.AI.Reasoning`
9
  reasoning disclosure, `tool.ran` becomes the tool block `ChatLive` renders
10
  tool activity with, and `turn.assistant` becomes an assistant message with
11
  the app's Markdown rendering. It is a shared function component rather
12
  than a private helper of the gym page so `OpenAgentsWeb.ThreadShowLive`
13
  can adopt the same rendering later instead of keeping a second vocabulary.
14
15
  Every payload field is client-written, so every read is defensive, exactly
16
  as `OpenAgentsWeb.ThreadShowLive` reads the same payloads: a missing or
17
  oddly typed field degrades the event to the neutral bounded raw row, an
18
  unknown event type takes the raw row too, and nothing here can crash the
19
  transcript around a malformed payload.
20
  """
21
22
  use OpenAgentsWeb, :html
23
24
  import OpenAgentsWeb.AI.Conversation, only: [message: 1, message_content: 1]
25
26
  import OpenAgentsWeb.AI.Reasoning,
27
    only: [
28
      reasoning: 1,
29
      reasoning_trigger: 1,
30
      reasoning_content: 1,
31
      tool: 1,
32
      tool_header: 1,
33
      tool_content: 1,
34
      tool_input: 1,
35
      tool_output: 1
36
    ]
37
38
  @bounded_json_characters 2_000
39
40
  @doc """
41
  One transcript entry, dispatched on its event type.
42
43
  `event` is an `OpenAgents.Threads.Event`. `id`, when given, prefixes the
44
  DOM ids of the inner components.
45
  """
46
  attr :id, :string, default: nil
47
  attr :event, :any, required: true
48
49
  def transcript_event(%{event: %{event_type: "turn.user"} = event} = assigns) do
50
    case text(event.payload) do
51
      nil ->
52
        raw_row(assigns)
53
54
      text ->
55
        assigns = assign(assigns, text: text, steered: steered?(event.payload))
56
57
        ~H"""
58
        <.message from="user" class="items-end">
59
          <.badge :if={@steered} variant={:warning} class="message-provenance">steered</.badge>
60
          <.message_content>
61
            <div class="whitespace-pre-wrap">{@text}</div>
62
          </.message_content>
63
        </.message>
64
        """
65
    end
66
  end
67
68
  def transcript_event(%{event: %{event_type: "turn.reasoning"} = event} = assigns) do
69
    case text(event.payload) do
70
      nil ->
71
        raw_row(assigns)
72
73
      text ->
74
        assigns = assign(assigns, :text, text)
75
76
        ~H"""
77
        <.reasoning id={@id && "#{@id}-reasoning"}>
78
          <.reasoning_trigger />
79
          <.reasoning_content text={@text} />
80
        </.reasoning>
81
        """
82
    end
83
  end
84
85
  def transcript_event(%{event: %{event_type: "tool.ran"} = event} = assigns) do
86
    case string(event.payload, ["tool", "name"]) do
87
      nil ->
88
        raw_row(assigns)
89
90
      tool_name ->
91
        assigns =
92
          assign(assigns,
93
            tool_name: tool_name,
94
            state: tool_state(string(event.payload, ["status"])),
95
            arguments: bounded_json(Map.get(event.payload, "arguments")),
96
            result: bounded_json(Map.get(event.payload, "result")),
97
            error: bounded_json(Map.get(event.payload, "error"))
98
          )
99
100
        ~H"""
101
        <.tool id={@id && "#{@id}-tool"} class="mb-0">
102
          <.tool_header type="dynamic-tool" tool_name={@tool_name} state={@state} />
103
          <.tool_content>
104
            <.tool_input :if={@arguments} input={@arguments} />
105
            <.tool_output :if={@result || @error} output={@result} error_text={@error} />
106
            <p :if={!@arguments && !@result && !@error} class="text-muted-foreground text-xs">
107
              The report carried no arguments and no result.
108
            </p>
109
          </.tool_content>
110
        </.tool>
111
        """
112
    end
113
  end
114
115
  def transcript_event(%{event: %{event_type: "turn.assistant"} = event} = assigns) do
116
    case text(event.payload) do
117
      nil ->
118
        raw_row(assigns)
119
120
      text ->
121
        assigns = assign(assigns, :text, text)
122
123
        ~H"""
124
        <.message from="assistant">
125
          <.message_content text={@text} />
126
        </.message>
127
        """
128
    end
129
  end
130
131
  def transcript_event(assigns), do: raw_row(assigns)
132
133
  # The neutral row: whatever the type, whatever the payload, it renders as
134
  # the recorded fact rather than crashing the transcript around it.
135
  defp raw_row(assigns) do
136
    assigns = assign(assigns, :json, bounded_json(assigns.event.payload))
137
138
    ~H"""
139
    <details class="rounded-md border border-border" data-transcript-raw="true">
140
      <summary class="flex cursor-pointer select-none items-center gap-2 px-3 py-2 text-xs">
141
        <code class="font-mono">{@event.event_type}</code>
142
        <span class="text-muted-foreground"><.time_ago at={@event.emitted_at} /></span>
143
      </summary>
144
      <pre
145
        :if={@json}
146
        class="overflow-x-auto border-t border-border px-3 py-2 font-mono text-xs"
147
      ><code phx-no-format>{@json}</code></pre>
148
    </details>
149
    """
150
  end
151
152
  # ── payload readers ──────────────────────────────────────────────────────
153
154
  defp text(payload), do: string(payload, ["text"])
155
156
  defp string(payload, keys) when is_map(payload) do
157
    Enum.find_value(keys, fn key ->
158
      case Map.get(payload, key) do
159
        value when is_binary(value) and value != "" -> value
160
        _other -> nil
161
      end
162
    end)
163
  end
164
165
  defp string(_payload, _keys), do: nil
166
167
  defp steered?(payload) when is_map(payload), do: Map.get(payload, "steered") == true
168
  defp steered?(_payload), do: false
169
170
  defp bounded_json(nil), do: nil
171
172
  defp bounded_json(value) do
173
    case Jason.encode(value, pretty: true) do
174
      {:ok, json} when byte_size(json) > @bounded_json_characters ->
175
        String.slice(json, 0, @bounded_json_characters) <> "\n…"
176
177
      {:ok, json} ->
178
        json
179
180
      {:error, _unencodable} ->
181
        inspect(value, limit: 50, printable_limit: @bounded_json_characters)
182
    end
183
  end
184
185
  # The client-written status words mapped onto the AI SDK tool-part states
186
  # `tool_header/1` reads, the same narrowing `ChatLive` applies to durable
187
  # step statuses. `tool.ran` is past tense, so an absent or unrecognized
188
  # status reads as completed rather than inventing an error.
189
  defp tool_state("ok"), do: "output-available"
190
  defp tool_state("success"), do: "output-available"
191
  defp tool_state("succeeded"), do: "output-available"
192
  defp tool_state("error"), do: "output-error"
193
  defp tool_state("failed"), do: "output-error"
194
  defp tool_state("refused"), do: "output-denied"
195
  defp tool_state("denied"), do: "output-denied"
196
  defp tool_state("running"), do: "input-available"
197
  defp tool_state(_other), do: "output-available"
198
end
lib/openagents_web/live/gym_live.ex modified +98 -6

@@ -1,17 +1,29 @@

1 1
defmodule OpenAgentsWeb.GymLive do
2 2
  @moduledoc """
3
  The Gym: graded benchmark runs of our agents, operator-only.
3
  The Gym: graded benchmark runs of our agents, operator-only, live.
4 4
5 5
  Read-only over `OpenAgents.Gym` — the harness runs elsewhere and posts
6
  results through `POST /api/v1/gym/runs`; this surface is the scoreboard
7
  that capability work (models, plugins, harness changes) is read against.
6
  results through `POST /api/v1/gym/runs` and the lifecycle routes; this
7
  surface is the scoreboard that capability work (models, plugins, harness
8
  changes) is read against. It subscribes to the gym topic, so a running
9
  suite appears the moment the harness registers it, its trial tally moves
10
  as trials report, and the run flips to the graded table in place when it
11
  finalizes — no reload anywhere.
12
8 13
  Operator-gated the same way `/chat` is: the route sits behind the
9 14
  `:operator` pipeline, the mount re-checks, and every event re-checks,
10 15
  because a long-lived socket outlives the decision that opened it.
16
17
  The graded table stays a stream. The running section is a bounded assign
18
  instead, because each entry carries a live trial tally that mutates on
19
  every `{:gym_trial, _}` broadcast, and the collection is bounded by the
20
  number of suites running at once rather than by history.
11 21
  """
12 22
13 23
  use OpenAgentsWeb, :live_view
14 24
25
  import OpenAgentsWeb.AI.Conversation, only: [shimmer: 1]
26
15 27
  alias OpenAgents.Accounts
16 28
  alias OpenAgents.Gym
17 29
  alias OpenAgents.Gym.Run

@@ -19,6 +31,10 @@ defmodule OpenAgentsWeb.GymLive do

19 31
  @impl true
20 32
  def mount(_params, _session, socket) do
21 33
    if Accounts.admin?(socket.assigns.current_user) do
34
      # Subscribe before the snapshot read, so a run or trial that lands
35
      # between the two arrives as a message rather than being missed.
36
      if connected?(socket), do: Gym.subscribe()
37
22 38
      {:ok, load(socket, nil)}
23 39
    else
24 40
      {:ok, redirect(socket, to: ~p"/")}

@@ -34,18 +50,58 @@ defmodule OpenAgentsWeb.GymLive do

34 50
    end
35 51
  end
36 52
53
  @impl true
54
  def handle_info({:gym_run, %Run{}}, socket) do
55
    # Start, finalize, abandon, and sweep all reload the page's two
56
    # sections under the current filter, which is what moves a finalized
57
    # run from the running section into the graded table in place.
58
    if Accounts.admin?(socket.assigns.current_user) do
59
      {:noreply, load(socket, socket.assigns.suite)}
60
    else
61
      {:noreply, redirect(socket, to: ~p"/")}
62
    end
63
  end
64
65
  def handle_info({:gym_trial, trial}, socket) do
66
    if Accounts.admin?(socket.assigns.current_user) do
67
      running =
68
        Enum.map(socket.assigns.running, fn entry ->
69
          if entry.run.id == trial.run_id, do: running_entry(entry.run), else: entry
70
        end)
71
72
      {:noreply, assign(socket, :running, running)}
73
    else
74
      {:noreply, redirect(socket, to: ~p"/")}
75
    end
76
  end
77
37 78
  defp presence(""), do: nil
38 79
  defp presence(suite) when is_binary(suite), do: suite
39 80
40 81
  defp load(socket, suite) do
41 82
    runs = Gym.list_runs(suite: suite)
83
    {running, finished} = Enum.split_with(runs, &(&1.status == "running"))
42 84
43 85
    socket
44 86
    |> assign(:page_title, "Gym")
45 87
    |> assign(:suite, suite)
46 88
    |> assign(:suites, Gym.suites())
89
    |> assign(:running, Enum.map(running, &running_entry/1))
47 90
    |> assign(:runs_empty?, runs == [])
48
    |> stream(:runs, runs, reset: true)
91
    |> assign(:table_empty?, finished == [])
92
    |> stream(:runs, finished, reset: true)
93
  end
94
95
  # A running run with its live trial tally: how many trials the harness
96
  # has reported and how many have passed so far.
97
  defp running_entry(run) do
98
    trials = Gym.list_trials(run)
99
100
    %{
101
      run: run,
102
      reported: length(trials),
103
      passed: Enum.count(trials, &(&1.state == "passed"))
104
    }
49 105
  end
50 106
51 107
  defp percent(nil), do: "—"

@@ -64,6 +120,9 @@ defmodule OpenAgentsWeb.GymLive do

64 120
    if(rest == 0, do: "#{minutes}m", else: "#{minutes}m #{rest}s")
65 121
  end
66 122
123
  defp elapsed_since(started_at),
124
    do: elapsed(max(DateTime.diff(DateTime.utc_now(), started_at, :second), 0))
125
67 126
  # A run still `running` (or swept to `abandoned`) has no grades yet.
68 127
  defp counts(passed, total) when is_integer(passed) and is_integer(total),
69 128
    do: "#{passed}/#{total}"

@@ -104,6 +163,37 @@ defmodule OpenAgentsWeb.GymLive do

104 163
          </select>
105 164
        </form>
106 165
166
        <section :if={@running != []} id="gym-running" class="space-y-3" aria-label="Running now">
167
          <h2 class="text-sm font-medium text-muted-foreground">Running now</h2>
168
          <.link
169
            :for={entry <- @running}
170
            navigate={~p"/gym/runs/#{entry.run.id}"}
171
            id={"gym-running-#{entry.run.id}"}
172
            class="block"
173
          >
174
            <.card class="text-sm transition-colors hover:border-primary">
175
              <div class="flex flex-wrap items-center gap-x-6 gap-y-2">
176
                <span class="font-mono">{entry.run.suite}</span>
177
                <span>
178
                  {entry.run.agent}
179
                  <span :if={entry.run.agent_version} class="text-muted-foreground">
180
                    @{entry.run.agent_version}
181
                  </span>
182
                </span>
183
                <span class="font-mono">{entry.run.model}</span>
184
                <span>{entry.run.lane || "—"}</span>
185
                <span class="tabular-nums" data-tally>
186
                  {entry.passed} passed / {entry.reported} reported
187
                </span>
188
                <span class="whitespace-nowrap text-muted-foreground">
189
                  {elapsed_since(entry.run.inserted_at)}
190
                </span>
191
                <.shimmer text="Running" tag="span" class="text-xs" />
192
              </div>
193
            </.card>
194
          </.link>
195
        </section>
196
107 197
        <div :if={@runs_empty?}>
108 198
          <.empty title="No runs recorded yet">
109 199
            No graded runs have been posted. The harness records one with

@@ -112,7 +202,7 @@ defmodule OpenAgentsWeb.GymLive do

112 202
          </.empty>
113 203
        </div>
114 204
115
        <div :if={!@runs_empty?} class="overflow-x-auto">
205
        <div :if={!@table_empty?} class="overflow-x-auto">
116 206
          <table class="table">
117 207
            <thead>
118 208
              <tr>

@@ -131,7 +221,9 @@ defmodule OpenAgentsWeb.GymLive do

131 221
            <tbody id="gym-runs" phx-update="stream">
132 222
              <tr :for={{id, run} <- @streams.runs} id={id}>
133 223
                <td class="whitespace-nowrap">
134
                  {Calendar.strftime(run.inserted_at, "%Y-%m-%d %H:%M")}
224
                  <.link navigate={~p"/gym/runs/#{run.id}"} class="hover:underline">
225
                    {Calendar.strftime(run.inserted_at, "%Y-%m-%d %H:%M")}
226
                  </.link>
135 227
                </td>
136 228
                <td class="font-mono text-sm">{run.suite}</td>
137 229
                <td>
lib/openagents_web/live/gym_run_live.ex added +392

@@ -0,0 +1,392 @@

1
defmodule OpenAgentsWeb.GymRunLive do
2
  @moduledoc """
3
  One Gym run, live: the run header, its trials as they report, and the
4
  selected trial's transcript streaming through the same conversation
5
  components `/chat` renders with.
6
7
  Operator-only the way `/gym` is: the route sits in the `:operator_chat`
8
  live session, the mount re-checks `OpenAgents.Accounts.admin?/1`, and
9
  every event re-checks it. An unknown run id redirects to `/gym` rather
10
  than confirming anything, matching how the operator surfaces route a
11
  reader back to the main flow.
12
13
  The transcript read path is `OpenAgents.Gym.fetch_trial_thread/1`: the
14
  viewer is any operator, not the thread's owner, so the read resolves only
15
  through a stored trial linkage that was ownership-verified at ingest
16
  (INVARIANTS THREAD-001, ADMIN-001). The snapshot-to-live order follows
17
  the projection protocol `OpenAgentsWeb.ThreadShowLive` documents:
18
  subscribe to the thread's topic first, then read the snapshot, then let
19
  buffered broadcasts drain with a monotonic `last_event_id` dedup, so an
20
  event is never dropped and never doubled. Trials on a lane that leaves no
21
  thread render a state-only placeholder instead.
22
  """
23
24
  use OpenAgentsWeb, :live_view
25
26
  import OpenAgentsWeb.AI.Conversation,
27
    only: [conversation: 1, conversation_content: 1, shimmer: 1]
28
29
  import OpenAgentsWeb.AI.ThreadTranscript, only: [transcript_event: 1]
30
31
  alias OpenAgents.Accounts
32
  alias OpenAgents.Gym
33
  alias OpenAgents.Gym.Run
34
  alias OpenAgents.Threads
35
36
  # Transcript pages are capped at 50 by the context; forty pages bounds the
37
  # snapshot at 2,000 events, the same bound `ThreadShowLive` holds.
38
  @maximum_pages 40
39
40
  @impl true
41
  def mount(%{"id" => run_id}, _session, socket) do
42
    if Accounts.admin?(socket.assigns.current_user) do
43
      # Subscribe before the snapshot read, so a trial report that lands
44
      # between the two arrives as a message rather than being missed.
45
      if connected?(socket), do: Gym.subscribe_run(run_id)
46
47
      case Gym.fetch_run(run_id) do
48
        {:ok, run} ->
49
          socket =
50
            socket
51
            |> assign(:page_title, "Gym run")
52
            |> assign(:run, run)
53
            |> assign(:trials, run.trials)
54
            |> assign(:selected_trial_id, nil)
55
            |> assign(:thread, nil)
56
            |> assign(:last_event_id, 0)
57
            |> assign(:transcript, :none)
58
            |> assign(:events_empty?, true)
59
            |> stream(:events, [])
60
            |> select_trial(default_trial(run.trials))
61
            |> restream_trials()
62
63
          {:ok, socket}
64
65
        :error ->
66
          {:ok, redirect(socket, to: ~p"/gym")}
67
      end
68
    else
69
      {:ok, redirect(socket, to: ~p"/")}
70
    end
71
  end
72
73
  @impl true
74
  def handle_event("select_trial", %{"id" => trial_id}, socket) do
75
    if Accounts.admin?(socket.assigns.current_user) do
76
      socket =
77
        case Enum.find(socket.assigns.trials, &(&1.id == trial_id)) do
78
          nil -> socket
79
          trial -> socket |> select_trial(trial) |> restream_trials()
80
        end
81
82
      {:noreply, socket}
83
    else
84
      {:noreply, redirect(socket, to: ~p"/")}
85
    end
86
  end
87
88
  @impl true
89
  def handle_info({:gym_run, %Run{} = run}, socket) do
90
    cond do
91
      !Accounts.admin?(socket.assigns.current_user) ->
92
        {:noreply, redirect(socket, to: ~p"/")}
93
94
      run.id == socket.assigns.run.id ->
95
        # The broadcast carries the run as stored, trials not loaded; the
96
        # trial list lives in its own assign, so only the header moves.
97
        {:noreply, assign(socket, :run, run)}
98
99
      true ->
100
        {:noreply, socket}
101
    end
102
  end
103
104
  def handle_info({:gym_trial, trial}, socket) do
105
    if Accounts.admin?(socket.assigns.current_user) do
106
      trials = upsert_trial(socket.assigns.trials, trial)
107
      socket = socket |> assign(:trials, trials) |> restream_trials()
108
109
      socket =
110
        cond do
111
          # The first reported trial becomes the selection, so an operator
112
          # watching an empty run is attached the moment work starts.
113
          socket.assigns.selected_trial_id == nil ->
114
            select_trial(socket, default_trial(trials))
115
116
          # The selected trial gained its thread link after selection.
117
          trial.id == socket.assigns.selected_trial_id and
118
            socket.assigns.transcript != :live and is_binary(trial.thread_id) ->
119
            select_trial(socket, trial)
120
121
          true ->
122
            socket
123
        end
124
125
      {:noreply, socket}
126
    else
127
      {:noreply, redirect(socket, to: ~p"/")}
128
    end
129
  end
130
131
  def handle_info({:thread_event, event}, socket) do
132
    cond do
133
      !Accounts.admin?(socket.assigns.current_user) ->
134
        {:noreply, redirect(socket, to: ~p"/")}
135
136
      socket.assigns.thread == nil or event.thread_id != socket.assigns.thread.id ->
137
        {:noreply, socket}
138
139
      event.id <= socket.assigns.last_event_id ->
140
        {:noreply, socket}
141
142
      true ->
143
        {:noreply,
144
         socket
145
         |> assign(:last_event_id, event.id)
146
         |> assign(:events_empty?, false)
147
         |> stream_insert(:events, event)}
148
    end
149
  end
150
151
  # ── selection ────────────────────────────────────────────────────────────
152
153
  defp default_trial(trials),
154
    do: Enum.find(trials, &(&1.state == "running")) || List.first(trials)
155
156
  defp select_trial(socket, nil) do
157
    socket
158
    |> detach_thread()
159
    |> assign(selected_trial_id: nil, transcript: :none, events_empty?: true, last_event_id: 0)
160
    |> stream(:events, [], reset: true)
161
  end
162
163
  defp select_trial(socket, trial) do
164
    socket = socket |> detach_thread() |> assign(:selected_trial_id, trial.id)
165
166
    if trial.thread_id == nil do
167
      socket
168
      |> assign(transcript: :no_thread, events_empty?: true, last_event_id: 0)
169
      |> stream(:events, [], reset: true)
170
    else
171
      attach_thread(socket, trial)
172
    end
173
  end
174
175
  defp attach_thread(socket, trial) do
176
    case Gym.fetch_trial_thread(trial.id) do
177
      {:ok, thread} ->
178
        # Attach the live subscriber before reading the snapshot: an append
179
        # that lands between the two arrives as a buffered message and is
180
        # deduped by id, so the gap cannot lose an event.
181
        if connected?(socket), do: Threads.subscribe(thread)
182
183
        events = transcript_snapshot(thread)
184
185
        socket
186
        |> assign(:thread, thread)
187
        |> assign(:transcript, :live)
188
        |> assign(:last_event_id, last_id(events))
189
        |> assign(:events_empty?, events == [])
190
        |> stream(:events, events, reset: true)
191
192
      :error ->
193
        socket
194
        |> assign(transcript: :unavailable, events_empty?: true, last_event_id: 0)
195
        |> stream(:events, [], reset: true)
196
    end
197
  end
198
199
  defp detach_thread(socket) do
200
    case socket.assigns[:thread] do
201
      nil ->
202
        socket
203
204
      thread ->
205
        if connected?(socket), do: Threads.unsubscribe(thread)
206
        assign(socket, :thread, nil)
207
    end
208
  end
209
210
  defp upsert_trial(trials, trial) do
211
    trials
212
    |> Enum.reject(&(&1.id == trial.id or &1.task == trial.task))
213
    |> then(&[trial | &1])
214
    |> Enum.sort_by(& &1.task)
215
  end
216
217
  defp restream_trials(socket) do
218
    socket
219
    |> assign(:trials_empty?, socket.assigns.trials == [])
220
    |> stream(:trials, socket.assigns.trials, reset: true)
221
  end
222
223
  # ── snapshot ─────────────────────────────────────────────────────────────
224
225
  defp transcript_snapshot(thread), do: transcript_snapshot(thread, nil, @maximum_pages, [])
226
227
  defp transcript_snapshot(_thread, _after_id, 0, pages),
228
    do: pages |> Enum.reverse() |> List.flatten()
229
230
  defp transcript_snapshot(thread, after_id, remaining, pages) do
231
    page = Threads.list_events(thread, after: after_id)
232
233
    case last_id(page) do
234
      0 -> transcript_snapshot(thread, after_id, 0, pages)
235
      last -> transcript_snapshot(thread, last, remaining - 1, [page | pages])
236
    end
237
  end
238
239
  defp last_id([]), do: 0
240
  defp last_id(events), do: List.last(events).id
241
242
  # ── presentation ─────────────────────────────────────────────────────────
243
244
  defp percent(nil), do: "—"
245
  defp percent(score), do: "#{Float.round(score * 100, 1)}%"
246
247
  defp status_variant("running"), do: :info
248
  defp status_variant("graded"), do: :success
249
  defp status_variant("abandoned"), do: :dim
250
  defp status_variant(_status), do: :default
251
252
  defp trial_variant("passed"), do: :success
253
  defp trial_variant("failed"), do: :danger
254
  defp trial_variant(_running_or_ungraded), do: :dim
255
256
  @impl true
257
  def render(assigns) do
258
    ~H"""
259
    <Layouts.app
260
      flash={@flash}
261
      sidebar_sections={assigns[:sidebar_sections]}
262
      current_scope={@current_scope}
263
    >
264
      <main id="gym-run" class="mx-auto w-full max-w-6xl space-y-6 px-4 py-10">
265
        <header class="space-y-3">
266
          <div class="flex flex-wrap items-center gap-3">
267
            <.badge id="gym-run-status" variant={status_variant(@run.status)}>
268
              {@run.status}
269
            </.badge>
270
            <span class="font-mono text-xs text-muted-foreground">{@run.id}</span>
271
          </div>
272
          <h1 class="text-2xl font-semibold tracking-tight">
273
            {@run.suite}
274
            <span class="text-muted-foreground">·</span>
275
            {@run.agent}<span
276
              :if={@run.agent_version}
277
              class="text-muted-foreground"
278
            >@{@run.agent_version}</span>
279
          </h1>
280
          <dl
281
            id="gym-run-facts"
282
            class="flex flex-wrap gap-x-6 gap-y-1 text-sm text-muted-foreground"
283
          >
284
            <div class="flex gap-1.5">
285
              <dt>Model</dt>
286
              <dd class="font-mono text-xs leading-5">{@run.model}</dd>
287
            </div>
288
            <div class="flex gap-1.5">
289
              <dt>Lane</dt>
290
              <dd>{@run.lane || "—"}</dd>
291
            </div>
292
            <div class="flex gap-1.5">
293
              <dt>Score</dt>
294
              <dd id="gym-run-score" class="tabular-nums">{percent(Run.score(@run))}</dd>
295
            </div>
296
            <div class="flex gap-1.5">
297
              <dt>Recipe</dt>
298
              <dd class="max-w-xs truncate font-mono text-xs leading-5" title={@run.recipe_digest}>
299
                {@run.recipe_digest}
300
              </dd>
301
            </div>
302
            <div class="flex gap-1.5">
303
              <dt>Started</dt>
304
              <dd><.time_ago at={@run.inserted_at} /></dd>
305
            </div>
306
            <div :if={@run.completed_at} class="flex gap-1.5">
307
              <dt>Completed</dt>
308
              <dd><.time_ago at={@run.completed_at} /></dd>
309
            </div>
310
          </dl>
311
        </header>
312
313
        <div class="grid gap-6 lg:grid-cols-[minmax(16rem,1fr)_2fr]">
314
          <section aria-label="Trials" class="space-y-3">
315
            <h2 class="text-sm font-medium text-muted-foreground">Trials</h2>
316
317
            <.empty :if={@trials_empty?} id="gym-run-trials-empty" title="No trials reported yet">
318
              Trials appear here as the harness launches them.
319
            </.empty>
320
321
            <div id="gym-run-trials" phx-update="stream" class="space-y-2">
322
              <button
323
                :for={{dom_id, trial} <- @streams.trials}
324
                id={dom_id}
325
                type="button"
326
                phx-click="select_trial"
327
                phx-value-id={trial.id}
328
                data-state={trial.state}
329
                data-selected={to_string(@selected_trial_id == trial.id)}
330
                class={[
331
                  "flex w-full items-center justify-between gap-3 rounded-md border px-3 py-2 text-left text-sm transition-colors",
332
                  if(@selected_trial_id == trial.id,
333
                    do: "border-primary bg-secondary",
334
                    else: "border-border hover:bg-secondary"
335
                  )
336
                ]}
337
              >
338
                <span class="min-w-0 truncate font-mono text-xs">{trial.task}</span>
339
                <%= if trial.state == "running" do %>
340
                  <.shimmer text="running" tag="span" class="shrink-0 text-xs" />
341
                <% else %>
342
                  <.badge variant={trial_variant(trial.state)} class="shrink-0">
343
                    {trial.state}
344
                  </.badge>
345
                <% end %>
346
              </button>
347
            </div>
348
          </section>
349
350
          <section aria-label="Transcript" class="space-y-3">
351
            <h2 class="text-sm font-medium text-muted-foreground">Transcript</h2>
352
353
            <%= case @transcript do %>
354
              <% :none -> %>
355
                <.empty id="gym-transcript-none" title="No trial selected">
356
                  Select a trial to read its transcript.
357
                </.empty>
358
              <% :no_thread -> %>
359
                <.empty id="gym-transcript-no-thread" title="No transcript">
360
                  This trial's lane left no transcript.
361
                </.empty>
362
              <% :unavailable -> %>
363
                <.empty id="gym-transcript-unavailable" title="Transcript unavailable">
364
                  The linked thread no longer exists.
365
                </.empty>
366
              <% :live -> %>
367
                <div class="flex h-[36rem] flex-col overflow-hidden rounded-md border border-border">
368
                  <.conversation id="gym-conversation" aria-label="Trial transcript">
369
                    <.conversation_content id="gym-conversation-content">
370
                      <.empty :if={@events_empty?} id="gym-transcript-empty" title="No events yet">
371
                        The transcript fills as the trial works.
372
                      </.empty>
373
                      <div id="gym-transcript-events" phx-update="stream" class="contents">
374
                        <div
375
                          :for={{dom_id, event} <- @streams.events}
376
                          id={dom_id}
377
                          data-kind={event.event_type}
378
                        >
379
                          <.transcript_event id={dom_id} event={event} />
380
                        </div>
381
                      </div>
382
                    </.conversation_content>
383
                  </.conversation>
384
                </div>
385
            <% end %>
386
          </section>
387
        </div>
388
      </main>
389
    </Layouts.app>
390
    """
391
  end
392
end
lib/openagents_web/route_authority.ex modified +3

@@ -200,6 +200,9 @@ defmodule OpenAgentsWeb.RouteAuthority do

200 200
  defp policy(%{path: "/gym"}),
201 201
    do: declaration(:operator, "configured operator GitHub ID", "gym:read", false)
202 202
203
  defp policy(%{path: "/gym/runs/:id"}),
204
    do: declaration(:operator, "configured operator GitHub ID", "gym:read", false)
205
203 206
  defp policy(%{path: "/admin/analytics"}),
204 207
    do: declaration(:operator, "configured operator GitHub ID", "analytics:read", false)
205 208
lib/openagents_web/router.ex modified +1

@@ -277,6 +277,7 @@ defmodule OpenAgentsWeb.Router do

277 277
      # than a second gating mechanism. Widening it is a decision, not a
278 278
      # default.
279 279
      live "/gym", GymLive, :index
280
      live "/gym/runs/:id", GymRunLive, :show
280 281
    end
281 282
  end
282 283
test/openagents/gym_test.exs modified +50

@@ -331,4 +331,54 @@ defmodule OpenAgents.GymTest do

331 331
      assert Gym.fetch_run("not-a-uuid") == :error
332 332
    end
333 333
  end
334
335
  describe "fetch_trial_thread/1" do
336
    test "reads a thread only through a stored, verified linkage" do
337
      bearer = github_user("gym-thread-reader")
338
      {:ok, thread} = Threads.open(bearer, "Run the linked trial")
339
      run = running_run()
340
341
      {:ok, linked} =
342
        Gym.record_trial(bearer, run, %{
343
          "task" => "linked",
344
          "state" => "running",
345
          "thread_id" => thread.id
346
        })
347
348
      assert {:ok, fetched} = Gym.fetch_trial_thread(linked.id)
349
      assert fetched.id == thread.id
350
    end
351
352
    test "a trial without a linkage is refused" do
353
      bearer = github_user("gym-thread-unlinked")
354
      run = running_run()
355
356
      {:ok, unlinked} =
357
        Gym.record_trial(bearer, run, %{"task" => "local-lane", "state" => "running"})
358
359
      assert Gym.fetch_trial_thread(unlinked.id) == :error
360
    end
361
362
    test "an unknown trial id is refused, however it is spelled" do
363
      assert Gym.fetch_trial_thread(Ecto.UUID.generate()) == :error
364
      assert Gym.fetch_trial_thread("not-a-uuid") == :error
365
    end
366
367
    test "a linkage whose thread was deleted with its account is refused" do
368
      bearer = github_user("gym-thread-deleted")
369
      {:ok, thread} = Threads.open(bearer, "Run then delete")
370
      run = running_run()
371
372
      {:ok, linked} =
373
        Gym.record_trial(bearer, run, %{
374
          "task" => "deleted",
375
          "state" => "running",
376
          "thread_id" => thread.id
377
        })
378
379
      {:ok, _deleted} = Repo.delete(thread)
380
381
      assert Gym.fetch_trial_thread(linked.id) == :error
382
    end
383
  end
334 384
end
test/openagents/threads/grant_token_reach_test.exs modified +2 -1

@@ -76,7 +76,8 @@ defmodule OpenAgents.Threads.GrantTokenReachTest do

76 76
    {:record_event, 3} => :thread_struct,
77 77
    {:record_events, 2} => :thread_struct,
78 78
    {:spend, 1} => :thread_struct,
79
    {:subscribe, 1} => :thread_struct
79
    {:subscribe, 1} => :thread_struct,
80
    {:unsubscribe, 1} => :thread_struct
80 81
  }
81 82
82 83
  # Every module that reaches a token-returning `OpenAgents.Threads` export.
test/openagents_web/live/gym_live_test.exs modified +40

@@ -60,6 +60,46 @@ defmodule OpenAgentsWeb.GymLiveTest do

60 60
    end
61 61
  end
62 62
63
  describe "live updates" do
64
    test "a running run appears, its tally moves, and it flips to graded in place", %{
65
      conn: conn
66
    } do
67
      conn = log_in_admin_user(conn, "gym-live-operator")
68
      {:ok, view, _html} = live(conn, ~p"/gym")
69
70
      bearer = github_user("gym-live-bearer")
71
72
      {:ok, run, false} =
73
        Gym.start_run(%{
74
          "suite" => "terminal-bench@2.0",
75
          "agent" => "openagents-coder",
76
          "model" => "ox-alpha",
77
          "lane" => "proxy",
78
          "tasks_total" => 2
79
        })
80
81
      running = view |> element("#gym-running-#{run.id}") |> render()
82
      assert running =~ "terminal-bench@2.0"
83
      assert running =~ "0 passed / 0 reported"
84
      assert running =~ ~p"/gym/runs/#{run.id}"
85
86
      {:ok, _trial} = Gym.record_trial(bearer, run, %{"task" => "hello", "state" => "passed"})
87
88
      assert view |> element("#gym-running-#{run.id}") |> render() =~ "1 passed / 1 reported"
89
90
      {:ok, graded} =
91
        Gym.finalize_run(run, %{
92
          "tasks_total" => 2,
93
          "tasks_passed" => 1,
94
          "recipe_digest" => "sha256:" <> String.duplicate("f", 64)
95
        })
96
97
      html = render(view)
98
      refute has_element?(view, "#gym-running-#{graded.id}")
99
      assert html =~ "50.0%"
100
    end
101
  end
102
63 103
  describe "runs" do
64 104
    test "recorded runs render with score, and the suite filter narrows", %{conn: conn} do
65 105
      _bench = record_run("terminal-bench@2.0", "d")
test/openagents_web/live/gym_run_live_test.exs added +216

@@ -0,0 +1,216 @@

1
defmodule OpenAgentsWeb.GymRunLiveTest do
2
  @moduledoc """
3
  `/gym/runs/:id` gates like every operator surface and streams the
4
  selected trial's transcript through the shared conversation components:
5
  the snapshot renders on mount, a later append arrives without reload, a
6
  malformed payload degrades to the neutral raw row, and a lane that left
7
  no thread renders a state-only placeholder.
8
  """
9
10
  use OpenAgentsWeb.ConnCase, async: false
11
12
  import Phoenix.LiveViewTest
13
14
  alias OpenAgents.Gym
15
  alias OpenAgents.Threads
16
17
  defp start_run(overrides \\ %{}) do
18
    {:ok, run, false} =
19
      Gym.start_run(
20
        Map.merge(
21
          %{
22
            "suite" => "terminal-bench@2.0",
23
            "agent" => "openagents-coder",
24
            "agent_version" => "0.3.5",
25
            "model" => "ox-alpha",
26
            "lane" => "proxy",
27
            "tasks_total" => 3
28
          },
29
          overrides
30
        )
31
      )
32
33
    run
34
  end
35
36
  describe "access" do
37
    test "an ordinary authenticated account is redirected", %{conn: conn} do
38
      run = start_run()
39
      conn = log_in_github_user(conn, "gym-run-ordinary")
40
41
      assert {:error, {:redirect, %{to: "/"}}} = live(conn, ~p"/gym/runs/#{run.id}")
42
    end
43
44
    test "an unauthenticated visitor is redirected", %{conn: conn} do
45
      run = start_run()
46
47
      assert {:error, {:redirect, %{to: "/"}}} = live(conn, ~p"/gym/runs/#{run.id}")
48
    end
49
50
    test "an unknown run id returns the operator to the scoreboard", %{conn: conn} do
51
      conn = log_in_admin_user(conn, "gym-run-unknown-operator")
52
53
      assert {:error, {:redirect, %{to: "/gym"}}} =
54
               live(conn, ~p"/gym/runs/#{Ecto.UUID.generate()}")
55
56
      assert {:error, {:redirect, %{to: "/gym"}}} = live(conn, ~p"/gym/runs/not-a-uuid")
57
    end
58
  end
59
60
  describe "the run page" do
61
    test "renders the header and updates the status in place on finalize", %{conn: conn} do
62
      run = start_run()
63
      conn = log_in_admin_user(conn, "gym-run-header-operator")
64
65
      {:ok, view, html} = live(conn, ~p"/gym/runs/#{run.id}")
66
67
      assert html =~ "terminal-bench@2.0"
68
      assert html =~ "openagents-coder"
69
      assert html =~ "ox-alpha"
70
      assert view |> element("#gym-run-status") |> render() =~ "running"
71
      assert has_element?(view, "#gym-run-trials-empty")
72
73
      {:ok, _graded} =
74
        Gym.finalize_run(run, %{
75
          "tasks_total" => 3,
76
          "tasks_passed" => 2,
77
          "recipe_digest" => "sha256:" <> String.duplicate("9", 64)
78
        })
79
80
      assert view |> element("#gym-run-status") |> render() =~ "graded"
81
      assert view |> element("#gym-run-score") |> render() =~ "66.7%"
82
    end
83
84
    test "a trial reported after mount appears and becomes the selection", %{conn: conn} do
85
      run = start_run()
86
      bearer = github_user("gym-run-late-bearer")
87
      conn = log_in_admin_user(conn, "gym-run-late-operator")
88
89
      {:ok, view, _html} = live(conn, ~p"/gym/runs/#{run.id}")
90
91
      {:ok, trial} =
92
        Gym.record_trial(bearer, run, %{"task" => "hello-world", "state" => "running"})
93
94
      row = view |> element("#trials-#{trial.id}") |> render()
95
      assert row =~ "hello-world"
96
      assert row =~ ~s(data-selected="true")
97
      assert has_element?(view, "#gym-transcript-no-thread")
98
    end
99
  end
100
101
  describe "the transcript" do
102
    defp linked_run(bearer) do
103
      {:ok, thread} = Threads.open(bearer, "Solve hello-world")
104
      run = start_run()
105
106
      {:ok, trial} =
107
        Gym.record_trial(bearer, run, %{
108
          "task" => "hello-world",
109
          "state" => "running",
110
          "thread_id" => thread.id
111
        })
112
113
      {run, trial, thread}
114
    end
115
116
    test "renders the snapshot and streams a live append through the chat components", %{
117
      conn: conn
118
    } do
119
      bearer = github_user("gym-run-transcript-bearer")
120
      {run, _trial, thread} = linked_run(bearer)
121
122
      {:ok, thread} = Threads.record_event(thread, "turn.user", %{"text" => "Fix the bug"})
123
124
      {:ok, thread} =
125
        Threads.record_event(thread, "turn.reasoning", %{"text" => "The bug is in the parser."})
126
127
      {:ok, thread} =
128
        Threads.record_event(thread, "tool.ran", %{
129
          "tool" => "read_file",
130
          "status" => "ok",
131
          "arguments" => %{"path" => "lib/parser.ex"},
132
          "result" => "defmodule Parser do"
133
        })
134
135
      conn = log_in_admin_user(conn, "gym-run-transcript-operator")
136
      {:ok, view, _html} = live(conn, ~p"/gym/runs/#{run.id}")
137
138
      user_row =
139
        view |> element("#gym-transcript-events [data-kind='turn.user']") |> render()
140
141
      assert user_row =~ "Fix the bug"
142
      assert user_row =~ ~s(data-from="user")
143
144
      assert view |> element("#gym-transcript-events [data-kind='turn.reasoning']") |> render() =~
145
               "The bug is in the parser."
146
147
      tool_row = view |> element("#gym-transcript-events [data-kind='tool.ran']") |> render()
148
      assert tool_row =~ "read_file"
149
      assert tool_row =~ "Completed"
150
      assert tool_row =~ "lib/parser.ex"
151
      assert tool_row =~ "defmodule Parser do"
152
153
      # The live append arrives without a reload, rendered as markdown.
154
      {:ok, _thread} =
155
        Threads.record_event(thread, "turn.assistant", %{"text" => "Fixed. It works **now**."})
156
157
      assistant_row =
158
        view |> element("#gym-transcript-events [data-kind='turn.assistant']") |> render()
159
160
      assert assistant_row =~ ~s(data-from="assistant")
161
      assert assistant_row =~ "<strong>now</strong>"
162
    end
163
164
    test "a malformed payload and an unknown type degrade to the neutral raw row", %{
165
      conn: conn
166
    } do
167
      bearer = github_user("gym-run-malformed-bearer")
168
      {run, _trial, thread} = linked_run(bearer)
169
170
      {:ok, thread} = Threads.record_event(thread, "turn.user", %{"no_text" => true})
171
      {:ok, _thread} = Threads.record_event(thread, "plugin.custom", %{"whatever" => [1, true]})
172
173
      conn = log_in_admin_user(conn, "gym-run-malformed-operator")
174
      {:ok, view, _html} = live(conn, ~p"/gym/runs/#{run.id}")
175
176
      textless =
177
        view
178
        |> element("#gym-transcript-events [data-kind='turn.user'] [data-transcript-raw]")
179
        |> render()
180
181
      assert textless =~ "no_text"
182
183
      unknown =
184
        view
185
        |> element("#gym-transcript-events [data-kind='plugin.custom'] [data-transcript-raw]")
186
        |> render()
187
188
      assert unknown =~ "plugin.custom"
189
    end
190
191
    test "selecting a trial moves the transcript, and a threadless lane says so", %{conn: conn} do
192
      bearer = github_user("gym-run-select-bearer")
193
      {run, linked, _thread} = linked_run(bearer)
194
195
      {:ok, local} =
196
        Gym.record_trial(bearer, run, %{"task" => "local-lane", "state" => "ungraded"})
197
198
      conn = log_in_admin_user(conn, "gym-run-select-operator")
199
      {:ok, view, _html} = live(conn, ~p"/gym/runs/#{run.id}")
200
201
      # The running linked trial is the default selection.
202
      assert view |> element("#trials-#{linked.id}") |> render() =~ ~s(data-selected="true")
203
      assert has_element?(view, "#gym-conversation")
204
205
      view |> element("#trials-#{local.id}") |> render_click()
206
207
      assert view |> element("#trials-#{local.id}") |> render() =~ ~s(data-selected="true")
208
      assert has_element?(view, "#gym-transcript-no-thread")
209
      refute has_element?(view, "#gym-conversation")
210
211
      view |> element("#trials-#{linked.id}") |> render_click()
212
213
      assert has_element?(view, "#gym-conversation")
214
    end
215
  end
216
end
test/openagents_web/operator_surface_test.exs modified +3

@@ -40,6 +40,7 @@ defmodule OpenAgentsWeb.OperatorSurfaceTest do

40 40
    # `/gym` landed while the rename was in flight; the forge targets move with
41 41
    # everything else.
42 42
    {"get", "/gym", "gym:read", false},
43
    {"get", "/gym/runs/:id", "gym:read", false},
43 44
    {"get", "/api/v1/admin/forge/targets", "deployments:promote", false},
44 45
    {"get", "/api/v1/admin/forge/targets/:id", "deployments:promote", false},
45 46
    {"get", "/api/operator/artifact-listings/:id/export", "artifact-catalog:operate", false},

@@ -100,6 +101,8 @@ defmodule OpenAgentsWeb.OperatorSurfaceTest do

100 101
    OpenAgentsWeb.ForumBoardLive => "widens the board listing to private boards",
101 102
    OpenAgentsWeb.ForumTopicLive => "widens the topic read and gates closing and hiding",
102 103
    OpenAgentsWeb.GymLive => "rechecks the operator on mount and on every event",
104
    OpenAgentsWeb.GymRunLive =>
105
      "rechecks the operator on mount and on every event before streaming a linked transcript",
103 106
    OpenAgentsWeb.GymRunController =>
104 107
      "rechecks the operator on every request before recording or listing gym runs",
105 108
    OpenAgentsWeb.HomeLive => "marks the session operator for the home surface",

This page updates live while a promote is in flight · changelog