Add the operator PostHog analytics surface

73ff25077ddf · AtlantisPleb · · parent fa4b79283164

Add the operator PostHog analytics surface

/admin/analytics pulls computed results from the PostHog REST API at
request time through a new OpenAgents.PostHog client (personal API key
over HogQL) so operators see trailing-24-hour aggregates without
opening PostHog and without a second aggregation authority.

The page renders activation funnel, chat turn outcomes and durations,
event volume, and top pages, and treats its degraded states as first
class: unconfigured credentials, an unanswered query with retry and no
stale numbers, and loading. The route is operator-gated and classified
in the route authority inventory as analytics:read.

Changelog: Operators can now read trailing-24-hour PostHog analytics at /admin/analytics when read credentials are configured.

Changelog-Category: feature
Changelog
Operators can now read trailing-24-hour PostHog analytics at /admin/analytics when read credentials are configured.
Changelog-Category
feature

Deploy story

What this commit did to the running system — joined from the forge receipt chain, the part a commit page elsewhere cannot show.

Not deployed through the forge lane

No push, promotion, build, or deploy receipt references this commit (receipts are scanned over a bounded recent window). Changes shipped by full node replacement carry their proof in the release gate receipt instead.

Changed files

  • modified config/config.exs
  • modified config/runtime.exs
  • modified docs/2026-08-21-posthog-integration-runbook.md
  • added lib/openagents/posthog.ex
  • added lib/openagents_web/live/admin_analytics_live.ex
  • modified lib/openagents_web/route_authority.ex
  • modified lib/openagents_web/router.ex
  • added test/openagents/posthog_test.exs
  • added test/openagents_web/live/admin_analytics_live_test.exs

Diff

9 files changed, +789 -0

config/config.exs modified +5

@@ -51,6 +51,11 @@ config :openagents,

51 51
  coding_jobs_dir: "/var/lib/openagents/coding-jobs",
52 52
  posthog_project_token: nil,
53 53
  posthog_api_host: "https://us.i.posthog.com",
54
  posthog_analytics: [
55
    personal_api_key: nil,
56
    project_id: nil,
57
    app_host: "https://us.posthog.com"
58
  ],
54 59
  work_workers_enabled: false,
55 60
  work: [enabled: false],
56 61
  scv_codex: [
config/runtime.exs modified +10

@@ -442,6 +442,16 @@ if runtime_role == :web do

442 442
    config :posthog, api_host: posthog_api_host
443 443
  end
444 444
445
  # The operator analytics surface reads computed results back from the
446
  # PostHog REST API. Absent credentials disable the read path only; capture
447
  # above is independent of it.
448
  config :openagents, :posthog_analytics,
449
    personal_api_key: optional_text.("OPENAGENTS_POSTHOG_PERSONAL_API_KEY"),
450
    project_id: optional_text.("OPENAGENTS_POSTHOG_PROJECT_ID"),
451
    app_host:
452
      optional_text.("OPENAGENTS_POSTHOG_APP_HOST") ||
453
        Application.get_env(:openagents, :posthog_analytics)[:app_host]
454
445 455
  github_oauth = Application.get_env(:openagents, :github_oauth, [])
446 456
447 457
  github_oauth =
docs/2026-08-21-posthog-integration-runbook.md modified +14

@@ -296,6 +296,20 @@ Work through this checklist in staging. Browser-side checks stay manual; every s

296 296
9. Check the browser console for CSP violations against the ingest host; fix `connect-src` if any appear.
297 297
10. Confirm each taxonomy event registered as an event definition with `read-data-schema`, so typos surface as missing definitions rather than silent zero-volume charts.
298 298
299
## Operator analytics surface
300
301
`/admin/analytics` gives operators the trailing-24-hour picture without opening PostHog. It pulls computed results from the PostHog REST API at request time through `OpenAgents.PostHog` (a personal API key over HogQL), so it adds no second aggregation authority: the numbers match what the PostHog app answers for the same window.
302
303
Settings (all optional; absent credentials disable the read path only, independently of capture):
304
305
| Setting | Requirement |
306
| --- | --- |
307
| `OPENAGENTS_POSTHOG_PERSONAL_API_KEY` | A `phx_...` personal API key created for this integration |
308
| `OPENAGENTS_POSTHOG_PROJECT_ID` | The numeric PostHog project id |
309
| `OPENAGENTS_POSTHOG_APP_HOST` | Optional; defaults to `https://us.posthog.com`. This is the app/API host, not the ingest host |
310
311
The page renders four bounded projections — activation funnel, chat turn outcomes and durations, event volume, top pages — plus its non-happy states as first-class UI: unconfigured credentials, an unanswered query (with retry, never stale numbers), and loading. It shows aggregates only; no conversation content is reachable from it. The route is operator-gated like the rest of `/admin` and classified in the route authority inventory as `analytics:read`.
312
299 313
### Rollout order
300 314
301 315
1. Steps 1-6 are implemented and covered by the test suite; they activate the moment a project token is configured.
lib/openagents/posthog.ex added +220

@@ -0,0 +1,220 @@

1
defmodule OpenAgents.PostHog do
2
  @moduledoc """
3
  Server-side read access to the project's own PostHog analytics.
4
5
  The operator analytics surface (`/admin/analytics`) pulls computed results
6
  from the PostHog REST API with a personal API key. This module is that
7
  boundary: it owns the HogQL for each bounded question, shapes the rows into
8
  plain maps, and never raises — a failed or unconfigured integration is an
9
  ordinary `{:error, reason}`, which the surface renders as a degraded state.
10
11
  Configuration (see `config/runtime.exs`):
12
13
  - `OPENAGENTS_POSTHOG_PERSONAL_API_KEY` — a `phx_...` personal API key.
14
    Absent means disabled: no request ever leaves, and the surface says so.
15
  - `OPENAGENTS_POSTHOG_PROJECT_ID` — the numeric PostHog project id.
16
  - `OPENAGENTS_POSTHOG_APP_HOST` — optional; defaults to
17
    `https://us.posthog.com`. This is the app/API host, not the event ingest
18
    host used by capture.
19
20
  The key is read-only analytics material. It grants whatever scopes the key
21
  was created with and must not be logged or echoed into errors.
22
  """
23
24
  require Logger
25
26
  @receive_timeout_ms 8_000
27
28
  @type shaped :: map()
29
30
  @doc """
31
  Whether the read path has both credentials configured. Capture being
32
  enabled does not imply this: the personal API key is a separate setting.
33
  """
34
  @spec enabled?() :: boolean()
35
  def enabled? do
36
    present?(settings()[:personal_api_key]) and integer_id?(settings()[:project_id])
37
  end
38
39
  @doc """
40
  Everything the operator analytics page shows, for the trailing 24 hours.
41
42
  Returns `{:ok, shaped}` with four bounded projections, or
43
  `{:error, :not_configured | :unavailable}`. Each projection runs as its own
44
  HogQL query; a failure of any one fails the whole pull, because partial
45
  numbers presented next to each other read as complete.
46
  """
47
  @spec overview() :: {:ok, shaped()} | {:error, :not_configured | :unavailable}
48
  def overview do
49
    if enabled?() do
50
      with {:ok, events} <- run(event_counts_sql(), "event_counts"),
51
           {:ok, funnel} <- run(funnel_sql(), "funnel"),
52
           {:ok, chat} <- run(chat_turns_sql(), "chat_turns"),
53
           {:ok, pages} <- run(top_pages_sql(), "top_pages") do
54
        {:ok,
55
         %{
56
           generated_at: DateTime.utc_now(),
57
           event_counts: shape_rows(events),
58
           funnel: shape_rows(funnel) |> List.first(%{}),
59
           chat_turns: shape_chat_turns(shape_rows(chat) |> List.first(%{})),
60
           top_pages: shape_rows(pages)
61
         }}
62
      end
63
    else
64
      {:error, :not_configured}
65
    end
66
  end
67
68
  # ── questions ────────────────────────────────────────────────────────────
69
70
  defp event_counts_sql do
71
    """
72
    SELECT event, count() AS count, uniq(person_id) AS people
73
    FROM events
74
    WHERE timestamp >= now() - INTERVAL 1 DAY
75
    GROUP BY event
76
    ORDER BY count DESC
77
    LIMIT 25
78
    """
79
    |> squash()
80
  end
81
82
  defp funnel_sql do
83
    """
84
    SELECT
85
      countIf(event = 'auth_started') AS auth_started,
86
      countIf(event = 'user_signed_up') AS user_signed_up,
87
      countIf(event = 'user_signed_in') AS user_signed_in,
88
      uniqIf(person_id, event = 'user_signed_up') AS identified_signups,
89
      countIf(event = 'chat_message_sent') AS chat_message_sent
90
    FROM events
91
    WHERE timestamp >= now() - INTERVAL 1 DAY
92
    """
93
    |> squash()
94
  end
95
96
  defp chat_turns_sql do
97
    """
98
    SELECT
99
      count() AS turns,
100
      countIf(properties.outcome = 'completed') AS completed,
101
      countIf(properties.outcome = 'failed') AS failed,
102
      countIf(properties.outcome = 'cancelled') AS cancelled,
103
      round(avg(properties.duration_ms)) AS avg_duration_ms,
104
      max(properties.duration_ms) AS max_duration_ms
105
    FROM events
106
    WHERE timestamp >= now() - INTERVAL 1 DAY AND event = 'chat_turn_completed'
107
    """
108
    |> squash()
109
  end
110
111
  defp top_pages_sql do
112
    """
113
    SELECT properties.$current_url AS url, count() AS views
114
    FROM events
115
    WHERE timestamp >= now() - INTERVAL 1 DAY AND event = '$pageview'
116
    GROUP BY properties.$current_url
117
    ORDER BY views DESC
118
    LIMIT 8
119
    """
120
    |> squash()
121
  end
122
123
  # ── shaping ──────────────────────────────────────────────────────────────
124
125
  defp shape_rows(%{"results" => results, "columns" => columns}) when is_list(results) do
126
    keys = Enum.map(columns, &to_string/1)
127
128
    Enum.map(results, fn row ->
129
      keys |> Enum.zip(List.wrap(row)) |> Map.new(fn {k, v} -> {k, scalar(v)} end)
130
    end)
131
  end
132
133
  defp shape_rows(_unexpected), do: []
134
135
  defp shape_chat_turns(row) do
136
    %{
137
      "turns" => count_value(row["turns"]),
138
      "completed" => count_value(row["completed"]),
139
      "failed" => count_value(row["failed"]),
140
      "cancelled" => count_value(row["cancelled"]),
141
      "avg_duration_ms" => duration_value(row["avg_duration_ms"]),
142
      "max_duration_ms" => duration_value(row["max_duration_ms"])
143
    }
144
  end
145
146
  defp scalar(value) when is_integer(value) or is_float(value), do: value
147
  defp scalar(value) when is_binary(value), do: value
148
  defp scalar(nil), do: nil
149
  defp scalar(value), do: to_string(value)
150
151
  # PostHog aggregates arrive as integers or floats depending on the
152
  # expression; counts round down and durations keep one decimal of sense by
153
  # staying numeric.
154
  defp count_value(value) when is_integer(value), do: value
155
  defp count_value(value) when is_float(value), do: trunc(value)
156
  defp count_value(_other), do: 0
157
158
  defp duration_value(value) when is_integer(value), do: value
159
  defp duration_value(value) when is_float(value), do: round(value)
160
  defp duration_value(_other), do: nil
161
162
  # ── transport ────────────────────────────────────────────────────────────
163
164
  defp run(sql, label) do
165
    settings = settings()
166
    url = "#{settings[:app_host]}/api/projects/#{settings[:project_id]}/query/"
167
168
    request_options =
169
      [
170
        json: %{query: %{kind: "HogQLQuery", query: sql}},
171
        auth: {:bearer, settings[:personal_api_key]},
172
        headers: [{"user-agent", "openagents-admin-analytics"}],
173
        receive_timeout: @receive_timeout_ms,
174
        retry: false
175
      ]
176
      |> Keyword.merge(settings[:request_options] || [])
177
178
    case Req.post(url, request_options) do
179
      {:ok, %Req.Response{status: 200, body: %{"results" => _} = body}} ->
180
        {:ok, body}
181
182
      {:ok, %Req.Response{status: status}} when status in [401, 403] ->
183
        Logger.warning("posthog_query_failed label=#{label} code=posthog_key_rejected")
184
        {:error, :unavailable}
185
186
      {:ok, %Req.Response{status: status}} ->
187
        Logger.warning("posthog_query_failed label=#{label} code=posthog_status_#{status}")
188
        {:error, :unavailable}
189
190
      {:error, _transport_error} ->
191
        Logger.warning("posthog_query_failed label=#{label} code=posthog_unreachable")
192
        {:error, :unavailable}
193
    end
194
  rescue
195
    error ->
196
      Logger.warning(
197
        "posthog_query_failed label=#{label} code=#{OpenAgents.OperationalLog.code(error)}"
198
      )
199
200
      {:error, :unavailable}
201
  end
202
203
  defp settings, do: Application.get_env(:openagents, :posthog_analytics, [])
204
205
  defp present?(value) when is_binary(value), do: String.trim(value) != ""
206
  defp present?(_value), do: false
207
208
  defp integer_id?(value) when is_integer(value), do: value > 0
209
210
  defp integer_id?(value) when is_binary(value) do
211
    case Integer.parse(String.trim(value)) do
212
      {id, ""} -> id > 0
213
      _invalid -> false
214
    end
215
  end
216
217
  defp integer_id?(_value), do: false
218
219
  defp squash(sql), do: sql |> String.split("\n") |> Enum.map(&String.trim/1) |> Enum.join(" ")
220
end
lib/openagents_web/live/admin_analytics_live.ex added +217

@@ -0,0 +1,217 @@

1
defmodule OpenAgentsWeb.AdminAnalyticsLive do
2
  @moduledoc """
3
  Operator view of product analytics, pulled server-side from PostHog.
4
5
  Every number here is computed by PostHog at request time through
6
  `OpenAgents.PostHog`, so the surface adds no second aggregation authority:
7
  what an operator sees matches what the PostHog app would answer for the same
8
  window. Read-only, operator-gated, and honest about its three non-happy
9
  states — unconfigured credentials, a failed pull, and loading.
10
11
  It shows operational facts about accounts as aggregates. It never renders
12
  conversation content, memory claims, or anything a person wrote.
13
  """
14
15
  use OpenAgentsWeb, :live_view
16
17
  alias OpenAgents.Accounts
18
  alias OpenAgents.PostHog
19
20
  @impl true
21
  def mount(_params, _session, socket) do
22
    if Accounts.admin?(socket.assigns.current_user) do
23
      socket =
24
        socket
25
        |> assign(:page_title, "Operator · Analytics")
26
        |> assign(:status, :loading)
27
        |> assign(:overview, nil)
28
29
      if connected?(socket) do
30
        send(self(), :load)
31
      end
32
33
      {:ok, socket}
34
    else
35
      {:ok, redirect(socket, to: ~p"/")}
36
    end
37
  end
38
39
  @impl true
40
  def handle_event("refresh", _params, socket) do
41
    # Re-checked per event, not only at mount: a long-lived socket outlives the
42
    # decision that opened it.
43
    if Accounts.admin?(socket.assigns.current_user) do
44
      {:noreply,
45
       socket
46
       |> assign(:status, :loading)
47
       |> load()}
48
    else
49
      {:noreply, redirect(socket, to: ~p"/")}
50
    end
51
  end
52
53
  @impl true
54
  def handle_info(:load, socket), do: {:noreply, load(socket)}
55
56
  defp load(%{assigns: %{status: :loading}} = socket) do
57
    case PostHog.overview() do
58
      {:ok, overview} ->
59
        assign(socket, status: :loaded, overview: overview)
60
61
      {:error, reason} when reason in [:not_configured, :unavailable] ->
62
        assign(socket, status: reason, overview: nil)
63
    end
64
  end
65
66
  # A refresh while a load is already resolving must not clobber the newer
67
  # state with an older response.
68
  defp load(socket), do: socket
69
70
  @impl true
71
  def render(assigns) do
72
    ~H"""
73
    <Layouts.app
74
      flash={@flash}
75
      sidebar_sections={assigns[:sidebar_sections]}
76
      current_scope={@current_scope}
77
      title="PostHog analytics"
78
    >
79
      <main id="admin-analytics-page" class="app-shell admin-shell">
80
        <section class="admin space-y-8" aria-labelledby="analytics-heading">
81
          <header class="admin-heading">
82
            <h1 id="analytics-heading">Product analytics</h1>
83
            <p>
84
              Computed live from PostHog over the trailing twenty-four hours. These are
85
              aggregate operational facts about usage; no conversation or memory content
86
              is readable from this page.
87
            </p>
88
            <div class="admin-totals">
89
              <.badge variant={:info}>TRAILING 24 HOURS</.badge>
90
              <.badge
91
                :if={@status == :loaded && @overview}
92
                variant={:dim}
93
                id="analytics-generated-at"
94
              >
95
                GENERATED {Calendar.strftime(@overview.generated_at, "%Y-%m-%d %H:%M UTC")}
96
              </.badge>
97
              <.text_button id="analytics-refresh" phx-click="refresh" disabled={@status == :loading}>
98
                {if(@status == :loading, do: "REFRESHING…", else: "REFRESH")}
99
              </.text_button>
100
            </div>
101
          </header>
102
103
          <.alert
104
            :if={@status == :not_configured}
105
            id="analytics-not-configured"
106
            appearance={:notice}
107
            variant={:warning}
108
          >
109
            This deployment has no PostHog read credentials configured. Set
110
            <.kbd>OPENAGENTS_POSTHOG_PERSONAL_API_KEY</.kbd>
111
            and
112
            <.kbd>OPENAGENTS_POSTHOG_PROJECT_ID</.kbd>
113
            to enable this surface.
114
          </.alert>
115
116
          <.alert
117
            :if={@status == :unavailable}
118
            id="analytics-unavailable"
119
            appearance={:notice}
120
            variant={:danger}
121
          >
122
            <div class="space-y-3">
123
              <p>PostHog did not answer the last query. Nothing on this page is stale data.</p>
124
              <.button id="analytics-retry" variant={:secondary} phx-click="refresh">
125
                TRY AGAIN
126
              </.button>
127
            </div>
128
          </.alert>
129
130
          <.alert :if={@status == :loading} id="analytics-loading" appearance={:row}>
131
            Querying PostHog for the trailing twenty-four hours…
132
          </.alert>
133
134
          <div :if={@status == :loaded && @overview} class="space-y-8">
135
            <section aria-labelledby="funnel-heading">
136
              <.card id="analytics-funnel">
137
                <h2 id="funnel-heading" class="card-title">Activation funnel</h2>
138
                <ul class="divide-y divide-border">
139
                  <li class="flex items-center justify-between gap-4 py-3">
140
                    <span>GitHub authorization started</span>
141
                    <span class="font-semibold">{@overview.funnel["auth_started"]}</span>
142
                  </li>
143
                  <li class="flex items-center justify-between gap-4 py-3">
144
                    <span>Accounts created</span>
145
                    <span class="font-semibold">{@overview.funnel["user_signed_up"]}</span>
146
                  </li>
147
                  <li class="flex items-center justify-between gap-4 py-3">
148
                    <span>Returning sign-ins</span>
149
                    <span class="font-semibold">{@overview.funnel["user_signed_in"]}</span>
150
                  </li>
151
                  <li class="flex items-center justify-between gap-4 py-3">
152
                    <span>First chat messages sent</span>
153
                    <span class="font-semibold">{@overview.funnel["chat_message_sent"]}</span>
154
                  </li>
155
                </ul>
156
              </.card>
157
            </section>
158
159
            <section aria-labelledby="turns-heading">
160
              <.card id="analytics-chat-turns">
161
                <h2 id="turns-heading" class="card-title">Chat turns</h2>
162
                <% turns = @overview.chat_turns %>
163
                <div class="flex flex-wrap items-center gap-3 pb-3">
164
                  <.badge>{turns["turns"]} TURNS</.badge>
165
                  <.badge :if={turns["completed"] > 0} variant={:success}>
166
                    {turns["completed"]} COMPLETED
167
                  </.badge>
168
                  <.badge :if={turns["failed"] > 0} variant={:danger}>
169
                    {turns["failed"]} FAILED
170
                  </.badge>
171
                  <.badge :if={turns["cancelled"] > 0} variant={:warning}>
172
                    {turns["cancelled"]} CANCELLED
173
                  </.badge>
174
                </div>
175
                <p class="text-muted-foreground">
176
                  Average turn {format_duration(turns["avg_duration_ms"])}; longest {format_duration(
177
                    turns["max_duration_ms"]
178
                  )}.
179
                </p>
180
              </.card>
181
            </section>
182
183
            <section aria-labelledby="events-heading">
184
              <.card id="analytics-event-volume">
185
                <h2 id="events-heading" class="card-title">Event volume</h2>
186
                <.table id="analytics-events-table" rows={@overview.event_counts}>
187
                  <:col :let={row} label="Event">{row["event"]}</:col>
188
                  <:col :let={row} label="Count">{row["count"]}</:col>
189
                  <:col :let={row} label="People">{row["people"]}</:col>
190
                </.table>
191
              </.card>
192
            </section>
193
194
            <section aria-labelledby="pages-heading">
195
              <.card id="analytics-top-pages">
196
                <h2 id="pages-heading" class="card-title">Top pages</h2>
197
                <.table id="analytics-pages-table" rows={@overview.top_pages}>
198
                  <:col :let={row} label="URL">{row["url"]}</:col>
199
                  <:col :let={row} label="Views">{row["views"]}</:col>
200
                </.table>
201
              </.card>
202
            </section>
203
          </div>
204
        </section>
205
      </main>
206
    </Layouts.app>
207
    """
208
  end
209
210
  defp format_duration(nil), do: "n/a"
211
212
  defp format_duration(ms) when is_number(ms) and ms >= 1_000 do
213
    "#{:erlang.float_to_binary(ms / 1_000, decimals: 1)}s"
214
  end
215
216
  defp format_duration(ms) when is_integer(ms), do: "#{ms}ms"
217
end
lib/openagents_web/route_authority.ex modified +3

@@ -112,6 +112,9 @@ defmodule OpenAgentsWeb.RouteAuthority do

112 112
  defp policy(%{path: "/logout"}),
113 113
    do: declaration(:authenticated_browser, "encrypted browser session", "session:delete", true)
114 114
115
  defp policy(%{path: "/admin/analytics"}),
116
    do: declaration(:operator, "configured operator GitHub ID", "analytics:read", false)
117
115 118
  defp policy(%{path: "/admin/forge"}),
116 119
    do: declaration(:operator, "configured operator GitHub ID", "forge:promote", true)
117 120
lib/openagents_web/router.ex modified +1

@@ -180,6 +180,7 @@ defmodule OpenAgentsWeb.Router do

180 180
        {OpenAgentsWeb.UserAuth, :ensure_admin}
181 181
      ] do
182 182
      live "/", AdminLive, :index
183
      live "/analytics", AdminAnalyticsLive, :index
183 184
      live "/forge", AdminForgeLive, :index
184 185
      live "/recordings", AdminRecordingsLive, :index
185 186
      live "/scv/accounts", AdminScvAccountsLive, :index
test/openagents/posthog_test.exs added +152

@@ -0,0 +1,152 @@

1
defmodule OpenAgents.PostHogTest do
2
  use ExUnit.Case, async: false
3
4
  alias OpenAgents.PostHog
5
6
  setup do
7
    original = Application.get_env(:openagents, :posthog_analytics)
8
9
    on_exit(fn ->
10
      if original == nil,
11
        do: Application.delete_env(:openagents, :posthog_analytics),
12
        else: Application.put_env(:openagents, :posthog_analytics, original)
13
    end)
14
15
    :ok
16
  end
17
18
  defp configure(overrides \\ []) do
19
    Application.put_env(
20
      :openagents,
21
      :posthog_analytics,
22
      Keyword.merge(
23
        [
24
          personal_api_key: "phx_test_key",
25
          project_id: 303_178,
26
          app_host: "https://posthog-api.internal",
27
          request_options: [plug: {Req.Test, __MODULE__}]
28
        ],
29
        overrides
30
      )
31
    )
32
  end
33
34
  describe "enabled?/0" do
35
    test "requires both a key and a positive numeric project id" do
36
      refute PostHog.enabled?()
37
38
      configure(personal_api_key: nil)
39
      refute PostHog.enabled?()
40
41
      configure(project_id: "0")
42
      refute PostHog.enabled?()
43
44
      configure(project_id: 303_178)
45
      assert PostHog.enabled?()
46
47
      configure(personal_api_key: "   ")
48
      refute PostHog.enabled?()
49
    end
50
  end
51
52
  describe "overview/0" do
53
    test "is not configured without credentials and sends nothing" do
54
      assert {:error, :not_configured} = PostHog.overview()
55
    end
56
57
    test "shapes the four projections from one pull" do
58
      configure()
59
60
      Req.Test.expect(__MODULE__, fn conn ->
61
        {:ok, body, conn} = Plug.Conn.read_body(conn)
62
        assert body =~ "uniq(person_id)"
63
        assert ["Bearer phx_test_key"] = Plug.Conn.get_req_header(conn, "authorization")
64
        assert conn.request_path == "/api/projects/303178/query/"
65
66
        Req.Test.json(conn, %{
67
          "columns" => ["event", "count", "people"],
68
          "results" => [["$pageview", 118, 8], ["chat_opened", 17, 3]]
69
        })
70
      end)
71
72
      Req.Test.expect(__MODULE__, fn conn ->
73
        {:ok, body, conn} = Plug.Conn.read_body(conn)
74
        assert body =~ "'auth_started'"
75
76
        Req.Test.json(conn, %{
77
          "columns" => [
78
            "auth_started",
79
            "user_signed_up",
80
            "user_signed_in",
81
            "identified_signups",
82
            "chat_message_sent"
83
          ],
84
          "results" => [[3, 1, 2, 1, 6]]
85
        })
86
      end)
87
88
      Req.Test.expect(__MODULE__, fn conn ->
89
        {:ok, body, conn} = Plug.Conn.read_body(conn)
90
        assert body =~ "'chat_turn_completed'"
91
92
        Req.Test.json(conn, %{
93
          "columns" => [
94
            "turns",
95
            "completed",
96
            "failed",
97
            "cancelled",
98
            "avg_duration_ms",
99
            "max_duration_ms"
100
          ],
101
          "results" => [[6, 6, 0, 0, 4525.0, 7414]]
102
        })
103
      end)
104
105
      Req.Test.expect(__MODULE__, fn conn ->
106
        {:ok, body, conn} = Plug.Conn.read_body(conn)
107
        assert body =~ "'$pageview'"
108
109
        Req.Test.json(conn, %{
110
          "columns" => ["url", "views"],
111
          "results" => [["https://openagents.com/", 20], [nil, 4]]
112
        })
113
      end)
114
115
      assert {:ok, overview} = PostHog.overview()
116
      assert %DateTime{} = overview.generated_at
117
118
      assert [%{"event" => "$pageview", "count" => 118, "people" => 8}] =
119
               Enum.slice(overview.event_counts, 0, 1)
120
121
      assert overview.funnel["user_signed_up"] == 1
122
      assert overview.funnel["chat_message_sent"] == 6
123
124
      assert overview.chat_turns == %{
125
               "turns" => 6,
126
               "completed" => 6,
127
               "failed" => 0,
128
               "cancelled" => 0,
129
               "avg_duration_ms" => 4525,
130
               "max_duration_ms" => 7414
131
             }
132
133
      assert [%{"url" => "https://openagents.com/", "views" => 20} | _] = overview.top_pages
134
    end
135
136
    test "a rejected key is unavailable, never raised" do
137
      configure()
138
139
      Req.Test.expect(__MODULE__, fn conn ->
140
        Plug.Conn.send_resp(conn, 401, "unauthorized")
141
      end)
142
143
      assert {:error, :unavailable} = PostHog.overview()
144
    end
145
146
    test "an unreachable host is unavailable, never raised" do
147
      configure(request_options: [])
148
149
      assert {:error, :unavailable} = PostHog.overview()
150
    end
151
  end
152
end
test/openagents_web/live/admin_analytics_live_test.exs added +167

@@ -0,0 +1,167 @@

1
defmodule OpenAgentsWeb.AdminAnalyticsLiveTest do
2
  @moduledoc """
3
  `/admin/analytics` gates like every operator surface, and its data path is
4
  honest about the states it can be in: unconfigured credentials, a PostHog
5
  that did not answer, and loaded aggregates.
6
7
  The assertions hold the same line as `/admin`: aggregate operational facts
8
  only. Nothing rendered here names conversation or memory content.
9
  """
10
11
  use OpenAgentsWeb.ConnCase, async: false
12
  import Phoenix.LiveViewTest
13
14
  setup do
15
    original = Application.get_env(:openagents, :posthog_analytics)
16
17
    on_exit(fn ->
18
      if original == nil,
19
        do: Application.delete_env(:openagents, :posthog_analytics),
20
        else: Application.put_env(:openagents, :posthog_analytics, original)
21
    end)
22
23
    :ok
24
  end
25
26
  describe "access" do
27
    test "the operator reaches the surface", %{conn: conn} do
28
      conn = log_in_admin_user(conn, "analytics-operator")
29
30
      {:ok, _view, html} = live(conn, ~p"/admin/analytics")
31
32
      assert html =~ "Product analytics"
33
    end
34
35
    test "an ordinary authenticated account is redirected and told nothing", %{conn: conn} do
36
      conn = log_in_github_user(conn, "analytics-ordinary")
37
38
      assert {:error, {:redirect, %{to: "/"}}} = live(conn, ~p"/admin/analytics")
39
40
      response = get(conn, ~p"/admin/analytics")
41
      assert redirected_to(response) == ~p"/"
42
    end
43
44
    test "an unauthenticated visitor is redirected", %{conn: conn} do
45
      assert {:error, {:redirect, %{to: "/"}}} = live(conn, ~p"/admin/analytics")
46
    end
47
  end
48
49
  describe "states" do
50
    test "without credentials the page says so instead of showing stale numbers",
51
         %{conn: conn} do
52
      Application.put_env(:openagents, :posthog_analytics,
53
        personal_api_key: nil,
54
        project_id: nil
55
      )
56
57
      conn = log_in_admin_user(conn, "analytics-unconfigured")
58
59
      {:ok, view, _html} = live(conn, ~p"/admin/analytics")
60
      render_async(view)
61
62
      assert has_element?(view, "#analytics-not-configured")
63
      refute has_element?(view, "#analytics-event-volume")
64
    end
65
66
    test "a failed pull offers retry and renders no numbers", %{conn: conn} do
67
      configure_posthog(fn conn ->
68
        Plug.Conn.send_resp(conn, 500, "boom")
69
      end)
70
71
      conn = log_in_admin_user(conn, "analytics-failed")
72
73
      {:ok, view, _html} = live(conn, ~p"/admin/analytics")
74
      render_async(view)
75
76
      assert has_element?(view, "#analytics-unavailable")
77
      refute has_element?(view, "#analytics-event-volume")
78
    end
79
80
    test "a successful pull renders bounded aggregates and refreshes", %{conn: conn} do
81
      Application.put_env(:openagents, :posthog_analytics,
82
        personal_api_key: "phx_test_key",
83
        project_id: 303_178,
84
        app_host: "https://posthog-api.internal",
85
        request_options: [plug: {Req.Test, __MODULE__}]
86
      )
87
88
      # One pull is four questions.
89
      Req.Test.expect(__MODULE__, 4, fn conn -> respond_by_query(conn) end)
90
91
      conn = log_in_admin_user(conn, "analytics-loaded")
92
93
      {:ok, view, _html} = live(conn, ~p"/admin/analytics")
94
      render_async(view)
95
96
      assert has_element?(view, "#analytics-generated-at")
97
      assert has_element?(view, "#analytics-funnel")
98
      assert has_element?(view, "#analytics-chat-turns")
99
      assert html = render(view)
100
      assert html =~ "$pageview"
101
      assert html =~ "https://openagents.com/"
102
103
      # A second full pull backs the refresh click.
104
      Req.Test.expect(__MODULE__, 4, fn conn -> respond_by_query(conn) end)
105
106
      assert view |> element("#analytics-refresh") |> render_click() =~ "TRAILING 24 HOURS"
107
      render_async(view)
108
      assert has_element?(view, "#analytics-generated-at")
109
    end
110
  end
111
112
  defp configure_posthog(handler) when is_function(handler, 1) do
113
    Application.put_env(:openagents, :posthog_analytics,
114
      personal_api_key: "phx_test_key",
115
      project_id: 303_178,
116
      app_host: "https://posthog-api.internal",
117
      request_options: [plug: {Req.Test, __MODULE__}]
118
    )
119
120
    Req.Test.expect(__MODULE__, handler)
121
  end
122
123
  # The client asks four questions in a fixed order; each stub answers by
124
  # matching the HogQL in the request body rather than relying on call order.
125
  defp respond_by_query(conn) do
126
    {:ok, body, conn} = Plug.Conn.read_body(conn)
127
128
    cond do
129
      body =~ "uniq(person_id)" ->
130
        Req.Test.json(conn, %{
131
          "columns" => ["event", "count", "people"],
132
          "results" => [["$pageview", 118, 8], ["chat_opened", 17, 3]]
133
        })
134
135
      body =~ "'auth_started'" ->
136
        Req.Test.json(conn, %{
137
          "columns" => [
138
            "auth_started",
139
            "user_signed_up",
140
            "user_signed_in",
141
            "identified_signups",
142
            "chat_message_sent"
143
          ],
144
          "results" => [[3, 1, 2, 1, 6]]
145
        })
146
147
      body =~ "'chat_turn_completed'" ->
148
        Req.Test.json(conn, %{
149
          "columns" => [
150
            "turns",
151
            "completed",
152
            "failed",
153
            "cancelled",
154
            "avg_duration_ms",
155
            "max_duration_ms"
156
          ],
157
          "results" => [[6, 6, 0, 0, 4525.0, 7414]]
158
        })
159
160
      true ->
161
        Req.Test.json(conn, %{
162
          "columns" => ["url", "views"],
163
          "results" => [["https://openagents.com/", 20]]
164
        })
165
    end
166
  end
167
end

This page updates live while a promote is in flight · changelog