Build runbook dashboards and verification through the PostHog MCP

76346d253ed8 · AtlantisPleb · · parent 75cde356d7cf

Build runbook dashboards and verification through the PostHog MCP

Step 8 now creates dashboards, insights, text tiles, and the rollout
annotation through MCP tools instead of the web app, with each
dashboard mapped to its query tool. Step 9 verification checks run as
SQL against the events table, and the feature-flag follow-up notes MCP
manageability.

Deploy story

What this commit did to the running system — joined from the forge receipt chain, the part a commit page elsewhere cannot show.

Not deployed through the forge lane

No push, promotion, build, or deploy receipt references this commit (receipts are scanned over a bounded recent window). Changes shipped by full node replacement carry their proof in the release gate receipt instead.

Changed files

  • modified docs/2026-08-21-posthog-integration-runbook.md

Diff

1 file changed, +34 -13

docs/2026-08-21-posthog-integration-runbook.md modified +34 -13

@@ -81,7 +81,7 @@ Out of scope for this effort:

81 81
82 82
- Session replay and self-driving (explicitly skipped).
83 83
- Error tracking. Note that the Elixir package enables exception capture by default; step 2 disables it so scope stays controlled. Revisit later as its own decision.
84
- Feature flags, surveys, revenue analytics, warehouse sources. Feature flags are the most likely follow-up; the SDK support comes free once step 2 is done.
84
- Feature flags, surveys, revenue analytics, warehouse sources. Feature flags are the most likely follow-up; the SDK support comes free once step 2 is done, and flag creation, evaluation testing, and rollout are all manageable through the PostHog MCP (`create-feature-flag` and related tools) when that decision lands.
85 85
- Installing the PostHog MCP server into coding agents. Useful later, unrelated to instrumentation.
86 86
87 87
### Prerequisites

@@ -268,29 +268,49 @@ Deliberate omissions:

268 268
- API read endpoints (`GET`) stay uninstrumented except where they represent product activation. Volume without signal costs money per event.
269 269
- Never capture message bodies, objective text, token material, ciphertexts, or raw query strings. The `$current_url` property contains query parameters; rely on the wrapper's redaction and keep sensitive routes out of custom properties.
270 270
271
### Step 8: Build the starter dashboards
271
### Step 8: Build the starter dashboards through the PostHog MCP
272 272
273
Create these in the PostHog app after the first events land:
273
Build every dashboard from your coding agent through the PostHog MCP server instead of clicking in the web app. Dashboard, insight, and annotation objects are all writable through MCP tools, so this step becomes a scripted session you can rerun and review like code.
274 274
275
1. **Web overview**: visitors, pageviews, sessions, bounce rate, top pages, referrers, UTMs, devices.
276
2. **Activation funnel**: `$pageview` → `auth_started` → `user_signed_up` → `chat_message_sent`.
277
3. **Engagement**: weekly active chatters, median `chat_turn_completed` duration, delegated-work completion rate.
278
4. **Product adoption**: issues and projects created per week, pushes received.
279
5. **Volume sanity**: total events per day by name, to catch runaway capture early.
275
Connect the MCP server first if your client does not already have it. The wizard's `mcp add` command edits AI-client configuration only and works regardless of project language, so it is safe to use here even though the wizard's integration flow itself does not support Phoenix:
276
277
```console
278
npx @posthog/wizard mcp add
279
```
280
281
Then drive the whole step through MCP tools:
282
283
1. **Create the containers.** Call `dashboard-create` once per dashboard below, or browse `dashboard-templates-list` and create from a template with `use_template` when one matches. Add section headers with `dashboard-create-text-tile`.
284
2. **Test each chart before saving it.** Run the underlying query with the read-only query tools: `query-trends`, `query-funnel`, `query-stickiness`, `query-web-overview`, and `query-web-stats`. Confirm the series names and volumes look right while the data is still cheap to inspect.
285
3. **Save passing queries as insights.** Call `insight-create` with the tested query and pass the dashboard IDs in its `dashboards` field. Re-run later with `dashboard-insights-run` to confirm rendering.
286
4. **Mark the rollout.** Call `annotation-create` with the integration date so later trend breaks are attributable to the instrumentation itself.
287
288
The five dashboards, mapped to their queries:
289
290
| Dashboard | Tiles | Query source |
291
| --- | --- | --- |
292
| Web overview | visitors, pageviews, sessions, bounce rate, top pages, referrers, UTMs, devices | `query-web-overview` plus `query-web-stats` breakdowns |
293
| Activation funnel | `$pageview` → `auth_started` → `user_signed_up` → `chat_message_sent` | `query-funnel` |
294
| Engagement | weekly active chatters, `chat_turn_completed` duration, delegated-work completion rate | `query-stickiness` on `chat_message_sent`, `query-trends` with a median aggregate, `query-funnel` |
295
| Product adoption | issues and projects created per week, pushes received | `query-trends` broken down by event name |
296
| Volume sanity | total events per day by name | `query-trends`, total count, breakdown by event |
297
298
Keep the dashboard definitions in the agent session transcript or commit them as a script; because creation goes through MCP calls, recreating the set in staging or after a project reset is mechanical.
280 299
281 300
### Step 9: Verify
282 301
283
Work through this checklist in staging:
302
Work through this checklist in staging. Browser-side checks stay manual; every server-side or ingestion check runs through the PostHog MCP against the events table, which beats tailing the web UI:
284 303
285
1. Cold load of `/` produces a `$pageview`; navigating to `/docs` through a LiveView link produces a second one with the correct `$current_url`.
286
2. Autocapture shows click events in the live events tail.
287
3. Complete a GitHub OAuth login: `auth_started`, `user_signed_up` (or `user_signed_in`), and the identify merge are visible; the person shows both the anonymous pre-login events and the identified ones.
304
1. Cold load of `/` produces a `$pageview`; navigating to `/docs` through a LiveView link produces a second one with the correct `$current_url`. Confirm in the browser and then with an `execute-sql` query filtering `event = '$pageview'` ordered by timestamp.
305
2. Autocapture lands click events. Query the last hour of events where `event LIKE '$autocapture%'`.
306
3. Complete a GitHub OAuth login: `auth_started`, `user_signed_up` (or `user_signed_in`) arrive, and the identified person shows both the anonymous pre-login events and the identified ones. Check person merging with a persons query on the distinct ID.
288 307
4. Send a chat message: `chat_message_sent` on the client and `chat_turn_completed` on the server attach to the same person.
289 308
5. Create an issue through the web UI and through the JSON API: both produce `issue_created` with a `surface` property distinguishing them.
290
6. Search captured event properties for token-shaped strings and message content; find nothing.
309
6. Search captured event properties for token-shaped strings and message content; find nothing. An `execute-sql` scan over recent event properties catches this faster than the UI.
291 310
7. Stop the app with `OPENAGENTS_POSTHOG_PROJECT_TOKEN` unset: boot succeeds and no network calls go to PostHog.
292 311
8. Run the test suite: `test_mode` drops events and no test asserts on outbound PostHog traffic.
293 312
9. Check the browser console for CSP violations against the ingest host; fix `connect-src` if any appear.
313
10. Confirm each taxonomy event registered as an event definition with `read-data-schema`, so typos surface as missing definitions rather than silent zero-volume charts.
294 314
295 315
### Rollout order
296 316

@@ -307,3 +327,4 @@ Work through this checklist in staging:

307 327
- [Phoenix guide](https://posthog.com/docs/libraries/phoenix)
308 328
- [JavaScript Web SDK](https://posthog.com/docs/libraries/js)
309 329
- [Identifying users](https://posthog.com/docs/getting-started/identify-users)
330
- [PostHog MCP server](https://posthog.com/docs/model-context-protocol)

This page updates live while a promote is in flight · changelog