Track full chat lifecycle and token usage in PostHog #102

Closed AtlantisPleb opened this 22h ago 1 comment

Outcome

The /chat surface reports every meaningful step of a conversation through PostHog, including assistant responses, token consumption, model/provider selection, queued messages, and failures.

Current behavior

lib/openagents_web/live/chat_live.ex already emits two events:

  • chat_message_sent when a user message starts a turn.
  • chat_turn_completed when a turn finishes.

These do not include the assistant message, token counts, model or provider, message queuing, streaming progress, or non-terminal errors.

Work

Audit lib/openagents_web/live/chat_live.ex and lib/openagents/chat/open_router.ex and emit the missing server-side PostHog events through OpenAgents.Analytics.capture/3:

  • chat_message_received — the assistant message was streamed or returned to the user.
  • chat_tokens_used — a turn consumed tokens; include input_tokens, output_tokens, model, provider, conversation_id, and turn_id.
  • chat_message_queued — a user message was queued while another turn was active.
  • chat_turn_failed — a turn ended in an error or non-completed status, with reason.
  • chat_stream_chunk — one assistant stream chunk arrived (rate-limited to avoid event spam, e.g. one event per second or per n chunks).
  • chat_tool_called — the assistant invoked a tool, with tool_name and turn_id.
  • chat_voice_started and chat_voice_ended — a voice session began or completed, with duration_ms.

OpenRouter token capture

Read lib/openagents/chat/open_router.ex and lib/openagents/chat/open_router/responses_stream_decoder.ex. When OpenRouter returns usage in a response or stream, capture chat_tokens_used once per turn, not per chunk. Do not duplicate counts across fallback paths.

Analytics queries

Update lib/openagents/posthog.ex so the operator analytics page shows:

  • total chat_tokens_used per model and provider in the last day.
  • total chat_message_received count.
  • chat_turn_failed rate.

Acceptance criteria

  • Every user message and assistant response is captured.
  • Token counts match the usage returned by OpenRouter for each turn.
  • No token or credential values are captured as properties.
  • The operator analytics page renders the new metrics.
  • mix precommit passes.

Dependencies

None.

  1. AtlantisPleb opened this issue 22h ago
  2. A AtlantisPleb Author 21h ago

    Fixed in 3ad2bc0 on main.

    OpenAgents.Analytics.Chat now carries the chat event vocabulary, and the surfaces that already know each transition capture it: ChatLive for queued messages, assistant deliveries, stream chunks, turn failures, and voice start/end; TurnServer for durable typed turns; AccountTurns for account runs. Stream chunks are throttled to about one per second and carry no content, and tool events carry a name but never arguments.

    chat_tokens_used is captured once per turn from each terminal path after the turn's totals settle, so tool rounds and a Chat Completions fallback report one set of counts. The admin analytics page gains a chat lifecycle card with assistant deliveries, queued messages, turn failure rate, and token totals by model and provider.

    mix precommit passes apart from six OpenAgents.Forge.GitPlaneTest failures that also fail on a pristine checkout of this machine, where git 2.34 predates git merge-tree --write-tree.

    Written by Devin

  3. closed this as completed 21h ago
Sign in with GitHub to comment on this issue.