Track full chat lifecycle and token usage in PostHog #102
- AtlantisPleb opened this issue 22h ago
-
A Author 21h ago Fixed in 3ad2bc0 on main.
OpenAgents.Analytics.Chatnow carries the chat event vocabulary, and the surfaces that already know each transition capture it:ChatLivefor queued messages, assistant deliveries, stream chunks, turn failures, and voice start/end;TurnServerfor durable typed turns;AccountTurnsfor account runs. Stream chunks are throttled to about one per second and carry no content, and tool events carry a name but never arguments.chat_tokens_usedis captured once per turn from each terminal path after the turn's totals settle, so tool rounds and a Chat Completions fallback report one set of counts. The admin analytics page gains a chat lifecycle card with assistant deliveries, queued messages, turn failure rate, and token totals by model and provider.mix precommitpasses apart from sixOpenAgents.Forge.GitPlaneTestfailures that also fail on a pristine checkout of this machine, where git 2.34 predatesgit merge-tree --write-tree. - closed this as completed 21h ago
Outcome
The
/chatsurface reports every meaningful step of a conversation through PostHog, including assistant responses, token consumption, model/provider selection, queued messages, and failures.Current behavior
lib/openagents_web/live/chat_live.exalready emits two events:chat_message_sentwhen a user message starts a turn.chat_turn_completedwhen a turn finishes.These do not include the assistant message, token counts, model or provider, message queuing, streaming progress, or non-terminal errors.
Work
Audit
lib/openagents_web/live/chat_live.exandlib/openagents/chat/open_router.exand emit the missing server-side PostHog events throughOpenAgents.Analytics.capture/3:chat_message_received— the assistant message was streamed or returned to the user.chat_tokens_used— a turn consumed tokens; includeinput_tokens,output_tokens,model,provider,conversation_id, andturn_id.chat_message_queued— a user message was queued while another turn was active.chat_turn_failed— a turn ended in an error or non-completed status, withreason.chat_stream_chunk— one assistant stream chunk arrived (rate-limited to avoid event spam, e.g. one event per second or per n chunks).chat_tool_called— the assistant invoked a tool, withtool_nameandturn_id.chat_voice_startedandchat_voice_ended— a voice session began or completed, withduration_ms.OpenRouter token capture
Read
lib/openagents/chat/open_router.exandlib/openagents/chat/open_router/responses_stream_decoder.ex. When OpenRouter returnsusagein a response or stream, capturechat_tokens_usedonce per turn, not per chunk. Do not duplicate counts across fallback paths.Analytics queries
Update
lib/openagents/posthog.exso the operator analytics page shows:chat_tokens_usedper model and provider in the last day.chat_message_receivedcount.chat_turn_failedrate.Acceptance criteria
usagereturned by OpenRouter for each turn.mix precommitpasses.Dependencies
None.