Chat top bar reports Reasoning 0 for turns that reasoned #116

Closed AtlantisPleb opened this 3d ago 1 comment

Outcome

When a /chat turn reasons, the console records the provider's reasoning token count and the running token list in the top bar reports it. A turn that visibly thought never reports Reasoning 0.

Current behavior

Issue #101 shipped the running token list in commit 589a703. On openagents.com/chat the top bar reads:

Input 3,244   Output 358   Reasoning 0   Cached 2,816   Total 3,602

The same turn rendered a Thought for 1 seconds block with a full reasoning summary, and the composer was set to High reasoning, so the turn did reason. Total equals Input plus Output exactly, which is the fallback in turn_total/1 for a response with no total_tokens.

Reasoning 0 is not the "kind not reported" path: sum_tokens/2 in lib/openagents_web/live/chat_console_live.ex drops a kind when every turn reports nil, so the list is showing a stored zero, not a missing value. Either the provider reports output_tokens_details.reasoning_tokens as 0, or the count is lost between the response and the turn record that OpenAgents.Chat.AccountTurns reads back.

Reading paths involved:

  • OpenAgents.Chat.OpenRouter.ResponsesStreamDecoder.merge_provider_metadata/2 copies response["usage"] onto the completion with Map.put_new/3.
  • OpenAgents.Chat.AccountTurns.usage_counts/1 derives reasoning from output_tokens_details.reasoning_tokens, then completion_tokens_details.reasoning_tokens.
  • OpenAgents.Providers.OpenAI.StreamDecoder.normalize_usage/1 reads the same field for the SCV path, under the different key name reasoning_output_tokens.

Contract

  • Establish what OpenRouter returns for this model on the Responses profile. Capture one raw response.completed usage envelope for a reasoning turn and record it in the issue, so the fix is grounded in a provider payload and not in an assumption about which field carries the count.
  • If the count arrives and is dropped, carry it end to end: response envelope, stored completion, usage_counts/1, the per-turn usage line, and the top bar sum.
  • If the provider reports no reasoning detail for the model, do not render Reasoning 0. A count the provider never measured must stay off the list, the same as any other unreported kind, and a zero must remain distinguishable from a missing value.
  • Do not derive, estimate, or back out a reasoning count from output tokens or from the reasoning summary's length. Provider-reported evidence only.
  • Cover the three cases in test/openagents_web/live/chat_console_test.exs and the turn tests: a reported positive count, a reported zero, and an absent detail map.

Notes

Reported from a production /chat conversation on 2026-08-23. Screenshot evidence is in the reporting session.

  1. AtlantisPleb opened this issue 3d ago
  2. A AtlantisPleb Author 3d ago

    Fixed in e9c3682.

    Two things were losing the count:

    • Each tool round of a turn is its own Responses request, and each response reports only what that round consumed. The last round's usage replaced the earlier counts, so reasoning the model spent before a tool call never reached the console. OpenAgents.Chat.OpenRouter now sums the counts every round reports.
    • Ox Alpha streams reasoning output while reporting reasoning_tokens: 0. That zero measures nothing, so OpenAgents.Chat.AccountTurns now stores no reasoning count for such a turn and the top bar omits the category instead of reading Reasoning 0. GPT-5-mini's positive counts still flow through unchanged.

    No count is inferred: only what the provider reports is stored.

  3. closed this as completed 3d ago
Sign in with GitHub to comment on this issue.