Surface cached-token splits through usage, grants, and ATIF #220

Closed AtlantisPleb opened this 2d ago 2 comments

Outcome

Cost figures stop overstating agentic workloads: the provider's cached-versus-fresh prompt-token split travels end to end — provider response, proxy usage record, grant metering, ATIF step metrics.extra (the spec's designated place), and the Gym's cost-per-accepted-outcome math.

Evidence

docs/terminalbench/2026-08-24-fix-git-run-analysis.md: the gemini-3.7-flash trial's 124,941 input tokens were mostly the same prefix re-sent every round — exactly the >95% prefix-reuse traffic shape AgentX measures — and OpenAI and Gemini both bill cached prefix tokens at a fraction of fresh. Our usage records carry only totals, so cost per accepted outcome is wrong on precisely the workloads that matter.

Contract

The proxy reads the provider's cache fields (OpenAI prompt_tokens_details.cached_tokens and the Gemini equivalent) and records them on grant usage beside the existing totals.
The CLI's usage report and transcript events carry the split; the ATIF export puts it in step metrics.extra per the spec's convention for provider-specific token metrics.
bench/post_gym_run.py prices cached and fresh separately once the catalog carries per-model pricing.
Providers that report no split record none — absent, never zero-as-fact.

AtlantisPleb opened this issue 2d ago
A AtlantisPleb Author 2d ago

Server side landed on main in cb314d1. The proxy reads the provider's cache fields where they exist — OpenAI's prompt_tokens_details.cached_tokens and Gemini's cachedContentTokenCount — and records them on grant usage beside the totals. The contract's load-bearing clause is honoured: a provider reporting no split records none, absent rather than zero, and the usage merge no longer coerces a missing field to 0 and adds it. 185 inference, provider, chat, and proxy tests green.

Remaining for this issue: the CLI's usage report and transcript events carrying the split, the ATIF export putting it in step metrics.extra, and bench/post_gym_run.py pricing cached and fresh separately (which waits on per-model pricing in the catalog).
A AtlantisPleb Author 2d ago
Complete end to end.
- Server (cb314d1): the proxy reads OpenAI's prompt_tokens_details.cached_tokens and Gemini's cachedContentTokenCount and records them on grant usage beside the totals.
- CLI (aa7a61de96): the thread lane's usage chunk and the transcript carry the split.
- ATIF (same commit): the export puts it in step metrics.extra, the spec's designated place for provider-specific token metrics.
The contract's load-bearing clause holds at every hop: a provider that reports no split produces none — no extra key at all rather than one filled with zeros — because a zero there claims 'measured, and it was none', which is not the same as 'nobody measured'. Tested for absence as well as presence. 835 CLI tests green.

Visible in real traffic now: a coder session read back 7,329 of 7,422 input tokens as cache reads, which is exactly the prefix-reuse shape that made the old cost figures overstate agentic work.

The one deferred piece is bench/post_gym_run.py pricing cached and fresh separately, which waits on per-model pricing in the catalog — that belongs to #199's pricing work rather than here.
closed this as completed 2d ago