Price and attribute a metered call to the model that served it, not the one requested #250

Closed AtlantisPleb opened this 5d ago

Evidence

Shipped in the release at 0bf2706, promoted 2026-08-27.

1 pushes receipt

The bug

config :openagents, :vercel_gateway_fallback_models (config/config.exs) tells
the Vercel AI Gateway to answer a failed google/gemini-3.7-flash call with
zai/glm-5.3, zai/glm-5.2, or openai/gpt-5.6-luna — and to return 200 when
one of them does. OpenAgents.Providers.VercelGateway sends that list as
providerOptions.gateway.models and never reads back which model answered, so
the whole metering path treated the requested model as the serving one:

  • OpenAgents.Inference.record_usage/2 priced the record against
    grant.model_id. Gemini carries rates and Luna carries none, so a Luna call
    was billed at Gemini's rates.
  • OpenAgents.Inference.Health.record_success/1 was called with the requested
    lane's id, so a lane that failed and was rescued by a fallback published
    available at GET /api/v1/models — the same failure #238 fixed.
  • The x-openagents-model header and every SSE chunk's model field named the
    requested model, so a client rendered a model that did not answer.

Evidence

OpenAgents.Providers.OpenRouter.StreamDecoder — which the gateway shares —
parses id, choices, and usage off each chat-completions chunk and drops
model entirely. Nothing else on the path could have known.

The size of the wrong number, from the shipped catalog:

merge_usage(%{}, %{"input_tokens" => 1_000_000, "output_tokens" => 100_000}, "gemini-3.7-flash")
#=> pricing_id: "placeholder.gemini-3.7-flash.v1", estimated_cost_microusd: 2_250_000

$2.25 of measured-looking cost for a call served by a lane this deployment has
no rates for at all. OpenAgents.Threads.spend/1 then totalled it as a known
figure rather than reporting cost.microusd = nil, which is exactly the
$0.00-beside-real-spend failure METER-001 exists to prevent.

The fix

The response says which model served it; read it back.

  • The chat-completions decoder carries model out as
    {:model_served, name} — once, and never guessed.
  • OpenAgents.Providers.Provider.substitutable?/0 (optional, false where not
    exported) says whether a lane's provider may answer with a different model.
    VercelGateway answers true exactly while a fallback list is configured, so
    silence from a lane that cannot substitute still means the model it asked for.
  • Every metered record carries served_model, and Pricing.price/2 prices
    against that rather than against the grant's model. Three values name no rate
    table and so price at nothing: unresolved (a substitutable lane disclosed
    nothing — naming the requested model would be a claim the deployment cannot
    support), mixed (one grant's calls served by more than one model; a single
    total cannot be charged at two rate tables), and any model outside the
    catalog.
  • Threads.spend/1 names the serving lane in cost.unpriced_models, so an
    operator is sent to price the lane that actually ran.
  • Health follows the lane that answered: a fallback-served call records a
    failure against the requested lane and a success against the serving lane
    where the catalog has it. An unresolved response records nothing — health that
    reports what it does not know is the fault being fixed, not a smaller version
    of it.
  • The header and chunks attribute the serving model, or the word unresolved.

METER-001 and PROVIDER-002 are amended in the same change.

Proof: test/openagents_web/controllers/inference_proxy_fallback_test.exs and
test/openagents/providers/open_router/stream_decoder_test.exs.

  1. AtlantisPleb opened this issue 5d ago
  2. AtlantisPleb closed this as completed in b2e5ee0 5d ago
Sign in with GitHub to comment on this issue.