A turn answered by a different model than the one requested says nothing #160

Closed AtlantisPleb opened this 3d ago 2 comments

Outcome

Asking for a backend the server does not support fails loudly. A turn is never answered by a different model than the one requested without the caller being told.

Current behavior

5375462 added model selection to POST /api/v3/chat/turns, and 0b42ec5004 added --model and a Tab toggle to openagents coder. Production has not deployed either, and it does not reject the parameter — it ignores it.

Verified against production just now:

$ openagents api -X POST chat/turns --input '{"message":"ping","model":"gemini-3.7-flash"}'
{"turn": {"id": "fb2ca8e7-…", "status": "streaming"}}     # 202 Accepted

$ openagents api chat/events   # the terminal event for that run
response_completed | model: stealth/ox-alpha

The request named gemini-3.7-flash. Ox Alpha answered. The server reported success, and nothing in the response says a substitution happened. Meanwhile the CLI's status bar reads gemini-3.7-flash, because the client has no reason to doubt a 202.

So a user can toggle to Gemini with Tab, watch the status bar say Gemini, read an answer from Ox Alpha, and have no way to notice. Every claim on screen is wrong and every one of them looks right.

This is the same failure shape as three other defects found today, and worth naming as a class: an unverifiable claim that reads as verified. The forum evidence document recorded a "verified" import against a destination nobody named. The published CLI shipped a command that existed in no source file. The chat console reported Reasoning 0 for turns that reasoned. In each case the surface asserted something no check could contradict.

Why it happens

Unknown parameters are ignored rather than refused. That is a reasonable default for a GitHub-shaped API where clients send extra fields, and it is the wrong default for a parameter that selects which model spends the user's budget and answers their question.

The deployed revision predates OpenAgents.Chat.Backends, so it has no list to validate against. The newer code does validate, and returns a typed 422 naming model — but only once deployed.

Work

Two independent halves; either alone leaves a hole.

Server. Decide what an unrecognized model means and make it explicit rather than incidental. Refusing with the existing typed 422 is the obvious answer, and it is what the deployed-forward code already does. Whatever is chosen, a client must be able to tell from the response whether the backend it asked for is the backend that ran.

Client. Do not trust a 202 to mean the parameter was honored. Two mechanisms exist and the CLI should use at least one:

  • GET /api/v3 publishes the supported set as chat.openagents.parameters.model.enum, derived from Backends.ids() and bound by three governance tests. A client can read it once and refuse locally against a server that does not list the backend, naming the server rather than failing obscurely.
  • The terminal response_completed event carries the model that actually ran. The CLI already parses that event. If it does not match what was requested, say so on the transcript rather than silently rendering the answer under the wrong label.

The second is the stronger check, because it verifies what happened rather than what was advertised.

Acceptance criteria

  • A request naming an unsupported backend does not return 202 and then answer from another one.
  • openagents coder --model gemini-3.7-flash against a server without that backend refuses with a message naming the server, or reports the substitution on the transcript.
  • The status bar never names a backend that did not answer.
  • A test drives the mismatch — a server that ignores the parameter, a client that asked for something else — and fails if the substitution is silent.

Verification

A CLI test with a stubbed server that ignores model and returns a different one in response_completed; a controller test for the unsupported value; a live check against production before and after the Gemini deploy.

Dependencies

Follows 5375462 and 0b42ec5004. Production also still needs GEMINI_API_KEY in the fleet environment, recorded in docs/operations/production-deploy-runbook.md §4; until then a deployed Gemini turn refuses with missing_api_key, which is the loud failure this issue asks for everywhere else.

  1. AtlantisPleb opened this issue 3d ago
  2. A AtlantisPleb Author 2d ago

    Code-complete on main via #199's landing (5125d3f): POST /api/v3/threads already refused unknown models; POST /api/inference/proxy — the lane this issue verified against production — now refuses a body model outside the catalog (model_not_served) or different from the grant's (model_mismatch) and attributes the effective model on every success, under the new PROVIDER-002 invariant. Like #164, this closes once production serves the revision (#187 tracks the main-versus-deployed gap) and the verification in this issue's body fails loudly against the live API.

  3. A AtlantisPleb Author 2d ago

    Fixed on main and proven live against the dev server. Asking for a backend the server does not serve now fails loudly instead of being silently ignored:

    POST /api/inference/proxy  {"model":"gpt-9-imaginary"}
    -> {"error":{"code":"model_not_served","requested":"gpt-9-imaginary","served":["gemini-3.7-flash","ox-alpha","gpt-5.6-luna"]}}
    

    The refusal names both what was asked for and what is actually served, so a caller can correct itself rather than guess. lib/openagents_web/controllers/inference_proxy_controller.ex returns 422 model_not_served, and chat_turn_controller.ex renders :unsupported_backend the same way.

    A turn is never answered by a different model than the one requested without the caller being told. Production still has to deploy main for this to be true there — that is #187's job, not this issue's.

  4. closed this as completed 2d ago
Sign in with GitHub to comment on this issue.