Serve a typed model catalog and route coder turns across house providers #199

Closed AtlantisPleb opened this 2d ago 4 comments

Outcome

The coder's first compute lane: server-held provider credits (Gemini, Z.ai, and peer services) behind POST /api/inference/proxy, fronted by a typed model catalog the CLI reads instead of guessing.

Contract

  • A catalog endpoint lists each offered model: id, provider, context ceiling, output ceiling, availability. The CLI renders selection from it.
  • Per-turn model selection is honored, and every turn records requested versus effective model. #160 closes as part of this work: an unsupported model fails loudly, and substitution is never silent.
  • Provider credentials live server-side only. Usage is attributed per provider and per grant.
  • Routing across providers is server policy (credit balances, availability), never client guesswork.

Source

docs/2026-08-24-coder-first-cloud-complements.md section 5, lane 1. Part of the Coder v1 release arc.

  1. AtlantisPleb opened this issue 2d ago
  2. A AtlantisPleb Author 2d ago

    Core landed on main as 5125d3f (with the grants re-mint route following as f6723c8, WAL receipt seq 301). GET /api/v3/models serves the typed catalog (id, provider, context_window, max_output, availability, default) from one config-driven source; a credential-less provider lists as unavailable and refuses selection with model_unavailable before any provider call; the proxy now refuses an unknown body model (model_not_served, echoing the served list) and a grant mismatch (model_mismatch) instead of silently ignoring the field, and every 200 attributes the effective model (x-openagents-model header plus model on every SSE chunk). PROVIDER-002 (no silent substitution) added to INVARIANTS.md. Remaining in scope here: the Gemini and Z.ai proxy-lane adapters — the report records exactly what each needs (Provider behaviour module, credential key in RuntimeConfig, lane entry, catalog entry, boundary-test classification; Chat.Gemini is the port source for the Gemini wire shape). 251 targeted tests green across the touched surfaces.

  3. A AtlantisPleb Author 2d ago

    Status against the four contract bullets, checked live against the dev server:

    • Catalog endpoint — shipped. GET /api/v1/models lists each offered model with id, provider, context ceiling, output ceiling, availability, and which is default.
    • Per-turn selection and loud failure — shipped. #160 is closed: an unsupported model returns 422 model_not_served naming both what was requested and what is served, so substitution is never silent.
    • Server-side credentials — shipped. Provider credentials live behind the proxy, and usage is attributed per grant.
    • Routing as server policy — partial, and the gap has teeth. Availability is a static property of the catalog entry rather than anything derived from whether the lane answers. gemini-3.7-flash reports "availability": "available" while every call to it fails upstream (#238), so a client that reads the catalog and trusts it picks a dead lane. Balance-aware routing across providers does not exist yet either.

    The remaining work for this issue is the fourth bullet, and #238 is the concrete instance of it. Leaving open for that.

  4. A AtlantisPleb Author 2d ago

    The fourth bullet — routing as server policy rather than client guesswork — has its first half on main in c23e2eb: availability is now derived from what the lane actually did, not only from whether a credential is configured, so a client reading the catalog is no longer told a dead lane is available (that was #238, now closed).

    What remains on this issue is balance-aware routing across providers: choosing a lane by credit balance as well as by health. The health signal that decision needs now exists.

  5. A AtlantisPleb Author 2d ago

    All four contract bullets are now shipped.

    • Catalog endpointGET /api/v1/models lists id, provider, context and output ceilings, availability, and the default.
    • Per-turn selection honoured, loud on unsupported — #160, closed: a model outside the served set returns 422 model_not_served naming both what was asked and what is served.
    • Server-side credentials, usage attributed per grant — and the cached-versus-fresh split now travels with it (#220).
    • Routing is server policy — landed in two halves: availability is derived from what a lane actually did rather than only from whether a credential exists (c23e2eb), and Models.select/0 now picks the lane for a caller that named none (c96a1ba), preferring a configured non-degraded lane and falling back to the catalog default when every lane is degraded.

    PROVIDER-002 was amended in the same change rather than left to drift, because the boundary is what makes this honest: selection applies only where nothing named a model — neither the mint nor the call. A named model is never substituted; a caller that named a lane gets that lane or an error, degraded or not, with a test pinning it. The effective model stays attributed on every response.

    Credit-balance routing is deliberately not included: balances are not modelled in this repository, and guessing at them would be worse than the fixed default this replaces. If you want balance-aware routing it wants its own issue and a source of balance truth. 4,482 tests green.

  6. closed this as completed 2d ago
Sign in with GitHub to comment on this issue.