Read the lane off the run, not out of the model id

96c921c07c9c · Claude Fable 5 · · parent 3782e743a2b3

Read the lane off the run, not out of the model id

A live local run caught two faults that only appear together. The first graded
run of tb2-quick reported two models where there is one: the completed trial
read `qwen3.8:27b-mtp-q8_0` from the coder's ATIF export, and the timed-out
trial recovered `ollama:qwen3.8:27b-mtp-q8_0` from Harbor's config, because the
recovery reproduced the adapter's local-lane prefix. That prefix is how the
adapter tells the CLI which lane to use and never reaches a trajectory, so
adding it invented a spelling nothing else in the run uses — and the run digest
would have pinned the pair, and a lane comparison would have called it a
confounder.

The second fault was underneath it. `unmetered_local_lane` was decided by
looking for `ollama:` in the model id, so the trial that spelled it bare priced
as `unknown_model` — a lane with no rates to have reads as a gap in the rate
catalog. Whether a lane bills metered tokens is a fact about the lane, and the
run records the lane, so `priceUsage` now takes it. The prefix stays as a
secondary signal for a caller pricing one trial with no lane in hand.

Two lane-comparison cases changed with it, and the change is the honest one.
They asserted a `worse` cost delta between a proxy row and a local row, which
was only reachable because a fixture pretended a local lane could be priced.
The comparison #34 actually asks for — house models through the proxy against a
local model — has a permanently unstatable cost delta in one direction. It now
reports `unpriced` with its reason, and the success-rate delta, which is
measured the same way on both sides, still compares.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SoZMfWRSGnf6FZX2Ar9rQ2
Co-Authored-By
Claude Fable 5 <noreply@anthropic.com>

Deploy story

What this commit did to the running system — joined from the forge receipt chain, the part a commit page elsewhere cannot show.

Not deployed through the forge lane

No push, promotion, build, or deploy receipt references this commit (receipts are scanned over a bounded recent window). Changes shipped by full node replacement carry their proof in the release gate receipt instead.

Changed files

  • modified packages/coder-effectiveness/src/compare.test.ts
  • modified packages/coder-effectiveness/src/effectiveness.test.ts
  • modified packages/coder-effectiveness/src/effectiveness.ts
  • modified packages/coder-effectiveness/src/harbor-job.ts
  • modified packages/coder-effectiveness/src/pricing.ts

Diff

5 files changed, +72 -25

packages/coder-effectiveness/src/compare.test.ts modified +24 -3

@@ -89,9 +89,29 @@ describe("lane against lane", () => {

89 89
    const comparison = compareRuns([proxy, local]).laneComparisons[0]!;
90 90
91 91
    expect(comparison.baselineLane).toBe("proxy");
92
    const lane = comparison.lanes.find((entry) => entry.lane === "local")!;
93
    expect(lane.costDelta?.direction).toBe("worse");
94 92
    expect(comparison.lanes.find((entry) => entry.lane === "proxy")!.costDelta).toBeNull();
93
    // Success rate compares across any two lanes; it is measured the same way
94
    // on both sides whatever either one costs.
95
    expect(
96
      comparison.lanes.find((entry) => entry.lane === "local")!.successRateDelta?.direction,
97
    ).toBe("worse");
98
  });
99
100
  test("refuses a cost delta against the local lane, which bills no metered tokens", () => {
101
    // This is the lane comparison #34 actually asks for — house models through
102
    // the proxy against a local model — and its cost delta is permanently
103
    // unstatable in one direction. Saying so is the point: a blank cell here
104
    // reads as "the same", and a zero would read as "free".
105
    const proxy = row("priced-lane", "proxy", "2026-08-25T10:00:00.000Z");
106
    const local = row("regressed-lane", "local", "2026-08-25T10:05:00.000Z");
107
108
    const lane = compareRuns([proxy, local]).laneComparisons[0]!.lanes.find(
109
      (entry) => entry.lane === "local",
110
    )!;
111
112
    expect(lane.costDelta?.direction).toBe("unpriced");
113
    expect(lane.costDelta?.absolute).toBeNull();
114
    expect(lane.costDelta?.reason).toContain("cost_unknown");
95 115
  });
96 116
97 117
  test("honours an explicit baseline lane", () => {

@@ -101,8 +121,9 @@ describe("lane against lane", () => {

101 121
    const comparison = compareRuns([proxy, local], { baselineLane: "local" }).laneComparisons[0]!;
102 122
103 123
    expect(comparison.baselineLane).toBe("local");
124
    expect(comparison.lanes.find((entry) => entry.lane === "local")!.costDelta).toBeNull();
104 125
    expect(comparison.lanes.find((entry) => entry.lane === "proxy")!.costDelta?.direction).toBe(
105
      "better",
126
      "unpriced",
106 127
    );
107 128
  });
108 129
packages/coder-effectiveness/src/effectiveness.test.ts modified +12 -4

@@ -221,12 +221,20 @@ describe("cost per accepted outcome", () => {

221 221
 * the run you most want to know the lane of.
222 222
 */
223 223
describe("a trial with no trajectory", () => {
224
  test("recovers the model from Harbor's own trial config", () => {
224
  test("recovers the model spelled the way the coder's own export spells it", () => {
225
    const result = report("timed-out-lane", "local");
226
227
    // Harbor spells it `ollama/qwen3.8:…`; the coder's ATIF export writes the
228
    // bare name. A run that mixed a completed trial with a killed one would
229
    // otherwise report two models where there is one, and the run digest would
230
    // pin the pair.
231
    expect(result.models).toEqual(["qwen3.8:27b-mtp-q8_0"]);
232
  });
233
234
  test("prices a bare local-lane id off the lane, not off a prefix in the id", () => {
235
    // Nothing in `qwen3.8:27b-mtp-q8_0` says it ran locally. The run does.
225 236
    const result = report("timed-out-lane", "local");
226 237
227
    expect(result.models).toEqual(["ollama:qwen3.8:27b-mtp-q8_0"]);
228
    // Harbor spells it `ollama/…`; the catalog id the adapter sends is
229
    // `ollama:…`, and a recovered id has to price like a read one.
230 238
    expect(result.perTrial[0]!.disposition).toBe("unmetered_local_lane");
231 239
    expect(result.perTrial[0]!.disposition).not.toBe("unknown_model");
232 240
  });
packages/coder-effectiveness/src/effectiveness.ts modified +1

@@ -135,6 +135,7 @@ export const summarizeRun = (

135 135
        cachedInputTokens: trial.cachedInputTokens,
136 136
      },
137 137
      catalog,
138
      run.lane,
138 139
    );
139 140
    if (cost.usd === null) {
140 141
      unpricedTrials += 1;
packages/coder-effectiveness/src/harbor-job.ts modified +14 -9

@@ -140,12 +140,19 @@ const readTrial = (dirName: string, trialDir: string, trialResult: unknown): Tri

140 140
 * and price as `unknown_model` for a reason that has nothing to do with pricing.
141 141
 *
142 142
 * Harbor records the model on every trial before the agent starts, spelled the
143
 * way its `--model` flag takes it. The adapter maps that spelling onto the
144
 * coder's catalog id, and this repeats that mapping so a recovered id prices
145
 * the same as one read from a trajectory. It is a second copy of a two-line
146
 * rule; the alternative is a model pin that disappears precisely when it
147
 * matters. Keep it in step with `_catalog_model` in
148
 * `bench/adapters/openagents_coder.py`.
143
 * way its `--model` flag takes it: `<provider>/<name>`. The id has to come back
144
 * spelled the way the CODER spells it, not the way Harbor does, because a run
145
 * mixing completed and killed trials would otherwise report two models where
146
 * there is one — and the report would list them, the digest would pin them, and
147
 * a lane comparison would treat the pair as a confounder. The coder's ATIF
148
 * export records the bare name (`qwen3.8:27b-mtp-q8_0`), so that is what a
149
 * recovered id is.
150
 *
151
 * Note that this deliberately does NOT reproduce the adapter's `ollama:` local
152
 * prefix. That prefix is how the adapter tells the CLI which lane to use; it
153
 * never reaches the trajectory, so adding it here would invent a spelling
154
 * nothing else in the run uses. Whether the lane bills metered tokens is a fact
155
 * about the lane, and the lane is on the run — see `priceUsage`.
149 156
 */
150 157
const modelFromTrialConfig = (trialConfig: unknown): string | null => {
151 158
  // The trial's own `config.json` holds the agent block at the top level; the

@@ -157,10 +164,8 @@ const modelFromTrialConfig = (trialConfig: unknown): string | null => {

157 164
  if (spelled === null) return null;
158 165
  const separator = spelled.indexOf("/");
159 166
  if (separator === -1) return spelled;
160
  const provider = spelled.slice(0, separator);
161 167
  const name = spelled.slice(separator + 1);
162
  if (provider === "ollama") return `ollama:${name}`;
163
  return name === "" ? provider : name;
168
  return name === "" ? spelled.slice(0, separator) : name;
164 169
};
165 170
166 171
/**
packages/coder-effectiveness/src/pricing.ts modified +21 -9

@@ -135,22 +135,34 @@ export interface CostResult {

135 135
  readonly reason: string;
136 136
}
137 137
138
/** The local lane runs on hardware you own; no per-token rate applies. */
139
const isLocalLaneModel = (modelId: string): boolean =>
140
  modelId.startsWith("ollama:") || modelId.startsWith("ollama/");
138
/**
139
 * Whether this trial ran on a lane that bills no metered tokens.
140
 *
141
 * The lane the run was executed on is the authority, and it is passed in
142
 * because it is a fact the run records rather than something to be guessed at.
143
 * The `ollama:` prefix is kept as a secondary signal for a caller pricing one
144
 * trial with no lane in hand, but it was never a good primary one: the coder's
145
 * own ATIF export writes the bare model name, so an id read from a trajectory
146
 * carries no prefix to spot and a whole local run priced as `unknown_model` —
147
 * which reads as a gap in the rate catalog rather than as a lane that has no
148
 * rates to be in it.
149
 */
150
const isUnmetered = (modelId: string, lane: string | null): boolean =>
151
  lane === "local" || modelId.startsWith("ollama:") || modelId.startsWith("ollama/");
141 152
142 153
/**
143 154
 * Price one trial's usage against a rate catalog.
144 155
 *
145
 * Unknown stays unknown. A model with no rate, a local lane with no per-token
146
 * rate at all, or usage missing either token dimension all return `usd: null`
147
 * with a disposition that names the reason. Nothing here falls back to a
148
 * conservative default rate: this function measures, it does not charge.
156
 * Unknown stays unknown. A model with no rate, a lane with no per-token rate at
157
 * all, or usage missing either token dimension all return `usd: null` with a
158
 * disposition that names the reason. Nothing here falls back to a conservative
159
 * default rate: this function measures, it does not charge.
149 160
 */
150 161
export const priceUsage = (
151 162
  modelId: string | null,
152 163
  usage: UsageForCost,
153 164
  catalog: Readonly<Record<string, ModelRateRow>> = CODER_RATE_CATALOG,
165
  lane: string | null = null,
154 166
): CostResult => {
155 167
  if (modelId === null || modelId === "") {
156 168
    return {

@@ -160,12 +172,12 @@ export const priceUsage = (

160 172
      reason: "the trial records no model id, so no rate can be selected",
161 173
    };
162 174
  }
163
  if (isLocalLaneModel(modelId)) {
175
  if (isUnmetered(modelId, lane)) {
164 176
    return {
165 177
      usd: null,
166 178
      disposition: "unmetered_local_lane",
167 179
      rateBasis: null,
168
      reason: `${modelId} runs on the local lane, which bills no metered tokens, so it has no per-token cost`,
180
      reason: `${modelId} ran on the local lane, which bills no metered tokens, so it has no per-token cost`,
169 181
    };
170 182
  }
171 183
  const row = catalog[modelId];

This page updates live while a promote is in flight · changelog