# bench-results

The append-only record of graded coder runs, one file per suite, one JSON line

per run. Issue

[#34](https://openagents.com/OpenAgentsInc/openagents/issues/34) asks for

results that append here with receipts; this directory is that store.

It is empty of rows. Every row must come from a real Harbor run, and no run has

been recorded yet. A seeded or example row would be a fabricated measurement

sitting in the file that the trend line reads.

## Appending a run

```sh

# 1. Run the suite. Harbor grades the trials.

bench/run-suite.sh bench/suites/tb2-cross-section.txt \

  --model openai/gpt-5.6-luna --jobs-dir /tmp/gym-jobs-run

# 2. Score it and record it.

pnpm run effectiveness:report -- /tmp/gym-jobs-run/<job-dir> \

  --suite tb2-cross-section --lane proxy \

  --thresholds packages/coder-effectiveness/thresholds/tb2-cross-section.json \

  --append bench-results/tb2-cross-section.jsonl

```

The report's exit code is unchanged by `--append`: `0` the gate passed, `1` a

floor was breached, `2` the gate was unverifiable. A fourth code, `3`, means the

run was scored but the store refused to record it, and it only ever replaces a

`0`.

## Reading it

```sh

pnpm run effectiveness:compare -- bench-results/tb2-cross-section.jsonl

```

Compare verifies the chain first and refuses to compare a store that does not

verify. It then produces two views:

- **Lanes** — the same suite across lanes at their most recent runs, each

  measured against a baseline lane.

- **Trend** — the same suite on one lane over time, the shape #34's acceptance

  clause asks for and the shape a regression appears in.

## Receipts, and why they are a chain

Each row carries a `receipt`, a `receipt:<sha256>` digest over that row's own

fields **and** the receipt of the row before it.

- Edit a figure in an existing row and that row's contents stop matching its

  receipt: a `receipt_mismatch` naming the row.

- Insert, remove, or reorder rows and a row's `previousReceipt` stops naming the

  row before it: a `chain_broken` naming the row.

That is what makes the file append-only in practice rather than by convention.

A benchmark history nobody can quietly rewrite is worth more than one that

merely has not been rewritten yet.

It is a hash chain, not a signature. A signature would answer "who wrote this",

which needs a key the tooling has no business holding. The chain answers "has

this been rewritten since it was written", which is the question a trend

actually asks. If a signing seam arrives later, it signs the head receipt and

this chain still holds underneath it.

## Unknown costs stay unknown

A row's `costPerAcceptedOutcomeUsd` is `null` when the run could not be priced,

and it carries the `costDisposition` and `costCoverage` that say why. It is

never `0`. `gpt-5.6-luna` — the lane the graded runs use most — is deliberately

unpriced in the forge model catalog, so writing a zero here would launder an

unmeasured lane into a free one at exactly the point where the figure stops

being read next to its reason.

A comparison follows the same rule: a delta against an unpriced side is

`unpriced`, never `0` and never "improved".

## Row fields

| Field                         | Meaning                                                                                                                                                                                  |

| ----------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |

| `runDigest`                   | The report's pin over suite, lane, tasks, CLI version, model, rates.                                                                                                                     |

| `suiteKey`                    | The narrower pin two rows must share to be comparable: suite, sorted task list, rate catalog. Lane, model, and CLI version are excluded, because those are the axes a comparison varies. |

| `jobId`                       | The Harbor job. A store refuses a job it already holds — re-scoring a run does not make it a second run.                                                                                 |

| `costPerAcceptedOutcomeUsd`   | Total run cost over accepted outcomes, failures included, or `null`.                                                                                                                     |

| `gateStatus`                  | `passed`, `failed`, `unverifiable`, or `null` when no thresholds file was given.                                                                                                         |

| `previousReceipt` / `receipt` | The chain.                                                                                                                                                                               |

The full type is `BenchResultRow` in

`packages/coder-effectiveness/src/results-store.ts`.

  "note": "Heuristic false-green LEADS, not findings. A finding requires a demonstrated reproduction (surviving mutation via mutation-runner). Do not treat a candidate as a confirmed false green. Coverage-theater leads may include tests that delegate their assertion to a custom helper the classifier does not recognise; verify before acting.",

  "sourceDigest": "sha256:dd810dd48c5bdbc9becd7fcc01dd41a4ca2abf0b2d6f6a545907247f6e3e8361",

  "summary": {

    "filesScanned": 2485,

    "filesScanned": 2488,

    "candidateCount": 16,

    "byMode": {

      "false_green_coverage_theater": 15,

{

  "schemaVersion": "1",

  "repository": "OpenAgentsInc/openagents",

  "sourceDigest": "sha256:88c82b2d87bcc50ede92cc36ec6d4b69b275bb598a0645b6f611e4ab69b72bab",

  "sourceDigest": "sha256:65e4ac0e1a556f2bfac1a35ea147861f876a3704f4e7b8cbeeeccf8292b0aeba",

  "surfaces": [

    {

      "id": "app:@openagentsinc/acceptance-runner",

      "oracles": [

        {

          "type": "test",

          "ref": "packages/coder-effectiveness (4 tracked test files)"

          "ref": "packages/coder-effectiveness (7 tracked test files)"

        }

      ],

      "obligation": {

    "test:assure-repo": "vp test --run packages/assure-repo/test",

    "audit:assure-repo": "node --import tsx packages/assure-repo/src/cli.ts audit-generate",

    "check:assure-repo-audit": "node --import tsx packages/assure-repo/src/cli.ts audit-check",

    "effectiveness:compare": "node --import tsx packages/coder-effectiveness/src/compare-cli.ts",

    "effectiveness:report": "node --import tsx packages/coder-effectiveness/src/cli.ts",

    "test:coder-effectiveness": "vp test --run packages/coder-effectiveness/src",

    "test:product-spec": "vp test --run packages/product-spec",

bench/run-suite.sh bench/suites/tb2-cross-section.txt \

  --model openai/gpt-5.6-luna --jobs-dir /tmp/gym-jobs-run

# 2. Score it.

# 2. Score it, and record it.

pnpm run effectiveness:report -- /tmp/gym-jobs-run/<job-dir> \

  --suite tb2-cross-section --lane proxy \

  --thresholds packages/coder-effectiveness/thresholds/tb2-cross-section.json

  --thresholds packages/coder-effectiveness/thresholds/tb2-cross-section.json \

  --append bench-results/tb2-cross-section.jsonl

# 3. Read the trend and the lanes.

pnpm run effectiveness:compare -- bench-results/tb2-cross-section.jsonl

```

Nothing new has to be produced by the coder for this to work. The suite reads

success floor and cannot be priced is `failed`, because something _was_

measured and it broke.

| Exit | Meaning                                     |

| ---- | ------------------------------------------- |

| 0    | Every applicable floor passed.              |

| 1    | A floor was breached.                       |

| 2    | The gate could not be verified. Not a pass. |

| Exit | Meaning                                                 |

| ---- | ------------------------------------------------------- |

| 0    | Every applicable floor passed.                          |

| 1    | A floor was breached.                                   |

| 2    | The gate could not be verified. Not a pass.             |

| 3    | The run was scored but `--append` refused to record it. |

Code 3 only ever replaces a 0. A breach or an unverifiable gate is the more

important finding, so a non-zero gate always outranks a bookkeeping refusal.

## The store

`--append <file>` writes the run into an append-only JSONL store — one line per

run, one file per suite, under `bench-results/`. Every row carries a

`receipt:<sha256>` over its own fields **and** the receipt of the row before it,

so editing a figure in place breaks that row's receipt and inserting or removing

a row breaks the next row's link. Both are named findings rather than silent

ones, which is what makes the file append-only in practice rather than by

convention.

Two refusals, both reported rather than thrown:

- `duplicate_job` — the store already holds this Harbor job. Re-scoring a run

  against different floors is a useful thing to do, and a second row is not what

  it produces; two rows for one execution would double-count it in every trend

  that follows.

- `chain_broken` — the store does not verify, so appending would extend a

  rewritten history and bury the break one row deeper.

An unpriced run records `null`, never `0`, and carries the disposition and

coverage that say why. See `bench-results/README.md`.

## Comparing runs

```sh

pnpm run effectiveness:compare -- bench-results/tb2-cross-section.jsonl

```

The chain is verified first, and a store that does not verify is refused rather

than compared: a trend over a rewritten history reads exactly like a trend over

an honest one. Then two views come off the same rows.

**Trend** is one suite on one lane over time — the shape #34's acceptance clause

asks for, and the shape a regression appears in. **Lanes** is one suite across

lanes at their most recent runs, each measured against a baseline.

Three rules keep both honest:

- **Only rows sharing a `suiteKey` are compared.** That key is the suite, the

  sorted task list, and the rate catalog. Lane, model, and CLI version are

  excluded, because those are the axes a comparison varies. Rows in other groups

  are reported as their own groups, never folded into one table with a footnote.

- **A cost delta needs both sides priced.** If either row is unpriced, the delta

  is `unpriced` — not zero, and not "improved". It is the same refusal the report

  makes about a single run, applied to the difference between two.

- **A comparison that varies two things says so.** A lane comparison whose rows

  carry different CLI versions is flagged `confounded`, as is a trend that also

  changes model, and any comparison touching a row priced from placeholder

  rates. Nothing is suppressed: a confounded comparison is often the only one

  available, and it is readable as long as it is labelled.

## Grading

| `crashed-verifier` | Ungraded trials stay out of both buckets.                         |

| `regressed-lane`   | A regression raises cost per accepted outcome and trips the gate. |

The store and comparison cases build their rows from the same fixtures, into a

temporary store, with the clock injected.

```sh

pnpm --dir packages/coder-effectiveness test

```

  per-model-family seam, the token-economy delta it measures is readable from

  this suite's rows without a change here.

- **Two consecutive scheduled runs and a caught live regression.** Issue #34's

  acceptance needs real Harbor runs on amd64 hardware; the fixture runs prove

  the grading and the arithmetic, not the schedule.

- **Appending results to `bench-results` with receipts.** The report is

  `--json`-shaped and ready for it; the store is not wired.

  acceptance needs real Harbor runs on amd64 hardware. The store and the

  comparison now hold the rows those runs will produce, and the fixture cases

  prove the trend reads a regression as a rise in cost per accepted outcome —

  but a fixture is not a schedule, and no real run has been recorded.

- **The 20-30 task suite.** The floors point at

  `bench/suites/tb2-cross-section.txt`, twelve tasks. The wider suite and the

  owned set drawn from this tracker's closed issues are not built.

  pricingFromModelsPayload,

} from "./pricing.ts";

import { renderReport } from "./render.ts";

import { appendResultRow } from "./results-store.ts";

import { evaluateThresholds, parseThresholds, type ThresholdGate } from "./thresholds.ts";

const USAGE = `Usage: coder-effectiveness report <job-dir> [options]

  --models <file>        A captured GET /api/v1/models body to price from,

                         instead of the pinned rate catalog. A model the served

                         catalog leaves unpriced stays unpriced here.

  --append <store>       Append this run to an append-only bench-results store,

                         chained to the receipt of the row before it. Refuses a

                         Harbor job the store already holds, and refuses to

                         extend a store that does not verify.

  --json                 Emit the report as JSON instead of text.

  -h, --help             Show this help.

Exit codes: 0 gate passed, 1 a floor was breached, 2 the gate was unverifiable.

An unverifiable gate is not a pass: a criterion could not be measured, most

often because the lane carries no published rate.`;

Exit codes: 0 gate passed, 1 a floor was breached, 2 the gate was unverifiable,

3 the run was scored but --append refused to record it. An unverifiable gate is

not a pass: a criterion could not be measured, most often because the lane

carries no published rate. A non-zero gate always outranks 3 — a breach matters

more than a bookkeeping refusal.`;

/**

 * Its own code, so a scheduled run whose result never reached the store is not

 * reported as a clean pass. It only ever replaces a 0: a gate that failed or

 * could not be verified is the more important finding.

 */

const APPEND_REFUSED_EXIT = 3;

interface Arguments {

  readonly jobDir: string;

  readonly lane: string;

  readonly thresholdsPath: string | null;

  readonly modelsPath: string | null;

  readonly appendPath: string | null;

  readonly json: boolean;

}

  let lane = "proxy";

  let thresholdsPath: string | null = null;

  let modelsPath: string | null = null;

  let appendPath: string | null = null;

  let json = false;

  for (let index = 0; index < argv.length; index += 1) {

      modelsPath = expectValue(argv, (index += 1), "--models");

      continue;

    }

    if (argument === "--append") {

      appendPath = expectValue(argv, (index += 1), "--append");

      continue;

    }

    if (argument.startsWith("-")) {

      throw new Error(`unknown option: ${argument}`);

    }

  if (lane !== "proxy" && lane !== "local") {

    throw new Error(`--lane must be proxy or local, got: ${lane}`);

  }

  return { jobDir, suite, lane, thresholdsPath, modelsPath, json };

  return { jobDir, suite, lane, thresholdsPath, modelsPath, appendPath, json };

};

const expectValue = (argv: ReadonlyArray<string>, index: number, option: string): string => {

            parseThresholds(JSON.parse(readFileSync(parsed.thresholdsPath, "utf8"))),

          );

    const appended =

      parsed.appendPath === null

        ? null

        : appendResultRow(parsed.appendPath, report, gate, {

            recordedAt: new Date().toISOString(),

          });

    process.stdout.write(

      parsed.json ? `${JSON.stringify({ report, gate }, null, 2)}\n` : renderReport(report, gate),

      parsed.json

        ? `${JSON.stringify({ report, gate, appended }, null, 2)}\n`

        : renderReport(report, gate),

    );

    return exitCodeFor(gate);

    if (appended !== null && !parsed.json) {

      process.stdout.write(

        appended.appended

          ? `\nAppended to ${parsed.appendPath!} as ${appended.row.receipt}\n`

          : `\nNot appended (${appended.refusal}): ${appended.reason}\n`,

      );

    }

    const gateCode = exitCodeFor(gate);

    if (gateCode !== 0) return gateCode;

    return appended !== null && !appended.appended ? APPEND_REFUSED_EXIT : 0;

  } catch (error) {

    process.stderr.write(`coder-effectiveness: ${(error as Error).message}\n`);

    return 1;

/**

 * `coder-effectiveness compare` — read a bench-results store and say what its

 * rows do and do not prove about each other.

 *

 *     pnpm run effectiveness:compare -- bench-results/tb2-cross-section.jsonl

 *

 * The chain is verified before anything is compared, and a store that does not

 * verify is refused rather than compared. A trend line computed over a history

 * that has been rewritten is worse than no trend line, because it reads exactly

 * like one that has not.

 *

 * EXIT CODES. 0 the store verified and was compared, 1 the store could not be

 * read, 2 the store does not verify. The second and third are different

 * findings: a missing file is an operator typo, a broken chain is evidence.

 */

import { compareRuns } from "./compare.ts";

import { renderComparison } from "./render-compare.ts";

import { readResultRows, verifyResultChain } from "./results-store.ts";

const USAGE = `Usage: coder-effectiveness compare <store> [options]

Arguments:

  <store>                An append-only bench-results JSONL store, the file

                         \`effectiveness:report --append\` writes.

Options:

  --suite <name>         Compare only rows recorded under this suite name.

  --baseline-lane <lane> Lane every other lane is measured against.

                         Default: proxy, or the first lane in the group.

  --json                 Emit the comparison as JSON instead of text.

  -h, --help             Show this help.

Exit codes: 0 compared, 1 the store could not be read, 2 the store does not

verify. A store that does not verify is not compared: its history was rewritten

after it was written, and a trend over a rewritten history reads exactly like a

trend over an honest one.`;

interface Arguments {

  readonly storePath: string;

  readonly suite: string | null;

  readonly baselineLane: string | null;

  readonly json: boolean;

}

const parseArguments = (argv: ReadonlyArray<string>): Arguments | "help" => {

  let storePath: string | null = null;

  let suite: string | null = null;

  let baselineLane: string | null = null;

  let json = false;

  for (let index = 0; index < argv.length; index += 1) {

    const argument = argv[index]!;

    if (argument === "--") continue;

    if (argument === "-h" || argument === "--help") return "help";

    if (argument === "--json") {

      json = true;

      continue;

    }

    if (argument === "--suite") {

      suite = expectValue(argv, (index += 1), "--suite");

      continue;

    }

    if (argument === "--baseline-lane") {

      baselineLane = expectValue(argv, (index += 1), "--baseline-lane");

      continue;

    }

    if (argument.startsWith("-")) throw new Error(`unknown option: ${argument}`);

    if (storePath !== null) throw new Error(`unexpected extra argument: ${argument}`);

    storePath = argument;

  }

  if (storePath === null) throw new Error("missing required <store> argument");

  return { storePath, suite, baselineLane, json };

};

const expectValue = (argv: ReadonlyArray<string>, index: number, option: string): string => {

  const value = argv[index];

  if (value === undefined || value.startsWith("-")) throw new Error(`${option} needs a value`);

  return value;

};

const main = (argv: ReadonlyArray<string>): number => {

  let parsed: Arguments | "help";

  try {

    parsed = parseArguments(argv);

  } catch (error) {

    process.stderr.write(`${(error as Error).message}\n\n${USAGE}\n`);

    return 1;

  }

  if (parsed === "help") {

    process.stdout.write(`${USAGE}\n`);

    return 0;

  }

  let rows;

  try {

    rows = readResultRows(parsed.storePath);

  } catch (error) {

    process.stderr.write(`coder-effectiveness: ${(error as Error).message}\n`);

    return 1;

  }

  const verdict = verifyResultChain(rows);

  if (!verdict.ok) {

    process.stderr.write(

      `coder-effectiveness: ${parsed.storePath} does not verify (${verdict.break.kind}).\n  ${verdict.break.detail}\nNothing was compared.\n`,

    );

    return 2;

  }

  const comparison = compareRuns(rows, {

    ...(parsed.suite === null ? {} : { suite: parsed.suite }),

    ...(parsed.baselineLane === null ? {} : { baselineLane: parsed.baselineLane }),

  });

  process.stdout.write(

    parsed.json

      ? `${JSON.stringify({ verified: verdict, comparison }, null, 2)}\n`

      : renderComparison(comparison),

  );

  return 0;

};

process.exitCode = main(process.argv.slice(2));

/**

 * Comparison: trends over time, lanes against each other, and the deltas this

 * suite refuses to state.

 *

 * Rows are built from the same checked-in fixture Harbor jobs the rest of the

 * suite reads, with the clock injected. No model is called and no Docker image

 * runs.

 */

import { fileURLToPath } from "node:url";

import { describe, expect, test } from "vite-plus/test";

import { compareRuns } from "./compare.ts";

import { summarizeRun } from "./effectiveness.ts";

import { readHarborJob } from "./harbor-job.ts";

import { CODER_RATE_CATALOG_VERSION } from "./pricing.ts";

import { renderComparison } from "./render-compare.ts";

import { buildResultRow, type BenchResultRow } from "./results-store.ts";

const fixture = (name: string): string =>

  fileURLToPath(new URL(`../fixtures/${name}`, import.meta.url));

const row = (name: string, lane: string, recordedAt: string): BenchResultRow =>

  buildResultRow(

    summarizeRun(

      readHarborJob(fixture(name), {

        suite: "tb2-cross-section",

        lane,

        rateCatalogVersion: CODER_RATE_CATALOG_VERSION,

      }),

    ),

    null,

    null,

    { recordedAt },

  );

describe("trend on one lane", () => {

  test("reads a regression between two consecutive runs as cost rising", () => {

    const first = row("priced-lane", "proxy", "2026-08-25T10:00:00.000Z");

    const second = row("regressed-lane", "proxy", "2026-08-25T11:00:00.000Z");

    const trends = compareRuns([first, second]).trends;

    expect(trends).toHaveLength(1);

    const step = trends[0]!.steps[0]!;

    expect(step.costDelta.direction).toBe("worse");

    expect(step.costDelta.absolute).toBeGreaterThan(0);

    expect(step.successRateDelta.direction).toBe("worse");

  });

  test("orders by when a run was recorded, not by the order rows were passed", () => {

    const later = row("regressed-lane", "proxy", "2026-08-25T11:00:00.000Z");

    const earlier = row("priced-lane", "proxy", "2026-08-25T10:00:00.000Z");

    const trend = compareRuns([later, earlier]).trends[0]!;

    expect(trend.rows.map((entry) => entry.recordedAt)).toEqual([

      "2026-08-25T10:00:00.000Z",

      "2026-08-25T11:00:00.000Z",

    ]);

    expect(trend.steps[0]!.costDelta.direction).toBe("worse");

  });

  test("produces no trend from a single run", () => {

    expect(compareRuns([row("priced-lane", "proxy", "2026-08-25T10:00:00.000Z")]).trends).toEqual(

      [],

    );

  });

});

describe("lane against lane", () => {

  test("measures every other lane against the baseline", () => {

    const proxy = row("priced-lane", "proxy", "2026-08-25T10:00:00.000Z");

    const local = row("regressed-lane", "local", "2026-08-25T10:05:00.000Z");

    const comparison = compareRuns([proxy, local]).laneComparisons[0]!;

    expect(comparison.baselineLane).toBe("proxy");

    const lane = comparison.lanes.find((entry) => entry.lane === "local")!;

    expect(lane.costDelta?.direction).toBe("worse");

    expect(comparison.lanes.find((entry) => entry.lane === "proxy")!.costDelta).toBeNull();

  });

  test("honours an explicit baseline lane", () => {

    const proxy = row("priced-lane", "proxy", "2026-08-25T10:00:00.000Z");

    const local = row("regressed-lane", "local", "2026-08-25T10:05:00.000Z");

    const comparison = compareRuns([proxy, local], { baselineLane: "local" }).laneComparisons[0]!;

    expect(comparison.baselineLane).toBe("local");

    expect(comparison.lanes.find((entry) => entry.lane === "proxy")!.costDelta?.direction).toBe(

      "better",

    );

  });

  test("compares each lane at its most recent run", () => {

    const stale = row("regressed-lane", "local", "2026-08-25T09:00:00.000Z");

    const fresh = row("priced-lane", "local", "2026-08-25T12:00:00.000Z");

    const proxy = row("priced-lane", "proxy", "2026-08-25T10:00:00.000Z");

    const comparison = compareRuns([stale, fresh, proxy], { baselineLane: "proxy" })

      .laneComparisons[0]!;

    expect(comparison.lanes.find((entry) => entry.lane === "local")!.row.recordedAt).toBe(

      "2026-08-25T12:00:00.000Z",

    );

  });

});

describe("deltas this suite refuses to state", () => {

  test("refuses a cost delta when either side could not be priced", () => {

    const unpriced = row("unpriced-lane", "local", "2026-08-25T10:00:00.000Z");

    const mixed = row("mixed-lane", "proxy", "2026-08-25T10:05:00.000Z");

    const comparison = compareRuns([unpriced, mixed]).laneComparisons[0]!;

    const delta = comparison.lanes.find((entry) => entry.lane === "local")!.costDelta!;

    expect(delta.direction).toBe("unpriced");

    expect(delta.absolute).toBeNull();

    expect(delta.reason).toContain("unmeasured cost");

  });

  test("refuses a success-rate delta when no verifier ran on one side", () => {

    const graded = row("priced-lane", "proxy", "2026-08-25T10:00:00.000Z");

    const crashed = { ...row("crashed-verifier", "local", "2026-08-25T10:05:00.000Z") };

    const forced: BenchResultRow = { ...crashed, suiteKey: graded.suiteKey, successRate: null };

    const comparison = compareRuns([graded, forced]).laneComparisons[0]!;

    const delta = comparison.lanes.find((entry) => entry.lane === "local")!.successRateDelta!;

    expect(delta.direction).toBe("unknown");

    expect(delta.absolute).toBeNull();

  });

});

describe("what a comparison will not fold together", () => {

  test("keeps runs of different task lists in different groups", () => {

    const four = row("priced-lane", "proxy", "2026-08-25T10:00:00.000Z");

    const three = row("unpriced-lane", "local", "2026-08-25T10:05:00.000Z");

    const comparison = compareRuns([four, three]);

    expect(comparison.laneComparisons).toEqual([]);

    expect(comparison.isolatedGroups).toBe(2);

  });

  test("scopes to one suite when asked", () => {

    const kept = row("priced-lane", "proxy", "2026-08-25T10:00:00.000Z");

    const other: BenchResultRow = {

      ...row("regressed-lane", "local", "2026-08-25T10:05:00.000Z"),

      suite: "other",

    };

    const comparison = compareRuns([kept, other], { suite: "tb2-cross-section" });

    expect(comparison.laneComparisons).toEqual([]);

    expect(comparison.isolatedGroups).toBe(1);

  });

  test("flags a lane comparison whose CLI version also varies", () => {

    const proxy = row("priced-lane", "proxy", "2026-08-25T10:00:00.000Z");

    const local: BenchResultRow = {

      ...row("regressed-lane", "local", "2026-08-25T10:05:00.000Z"),

      agentVersions: ["9.9.9"],

    };

    const comparison = compareRuns([proxy, local]).laneComparisons[0]!;

    expect(comparison.confounders.some((note) => note.includes("CLI version also varies"))).toBe(

      true,

    );

  });

  test("flags a trend priced from placeholder rates", () => {

    const first = row("priced-lane", "proxy", "2026-08-25T10:00:00.000Z");

    const second = row("regressed-lane", "proxy", "2026-08-25T11:00:00.000Z");

    const step = compareRuns([first, second]).trends[0]!.steps[0]!;

    expect(step.confounders.some((note) => note.includes("operator placeholder"))).toBe(true);

  });

});

describe("rendering", () => {

  test("prints an unpriced lane as unpriced rather than as a blank cell", () => {

    const unpriced = row("unpriced-lane", "local", "2026-08-25T10:00:00.000Z");

    const mixed = row("mixed-lane", "proxy", "2026-08-25T10:05:00.000Z");

    const text = renderComparison(compareRuns([unpriced, mixed]));

    expect(text).toContain("unpriced");

    expect(text).not.toContain("$0.0000");

  });

  test("says why an empty store has nothing to compare", () => {

    const text = renderComparison(compareRuns([]));

    expect(text).toContain("Nothing to compare");

    expect(text).toContain("The store holds no rows");

  });

  test("prints the trend a regression shows up in", () => {

    const first = row("priced-lane", "proxy", "2026-08-25T10:00:00.000Z");

    const second = row("regressed-lane", "proxy", "2026-08-25T11:00:00.000Z");

    const text = renderComparison(compareRuns([first, second]));

    expect(text).toContain("Trend — tb2-cross-section on the proxy lane");

    expect(text).toContain("worse");

  });

});

/**

 * Comparison: what two rows in the results store may honestly be said about

 * each other, and what they may not.

 *

 * #34 asks for two things off the same store. **Trend** is the same suite on

 * the same lane over time — the shape "two consecutive scheduled runs producing

 * comparable rows" needs, and the shape a regression shows up in. **Lane

 * comparison** is the same suite across lanes at one moment: the same tasks

 * through the proxy and through a local model, which is the comparative view

 * the issue says falls out for free.

 *

 * The rules that keep both honest:

 *

 * 1. **Only rows sharing a `suiteKey` are compared.** Different task lists are

 *    different measurements. Rows in other groups are reported as their own

 *    groups, never folded into one table with a footnote.

 * 2. **A cost delta needs both sides priced.** If either row's cost per

 *    accepted outcome is `null`, the delta is `unpriced`, not zero and not

 *    "improved". This is the same refusal the report makes about a single run,

 *    applied to the difference between two.

 * 3. **A comparison that varies two things says so.** Lane comparison assumes

 *    the CLI is held still while the lane changes; a group whose rows carry

 *    different agent versions is flagged `confounded`. Trend assumes the lane

 *    is held still while the CLI changes, so a trend that also changes model is

 *    flagged the same way. Neither is suppressed — a confounded comparison is

 *    often the only one available, and it is readable as long as it is labelled.

 * 4. **Deltas are directional facts, not verdicts.** Cost per accepted outcome

 *    going up is `worse` and success rate going up is `better`, and this module

 *    says which without deciding whether the run passes. That is the gate's

 *    job, and it already has a third verdict for the cases nothing measured.

 */

import type { BenchResultRow } from "./results-store.ts";

export type DeltaDirection = "better" | "worse" | "unchanged" | "unpriced" | "unknown";

export interface Delta {

  readonly from: number | null;

  readonly to: number | null;

  readonly absolute: number | null;

  /** Change as a fraction of `from`. `null` when `from` is 0 or unknown. */

  readonly relative: number | null;

  readonly direction: DeltaDirection;

  readonly reason: string;

}

export interface LaneRow {

  readonly lane: string;

  readonly row: BenchResultRow;

  /** `null` on the baseline lane itself. */

  readonly costDelta: Delta | null;

  readonly successRateDelta: Delta | null;

}

export interface LaneComparison {

  readonly suiteKey: string;

  readonly suite: string;

  readonly tasks: ReadonlyArray<string>;

  readonly baselineLane: string;

  readonly lanes: ReadonlyArray<LaneRow>;

  /** Set when something other than the lane also varies across these rows. */

  readonly confounders: ReadonlyArray<string>;

}

export interface TrendStep {

  readonly from: BenchResultRow;

  readonly to: BenchResultRow;

  readonly costDelta: Delta;

  readonly successRateDelta: Delta;

  readonly confounders: ReadonlyArray<string>;

}

export interface LaneTrend {

  readonly suiteKey: string;

  readonly suite: string;

  readonly lane: string;

  readonly rows: ReadonlyArray<BenchResultRow>;

  readonly steps: ReadonlyArray<TrendStep>;

}

export interface Comparison {

  readonly laneComparisons: ReadonlyArray<LaneComparison>;

  readonly trends: ReadonlyArray<LaneTrend>;

  /** Suite keys the store holds that no other row is comparable to. */

  readonly isolatedGroups: number;

}

/**

 * Compare a store's rows.

 *

 * `lower is better` is passed per metric rather than inferred, because the two

 * metrics here disagree: cost per accepted outcome improves downward and

 * success rate improves upward, and a single "delta" helper that guessed would

 * eventually guess wrong on a third metric.

 */

export const compareRuns = (

  rows: ReadonlyArray<BenchResultRow>,

  options: { readonly suite?: string; readonly baselineLane?: string } = {},

): Comparison => {

  const scoped =

    options.suite === undefined ? rows : rows.filter((row) => row.suite === options.suite);

  const groups = groupBy(scoped, (row) => row.suiteKey);

  const laneComparisons: Array<LaneComparison> = [];

  const trends: Array<LaneTrend> = [];

  let isolatedGroups = 0;

  for (const group of groups) {

    const latestByLane = latestPerLane(group);

    if (latestByLane.length < 2) {

      isolatedGroups += 1;

    } else {

      laneComparisons.push(laneComparisonOf(latestByLane, options.baselineLane));

    }

    for (const laneRows of groupBy(group, (row) => row.lane)) {

      if (laneRows.length < 2) continue;

      trends.push(trendOf(laneRows));

    }

  }

  return { laneComparisons, trends, isolatedGroups };

};

/**

 * One row per lane: the most recently recorded.

 *

 * A lane with several rows in the store is a lane that has been run more than

 * once, and the lane comparison wants where it stands now. Its history is not

 * discarded — that is what the trend reads.

 */

const latestPerLane = (rows: ReadonlyArray<BenchResultRow>): ReadonlyArray<BenchResultRow> =>

  groupBy(rows, (row) => row.lane)

    .map((laneRows) =>

      laneRows.reduce((latest, row) => (row.recordedAt >= latest.recordedAt ? row : latest)),

    )

    .toSorted((left, right) => left.lane.localeCompare(right.lane));

const laneComparisonOf = (

  rows: ReadonlyArray<BenchResultRow>,

  requestedBaseline: string | undefined,

): LaneComparison => {

  const baseline =

    rows.find((row) => row.lane === requestedBaseline) ??

    rows.find((row) => row.lane === "proxy") ??

    rows[0]!;

  return {

    suiteKey: baseline.suiteKey,

    suite: baseline.suite,

    tasks: baseline.tasks,

    baselineLane: baseline.lane,

    lanes: rows.map((row) => ({

      lane: row.lane,

      row,

      costDelta: row === baseline ? null : costDelta(baseline, row),

      successRateDelta: row === baseline ? null : successRateDelta(baseline, row),

    })),

    confounders: confoundersOf(rows, "lane"),

  };

};

const trendOf = (laneRows: ReadonlyArray<BenchResultRow>): LaneTrend => {

  const ordered = laneRows.toSorted((left, right) =>

    left.recordedAt.localeCompare(right.recordedAt),

  );

  const steps: Array<TrendStep> = [];

  for (let index = 1; index < ordered.length; index += 1) {

    const from = ordered[index - 1]!;

    const to = ordered[index]!;

    steps.push({

      from,

      to,

      costDelta: costDelta(from, to),

      successRateDelta: successRateDelta(from, to),

      confounders: confoundersOf([from, to], "recordedAt"),

    });

  }

  return {

    suiteKey: ordered[0]!.suiteKey,

    suite: ordered[0]!.suite,

    lane: ordered[0]!.lane,

    rows: ordered,

    steps,

  };

};

/**

 * Cost per accepted outcome, where up is worse.

 *

 * An unpriced side makes the delta `unpriced` and carries the disposition that

 * explains why, so a reader sees "the local lane bills no metered tokens"

 * rather than an empty cell.

 */

const costDelta = (from: BenchResultRow, to: BenchResultRow): Delta => {

  if (from.costPerAcceptedOutcomeUsd === null || to.costPerAcceptedOutcomeUsd === null) {

    const unpriced = from.costPerAcceptedOutcomeUsd === null ? from : to;

    return {

      from: from.costPerAcceptedOutcomeUsd,

      to: to.costPerAcceptedOutcomeUsd,

      absolute: null,

      relative: null,

      direction: "unpriced",

      reason: `no cost delta: the ${unpriced.lane} lane reports ${unpriced.costDisposition}, and a delta against an unmeasured cost would be an invention`,

    };

  }

  return numericDelta(from.costPerAcceptedOutcomeUsd, to.costPerAcceptedOutcomeUsd, "lower", {

    better: "cost per accepted outcome fell",

    worse: "cost per accepted outcome rose",

  });

};

/** Success rate, where up is better. `null` on either side is unknown. */

const successRateDelta = (from: BenchResultRow, to: BenchResultRow): Delta => {

  if (from.successRate === null || to.successRate === null) {

    return {

      from: from.successRate,

      to: to.successRate,

      absolute: null,

      relative: null,

      direction: "unknown",

      reason:

        "no success-rate delta: a run with no graded trials has no success rate rather than a zero one",

    };

  }

  return numericDelta(from.successRate, to.successRate, "higher", {

    better: "success rate rose",

    worse: "success rate fell",

  });

};

const numericDelta = (

  from: number,

  to: number,

  betterWhen: "lower" | "higher",

  reasons: { readonly better: string; readonly worse: string },

): Delta => {

  const absolute = to - from;

  const relative = from === 0 ? null : absolute / from;

  const improved = betterWhen === "lower" ? absolute < 0 : absolute > 0;

  return {

    from,

    to,

    absolute,

    relative,

    direction: absolute === 0 ? "unchanged" : improved ? "better" : "worse",

    reason: absolute === 0 ? "unchanged" : improved ? reasons.better : reasons.worse,

  };

};

/**

 * What else varies across the rows being compared, besides the axis being

 * compared on.

 *

 * Named rather than counted, so the reader can decide whether the confound

 * matters. Comparing two lanes on different CLI versions is still worth doing;

 * reading it as a clean lane comparison is not.

 */

const confoundersOf = (

  rows: ReadonlyArray<BenchResultRow>,

  axis: "lane" | "recordedAt",

): ReadonlyArray<string> => {

  const notes: Array<string> = [];

  const versions = distinct(rows.flatMap((row) => row.agentVersions));

  const models = distinct(rows.flatMap((row) => row.models));

  const bases = distinct(rows.map((row) => row.rateBasis));

  if (versions.length > 1) {

    notes.push(`CLI version also varies (${versions.join(", ")})`);

  }

  if (axis === "recordedAt" && models.length > 1) {

    notes.push(`model also varies (${models.join(", ")})`);

  }

  if (bases.includes("operator_placeholder")) {

    notes.push(

      "at least one row is priced from operator placeholder rates, so its cost is provisional",

    );

  }

  return notes;

};

const distinct = (values: ReadonlyArray<string | null>): ReadonlyArray<string> =>

  [...new Set(values.filter((value): value is string => value !== null))].toSorted();

const groupBy = <T>(

  values: ReadonlyArray<T>,

  key: (value: T) => string,

): ReadonlyArray<ReadonlyArray<T>> => {

  const buckets = new Map<string, Array<T>>();

  for (const value of values) {

    const bucket = buckets.get(key(value));

    if (bucket === undefined) buckets.set(key(value), [value]);

    else bucket.push(value);

  }

  return [...buckets.values()];

};

export {

  type Comparison,

  compareRuns,

  type Delta,

  type DeltaDirection,

  type LaneComparison,

  type LaneRow,

  type LaneTrend,