Efficiency
Does routing work through OpenAgents beat handing it straight to Claude Code or Codex? The same pinned tasks run through each, an independent check decides every pass, and these are the numbers, wins and losses alike.
Findings
- Headline: against raw Claude Code, the routed default was 68% cheaper and took 83% longer, so routing does not win on time yet (passed 21/21 against 21/21).
- raw Codex against raw Claude Code on 7 tasks: cost 0.37× (0.33–0.41) — 63% cheaper; time 1.11× (0.97–1.26) — no measurable difference.
- routed default (Codex, recipe on) against raw Claude Code on 7 tasks: cost 0.32× (0.28–0.35) — 68% cheaper; time 1.83× (1.61–2.07) — 83% slower.
- lean Claude Code session against raw Claude Code on 7 tasks: cost 0.45× (0.39–0.52) — 55% cheaper; time 0.86× (0.74–0.98) — 14% faster.
- routed default (Codex, recipe on) against raw Codex on 7 tasks: cost 0.85× (0.78–0.92) — 15% cheaper; time 1.65× (1.52–1.79) — 65% slower.
A ratio below 1 favors the arm. “No measurable difference” means the 95% interval includes 1.
2026-10-03b
Standing study (#10162): raw Claude Code, raw Codex, routed default, lean session after the start-up cut (#10279), at 0565629714 · 84 runs on 7 tasks · rows and write-up
| Arm | Runs | Passed (95%) | Cost per checked result (95%) | Time to a checked result, median (95%) | Cost against raw Claude Code (95%) | Time against raw Claude Code (95%) |
|---|
| raw Claude Code | 21 | 21/21 (85–100%) | $0.297 ($0.273–$0.326) | 41 s (35 s–44 s) | 1 (baseline) | 1 (baseline) |
|---|
| raw Codex | 21 | 21/21 (85–100%) | $0.110 ($0.105–$0.117) | 37 s (34 s–40 s) | 0.37× (0.33–0.41) | 1.11× (0.97–1.26) |
|---|
| routed default (Codex, recipe on) | 21 | 21/21 (85–100%) | $0.094 ($0.089–$0.100) | 75 s (70 s–77 s) | 0.32× (0.28–0.35) | 1.83× (1.61–2.07) |
|---|
| lean Claude Code session | 21 | 21/21 (85–100%) | $0.133 ($0.120–$0.150) | 34 s (29 s–39 s) | 0.45× (0.39–0.52) | 0.86× (0.74–0.98) |
|---|
Class: repository
| Arm | Runs | Passed (95%) | Cost per checked result (95%) | Time to a checked result, median (95%) |
|---|
| raw Claude Code | 9 | 9/9 (70–100%) | $0.254 ($0.239–$0.276) | 41 s (35 s–44 s) |
|---|
| raw Codex | 9 | 9/9 (70–100%) | $0.062 ($0.057–$0.066) | 37 s (34 s–40 s) |
|---|
| routed default (Codex, recipe on) | 9 | 9/9 (70–100%) | $0.062 ($0.055–$0.069) | 74 s (70 s–77 s) |
|---|
| lean Claude Code session | 9 | 9/9 (70–100%) | $0.088 ($0.083–$0.094) | 33 s (29 s–39 s) |
|---|
Class: terminal-bench
| Arm | Runs | Passed (95%) | Cost per checked result (95%) | Time to a checked result, median (95%) |
|---|
| raw Claude Code | 12 | 12/12 (76–100%) | $0.329 ($0.289–$0.381) | 37 s (36 s–73 s) |
|---|
| raw Codex | 12 | 12/12 (76–100%) | $0.147 ($0.138–$0.158) | 55 s (53 s–74 s) |
|---|
| routed default (Codex, recipe on) | 12 | 12/12 (76–100%) | $0.118 ($0.110–$0.126) | 106 s (96 s–136 s) |
|---|
| lean Claude Code session | 12 | 12/12 (76–100%) | $0.167 ($0.145–$0.196) | 43 s (40 s–53 s) |
|---|
Earlier studies
Each compared only within itself: the code, arms, and tasks changed between them.
2026-10-03
Standing study (#10162): raw Claude Code, raw Codex, routed default, lean session, at 6b4dbe827d · 84 runs on 7 tasks · rows and write-up
| Arm | Runs | Passed (95%) | Cost per checked result (95%) | Time to a checked result, median (95%) | Cost against raw Claude Code (95%) | Time against raw Claude Code (95%) |
|---|
| raw Claude Code | 21 | 21/21 (85–100%) | $0.270 ($0.259–$0.279) | 42 s (32 s–53 s) | 1 (baseline) | 1 (baseline) |
|---|
| raw Codex | 21 | 21/21 (85–100%) | $0.102 ($0.098–$0.107) | 38 s (37 s–42 s) | 0.38× (0.36–0.40) | 1.17× (1.09–1.26) |
|---|
| routed default (Codex, recipe on) | 21 | 21/21 (85–100%) | $0.085 ($0.082–$0.089) | 79 s (73 s–83 s) | 0.32× (0.30–0.33) | 1.90× (1.78–2.04) |
|---|
| lean Claude Code session | 21 | 21/21 (85–100%) | $0.169 ($0.153–$0.192) | 44 s (44 s–50 s) | 0.63× (0.56–0.71) | 1.34× (1.19–1.53) |
|---|
2026-10-02-prompt-cache
#10244: routed Claude loop after the prompt-cache fix, at 9513636c45 · 18 runs on 3 tasks · rows and write-up
| Arm | Runs | Passed (95%) | Cost per checked result (95%) | Time to a checked result, median (95%) | Cost against raw Claude Code (95%) | Time against raw Claude Code (95%) |
|---|
| raw Claude Code | 9 | 9/9 (70–100%) | $0.309 ($0.269–$0.381) | 54 s (51 s–67 s) | 1 (baseline) | 1 (baseline) |
|---|
| routed Claude loop | 9 | 9/9 (70–100%) | $0.249 ($0.232–$0.263) | 104 s (78 s–107 s) | 0.81× (0.66–0.95) | 1.38× (1.02–1.82) |
|---|
2026-10-02-shadow-baseline
#10209: routed Microcoder loop, recipe on and off, at feb4bc8270; lean Claude session (#10246) at 6b4dbe827d · 189 runs on 7 tasks · rows and write-up
| Arm | Runs | Passed (95%) | Cost per checked result (95%) | Time to a checked result, median (95%) | Cost against raw Claude Code (95%) | Time against raw Claude Code (95%) |
|---|
| raw Claude Code | 21 | 21/21 (85–100%) | $0.297 ($0.288–$0.306) | 45 s (37 s–54 s) | 1 (baseline) | 1 (baseline) |
|---|
| raw Codex | 21 | 21/21 (85–100%) | $0.113 ($0.109–$0.119) | 62 s (56 s–67 s) | 0.38× (0.36–0.40) | 1.50× (1.17–1.85) |
|---|
| lean Claude Code session | 21 | 21/21 (85–100%) | $0.181 ($0.169–$0.191) | 46 s (40 s–46 s) | 0.61× (0.57–0.65) | 1.09× (0.82–1.42) |
|---|
| routed Claude loop, recipe off | 21 | 20/21 (77–99%) | $0.500 ($0.434–$0.574) | 83 s (60 s–92 s) | 1.60× (1.44–1.75) | 1.47× (1.15–1.86) |
|---|
| routed Claude loop | 21 | 21/21 (85–100%) | $0.501 ($0.434–$0.579) | 64 s (53 s–72 s) | 1.68× (1.46–1.95) | 1.39× (1.03–1.90) |
|---|
| routed-codex-loop | 21 | 21/21 (85–100%) | $0.089 ($0.083–$0.097) | 78 s (73 s–89 s) | 0.30× (0.28–0.33) | 1.50× (1.18–1.86) |
|---|
| routed Codex, recipe off | 21 | 21/21 (85–100%) | $0.161 ($0.153–$0.170) | 99 s (85 s–114 s) | 0.54× (0.51–0.58) | 1.83× (1.40–2.37) |
|---|
| routed Codex | 21 | 21/21 (85–100%) | $0.260 ($0.238–$0.284) | 124 s (112 s–152 s) | 0.87× (0.79–0.96) | 2.61× (2.03–3.31) |
|---|
| routed-codex-session | 21 | 21/21 (85–100%) | $0.107 ($0.102–$0.111) | 68 s (68 s–82 s) | 0.36× (0.34–0.38) | 1.58× (1.24–1.96) |
|---|
Decisions
Coder turns Jev's probabilities into actions at fixed thresholds: whether a task is hard, which checks to keep, which guidance to include. Each decision is now recorded with its probability and joined to the run's independent check. A nightly refit proposes a new threshold and adopts it only when it beats the default on held-out runs.
Not enough data yet: no decision has 20 checked runs. Accuracy and reliability appear here once one does.
| Setting | Default | In effect | Labelled runs | Status |
|---|
| evidence.yes (default never measured) | 0.5 | 0.5 | 0 | not enough data yet: needs 40 fit and 20 held-out labelled samples |
| issue_turn.plain (default never measured) | 0.2 | 0.2 | 0 | not enough data yet: needs 40 fit and 20 held-out labelled samples |
| recipe.check_keep | 0.7 | 0.7 | 0 | not enough data yet: needs 40 fit and 20 held-out labelled samples |
| recipe.hard | 0.8 | 0.8 | 0 | not enough data yet: needs 40 fit and 20 held-out labelled samples |
| system.select (default never measured) | 0.5 | 0.5 | 0 | not enough data yet: needs 40 fit and 20 held-out labelled samples |
| terminal.asks_only | 0.6 | 0.6 | 0 | not enough data yet: needs 40 fit and 20 held-out labelled samples |
Accuracy is measured against the run's independent check, a proxy for whether each decision was right; the hard decision is measured against whether the run took more than 10 minutes. See the calibration plan.
Method
- Tasks. A pinned set: Terminal-Bench 2.1 tasks moved onto a host, and real fixes from public repositories merged after the models' training cutoff, each asked as an issue. The set is named in every row, and rows from different sets are never pooled.
- Arms. Raw Claude Code (
claude -p on its own defaults), raw Codex (codex exec on the routed default's model and effort), and OpenAgents' routed paths (openagents chat send): the shipped default, and Claude Code run as one lean, briefed session. - Checks. A pass is the task's own tests or the fix commit's own test file, run afterwards on the work the run left, never the agent's own claim. Every check fails on the untouched task and passes on the reference fix.
- Cost. List price for the engine plus Jev, as each run recorded it, not a bill. Cost per checked result is everything an arm spent, failures included, over its passes. The chat router's own model call is not metered and not counted.
- Time. Wall time from the command's start to the result settling. Time to a checked result is the median over passing runs.
- Intervals. Pass rates use Wilson's 95% interval. Costs, times, and ratios use a 95% bootstrap that resamples trials within each task. Ratios against raw Claude Code are sums over tasks of per-task means.
Read the cost audit, the first shadow-baseline measurement, and the standing study's runbook; the rows behind every number on this page are in the repository. On your own computer, openagents efficiency adds your routed runs and shadow baselines.