Outcome
Measure provider and fleet efficiency in accepted outcomes per kilowatt-hour and cost per accepted outcome, with latency and quality distributions.
Current behavior
Issue #43 tracks productive token usage, and #56 plans provider overflow routing. Neither metric identifies whether work was accepted or accounts for energy, verification, retries, and failed outcomes.
Source: docs/episode-triage.md, episodes 232, 237, 243, and 259.
Contract
- Define the accepted-outcome denominator through the project 3 contract.
- Attribute model, computer, verifier, checkpoint, and retry usage to one outcome attempt.
- Record measured energy where available and labeled estimates where direct measurement is unavailable.
- Publish P50, P90, and P99 latency, cost, energy, and acceptance rate by admitted provider and workflow class.
- Separate generation from verification cost and energy.
- Keep raw customer content, prompts, code, and private provider identifiers out of metrics.
- Give routing policy a bounded, freshness-stamped efficiency input; do not let the metric bypass quality or authority gates.
Acceptance criteria
- One completed, failed, and retried workflow produce distinct, reconcilable measurements.
- A result cannot improve the metric unless it reaches accepted-outcome state.
- Estimated energy is visibly different from measured energy and exposes its method.
- Provider totals reconcile with billing and usage receipts within a documented tolerance.
- The status and API projections remain privacy safe and avoid false precision.
Verification
Use a pinned workload corpus across at least two admitted providers. Test attribution, retries, verifier cost, stale estimates, privacy, reconciliation, and routing consumption.
Dependencies
Depends on #43, #56, and the accepted-outcome contract in project 3.
Outcome
Measure provider and fleet efficiency in accepted outcomes per kilowatt-hour and cost per accepted outcome, with latency and quality distributions.
Current behavior
Issue #43 tracks productive token usage, and #56 plans provider overflow routing. Neither metric identifies whether work was accepted or accounts for energy, verification, retries, and failed outcomes.
Source:
docs/episode-triage.md, episodes 232, 237, 243, and 259.Contract
Acceptance criteria
Verification
Use a pinned workload corpus across at least two admitted providers. Test attribution, retries, verifier cost, stale estimates, privacy, reconciliation, and routing consumption.
Dependencies
Depends on #43, #56, and the accepted-outcome contract in project 3.