Measure accepted outcomes per kilowatt-hour and per dollar #83

Open AtlantisPleb opened this 23h ago

Outcome

Measure provider and fleet efficiency in accepted outcomes per kilowatt-hour and cost per accepted outcome, with latency and quality distributions.

Current behavior

Issue #43 tracks productive token usage, and #56 plans provider overflow routing. Neither metric identifies whether work was accepted or accounts for energy, verification, retries, and failed outcomes.

Source: docs/episode-triage.md, episodes 232, 237, 243, and 259.

Contract

  • Define the accepted-outcome denominator through the project 3 contract.
  • Attribute model, computer, verifier, checkpoint, and retry usage to one outcome attempt.
  • Record measured energy where available and labeled estimates where direct measurement is unavailable.
  • Publish P50, P90, and P99 latency, cost, energy, and acceptance rate by admitted provider and workflow class.
  • Separate generation from verification cost and energy.
  • Keep raw customer content, prompts, code, and private provider identifiers out of metrics.
  • Give routing policy a bounded, freshness-stamped efficiency input; do not let the metric bypass quality or authority gates.

Acceptance criteria

  • One completed, failed, and retried workflow produce distinct, reconcilable measurements.
  • A result cannot improve the metric unless it reaches accepted-outcome state.
  • Estimated energy is visibly different from measured energy and exposes its method.
  • Provider totals reconcile with billing and usage receipts within a documented tolerance.
  • The status and API projections remain privacy safe and avoid false precision.

Verification

Use a pinned workload corpus across at least two admitted providers. Test attribution, retries, verifier cost, stale estimates, privacy, reconciliation, and routing consumption.

Dependencies

Depends on #43, #56, and the accepted-outcome contract in project 3.

  1. AtlantisPleb opened this issue 23h ago
Sign in with GitHub to comment on this issue.