Stand up the AgentX serving replay against the compute mix #33

Closed AtlantisPleb opened this 2d ago 2 comments

Outcome

A bench/ lane in this repository runs the AgentX trace replay (AIPerf inferencex-agentx-mvp, pinned dataset and harness versions) against the compute mix and produces a receipted, repeatable comparison: each model-catalog entry through POST /api/inference/proxy versus the same upstream model direct, at bounded concurrency ladders, reporting TTFT, TPOT, tokens per second, and cost per million tokens derived from the same grant usage records the leaderboard consumes.

Contract

Recipes are data: pinned dataset digest, harness version, target, concurrency ladder, duration. A run's exact command, config digest, and results land as artifacts in a bench-results forge repository, and the run itself is a thread so the transcript is the provenance.
InferenceX discipline adopted wholesale: scores and metrics validate against floors, evidence stages before any gate evaluates, and smoke/fast runs are marked bring-up evidence, never publishable.
Hosted-provider terms respected: bounded concurrency against third-party APIs; full ladders only against endpoints we own.
The proxy's whole-body SSE buffering is a known TTFT caveat — quantify it, do not paper over it.

Acceptance

One honest comparison table for two catalog models with proxy overhead quantified, reproduced by a second run from the same recipe.

Source

OpenAgentsInc/openagents.com docs 2026-08-24-benchmark-workbench-agentx.md section 3. Reference clone: projects/repos/InferenceX (read docs/eval-agentx-procedures.md there first).

AtlantisPleb opened this issue 2d ago
A AtlantisPleb Author 2d ago

The Gym's ingest door now exists (OpenAgentsInc/openagents.com 310dac1): when this bench lane completes a run, post the graded result to POST /api/v3/gym/runs — fields suite, agent, agent_version, model, lane, tasks_total, tasks_passed, input_tokens, output_tokens, cost_microusd, duration_seconds, recipe_digest (unique; a retried upload replays with 200 instead of duplicating), report (bounded map for per-task rows). Auth: a forge:write token whose account holds operator standing. The /gym page renders whatever lands there.
A AtlantisPleb Author 1d ago

Closing as wontdo for now: the AgentX serving replay is deferred until the compute mix question comes back around. Revisit later.
closed this as completed 1d ago