| 7 |
7
|
|
from the clone at `projects/repos/harbor` (v0.16.x, Apache-style OSS from
|
| 8 |
8
|
|
the Terminal-Bench creators).
|
| 9 |
9
|
|
|
|
10
|
+ |
## 0. The Gym framing
|
|
11
|
+ |
|
|
12
|
+ |
This is not only measurement. Harbor and Terminal-Bench are the **Gym**:
|
|
13
|
+ |
the training ground where the coder's capability work proves itself
|
|
14
|
+ |
against graded tasks, the way a model proves itself against an RL
|
|
15
|
+ |
environment. Concretely, three loops run through the same suites:
|
|
16
|
+ |
|
|
17
|
+ |
1. **Extend the coder.** A new capability lands with a Gym delta: the
|
|
18
|
+ |
suite score before and after, on the same recipe. A capability that
|
|
19
|
+ |
moves no score is questioned; a graded task class the coder cannot
|
|
20
|
+ |
pass yet is a capability backlog item.
|
|
21
|
+ |
2. **Test multi-model swapping.** The same suite per compute-mix lane —
|
|
22
|
+ |
each catalog model through the proxy, and Fable through the
|
|
23
|
+ |
foreign-harness delegation lane beside the stock claude-code adapter —
|
|
24
|
+ |
turns "which lane should this task class route to" into an
|
|
25
|
+ |
evidence-backed policy instead of a vibe, and eventually feeds the
|
|
26
|
+ |
routing policy itself.
|
|
27
|
+ |
3. **Test plugins.** The same suite with and without a plugin loaded;
|
|
28
|
+ |
provenance-stamped `tool.ran` steps in the ATIF make the delta
|
|
29
|
+ |
attributable to the exact plugin digest. A plugin's Gym delta is the
|
|
30
|
+ |
registry's first honest quality signal.
|
|
31
|
+ |
|
|
32
|
+ |
RewardKit closes the far end of the loop: the same trials become RL
|
|
33
|
+ |
rollouts when optimization work begins — candidates, never
|
|
34
|
+ |
auto-promotions, per the standing law.
|
|
35
|
+ |
|
|
36
|
+ |
**The surface** shipped with this plan (`310dac1`): `gym_runs` records
|
|
37
|
+ |
graded runs idempotently by recipe digest, the bench harness posts them
|
|
38
|
+ |
through `POST /api/v3/gym/runs`, and `/gym` — a sidebar entry under
|
|
39
|
+ |
Projects — renders the scoreboard. Operator-only on every path for now:
|
|
40
|
+ |
the operator allowlist is the whitelist, deliberately, rather than a
|
|
41
|
+ |
second gating mechanism; widening it later is a decision, not a default.
|
|
42
|
+ |
The Gym project (project 14) tracks the lane.
|
|
43
|
+ |
|
| 10 |
44
|
|
## 1. What Harbor is
|
| 11 |
45
|
|
|
| 12 |
46
|
|
Harbor is the official harness for Terminal-Bench 2.0 and a general
|