docs(coder): correct P3 against the code, and add K1 on premise provenance

846597cd81b9 · AtlantisPleb · · parent 3a3f31e8af23

docs(coder): correct P3 against the code, and add K1 on premise provenance

P3 claimed the capability tool carries each plugin's name and first sentence
in a twelve-slot standing description. Both harnesses do the opposite and
say so in comments: the tool is constant-size on purpose (#42), the catalog
is searched rather than enumerated, and a search returns at most five
matches ranked by overlap with each manifest's name and description. The
claim came from a design document in the openagents.com repository and
entered the ledger as adopted without being read against plugins.rs or
coder-capability.ts. The corrected entry names a real tunable surface: a
manifest's own description is what decides its rank.

Add K1 recording the pattern behind that error and two others found by
agents working from briefs written the same way — a tui.rs claim lifted from
a teardown of the TypeScript UI, and catalog lines described as literals
when they are manifest fields. A brief is evidence about its author's
reading, not about the code, and a premise now carries the file and symbol
it came from.

Also correct the plugin count in the analysis: thirteen in-tree, not twelve.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K7q2vA5LJroLTR6ZFbRq6j
Co-Authored-By
Claude Fable 5 <noreply@anthropic.com>

Deploy story

What this commit did to the running system — joined from the forge receipt chain, the part a commit page elsewhere cannot show.

Not deployed through the forge lane

No push, promotion, build, or deploy receipt references this commit (receipts are scanned over a bounded recent window). Changes shipped by full node replacement carry their proof in the release gate receipt instead.

Changed files

  • modified docs/coder/2026-08-26-dspy-gepa-coder-optimization.md
  • modified docs/coder/best-practices.md

Diff

2 files changed, +72 -17

docs/coder/2026-08-26-dspy-gepa-coder-optimization.md modified +14 -10

@@ -77,9 +77,9 @@ analogical. The correspondence:

77 77
| DSPy concept | Coder counterpart | State |
78 78
| --- | --- | --- |
79 79
| Signature (typed I/O contract) | Plugin manifest input/output schemas, host-validated | Shipped |
80
| Module (a callable unit) | WASM plugin, content-addressed by digest | Shipped, twelve in-tree |
80
| Module (a callable unit) | WASM plugin, content-addressed by digest | Shipped, thirteen in-tree |
81 81
| Program (composed modules) | The coder harness: system prompt + tools + catalog + plugins per turn | Shipped |
82
| Demos / instructions (optimizable text) | System prompt, tool descriptions, the twelve catalog lines, knowledge-base stances | Shipped, but **compiled into code** |
82
| Demos / instructions (optimizable text) | System prompt and tool descriptions (literals in both CLIs); plugin manifest descriptions (already data) | Shipped; the prompt and tool text still compiled in |
83 83
| Retrieval rail | `knowledge-base` plugin: corpus queried every turn, attached as a bracketed note | Shipped, with a governed promotion path |
84 84
| Metric | Harbor verifier + ATIF token metrics per trial | Shipped (the Gym) |
85 85
| Trace | ATIF trajectory per trial | Shipped |

@@ -93,11 +93,15 @@ Two entries deserve emphasis:

93 93
  reviewed-like-content with stable ids and a promotion path from system
94 94
  memory. This is the cleanest channel an optimizer could write
95 95
  candidates into — and the one with governance already in place.
96
- **The catalog lines are the selection policy DSPy would tune.** Twelve
97
  slots, one sentence each, contested (best practice P3). Which plugin a
98
  model reaches for on which task class is a function of that text today;
99
  the registry strategy's "selection loop" is this surface under
100
  optimization.
96
- **Manifest descriptions are the selection policy DSPy would tune.** The
97
  `capability` tool is constant-size by design (#42): it searches the
98
  catalog rather than enumerating it, returning at most five matches ranked
99
  by overlap with each manifest's `name` and `description`, plus a count of
100
  the remainder. So the tunable text is each plugin's own manifest
101
  description, and the policy is its rank on a given query — which is the
102
  registry strategy's "selection loop" already sitting in data rather than
103
  in code. Corrected 2026-08-26; this bullet previously described twelve
104
  standing catalog slots, which neither harness has (best practice K1).
101 105
102 106
What we do not have — and did not have in DSE either — is the optimizer.
103 107
The rest of the machine is more complete than DSE's ever was, because the

@@ -113,7 +117,7 @@ Every argument has a concrete referent here:

113 117
114 118
| GEPA argument | Gym referent |
115 119
| --- | --- |
116
| `seed_candidate` | The current text surfaces: system prompt, tool descriptions, catalog lines, stances |
120
| `seed_candidate` | The current text surfaces: system prompt, tool descriptions, manifest descriptions, stances |
117 121
| `trainset` (dev) | `tb2-quick`, plus `owned-closed-issues` once its environments exist |
118 122
| `valset` (holdout) | `tb2-cross-section` — run sparingly, exactly as the runbook already directs |
119 123
| metric | Verifier acceptance minus a token-cost term, read from the trial's `result.json` and ATIF metrics |

@@ -161,7 +165,7 @@ adapter is Python, and `bench/` is where both live. Concretely:

161 165
  data the optimizer can diff and the build can verify, rather than string
162 166
  literals inside `crates/coder-lite` and `packages/openagents-cli`. The
163 167
  knowledge base already has this shape (corpus file → build → digest);
164
  the system prompt, tool descriptions, and catalog lines need the same
168
  the system prompt and tool descriptions need the same
165 169
  treatment. This staging is also what lets a human cycle (#119) and an
166 170
  optimizer cycle produce identical artifact shapes.
167 171

@@ -179,7 +183,7 @@ Two new tracker issues, sequenced against the existing loop set

179 183
180 184
1. **Stage the optimizable text surfaces as artifacts**
181 185
   (OpenAgentsInc/openagents#122). Extract system prompt, tool
182
   descriptions, and catalog lines into versioned data with digests,
186
   descriptions, and manifest descriptions into versioned data with digests,
183 187
   consumed by both CLIs; define the candidate-diff format. Prerequisite
184 188
   for the optimizer; makes #119's manual levers diffable artifacts too.
185 189
2. **A GEPA lane over the Gym** (OpenAgentsInc/openagents#123).
docs/coder/best-practices.md modified +58 -7

@@ -160,13 +160,28 @@ coder's core tools under the permission profile.

160 160
refuses undeclared imports and stale digests; review checks the manifest
161 161
against the shape rules.
162 162
163
### P3. A catalog line is contested space — `adopted`
164
165
The capability tool carries each installed plugin's name and first sentence,
166
capped at twelve. A plugin competes for that slot with everything installed;
167
its first sentence must earn a model's attention on the turns where the
168
plugin applies.
169
**Provenance:** harvest. **Detection:** review question when the A/B shows
163
### P3. A manifest's own words are its ranking, and rank is contested — `adopted`
164
165
The `capability` tool is constant-size on purpose (#42): the catalog is
166
searched, never enumerated in the standing prompt, so the prompt does not
167
grow as plugins are installed. A search returns at most `SEARCH_LIMIT` (5)
168
matches, ranked by overlap with each manifest's `name` and `description`,
169
plus a count of the remainder. A plugin does not compete for a standing
170
slot — it competes for rank in a five-result search, and the text deciding
171
that rank is its own manifest description. Write it as the words a model
172
would use when it needs the plugin.
173
174
**Corrected 2026-08-26.** This entry previously read "The capability tool
175
carries each installed plugin's name and first sentence, capped at twelve."
176
That is false in both harnesses. It was carried from a design document in
177
the openagents.com repository into the ledger as `adopted` without ever
178
being read against the code — the exact laundering the provenance field
179
exists to prevent (see K1).
180
181
**Provenance:** `crates/openagents-cli/src/plugins.rs` (`SEARCH_LIMIT`,
182
`capability_tool_definition`) and
183
`packages/openagents-cli/src/coder-capability.ts`, both read 2026-08-26.
184
**Detection:** review question when the A/B shows
170 185
a plugin installed but never invoked on tasks it should have served.
171 186
172 187
## Optimization

@@ -234,6 +249,42 @@ suite cannot carry (ledger M5).

234 249
**Provenance:** dspy-gepa. **Detection:** candidate evidence without a
235 250
stated trial count is not a promotion argument.
236 251
252
## Knowledge
253
254
### K1. A claim's provenance is where it was verified, not where it was read — `adopted`
255
256
A design document, an issue body, a teardown, or a task brief is evidence
257
about its author's reading. It is not evidence about the code. Before a
258
claim enters this ledger, an issue's premises, or a brief handed to another
259
agent, read it against the source and cite the file and symbol. A claim
260
sourced to another document must say so and stays `proposed` until someone
261
checks it.
262
263
**Provenance:** three failures on 2026-08-26, all in briefs and ledger
264
entries written from a fast survey and handed out as authoritative:
265
266
- P3 asserted a twelve-slot catalog carrying each plugin's first sentence.
267
  Both harnesses are constant-size by design and search the catalog instead;
268
  the claim came from a document in the openagents.com repository and was
269
  never read against `plugins.rs` or `coder-capability.ts`.
270
- The #117 brief asserted a `tui.rs` fixed-row template with fleet/composer/
271
  spacer constants. That sentence describes `coder-ui.ts`; it was lifted
272
  from a Claude Code teardown comparing the TypeScript UI and attached to a
273
  Rust file. Four further premises in the same brief also did not hold, one
274
  of which had the port direction backwards — the TypeScript composer is a
275
  flat string with no cursor index, history, or completion, so capabilities
276
  the brief called owed were already present.
277
- The #122 brief asserted catalog lines were literals in the two CLIs. They
278
  are `description` fields in `plugins/<id>/manifest.json`; grep for one
279
  returns zero source files. The same brief said twelve plugins; there are
280
  thirteen.
281
282
**Detection:** every premise in an issue or brief carries the file and
283
symbol it was read from. A premise without one is a question for the agent
284
to verify, and must be marked as such rather than stated flat. When an agent
285
reports a premise false, the correction lands in the source document — an
286
uncorrected brief re-teaches the error to the next reader.
287
237 288
## Repository
238 289
239 290
### R1. Pack with `pnpm pack`, never `npm pack` — `adopted`

This page updates live while a promote is in flight · changelog