Perform the five exit rehearsals that had never been run

619260d4e546 · AtlantisPleb · · parent 22ef99926d44

Perform the five exit rehearsals that had never been run

Rehearsals 2 through 6 ran against the live forge on revision
46cf8a5a. Every row of the status table now names a date and a
result, and docs/2026-08-25-forge-exit-rehearsals-2-to-6.md records
what each step produced and which half of it ran against production.

Three steps mutate state — rehearsal 2's tampering, rehearsal 3's
rebuild, and any rotation in rehearsal 4 — so they ran against a
forge the test process builds and are labeled local. A rehearsal
that cannot safely run its destructive step against production has
a permanent gap, and saying so beats implying the step was taken.

Three defects were filed rather than repaired inside the record.
#251: the three nodes disagree about the same repository, because
the WAL is shared and the projection is node-local, so a node that
has not replayed yet reports the two findings that mean tampering.
That blocks #179's scheduled verification pass. #253: production
never provisioned MACHINE_TOKEN_ENCRYPTION_KEY, so the pairing vault
still runs on the GitHub vault's key and VAULT-001's separation is
not deployed. #252: POST /api/v1/responses reached the route
authority unclassified and EXIT-001's derived coverage was red on
main.

#252 is answered here because it is a red on main rather than a
finding to carry: the stub reaches no repo and no schema, so the
family is not_user_data, with a note saying the entry is revisited
the day a provider stands behind the route.

Rehearsal 4 turns out to have an executable proof after all —
test/openagents/forge/key_rotation_test.exs — which closes #180's
third acceptance criterion. Rehearsal 5's stated limit is now
measured: the WAL bucket has no object versioning and seven days of
soft delete, so the storage keeps no history of itself.

Closes #180.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KnhfrafYx5ZGaMbzZEJQ2d
Co-Authored-By
Claude Opus 5 (1M context) <noreply@anthropic.com>
Closes
#180

Deploy story

What this commit did to the running system — joined from the forge receipt chain, the part a commit page elsewhere cannot show.

pushed
by user · WAL seq 395 · 2026-08-25T15:17:56.935192Z

Changed files

  • added docs/2026-08-25-forge-exit-rehearsals-2-to-6.md
  • modified docs/forge-exit-rehearsals.md
  • modified lib/openagents/data_rights/export_inventory.ex

Diff

3 files changed, +612 -17

docs/2026-08-25-forge-exit-rehearsals-2-to-6.md added +462

@@ -0,0 +1,462 @@

1
# Forge exit rehearsals 2 through 6, performed
2
3
**Date:** 2026-08-25
4
**Issue:** #180
5
**Runbook:** `docs/forge-exit-rehearsals.md` defines the six rehearsals. This
6
document records what running five of them produced.
7
**Companion:** `docs/forge-operator-independence.md` states the trust boundary
8
they rehearse against.
9
10
Rehearsal 1 was performed on 2026-08-23, failed, and was filed as #179. The
11
other five had never been run outside the test suite. This record covers
12
rehearsals 2 through 6 against the live forge on revision
13
`46cf8a5aea3791936c22e82c145a9a8dd734374d`, deployed on all three fleet nodes.
14
15
## How to read this record
16
17
Every rehearsal is split into what ran against the live forge and what ran
18
against a forge a test process builds. The split matters, because #179 exists
19
precisely because six invariants were green against a forge the test built
20
while the live one could not serve a full clone. A claim proven only in a test
21
process is labeled as such here rather than being allowed to stand for the
22
live one.
23
24
Rehearsals must be read-only against production. Two documented steps mutate
25
state — rehearsal 3's `rebuild/1`, which discards a projection, and every
26
tamper in rehearsal 2's step 2 — so both were performed locally and are
27
labeled local. Nothing in this record deleted a repository, force-pushed,
28
mirrored, rotated a credential, or wrote to the WAL.
29
30
## Summary
31
32
| Rehearsal | Verdict | Defects filed |
33
| --- | --- | --- |
34
| 2. Detect a forged, missing, reordered, or mismatched receipt | Pass, with one new defect | #251; #190 still reproduces |
35
| 3. Mirror divergence | Pass, and it confirms #188 exactly | none new |
36
| 4. Key rotation | Pass in the test process, one deployment gap | #253 |
37
| 5. Operator loss | Recovery passes; the WAL remains unobtainable | none new |
38
| 6. Partial export | Pass live, red on `main` | #252 |
39
40
Three defects were filed. None of them was repaired inside this record.
41
42
## Rehearsal 2: detect a forged, missing, reordered, or mismatched receipt
43
44
### Step 1, live: the log and the projection agree
45
46
On one fleet node:
47
48
```
49
OpenAgents.Forge.Verification.verify("OpenAgentsInc/openagents.com")
50
#=> {:ok,
51
#=>  %{repo: "OpenAgentsInc/openagents.com",
52
#=>    storage_key: "ecd89cf6-f602-479f-9f47-266307345aaa",
53
#=>    entries: 382,
54
#=>    chained_from: 279,
55
#=>    head: %{seq: 381, link: "f663eb135c5fd51177024b803b0d7732930ae9a643fc9a8e7022d963a36e4949"},
56
#=>    findings: []}}
57
```
58
59
Reading the log directly gives the same shape: 382 entries, 103 of them
60
carrying a link, which is the contiguous suffix 279 through 381. Entries below
61
279 predate the chain, which `EXIT-005` calls history rather than tampering.
62
63
**Verdict: pass.**
64
65
### Step 1, live: #190 still reproduces
66
67
The rehearsal tells an operator to pass `{storage_key}`. The value the forge
68
offers for that placeholder is still wrong:
69
70
```
71
OpenAgents.Forge.Repos.allowed_repos()
72
#=> ["openagents.com"]
73
74
OpenAgents.Forge.Verification.verify("openagents.com")
75
#=> {:error, %{entries: 0, storage_key: "openagents.com",
76
#=>            findings: [%{code: "wal_unreadable", detail: %{"reason" => ":not_found"}}]}}
77
```
78
79
`verify/2` now accepts an `owner/name` path, which is #190's first acceptance
80
criterion and the reason step 1 above works at all. The other two are
81
outstanding. `allowed_repos/0` still returns a name no verifier can use, and
82
`/var/lib/openagents/forge/repos` still holds `openagents.com.git` beside the
83
six UUID-keyed directories that serve real repositories. An operator following
84
the runbook literally still reaches the report that means "your write-ahead log
85
is missing" for a log that is intact.
86
87
**Verdict: partial, tracked by #190.** No new issue.
88
89
### Step 3, live: the published anchor
90
91
`GET /.well-known/openagents-forge-anchor.json` answers `200`. The document
92
fetched over HTTPS before the check carried `anchor_seq: 2`, `published_at:
93
2026-08-25T13:52:41Z`, `signed: false`, `witnessed: false`, and this
94
repository at `head_seq: 375`, `head_link: "19c7a2c5…"`, `chained_from: 279`,
95
over 376 entries.
96
97
By the time the check ran the log had advanced past that sequence, which is
98
what makes the check meaningful: the anchor commits to a prefix, and the
99
verifier was asked about it afterwards.
100
101
| Anchor supplied | Result |
102
| --- | --- |
103
| `%{seq: 375, link: "19c7a2c5…"}`, the published value | `{:ok, findings: []}` |
104
| The same sequence with one hex digit changed | `anchor_mismatch`, naming the anchored and recorded values |
105
| `%{seq: 100, …}`, below `chained_from` | `anchor_unreachable`, `"entry carries no link"` |
106
| `%{seq: 99_999_999, …}` | `anchor_unreachable`, `"no entry at this sequence"` |
107
108
The option is honored rather than accepted and ignored, and the two
109
unreachable cases are distinguished by reason rather than collapsed.
110
111
**Verdict: pass.**
112
113
### Step 2, local: each finding code
114
115
Step 2 asks that each way the served state can disagree with the record be
116
reported distinctly. Producing those disagreements means tampering with a
117
repository, so this ran against a forge the test process builds:
118
119
```sh
120
mix test test/openagents/forge/independence_test.exs
121
```
122
123
Thirty-seven tests, all green. They name each code the step lists:
124
`entry_digest_mismatch` from a rewritten entry, `entry_sequence_broken` from a
125
removed one, `entry_object_missing` from an entry the store cannot produce,
126
`served_refs_diverged` from a ref moved or added without a push,
127
`object_missing` and `object_unreachable` from a lost cache and an unwalkable
128
history, and `chain_link_mismatch` and `chain_link_missing` from a rewrite that
129
leaves the chain alone and a link removed mid-log.
130
131
**Verdict: pass, in the test process only.** No tamper was performed against
132
the live forge, and this record does not claim one was.
133
134
### New defect: the three nodes disagree, and lag reads as tampering
135
136
Running step 1 on one node passes. Running it on all three at the same moment
137
gives three different answers.
138
139
| Time (UTC) | Repository | Node A | Node B | Node C |
140
| --- | --- | --- | --- | --- |
141
| 14:38–14:40 | `OpenAgentsInc/openagents.com` | clean, 382 entries | diverged, 383 | clean, 383 |
142
| 14:38–14:40 | `OpenAgentsInc/openagents` | diverged | clean | diverged |
143
| 14:57–14:58 | `OpenAgentsInc/openagents.com` | clean, 391 | clean, 392 | diverged, 393 |
144
| 14:57–14:58 | `OpenAgentsInc/openagents` | diverged | clean | diverged twice |
145
146
"Diverged" is `served_refs_diverged` plus `object_missing` — the two findings
147
that mean the forge is serving something other than what was pushed.
148
149
The cause is that the WAL is shared and the projection is not. The adapter is
150
`OpenAgents.Forge.WAL.Gcs` against one bucket; the bare repositories sit on
151
each node's own disk. A node that has not replayed an entry yet is
152
indistinguishable, to `verify/2`, from a node whose projection was altered.
153
154
It converges without help. Entry 136 of `OpenAgentsInc/openagents` was written
155
at 14:32:33Z; node A still reported the divergence at 14:38:25Z and
156
was clean by 14:42:35Z, with its WAL head and its on-disk `refs/heads/main`
157
both reading `4773472f`. Eight consecutive `git ls-remote` calls against the
158
same repository returned one sha, so no client-visible flapping was observed.
159
160
This matters most for #179, whose third acceptance criterion asks for a
161
scheduled verification pass publishing its findings. Built today, that pass
162
would publish tampering findings for a healthy forge most of the time.
163
164
Filed as **#251**.
165
166
## Rehearsal 3: mirror divergence
167
168
### Step 1, live: compare heads
169
170
```sh
171
git ls-remote https://openagents.com/OpenAgentsInc/openagents.com.git > forge-refs.txt
172
git ls-remote https://github.com/OpenAgentsInc/openagents.com.git > gh-refs.txt
173
diff <(sort forge-refs.txt) <(sort gh-refs.txt)
174
```
175
176
Twenty-five refs on each side, byte-identical, `refs/heads/main` at
177
`e57f5ea8b1666ccb69fc4c626f54cf41b34a0ebe` on both.
178
179
**Verdict: pass.** Ref maps agree.
180
181
### Step 1, live: histories do not agree, and #188 is confirmed at today's numbers
182
183
Identical ref maps are not identical history. Both sides were cloned in full:
184
185
| Source | Commits on `main` | `git fsck` | Root commit | Holds `c91327d6` |
186
| --- | --- | --- | --- | --- |
187
| The forge | 423 | clean | `eda094c6` | no |
188
| GitHub mirror | 730 | clean | `a352f78e` | yes, with 307 commits behind it |
189
190
The forge clone now succeeds, which is #179's first acceptance criterion. It
191
carries a `.git/shallow` file naming five boundaries, so the clone is honest
192
about where history stops rather than aborting. `git fsck` is clean.
193
194
The 307-commit gap is exactly what #188 records: 730 minus 423. Those commits
195
are on the mirror and in no WAL, so no rebuild produces them. For everything
196
pushed since the seed the forge is canonical and the mirror is lossy; for
197
everything before it the relation is inverted.
198
199
### Step 2, live: a mirror is configured
200
201
The live node answers:
202
203
```
204
Application.get_env(:openagents, :forge_mirror_urls, %{}) |> Map.keys()
205
#=> ["openagents.com"]
206
```
207
208
and `GET /api/status` publishes `forge.mirror` as
209
`{"repo": "openagents.com", "state": "current", "lagging_minutes": null}`.
210
211
Both halves of #188's first finding hold. `INVARIANTS.md` and `CLAUDE.md` have
212
since been corrected, and `test/openagents/forge/independence_test.exs` carries
213
"does not claim a mirror is unconfigured while runtime configures one", which
214
reads the configuration rather than asserting the empty default. That is #188's
215
first acceptance criterion met; its second and third remain open.
216
217
**Verdict: pass, and it reproduces #188.** No new issue.
218
219
### Step 3, local: recovery comes from the WAL
220
221
Step 3 calls `OpenAgents.Forge.Sync.rebuild/1`, which discards the local bare
222
repository and re-materializes it from sequence zero. That is a mutation, so it
223
was **not run against production**. Its behavior was exercised locally instead,
224
by three tests in `test/openagents/forge/independence_test.exs`:
225
226
- "the WAL rebuilds the repository and re-derives receipts the database lost"
227
- "the rebuild path never reads the mirror"
228
- "the mirror restores source and cannot restore the push record"
229
230
All three pass.
231
232
**Verdict: pass, in the test process only.** Whether `rebuild/1` succeeds
233
against the live 423-commit projection is untested, and this record does not
234
claim it.
235
236
## Rehearsal 4: key rotation
237
238
The runbook records this rehearsal's executable proof as "None". That is stale.
239
`test/openagents/forge/key_rotation_test.exs` exists and covers both halves of
240
what #180 asks for. Twelve tests, all green.
241
242
### Local: no rotation invalidates a receipt
243
244
- "the WAL chain link is unkeyed and reproducible from the entry alone"
245
- "the verifier was compiled against no secret and no vault"
246
247
A forge receipt depends on no key, so the first half of #180's contract holds
248
by construction rather than by procedure.
249
250
### Local: a rotation in the wrong order is refused
251
252
Four tests cover the reputation issuer key's ordering rule:
253
254
- retiring forward keeps an attestation verified and the successor issues
255
- retiring backward is refused, and the refusal names the attestation it would
256
  have unverified (#191)
257
- a key that signed nothing is bounded by its own activation
258
- a future-dated retirement beyond the clock-skew allowance is refused
259
260
There is no override, which is the point: a backdated retirement is the single
261
`UPDATE` that silently unverifies published signatures.
262
263
### Local: the three vaults rotate differently, and each says how
264
265
- the GitHub vault opens envelopes sealed under a retired key
266
- it refuses an envelope whose key left the keyring
267
- a GitHub key rotation does not orphan the machine pairing vault (#192)
268
- the voice recording vault has no keyring, so its key cannot rotate without
269
  stranding what it sealed
270
271
### Live, read-only: what is actually deployed
272
273
Read from the node without printing any key material:
274
275
| Secret | Live state |
276
| --- | --- |
277
| `GITHUB_TOKEN_ENCRYPTION_KEY` | set, key id `production-legacy-v1` |
278
| `GITHUB_TOKEN_DECRYPTION_KEYS_JSON` | empty |
279
| `MACHINE_TOKEN_ENCRYPTION_KEY` | unset |
280
| `VOICE_RECORDING_ENCRYPTION_KEY` | set, and a different key |
281
| Forge operator token | configured |
282
| Admitted reputation issuer keys | none |
283
284
Three of these are worth stating plainly.
285
286
**The pairing vault is not separated in production.** With
287
`MACHINE_TOKEN_ENCRYPTION_KEY` unset, the runtime falls through to the boot
288
bridge and configures the pairing vault with the GitHub vault's active key. The
289
two configured values hash identically on the node. #192's fix is deployed and
290
correct; the secret it needs was never provisioned, so the bridge meant to last
291
one deploy is the production configuration. `VAULT-001` says each vault seals
292
under its own key, and two of the three currently do not. Filed as **#253**.
293
294
**No key has ever been rotated here.** The GitHub retired keyring is empty and
295
the active key id is `production-legacy-v1`, so the documented rotation — add
296
the old key to the keyring, activate the new one, rewrap with
297
`OpenAgents.Accounts.rotate_github_tokens!/0` — has never run against
298
production. That is a fact rather than a defect, and it is the reason this
299
rehearsal's live half is an inventory instead of a rotation.
300
301
**No attestation has been signed here.** `OpenAgents.Reputation.keys()` returns
302
none on the live node, so the issuer rotation path has no production
303
population. The ordering rule is proven in the test process and has never been
304
exercised against a real attestation.
305
306
**Verdict: pass in the test process; one deployment gap, filed as #253.** No
307
key was rotated against production, and this record does not claim one was.
308
Rehearsal 4 now has an executable proof, which closes #180's third acceptance
309
criterion.
310
311
## Rehearsal 5: operator loss
312
313
### Local: a second operator recovers from the WAL
314
315
`rebuild/1` re-materializes a repository from sequence zero with no mirror
316
input, and the receipts a database lost are re-derived from the log. Both are
317
proven in the test process by the three tests rehearsal 3's step 3 names. That
318
is the source half, and it holds.
319
320
The metadata half is rehearsal 6's export, and the credential half is rehearsal
321
4's inventory. Both are recorded above.
322
323
### Live, read-only: what a second operator cannot obtain
324
325
The runbook says the WAL "lives in storage the current operator controls, and
326
this repository contains nothing that changes that". Read from the node and
327
from the storage API, that is exact, and it is more specific than the sentence
328
suggests:
329
330
- The adapter is `OpenAgents.Forge.WAL.Gcs`, against the bucket
331
  one bucket in the operator's own cloud project, in one region.
332
- The bucket has uniform bucket-level access, so reaching it requires an IAM
333
  grant only the current operator can make. There is no anonymous read path and
334
  no second copy anywhere a stranger can reach.
335
- The bucket has **no object versioning**. Soft delete retains a deleted object
336
  for seven days and nothing longer. An operator who rewrote an entry a week
337
  ago leaves no earlier copy in the bucket for anyone to compare against.
338
- The bare repositories are node-local, on each node's own disk, so they are a
339
  projection rather than a second copy of the record.
340
341
So the honest statement is stronger than "the operator controls the storage".
342
The storage keeps no history of itself. The only thing outside it that commits
343
to the log's contents is the anchor published at
344
`/.well-known/openagents-forge-anchor.json`, and that document is served by the
345
same operator and witnessed by nobody — it says so itself, and
346
`GET /api/status` reports `anchor_witnessed: false`. A reader who kept a copy
347
of it can refute a later rewrite; a reader who did not cannot, and neither can
348
a second operator.
349
350
**Verdict: recovery passes in the test process. The rehearsal's stated limit
351
holds and is now measured rather than asserted.** No new issue: #151 already
352
carries the witness.
353
354
## Rehearsal 6: partial export
355
356
### Steps 1 and 2, local: the document says what it leaves out
357
358
Performed through the route a person uses, against a forge the test process
359
builds, because the live route requires a session and this record does not
360
claim to have opened one. `GET /data/export/account` answers `200` and returns
361
a document with fifteen top-level keys.
362
363
`"bounds"` names fourteen caps:
364
365
```json
366
{"attestations": 5000, "box_run_output_bytes": 65536, "box_runs": 2000,
367
 "deployments": 2000, "forum_posts": 10000, "forum_tips": 2000,
368
 "forum_topics": 2000, "issue_dependencies": 5000, "pull_requests": 5000,
369
 "push_receipts": 10000, "stack_entries": 10000, "stacks": 2000,
370
 "thread_events": 10000, "threads": 1000}
371
```
372
373
Ten `*_truncated` flags travel with the collections they bound, across
374
`boxes`, `deployments`, `forum`, `push_receipts`, and `threads`.
375
376
`"not_included"` names four families with a mechanism and a reason each:
377
`conversation` through `GET /data/export` and `GET /data/export/atif`,
378
`repository_content` through the authenticated Git transport,
379
`repository_identity` for a repository the account can no longer read, and
380
`forum` for posts under an unclaimed legacy `actor_ref`.
381
382
The sealed variant behaves the same. `GET /data/export/account?recipient=…`
383
returns a body beginning `age-encryption.org/v1`, with a `.json.age` filename,
384
and decrypts to the same `bounds`.
385
386
**Verdict: pass.**
387
388
### Step 1, local: a flag actually turns true
389
390
Every `*_truncated` assertion in the committed suite is a `refute`. Nothing
391
proved the positive case, so the central claim — that a document which cannot
392
return everything says so — had no proof that could fail for it.
393
394
A Box run with 70,000 bytes of output was seeded and read back through the
395
route:
396
397
| Measurement | Value |
398
| --- | --- |
399
| Output produced | 70,000 bytes |
400
| `bounds.box_run_output_bytes` | 65,536 |
401
| Output exported | 65,536 bytes |
402
| `run.output_truncated` | `true` |
403
| `run.output_byte_size` | 70,000 |
404
405
The document reports the full size beside the capped bytes and flags the cut.
406
407
**Verdict: pass.** The proof gap is real and remains: this observation lives in
408
this record rather than in a committed test, so nothing turns red if the flag
409
stops being set. `docs/2026-08-24-invariant-proof-audit.md` is the register for
410
that class of residue.
411
412
### Step 3, live: the ledger and the disclosure agree
413
414
`GET /api/status` publishes:
415
416
```json
417
"export": {"families": 35, "portable": 24, "partial": 0, "blocked": 1,
418
           "not_user_data": 10,
419
           "gaps": [{"family": "trace", "issue": 217, "status": "blocked"}]}
420
```
421
422
and the deployed `OpenAgents.DataRights.ExportInventory` reports the same 35
423
families with the same split and the same single gap. `api_family_drift/0`
424
returns `%{stale: [], unclassified: []}` on the node.
425
426
**Verdict: pass, live.**
427
428
### Step 3, on `main`: the coverage proof is red
429
430
The same comparison fails on `main` at `2a9bd5e`:
431
432
```
433
assert ExportInventory.api_family_drift() == %{unclassified: [], stale: []}
434
left:  %{stale: [], unclassified: [:response]}
435
```
436
437
`bf115c0` added `POST /api/v1/responses` to `OpenAgentsWeb.ApiRouteAuthority`
438
with the family `:response` and no ledger entry. This is `EXIT-001`'s derived
439
coverage doing what it exists to do — a family reached `/api/v1` without anyone
440
deciding whether a user can export it, and the build failed. What remains is
441
the decision. Filed as **#252**, and answered in a separate commit rather than
442
inside this record: `OpenAgentsWeb.ResponsesController` reaches no repo and no
443
schema, so the family is `not_user_data` with a note saying the entry is
444
revisited the day a provider stands behind the stub.
445
446
**Verdict: pass against the live forge, fail against `main` when the rehearsal
447
ran.**
448
449
## What these rehearsals still do not cover
450
451
The runbook's own list is unchanged and still accurate: withholding,
452
confidentiality, and attribution of operator reads. Two additions from
453
performing them:
454
455
- **No tamper was performed against the live forge.** Every finding code in
456
  rehearsal 2's step 2 is proven in a test process. The live half proves the
457
  verifier runs and reports clean, not that it would report a real tamper here.
458
- **No key was rotated and no projection was rebuilt against production.**
459
  Rehearsals 4 and 5 measured what is deployed and proved the procedures
460
  elsewhere. A rehearsal that cannot safely run its own destructive step
461
  against production is a rehearsal with a permanent gap, and saying so is
462
  better than implying the step was taken.
docs/forge-exit-rehearsals.md modified +131 -17

@@ -28,14 +28,20 @@ that reads as executable.

28 28
| Rehearsal | Executable proof | Performed against the live forge |
29 29
| --- | --- | --- |
30 30
| 1. Restore a repository and its work history | `EXIT-004`, `EXIT-001` | 2026-08-23 — **failed**, see #179; steps 1 and 2 re-run 2026-08-25 and passed |
31
| 2. Detect a forged, missing, reordered, or mismatched receipt | `EXIT-002`, `EXIT-005` | Steps 1 and 3 performed 2026-08-25 |
32
| 3. Mirror divergence | `EXIT-003` | No |
33
| 4. Key rotation | None | No |
34
| 5. Operator loss | `EXIT-003` | No |
35
| 6. Partial export | `EXIT-001` | No |
36
37
Four of the six have never been run outside the test suite, which #180 carries.
38
That is the honest state.
31
| 2. Detect a forged, missing, reordered, or mismatched receipt | `EXIT-002`, `EXIT-005` | 2026-08-25 — steps 1 and 3 pass; step 2 is local only; **new defect #251**, and #190 still reproduces |
32
| 3. Mirror divergence | `EXIT-003` | 2026-08-25 — steps 1 and 2 pass and confirm #188; step 3 is local only |
33
| 4. Key rotation | `test/openagents/forge/key_rotation_test.exs` | 2026-08-25 — passes in the test process; **new defect #253** |
34
| 5. Operator loss | `EXIT-003` | 2026-08-25 — recovery passes in the test process; the WAL remains unobtainable, now measured |
35
| 6. Partial export | `EXIT-001` | 2026-08-25 — passes against the live forge; **new defect #252**, found red on `main` and since classified |
36
37
Every row now names a date and a result.
38
`docs/2026-08-25-forge-exit-rehearsals-2-to-6.md` records what running
39
rehearsals 2 through 6 produced, and which half of each ran against the live
40
forge rather than against a forge a test process builds. Three steps mutate
41
state — rehearsal 2's tampering, rehearsal 3's rebuild, and any rotation in
42
rehearsal 4 — so they were performed locally and are labeled local there. A
43
rehearsal that cannot safely run its destructive step against production has a
44
permanent gap, and that record says so rather than implying the step was taken.
39 45
40 46
## Release re-check, 2026-08-25
41 47

@@ -291,6 +297,17 @@ false`. A consistent rewrite is refuted only for a reader who kept a copy of

291 297
that document or a receipt line; nobody is attesting to it on the operator's
292 298
behalf, and #151 carries the witness.
293 299
300
**Step 2 has been performed only locally, and step 1 found a defect.** The
301
later pass recorded in `docs/2026-08-25-forge-exit-rehearsals-2-to-6.md`
302
verified the same log against the anchor the forge had already published, ran
303
step 2's tampering against a forge the test process builds rather than against
304
production, and ran step 1 on all three fleet nodes at once. The nodes gave
305
three different answers: the WAL is shared and the projection is node-local, so
306
a node that has not replayed an entry yet reports `served_refs_diverged` and
307
`object_missing`, which are the findings that mean tampering. #251 carries it,
308
and it blocks #179's scheduled verification pass. #190 also still reproduces:
309
`OpenAgents.Forge.Repos.allowed_repos/0` returns a name `verify/2` cannot use.
310
294 311
## 3. Mirror divergence
295 312
296 313
**Proves:** the GitHub mirror is never an input to recovery, and divergence is

@@ -308,9 +325,11 @@ reported rather than reconciled silently.

308 325
   git ls-remote https://github.com/{owner}/{repo}.git refs/heads/main
309 326
   ```
310 327
311
2. A difference is expected whenever the forge is ahead. Automatic mirroring is
312
   not configured, so GitHub stays at whatever was last pushed to it.
313
   `OpenAgents.Forge.MirrorWatch` publishes freshness on `/status`.
328
2. A difference is expected whenever the forge is ahead. A mirror **is**
329
   configured for `openagents.com` — `OPENAGENTS_FORGE_MIRROR_URLS_JSON` sets
330
   it, and #188 corrected the contracts that said otherwise — so read
331
   `forge.mirror` on `/status`, which `OpenAgents.Forge.MirrorWatch`
332
   publishes, rather than assuming GitHub is frozen.
314 333
315 334
3. Confirm the divergence changes nothing about authority. Rebuild from the WAL
316 335
   and check the head is unchanged:

@@ -327,10 +346,34 @@ reported rather than reconciled silently.

327 346
   returns `:ok` on success and a typed error when the WAL cannot produce a
328 347
   servable projection. `EXIT-003` turns red if a mirror input is added.
329 348
349
### Result, 2026-08-25
350
351
**Steps 1 and 2 pass against the live forge.** Both remotes advertise the same
352
twenty-five refs, byte for byte, with `refs/heads/main` at
353
`e57f5ea8b1666ccb69fc4c626f54cf41b34a0ebe` on each, and the node reports
354
`openagents.com` as the one configured mirror.
355
356
Identical ref maps are not identical history, and the gap is #188's. A full
357
clone of each side: the forge serves 423 commits on `main`, `git fsck` clean,
358
grafted at five `shallow` boundaries with `eda094c6` as its root; GitHub serves
359
730, `git fsck` clean, rooted at `a352f78e`. The 307-commit difference is on
360
the mirror and in no WAL. For everything pushed since the seed the forge is
361
canonical and the mirror is lossy; for everything before it the relation is
362
inverted, which is what #188 asks the contracts to say.
363
364
**Step 3 was not run against production.** `rebuild/1` discards a projection,
365
and a rehearsal must be read-only against the forge people push to. Its
366
behavior was exercised locally instead, by three tests in
367
`test/openagents/forge/independence_test.exs`. Whether it succeeds against the
368
live 423-commit projection is untested.
369
`docs/2026-08-25-forge-exit-rehearsals-2-to-6.md` records both halves.
370
330 371
## 4. Key rotation
331 372
332
**Proves:** nothing yet. This rehearsal is written and has never been
333
performed.
373
**Proves:** that no rotation invalidates an already-issued receipt, and that a
374
rotation performed in the wrong order is refused rather than silently
375
invalidating history. `test/openagents/forge/key_rotation_test.exs` is the
376
proof, and the rehearsal was performed on 2026-08-25.
334 377
335 378
The forge holds several key-like secrets and they rotate differently:
336 379

@@ -369,8 +412,11 @@ The forge holds several key-like secrets and they rotate differently:

369 412
    `MACHINE_TOKEN_ENCRYPTION_KEY`) rotates with bounded loss: at most one
370 413
    ten-minute window of unclaimed pairings becomes unreadable, and a person
371 414
    retries the pairing. There is no keyring because no record outlives the
372
    window. Rotating the GitHub key no longer touches this vault; #192
373
    records that it once did, silently.
415
    window. The code no longer reaches the GitHub key, which is #192's fix,
416
    but production has not provisioned `MACHINE_TOKEN_ENCRYPTION_KEY`, so the
417
    boot bridge in `config/runtime.exs` still configures this vault with the
418
    GitHub vault's active key. Until that secret is set, a GitHub key rotation
419
    does move this vault's key. #253 carries it.
374 420
  - `OpenAgents.Voice.RecordingVault` (call audio,
375 421
    `VOICE_RECORDING_ENCRYPTION_KEY`) rotates with permanent loss: one key,
376 422
    no key id, no keyring, so recordings sealed under the retired key never

@@ -381,10 +427,30 @@ The forge holds several key-like secrets and they rotate differently:

381 427
every already-issued receipt verifiable, and that a rotation performed in the
382 428
wrong order is refused rather than silently invalidating history.
383 429
430
### Result, 2026-08-25
431
432
**Both halves pass in the test process.** A forge receipt depends on no key —
433
the WAL chain link is unkeyed and reproducible from the entry alone, and the
434
verifier is compiled against no secret and no vault — so no rotation can
435
invalidate one. The reputation issuer key's ordering rule refuses a backward
436
retirement and names the attestation it would have unverified, refuses one
437
earlier than a silent key's activation, and refuses one beyond the clock-skew
438
allowance. Each vault's rotation outcome is proven separately. Twelve tests,
439
all green.
440
441
**No key was rotated against production.** The live half is an inventory read
442
without printing key material, and it found one gap: `VOICE_RECORDING_ENCRYPTION_KEY`
443
is set and distinct, but `MACHINE_TOKEN_ENCRYPTION_KEY` is unset, so the
444
pairing vault runs on the GitHub vault's key (#253). It also found that the
445
GitHub retired keyring is empty and no reputation issuer key is admitted, so
446
neither rotation has ever run here.
447
`docs/2026-08-25-forge-exit-rehearsals-2-to-6.md` records the inventory.
448
384 449
## 5. Operator loss
385 450
386
**Proves:** nothing yet outside `EXIT-003`, which shows recovery comes from the
387
WAL and never from the mirror.
451
**Proves:** that the source and the receipts come back from the WAL alone.
452
`test/openagents/forge/independence_test.exs` proves it in a test process, and
453
the rehearsal was performed on 2026-08-25.
388 454
389 455
**What a rehearsal must establish:** that a second operator, starting from the
390 456
WAL and the exported metadata alone, brings the forge back without treating

@@ -396,6 +462,25 @@ depends on rehearsal 4.

396 462
WAL at all. It lives in storage the current operator controls, and this
397 463
repository contains nothing that changes that.
398 464
465
### Result, 2026-08-25
466
467
**Recovery passes in the test process**, through the three tests rehearsal 3's
468
step 3 names: the WAL rebuilds the repository and re-derives receipts a
469
database lost, the rebuild path reads no mirror, and the mirror restores source
470
without restoring the push record.
471
472
**The limit above is now measured rather than asserted, and it is stronger than
473
it reads.** The WAL adapter is `OpenAgents.Forge.WAL.Gcs` against one bucket in
474
the operator's own cloud project, with uniform bucket-level access, so reaching
475
it needs a grant only the current operator can make. That bucket has no object
476
versioning; soft delete retains a deleted object for seven days and nothing
477
longer. So the storage keeps no history of itself, and an operator who rewrote
478
an entry a week ago leaves no earlier copy in it. The only commitment outside
479
that bucket is the anchor at `/.well-known/openagents-forge-anchor.json`,
480
served by the same operator and witnessed by nobody (#151). A reader who kept a
481
copy of it can refute a later rewrite; a second operator starting from scratch
482
cannot.
483
399 484
## 6. Partial export
400 485
401 486
**Proves:** an export that cannot return everything says so rather than

@@ -414,6 +499,35 @@ returning a shorter document that reads as complete.

414 499
   enforces against the surface in both directions, and against the same
415 500
   counts published at `GET /api/status` under `independence.export`.
416 501
502
### Result, 2026-08-25
503
504
**Step 3 passes against the live forge.** `GET /api/status` publishes 35
505
families — 24 portable, 0 partial, 1 blocked, 10 not user data — with the
506
`trace` family as the one declared gap, and the deployed ledger reports the
507
same split and no drift.
508
509
**Step 3 fails against `main`.** The same comparison, run in the test suite,
510
reports `unclassified: [:response]`: `POST /api/v1/responses` reached the route
511
authority without a ledger entry. That is `EXIT-001`'s
512
derived coverage failing the build, which is what it is for. #252 carries the
513
decision it forced: the stub records nothing, so the family is
514
`not_user_data`, and the entry says it is revisited when a provider stands
515
behind the route.
516
517
**Steps 1 and 2 pass, exercised through the route against a local forge**
518
rather than a live session. The document carries a `bounds` map of fourteen
519
caps, ten `*_truncated` flags across five collections, and a `"not_included"`
520
list naming four families with a mechanism and a reason each. The sealed
521
variant returns an `age-encryption.org/v1` body that decrypts to the same
522
bounds.
523
524
**One flag was observed positive, and no committed test does that.** Every
525
`*_truncated` assertion in the suite is a `refute`, so the claim this rehearsal
526
exists to check had no proof that could fail for it. A Box run of 70,000 bytes
527
exports 65,536 with `output_truncated: true` and `output_byte_size: 70000`. The
528
observation lives in `docs/2026-08-25-forge-exit-rehearsals-2-to-6.md` rather
529
than in a test, so the proof gap remains.
530
417 531
## What these rehearsals do not cover
418 532
419 533
- **Withholding.** No rehearsal detects an operator who serves nothing or
lib/openagents/data_rights/export_inventory.ex modified +19

@@ -373,6 +373,25 @@ defmodule OpenAgents.DataRights.ExportInventory do

373 373
      issue: nil,
374 374
      note: "The API's own route and extension inventory."
375 375
    },
376
    %{
377
      family: :response,
378
      api?: true,
379
      status: :not_user_data,
380
      mechanism: nil,
381
      proof: nil,
382
      issue: nil,
383
      # Classified the moment the route landed, which is the only time this
384
      # answer is cheap. `OpenAgentsWeb.ResponsesController` reaches no repo
385
      # and no schema: it answers every request with one acknowledged message
386
      # and writes nothing, so there is no record to give back. This entry
387
      # stops being true the day a provider stands behind the stub and a
388
      # response is persisted, and the classification is revisited then rather
389
      # than inherited (#252).
390
      note:
391
        "POST /api/v1/responses is a stub. It consults no model and records " <>
392
          "nothing, so it holds no record an account authors. Revisit this " <>
393
          "entry when a provider stands behind it and a response is stored."
394
    },
376 395
    %{
377 396
      family: :model,
378 397
      api?: true,

This page updates live while a promote is in flight · changelog