Perform the five exit rehearsals that had never been run

2cc7123460a2 · AtlantisPleb · · parent 18cb0ca26fda

Perform the five exit rehearsals that had never been run

All six now have a date and a result against the live forge, and five of
them found something. #180.

The common cause behind three findings is that `main` is 57 commits
ahead of the deployed revision and every exit surface #94 audits is in
those 57 commits: `GET /data/export/account` answers 404, the
`independence` section EXIT-006 publishes is absent because the module
does not load, and `Verification.verify` has arity 1, so the anchor
check that distinguishes EXIT-005's tamper-evidence from EXIT-002's
operator-agrees-with-operator cannot be run at all. Of the 275 entries
in the live log, none carries a chain link. #187.

Rehearsal 2 is the sharpest result. `verify/1` reports zero findings on
the repository a full clone aborts on, at the same hour, on the same
node — a green verifier and an unservable repository. It also corrects
boundaries, and the projection has no `shallow` file, so the log holds
the boundary and a stale cache does not. The graft is recoverable from
the WAL; the 307 pre-seed commits are not. Its documented first step
does not work either, because the name the configuration admits has no
WAL behind it and a stale bare repository sits under it. #190.

Rehearsal 3 found a mirror configured in production while EXIT-003 and
CLAUDE.md both say none is, and that this mirror holds 307 commits —
51% of `main` — the forge cannot serve and the WAL never held. For the
pre-seed history the mirror is strictly richer rather than strictly
lossy, which is the outcome EXIT-003 exists to prevent, reached from a
direction it cannot watch: nothing was added to the recovery path. Its
third step names `Sync.rebuild/1`, which exists neither on the deployed
build nor on `main`. #188, #189.

Rehearsal 4 had no executable proof and now has one. No forge receipt
depends on any key, which is asserted rather than assumed by rotating
every secret under a computed chain link and by reading the verifier's
compiled callees. Two families fail the other half: a backdated
`retire_key/2` silently flips an already-signed attestation to
unverified while its signature stays valid, and the machine pairing
vault reads the GitHub vault's active key with no key id and no
keyring, so the documented GitHub rotation destroys outstanding
pairings. Both are pinned with their issue numbers, and each pin was
checked by implementing the fix and confirming the pin went red. #191,

Rehearsal 5 cannot be completed from here and says so plainly. The WAL
lives in one bucket in the operator's own project, with no second
custodian and no mechanism in this repository to establish one, so a
second operator obtains no refs, no objects, no sequences, no
principals, and no push times. What they would get is the mirror:
today, more source than the WAL holds and no provenance for any of it.

Rehearsal 6 could not reach the route it exists to exercise. EXIT-001
is the most thoroughly proven of the six, and the document it proves
cannot be downloaded from the forge it describes.

Every one of EXIT-001 through EXIT-006 was green throughout. Each runs
against a forge the test process builds; a rehearsal runs against the
one people use.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DTmy4SEXrHXouw5sZbs3f4
Co-Authored-By
Claude Opus 5 (1M context) <noreply@anthropic.com>

Deploy story

What this commit did to the running system — joined from the forge receipt chain, the part a commit page elsewhere cannot show.

Not deployed through the forge lane

No push, promotion, build, or deploy receipt references this commit (receipts are scanned over a bounded recent window). Changes shipped by full node replacement carry their proof in the release gate receipt instead.

Changed files

  • modified INVARIANTS.md
  • modified docs/forge-exit-rehearsals.md
  • modified docs/forge-operator-independence.md
  • modified ops/ci/allowed-sarah-references.txt
  • added test/openagents/forge/key_rotation_test.exs

Diff

5 files changed, +742 -29

INVARIANTS.md modified +24 -8

@@ -3874,8 +3874,17 @@ cache. This proves that divergence between the WAL and what is served is

3874 3874
detectable. The first is about replay; the second is about detection, and
3875 3875
neither substitutes for the other.
3876 3876
3877
A receipt also depends on no key, which is what makes it survive every key
3878
rotation this forge can perform. The chain link is unkeyed `sha256` and the
3879
verifier was compiled against no vault, no `OpenAgents.ApiTokens`, and no
3880
`OpenAgents.Reputation`; `test/openagents/forge/key_rotation_test.exs` asserts
3881
both, so a receipt cannot quietly acquire a key dependency. That proof is
3882
rehearsal 4 of `docs/forge-exit-rehearsals.md`, which had no executable proof
3883
before #180.
3884
3877 3885
Evidence: `OpenAgents.Forge.Verification`, `OpenAgents.Forge.WAL`,
3878
`OpenAgents.Forge.Repos`, and `test/openagents/forge/independence_test.exs`.
3886
`OpenAgents.Forge.Repos`, `test/openagents/forge/independence_test.exs`, and
3887
`test/openagents/forge/key_rotation_test.exs`.
3879 3888
3880 3889
### EXIT-003 — Recovery comes from the WAL, and the mirror is strictly lossy
3881 3890

@@ -3922,12 +3931,19 @@ key only PostgreSQL could produce would not survive `reconcile_receipts/1`,

3922 3931
which rebuilds this table from the WAL alone. The absence is the invariant
3923 3932
holding, not an omission.
3924 3933
3925
Two operational facts bound the claim. `:forge_mirror_urls` is empty in
3926
`config/config.exs` and set by no environment, so no mirror runs today and
3927
GitHub holds whatever was last pushed to it directly, which is the trade
3928
`REPOSITORY-002` records. And `mirror_now/1` is a force push of every ref, so
3929
configuring a mirror overwrites what direct pushes left there rather than
3930
merging with it.
3934
Two operational facts bound the claim, and the first was stated wrongly here
3935
until #180's rehearsal 3 checked it against production. `:forge_mirror_urls` is
3936
empty in `config/config.exs`, but `config/runtime.exs` reads
3937
`OPENAGENTS_FORGE_MIRROR_URLS_JSON` and that variable is set in production: the
3938
live node reports one configured mirror, and `/status` publishes its freshness.
3939
A mirror runs. And `mirror_now/1` is a force push of every ref, so the
3940
configured mirror overwrites what direct pushes left there rather than merging
3941
with it, which matters more now than it did while the claim was that none ran.
3942
#188 carries both, together with the finding that the mirror currently holds
3943
307 commits of this repository's history the WAL never held — so for the
3944
pre-seed history the mirror is strictly *richer* rather than strictly lossy,
3945
which this invariant does not watch for and cannot detect, because nothing was
3946
added to the recovery path.
3931 3947
3932 3948
Evidence: `OpenAgents.Forge.Sync`, `OpenAgents.Forge.Pushes`,
3933 3949
`OpenAgents.Forge.PushReceipt`, `OpenAgents.Forge.Verification`, and

@@ -4855,7 +4871,7 @@ contract; the invariant prose above defines the assertion, not the filename.

4855 4871
| REPOSITORY-002 | `ops/ci/push-remote-check.sh`, `ops/dev/install-push-guard.sh`, `test/openagents/push_remote_contract_test.exs` |
4856 4872
| REPOSITORY-003 | `test/openagents/forge/wal_replay_test.exs`, `test/openagents/forge/sync_test.exs`, `test/openagents/forge/independence_test.exs` |
4857 4873
| EXIT-001 | `test/openagents/data_rights/export_inventory_test.exs`, `test/openagents/data_rights/account_export_test.exs` |
4858
| EXIT-002 | `test/openagents/forge/independence_test.exs` |
4874
| EXIT-002 | `test/openagents/forge/independence_test.exs`, `test/openagents/forge/key_rotation_test.exs` |
4859 4875
| EXIT-003 | `test/openagents/forge/independence_test.exs` |
4860 4876
| EXIT-004 | `test/openagents/forge/independence_test.exs` |
4861 4877
| EXIT-005 | `test/openagents/forge/independence_test.exs`, `test/openagents/forge/wal_test.exs`, `test/openagents/forge/git_http_test.exs`, `test/openagents_web/controllers/push_receipt_controller_test.exs`, `test/openagents_web/controllers/forge_anchor_controller_test.exs` |
docs/forge-exit-rehearsals.md modified +324 -14

@@ -19,15 +19,22 @@ ran it and recorded the result here. Anything else says so.

19 19
20 20
| Rehearsal | Executable proof | Performed against the live forge |
21 21
| --- | --- | --- |
22
| 1. Restore a repository and its work history | `EXIT-004`, `EXIT-001` | 2026-08-23 — **failed**, see #179 |
23
| 2. Detect a forged, missing, reordered, or mismatched receipt | `EXIT-002`, `EXIT-005` | No |
24
| 3. Mirror divergence | `EXIT-003` | No |
25
| 4. Key rotation | None | No |
26
| 5. Operator loss | `EXIT-003` | No |
27
| 6. Partial export | `EXIT-001` | No |
28
29
Five of the six have never been run outside the test suite, which #180 carries.
30
That is the honest state, and the one that was run failed.
22
| 1. Restore a repository and its work history | `EXIT-004`, `EXIT-001` | 2026-08-23 — **failed**, #179; re-run 2026-08-24 — **still failed**, #187 |
23
| 2. Detect a forged, missing, reordered, or mismatched receipt | `EXIT-002`, `EXIT-005` | 2026-08-24 — **performed, and the result is worse than a failure**: #187, #190 |
24
| 3. Mirror divergence | `EXIT-003` | 2026-08-24 — **performed, failed**: #188, #189 |
25
| 4. Key rotation | `test/openagents/forge/key_rotation_test.exs` | 2026-08-24 — **performed, partly failed**: #191, #192 |
26
| 5. Operator loss | `EXIT-003` | 2026-08-24 — **performed as far as it can be**; the blocking half is named below |
27
| 6. Partial export | `EXIT-001` | 2026-08-24 — **performed, failed**: #187 |
28
29
All six have now been run against the live forge. Five of them found something,
30
and one finding is common to three of them: `main` is 57 commits ahead of the
31
deployed revision `6d421b3c7ffe`, and the exit surfaces #94 audits are all in
32
those 57 commits. #187 carries that.
33
34
The general lesson is the one #179 already taught, now with five more
35
instances. Every invariant `EXIT-001` through `EXIT-006` was green throughout,
36
because each runs against a forge the test process builds. A rehearsal runs
37
against the forge people use, and that is the entire difference.
31 38
32 39
## 1. Restore a repository and its bounded work history
33 40

@@ -117,7 +124,51 @@ is why `EXIT-004` stayed green throughout.

117 124
`OpenAgents.Forge.Sync` reconciles the graft against the objects a projection
118 125
actually holds, so a repository that cannot be walked repairs itself from the
119 126
WAL. The pre-seed history remains outside this forge and is not recoverable
120
from it. Rerun step 1 to confirm the live forge serves a full clone.
127
from it.
128
129
### Re-run, 2026-08-24
130
131
**Step 1 failed again, and it will keep failing until a release is promoted.**
132
The repair is on `main`; the live forge runs `6d421b3c7ffe`, which predates
133
it. Verbatim:
134
135
```
136
$ git clone https://openagents.com/OpenAgentsInc/openagents.com.git restored
137
Cloning into 'restored'...
138
remote: error: Could not read c91327d60c520d11133ddcc6cb3304784f2f0481
139
remote: fatal: Failed to traverse parents of commit eda094c6ae9f100060b96cd93bad9e4ecd117e94
140
remote: aborting due to possible repository corruption on the remote side.
141
fatal: early EOF
142
fatal: fetch-pack: invalid index-pack output
143
```
144
145
Four of the five branches #179 recorded still fail at depth 2:
146
`codex/github-backed-repositories`, `codex/posthog-integration`,
147
`components-gallery`, and `repos-ui`. `codex/admin-posthog-analytics` now
148
succeeds, because something pushed across its boundary in the interval.
149
150
**One sentence of #179's diagnosis is wrong, and rehearsal 2 is what found
151
it.** #179 says the seed "was written before WAL entries carried a `shallow`
152
key. The log therefore records no boundary." The live WAL says otherwise. Its
153
seq 0 entry carries a `shallow` key naming five boundary commits, `eda094c6`
154
among them, and the served projection has no `shallow` file at all:
155
156
```
157
File.exists?("/var/lib/openagents/forge/repos/ecd89cf6-….git/shallow")
158
#=> false
159
```
160
161
The log holds the boundary; the projection does not. The deployed replay does
162
write the graft — `OpenAgents.Forge.Sync.write_shallow_boundaries/2` exists at
163
`6d421b3c7ffe` — but only while applying an entry, and this cache was
164
materialized before that code existed and has applied nothing since that would
165
rewrite it. So the repair `main` carries is the right one, and the reason it is
166
needed is a stale projection rather than a silent log.
167
168
That distinction matters for what is recoverable. The graft is recoverable from
169
the WAL, so the repository becomes cloneable, with history that honestly stops
170
at the boundary. The 307 pre-seed commits are not in the WAL and are not
171
recoverable from it. See rehearsal 3 for where they are.
121 172
122 173
## 2. Detect a forged, missing, reordered, or mismatched receipt
123 174

@@ -154,6 +205,82 @@ published yet (#168). `GET /api/status` reports this as

154 205
   A rewritten prefix reports `anchor_mismatch`. Without the anchor argument
155 206
   the same log reports clean, which is the whole point of publishing one.
156 207
208
### Result, 2026-08-24
209
210
Performed on `sarah-fleet-1` through
211
`docker exec openagents /app/bin/openagents rpc`. Steps 1 and 2 ran. Step 3
212
could not.
213
214
**Step 1 does not work as written.** The rehearsal says `{storage_key}` and the
215
forge answers with a name:
216
217
```
218
> OpenAgents.Forge.Repos.allowed_repos()
219
["openagents.com"]
220
221
> OpenAgents.Forge.Verification.verify("openagents.com")
222
{:error, %{entries: 0, repo: "openagents.com",
223
           findings: [%{code: "wal_unreadable", detail: %{"reason" => ":not_found"}}]}}
224
```
225
226
`wal_unreadable` is what a verifier says when the write-ahead log is gone. The
227
log is intact; the name is wrong. The served repository is keyed by a UUID, and
228
a stale bare repository sits under the name holding one ref at a commit `main`
229
passed long ago. Filed as #190.
230
231
**Step 2 reports clean on a repository that cannot be cloned.**
232
233
```
234
> OpenAgents.Forge.Verification.verify("ecd89cf6-f602-479f-9f47-266307345aaa")
235
tag=:ok
236
entries=275
237
finding_count=0
238
codes=%{}
239
```
240
241
Zero findings, at the same hour a full clone of that repository aborts. This is
242
the sharpest available statement of why rehearsals exist. The deployed verifier
243
checks that every ref tip resolves, and every ref tip does; it does not walk
244
the tips into their ancestors, because the walk is part of the `EXIT-004`
245
amendment on `main`. A green verifier and an unservable repository, at the same
246
moment, on the same node.
247
248
**The chain `EXIT-005` describes is not running.** Of the 275 entries in the
249
live log, **none carries a link** — including the entry written twelve minutes
250
before the check:
251
252
```
253
> entries=275 linked=0
254
> last=%{"format" => "receive_pack", "object" => "entries/00000274-3f1807e0e409",
255
         "principal" => "user:af9e…", "pushed_at" => "2026-08-24T03:53:33.604547Z", "seq" => 274}
256
```
257
258
Confirmed independently: `OpenAgents.Forge.WAL.chain_link/2` and `entry_link/1`
259
do not exist on the deployed build, with the module loaded first, because
260
`function_exported?/3` answers `false` for a module nobody has loaded and that
261
is an easy way to draw the wrong conclusion.
262
263
**Step 3 could not be performed at all.** The deployed `Verification.verify`
264
has arity 1:
265
266
```
267
> OpenAgents.Forge.Verification.__info__(:functions) |> Keyword.get_values(:verify)
268
[1]
269
```
270
271
There is no `:anchor` option to pass, and there would be nothing to pass to it:
272
an anchor is a `link`, and no entry has one. The one check that distinguishes
273
`EXIT-005`'s tamper-evidence from `EXIT-002`'s "the operator agrees with the
274
operator" is unavailable on the live forge. `Application.get_env(:openagents,
275
:forge_wal_anchor)` is `nil`, which is what `EXIT-006` would publish if
276
`EXIT-006` were deployed; `OpenAgents.Forge.Independence` does not load on the
277
node either. Filed as #187.
278
279
**What this rehearsal proves today:** that `verify/1` runs against production
280
storage without a database and reports the shape it promises. That is real and
281
it is less than the rehearsal claims. Every disagreement it can detect other
282
than the five ref-and-entry findings is unavailable here.
283
157 284
## 3. Mirror divergence
158 285
159 286
**Proves:** the GitHub mirror is never an input to recovery, and divergence is

@@ -185,10 +312,61 @@ reported rather than reconciled silently.

185 312
   The rebuild path takes no mirror input. `EXIT-003` turns red if one is
186 313
   added.
187 314
315
### Result, 2026-08-24
316
317
**Step 1: no divergence, and that is not the good news it sounds like.** The
318
two ref maps are identical — 25 refs, same shas, `refs/heads/main` at
319
`773ad680` on both.
320
321
**Step 2 contradicts the contract.** `EXIT-003` states as an operational fact
322
that `:forge_mirror_urls` "is empty in `config/config.exs` and set by no
323
environment, so no mirror runs today", and `CLAUDE.md` repeats it. The live
324
node disagrees:
325
326
```
327
> Application.get_env(:openagents, :forge_mirror_urls, %{}) |> Map.keys()
328
["openagents.com"]
329
```
330
331
`GET /api/status` says so too, publishing `forge.mirror` as
332
`{"repo": "openagents.com", "state": "current"}`, which
333
`OpenAgents.Forge.MirrorWatch` emits only for a configured repo. A mirror runs,
334
and `mirror_now/1` is a force push of every ref. Filed as #188.
335
336
**And the mirror is the only complete copy of half this repository.**
337
338
| Source | `main` commits | `git fsck` | Holds `c91327d6` |
339
| --- | --- | --- | --- |
340
| GitHub mirror | 603 | clean | yes |
341
| The forge | 296, from the seed forward | full clone aborts | no |
342
343
307 commits — 51% of `main` — exist on GitHub and nowhere else this forge can
344
reach. They are not in the WAL, so no rebuild produces them. `EXIT-003` says
345
recovery comes from the WAL and the mirror is strictly lossy; for the pre-seed
346
history the relation is inverted, and the mirror is strictly richer. That is
347
the outcome `EXIT-003` exists to prevent, arrived at from a direction the
348
invariant does not watch: not a fallback someone added to the recovery path,
349
but a projection that never held the history in the first place. Also #188.
350
351
**Step 3 could not be performed, because the function does not exist.**
352
353
```
354
> OpenAgents.Forge.Sync.__info__(:functions) |> Keyword.keys()
355
[:ensure_cluster_fresh, :ensure_fresh, :ensure_fresh!, :replay_missing, :with_repo_lock]
356
```
357
358
`OpenAgents.Forge.Sync.rebuild/1` is not on the deployed build and is not on
359
`main`. This document and #179 both instruct an operator to run it. Filed as
360
#189.
361
188 362
## 4. Key rotation
189 363
190
**Proves:** nothing yet. This rehearsal is written and has never been
191
performed.
364
**Proves:** that no forge receipt depends on any key this forge holds, so no
365
rotation can invalidate one; and, for each key-like secret, whether rotating it
366
loses data and whether the wrong order is refused.
367
368
**Executable proof:** `test/openagents/forge/key_rotation_test.exs`, added by
369
#180. This rehearsal had none until then.
192 370
193 371
The forge holds several key-like secrets and they rotate differently:
194 372

@@ -212,6 +390,63 @@ The forge holds several key-like secrets and they rotate differently:

212 390
every already-issued receipt verifiable, and that a rotation performed in the
213 391
wrong order is refused rather than silently invalidating history.
214 392
393
### Result, 2026-08-24
394
395
The first half holds everywhere. The second holds in one family of four.
396
397
| Family | Rotation loses nothing | Wrong order refused |
398
| --- | --- | --- |
399
| Forge operator token | yes | not applicable — there is no order |
400
| Account `oa_pat_` tokens | yes | not applicable |
401
| Reputation issuer key | forward, yes | **no** — #191 |
402
| GitHub token vault | yes | yes |
403
| Machine pairing vault | **no** — #192 | no |
404
| Voice recording vault | **no** | no |
405
406
**No forge receipt depends on a key at all**, which makes the positive claim
407
true and worth stating plainly rather than triumphantly. `OpenAgents.Forge.WAL`'s
408
chain link is unkeyed `sha256` over a domain tag and the entry's own fields,
409
and `OpenAgents.Forge.Verification` was compiled against no vault, no
410
`OpenAgents.ApiTokens`, and no `OpenAgents.Reputation`. The proof rotates every
411
key-like secret in the application underneath a computed link and asserts the
412
link is unmoved. A push made with the operator token records the literal
413
`operator:forge-token`, which is written at push time rather than derived from
414
the secret, so no rotation makes a past push attributable to a person or takes
415
attribution away.
416
417
**The reputation issuer key fails the second half.** `retire_key/2` accepts any
418
timestamp and validates it against nothing. Retiring a key at a moment at or
419
before an attestation it already signed flips that attestation to
420
`"verified" => false` while `"signature" => %{"valid" => true}` — a valid
421
signature over an unaltered claim, reported as unverified, by one `UPDATE`
422
against a row the operator controls. The forward edge *is* guarded:
423
`require_active_key/2` refuses issuance under a key that is not yet active.
424
Only retirement is open. Filed as #191.
425
426
**The machine pairing vault fails the first half, and the coupling is not
427
written down anywhere.** `OpenAgents.Machines.TokenVault.key/0` reads
428
`:github_token_encryption_key` — the GitHub vault's *active* key — and its
429
envelope carries no key id and consults no keyring. So the documented GitHub
430
rotation in `docs/github-auth-plan.md`, performed in the documented order,
431
makes every outstanding pairing ciphertext permanently unopenable. The blast
432
radius is bounded: pairings live ten minutes and both terminal transitions null
433
the column, so at most ten minutes of unclaimed pairings are lost. Filed as
434
#192. `OpenAgents.Voice.RecordingVault` has the same shape with its own key and
435
no rewrap path; `test/openagents/voice/recordings_test.exs:319` already pins
436
that a wrong key fails closed, so what was missing was the statement that
437
rotating it is unrecoverable rather than the behaviour.
438
439
**The GitHub token vault is the one that gets this right**, and it is worth
440
saying why rather than only that it does. The key id is inside the envelope and
441
bound into the AAD, up to sixteen prior keys stay readable, `rotate_github_tokens!/0`
442
rewraps inside one transaction, and an invalid keyring raises at boot in `:prod`.
443
Performed in the wrong order it fails closed and the rewrap rolls back, rather
444
than writing a row nobody can open.
445
446
Both failures are pinned by tests that name their issue, so a fix turns the
447
test red instead of passing unnoticed. Each pin was mutation-checked by
448
implementing the fix and confirming the pin failed.
449
215 450
## 5. Operator loss
216 451
217 452
**Proves:** nothing yet outside `EXIT-003`, which shows recovery comes from the

@@ -227,6 +462,39 @@ depends on rehearsal 4.

227 462
WAL at all. It lives in storage the current operator controls, and this
228 463
repository contains nothing that changes that.
229 464
465
### Result, 2026-08-24
466
467
Performed as far as it goes, which is not far, and the boundary is now concrete
468
rather than abstract.
469
470
**What a second operator would need, named exactly.** The WAL adapter is
471
`OpenAgents.Forge.WAL.Gcs` and the bucket is `sarah-forge-wal`, in the Google
472
Cloud project the current operator owns. There is no second custodian, no
473
escrow, no copy anywhere else, and no mechanism in this repository by which one
474
could be established. A second operator starting from nothing obtains **no
475
refs, no objects, no sequences, no principals, and no push times**, because all
476
of it lives behind one IAM boundary.
477
478
**What that leaves them.** The GitHub mirror, which carries every commit, tree,
479
blob, tag, and advertised ref and no record of who pushed what or when — and
480
which, as rehearsal 3 found, currently carries 307 commits of history the WAL
481
never held. So a second operator restoring from the mirror alone would today
482
recover *more source* and *no provenance*: a complete-looking repository with
483
no evidence attached to any of it. `EXIT-003` proves both halves of that trade
484
in a test process; this rehearsal is where it becomes a fact about this forge.
485
486
**The metadata half was not exercised**, because the account export it depends
487
on returns `404` on the live forge — see rehearsal 6. **The credential half was
488
not exercised**, because it depends on rehearsal 4, which found two families
489
that do not survive rotation at all.
490
491
**This rehearsal cannot be completed from here.** Completing it needs a WAL
492
copy held somewhere the current operator does not solely control, which is an
493
owner action and an infrastructure decision, not a code change. It is the same
494
missing thing `#151` and `#168` name for the anchor, one level up: an anchor
495
proves the log was not rewritten, and a second custodian is what makes the log
496
obtainable at all. Neither exists today and no rehearsal changes that.
497
230 498
## 6. Partial export
231 499
232 500
**Proves:** an export that cannot return everything says so rather than

@@ -245,11 +513,53 @@ returning a shorter document that reads as complete.

245 513
   enforces against the surface in both directions, and against the same
246 514
   counts published at `GET /api/status` under `independence.export`.
247 515
516
### Result, 2026-08-24
517
518
**Step 1 could not be performed. The route does not exist on the live forge.**
519
520
```
521
$ curl -s -o /dev/null -w '%{http_code}\n' https://openagents.com/data/export/account
522
404
523
$ curl -s -o /dev/null -w '%{http_code}\n' https://openagents.com/data/export
524
302
525
```
526
527
`302` is the sign-in redirect, which is what an authenticated route answers to
528
an anonymous caller. `404` is what a route that was never compiled answers.
529
`GET /data/export/account` landed in `b061b99`, after the deployed revision.
530
531
**Step 3 could not be performed either**, for the same reason one level up:
532
`/api/status` publishes no `independence` section, because
533
`OpenAgents.Forge.Independence` is not on the deployed build. So the counts
534
this step says to compare against do not exist, and neither does the
535
`independence.export.gaps` list. Filed as #187.
536
537
**What was exercised instead.** The bounds and `not_included` properties this
538
rehearsal checks are proven in
539
`test/openagents/data_rights/account_export_test.exs` against the ledger in
540
both directions, and that proof is green. What a green proof cannot tell you is
541
whether the route is reachable, and that is the whole content of this result:
542
`EXIT-001` is the most thoroughly proven of the six exit invariants, and the
543
document it proves cannot be downloaded from the forge it describes.
544
545
**One thing changed here rather than only being found.** #178 landed while this
546
rehearsal was being performed, so the route now accepts an `age` recipient and
547
returns a document encrypted to a key the operator does not hold. That widens
548
what step 1 will check once a release carries it: the export can now be taken
549
without the operator being able to read the file that carries it. The store it
550
was built from stays plaintext, which `GET /api/status` publishes beside it —
551
see `docs/2026-08-24-private-export-encryption.md`.
552
248 553
## What these rehearsals do not cover
249 554
250 555
- **Withholding.** No rehearsal detects an operator who serves nothing or
251 556
  serves stale state.
252
- **Confidentiality.** No export is encrypted to a key the recipient holds, and
253
  no column in this repository is encrypted at rest (#178).
557
- **Confidentiality of the store.** The account export can now be encrypted to
558
  a key the recipient holds (#178), which protects the file and not the
559
  database it was read from. No column in this repository is encrypted at rest
560
  (#193), so the operator holds the plaintext every export is built from.
254 561
- **Attribution of operator reads.** No operator read is audited, so no
255 562
  rehearsal can show one did not happen.
563
- **The gap between a proven invariant and a deployed one.** Nothing reports
564
  it. Five of the six rehearsals above ran into it, and each found it by hand.
565
  #187.
docs/forge-operator-independence.md modified +27 -7

@@ -68,7 +68,10 @@ every other.

68 68
  hand-rolled vaults seal three specific fields — GitHub access tokens, machine
69 69
  pairing tokens, and voice recording chunks — and each takes its key from the
70 70
  application environment, which is the operator's environment. They defend
71
  against a stolen database dump, not against the operator.
71
  against a stolen database dump, not against the operator. Only one of the
72
  three can be rotated without losing what it sealed (#192). #193 carries the
73
  at-rest question, and #178 decided the export path separately because losing
74
  an export key costs nothing while losing a storage key would be permanent.
72 75
- **Every repository's contents.** Git objects live unencrypted in the node's
73 76
  bare repositories and in the WAL. Whatever protection exists is disk-level
74 77
  and object-storage-level, which is to say transparent to whoever runs the

@@ -332,17 +335,34 @@ status page has hidden them.

332 335
`docs/forge-exit-rehearsals.md` defines six rehearsals — restore, receipt
333 336
verification, mirror divergence, key rotation, operator loss, and partial
334 337
export — with what each proves, what it cannot, and whether anyone has run it.
335
Five of the six have never been run outside the test suite, and the one that
336
was run against the live forge failed: a full clone of this repository aborts
337
on a missing object 275 commits behind `main` (#179). `EXIT-004` was green
338
throughout, because it runs against a forge the test builds and never against
339
the one people clone from.
338
339
All six have now been run against the live forge (#180, 2026-08-24). Five found
340
something. The restore rehearsal found that a full clone of this repository
341
aborts on a missing object (#179); the receipt rehearsal found that the
342
verifier reports zero findings on that same repository at the same hour, and
343
that no entry in the live log carries the chain link `EXIT-005` describes; the
344
mirror rehearsal found a configured mirror two contracts said did not exist,
345
and 307 commits that live only on it (#188); the rotation rehearsal found two
346
key families that lose data or history when rotated (#191, #192); and the
347
export rehearsal found that `GET /data/export/account` answers `404` on the
348
live forge (#187). The operator-loss rehearsal cannot be completed from inside
349
this repository at all, and says so.
350
351
Every one of `EXIT-001` through `EXIT-006` was green throughout. Each runs
352
against a forge the test process builds; a rehearsal runs against the forge
353
people use, and that is the entire difference.
340 354
341 355
## Open gaps
342 356
343 357
| Gap | Issue |
344 358
| --- | --- |
345 359
| The live forge cannot serve a full clone of its own repository | #179 |
360
| The exit surfaces are proven on `main` and absent from the deployed forge | #187 |
346 361
| The published WAL anchor is served by the operator and witnessed by nobody, so a consistent rewrite is caught only by a reader who kept a copy | #151 |
347 362
| No column is encrypted at rest, so the operator reads the source every export is built from | #193 |
348
| Five of six exit rehearsals have never been performed | #180 |
363
| A mirror is configured while two contracts say none is, and it holds 307 commits the forge cannot serve | #188 |
364
| The documented WAL rebuild command does not exist | #189 |
365
| `verify/1` cannot be reached from the repository name the configuration admits | #190 |
366
| A backdated issuer-key retirement silently unverifies attestations | #191 |
367
| The machine pairing vault cannot survive a GitHub key rotation | #192 |
368
| No WAL copy exists outside storage the operator solely controls, so operator loss is unrehearsable | #151, #168 |
ops/ci/allowed-sarah-references.txt modified +9

@@ -64,3 +64,12 @@

64 64
# Persona-specific recording layout and source-manifest vocabulary.
65 65
^(assets|lib|test)/.*:[0-9]+:.*mic_left_sarah_right
66 66
^(assets|lib|test)/.*:[0-9]+:.*(sarah_authored_speech|(final|prepared)_sarah_script|sarah-(corpus|runbook|contracts|knowledge))
67
68
# Production infrastructure still carries pre-rename names: the fleet instances
69
# behind openagents.com are sarah-fleet-1..3, the WAL bucket is sarah-forge-wal,
70
# and the Cloud SQL instance is sarah-postgres. The exit rehearsals record where
71
# the WAL actually lives and which host each command ran on, because "a second
72
# operator obtains nothing" is a claim about a named bucket in a named project
73
# rather than a general sentiment. Renaming the infrastructure is a separate
74
# operator action; naming it accurately here is not optional.
75
^docs/forge-exit-rehearsals\.md:[0-9]+:.*sarah-(fleet-[0-9]|forge-wal|postgres)
test/openagents/forge/key_rotation_test.exs added +358

@@ -0,0 +1,358 @@

1
defmodule OpenAgents.Forge.KeyRotationTest do
2
  @moduledoc """
3
  Rehearsal 4 of `docs/forge-exit-rehearsals.md`, which had no executable
4
  proof at all until #180 performed it.
5
6
  The rehearsal asks two things of every key-like secret this forge holds:
7
  that rotating it leaves already-issued receipts verifiable, and that a
8
  rotation performed in the wrong order is refused rather than silently
9
  invalidating history.
10
11
  The first holds everywhere. The second holds in one of the four families,
12
  and the two places it does not are pinned here with the issues that carry
13
  them, so a fix turns a test red instead of passing unnoticed. A rehearsal
14
  that recorded only the half that works would be the kind of claim `EXIT-006`
15
  exists to prevent.
16
17
  | Family | Rotation loses nothing | Wrong order refused |
18
  | --- | --- | --- |
19
  | Forge operator token | yes — the principal is a literal, not a derivation | not applicable; there is no order |
20
  | Account `oa_pat_` tokens | yes — digest-only, no key under them | not applicable |
21
  | Reputation issuer key | forward, yes | **no** — #191 |
22
  | GitHub token vault | yes — key id in the envelope, keyring for the old ones | yes |
23
  | Machine pairing vault | **no** — #192 | no |
24
  | Voice recording vault | **no** — no key id, no keyring | no |
25
26
  Forge push receipts sit underneath all of it and depend on none of it:
27
  `OpenAgents.Forge.WAL`'s chain is unkeyed `sha256` and
28
  `OpenAgents.Forge.Verification` reads no secret, so no rotation in this
29
  table can change a `verify/1` verdict. That is asserted rather than assumed,
30
  because "rotation leaves every already-issued receipt verifiable" is only
31
  worth publishing if something checks that receipts never acquired a key
32
  dependency.
33
  """
34
35
  use OpenAgents.DataCase, async: false
36
37
  import OpenAgents.CompensationFixtures
38
  import OpenAgents.IssuesFixtures
39
40
  alias OpenAgents.Accounts.TokenVault, as: GitHubVault
41
  alias OpenAgents.Forge.{Verification, WAL}
42
  alias OpenAgents.Machines.TokenVault, as: MachineVault
43
  alias OpenAgents.Reputation
44
  alias OpenAgents.Reputation.Claim
45
  alias OpenAgents.Voice.RecordingVault
46
47
  describe "forge receipts depend on no key, so no rotation can invalidate one" do
48
    test "the WAL chain link is unkeyed and reproducible from the entry alone" do
49
      entry = %{
50
        "seq" => 1,
51
        "object" => "entries/00000001-abcdef012345",
52
        "format" => "receive_pack",
53
        "principal" => "operator:forge-token",
54
        "pushed_at" => "2026-08-24T00:00:00.000000Z",
55
        "refs" => %{"refs/heads/main" => String.duplicate("a", 40)}
56
      }
57
58
      assert {:ok, link} = WAL.chain_link("", entry)
59
      assert {:ok, ^link} = WAL.chain_link("", entry)
60
      assert link =~ ~r/\A[0-9a-f]{64}\z/
61
62
      # It commits to the entry's contents, so it is a hash of the record and
63
      # not a token issued beside it.
64
      assert {:ok, other} = WAL.chain_link("", %{entry | "principal" => "user:someone"})
65
      refute other == link
66
67
      # Every key-like secret in the application changes underneath it and the
68
      # link is unmoved, because none of them is an input.
69
      rotate_every_secret(fn ->
70
        assert {:ok, ^link} = WAL.chain_link("", entry)
71
      end)
72
    end
73
74
    test "the verifier was compiled against no secret and no vault" do
75
      callees = external_calls(Verification)
76
77
      for module <- [GitHubVault, MachineVault, RecordingVault, Reputation, OpenAgents.ApiTokens] do
78
        refute module in callees,
79
               "#{inspect(module)} reached OpenAgents.Forge.Verification. A receipt that " <>
80
                 "depends on a key stops being verifiable the moment that key rotates, " <>
81
                 "which is what rehearsal 4 exists to rule out."
82
      end
83
    end
84
  end
85
86
  describe "the reputation issuer key" do
87
    setup do
88
      repository = repository_fixture()
89
      issue = issue_fixture(repository)
90
      assert {:ok, policy} = Reputation.admit_policy(operator())
91
      keypair = Claim.generate_keypair()
92
93
      assert {:ok, key} =
94
               Reputation.admit_key(%{public_key: keypair.public_key, issuer: "verifier"})
95
96
      %{
97
        repository: repository,
98
        issue: issue,
99
        policy: policy,
100
        key: key,
101
        signer: %{key_id: key.key_id, private_key: keypair.private_key}
102
      }
103
    end
104
105
    test "retiring forward keeps an attestation verified, and the successor issues", context do
106
      assert {:ok, attestation} = issue(context)
107
      assert {:ok, _retired} = Reputation.retire_key(context.key, DateTime.utc_now())
108
109
      report = Reputation.verify(attestation)
110
      assert report["signature"]["valid"]
111
      assert report["signature"]["key_status"] == "retired"
112
      assert report["verified"]
113
114
      assert {:error, :signing_key_retired} = issue(context)
115
    end
116
117
    test "retiring backward silently unverifies what the key already signed (#191)", context do
118
      assert {:ok, attestation} = issue(context)
119
      assert Reputation.verify(attestation)["verified"]
120
121
      backdated = DateTime.add(attestation.attested_at, -1, :second)
122
      assert {:ok, _retired} = Reputation.retire_key(context.key, backdated)
123
124
      report = Reputation.verify(attestation)
125
126
      # The signature is still valid over an unaltered claim. Only the window
127
      # moved, and it moved because one UPDATE against a row the operator
128
      # controls said so. Nothing refused it and nothing recorded it.
129
      assert report["digest_match"]
130
      assert report["signature"]["valid"]
131
      refute report["signature"]["key_active_at_attestation"]
132
      refute report["verified"]
133
    end
134
135
    test "the forward edge is the one that is guarded", context do
136
      # Issuance under a key that is not active is refused, so the asymmetry
137
      # in #191 is specifically the retirement edge rather than a missing
138
      # window check.
139
      future = DateTime.add(DateTime.utc_now(), 3600, :second)
140
      keypair = Claim.generate_keypair()
141
142
      assert {:ok, key} =
143
               Reputation.admit_key(%{
144
                 public_key: keypair.public_key,
145
                 issuer: "verifier",
146
                 activated_at: future
147
               })
148
149
      assert {:error, :signing_key_retired} =
150
               issue(%{context | signer: %{key_id: key.key_id, private_key: keypair.private_key}})
151
    end
152
  end
153
154
  describe "the three hand-rolled vaults" do
155
    test "the GitHub vault opens envelopes sealed under a retired key" do
156
      first = Base.encode64(:crypto.strong_rand_bytes(32))
157
      second = Base.encode64(:crypto.strong_rand_bytes(32))
158
159
      sealed =
160
        with_env(
161
          [github_token_encryption_key: first, github_token_encryption_key_id: "first"],
162
          fn ->
163
            assert {:ok, sealed, "first"} = GitHubVault.seal_with_metadata("gho_original")
164
            sealed
165
          end
166
        )
167
168
      # The documented order: the retiring key enters the keyring before the
169
      # successor becomes active.
170
      with_env(
171
        [
172
          github_token_encryption_key: second,
173
          github_token_encryption_key_id: "second",
174
          github_token_decryption_keys: %{"first" => first}
175
        ],
176
        fn ->
177
          assert {:ok, "gho_original"} = GitHubVault.open(sealed)
178
          assert {:ok, "first"} = GitHubVault.key_id(sealed)
179
        end
180
      )
181
    end
182
183
    test "the GitHub vault refuses to open an envelope whose key left the keyring" do
184
      first = Base.encode64(:crypto.strong_rand_bytes(32))
185
      second = Base.encode64(:crypto.strong_rand_bytes(32))
186
187
      sealed =
188
        with_env(
189
          [github_token_encryption_key: first, github_token_encryption_key_id: "first"],
190
          fn ->
191
            assert {:ok, sealed, "first"} = GitHubVault.seal_with_metadata("gho_original")
192
            sealed
193
          end
194
        )
195
196
      # The wrong order: the successor is activated and the predecessor was
197
      # never added to the keyring. This is the one family where the wrong
198
      # order fails closed rather than losing data quietly, and the rewrap
199
      # that would follow rolls back rather than writing an unopenable row.
200
      with_env(
201
        [
202
          github_token_encryption_key: second,
203
          github_token_encryption_key_id: "second",
204
          github_token_decryption_keys: %{}
205
        ],
206
        fn ->
207
          assert {:error, reason} = GitHubVault.open(sealed)
208
          assert reason in [:token_unsealable, :token_vault_not_configured]
209
        end
210
      )
211
    end
212
213
    test "the machine pairing vault reads the GitHub vault's active key (#192)" do
214
      first = Base.encode64(:crypto.strong_rand_bytes(32))
215
      second = Base.encode64(:crypto.strong_rand_bytes(32))
216
217
      sealed =
218
        with_env([github_token_encryption_key: first], fn ->
219
          assert {:ok, sealed} = MachineVault.seal("smct_pairing")
220
          assert {:ok, "smct_pairing"} = MachineVault.open(sealed)
221
          sealed
222
        end)
223
224
      # Rotating the GitHub key — the documented procedure, performed in the
225
      # documented order — orphans this envelope, because the machine vault
226
      # carries no key id and consults no keyring. The blast radius is the ten
227
      # minutes of unclaimed pairings the lifetime allows, which is why this
228
      # is filed rather than treated as an incident.
229
      with_env(
230
        [
231
          github_token_encryption_key: second,
232
          github_token_encryption_key_id: "second",
233
          github_token_decryption_keys: %{"first" => first}
234
        ],
235
        fn ->
236
          assert {:error, :token_unsealable} = MachineVault.open(sealed)
237
        end
238
      )
239
    end
240
241
    test "the voice recording vault has no keyring either, so its key cannot rotate" do
242
      first = Base.encode64(:crypto.strong_rand_bytes(32))
243
      second = Base.encode64(:crypto.strong_rand_bytes(32))
244
      recording = Ecto.UUID.generate()
245
246
      sealed =
247
        with_env([voice_recording_encryption_key: first], fn ->
248
          assert {:ok, sealed} = RecordingVault.seal("audio", recording, 1)
249
          sealed
250
        end)
251
252
      with_env([voice_recording_encryption_key: second], fn ->
253
        assert {:error, :chunk_unsealable} = RecordingVault.open(sealed, recording, 1)
254
      end)
255
    end
256
  end
257
258
  describe "the forge operator token" do
259
    test "rotating it changes no principal a past push already recorded" do
260
      # `operator:forge-token` is a literal `OpenAgents.Forge.GitHTTP` writes
261
      # at push time, not a derivation from the secret, so a rotation cannot
262
      # make a past push attributable to a person and cannot make it
263
      # unattributable either. There is no ordering to get wrong, and no
264
      # overlap window: the cutover is hard, and a pusher holding the old
265
      # value is refused rather than accepted under a stale principal.
266
      previous = Application.get_env(:openagents, :forge_operator_token)
267
      on_exit(fn -> restore(:forge_operator_token, previous) end)
268
269
      entry = %{
270
        "seq" => 7,
271
        "object" => "entries/00000007-0123456789ab",
272
        "format" => "receive_pack",
273
        "principal" => "operator:forge-token",
274
        "pushed_at" => "2026-08-24T00:00:00.000000Z",
275
        "refs" => %{"refs/heads/main" => String.duplicate("b", 40)}
276
      }
277
278
      Application.put_env(:openagents, :forge_operator_token, "before-rotation")
279
      assert {:ok, link} = WAL.chain_link("", entry)
280
281
      Application.put_env(:openagents, :forge_operator_token, "after-rotation")
282
      assert {:ok, ^link} = WAL.chain_link("", entry)
283
      assert entry["principal"] == "operator:forge-token"
284
    end
285
  end
286
287
  defp issue(context) do
288
    decision = outcome_decision_fixture()
289
290
    Reputation.issue(context.policy, context.signer, %{
291
      event_type: "completion",
292
      subject_id: "actor:builder",
293
      outcome: %{kind: "compensation_outcome_decision", ref: decision.decision_receipt_ref},
294
      repository: context.repository,
295
      issue_number: context.issue.number,
296
      revision: String.duplicate("a", 40),
297
      artifact_digest: OpenAgents.Provenance.Canonical.digest!(%{"nonce" => Claim.nonce()}),
298
      confidence_ppm: 900_000,
299
      transparency_tier: "public",
300
      evidence: [
301
        %{
302
          kind: "outcome",
303
          ref: decision.decision_receipt_ref,
304
          digest: decision.outcome_digest,
305
          observed_at: DateTime.to_iso8601(DateTime.utc_now())
306
        }
307
      ]
308
    })
309
  end
310
311
  defp operator do
312
    %{
313
      authenticated: true,
314
      actor_id: "operator:test",
315
      auth_method: "test_session",
316
      approval_receipt_ref: "rotation:#{System.unique_integer([:positive])}"
317
    }
318
  end
319
320
  defp rotate_every_secret(assertion) do
321
    keys = [
322
      :forge_operator_token,
323
      :github_token_encryption_key,
324
      :github_token_encryption_key_id,
325
      :voice_recording_encryption_key
326
    ]
327
328
    previous = Enum.map(keys, &{&1, Application.get_env(:openagents, &1)})
329
    on_exit(fn -> Enum.each(previous, fn {key, value} -> restore(key, value) end) end)
330
331
    for key <- keys do
332
      Application.put_env(:openagents, key, Base.encode64(:crypto.strong_rand_bytes(32)))
333
    end
334
335
    assertion.()
336
  end
337
338
  defp with_env(settings, function) do
339
    previous =
340
      Enum.map(settings, fn {key, _value} -> {key, Application.get_env(:openagents, key)} end)
341
342
    Enum.each(settings, fn {key, value} -> Application.put_env(:openagents, key, value) end)
343
344
    try do
345
      function.()
346
    after
347
      Enum.each(previous, fn {key, value} -> restore(key, value) end)
348
    end
349
  end
350
351
  defp restore(key, nil), do: Application.delete_env(:openagents, key)
352
  defp restore(key, value), do: Application.put_env(:openagents, key, value)
353
354
  defp external_calls(module) do
355
    {:ok, {^module, [imports: imports]}} = :beam_lib.chunks(:code.which(module), [:imports])
356
    Enum.map(imports, &elem(&1, 0))
357
  end
358
end

This page updates live while a promote is in flight · changelog