Perform the five exit rehearsals that have never been run #180

Closed AtlantisPleb opened this 6d ago 1 comment

Evidence

Shipped in the release at 0bf2706, promoted 2026-08-27.

1 pushes receipt

Problem

docs/forge-exit-rehearsals.md defines six rehearsals against the forge's exit properties. One has been performed. Five have not, and the status table in that document says so rather than implying otherwise.

The one that was performed found a real defect on the first step (#179), which is the argument for performing the rest: EXIT-001 through EXIT-006 were all green while the live forge could not serve a full clone of its own repository, because every proof runs against a forge the test process builds.

Rehearsal	Executable proof	Performed
1. Restore a repository and its work history	`EXIT-004`, `EXIT-001`	Yes — failed, #179
2. Detect a forged, missing, reordered, or mismatched receipt	`EXIT-002`, `EXIT-005`	No
3. Mirror divergence	`EXIT-003`	No
4. Key rotation	None	No
5. Operator loss	`EXIT-003`	No
6. Partial export	`EXIT-001`	No

Rehearsal 4 has no executable proof at all. Rotating the forge operator token, an account's oa_pat_ tokens, the reputation issuer key, and the three hand-rolled vaults are four different procedures with four different failure modes, and none of them has been exercised.

Contract

Perform rehearsals 2 through 6 against the live forge, and record the result of each in docs/forge-exit-rehearsals.md — including the ones that fail. A rehearsal that finds nothing is a result; a rehearsal that is skipped is not.
Where a rehearsal finds a defect, file it rather than repairing it inside the rehearsal record, so the record stays a record.
For rehearsal 4, establish that a rotation of each key leaves every already-issued receipt verifiable, and that a rotation performed in the wrong order is refused rather than silently invalidating history.
For rehearsal 5, state plainly what a second operator cannot obtain. The WAL lives in storage the current operator controls, and no rehearsal changes that.

Acceptance criteria

Every row of the rehearsal status table names a date and a result.
Any defect found is filed with its reproduction.
Rehearsal 4 either gains an executable proof or records why it cannot have one.

Parent: #94.

AtlantisPleb opened this issue 6d ago
AtlantisPleb closed this as completed in 619260d 5d ago

A AtlantisPleb Author 5d ago

Rehearsals 2 through 6 were performed on 2026-08-25 against the live forge on revision 46cf8a5aea3791936c22e82c145a9a8dd734374d, deployed on all three fleet nodes. docs/2026-08-25-forge-exit-rehearsals-2-to-6.md records every step, its output, and its verdict, and the status table in docs/forge-exit-rehearsals.md now names a date and a result on every row.

Rehearsal	Verdict	Filed
2. Receipt verification	Steps 1 and 3 pass live; step 2 local only	#251; #190 still reproduces
3. Mirror divergence	Steps 1 and 2 pass live and confirm #188; step 3 local only	none new
4. Key rotation	Passes in the test process; live half is a read-only inventory	#253
5. Operator loss	Recovery passes in the test process; the limit is now measured	none new
6. Partial export	Passes live; was red on `main`	#252

What each one produced

Rehearsal 2. verify/2 on a fleet node reports findings: [] over 382 entries with chained_from: 279 and a head link at seq 381. The anchor published at /.well-known/openagents-forge-anchor.json was fetched over HTTPS first and then used as the anchor: the published %{seq: 375, link: "19c7a2c5…"} verifies clean, one changed hex digit reports anchor_mismatch, a sequence below chained_from reports anchor_unreachable with "entry carries no link", and a sequence past the end reports it with "no entry at this sequence". Step 2 needs tampering, so it ran against a forge the test process builds — 37 tests, all green, naming each finding code. No tamper was performed against production.

Rehearsal 3. Both remotes advertise the same 25 refs byte for byte, refs/heads/main identical. Full clones of each: the forge serves 423 commits on main, git fsck clean, grafted at five shallow boundaries with eda094c6 as its root; GitHub serves 730, clean, rooted at a352f78e. The 307-commit gap is exactly #188 at today's numbers, and the node confirms openagents.com as the one configured mirror. Step 3 calls rebuild/1, which discards a projection, so it was not run against production.

Rehearsal 4. This rehearsal already has an executable proof, which the runbook recorded as "None": test/openagents/forge/key_rotation_test.exs, 12 tests, all green. It covers both halves of this issue's contract — the WAL chain link is unkeyed so no rotation invalidates a receipt, and a backward retirement is refused naming the attestation it would have unverified, as are one before a silent key's activation and one beyond the clock-skew allowance. That closes the third acceptance criterion. No key was rotated against production; the live half is an inventory read without printing key material, and it found that MACHINE_TOKEN_ENCRYPTION_KEY is unset (#253), the GitHub retired keyring is empty, and no reputation issuer key is admitted — so neither rotation has ever run here.

Rehearsal 5. Recovery from the WAL passes in the test process. The limit this issue asks to be stated plainly is now measured rather than asserted, and it is stronger than the runbook reads: the WAL adapter is OpenAgents.Forge.WAL.Gcs against one bucket in the operator's own cloud project with uniform bucket-level access, and that bucket has no object versioning — soft delete retains a deleted object for seven days and nothing longer. The storage keeps no history of itself, so an operator who rewrote an entry a week ago leaves no earlier copy in it. The only commitment outside that bucket is the anchor, served by the same operator and witnessed by nobody (#151).

Rehearsal 6. GET /api/status publishes 35 families — 24 portable, 0 partial, 1 blocked, 10 not user data — and the deployed ledger reports the same split with no drift. Steps 1 and 2 were exercised through the route against a local forge: 14 caps in bounds, 10 *_truncated flags, 4 families in not_included each with a mechanism and a reason, and a sealed ?recipient= variant that decrypts to the same bounds. One flag was observed positive — a 70,000-byte Box run exports 65,536 with output_truncated: true and output_byte_size: 70000 — which no committed test does, since every *_truncated assertion in the suite is a refute. That proof gap is recorded rather than closed.

Defects filed

#251 — the three nodes disagree about the same repository at the same instant. The WAL is shared and the projection is node-local, so a node that has not replayed an entry yet reports served_refs_diverged and object_missing, which are the findings that mean tampering. Reproduced twice, minutes apart, in both directions. It converges on its own within minutes. It blocks #179's third acceptance criterion, because a scheduled pass built today would publish tampering findings for a healthy forge.
#253 — production never provisioned MACHINE_TOKEN_ENCRYPTION_KEY, so config/runtime.exs still falls through to the boot bridge and the pairing vault runs on the GitHub vault's key. #192's fix is deployed and correct; the secret it needs was not set, so VAULT-001's separation is not deployed and this runbook's "rotating the GitHub key no longer touches this vault" was false against production. An owner action rather than a code change.
#252 — POST /api/v1/responses reached OpenAgentsWeb.ApiRouteAuthority with the family :response and no ledger entry, so EXIT-001's derived coverage was red on main. Answered rather than carried, because a red on main is not a finding to file and leave: the stub reaches no repo and no schema, so the family is not_user_data with a note saying the entry is revisited the day a provider stands behind the route.

#190 still reproduces: verify/2 now accepts an owner/name path, which is its first acceptance criterion, but OpenAgents.Forge.Repos.allowed_repos/0 still returns a name no verifier can use and the stale openagents.com.git bare repository still sits beside the real ones.

What was deliberately not done

Every rehearsal was read-only against production. Nothing deleted a repository, force-pushed, mirrored, rotated a credential, or wrote to the WAL. Three steps mutate state — rehearsal 2's tampering, rehearsal 3's rebuild, and any rotation in rehearsal 4 — and each ran against a forge the test process builds, labeled local in the record. A rehearsal that cannot safely run its own destructive step against production has a permanent gap, and the record says so rather than implying the step was taken.

Commit 619260d.