Say how the production image is actually built, and what to verify Deploying c67cf39 walked into four places where the runbook described a procedure that no longer exists, or a check that cannot pass. Step 3 sent the operator to `ops/deploy/build-image.sh`, which cannot produce a production image on an Apple Silicon workstation: it boots the amd64 image to check the packaged revision, that boot runs emulated, and the Erlang VM dies at kernel start. `ab5395c` added `ops/deploy/build-image-cloud.sh` for exactly this and the runbook never mentioned it, so the documented path spends a build to reach a wall the repository already knew about. Both scripts are described now, with the reason to prefer each. `ops/deploy/release-to-production.sh` drives promote, roll, and settle, and did not appear anywhere. An operator following the runbook literally would drive three steps by hand that one script already does, including the refusals that keep a half-rolled fleet from reporting success. Step 6 said to run one migration job without saying how to run one. It is the startup template's `migrate-now`, invoked as `sudo bash`, because the node mounts /tmp noexec and executing the file directly fails with a message about permissions rather than about the mount. The step now also says to verify what a migration wrote rather than read its exit status: this release moved an account allowance onto a column and backfilled it, and a migration that reported success while writing the new figure across existing rows would have re-priced every live account with nothing in the output to show it. Step 9 asked for two things that cannot be observed. Neither `/status` nor `/api/status` renders an image digest, so the check now points at the settled target's rolling authority and at the per-node `image_matches_live`, which is where that identity is actually recorded. And `raft=3` is wrong while `OPENAGENTS_FEATURE_RA` is false: the fleet reports `raft=0`, so the runbook was asking the operator to treat a healthy roll as a failed one. Step 1 gains the untracked-file rule the gate enforces but did not state, and the note that main advancing mid-deploy does not invalidate the candidate -- it means the later commits are not deployed, which belongs in the report.
Say how the production image is actually built, and what to verify
Deploy story
What this commit did to the running system — joined from the forge receipt chain, the part a commit page elsewhere cannot show.
- pushed
- by user · WAL seq 453 · 2026-08-26T16:38:40.594798Z
Changed files
-
modified
docs/operations/production-deploy-runbook.md
Diff
1 file changed, +125 -18
docs/operations/production-deploy-runbook.md modified +125 -18
@@ -25,6 +25,31 @@ Two invariants govern everything:
| 25 | 25 |
|
| 26 | 26 |
|
| 27 | 27 |
|
| 28 |
|
|
| 29 |
|
|
| 30 |
|
|
| 31 |
|
|
| 32 |
|
|
| 33 |
|
|
| 34 |
|
|
| 35 |
|
|
| 36 |
|
|
| 37 |
|
|
| 38 |
|
|
| 39 |
|
|
| 40 |
|
|
| 41 |
|
|
| 42 |
|
|
| 43 |
|
|
| 44 |
|
|
| 45 |
|
|
| 46 |
|
|
| 47 |
|
|
| 48 |
|
|
| 49 |
|
|
| 50 |
|
|
| 51 |
|
|
| 52 |
|
|
| 28 | 53 |
|
| 29 | 54 |
|
| 30 | 55 |
|
@@ -61,13 +86,28 @@ machine:
| 61 | 86 |
|
| 62 | 87 |
|
| 63 | 88 |
|
| 64 |
|
|
| 89 |
|
|
| 90 |
|
|
| 91 |
|
|
| 92 |
|
|
| 93 |
|
|
| 94 |
|
|
| 95 |
|
|
| 96 |
|
|
| 97 |
|
|
| 98 |
|
|
| 65 | 99 |
|
| 66 |
|
|
| 100 |
|
|
| 101 |
|
|
| 67 | 102 |
|
| 68 | 103 |
|
| 69 | 104 |
|
| 70 | 105 |
|
| 106 |
|
|
| 107 |
|
|
| 108 |
|
|
| 109 |
|
|
| 110 |
|
|
| 71 | 111 |
|
| 72 | 112 |
|
| 73 | 113 |
|
@@ -89,25 +129,38 @@ commit whose gate did not complete.
| 89 | 129 |
|
| 90 | 130 |
|
| 91 | 131 |
|
| 132 |
|
|
| 133 |
|
|
| 92 | 134 |
|
| 93 |
|
|
| 135 |
|
|
| 94 | 136 |
|
| 95 | 137 |
|
| 96 |
|
|
| 97 |
|
|
| 98 |
|
|
| 99 |
|
|
| 100 |
|
|
| 101 |
|
|
| 102 |
|
|
| 138 |
|
|
| 139 |
|
|
| 140 |
|
|
| 141 |
|
|
| 142 |
|
|
| 143 |
|
|
| 144 |
|
|
| 145 |
|
|
| 146 |
|
|
| 147 |
|
|
| 148 |
|
|
| 149 |
|
|
| 150 |
|
|
| 151 |
|
|
| 152 |
|
|
| 153 |
|
|
| 154 |
|
|
| 155 |
|
|
| 156 |
|
|
| 103 | 157 |
|
| 104 | 158 |
|
| 105 |
|
|
| 106 |
|
|
| 159 |
|
|
| 160 |
|
|
| 161 |
|
|
| 107 | 162 |
|
| 108 | 163 |
|
| 109 |
|
|
| 110 |
|
|
| 111 | 164 |
|
| 112 | 165 |
|
| 113 | 166 |
|
@@ -279,6 +332,44 @@ Run exactly one migration job for the release; nodes must not race it.
| 279 | 332 |
|
| 280 | 333 |
|
| 281 | 334 |
|
| 335 |
|
|
| 336 |
|
|
| 337 |
|
|
| 338 |
|
|
| 339 |
|
|
| 340 |
|
|
| 341 |
|
|
| 342 |
|
|
| 343 |
|
|
| 344 |
|
|
| 345 |
|
|
| 346 |
|
|
| 347 |
|
|
| 348 |
|
|
| 349 |
|
|
| 350 |
|
|
| 351 |
|
|
| 352 |
|
|
| 353 |
|
|
| 354 |
|
|
| 355 |
|
|
| 356 |
|
|
| 357 |
|
|
| 358 |
|
|
| 359 |
|
|
| 360 |
|
|
| 361 |
|
|
| 362 |
|
|
| 363 |
|
|
| 364 |
|
|
| 365 |
|
|
| 366 |
|
|
| 367 |
|
|
| 368 |
|
|
| 369 |
|
|
| 370 |
|
|
| 371 |
|
|
| 372 |
|
|
| 282 | 373 |
|
| 283 | 374 |
|
| 284 | 375 |
|
@@ -329,14 +420,30 @@ and keeps every recorded observation.
| 329 | 420 |
|
| 330 | 421 |
|
| 331 | 422 |
|
| 332 |
|
|
| 333 |
|
|
| 334 |
|
|
| 423 |
|
|
| 424 |
|
|
| 425 |
|
|
| 426 |
|
|
| 427 |
|
|
| 428 |
|
|
| 429 |
|
|
| 430 |
|
|
| 431 |
|
|
| 432 |
|
|
| 335 | 433 |
|
| 336 |
|
|
| 434 |
|
|
| 435 |
|
|
| 337 | 436 |
|
| 338 | 437 |
|
| 339 | 438 |
|
| 439 |
|
|
| 440 |
|
|
| 441 |
|
|
| 442 |
|
|
| 443 |
|
|
| 444 |
|
|
| 445 |
|
|
| 446 |
|
|
| 340 | 447 |
|
| 341 | 448 |
|
| 342 | 449 |
|