Say how the production image is actually built, and what to verify

8f282e9f8b27 · AtlantisPleb · · parent 815a5b062f01

Say how the production image is actually built, and what to verify

Deploying c67cf39 walked into four places where the runbook described a
procedure that no longer exists, or a check that cannot pass.

Step 3 sent the operator to `ops/deploy/build-image.sh`, which cannot produce a
production image on an Apple Silicon workstation: it boots the amd64 image to
check the packaged revision, that boot runs emulated, and the Erlang VM dies at
kernel start. `ab5395c` added `ops/deploy/build-image-cloud.sh` for exactly this
and the runbook never mentioned it, so the documented path spends a build to
reach a wall the repository already knew about. Both scripts are described now,
with the reason to prefer each.

`ops/deploy/release-to-production.sh` drives promote, roll, and settle, and did
not appear anywhere. An operator following the runbook literally would drive
three steps by hand that one script already does, including the refusals that
keep a half-rolled fleet from reporting success.

Step 6 said to run one migration job without saying how to run one. It is the
startup template's `migrate-now`, invoked as `sudo bash`, because the node
mounts /tmp noexec and executing the file directly fails with a message about
permissions rather than about the mount. The step now also says to verify what
a migration wrote rather than read its exit status: this release moved an
account allowance onto a column and backfilled it, and a migration that
reported success while writing the new figure across existing rows would have
re-priced every live account with nothing in the output to show it.

Step 9 asked for two things that cannot be observed. Neither `/status` nor
`/api/status` renders an image digest, so the check now points at the settled
target's rolling authority and at the per-node `image_matches_live`, which is
where that identity is actually recorded. And `raft=3` is wrong while
`OPENAGENTS_FEATURE_RA` is false: the fleet reports `raft=0`, so the runbook
was asking the operator to treat a healthy roll as a failed one.

Step 1 gains the untracked-file rule the gate enforces but did not state, and
the note that main advancing mid-deploy does not invalidate the candidate --
it means the later commits are not deployed, which belongs in the report.

Deploy story

What this commit did to the running system — joined from the forge receipt chain, the part a commit page elsewhere cannot show.

pushed
by user · WAL seq 453 · 2026-08-26T16:38:40.594798Z

Changed files

  • modified docs/operations/production-deploy-runbook.md

Diff

1 file changed, +125 -18

docs/operations/production-deploy-runbook.md modified +125 -18

@@ -25,6 +25,31 @@ Two invariants govern everything:

25 25
- **One immutable image digest.** Deployment identity is a `sha256:` digest,
26 26
  never a mutable tag.
27 27
28
## The short path
29
30
Steps 5, 7, and 8 — promote, roll, and settle — are driven by one script:
31
32
```sh
33
ops/deploy/release-to-production.sh <full-sha>
34
```
35
36
It reads the gate receipt rather than rerunning the gate, resolves the image
37
by digest, promotes and waits for the forge to write a complete build receipt,
38
replaces one node at a time, and settles the target. It refuses rather than
39
guesses: no passing gate receipt, an image whose embedded revision is not the
40
Git SHA, or a node that returns on the wrong revision each stop it where it
41
stands, because a half-rolled fleet that reports success is worse than one that
42
stops. It also skips a node that already serves the SHA with the digest
43
pinned, because restarting such a node serves `502` for the length of the
44
restart for no gain.
45
46
If it refuses, do not work around the refusal. Read what it refused on.
47
48
The script does not run the release gate (step 2), build the image (step 3),
49
or run the migration job (step 6). Do those first. The remaining sections
50
describe what the script automates, and what to do when a step needs to be
51
driven by hand.
52
28 53
## 0. One-time workstation prerequisites
29 54
30 55
The gate fails fast when any of these is missing. Set them up once per

@@ -61,13 +86,28 @@ machine:

61 86
62 87
## 1. Pick the candidate
63 88
64
1. Work from a clean worktree: `git status --porcelain` prints nothing.
89
1. Work from a clean worktree. Both the gate and the image build check
90
   `git status --porcelain --untracked-files=all`, so an untracked file stops
91
   them just as a modified one does. A long-lived checkout collects logs, PID
92
   files, and other agents' worktree directories, and none of it is yours to
93
   move or delete. Add those paths to `.git/info/exclude` instead: nothing is
94
   moved, modified, or committed, and only your own `git status` changes.
95
   Restore the file when the deploy is done. Before doing that, confirm the
96
   paths cannot reach the image — the `Dockerfile` copies `VERSION`,
97
   `mix.exs`, `mix.lock`, `config/`, `priv`, `lib`, `rel`, and `assets`, and
98
   nothing else.
65 99
2. The candidate is the exact SHA of forge `main`. Fetch and confirm
66
   `git rev-parse HEAD` equals `git rev-parse origin/main`.
100
   `git rev-parse HEAD` equals `git rev-parse openagents/main`. The forge
101
   remote is `openagents`; `origin` is the GitHub mirror.
67 102
3. If you have local commits, push them first: `git push openagents HEAD:main`
68 103
   (never GitHub; the push guard refuses non-forge pushes).
69 104
4. Record the full SHA. It appears in every later step.
70 105
106
`main` advancing while you deploy does not invalidate a candidate you have
107
already gated, built, and rolled. It means the commits that landed after it
108
are not deployed. Say so in the report rather than implying the fleet runs the
109
tip.
110
71 111
## 2. Run the release gate
72 112
73 113
```sh

@@ -89,25 +129,38 @@ commit whose gate did not complete.

89 129
90 130
## 3. Build and publish the immutable image
91 131
132
On an Apple Silicon workstation, build on Cloud Build:
133
92 134
```sh
93
ops/deploy/build-image.sh openagents:<full-sha>
135
ops/deploy/build-image-cloud.sh <full-sha>
94 136
```
95 137
96
The script verifies the gate receipt (`ops/ci/gate.sh --verify`), builds the
97
`final` Docker target for `linux/amd64`, boots the packaged release to check
98
that the embedded `OpenAgents.BuildInfo.revision()` equals the SHA, and writes
99
a receipt to `.git/openagents/images/<sha>.json`.
100
101
Publish to Artifact Registry and capture the **registry** digest — the digest
102
printed by `docker push` is the deployment identity, not the local image ID:
138
Cloud Build workers are native `amd64`, so both the build and the packaged
139
revision check run for real. The script refuses before it spends a build if the
140
worktree is not at the exact SHA, if the worktree is not clean, or if the tag
141
already exists. That last refusal matters: tags in this repository are
142
immutable, so a wrong image cannot be replaced and the SHA is unusable
143
forever. The script builds the `final` target, asserts that the packaged
144
`OpenAgents.BuildInfo.revision()` equals the SHA, and pushes to Artifact
145
Registry itself. No separate `docker push` step is needed.
146
147
`ops/deploy/build-image.sh` does the same work locally, and on a native
148
`amd64` machine it is still the shorter path. Do not reach for it on Apple
149
Silicon. It boots the `linux/amd64` image to check the packaged revision, that
150
boot runs under emulation, and the Erlang VM cannot start there. It dies at
151
kernel start with `failed_to_start_child,user,nouser`, which is reproducible on
152
the bare `hexpm/elixir` base image and unaffected by `-noinput` or `TERM=dumb`.
153
A cached layer can hide this until the Docker cache is cleared.
154
155
Record the published `sha256:` digest — it is the deployment identity, not the
156
local image ID:
103 157
104 158
```sh
105
docker tag <local-digest> us-central1-docker.pkg.dev/openagents-staging-20260820/openagents-staging/openagents:<full-sha>
106
docker push us-central1-docker.pkg.dev/openagents-staging-20260820/openagents-staging/openagents:<full-sha>
159
gcloud artifacts docker images describe \
160
  us-central1-docker.pkg.dev/openagents-staging-20260820/openagents-staging/openagents:<full-sha> \
161
  --format='value(image_summary.digest)' --project openagentsgemini
107 162
```
108 163
109
Record the pushed `sha256:` digest.
110
111 164
## 4. Confirm runtime secrets and environment
112 165
113 166
The fleet startup script (instance metadata key `startup-script` on

@@ -279,6 +332,44 @@ Run exactly one migration job for the release; nodes must not race it.

279 332
lineage bridge first, and require the lineage classification and integrity
280 333
checks to pass before touching the fleet.
281 334
335
Fill the startup template with the digest you published and run its
336
`migrate-now` subcommand on one node. This runs the release's own
337
`OpenAgents.Release.migrate()` in a throwaway container against the new image,
338
and leaves the node's running container alone:
339
340
```sh
341
sed "s|__IMAGE_DIGEST__|<digest>|g" ops/deploy/fleet-startup.template.sh > /tmp/startup.sh
342
gcloud compute scp /tmp/startup.sh sarah-fleet-1:/tmp/startup.sh \
343
  --zone us-central1-a --project openagentsgemini
344
gcloud compute ssh sarah-fleet-1 --zone us-central1-a --project openagentsgemini \
345
  --tunnel-through-iap --command="sudo bash /tmp/startup.sh migrate-now"
346
```
347
348
Invoke it as `sudo bash /tmp/startup.sh`, not `sudo /tmp/startup.sh`. The
349
node mounts `/tmp` `noexec`, so executing the file directly fails with
350
`Permission denied` and says nothing about why.
351
352
The fleet also sets `OPENAGENTS_MIGRATE_ON_BOOT=true`, so the first replaced
353
node would migrate during boot anyway, serialized by the same advisory lock
354
(RELEASE-001). Running the job first is still worth the extra step: it puts
355
the schema change before the roll, where you can verify it while the fleet is
356
still serving the previous release and nothing has been disturbed.
357
358
Verify what the migration wrote, rather than reading its exit status. A
359
migration that reports success can still have written the wrong values, and a
360
data-bearing migration is the one place where that is expensive and silent.
361
Query the affected rows directly:
362
363
```sh
364
docker exec openagents /app/bin/openagents rpc 'Code.eval_file("/tmp/check.exs")'
365
```
366
367
For a column added with a backfill, check three things: the row values the
368
backfill was supposed to write, the column's `is_nullable` and
369
`column_default` in `information_schema.columns`, and any constraint the
370
migration created. Capture the same counts before the migration runs so the
371
after-state has something to be compared against.
372
282 373
## 7. Rolling replacement
283 374
284 375
`OpenAgents.Forge.RollingReplacement.run/2` replaces one node at a time. Its

@@ -329,14 +420,30 @@ and keeps every recorded observation.

329 420
330 421
Do not report success without all of the following:
331 422
332
- `/status` and `/api/status` return the exact SHA and image digest.
333
- All three nodes report the same revision and digest; `beam=3`, `raft=3`,
334
  quorum holds.
423
- `/status` and `/api/status` return the exact SHA as `revision`. Neither
424
  renders the image digest, so read the digest from the settled target's
425
  `details.rolling_authority`, which records the authorized digest and the
426
  identity each node was observed at. In `/api/status`, every entry of `nodes`
427
  reporting `boot.reason: "image_matches_live"` is the node-level statement
428
  that it runs the authorized digest, because boot convergence admits against
429
  the published identity rather than the SHA alone.
430
- All three nodes report the same revision; `beam=3` and quorum holds.
431
  `raft` is `0` while `OPENAGENTS_FEATURE_RA` is `false`, which is the current
432
  fleet setting — do not read `raft=0` as a failed roll.
335 433
- All three backends of `sarah-backend` are healthy.
336
- The migration appears in `schema_migrations`.
434
- The migration appears in `schema_migrations`, and its data effects are
435
  verified per step 6 rather than assumed from its exit status.
337 436
- Login, a typed chat turn, and a durable reload work.
338 437
- Issues, git clone/fetch, and a read-only computer job work.
339 438
- New configuration is live without exposing secret values.
439
- Any static asset the release changed is verified from the **served bytes**,
440
  not from the repository. Fetch it and compare its hash against the file at
441
  the deployed SHA:
442
443
  ```sh
444
  curl -fsS https://openagents.com/install.sh | shasum -a 256
445
  shasum -a 256 priv/static/install.sh
446
  ```
340 447
341 448
## Load balancer health check
342 449

This page updates live while a promote is in flight · changelog