OpenAgents

Commit

Add the end-to-end production deploy runbook

7fd39028f8ac · Devin AI · · parent 2bb4381276e1 

Add the end-to-end production deploy runbook

One checklist from candidate selection through the release gate, image
build and publish, secret wiring, promotion, migration, rolling
replacement, settlement, and verification, with the known failure
modes and their fixes.

Co-Authored-By: Christopher David <chris@openagents.com>
Co-Authored-By
Christopher David <chris@openagents.com>

Deploy story

What this commit did to the running system — joined from the forge receipt chain, the part a commit page elsewhere cannot show.

Not deployed through the forge lane

No push, promotion, build, or deploy receipt references this commit (receipts are scanned over a bounded recent window). Changes shipped by full node replacement carry their proof in the release gate receipt instead.

Changed files

  • added docs/operations/production-deploy-runbook.md

Diff

1 file changed, +205 -0

docs/operations/production-deploy-runbook.md added +205

@@ -0,0 +1,205 @@

1 
# Production deploy runbook
2
3 
This is the single end-to-end procedure for deploying `openagents.com` to the
4 
production fleet. Follow it top to bottom. Every other operations document is
5 
reference material; this one is the checklist.
6
7 
The pipeline is:
8
9 
```
10 
push candidate to forge main
11 
  → release gate (exact SHA)
12 
  → immutable image build and publish
13 
  → promote on /admin/forge
14 
  → migration job
15 
  → rolling replacement (structural) or direct deploy (BEAM-only)
16 
  → settle the target
17 
  → verify production
18 
```
19
20 
Two invariants govern everything:
21
22 
- **One exact Git SHA.** The gate receipt, the image, the promotion, and the
23 
  rolling replacement all name the same full commit SHA. If `main` advances,
24 
  you have a new candidate and you start over at the gate.
25 
- **One immutable image digest.** Deployment identity is a `sha256:` digest,
26 
  never a mutable tag.
27
28 
## 0. One-time workstation prerequisites
29
30 
The gate fails fast when any of these is missing. Set them up once per
31 
machine:
32
33 
1. Tools on `PATH`: `jq`, `mix`, `npm`, `docker`, and `terraform` satisfying
34 
   `>= 1.11, < 2.0` (the staging infrastructure gate checks
35 
   `ops/staging/infra/versions.tf`).
36 
2. JavaScript dependencies: run `npm ci --prefix assets`. The relup stage runs
37 
   `mix assets.deploy`, which needs `assets/node_modules` (for example
38 
   `posthog-js`). A missing install fails the relup stage, not the javascript
39 
   stage.
40 
3. A local PostgreSQL server with the `vector` extension available, plus a
41 
   disposable database the gate may trash:
42
43 
   ```sh
44 
   createdb openagents_release_smoke
45 
   psql -d openagents_release_smoke -c 'CREATE EXTENSION IF NOT EXISTS vector;'
46 
   ```
47
48 
   The `OPENAGENTS_RELEASE_SMOKE_DISPOSABLE=1` flag asserts the database is
49 
   disposable; it does not create it. The version-chain and release-smoke
50 
   stages boot real releases against this database and fail during startup if
51 
   it does not exist.
52 
4. Operator gcloud auth with compute and Secret Manager access in
53 
   `openagentsgemini`, and Docker configured for Artifact Registry:
54
55 
   ```sh
56 
   gcloud auth login <operator>@openagents.com --no-browser
57 
   gcloud auth configure-docker us-central1-docker.pkg.dev
58 
   ```
59
60 
## 1. Pick the candidate
61
62 
1. Work from a clean worktree: `git status --porcelain` prints nothing.
63 
2. The candidate is the exact SHA of forge `main`. Fetch and confirm
64 
   `git rev-parse HEAD` equals `git rev-parse origin/main`.
65 
3. If you have local commits, push them first: `git push openagents HEAD:main`
66 
   (never GitHub; the push guard refuses non-forge pushes).
67 
4. Record the full SHA. It appears in every later step.
68
69 
## 2. Run the release gate
70
71 
```sh
72 
OPENAGENTS_RELEASE_SMOKE_DISPOSABLE=1 \
73 
OPENAGENTS_RELEASE_SMOKE_DATABASE_URL='ecto://USER:PASSWORD@127.0.0.1/openagents_release_smoke' \
74 
ops/ci/gate.sh
75 
```
76
77 
The gate runs, in order: `compile`, `production_compile`, `precommit`,
78 
`cluster`, `javascript`, `direct_transaction`, `relup`, `version_chain`,
79 
`interrupted_install`, `rolling_replacement`, `contracts`, `staging_infra`,
80 
and `release_smoke`. It writes a receipt keyed to the exact SHA under
81 
`.git/openagents/`. A receipt from a different SHA — even a docs-only parent —
82 
is invalid.
83
84 
Budget 30 to 60 minutes. If a stage fails, fix the cause, push the fix to
85 
forge `main`, and rerun the whole gate against the new SHA. Do not deploy a
86 
commit whose gate did not complete.
87
88 
## 3. Build and publish the immutable image
89
90 
```sh
91 
ops/deploy/build-image.sh openagents:<full-sha>
92 
```
93
94 
The script verifies the gate receipt (`ops/ci/gate.sh --verify`), builds the
95 
`final` Docker target for `linux/amd64`, boots the packaged release to check
96 
that the embedded `OpenAgents.BuildInfo.revision()` equals the SHA, and writes
97 
a receipt to `.git/openagents/images/<sha>.json`.
98
99 
Publish to Artifact Registry and capture the **registry** digest — the digest
100 
printed by `docker push` is the deployment identity, not the local image ID:
101
102 
```sh
103 
docker tag <local-digest> us-central1-docker.pkg.dev/openagents-staging-20260820/openagents-staging/openagents:<full-sha>
104 
docker push us-central1-docker.pkg.dev/openagents-staging-20260820/openagents-staging/openagents:<full-sha>
105 
```
106
107 
Record the pushed `sha256:` digest.
108
109 
## 4. Confirm runtime secrets and environment
110
111 
The fleet startup script (instance metadata key `startup-script` on
112 
`sarah-fleet-1/2/3`) resolves secrets from Secret Manager at boot and passes
113 
them to the app container through the `ENV_NAMES` array. A new runtime
114 
environment variable needs three things:
115
116 
1. A Secret Manager secret in `openagentsgemini`, with
117 
   `roles/secretmanager.secretAccessor` granted to
118 
   `oa-mvp-automation@openagentsgemini.iam.gserviceaccount.com`.
119 
2. An `export NAME="$(secret <secret-name>)"` line in the startup script.
120 
3. The name added to `ENV_NAMES`.
121
122 
Edit one copy of the script, `diff` it against each instance's live metadata
123 
(all three must be identical), then apply with:
124
125 
```sh
126 
gcloud compute instances add-metadata sarah-fleet-N --zone <zone> \
127 
  --project openagentsgemini \
128 
  --metadata-from-file startup-script=<file>
129 
```
130
131 
Metadata changes take effect at the next instance reset, which the rolling
132 
replacement performs. Never print secret values.
133
134 
## 5. Promote and classify
135
136 
Promote the exact SHA through `/admin/forge`. The target moves
137 
`promoted → building → built`, and the classifier decides the deployment
138 
route:
139
140 
- **Direct deploy**: the diff touches only allowlisted BEAM modules. The
141 
  forge hot loop handles it; watch the target go `deploying → live`.
142 
- **`needs_rolling_replace`**: anything structural — `config/*`, migrations,
143 
  dependencies, assets, ERTS, or release-private files. Continue below.
144
145 
Keep the classification receipt.
146
147 
## 6. Migration job
148
149 
Run exactly one migration job for the release; nodes must not race it.
150 
`bin/migrate` takes the release advisory lock. Apply any reviewed migration
151 
lineage bridge first, and require the lineage classification and integrity
152 
checks to pass before touching the fleet.
153
154 
## 7. Rolling replacement
155
156 
`OpenAgents.Forge.RollingReplacement.run/2` replaces one node at a time. Its
157 
request names the exact current and previous SHAs, the exact current and
158 
previous image digests, the expected node list, and the fleet size. For each
159 
node it removes readiness, drains to zero active work, verifies remaining
160 
capacity and quorum, resets the instance with the target digest (the GCP
161 
provider sets `openagents-image`, `openagents-image-digest`, and
162 
`openagents-sha` metadata), then waits for membership, readiness, boot
163 
convergence, database readiness, and the exact SHA and digest before moving
164 
on.
165
166 
Order: `sarah-fleet-1` (us-central1-a), `sarah-fleet-2` (us-central1-b),
167 
`sarah-fleet-3` (us-central1-c). Two healthy nodes must exist before each
168 
replacement. If a node fails to rejoin, the coordinator rolls back to the
169 
previous digest; wait for full health and stop — do not continue the roll.
170
171 
## 8. Settle the target
172
173 
```elixir
174 
OpenAgents.Forge.Targets.finish_rolling_replacement(target_id, rolling_result)
175 
```
176
177 
Settlement demands the newest target in `needs_rolling_replace`, a complete
178 
verified build receipt, and a rolling result whose SHA and image identities
179 
match the target. Success flips the target to `live` and inserts the immutable
180 
deployment receipt.
181
182 
## 9. Verify production
183
184 
Do not report success without all of the following:
185
186 
- `/status` and `/api/status` return the exact SHA and image digest.
187 
- All three nodes report the same revision and digest; `beam=3`, `raft=3`,
188 
  quorum holds.
189 
- All three backends of `sarah-backend` are healthy.
190 
- The migration appears in `schema_migrations`.
191 
- Login, a typed chat turn, and a durable reload work.
192 
- Issues, git clone/fetch, and a read-only computer job work.
193 
- New configuration is live without exposing secret values.
194
195 
## Known failure modes
196
197 
| Symptom | Cause | Fix |
198 
| --- | --- | --- |
199 
| `cluster` stage: `report["consistent"] == true` fails | A `local_report` dependency crashes on peers where the app is loaded but not started | Health-report callees must fail closed (catch `:exit`), never raise |
200 
| `relup` stage: esbuild cannot resolve a package | `assets/node_modules` missing | `npm ci --prefix assets` |
201 
| `version_chain` / `release_smoke`: `invalid_catalog_name` | Disposable database does not exist | `createdb` + `CREATE EXTENSION vector` |
202 
| `staging_infra`: terraform missing or version unsupported | No terraform, or version outside `>= 1.11, < 2.0` | Install a supported terraform |
203 
| Push rejected (non-fast-forward) | Forge `main` advanced | Fetch, rebase your commit, push, restart the gate on the new SHA |
204 
| `gate.sh --verify` fails in `build-image.sh` | Receipt is for a different SHA | Rerun the gate on the exact candidate |
205 
| App boots without a new variable | Name missing from `ENV_NAMES` in the startup script | Add the export and the `ENV_NAMES` entry, re-apply metadata |

This page updates live while a promote is in flight · changelog