Update production rollout documentation

7a2e8fe52060 · AtlantisPleb · · parent e2bcca0a3424

Update production rollout documentation

Deploy story

What this commit did to the running system — joined from the forge receipt chain, the part a commit page elsewhere cannot show.

Not deployed through the forge lane

No push, promotion, build, or deploy receipt references this commit (receipts are scanned over a bounded recent window). Changes shipped by full node replacement carry their proof in the release gate receipt instead.

Changed files

  • modified docs/operations/production-cutover.md
  • modified docs/operations/release-deployment-fallbacks.md

Diff

2 files changed, +24 -18

docs/operations/production-cutover.md modified +17 -12

@@ -2,8 +2,8 @@

2 2
3 3
Date: 2026-08-20
4 4
5
Status: Production is fenced until the isolated staging gates and database
6
rehearsal pass for one exact candidate.
5
Status: Production cut over to OpenAgents on 2026-08-21. Use this runbook for
6
the retained rollback boundary and later fleet replacements.
7 7
8 8
This runbook replaces the Sarah release on the existing three-node production
9 9
fleet. It preserves the `openagents.com` load balancer and the `sarah`

@@ -20,14 +20,20 @@ Production currently uses these resources in `openagentsgemini`:

20 20
  `sarah-fleet-3` in three zones.
21 21
- Global backend service `sarah-backend` and the existing `openagents.com`
22 22
  HTTPS frontend.
23
- The Sarah release as the serving application and the idle Sarah Cloud Run
24
  service as the application rollback target.
23
- The OpenAgents release as the serving application and the retained Sarah
24
  image and startup metadata as rollback inputs.
25 25
26 26
Do not rename these provider resources during the application cutover. Rename
27 27
them only in a later infrastructure migration after OpenAgents passes its
28 28
production observation window.
29 29
30
## Production admission conditions
30
## Initial cutover admission conditions
31
32
The following conditions record the initial cutover standard. Current rolling
33
releases do not require a fixed-duration soak. Each rollout must keep two
34
healthy nodes available, verify the exact revision locally on the replaced
35
node, wait for load-balancer health, and stop before the next node if a check
36
fails.
31 37
32 38
Use one exact Git SHA and application image digest for every condition. Do not
33 39
waive a failed condition by rebuilding from another commit during the cutover.

@@ -39,10 +45,8 @@ waive a failed condition by rebuilding from another commit during the cutover.

39 45
  validator. Staging does not use a production database, service account,
40 46
  bucket, or secret.
41 47
- The candidate passes all Gate 14 cases on staging.
42
- The same candidate passes controlled failure injection and a measured
43
  15-minute soak on the pinned `openagents-staging-release` service. Later
44
  commits can deploy to another staging service without replacing this release
45
  candidate or resetting its clock.
48
- The same candidate passes controlled failure injection and staging health
49
  checks on the pinned `openagents-staging-release` service.
46 50
- A fresh production backup restores into a disposable Cloud SQL instance.
47 51
  The migration-lineage bridge, all remaining migrations, candidate startup,
48 52
  last-known-good startup, row-count checks, and integrity checks pass there.

@@ -174,6 +178,7 @@ through the observation window. Then rotate transitional credentials, remove

174 178
the old restart path, enable deletion protection on the fleet instances, and
175 179
schedule provider-resource renames as a separate infrastructure change.
176 180
177
Do not declare production ready until the staging soak and the disposable-copy
178
migration rehearsal complete for the exact candidate. Local parity and a
179
healthy staging deploy are necessary but do not replace those conditions.
181
Production is live. For a later rolling release, preserve the existing database
182
and persistent repository volumes, use one immutable image digest, replace one
183
node at a time, and verify all three load-balancer backends before closing the
184
release.
docs/operations/release-deployment-fallbacks.md modified +7 -6

@@ -2,7 +2,8 @@

2 2
3 3
Date: 2026-08-20
4 4
5
Status: Implemented and locally proven; keep both lanes disabled until the Gate 12 isolated staging fleet exists
5
Status: Implemented. The isolated staging fleet uses the coordinated rolling
6
lane, and production uses the same one-node-at-a-time health boundary.
6 7
7 8
## Purpose
8 9

@@ -15,9 +16,9 @@ classifier must choose one strategy for the complete candidate:

15 16
  configuration, migration, module-deletion, or otherwise unclassified
16 17
  changes.
17 18
18
Neither lane is production-approved. Gate 12 must connect the provider-neutral
19
coordinators to an isolated three-node staging fleet and retain staging-only
20
receipts before anyone enables a deployment worker.
19
The relup lane is not production-approved. Use rolling replacement in
20
production only with explicit operator authority, an immutable image digest,
21
two remaining healthy nodes, and exact revision checks after each replacement.
21 22
22 23
## Local release gate
23 24

@@ -207,5 +208,5 @@ evidence for:

207 208
  node;
208 209
- additive migration compatibility while both application versions run.
209 210
210
Production remains out of scope until the later staging, rollback, and soak
211
gates pass and receive separate approval.
211
Production rolling replacement requires explicit operator authority. Do not
212
infer that authority from a successful staging rollout.

This page updates live while a promote is in flight · changelog