Production deploy runbook

This is the single end-to-end procedure for deploying openagents.com to the production fleet. Follow it top to bottom. Every other operations document is reference material; this one is the checklist.

The pipeline is:

push candidate to forge main
  → release gate (exact SHA)
  → immutable image build and publish
  → promote on /admin/forge
  → migration job
  → rolling replacement (structural) or direct deploy (BEAM-only)
  → settle the target
  → verify production

Two invariants govern everything:

• One exact Git SHA. The gate receipt, the image, the promotion, and the rolling replacement all name the same full commit SHA. If main advances, you have a new candidate and you start over at the gate.
• One immutable image digest. Deployment identity is a sha256: digest, never a mutable tag.

The short path

Steps 5, 7, and 8 — promote, roll, and settle — are driven by one script:

ops/deploy/release-to-production.sh <full-sha>

It reads the gate receipt rather than rerunning the gate, resolves the image by digest, promotes and waits for the forge to write a complete build receipt, replaces one node at a time, and settles the target. It refuses rather than guesses: no passing gate receipt, an image whose embedded revision is not the Git SHA, or a node that returns on the wrong revision each stop it where it stands, because a half-rolled fleet that reports success is worse than one that stops. It also skips a node that already serves the SHA with the digest pinned, because restarting such a node serves 502 for the length of the restart for no gain.

If it refuses, do not work around the refusal. Read what it refused on.

The script does not run the release gate (step 2), build the image (step 3), or run the migration job (step 6). Do those first. The remaining sections describe what the script automates, and what to do when a step needs to be driven by hand.

0. One-time workstation prerequisites

The gate fails fast when any of these is missing. Set them up once per machine:

1. Tools on PATH: jq, mix, npm, docker, git version 2.38 or later (the stack merge and git-plane tests run git merge-tree --write-tree, which older git does not support), and terraform satisfying >= 1.11, < 2.0 (the staging infrastructure gate checks ops/staging/infra/versions.tf).

2. JavaScript dependencies: run npm ci --prefix assets. The relup stage runs mix assets.deploy, which needs assets/node_modules (for example posthog-js). A missing install fails the relup stage, not the javascript stage.

3. A local PostgreSQL server with the vector extension available, plus a disposable database the gate may trash:

   createdb openagents_release_smoke
   psql -d openagents_release_smoke -c 'CREATE EXTENSION IF NOT EXISTS vector;'
   

   The OPENAGENTS_RELEASE_SMOKE_DISPOSABLE=1 flag asserts the database is disposable; it does not create it. The version-chain and release-smoke stages boot real releases against this database and fail during startup if it does not exist.

4. Operator gcloud auth with compute and Secret Manager access in openagentsgemini, and Docker configured for Artifact Registry:

   gcloud auth login <operator>@openagents.com --no-browser
   gcloud auth configure-docker us-central1-docker.pkg.dev
   

1. Pick the candidate

1. Work from a clean worktree. Both the gate and the image build check git status --porcelain --untracked-files=all, so an untracked file stops them just as a modified one does. A long-lived checkout collects logs, PID files, and other agents' worktree directories, and none of it is yours to move or delete. Add those paths to .git/info/exclude instead: nothing is moved, modified, or committed, and only your own git status changes. Restore the file when the deploy is done. Before doing that, confirm the paths cannot reach the image — the Dockerfile copies VERSION, mix.exs, mix.lock, config/, priv, lib, rel, and assets, and nothing else.
2. The candidate is the exact SHA of forge main. Fetch and confirm git rev-parse HEAD equals git rev-parse openagents/main. The forge remote is openagents; origin is the GitHub mirror.
3. If you have local commits, push them first: git push openagents HEAD:main (never GitHub; the push guard refuses non-forge pushes).
4. Record the full SHA. It appears in every later step.

main advancing while you deploy does not invalidate a candidate you have already gated, built, and rolled. It means the commits that landed after it are not deployed. Say so in the report rather than implying the fleet runs the tip.

2. Run the release gate

OPENAGENTS_RELEASE_SMOKE_DISPOSABLE=1 \
OPENAGENTS_RELEASE_SMOKE_DATABASE_URL='ecto://USER:PASSWORD@127.0.0.1/openagents_release_smoke' \
ops/ci/gate.sh

The gate runs, in order: compile, production_compile, precommit, cluster, javascript, direct_transaction, relup, version_chain, interrupted_install, rolling_replacement, contracts, staging_infra, and release_smoke. It writes a receipt keyed to the exact SHA under .git/openagents/. A receipt from a different SHA — even a docs-only parent — is invalid.

Budget 30 to 60 minutes. If a stage fails, fix the cause, push the fix to forge main, and rerun the whole gate against the new SHA. Do not deploy a commit whose gate did not complete.

3. Build and publish the immutable image

On an Apple Silicon workstation, build on Cloud Build:

ops/deploy/build-image-cloud.sh <full-sha>

Cloud Build workers are native amd64, so both the build and the packaged revision check run for real. The script refuses before it spends a build if the worktree is not at the exact SHA, if the worktree is not clean, or if the tag already exists. That last refusal matters: tags in this repository are immutable, so a wrong image cannot be replaced and the SHA is unusable forever. The script builds the final target, asserts that the packaged OpenAgents.BuildInfo.revision() equals the SHA, and pushes to Artifact Registry itself. No separate docker push step is needed.

ops/deploy/build-image.sh does the same work locally, and on a native amd64 machine it is still the shorter path. Do not reach for it on Apple Silicon. It boots the linux/amd64 image to check the packaged revision, that boot runs under emulation, and the Erlang VM cannot start there. It dies at kernel start with failed_to_start_child,user,nouser, which is reproducible on the bare hexpm/elixir base image and unaffected by -noinput or TERM=dumb. A cached layer can hide this until the Docker cache is cleared.

Record the published sha256: digest — it is the deployment identity, not the local image ID:

gcloud artifacts docker images describe \
  us-central1-docker.pkg.dev/openagents-staging-20260820/openagents-staging/openagents:<full-sha> \
  --format='value(image_summary.digest)' --project openagentsgemini

4. Confirm runtime secrets and environment

The fleet startup script (instance metadata key startup-script on sarah-fleet-1/2/3) resolves secrets from Secret Manager at boot and passes them to the app container through the ENV_NAMES array. A new runtime environment variable needs three things:

1. A Secret Manager secret in openagentsgemini, with roles/secretmanager.secretAccessor granted to oa-mvp-automation@openagentsgemini.iam.gserviceaccount.com.
2. An export NAME="$(secret <secret-name>)" line in the startup script.
3. The name added to ENV_NAMES.

Edit one copy of the script, diff it against each instance's live metadata (all three must be identical), then apply with:

gcloud compute instances add-metadata sarah-fleet-N --zone <zone> \
  --project openagentsgemini \
  --metadata-from-file startup-script=<file>

Metadata changes take effect at the next instance reset, which the rolling replacement performs. Never print secret values.

Pending: GEMINI_API_KEY

The Gemini 3.7 Flash chat backend needs GEMINI_API_KEY in the fleet environment. Step 1 is already done: the secret is openagents-gemini-api-key in openagentsgemini. Steps 2 and 3 are not, and until they are, a production turn that names gemini-3.7-flash fails with missing_api_key rather than answering. The rest of the API is unaffected, and no other backend changes.

Add to the startup script:

export GEMINI_API_KEY="$(secret openagents-gemini-api-key)"

and add GEMINI_API_KEY to ENV_NAMES. Confirm the accessor grant first:

CLOUDSDK_CONFIG=<sa-config> gcloud secrets get-iam-policy openagents-gemini-api-key \
  --project openagentsgemini

5. Promote and classify

Promote the exact SHA through /admin/forge, or through the operator API described in Promote without a browser. Both paths call OpenAgents.Forge.Promotion, so they apply one policy and write one kind of receipt. The target moves promoted → building → built, and the classifier decides the deployment route:

OpenAgents.Forge.DeploymentLane.classify/2 chooses the lane, from three inputs and before any node is touched: the build manifest's structural classification, the hot-load allowlist, and the fleet's own relup topology verdict, read from every member by fleet_topology/1.

• Direct deploy: the diff touches only allowlisted BEAM modules. The forge hot loop handles it; watch the target go deploying → live.
• needs_rolling_replace: anything structural — config/*, migrations, dependencies, assets, ERTS, or release-private files. Continue below.
• Relup: an application transition between two concrete X.Y.Z versions, on a fleet whose topology supports it. Not reachable today, and the classification says which of the two conditions failed. See below.

Keep the classification receipt. On a needs_rolling_replace target the details map carries reasons and the topology verdict the lane was chosen against:

openagents api admin/forge/targets/$TARGET_ID

The verdict names counts of fleet members, never their names, and each application whose top process OTP release handling cannot inspect. Two reasons are worth reading directly:

• topology_incompatible:libring:HashRing.Supervisor — the fleet cannot take the relup lane. HashRing.App.start/2 returns a DynamicSupervisor, so :release_handler.install_release/1 cannot find that application's top supervisor. The candidate is classified onto rolling replacement rather than entering the relup lane and refusing on its first step. INVARIANTS.md, RELEASE-008 explains the refusal that remains underneath as the backstop, and RELEASE-009 the classification in front of it.
• relup_lane_unadmitted — the fleet could have taken the lane, but the hot loop does not admit it. RELEASE-005 keeps the relup workers disabled until isolated staging proves their provider and topology, and OpenAgents.Forge.RelupDeployment.run/2 still has no production caller.

The verdict is read from the fleet at classification time rather than published by the release gate, because it describes the running fleet and not the candidate bytes: the gate runs on a builder that is not the fleet, and a node can restart into a different application set in between. The gate keeps proving the RELEASE-008 refusal in its relup_topology stage.

Promote without a browser

Release tooling, scripts, and incident recovery use the operator API instead of a browser session. It requires the deployments:promote scope and live operator standing, checked on every request. forge:write cannot promote a fleet target, and neither can the tenant plane's deployments:write.

Bootstrap a credential through the device authorization flow, which asks for the scope by name:

curl --request POST \
  --header "Content-Type: application/json" \
  --data '{"scope": "deployments:promote"}' \
  https://openagents.com/api/v1/device/authorizations

Open /device in a signed-in operator browser, confirm the code, and read the requested scope shown on the approval page. Then exchange the device code:

curl --request POST \
  --header "Content-Type: application/json" \
  --data '{"device_code": "'"$DEVICE_CODE"'"}' \
  https://openagents.com/api/v1/device/authorizations/token

Approval is refused unless the approving account is a current operator. A privileged credential expires in at most 7 days, so release tooling re-authorizes rather than holding long-lived fleet authority.

Promote one exact, already-pushed SHA:

curl --request POST \
  --header "Authorization: Bearer $OPENAGENTS_PROMOTE_TOKEN" \
  --header "Content-Type: application/json" \
  --data '{
    "repo": "openagents.com",
    "sha": "'"$SHA"'",
    "environment": "production",
    "idempotency_key": "release-2026-08-23-0001",
    "expected_current_target_id": "'"$CURRENT_TARGET_ID"'"
  }' \
  https://openagents.com/api/v1/admin/forge/targets

The response is 202 Accepted with the immutable target ID, the exact SHA, the initial status, a status_url, and the request ID. It does not wait for the build or the deployment. Poll the status URL until terminal is true, or until status is needs_rolling_replace and you continue at step 7:

openagents api admin/forge/targets/$TARGET_ID
openagents api admin/forge/targets

Refusals carry the shared /api/v1 error envelope. Three codes matter during an incident:

• idempotency_conflict (409) — that key already names different bytes. Pick a new key rather than reusing one.
• precondition_failed (409) — someone promoted while you were preparing. Re-read the current target before deciding again.
• unknown_commit (422) — the SHA is not in the forge. Push it first; a push never promotes itself.

6. Migration job

Run exactly one migration job for the release; nodes must not race it. bin/migrate takes the release advisory lock. Apply any reviewed migration lineage bridge first, and require the lineage classification and integrity checks to pass before touching the fleet.

Fill the startup template with the digest you published and run its migrate-now subcommand on one node. This runs the release's own OpenAgents.Release.migrate() in a throwaway container against the new image, and leaves the node's running container alone:

sed "s|__IMAGE_DIGEST__|<digest>|g" ops/deploy/fleet-startup.template.sh > /tmp/startup.sh
gcloud compute scp /tmp/startup.sh sarah-fleet-1:/tmp/startup.sh \
  --zone us-central1-a --project openagentsgemini
gcloud compute ssh sarah-fleet-1 --zone us-central1-a --project openagentsgemini \
  --tunnel-through-iap --command="sudo bash /tmp/startup.sh migrate-now"

Invoke it as sudo bash /tmp/startup.sh, not sudo /tmp/startup.sh. The node mounts /tmp noexec, so executing the file directly fails with Permission denied and says nothing about why.

The fleet also sets OPENAGENTS_MIGRATE_ON_BOOT=true, so the first replaced node would migrate during boot anyway, serialized by the same advisory lock (RELEASE-001). Running the job first is still worth the extra step: it puts the schema change before the roll, where you can verify it while the fleet is still serving the previous release and nothing has been disturbed.

Verify what the migration wrote, rather than reading its exit status. A migration that reports success can still have written the wrong values, and a data-bearing migration is the one place where that is expensive and silent. Query the affected rows directly.

For a column added with a backfill, check three things: the row values the backfill was supposed to write, the column's is_nullable and column_default in information_schema.columns, and any constraint the migration created.

docker exec openagents /app/bin/openagents rpc 'OpenAgents.Repo.query!("SELECT count(*) FROM <table> WHERE <column> IS NULL") |> IO.inspect()'

docker exec openagents /app/bin/openagents rpc 'OpenAgents.Repo.query!("SELECT is_nullable, column_default FROM information_schema.columns WHERE table_name = $1 AND column_name = $2", ["<table>", "<column>"]) |> IO.inspect()'

docker exec openagents /app/bin/openagents rpc 'OpenAgents.Repo.query!("SELECT c.conname, pg_get_constraintdef(c.oid) FROM pg_constraint c JOIN pg_class t ON t.oid = c.conrelid WHERE t.relname = $1", ["<table>"]) |> IO.inspect()'

Write the query out rather than staging a script and evaluating it. A step that tells you to run a file it never shows you is a step you have to invent, and what you invent is what goes unreviewed.

Capture the same counts before the migration runs so the after-state has something to be compared against.

7. Rolling replacement

OpenAgents.Forge.RollingReplacement.run/2 replaces one node at a time. Its request names the Forge target id, the exact current and previous SHAs, the exact current and previous image digests, the expected node list, and the fleet size.

Before it touches the fleet, the coordinator publishes the authorized rolling identity onto that target. Boot convergence reads the published record, so a node that boots into the authorized image is admitted on its first convergence attempt and stays in the load balancer for the rest of the roll. Leave OPENAGENTS_FEATURE_BOOT_CONVERGENCE on, and do not restart OpenAgents.Forge.BootConverge.

For each node the coordinator removes readiness, drains to zero active work, verifies remaining capacity and quorum, resets the instance with the target digest (the GCP provider sets openagents-image, openagents-image-digest, and openagents-sha metadata), then waits for membership, readiness, boot convergence, database readiness, and the exact SHA and digest before moving on. It records each node's observed SHA and image digest against the published authority as that node rejoins.

Order: sarah-fleet-1 (us-central1-a), sarah-fleet-2 (us-central1-b), sarah-fleet-3 (us-central1-c). Two healthy nodes must exist before each replacement. If a node fails to rejoin, the coordinator rolls back to the previous digest; wait for full health and stop — do not continue the roll.

8. Settle the target

OpenAgents.Forge.Targets.finish_rolling_replacement(target_id, rolling_result)

Settlement demands the newest target in needs_rolling_replace, a complete verified build receipt, a rolling result whose SHA and image identities match the published authority and its exact expected node set, and an exact-identity observation from every one of those nodes. Success flips the target to live and inserts the immutable deployment receipt.

rolling_nodes_not_converged means a node is not on the authorized SHA and image digest. The target stays needs_rolling_replace, which is the recoverable state: read details.rolling_authority.observed on the target row to see exactly which node reported which identity, fix that node, and rerun run/2 with the same request. Republishing the same identity resumes the roll and keeps every recorded observation.

9. Verify production

Do not report success without all of the following:

• /status and /api/status return the exact SHA as revision. Neither renders the image digest, so read the digest from the settled target's details.rolling_authority, which records the authorized digest and the identity each node was observed at. In /api/status, every entry of nodes reporting boot.reason: "image_matches_live" is the node-level statement that it runs the authorized digest, because boot convergence admits against the published identity rather than the SHA alone.

• All three nodes report the same revision; beam=3 and quorum holds. raft is 0 while OPENAGENTS_FEATURE_RA is false, which is the current fleet setting — do not read raft=0 as a failed roll.

• All three backends of sarah-backend are healthy.

• The migration appears in schema_migrations, and its data effects are verified per step 6 rather than assumed from its exit status.

• Login, a typed chat turn, and a durable reload work.

• Issues, git clone/fetch, and a read-only computer job work.

• New configuration is live without exposing secret values.

• Any static asset the release changed is verified from the served bytes, not from the repository. Fetch it and compare its hash against the file at the deployed SHA:

  curl -fsS https://openagents.com/install.sh | shasum -a 256
  shasum -a 256 priv/static/install.sh
  

Load balancer health check

The global backend service sarah-backend decides which nodes receive traffic. The health check it references determines whether a broken node leaves rotation:

• A TCP check (sarah-hc-tcp) only verifies that port 8080 accepts connections. A node that answers every request with 500 still passes, so it keeps serving a third of traffic while broken. Issue #105 records an incident where this happened.
• The HTTP check sarah-hc-healthz probes GET /healthz on port 8080 and requires 200. The application returns 503 from /healthz when the database is unreachable, boot has not converged, the deployment participant is not ready, admission is not ready, or the forge cache is not ready, so an unhealthy node drops out of rotation.

Requirements before pointing the backend at the HTTP check:

1. The running image must exclude /health and /healthz from force_ssl in config/prod.exs. GCP HTTP probes are plain HTTP; Plug.SSL answers them with a 301 redirect, which counts as a failed probe. If every node redirects, the backend loses all members at once and the site serves 502. This happened on 2026-08-23.

2. Verify the exact probe shape on every node first. curl http://localhost:8080/healthz is not sufficient because Plug.SSL excludes the localhost host. Use a non-localhost host header:

   curl -s -o /dev/null -w '%{http_code}' \
     -H 'Host: <node-external-ip>' http://localhost:8080/healthz
   

   Require 200 on all three nodes before swapping.

3. Swap, then watch gcloud compute backend-services get-health sarah-backend --global. If any node goes UNHEALTHY unexpectedly, revert immediately:

   gcloud compute backend-services update sarah-backend --global \
     --health-checks sarah-hc-tcp --global-health-checks
   

The backend also reuses idle keepalive connections to nodes for up to 610 seconds. The server's keepalive timeout must exceed that window — config/runtime.exs sets thousand_island_options: [read_timeout: 660_000] on the production endpoint. With a shorter timeout the server closes idle connections the load balancer is about to reuse, and clients receive intermittent 502 responses logged as backend_connection_closed_before_data_sent_to_client. LiveView long-poll clients hit this most often because they issue many sequential requests. Request logging is enabled on sarah-backend (sample rate 1.0), so gcloud logging read 'httpRequest.status=502' shows the statusDetails for any recurrence.

With the HTTP check active, a structural rolling replacement depends on boot convergence admitting a node that runs the authorized rolling image. Because run/2 publishes that identity before the first replacement, nodes replaced before settlement report image_matches_rolling_target and stay in rotation; settlement through finish_rolling_replacement/2 flips the target live and the worker's own next convergence attempt reports image_matches_live. No flag change and no restart of OpenAgents.Forge.BootConverge is part of this path.

Admission is bound to the published identity, not to the SHA alone: a node whose image digest is not the authorized one, or a needs_rolling_replace target with no published authority, admits nothing and the node correctly returns 503. If every node returns 503 at the start of a roll, check that run/2 actually published the authority — read details.rolling_authority on the target row — rather than disabling boot convergence.

Images older than commit a79bfce degrade with artifact_not_direct instead and return 503 for the whole roll, which would empty the backend. The 2026-08-22 rollout of 45f6fff3222c432f248ca831a6fbc882c0fc6206 ran on such an image and needed a temporary boot-convergence disable plus a manual restart of OpenAgents.Forge.BootConverge. That workaround is retired: do not use it.

Known failure modes

Symptom	Cause	Fix
cluster stage: report["consistent"] == true fails	A local_report dependency crashes on peers where the app is loaded but not started	Health-report callees must fail closed (catch :exit), never raise
relup stage: esbuild cannot resolve a package	assets/node_modules missing	npm ci --prefix assets
version_chain / release_smoke: invalid_catalog_name	Disposable database does not exist	createdb + CREATE EXTENSION vector
staging_infra: terraform missing or version unsupported	No terraform, or version outside >= 1.11, < 2.0	Install a supported terraform
Push rejected (non-fast-forward)	Forge main advanced	Fetch, rebase your commit, push, restart the gate on the new SHA
gate.sh --verify fails in build-image.sh	Receipt is for a different SHA	Rerun the gate on the exact candidate
App boots without a new variable	Name missing from ENV_NAMES in the startup script	Add the export and the ENV_NAMES entry, re-apply metadata
Target build fails with invalid_module_name	A compiled module falls outside the artifact allowlist in OpenAgents.Forge.BuildArtifact (OpenAgents.*, OpenAgentsWeb.*, allowlisted protocol implementations, Mix.Tasks.Openagents.*)	Rename the module into an allowlisted namespace, or extend the pattern for a new generated-implementation family. test/openagents/forge/build_artifact_namespace_test.exs catches this in precommit
Target build fails with invalid_module_name even though every source module is allowlisted	The build cache at $OPENAGENTS_FORGE_BUILD_DIR/cache/_build/prod persists across builds, and a stale BEAM for a renamed or deleted module is still in lib/openagents/ebin	Since the application-resource packaging fix, BuildWorker.read_candidate_beams/1 packages only modules listed in the generated openagents.app, so stale BEAMs are ignored. On an older build, delete the stale .beam from the cache ebin on the node that ran the build and re-promote
Intermittent git-over-HTTP 500 on push and fetch	One fleet node has a structurally invalid bare-repository cache (for example HEAD present but refs/ missing), and Repos.ensure_repo_at! crashed on it. The load balancer alternates between healthy nodes and the broken one	Since the quarantine fix in OpenAgents.Forge.Repos, the node moves the invalid cache to <repo>.git.corrupt-<n> and re-materializes from the WAL on the next read. On an older build, find the node whose log shows fatal: not a git repository with a MatchError from Repos.set_default_branch!/2, move the cache directory aside inside the container, and let WAL replay rebuild it (docs/operations/forge-cache-recovery.md)