Production deployment time audit

Date: August 27, 2026

Status: Current-state audit and improvement plan

Scope: Production releases for openagents.com

Executive summary

The deployment system has three valid lanes, but recent production work has used rolling replacement most often. Of 47 live targets with deployment receipts since August 20, 2026, 26 used rolling replacement, 4 used direct load, and 17 predate reliable lane typing or used the relup path. Across all 70 targets in the period, the classifier marked 39 for rolling replacement and 19 as direct-load candidates. The remaining 12 targets had no usable classification.

Rolling replacement is often correct because recent candidates combine assets, configuration, migrations, release files, or other structural changes with application code. It is not the correct default for every release. Two recent candidates that the classifier identified as direct-load candidates were rolled manually. That choice spent minutes replacing containers for changes that the direct-load transaction could have applied in less than one second.

The measured lane differences are large:

Measurement	Sample	Minimum	Median	Maximum
Direct load, promotion to terminal state	4	10.5 s	33.6 s	151.0 s
Direct-load fleet transaction	4	0.29 s	0.39 s	0.88 s
Rolling replacement, promotion to live	26	3.1 min	8.6 min	39.4 min
Forge candidate build for rolling targets	26	8.4 s	2.0 min	2.7 min
Release gate wall time, most recent 15 receipts	15	1 s	1.7 min	7.6 min
Production image build, successful recent builds	26	4.1 min	5.2 min	8.9 min

These measurements show three immediate priorities:

Select the lane before starting structural-release work. Use direct load for an allowlisted BEAM-only change, relup for a compatible application upgrade outside the allowlist, and rolling replacement only when the candidate changes the runtime structure.
Automate promotion with a narrowly scoped operator credential. Recent direct deployments spent 9 to 11 minutes waiting for a manual promotion even though promotion-to-live took 29 to 38 seconds.
Instrument and streamline rolling replacement. The current script rebuilds and pulls an image, restarts three nodes serially, and polls each node every 20 seconds. The durable receipt does not retain the time spent in each node phase, so it cannot identify whether image pulls, container startup, clustering, or polling dominates the roll.

Method and limitations

This audit uses the following evidence:

Production target and deployment receipts from August 20 through August 27, 2026.
Local exact-SHA release gate receipts under .git/openagents/release-gate/.
Successful Cloud Build records for the production image repository.
The deployment policy in docs/deploymodes.md and the executable behavior in ops/ci/gate.sh, ops/deploy/build-image-cloud.sh, ops/deploy/release-to-production.sh, and ops/deploy/fleet-startup.template.sh.

The measurements have four limitations:

promotion-to-live starts when an operator promotes a pushed target. It does not include the release gate, image build, or time waiting for promotion.
push-to-live includes operator delay. It measures what a developer feels, but it does not isolate system execution time.
Release gate stage durations are cached by input hash. A receipt's total wall time describes that invocation, while an individual cached stage can report the duration of an earlier execution. Do not add the stage durations to reconstruct the total.
Many manually settled rolling receipts record a deployment duration of zero or a rounded assumption. This audit therefore uses target lifecycle times for rolling replacement and does not report a receipt-level rollout median.

How the lanes should work

Use the narrowest lane that preserves the production invariants in docs/deploymodes.md.

Lane	Use it when	Do not use it when	Expected production work
Direct load	Every changed module is in the direct-load allowlist; the candidate changes only added or modified BEAM files; the application specification, release version, dependencies, toolchain, and structural paths are unchanged	The candidate deletes a module or changes assets, configuration, migrations, native code, dependencies, or release structure	Build the changed modules, prepare every node, deploy to a canary, and commit the same artifact set across the fleet without restarting containers
Relup	A pure Elixir application change is outside the direct allowlist, needs an OTP code-change callback, or changes supervised state compatibly; the forward and reverse release packages pass the topology preflight	The candidate changes assets, dependencies, migrations, native code, or has no safe upgrade path	Install the exact relup package one node at a time and retain the reverse package
Rolling replacement	The candidate changes `mix.exs`, `mix.lock`, runtime configuration, assets, migrations or other release `priv` content, `rel`, native code, the Dockerfile, dependencies, or the toolchain	The classifier proves the candidate is an allowlisted BEAM-only change	Build one immutable image and replace one healthy node at a time

Published documentation under priv/docs/** is compiled into the allowlisted documentation catalog and can use direct load when it causes no other structural change. Source documentation, comments, and other uncompiled files can produce a zero-module candidate and might require no production action.

The decision order is:

Use direct load when the classifier and preflight permit it.
Use relup when direct load is not permitted and the OTP upgrade is compatible.
Use rolling replacement for structural candidates or when either narrower lane refuses the candidate.

Never override a refusal to obtain a faster release. A refusal is evidence that the candidate does not satisfy that lane's safety contract.

Where the current time goes

Direct load

The fleet transaction is already fast. The most recent production direct load deployed one module across all three nodes in 381 ms. Its full promotion-to-live interval was 28.9 seconds, including the Forge build.

The variable cost is the candidate build. In the four typed direct-load receipts, the build took 10.0 to 149.4 seconds, with a median of 32.5 seconds. The operator queue can cost more than the entire deployment. Three recent direct releases took 9.0, 9.1, and 11.3 minutes from push to live, while their promotion-to-live intervals were 151.0, 38.3, and 28.9 seconds.

Direct-load optimization should therefore target promotion latency and Forge builder latency. Optimizing the subsecond fleet transaction will not improve the developer's wait materially.

Release gate

The most recent 15 gate receipts took 1 to 454 seconds, with a median of 103 seconds. A one-second gate means that every stage reused exact input evidence; it does not mean the full test suite ran in one second. The broadest stage is mix precommit: its cache input is the entire tree, so a new commit invalidates it. A recent uncached run took about 4.2 minutes and executed 5,129 checks.

The production runbook's 30-to-60-minute gate estimate no longer describes the recent measured range. Update that estimate only after the gate receipt reports enough uncached samples to publish percentiles by candidate type.

Production image build

Across 26 successful recent builds, Cloud Build took 246 to 536 seconds from submission to completion, with a median of 314 seconds. The median queue delay was 52 seconds, and the median build execution was 259 seconds.

The image path compiles the application and assets inside Docker, verifies the packaged revision, and pushes the immutable image. A structural release also causes the Forge builder to compile the candidate after promotion. Those two builds currently run serially in the operator procedure and repeat some work.

Three-node rolling replacement

After promotion, rolling targets took a median of 8.6 minutes to reach live. The Forge candidate build accounts for about 2.0 minutes of that median. The remaining interval includes operator coordination, three sequential node replacements, and settlement.

For every node, the startup path performs the following costly operations:

Stop and remove the application container.
Prune unused images.
Pull the new production image.
Restart the Cloud SQL proxy and wait for it.
Start the application container.
Remove, prune, pull, and restart the builder container.
Rejoin the distributed Erlang cluster in a background loop.

The release script checks every other node, updates instance metadata, starts the startup runner, and then checks health every 20 seconds. Across three nodes, fixed polling alone adds up to 60 seconds of detection delay and an expected delay of about 30 seconds. Because the receipt stores only ready for each node, it cannot distinguish this delay from image transfer or startup.

Why rolling replacement has become the default in practice

The observed behavior has four causes:

Candidates often bundle application code with assets, configuration, migrations, or release files. One structural path makes the entire candidate a rolling replacement.
Operators have not had a local, pre-promotion lane forecast that explains the classifier result before they spend time on a gate and image.
The runbook presents the image pipeline as the primary short path even though direct load does not need a production image or node restart.
Operators have used rolling replacement as a familiar fallback, including for at least two candidates that production classified as direct-load candidates.

The safety rules are not the problem. The workflow exposes the lane too late and makes the broadest lane easier to start than the narrow lanes.

Improvement plan

Priority 0: Improve iteration time

Add a preflight deployment plan command. Given the current live SHA and a candidate SHA, print the selected lane, every structural reason, changed modules, deletions, version requirements, and required artifacts. Use the same classifier code as production. Make this the first runbook step.
Automate explicit promotion. Add a CLI command that obtains a short-lived deployments:promote credential, promotes one exact SHA, and waits for the target receipt. Keep promotion separate from push. This change removes the largest measured direct-load delay without weakening the approval boundary.
Make the release command lane-aware. For a direct candidate, run the direct preflight and promotion flow without building a container image. For a relup candidate, build and verify the exact release package. Start the image path only for a structural candidate.
Keep independent changes in separate candidates. Do not attach an asset, configuration, migration, or dependency change to an otherwise direct-load application change when the changes can ship independently. The structural file forces the combined candidate into rolling replacement.
Report developer and system latency separately. Publish push-to-live, push-to-promotion, promotion-to-build-complete, deployment transaction, and promotion-to-live. This split makes operator delay visible without hiding the developer's actual wait.

Priority 1: Speed up full rolling replacement

Persist phase and node timings. Record gate start and finish, Cloud Build queue and execution, Forge build, authorization, and settlement. For every node, record drain, image pull, application start, health, cluster rejoin, and builder start. Replace zero-duration manual receipts with measured monotonic durations.
Pre-pull the immutable image. Pull and verify the target digest on all three nodes while the old containers still serve traffic. Drain and restart one node only after every node has the verified image. Retain the previous digest for rollback and do not prune it before the new node is healthy.
Stop restarting unchanged support containers. Keep the Cloud SQL proxy running when its image and arguments are unchanged. Restart the builder only when its image digest or configuration changed. These processes do not need to share the application container's release cadence.
Use adaptive health polling. Poll every 2 seconds during the first 30 seconds, then back off to 5 seconds with jitter while preserving the current 10-minute refusal timeout. This removes most fixed detection delay without weakening readiness checks.
Run independent builds concurrently. After the exact-SHA gate passes, start the immutable image build and Forge candidate build together for a forecast structural candidate. Bind both receipts to the same SHA and do not authorize replacement until both succeed. At current medians, serial builds cost about 7.2 minutes; overlap can approach the slower 5.2-minute build.
Use one rolling coordinator. Move the shell loop behind the existing rolling-replacement coordinator so authorization, other-node health, idempotent node skipping, timing, and settlement share one implementation. Keep the script as an operator entry point instead of a second state machine.

Priority 2: Reduce structural-release frequency

Review the direct-load allowlist with evidence. Add a namespace only after tests prove state compatibility, rollback, and fleet convergence. Do not widen the allowlist to make the classifier quiet.
Evaluate an immutable asset deployment lane. Fingerprinted CSS and JavaScript could move independently if the application serves an atomic asset manifest and retains the previous bundle through rollback. Until that design exists and has executable invariants, assets remain structural.
Reduce duplicated compilation. Let the image build consume a verified candidate artifact or let both builders share content-addressed compilation inputs. Preserve independent revision verification at the final image.
Split the builder lifecycle from the web release. Pin a builder digest separately so routine web rolls do not pull and restart an unchanged builder on every node.

Proposed targets

Use the following targets after phase timing lands and produces at least 20 valid samples per lane:

Metric	Initial target
Direct-load push to promotion	p50 under 15 s
Direct-load promotion to live	p50 under 45 s; p95 under 2 min
Direct-load fleet transaction	p95 under 2 s
Relup package ready to fleet live	p50 under 90 s
Rolling promotion to live	p50 under 6 min; p95 under 15 min
Production image submission to published digest	p50 under 4 min
Per-node image-ready to healthy	p50 under 45 s

Do not set an end-to-end rolling target until the receipts include the gate, image build, operator wait, and per-node phases. The current data can measure those components separately, but it cannot reconstruct every historical full release reliably.

Recommended runbook shape

Replace the current one-size-fits-all sequence with this operator flow:

Select an exact candidate SHA and run the local deployment-plan preflight.
Run mix precommit and the qualification required for the selected lane.
Push the candidate to the forge.
Promote the exact SHA with the scoped command.
Follow the lane chosen by the production classifier:
- Direct load: wait for the transactional receipt and verify production.
- Relup: install the verified package one node at a time, then verify and settle.
- Rolling replacement: build or resolve the immutable image, pre-pull it, replace one node at a time, then verify and settle.
Confirm the public revision, three-node convergence, and durable receipt.

This flow keeps push, promotion, and deployment separate. It makes the common BEAM-only path fast and preserves rolling replacement as the safe path for structural change.

Code

docs/2026-08-27-production-deployment-time-audit.md