transaction protocol — `live` Forge target and deployment receipt recorded,

   boot convergence satisfied, process uptimes unbroken. The operator assembled

   and applied the artifact manually; no receipted automated deploy exists yet.

2. **"Nothing needs re-implementing" holds for the hot-load loop only.** The

   general relup path was, at measurement time, unconnected: `BuildArtifact`

   classifies only `direct_candidate` and `needs_rolling_replace`, and

   `RelupDeployment` accepted only the fixed `0.1.0 → 0.2.0` proof transition.

   Later the same day the relup **mechanism** was generalized — coordinator

   admission, appup generation from the two builds' compiled modules,

   packaging, and install proofs; see

   [`docs/operations/forge-hot-loop.md`](operations/forge-hot-loop.md). The

   **lane** is still not connected: no classifier emits a relup class,

   `RelupDeployment.run/2` has no caller outside tests, the release gate runs

   neither `ops/forge/package-relup.sh` nor

   `ops/relup-proof/install-proof.sh`, and no receipt binds a package to the

   revision it was built from. Those are engineering gaps, not operator ones.

   Production approval for the lane also remains an open operator decision.

2. **The general relup lane now carries production traffic.** The mechanism

   now admits arbitrary semantic version pairs, generates appup instructions

   from exact compiled-module diffs, and emits digest-addressed packages.

   `OpenAgents.Forge.RelupPackage` binds the package to the running source

   revision, target revision, operating system and architecture, release

   versions, state schemas, and target artifact digest before it calls

   `RelupDeployment`. On 2026-08-21, production upgraded three nodes from

   `0.2.0@81e4c25` to `0.2.1@9763bf7` in 48.838 seconds without restarting the

   BEAM. The classifier still does not select relup automatically; an operator

   builds and submits the package when direct loading refuses a compatible

   application release. See

   [`docs/operations/forge-hot-loop.md`](operations/forge-hot-loop.md).

---

Date: 2026-08-21

Status: Enablement runbook for the automated direct-load deploy loop

Status: Relup and mirroring are active in production. The automated direct-load

loop still requires fleet enablement.

This is the operator procedure for the fast deployment lane: push to the owned

forge, promote, and watch a code-only change go live across the fleet in

seconds without an image build. It also records why the loop reads as

disconnected today (`forge.loop.last_ms: null`, mirror `off`) and exactly what

flips it on.

seconds without an image build. It also describes the production relup path,

the independent GitHub mirror repair worker, and the remaining direct-load

enablement work.

## Verified state of the lanes

| --- | --- | --- |

| Direct BEAM transaction on production | Works | `fa4b792` loaded across three nodes via the transaction protocol; `live` target and deployment receipt recorded; uptimes unbroken |

| Automated push → promote → build → hot-load loop | Not yet operating | No receipted automated deploy exists, so `/api/status` reports `loop.last_ms: null` and `push_to_live_ms: null` |

| General relup lane | Manual tools only | `RelupDeployment` admits any distinct `X.Y.Z` pair, but the classifier emits only `direct_candidate` or `needs_rolling_replace`, `RelupDeployment.run/2` has no caller outside tests, and the release gate runs neither packaging script. Relups are not production-approved (`release-deployment-fallbacks.md`) |

| General relup lane | Active in production | `RelupPackage` binds source and target revisions, release versions, state schemas, target system, and artifact digests before `RelupDeployment` upgrades one node at a time. Production upgraded `0.2.0@81e4c25` to `0.2.1@9763bf7` in 48.838 seconds without restarting the BEAM. |

| Forge-to-GitHub mirror | Active in production | The production mirror uses a write-enabled deploy key. `MirrorWatch` runs independently of the deploy lane, repairs drift every five minutes, and reports freshness. Forge and GitHub exposed 10 identical refs after the production drill. |

| Rolling image replacement | Works, default for structural changes | Current release tooling path |

Two consequences worth stating plainly:

- Code-only changes do **not** require an image roll once the loop is enabled;

  the whole web layer is allowlisted.

- The current `main` (PostHog analytics, `73ff250`) touches

  `config/config.exs` and `config/runtime.exs`, which the classifier must

  refuse as structural. That change rides rolling replacement correctly.

- Changes to `config/config.exs` or `config/runtime.exs` remain structural.

  The classifier must refuse them for direct loading and route them to a full

  release path.

## Why the metrics read null

  application produced a receipt without timing, so the projection has no

  sample yet. The first automated loop deploy populates both.

- Mirror state `off` means no repository has a configured mirror URL

  (`OPENAGENTS_FORGE_MIRROR_URLS` defaults to empty). Mirroring feeds GitHub;

  it plays no part in the deploy loop.

  (`OPENAGENTS_FORGE_MIRROR_URLS_JSON` defaults to an empty map). Mirroring

  feeds GitHub; it plays no part in the deploy loop.

## The chain, as wired

   `forge_build_dir`, and `forge_artifact_dir` on durable state disks. Runtime

   configuration validates these when the deploy lane is enabled.

4. **Enable the deploy lane flag** (`OPENAGENTS_FEATURE_FORGE_DEPLOY=true`).

   This starts `Builder`, `HotLoader`, `Janitor`, and `MirrorWatch` under

   `OpenAgents.Forge.Supervisor`. Boot convergence is already proven on this

   fleet.

   This starts `Builder`, `HotLoader`, and `Janitor` under

   `OpenAgents.Forge.Supervisor`. `MirrorWatch` runs under the same supervisor

   regardless of this flag. Boot convergence is already proven on this fleet.

5. **Allowlist**: no change needed. Baked configuration already admits the

   whole `OpenAgentsWeb.` layer plus `OpenAgents.Changelog`,

   `OpenAgents.Forge.Browse`, `OpenAgents.BuildInfo`, and the scratch prefix,

A `reverted` outcome still warrants checking fleet convergence even though

this design captures each node's prior object code for exact rollback.

## The relup lane generalized, but nothing drives it

## Use the production relup lane

As of 2026-08-21 (later), the mechanism handles arbitrary version pairs:

The mechanism handles arbitrary version pairs:

1. **Coordinator admission is general.** `RelupDeployment` admits any distinct

   `X.Y.Z` pair whose state versions stay within `[1, 2]` and never regress —

   state schema. `mix openagents.relup` then checks the generated relup against

   the same diff, so packaging fails rather than shipping a relup that would

   install part of a revision.

3. **Packaging and install proofs exist as tools.**

3. **Packaging and install proofs produce deployable artifacts.**

   `ops/forge/package-relup.sh --from-version A --to-version B [--from-rev]

   [--to-rev]` builds both releases in isolated worktrees, generates the

   two-way relup, embeds it, and emits digest-addressed tarballs plus a

   refuses to run unless `OPENAGENTS_RELUP_PROOF_DISPOSABLE=1`, the URL host is

   loopback, and the database name contains `proof`, `smoke`, or `test`.

What is not true yet, and is engineering work rather than operator work:

- **No code calls the lane.** `RelupDeployment.run/2` has no caller outside

  tests. Nothing classifies a candidate as a relup, builds the request, or

  triggers a fleet relup; an operator drives the two scripts by hand.

- **The release gate does not run either script.** `ops/ci/gate.sh` runs

  `ops/relup-proof/run.sh`, `version-chain.sh`, and `kill-during-install.sh`,

  which exercise the pinned `0.1.0 → 0.2.0` pair only. Neither

  `package-relup.sh` nor `install-proof.sh` is a gate stage, so no receipt

  binds a general pair to a candidate SHA.

- **Nothing binds a package to its gate receipt.** `package.json` records the

  revisions it was built from, but `RelupDeployment` verifies the receipt for

  `request.sha` without checking that the artifact came from that revision.

Operator work still required before the lane carries production traffic:

- Production approval recorded against

  [`docs/operations/release-deployment-fallbacks.md`](release-deployment-fallbacks.md),

  which remains the authority that relups are not production-approved.

- A staging rehearsal of `package-relup.sh` followed by `install-proof.sh`

  against a disposable database, before any fleet use.

`OpenAgents.Forge.RelupPackage` closes the package-to-deployment boundary. It

rejects a package unless its source revision matches the running revision, its

target system matches the node, and its target tar matches the manifest

digest. It then constructs the bounded fleet request and calls

`RelupDeployment.run/2`. The coordinator verifies the target gate receipt,

rechecks exact membership between nodes, verifies the installed target

revision, and keeps the reverse release when it makes the target permanent.

Use patch versions for routine releases, such as `0.2.0` to `0.2.1`. Change

the minor version only when the release introduces a deliberate compatibility

or feature boundary. Never reuse a release version for different bytes;

`RelupNode` records the artifact digest for every unpacked version and rejects

a conflicting reuse.

The production drill on 2026-08-21 upgraded three nodes from

`0.2.0@81e4c25` to `0.2.1@9763bf7` in 48.838 seconds. Every node returned

`permanent`, retained `0.2.0` as `old`, reported the exact target revision,

and preserved its uptime. The operator used the documented emergency gate

override because the release was an explicitly authorized recovery and

enablement operation. Use a normal exact-SHA gate receipt for routine relups.

Note on scope: hot-load diffs and relups remain different artifact classes.

A BEAM-diff artifact cannot drive `release_handler`; only a full release

package can. That is why the build lane's classification stays two-class and

the relup lane consumes its own packages.

## Operate the GitHub mirror

Set `OPENAGENTS_FORGE_MIRROR_URLS_JSON` to a repository-to-URL JSON object.

Use a credential-free SSH URL such as

`ssh://github.com/OpenAgentsInc/openagents.com.git`; provide authentication

through a node-side SSH key and `GIT_SSH_COMMAND`. Never put a credential in

the URL or instance metadata.

`Pushes.mirror_storage_key/1` resolves the logical repository name to its

canonical storage UUID before reading refs. This distinction matters for

migrated repositories whose display name and storage key differ.

`MirrorWatch` runs even when `OPENAGENTS_FEATURE_FORGE_DEPLOY=false`, compares

the canonical `main` ref every five minutes, retries a full mirror push on

drift, and publishes `current` or `lagging` status.

After a rollout or storage repair, force convergence before validating the

mirror:

```elixir

storage_key = OpenAgents.Forge.Pushes.mirror_storage_key("openagents.com")

:ok = OpenAgents.Forge.Sync.ensure_cluster_fresh(storage_key)

:ok = OpenAgents.Forge.Pushes.mirror_now("openagents.com")

```

Compare complete, sorted `git ls-remote` output from Forge and GitHub. Do not

accept a matching `main` branch as proof if another branch or tag differs.

## References

- [Hot deploy gap audit](../2026-08-21-hot-deploy-gap-audit.md)

Date: 2026-08-20

Status: Implemented. The isolated staging fleet uses the coordinated rolling

lane, and production uses the same one-node-at-a-time health boundary.

Status: Implemented. Production uses the relup lane for compatible application

releases and the coordinated rolling lane for structural changes.

## Purpose

Use this runbook when a candidate cannot use the direct BEAM transaction. The

classifier must choose one strategy for the complete candidate:

- Use a relup only for the supported `0.1.0` to `0.2.0` application transition

  and its tested reverse transition.

- Use a relup for a packaged `X.Y.Z` transition when the generated appup,

  target system, exact revisions, artifact digests, state schemas, and reverse

  path pass validation.

- Use rolling replacement for ERTS, OTP, dependency, native-code, asset,

  configuration, migration, module-deletion, or otherwise unclassified

  changes.

The relup lane is not production-approved. Use rolling replacement in

production only with explicit operator authority, an immutable image digest,

two remaining healthy nodes, and exact revision checks after each replacement.

Production relups and rolling replacements require explicit operator authority.

For a rolling replacement, also require an immutable image digest, two

remaining healthy nodes, and exact revision checks after each replacement.

## Local release gate

## Relup lane

### Supported transition

### Supported transitions

The repository owns one explicit two-way transition:

`RelupDeployment` accepts any distinct semantic `X.Y.Z` pair when the state

schema remains in the supported range and does not regress. The generated

appup remains the authoritative compatibility check. If it cannot describe

the complete module change, package generation or

`release_handler.check_install_release/1` refuses the transition.

| Direction | Application version | State schema |

| --- | --- | --- |

| Upgrade | `0.1.0` to `0.2.0` | 1 to 2 |

| Downgrade | `0.2.0` to `0.1.0` | 2 to 1 |

| Upgrade | `X.Y.Z` to a distinct semantic version | 1 or 2 to the same or a higher supported schema |

| Downgrade | The embedded reverse transition | The target schema back to the packaged source schema |

Use patch versions for routine releases. Reserve a minor-version change for a

deliberate compatibility or feature boundary. Never publish different bytes

under an existing version.

`OpenAgents.ReleaseState` is a supervised, long-lived process with a versioned

`OpenAgents.ReleaseState.State` struct. `code_change/3` preserves the PID and

reverse, and re-upgrade state paths have focused tests and a real packaged-node

proof.

Build the release pair and relup:

Build a digest-addressed release pair and relup:

```sh

ops/relup-proof/run.sh

ops/forge/package-relup.sh \

  --from-version 0.2.0 \

  --to-version 0.2.1 \

  --from-rev SOURCE_SHA \

  --to-rev TARGET_SHA \

  --out-dir /path/to/package

```

The script builds explicit release versions, generates both directions with

`mix openagents.relup`, embeds the generated `relup` in the candidate tar, and

caches checksummed proof artifacts under `.git/openagents/relup-proof/<sha>/`.

It never deletes a shared release directory.

The script builds each exact revision in an isolated worktree, generates both

directions with `mix openagents.relup`, embeds the generated `relup` in the

candidate tar, and writes a `package.json` manifest with source and target

revisions, versions, state schemas, target system, and SHA-256 digests. It

never deletes a shared release directory.

`OpenAgents.Forge.RelupPackage.deploy/2` validates that manifest against the

running revision and system architecture before it reads the bounded target

artifact and calls `RelupDeployment`. A stale package cannot upgrade a fleet

that another deployment has already replaced.

Run the live state and interruption drills against a disposable database:

`OpenAgents.Forge.RelupDeployment` performs this sequence:

1. Verify the complete local gate receipt for the candidate SHA.

2. Snapshot the exact sorted member set and configured fleet size.

3. Stage one node's release tar in a digest-addressed cache.

4. Verify the cached and consumable tar digests.

5. Restore the consumable tar from cache, then unpack it.

6. Generate version-specific runtime configuration and run

1. Validate the package source revision, target system, and target artifact

   digest.

2. Verify the complete local gate receipt for the candidate SHA.

3. Snapshot the exact sorted member set and configured fleet size.

4. Stage one node's release tar in a digest-addressed cache.

5. Verify the cached and consumable tar digests.

6. Restore the consumable tar from cache, then unpack it.

7. Generate version-specific runtime configuration and run

   `release_handler.check_install_release/1`.

7. Install the candidate without changing the permanent release.

8. Verify current release status, application readiness, and the expected

   migrated state schema.

9. Make the release permanent, verify permanence, and recheck exact fleet

8. Install the candidate without changing the permanent release.

9. Verify current release status, exact target revision, application readiness,

   and the expected migrated state schema.

10. Make the release permanent, verify permanence, and recheck exact fleet

   membership.

10. Start the next node only after the previous node passes every check.

11. Start the next node only after the previous node passes every check.

The 2026-08-21 production drill upgraded three nodes from

`0.2.0@81e4c25` to `0.2.1@9763bf7` in 48.838 seconds. All nodes returned

`permanent`, retained `0.2.0` as the reverse release, and preserved their

uptimes.

If post-install health or permanence verification fails, the coordinator

installs the reverse relup, verifies the prior release and state schema, restores