#!/bin/sh

set -eu

repo_root=$(git rev-parse --show-toplevel)

exec "$repo_root/ops/ci/gate.sh"

### RELEASE-004 — CI runs on owned infrastructure only, and gates every release

Status: Proposed

Status: Current

The target release gate permits no hosted CI: no GitHub Actions workflows, no

GitHub-hosted or third-party runners, and no repository automation, secrets, or

scheduling handed to external CI compute. All checks run on owned machines.

The full matrix must bind unit, browser, distributed cluster, coverage,

release, relup, version-chain, failure, and staging evidence to the exact

candidate SHA, and every deploy command must refuse a stale or absent receipt.

This complete deploy refusal is proposed, not implemented. The repository

currently has an exact-SHA baseline receipt, merged coverage, release smoke,

and relup proof primitives, but Gate 12 must compose them into the final owned

release gate and bind every deployment entry point to its receipt.

Current progress evidence: `ops/ci/baseline.sh`, `ops/ci/coverage.sh`,

`ops/ci/release-smoke.sh`, `ops/relup-proof/run.sh`,

`ops/relup-proof/version-chain.sh`, and

The full local matrix binds unit, browser, distributed cluster, direct

transaction, release, relup, version-chain, interrupted-install, rolling

replacement, and repository-contract evidence to the exact candidate SHA.

Relup and rolling coordinators refuse a stale or absent receipt before they

change a node. `.githooks/pre-push` invokes the same gate. A bounded, logged

emergency override exists only for operator-directed recovery.

Staging evidence remains a separate later gate. A local receipt does not claim

that the candidate passed staging or authorize a production release.

Evidence: `ops/ci/gate.sh`, `.githooks/pre-push`,

`OpenAgents.Forge.GateReceipt`, `OpenAgents.Forge.GateReceiptTest`,

`ops/relup-proof/run.sh`, `ops/relup-proof/version-chain.sh`, and

`ops/relup-proof/kill-during-install.sh`.

### RELEASE-005 — Every code change has a fail-closed deployment class

Status: Current

Direct BEAM candidates use an exact-fleet prepare, canary, apply, verify,

commit, and rollback transaction. The supported `0.1.0` to `0.2.0` application

transition uses a two-way relup, versioned process state, node-by-node health

checks, and reverse installation. Every structural or unclassified candidate

uses digest-addressed rolling replacement with readiness drain, remaining-

capacity and quorum checks, exact rejoin verification, and last-known-good

image recovery. A failed relup or replacement aborts before another node

changes.

All deployment workers remain disabled until isolated staging proves their

complete provider and topology. Current means that the local mechanism and its

refusal and recovery paths exist; it does not authorize staging or production.

Evidence: `OpenAgents.Forge.Deployment`, `OpenAgents.Forge.RelupDeployment`,

`OpenAgents.Forge.RelupNode`, `OpenAgents.ReleaseState`,

`OpenAgents.Forge.RollingReplacement`,

`test/openagents/forge/relup_deployment_test.exs`,

`test/openagents/forge/rolling_replacement_test.exs`, and

`docs/operations/release-deployment-fallbacks.md`.

### STATUS-001 — The status page publishes one bounded, content-free projection

Status: Current

| RELEASE-001 | `ops/ci/release-smoke.sh`, `test/openagents_web/controllers/health_controller_test.exs` |

| RELEASE-002 | `test/openagents/github_oauth/runtime_config_test.exs`, `ops/ci/reference-check.sh` |

| RELEASE-003 | `test/openagents_web/allowed_origins_test.exs`, `ops/ci/release-smoke.sh` |

| RELEASE-004 | Proposed; current primitives are listed in its entry above. |

| RELEASE-004 | `ops/ci/gate.sh`, `test/openagents/forge/gate_receipt_test.exs` |

| RELEASE-005 | `test/openagents/forge/relup_deployment_test.exs`, `test/openagents/forge/rolling_replacement_test.exs` |

| STATUS-001 | `test/openagents/network_status_test.exs`, `test/openagents_web/live/network_status_live_test.exs` |

| TRANSPARENCY-001 | `test/openagents/forge/visibility_test.exs`, `test/openagents/forge/browse_test.exs`, `test/openagents_web/live/code_live_test.exs` |

- Issues, comments, labels, milestones, projects, public status, changelog, and

  bounded source-browsing surfaces.

- Git HTTP, push receipts, promotion targets, build receipts, and local BEAM

  deployment primitives.

- An owned local test gate covering browser JavaScript, the Phoenix application,

  distributed cluster cases, merged coverage, and a disposable production

  release smoke test.

  deployment primitives, including transactional direct loading, two-way

  relups, and provider-neutral rolling replacement.

- An owned exact-SHA release gate covering browser JavaScript, the Phoenix

  application, distributed cluster cases, direct transactions, relup and

  interruption recovery, rolling replacement, repository contracts, and a

  disposable packaged-release smoke test.

"Implemented" means the code and local tests exist. It does not mean the

feature has passed staging, security review, failure injection, or a soak.

### Planned or blocked on hardening

- Repository-backed tenant isolation for every issue and project record.

- Complete route-authority, token-lifecycle, recovery, build-isolation, and

  transactional fleet-deployment gates.

- Separate web and distributed staging lanes, a full regression matrix,

  failure-injection drills, and a 48-hour soak.

- Any production rollout. Production remains explicitly out of scope until the

mix phx.server

```

Before committing, run the repository-owned gate:

Before committing, run the fast repository checks:

```sh

mix precommit

```

Distributed, merged-coverage, relup, and release-smoke checks live under

`ops/` and are composed by the exact-SHA baseline gate while hardening is in

progress. This repository deliberately has no hosted CI configuration.

Before pushing a release candidate, provision a disposable database and run

`ops/ci/gate.sh` as documented in

[the release deployment fallback runbook](docs/operations/release-deployment-fallbacks.md).

This repository deliberately has no hosted CI configuration.

## Contributing and source control

**Exit criteria:** Every supported change has a safe deployment class, a tested

rollback or recovery path, and an exact-SHA gate receipt.

### Gate 11 implementation status

Implemented locally on 2026-08-20:

- Replaced the relup placeholders with explicit `0.1.0` and `0.2.0` release

  builds, a generated forward and reverse `relup`, and a packaged candidate

  that carries the generated release instructions.

- Added `OpenAgents.ReleaseState` and its versioned state struct. Focused and

  packaged-node proofs preserve the process PID and observations through

  upgrade, downgrade, and re-upgrade.

- Added a deterministic install barrier and a live recovery proof that kills

  the emulator after the point of no return, verifies that the prior permanent

  release boots, re-stages the consumed immutable tar, and completes the retry.

- Added `OpenAgents.Forge.RelupNode` and

  `OpenAgents.Forge.RelupDeployment` for digest verification, immutable cache

  retention, stage restoration, preflight, one-node-at-a-time installation,

  health and state checks, permanence, reverse relup, and fleet-wide reversal

  of already-upgraded nodes after a later failure.

- Added `OpenAgents.Forge.RollingReplacement` and the staging-provider

  behavior. The coordinator removes one node from readiness, waits for drain,

  checks remaining capacity and quorum, replaces by image digest, and requires

  membership, boot, database, SHA, digest, and readiness convergence before it

  continues. Unsafe pre-replacement capacity restores readiness; failed rejoin

  restores the last-known-good image and stops the rollout.

- Added an exact-SHA receipt verifier, the owned `ops/ci/gate.sh`, a pre-push

  hook, and a receipt-gated immutable local image builder. The release gate

  composes warning-free compilation, precommit, distributed, browser, direct,

  relup, interruption, rolling, contract, and packaged startup checks.

- Documented operation and additive migration constraints in the

  [release deployment fallback runbook](operations/release-deployment-fallbacks.md).

- Retained the direct lane's existing end-to-end push, build, canary, fleet,

  and live proof with the exact `OpenAgents.BuildInfo` allowlist entry. The

  broader allowlist remains unchanged and disabled outside tests.

The implementation is ready for an exact-commit local gate. Keep every deploy

lane disabled until Gate 12 provides an isolated distributed staging fleet and

an infrastructure-specific rolling provider. No staging or production

environment changed during this gate.

## Gate 12: Build an isolated staging environment

The existing Cloud Run staging service can validate the web application, OAuth,

# Release deployment fallbacks

Date: 2026-08-20

Status: Implemented and locally proven; keep both lanes disabled until the Gate 12 isolated staging fleet exists

## Purpose

Use this runbook when a candidate cannot use the direct BEAM transaction. The

classifier must choose one strategy for the complete candidate:

- Use a relup only for the supported `0.1.0` to `0.2.0` application transition

  and its tested reverse transition.

- Use rolling replacement for ERTS, OTP, dependency, native-code, asset,

  configuration, migration, module-deletion, or otherwise unclassified

  changes.

Neither lane is production-approved. Gate 12 must connect the provider-neutral

coordinators to an isolated three-node staging fleet and retain staging-only

receipts before anyone enables a deployment worker.

## Local release gate

Install the repository hook once on each owned development or release machine:

```sh

git config core.hooksPath .githooks

```

Provision a disposable PostgreSQL database with pgvector already installed,

then run the complete exact-SHA gate:

```sh

OPENAGENTS_RELEASE_SMOKE_DISPOSABLE=1 \

OPENAGENTS_RELEASE_SMOKE_DATABASE_URL='ecto://USER:PASSWORD@HOST/DATABASE' \

ops/ci/gate.sh

```

The gate requires a clean worktree. It runs warning-free compilation,

`mix precommit`, distributed tests, browser tests, the direct transaction,

forward and reverse relup proofs, interrupted-install recovery, rolling

replacement tests, repository contracts, and packaged release startup. It

writes a content-free receipt to

`.git/openagents/release-gate-receipts/<full-sha>.json` only after every stage

passes without an automatic retry.

Both deployment coordinators call `OpenAgents.Forge.GateReceipt.verify/2`

before changing a node. A new commit has a new SHA and therefore invalidates

the prior receipt. The emergency override is an explicit function option with

a bounded reason; it emits a warning and exists only for operator-directed

recovery. Do not use it for an ordinary release.

## Relup lane

### Supported transition

The repository owns one explicit two-way transition:

| Direction | Application version | State schema |

| --- | --- | --- |

| Upgrade | `0.1.0` to `0.2.0` | 1 to 2 |

| Downgrade | `0.2.0` to `0.1.0` | 2 to 1 |

`OpenAgents.ReleaseState` is a supervised, long-lived process with a versioned

`OpenAgents.ReleaseState.State` struct. `code_change/3` preserves the PID and

bounded observations in both directions. `rel/openagents.appup.exs` names the

advanced update explicitly. Do not add a version transition until its forward,

reverse, and re-upgrade state paths have focused tests and a real packaged-node

proof.

Build the release pair and relup:

```sh

ops/relup-proof/run.sh

```

The script builds explicit release versions, generates both directions with

`mix openagents.relup`, embeds the generated `relup` in the candidate tar, and

caches checksummed proof artifacts under `.git/openagents/relup-proof/<sha>/`.

It never deletes a shared release directory.

Run the live state and interruption drills against a disposable database:

```sh

OPENAGENTS_RELUP_PROOF_DISPOSABLE=1 \

OPENAGENTS_RELUP_PROOF_DATABASE_URL='ecto://USER:PASSWORD@HOST/DATABASE' \

ops/relup-proof/version-chain.sh

OPENAGENTS_RELUP_PROOF_DISPOSABLE=1 \

OPENAGENTS_RELUP_PROOF_DATABASE_URL='ecto://USER:PASSWORD@HOST/DATABASE' \

ops/relup-proof/kill-during-install.sh

```

### Fleet sequence

`OpenAgents.Forge.RelupDeployment` performs this sequence:

1. Verify the complete local gate receipt for the candidate SHA.

2. Snapshot the exact sorted member set and configured fleet size.

3. Stage one node's release tar in a digest-addressed cache.

4. Verify the cached and consumable tar digests.

5. Restore the consumable tar from cache, then unpack it.

6. Generate version-specific runtime configuration and run

   `release_handler.check_install_release/1`.

7. Install the candidate without changing the permanent release.

8. Verify current release status, application readiness, and the expected

   migrated state schema.

9. Make the release permanent, verify permanence, and recheck exact fleet

   membership.

10. Start the next node only after the previous node passes every check.

If post-install health or permanence verification fails, the coordinator

installs the reverse relup, verifies the prior release and state schema, restores

the prior permanent release, and aborts before touching another node. If the

reverse path fails, keep the node out of readiness and move to operator-directed

rolling recovery.

`OpenAgents.Forge.RelupNode` retains the immutable tar in

`releases/.openagents-relup-cache/<sha256>.tar.gz`. It copies those exact bytes

back to the filename consumed by `unpack_release/1` before every attempt. This

restaging step is mandatory after an interrupted install.

## Rolling replacement lane

Build a local immutable image only after the exact release gate passes:

```sh

ops/deploy/build-image.sh openagents:<full-sha>

```

The script records the content-addressed `sha256:` image ID under

`.git/openagents/images/<full-sha>.json`. A mutable tag is a convenience label,

not deployment identity. Pass only the digest to the replacement provider.

`OpenAgents.Forge.RollingReplacement` owns the provider-neutral rollout. Gate

12 must implement `OpenAgents.Forge.RollingProvider` for the isolated staging

infrastructure. Keep machine inventory, credentials, addresses, and provider

resource names outside the repository.

For each node, the coordinator performs this sequence:

1. Verify the exact-SHA release receipt and exact initial member set.

2. Remove the node from external readiness.

3. Drain until no local work singleton remains.

4. Verify that the remaining nodes meet the configured ready-capacity floor

   and retain quorum.

5. Replace the node with the target image digest.

6. Wait for exact BEAM membership, readiness, boot convergence, database

   access, source SHA, and image digest.

7. Recheck exact fleet membership before selecting another node.

If a node does not rejoin, the coordinator asks the provider to restore the

last-known-good SHA and digest, waits for that node's full health, records the

recovery result, and aborts. It never replaces a second node while the first is

missing or unhealthy.

## Database compatibility

Keep schema changes additive while old and new releases overlap. During a

relup or rolling rollout, both versions must be able to read and write the same

schema. Do not combine a reversible application transition with a destructive

column removal, type contraction, trigger removal, or irreversible data

rewrite. Ship contract migrations only in a later release after rollback to

the old application version is no longer supported and staging has proven the

new rollback boundary.

## Staging admission

Do not run either fleet lane against the current web-only staging service. Gate

12 must first provide stable node identities, private distribution, a separate

staging database instance, durable release artifacts, node-local caches,

readiness removal, drain support, and a provider implementation. Then retain

evidence for:

- one complete allowlisted push-to-build-to-canary-to-fleet-to-live direct

  transaction;

- one relup upgrade, downgrade, re-upgrade, health-triggered reverse, and

  interrupted-install retry;

- one no-op digest-addressed rolling rollout;

- one failed rejoin that restores the prior digest and stops before the next

  node;

- additive migration compatibility while both application versions run.

Production remains out of scope until the later staging, rollback, and soak

gates pass and receive separate approval.

defmodule Mix.Tasks.Openagents.Relup do

  @moduledoc """

  Generates a two-way `relup` in an explicit output directory.

      mix openagents.relup --target /path/to/0.2.0/openagents \

        --from /path/to/0.1.0/openagents --outdir /tmp/proof

  """

  use Mix.Task

  @shortdoc "Generates an explicit two-way OpenAgents relup"

  @switches [target: :string, from: :string, outdir: :string]

  @impl true

  def run(arguments) do

    {options, remaining, invalid} = OptionParser.parse(arguments, strict: @switches)

    if remaining != [] or invalid != [] do

      Mix.raise("openagents.relup received unsupported arguments")

    end

    target = required_path!(options, :target)

    from = required_path!(options, :from)

    outdir = required_directory!(options, :outdir)

    paths =

      [target, from]

      |> Enum.flat_map(fn release ->

        release

        |> Path.join("../../../lib/*/ebin")

        |> Path.expand()

        |> Path.wildcard()

      end)

      |> Enum.uniq()

      |> Enum.map(&to_charlist/1)

    case :systools.make_relup(

           to_charlist(target),

           [to_charlist(from)],

           [to_charlist(from)],

           path: paths,

           outdir: to_charlist(outdir),

           silent: true,

           warnings_as_errors: true

         ) do

      {:ok, _relup, _module, []} ->

        Mix.shell().info("Generated #{Path.join(outdir, "relup")}")

      {:ok, _relup, module, warnings} ->

        Mix.raise(

          "relup generation produced warnings: #{format(module, :format_warning, warnings)}"

        )

      {:error, module, reason} ->

        Mix.raise("relup generation failed: #{format(module, :format_error, reason)}")

      other ->

        Mix.raise("relup generation returned #{inspect(other)}")

    end

  end

  defp required_path!(options, key) do

    path = options |> Keyword.fetch!(key) |> Path.expand()

    if File.regular?(path <> ".rel"),

      do: path,

      else: Mix.raise("--#{key} must name a release resource without the .rel suffix")

  end

  defp required_directory!(options, key) do

    path = options |> Keyword.fetch!(key) |> Path.expand()

    File.mkdir_p!(path)

    path

  end

  defp format(module, function, value) do

    module

    |> apply(function, [value])

    |> IO.iodata_to_binary()

  end

end

    children = [

      OpenAgentsWeb.Telemetry,

      OpenAgents.Repo,

      OpenAgents.ReleaseState,

      # Deployment identity and boot convergence must settle before cluster

      # discovery or the endpoint can make this node externally reachable.

      OpenAgents.Forge.DeploymentNode,

        ~U[2026-08-19 05:38:00Z],

        "d82d174",

        "feature",

        "openagents.com/status became the public network page: fleet, quorum, and the live rollout pipeline with push→live timing."

        "The status surface gained bounded fleet, quorum, rollout, and push-to-live projections for staging validation."

      ),

      e(

        ~U[2026-08-19 05:13:00Z],

        ~U[2026-08-19 04:01:00Z],

        "70c34ba",

        "forge",

        "The hot-load deploy lane shipped: push → promote → incremental build → canary → fleet-wide live, receipted; first loop 13.2 s push→live."

        "The local hot-load proof covers push → promote → incremental build → canary → fleet commit with immutable receipts."

      ),

      e(

        ~U[2026-08-19 03:44:00Z],

        "677c9ba",

        "forge",

        "Sarah serves her own git: smart-HTTP forge at openagents.com/git, pushes acked only after WAL persist to object storage."

        "The forge implements smart HTTP and acknowledges a push only after durable WAL persistence."

      ),

      e(

        ~U[2026-08-19 03:41:00Z],

        "eb93e53",

        "infra",

        "Releases are gated by owned chaos drills — node kill, Raft partition, SIGKILL-mid-upgrade — on our own machines, no hosted CI."

        "The owned local gate includes node, Raft, and interrupted-upgrade drills without hosted CI."

      ),

      e(

        ~U[2026-08-19 02:47:00Z],

        "c0a96f1",

        "infra",

        "Live in-place hot code upgrade (relup) proven on the fleet: 0.1.0 → 0.2.0 with state migrated and no restart."

        "The relup foundation can generate a 0.1.0 to 0.2.0 state migration for later isolated-fleet proof."

      ),

      # ── 2026-08-18 — the Immortal Sarah build day

        ~U[2026-08-19 00:43:00Z],

        "2071eaf",

        "infra",

        "Zero-downtime rolling node replacement: drain, replace, rejoin — the cluster never loses quorum during a deploy."

        "The rolling-replacement contract defines drain, replace, rejoin, and quorum checks for later isolated-fleet proof."

      ),

      e(

        ~U[2026-08-18 23:21:00Z],

        ~U[2026-08-18 21:12:00Z],

        "f6afd61",

        "infra",

        "Sarah became a 3-node clustered BEAM fleet on GCE — the single-instance Cloud Run era ended."

        "A local three-node BEAM proof established membership, quorum, and state-handoff contracts for isolated staging."

      ),

      e(

        ~U[2026-08-18 23:13:00Z],

  @erpc_timeout_ms 5_000

  @doc "The currently expected relup marker for live nodes."

  def relup_marker, do: "v2-relup-live"

  def relup_marker, do: "v2-relup-capable"

  @doc "All cluster members, including this node."

  @spec members() :: [node()]

defmodule OpenAgents.Forge.GateReceipt do

  @moduledoc """

  Verifies the content-free, exact-SHA receipt required by release deployments.

  Receipts live under `.git`, not in the source tree. Changing the checked-out

  commit therefore invalidates the receipt without creating a commit that can

  attest to itself.

  """

  require Logger

  @schema "openagents.release-gate.v1"

  @sha_pattern ~r/\A[0-9a-f]{40}\z/

  @maximum_bytes 65_536

  @required_stages ~w(

    compile

    precommit

    cluster

    javascript

    direct_transaction

    relup

    version_chain

    interrupted_install

    rolling_replacement

    contracts

    release_smoke

  )

  @doc "Verify the local release-gate receipt for `sha`."

  def verify(sha, opts \\ [])

  def verify(sha, opts) when is_binary(sha) do

    if Regex.match?(@sha_pattern, sha) do

      case Keyword.get(opts, :emergency_override) do

        reason when is_binary(reason) and byte_size(reason) in 8..512 ->

          Logger.warning("release_gate_emergency_override sha=#{sha} reason=#{reason}")

          {:ok, %{schema: "openagents.release-gate.emergency.v1", git_sha: sha}}

        _no_override ->

          verify_receipt(sha, opts)

      end

    else

      {:error, :invalid_git_sha}

    end

  end

  def verify(_sha, _opts), do: {:error, :invalid_git_sha}

  @doc "Return the default receipt path for an exact Git SHA."

  def path(sha, opts \\ []) do

    root = Keyword.get_lazy(opts, :repo_root, &repo_root!/0)

    Path.join([root, ".git", "openagents", "release-gate-receipts", "#{sha}.json"])

  end

  defp verify_receipt(sha, opts) do

    with true <- Regex.match?(@sha_pattern, sha) or {:error, :invalid_git_sha},

         {:ok, stat} <- File.stat(path(sha, opts)),

         true <-

           (stat.type == :regular and stat.size <= @maximum_bytes) or

             {:error, :invalid_receipt_file},

         {:ok, bytes} <- File.read(path(sha, opts)),

         {:ok, receipt} <- Jason.decode(bytes),

         :ok <- validate(receipt, sha) do

      {:ok, receipt}

    else

      {:error, :enoent} -> {:error, :missing_gate_receipt}

      {:error, %Jason.DecodeError{}} -> {:error, :invalid_gate_receipt_json}

      {:error, reason} -> {:error, reason}

      false -> {:error, :invalid_gate_receipt}

    end

  end

  defp validate(receipt, sha) when is_map(receipt) do

    stages = Map.get(receipt, "stages", %{})

    cond do

      receipt["schema"] != @schema -> {:error, :wrong_gate_receipt_schema}

      receipt["git_sha"] != sha -> {:error, :stale_gate_receipt}

      receipt["status"] != "passed" -> {:error, :gate_not_passed}

      not complete_stages?(stages) -> {:error, :incomplete_gate_receipt}

      true -> :ok

    end

  end

  defp validate(_receipt, _sha), do: {:error, :invalid_gate_receipt}

  defp complete_stages?(stages) when is_map(stages) do

    Enum.all?(@required_stages, fn stage -> get_in(stages, [stage, "status"]) == "passed" end)

  end

  defp complete_stages?(_stages), do: false

  defp repo_root! do

    case System.cmd("git", ["rev-parse", "--show-toplevel"], stderr_to_stdout: true) do

      {root, 0} -> String.trim(root)

      {_output, _status} -> raise "release gate verification requires a Git worktree"

    end

  end

end

defmodule OpenAgents.Forge.RelupDeployment do

  @moduledoc """

  Upgrades an exact fleet one node at a time through a two-way OTP relup.

  The coordinator rechecks membership between nodes, makes a candidate

  permanent only after health and state verification, and installs the reverse

  relup when post-install health fails.

  """

  alias OpenAgents.Forge.GateReceipt

  alias OpenAgents.Forge.RelupNode

  @sha_pattern ~r/\A[0-9a-f]{40}\z/

  @digest_pattern ~r/\A[0-9a-f]{64}\z/

  @default_timeout_ms 120_000

  @doc "Deploy one two-way relup across the exact expected fleet."

  def run(request, opts \\ []) do

    with :ok <- validate_request(request),

         {:ok, _receipt} <- gate_verify(request.sha, opts),

         {:ok, nodes} <- snapshot_members(request, opts) do

      deploy_nodes(nodes, request, opts, %{}, [])

    end

  end

  defp deploy_nodes([], request, _opts, results, _completed) do

    {:ok, public_result(request, "live", results, nil)}

  end

  defp deploy_nodes([node | remaining], request, opts, results, completed) do

    with :ok <- stable_membership(request, opts) do

      case deploy_node(node, request, opts) do

        {:ok, result} ->

          case stable_membership(request, opts) do

            :ok ->

              deploy_nodes(

                remaining,

                request,

                opts,

                Map.put(results, to_string(node), result),

                [node | completed]

              )

            {:error, reason} ->

              fail_deployment(node, reason, [node | completed], request, opts, results)

          end

        {:error, reason} ->

          fail_deployment(node, reason, completed, request, opts, results)

      end

    else

      {:error, reason} ->

        fail_deployment(node, reason, completed, request, opts, results)

    end

  end

  defp fail_deployment(node, reason, rollback_nodes, request, opts, results) do

    {rollback_results, rollback_status} = rollback_completed(rollback_nodes, request, opts)

    failure =

      results

      |> Map.merge(rollback_results)

      |> Map.put(to_string(node), safe_code(reason))

    error_code =

      if rollback_status == :ok,

        do: safe_code(reason),

        else: "fleet_reverse_failed:" <> safe_code(reason)

    {:error, public_result(request, "failed", failure, error_code)}

  end

  defp rollback_completed(nodes, request, opts) do

    Enum.reduce(nodes, {%{}, :ok}, fn node, {results, status} ->

      case call(node, :reverse, [request], opts) do

        {:ok, _result} ->

          {Map.put(results, to_string(node), "reversed"), status}

        {:error, reason} ->

          {Map.put(results, to_string(node), "reverse_failed:" <> safe_code(reason)), :error}

      end

    end)

  end

  defp deploy_node(node, request, opts) do

    preinstall_steps = [:stage, :verify_stage, :unpack, :check_install]

    with :ok <- run_steps(node, preinstall_steps, request, opts),

         {:ok, _result} <- call(node, :install, [request], opts) do

      finish_installed_node(node, request, opts)

    end

  end

  defp finish_installed_node(node, request, opts) do

    with {:ok, _result} <- call(node, :verify, [request, :current], opts),

         {:ok, _result} <- call(node, :make_permanent, [request], opts),

         {:ok, _result} <- call(node, :verify, [request, :permanent], opts) do

      {:ok, "permanent"}

    else

      {:error, reason} -> reverse_after_failure(node, request, reason, opts)

    end

  end

  defp reverse_after_failure(node, request, reason, opts) do

    case call(node, :reverse, [request], opts) do

      {:ok, _result} ->

        {:error, {:node_reversed, safe_code(reason)}}

      {:error, reverse_reason} ->

        {:error, {:reverse_failed, safe_code(reason), safe_code(reverse_reason)}}

    end

  end

  defp run_steps(node, steps, request, opts) do

    Enum.reduce_while(steps, :ok, fn step, :ok ->

      case call(node, step, [request], opts) do

        {:ok, _result} -> {:cont, :ok}

        {:error, reason} -> {:halt, {:error, {step, reason}}}

      end

    end)

  end

  defp snapshot_members(request, opts) do

    current = members(opts)

    cond do

      current != request.expected_nodes -> {:error, :fleet_membership_mismatch}

      length(current) != request.expected_fleet_size -> {:error, :fleet_size_mismatch}

      true -> {:ok, current}

    end

  end

  defp stable_membership(request, opts) do

    if members(opts) == request.expected_nodes, do: :ok, else: {:error, :membership_changed}

  end

  defp members(opts) do

    Keyword.get(opts, :members, &OpenAgents.Cluster.members/0).()

    |> Enum.sort()

  end

  defp call(node, function, arguments, opts) do

    rpc = Keyword.get(opts, :rpc, &default_rpc/5)

    case rpc.(node, RelupNode, function, arguments, timeout(opts)) do

      {:ok, _result} = success -> success

      {:error, _reason} = error -> error

      other -> {:error, {:unexpected_rpc_result, safe_code(other)}}

    end

  rescue

    error -> {:error, {:rpc_exception, safe_code(error)}}

  catch

    kind, reason -> {:error, {:rpc_exit, safe_code({kind, reason})}}

  end

  defp default_rpc(node, module, function, arguments, timeout) do

    if node == Node.self() do

      apply(module, function, arguments)

    else

      :erpc.call(node, module, function, arguments, timeout)

    end

  end

  defp gate_verify(sha, opts) do

    case Keyword.fetch(opts, :gate_verifier) do

      {:ok, verifier} -> verifier.(sha)

      :error -> GateReceipt.verify(sha, Keyword.get(opts, :gate_receipt_options, []))

    end

  end

  defp validate_request(request) when is_map(request) do

    cond do

      not Regex.match?(@sha_pattern, Map.get(request, :sha, "")) ->

        {:error, :invalid_git_sha}

      not Regex.match?(@digest_pattern, Map.get(request, :artifact_digest, "")) ->

        {:error, :invalid_artifact_digest}

      not is_binary(Map.get(request, :artifact_bytes)) ->

        {:error, :invalid_artifact}

      Map.get(request, :release_name) != "openagents" ->

        {:error, :invalid_release_name}

      Map.get(request, :from_version) != "0.1.0" ->

        {:error, :unsupported_from_version}

      Map.get(request, :to_version) != "0.2.0" ->

        {:error, :unsupported_to_version}

      Map.get(request, :from_state_version) != 1 ->

        {:error, :unsupported_from_state_version}

      Map.get(request, :to_state_version) != 2 ->

        {:error, :unsupported_to_state_version}

      not is_list(Map.get(request, :expected_nodes)) ->

        {:error, :invalid_expected_nodes}

      Map.get(request, :expected_nodes) != Enum.sort(Map.get(request, :expected_nodes)) ->

        {:error, :unsorted_expected_nodes}

      Map.get(request, :expected_fleet_size) != length(Map.get(request, :expected_nodes)) ->

        {:error, :invalid_expected_fleet_size}

      true ->

        :ok

    end

  end

  defp validate_request(_request), do: {:error, :invalid_request}

  defp public_result(request, status, node_results, error_code) do

    %{

      schema: "openagents.relup-deployment.v1",

      sha: request.sha,

      artifact_digest: request.artifact_digest,

      from_version: request.from_version,

      to_version: request.to_version,

      status: status,

      node_results: node_results,

      error_code: error_code

    }

  end

  defp timeout(opts), do: Keyword.get(opts, :timeout_ms, @default_timeout_ms)

  defp safe_code(reason), do: OpenAgents.OperationalLog.code(reason)

end

defmodule OpenAgents.Forge.RelupNode do

  @moduledoc """

  Performs one node's verified OTP release-handler transaction.

  The immutable artifact remains in a digest-addressed cache. Before every

  unpack attempt, this module restores the release tar to the path consumed by

  `release_handler`, so an interrupted install can retry from the same bytes.

  """

  alias OpenAgents.ReleaseState

  @digest_pattern ~r/\A[0-9a-f]{64}\z/

  @version_pattern ~r/\A[0-9]+\.[0-9]+\.[0-9]+(?:[-+][0-9A-Za-z.-]+)?\z/

  @maximum_artifact_bytes 536_870_912

  @doc "Cache and stage an immutable release artifact."

  def stage(request, opts \\ []) do

    with :ok <- validate_request(request),

         true <-

           byte_size(request.artifact_bytes) <= @maximum_artifact_bytes or

             {:error, :artifact_too_large},

         true <-

           digest(request.artifact_bytes) == request.artifact_digest or

             {:error, :artifact_digest_mismatch},

         :ok <- ensure_release_directories(opts),

         :ok <- persist_cache(request, opts),

         :ok <- restage(request, opts) do

      {:ok, %{"phase" => "staged", "artifact_digest" => request.artifact_digest}}

    end

  end

  @doc "Verify that both cached and staged artifacts match the request."

  def verify_stage(request, opts \\ []) do

    with :ok <- verify_file(cache_path(request, opts), request.artifact_digest),

         :ok <- verify_file(stage_path(request, opts), request.artifact_digest) do

      {:ok, %{"phase" => "stage_verified"}}

    end

  end

  @doc "Restore the consumable tar and unpack it when needed."

  def unpack(request, opts \\ []) do

    with :ok <- restage(request, opts) do

      if release_known?(request.to_version, opts) do

        {:ok, %{"phase" => "unpacked", "restaged" => true}}

      else

        expected_version = to_charlist(request.to_version)

        case handler_call(opts, :unpack_release, [to_charlist(release_basename(request))]) do

          {:ok, ^expected_version} ->

            {:ok, %{"phase" => "unpacked", "restaged" => true}}

          {:error, reason} ->

            {:error, {:unpack_failed, safe_code(reason)}}

          _other ->

            {:error, :unexpected_unpack_result}

        end

      end

    end

  end

  @doc "Generate runtime configuration and preflight the relup."

  def check_install(request, opts \\ []) do

    with :ok <- generate_config(request.to_version, opts) do

      expected_from = to_charlist(request.from_version)

      case handler_call(opts, :check_install_release, [to_charlist(request.to_version)]) do

        {:ok, ^expected_from, _description} ->

          {:ok, %{"phase" => "checked"}}

        {:error, reason} ->

          {:error, {:check_failed, safe_code(reason)}}

        _other ->

          {:error, :unexpected_check_result}

      end

    end

  end

  @doc "Install the unpacked candidate without making it permanent."

  def install(request, opts \\ []) do

    expected_from = to_charlist(request.from_version)

    case handler_call(opts, :install_release, [to_charlist(request.to_version)]) do

      {:ok, ^expected_from, _description} ->

        {:ok, %{"phase" => "installed"}}

      {:continue_after_restart, ^expected_from, _description} ->

        {:ok, %{"phase" => "restart_required"}}

      {:error, reason} ->

        {:error, {:install_failed, safe_code(reason)}}

      _other ->

        {:error, :unexpected_install_result}

    end

  end

  @doc "Verify the running release, application health, and migrated state."

  def verify(request, permanence \\ :current, opts \\ []) do

    expected_status = if permanence == :permanent, do: :permanent, else: :current

    with true <-

           release_status(request.to_version, opts) == expected_status or

             {:error, :release_status_mismatch},

         %{"ready" => true} <- health_report(opts),

         %{schema_version: schema} <- state_snapshot(opts),

         true <- schema == request.to_state_version or {:error, :state_version_mismatch} do

      {:ok, %{"phase" => "verified", "permanence" => to_string(permanence)}}

    else

      {:error, reason} -> {:error, reason}

      _unhealthy -> {:error, :health_check_failed}

    end

  end

  @doc "Make the candidate release permanent."

  def make_permanent(request, opts \\ []) do

    case handler_call(opts, :make_permanent, [to_charlist(request.to_version)]) do

      :ok -> {:ok, %{"phase" => "permanent"}}

      {:error, reason} -> {:error, {:make_permanent_failed, safe_code(reason)}}

      _other -> {:error, :unexpected_make_permanent_result}

    end

  end

  @doc "Install and verify the reverse relup, then restore permanence."

  def reverse(request, opts \\ []) do

    with {:ok, _result} <- reverse_install(request, opts),

         :ok <- verify_reverse_health(request, opts),

         :ok <- reverse_permanent(request, opts),

         true <-

           release_status(request.from_version, opts) == :permanent or

             {:error, :reverse_not_permanent} do

      {:ok, %{"phase" => "reversed", "restored" => true}}