dependencies, ERTS, native code, migrations, assets, or another structural

   change that cannot use a relup.

3. Settle the original target with the fallback result so its verified build

   becomes the baseline for later direct-load classification.

   becomes the baseline for later direct-load classification. Call

   `OpenAgents.Forge.Targets.finish_relup_deployment/2` for a relup or

   `OpenAgents.Forge.Targets.finish_rolling_replacement/2` for a rolling

   replacement.

## Failure modes

- Forge and GitHub exposed `b3ae6c6` as `main`, and the public status endpoint

  reported the mirror `current`. Every node also reported the same hot

  artifact digest and target SHA.

- The documentation and CLI parity release upgraded the three-node fleet from

  `0.2.0@bd6c808` to `0.2.1@cf23f84` through a packaged relup in 53.876

  seconds. Every node returned `permanent`, the exact-SHA gate passed all 13

  stages and 2,044 tests, and the retained Forge target was settled with the

  relup package and manifest digests.

- Forge classified the activation change as `needs_rolling_replace`, and the

  operator settled it as `live` only after every node reported revision

  `3479f12`, complete cluster membership, and local health. Later compatible

  repository-owned fleet template now remove those retired containers and

  replace the disposable builder sidecar before pulling a new builder image.

The activation did not change the application version from `0.2.0`. Keep that

version for direct loads. For full compatible packages, increment only the

patch component. Change the minor component only for a planned compatibility

boundary, and never increment a release version merely to record a source

commit.

The initial activation did not change the application version from `0.2.0`.

Keep a release version unchanged for direct loads. For full compatible

packages, increment only the patch component. Change the minor component only

for a planned compatibility boundary, and never increment a release version

merely to record a source commit.

After a rollout or storage repair, force convergence before validating the

mirror:

reverse path fails, keep the node out of readiness and move to operator-directed

rolling recovery.

After the coordinator returns, settle its bounded result against the retained

Forge target:

```elixir

OpenAgents.Forge.Targets.finish_relup_deployment(target_id, relup_result)

```

Settlement accepts only the newest target in `needs_rolling_replace`, requires

the target's complete verified build receipt, and verifies the target SHA,

source SHA, package manifest digest, target artifact digest, release versions,

duration, and per-node permanence. It changes the target to `live` or `failed`

and inserts a second immutable deployment receipt in one transaction. A

successful settlement makes that build the baseline for later direct-load

classification.

`OpenAgents.Forge.RelupNode` retains the immutable tar in

`releases/.openagents-relup-cache/<sha256>.tar.gz`. It copies those exact bytes

back to the filename consumed by `unpack_release/1` before every attempt. This

recovery result, and aborts. It never replaces a second node while the first is

missing or unhealthy.

After the coordinator returns, settle its bounded result against the Forge

target:

After the rolling coordinator returns, settle its bounded result against the

Forge target:

```elixir

OpenAgents.Forge.Targets.finish_rolling_replacement(target_id, rolling_result)

      not Regex.match?(@sha_pattern, Map.get(request, :sha, "")) ->

        {:error, :invalid_git_sha}

      not Regex.match?(@sha_pattern, Map.get(request, :from_revision, "")) ->

        {:error, :invalid_from_git_sha}

      not Regex.match?(@digest_pattern, Map.get(request, :artifact_digest, "")) ->

        {:error, :invalid_artifact_digest}

      not Regex.match?(@digest_pattern, Map.get(request, :package_manifest_digest, "")) ->

        {:error, :invalid_package_manifest_digest}

      not is_binary(Map.get(request, :artifact_bytes)) ->

        {:error, :invalid_artifact}

    %{

      schema: "openagents.relup-deployment.v1",

      sha: request.sha,

      from_revision: request.from_revision,

      artifact_digest: request.artifact_digest,

      package_manifest_digest: request.package_manifest_digest,

      from_version: request.from_version,

      to_version: request.to_version,

      status: status,

  def load(directory, opts \\ [])

  def load(directory, opts) when is_binary(directory) do

    with {:ok, manifest} <- read_manifest(directory),

    with {:ok, manifest, package_manifest_digest} <- read_manifest(directory),

         :ok <- validate_manifest(manifest, opts),

         {:ok, artifact_bytes} <- read_artifact(directory, manifest),

         :ok <- verify_artifact(artifact_bytes, manifest["to_artifact_digest"]) do

      {:ok,

       %{

         sha: manifest["to_revision"],

         from_revision: manifest["from_revision"],

         release_name: manifest["release_name"],

         from_version: manifest["from_version"],

         to_version: manifest["to_version"],

         to_state_version: manifest["to_state_version"],

         artifact_bytes: artifact_bytes,

         artifact_digest: manifest["to_artifact_digest"],

         package_manifest_digest: package_manifest_digest,

         expected_nodes: nodes,

         expected_fleet_size: length(nodes)

       }}

  @doc "Validate a relup package and deploy it one node at a time."

  def deploy(directory, opts \\ []) do

    started_at = System.monotonic_time(:millisecond)

    with {:ok, request} <- load(directory, opts) do

      deployment_opts =

        Keyword.drop(opts, [:current_revision, :expected_nodes, :system_architecture])

      RelupDeployment.run(request, deployment_opts)

      case RelupDeployment.run(request, deployment_opts) do

        {status, result} when status in [:ok, :error] and is_map(result) ->

          {status,

           Map.put(

             result,

             :duration_ms,

             System.monotonic_time(:millisecond) - started_at

           )}

        other ->

          other

      end

    end

  end

         true <- stat.type == :regular and stat.size <= @maximum_manifest_bytes,

         {:ok, bytes} <- File.read(path),

         {:ok, manifest} <- Jason.decode(bytes) do

      {:ok, manifest}

      {:ok, manifest, digest(bytes)}

    else

      {:error, :enoent} -> {:error, :missing_package_manifest}

      {:error, %Jason.DecodeError{}} -> {:error, :invalid_package_manifest_json}

  another promotion, receipted like any other. Status advances through the

  deploy lane (`promoted → building → built → deploying →

  live | failed | reverted | needs_rolling_replace`) with bounded details

  at every step. An operator-approved rolling replacement can settle a

  `needs_rolling_replace` target as `live` or `failed` with a second immutable

  receipt. The `forge:target` broadcast is what wakes the builder.

  at every step. An operator-approved relup or rolling replacement can settle

  a `needs_rolling_replace` target as `live` or `failed` with a second

  immutable receipt. The `forge:target` broadcast is what wakes the builder.

  """

  import Ecto.Query

    end

  end

  @doc """

  Settle an operator-approved relup against its verified Forge build.

  The relup coordinator returns the bounded result passed to this function.

  Settlement succeeds only for the newest target, only after Forge classified

  it as `needs_rolling_replace`, and only when a complete build receipt exists.

  The target update and the second immutable deployment receipt commit in one

  transaction. The original classification receipt remains intact.

  """

  def finish_relup_deployment(target_id, relup_result) when is_map(relup_result) do

    result =

      :global.trans({{:forge_target_deploy, target_id}, self()}, fn ->

        Repo.transaction(fn ->

          target = Repo.get(Target, target_id, lock: "FOR UPDATE") || Repo.rollback(:not_found)

          current_id =

            Target

            |> where([t], t.repo == ^target.repo)

            |> order_by([t], desc: t.inserted_at)

            |> limit(1)

            |> select([t], t.id)

            |> Repo.one()

          if current_id != target.id, do: Repo.rollback(:superseded_target)

          unless target.status == "needs_rolling_replace" do

            requested_status = result_value(relup_result, :status) || "invalid"

            Repo.rollback({:invalid_transition, target.status, requested_status})

          end

          relup =

            case validate_relup_result(relup_result, target.sha) do

              {:ok, relup} -> relup

              {:error, reason} -> Repo.rollback(reason)

            end

          build =

            BuildReceipt

            |> where([b], b.target_id == ^target.id and b.status == "complete")

            |> order_by([b], desc: b.inserted_at)

            |> limit(1)

            |> Repo.one()

          if is_nil(build), do: Repo.rollback(:complete_build_receipt_not_found)

          deployment_id = Ecto.UUID.generate()

          now = DateTime.utc_now()

          details = %{

            "artifact_digest" => relup.artifact_digest,

            "build_artifact_digest" => build.artifact_digest,

            "deployment_id" => deployment_id,

            "deployment_lane" => "relup",

            "from_revision" => relup.from_revision,

            "from_version" => relup.from_version,

            "package_manifest_digest" => relup.package_manifest_digest,

            "relup_duration_ms" => relup.duration_ms,

            "relup_error_code" => relup.error_code,

            "relup_node_results" => relup.node_results,

            "to_version" => relup.to_version

          }

          target =

            target

            |> Target.status_changeset(relup.status, details)

            |> Repo.update!()

          receipt_attrs = %{

            artifact_digest: relup.artifact_digest,

            completed_at: now,

            deployment_id: deployment_id,

            error_code: relup.error_code,

            expected_nodes: relup.expected_nodes,

            manifest_digest: relup.package_manifest_digest,

            modules: build.modules,

            node_results: relup.node_results,

            nodes:

              Enum.map(relup.expected_nodes, fn node ->

                "#{node}=#{relup.node_results[node]}"

              end),

            push_to_live_ms: relup.duration_ms,

            repo: target.repo,

            result: relup.status,

            rollback_verified:

              relup.status == "failed" and

                Enum.all?(relup.node_results, fn {_node, status} -> status == "reversed" end),

            sha: target.sha,

            started_at: DateTime.add(now, -relup.duration_ms, :millisecond),

            target_id: target.id

          }

          receipt =

            %DeployReceipt{}

            |> DeployReceipt.changeset(receipt_attrs)

            |> Repo.insert()

            |> case do

              {:ok, receipt} -> receipt

              {:error, changeset} -> Repo.rollback({:invalid_receipt, changeset})

            end

          %{target: target, receipt: receipt}

        end)

      end)

    with {:ok, %{target: target} = committed} <- result do

      broadcast_status(target)

      {:ok, committed}

    end

  end

  def finish_relup_deployment(_target_id, _relup_result),

    do: {:error, :invalid_relup_result}

  @doc """

  Settle an operator-approved rolling replacement against its verified build.

  def finish_rolling_replacement(_target_id, _rolling_result),

    do: {:error, :invalid_rolling_result}

  defp validate_relup_result(result, target_sha) do

    schema = result_value(result, :schema)

    sha = result_value(result, :sha)

    from_revision = result_value(result, :from_revision)

    artifact_digest = result_value(result, :artifact_digest)

    package_manifest_digest = result_value(result, :package_manifest_digest)

    from_version = result_value(result, :from_version)

    to_version = result_value(result, :to_version)

    status = result_value(result, :status)

    node_results = result_value(result, :node_results)

    error_code = result_value(result, :error_code)

    duration_ms = result_value(result, :duration_ms)

    cond do

      schema != "openagents.relup-deployment.v1" ->

        {:error, :invalid_relup_result}

      sha != target_sha ->

        {:error, :relup_sha_mismatch}

      not valid_sha?(from_revision) ->

        {:error, :invalid_relup_result}

      not valid_digest?(artifact_digest) or not valid_digest?(package_manifest_digest) ->

        {:error, :invalid_relup_result}

      not valid_version?(from_version) or not valid_version?(to_version) or

          from_version == to_version ->

        {:error, :invalid_relup_result}

      status not in ~w(live failed) ->

        {:error, :invalid_relup_result}

      not valid_node_results?(node_results) ->

        {:error, :invalid_relup_result}

      not valid_duration?(duration_ms) ->

        {:error, :invalid_relup_result}

      status == "live" and

          (Enum.any?(node_results, fn {_node, node_status} -> node_status != "permanent" end) or

             not is_nil(error_code)) ->

        {:error, :invalid_relup_result}

      status == "failed" and not bounded_error?(error_code) ->

        {:error, :invalid_relup_result}

      true ->

        {:ok,

         %{

           artifact_digest: artifact_digest,

           duration_ms: duration_ms,

           error_code: error_code,

           expected_nodes: node_results |> Map.keys() |> Enum.sort(),

           from_revision: from_revision,

           from_version: from_version,

           node_results: node_results,

           package_manifest_digest: package_manifest_digest,

           status: status,

           to_version: to_version

         }}

    end

  end

  defp validate_rolling_result(result, target_sha) do

    schema = result_value(result, :schema)

    sha = result_value(result, :sha)

  defp valid_sha?(value) when is_binary(value), do: Regex.match?(~r/\A[0-9a-f]{40}\z/, value)

  defp valid_sha?(_value), do: false

  defp valid_digest?(value) when is_binary(value),

    do: Regex.match?(~r/\A[0-9a-f]{64}\z/, value)

  defp valid_digest?(_value), do: false

  defp valid_version?(value) when is_binary(value),

    do: Regex.match?(~r/\A[0-9]+\.[0-9]+\.[0-9]+(?:[-+][0-9A-Za-z.-]+)?\z/, value)

  defp valid_version?(_value), do: false

  defp valid_duration?(value), do: is_integer(value) and value in 0..86_400_000

  defp valid_image_digest?(value) when is_binary(value),

    do: Regex.match?(~r/\Asha256:[0-9a-f]{64}\z/, value)

  alias OpenAgents.Test.ReleaseHandler

  @sha String.duplicate("a", 40)

  @from_sha String.duplicate("c", 40)

  @digest String.duplicate("b", 64)

  @manifest_digest String.duplicate("d", 64)

  @nodes [:first@local, :second@local, :third@local]

  test "upgrades and makes each node permanent before touching the next" do

  defp request do

    %{

      sha: @sha,

      from_revision: @from_sha,

      release_name: "openagents",

      from_version: "0.1.0",

      to_version: "0.2.0",

      to_state_version: 2,

      artifact_bytes: "artifact",

      artifact_digest: @digest,

      package_manifest_digest: @manifest_digest,

      expected_nodes: @nodes,

      expected_fleet_size: 3

    }

               )

    end

    test "refuses a malformed source revision or package manifest digest" do

      assert {:error, :invalid_from_git_sha} =

               RelupDeployment.run(request() |> Map.put(:from_revision, "not-a-sha"))

      assert {:error, :invalid_package_manifest_digest} =

               RelupDeployment.run(request() |> Map.put(:package_manifest_digest, "short"))

    end

    test "refuses state version regression and unsupported state versions" do

      regression =

        request()

             )

    assert request.sha == @to_sha

    assert request.from_revision == @from_sha

    assert request.from_version == "0.2.0"

    assert request.to_version == "0.3.0"

    assert request.expected_nodes == [:first@local, :second@local]

    assert request.artifact_bytes == "target release"

    assert request.package_manifest_digest =~ ~r/\A[0-9a-f]{64}\z/

  end

  test "refuses a package for another revision, platform, or artifact" do

             Targets.finish_rolling_replacement(target.id, rolling_result(sha, "live"))

  end

  test "relup settlement makes the verified package the live baseline", %{sha: sha} do

    {:ok, target} = Targets.promote("demo", sha, "operator:test")

    {:ok, _building} = Targets.advance(target.id, "building")

    {:ok, _built} = Targets.advance(target.id, "built")

    {:ok, _relup} = Targets.advance(target.id, "needs_rolling_replace")

    build_digest = String.duplicate("a", 64)

    package_digest = String.duplicate("c", 64)

    package_manifest_digest = String.duplicate("d", 64)

    insert_build_receipt!(

      target,

      %{"classification" => "needs_rolling_replace", "source_sha" => sha},

      build_digest

    )

    result =

      relup_result(sha, "live")

      |> Map.put(:artifact_digest, package_digest)

      |> Map.put(:package_manifest_digest, package_manifest_digest)

    assert {:ok, %{target: live, receipt: receipt}} =

             Targets.finish_relup_deployment(target.id, result)

    assert live.status == "live"

    assert live.details["deployment_lane"] == "relup"

    assert live.details["artifact_digest"] == package_digest

    assert live.details["build_artifact_digest"] == build_digest

    assert receipt.result == "live"

    assert receipt.artifact_digest == package_digest

    assert receipt.manifest_digest == package_manifest_digest

    assert receipt.push_to_live_ms == 53_876

    assert receipt.node_results == %{

             "openagents@10.42.0.11" => "permanent",

             "openagents@10.42.0.12" => "permanent"

           }

  end

  test "relup settlement refuses a nonpermanent live result", %{sha: sha} do

    {:ok, target} = Targets.promote("demo", sha, "operator:test")

    {:ok, _building} = Targets.advance(target.id, "building")

    {:ok, _built} = Targets.advance(target.id, "built")

    {:ok, _relup} = Targets.advance(target.id, "needs_rolling_replace")

    insert_build_receipt!(target, %{"source_sha" => sha}, String.duplicate("a", 64))

    result =

      relup_result(sha, "live")

      |> put_in([:node_results, "openagents@10.42.0.12"], "current")

    assert {:error, :invalid_relup_result} =

             Targets.finish_relup_deployment(target.id, result)

    assert Repo.get!(OpenAgents.Forge.Target, target.id).status == "needs_rolling_replace"

  end

  test "rolling replacement records a complete large module inventory", %{sha: sha} do

    {:ok, target} = Targets.promote("demo", sha, "operator:test")

    {:ok, _building} = Targets.advance(target.id, "building")

      recovery: nil

    }

  end

  defp relup_result(sha, status) do

    %{

      schema: "openagents.relup-deployment.v1",

      sha: sha,

      from_revision: String.duplicate("b", 40),

      artifact_digest: String.duplicate("c", 64),

      package_manifest_digest: String.duplicate("d", 64),

      from_version: "0.2.0",

      to_version: "0.2.1",

      status: status,

      node_results: %{

        "openagents@10.42.0.11" => "permanent",

        "openagents@10.42.0.12" => "permanent"

      },

      error_code: nil,

      duration_ms: 53_876

    }

  end

end