Status: Current

Direct BEAM candidates use an exact-fleet prepare, canary, apply, verify,

commit, and rollback transaction. The supported `0.1.0` to `0.2.0` application

transition uses a two-way relup, versioned process state, node-by-node health

checks, and reverse installation. Every structural or unclassified candidate

uses digest-addressed rolling replacement with readiness drain, remaining-

capacity and quorum checks, exact rejoin verification, and last-known-good

image recovery. A failed relup or replacement aborts before another node

changes.

commit, and rollback transaction. An application transition between any two

concrete `X.Y.Z` versions uses a two-way relup, versioned process state,

node-by-node health checks, and reverse installation. Every structural or

unclassified candidate uses digest-addressed rolling replacement with readiness

drain, remaining-capacity and quorum checks, exact rejoin verification, and

last-known-good image recovery. A failed relup or replacement aborts before

another node changes.

A packaged relup describes the two revisions it was built from. The appup is

derived from both builds' compiled modules, so the relup carries an instruction

for every module whose code differs, and packaging refuses when the generated

relup omits one. A node therefore cannot install part of a revision while

reporting itself converged. A version that a node already unpacked from

different artifact bytes is refused rather than reused.

The reverse path restores the `from` release rather than a fixed one. Each

direction's target state schema travels in the appup, so a downgrade migrates

process state to the schema the `from` release compiled — including a pair

whose schemas match, which keeps its schema instead of being forced back to

schema 1. Reverse verification checks release status, node readiness, and that

exact schema before it restores permanence, and a downgrade that is given no

target schema refuses instead of guessing.

All deployment workers remain disabled until isolated staging proves their

complete provider and topology. Current means that the local mechanism and its

refusal and recovery paths exist; it does not authorize staging or production.

`OpenAgents.Forge.RelupDeployment.run/2` has no production caller, and the

release gate runs neither `ops/forge/package-relup.sh` nor

`ops/relup-proof/install-proof.sh`.

Evidence: `OpenAgents.Forge.Deployment`, `OpenAgents.Forge.RelupDeployment`,

`OpenAgents.Forge.RelupNode`, `OpenAgents.ReleaseState`,

`OpenAgents.Forge.RollingReplacement`,

`OpenAgents.Release.Appup`, `OpenAgents.Forge.RollingReplacement`,

`test/openagents/forge/relup_deployment_test.exs`,

`test/openagents/forge/relup_node_test.exs`,

`test/openagents/release/appup_test.exs`,

`test/openagents/cluster/code_change_test.exs`,

`test/openagents/forge/rolling_replacement_test.exs`, and

`docs/operations/release-deployment-fallbacks.md`.

| RELEASE-002 | `test/openagents/github_oauth/runtime_config_test.exs`, `ops/ci/reference-check.sh` |

| RELEASE-003 | `test/openagents_web/allowed_origins_test.exs`, `ops/ci/release-smoke.sh` |

| RELEASE-004 | `ops/ci/gate.sh`, `test/openagents/forge/gate_receipt_test.exs` |

| RELEASE-005 | `test/openagents/forge/relup_deployment_test.exs`, `test/openagents/forge/rolling_replacement_test.exs` |

| RELEASE-005 | `test/openagents/forge/relup_deployment_test.exs`, `test/openagents/forge/relup_node_test.exs`, `test/openagents/release/appup_test.exs`, `test/openagents/cluster/code_change_test.exs`, `test/openagents/forge/rolling_replacement_test.exs` |

| STATUS-001 | `test/openagents/network_status_test.exs`, `test/openagents_web/live/network_status_live_test.exs` |

| TRANSPARENCY-001 | `test/openagents/forge/visibility_test.exs`, `test/openagents/forge/browse_test.exs`, `test/openagents_web/live/code_live_test.exs` |

| REPOSITORY-001 | `test/openagents/repository_lifecycle_test.exs`, `test/openagents/repositories/provisioner_test.exs`, `test/openagents_web/controllers/repository_controller_test.exs`, `test/openagents/forge/git_http_test.exs` |

Meanwhile every production deploy rides the structural class: one immutable image built locally under amd64 emulation (~9 minutes for commit `eda094c`), pushed to Artifact Registry, qualified on staging, then rolled one node at a time (~7 minutes). Push-to-live measured about 23 minutes on 2026-08-21 for a release whose changes were mostly hot-loadable LiveView and template work — the exact class Sarah moved in seconds.

Closing the gap is operator work, not engineering work: apply the staging Terraform, run Gates 12–15 against the real builder image, pin the builder digest into fleet metadata, flip the feature flags, then extend the runbooks with the consolidated decision table and drill lessons listed in section 5.

Closing the gap for the direct-load lane is operator work: apply the staging Terraform, run Gates 12–15 against the real builder image, pin the builder digest into fleet metadata, flip the feature flags, then extend the runbooks with the consolidated decision table and drill lessons listed in section 5. The general relup lane is the exception — it still needs engineering work, described in the addendum below.

### Addendum: post-measurement production events (2026-08-21, later)

   general relup path was, at measurement time, unconnected: `BuildArtifact`

   classifies only `direct_candidate` and `needs_rolling_replace`, and

   `RelupDeployment` accepted only the fixed `0.1.0 → 0.2.0` proof transition.

   Later the same day the relup lane was connected for general version pairs —

   coordinator admission, appup generation, packaging, and install proofs; see

   [`docs/operations/forge-hot-loop.md`](operations/forge-hot-loop.md).

   Production approval for the lane remains an open operator decision.

   Later the same day the relup **mechanism** was generalized — coordinator

   admission, appup generation from the two builds' compiled modules,

   packaging, and install proofs; see

   [`docs/operations/forge-hot-loop.md`](operations/forge-hot-loop.md). The

   **lane** is still not connected: no classifier emits a relup class,

   `RelupDeployment.run/2` has no caller outside tests, the release gate runs

   neither `ops/forge/package-relup.sh` nor

   `ops/relup-proof/install-proof.sh`, and no receipt binds a package to the

   revision it was built from. Those are engineering gaps, not operator ones.

   Production approval for the lane also remains an open operator decision.

---

| --- | --- | --- |

| Direct BEAM transaction on production | Works | `fa4b792` loaded across three nodes via the transaction protocol; `live` target and deployment receipt recorded; uptimes unbroken |

| Automated push → promote → build → hot-load loop | Not yet operating | No receipted automated deploy exists, so `/api/status` reports `loop.last_ms: null` and `push_to_live_ms: null` |

| General relup lane | Proof harness only | The classifier emits only `direct_candidate` or `needs_rolling_replace`; `RelupDeployment` admits only the fixed `0.1.0 → 0.2.0` proof transition while production already runs `0.2.0`. Relups are not production-approved (`release-deployment-fallbacks.md`) |

| General relup lane | Manual tools only | `RelupDeployment` admits any distinct `X.Y.Z` pair, but the classifier emits only `direct_candidate` or `needs_rolling_replace`, `RelupDeployment.run/2` has no caller outside tests, and the release gate runs neither packaging script. Relups are not production-approved (`release-deployment-fallbacks.md`) |

| Rolling image replacement | Works, default for structural changes | Current release tooling path |

Two consequences worth stating plainly:

A `reverted` outcome still warrants checking fleet convergence even though

this design captures each node's prior object code for exact rollback.

## The relup lane is now connected

## The relup lane generalized, but nothing drives it

As of 2026-08-21 (later), the three gaps recorded below are closed in code:

As of 2026-08-21 (later), the mechanism handles arbitrary version pairs:

1. **Coordinator admission is general.** `RelupDeployment` admits any distinct

   `X.Y.Z` pair whose state versions stay within `[1, 2]` and never regress —

   matching what `RelupNode` already enforced per node. The packaged appup on

   the target nodes remains the real gate: `check_install_release` refuses

   honestly when no relup can be produced between two versions.

2. **Appup generation works for any admitted pair.** `rel/openagents.appup.exs`

   generates forward and reverse instructions from `RELUP_FROM`/`RELUP_TO`

   for arbitrary distinct versions instead of raising outside the proof

   transition.

2. **Appup generation describes the pair it was built from.**

   `rel/openagents.appup.exs` asks `OpenAgents.Release.Appup` to diff the two

   builds' compiled modules and emits one instruction per module that differs,

   plus the advanced `ReleaseState` update carrying each direction's target

   state schema. `mix openagents.relup` then checks the generated relup against

   the same diff, so packaging fails rather than shipping a relup that would

   install part of a revision.

3. **Packaging and install proofs exist as tools.**

   `ops/forge/package-relup.sh --from-version A --to-version B [--from-rev]

   [--to-rev]` builds both releases in isolated worktrees, generates the

   two-way relup, embeds it, and emits digest-addressed tarballs plus a

   `package.json` ready for deployment requests.

   `ops/relup-proof/install-proof.sh` then proves the pair against a live

   `ops/relup-proof/install-proof.sh` then proves that package against a live

   single-node release: forward install, permanent commit, reverse rollback,

   and re-upgrade, asserting `ReleaseState` observations survive every

   transition. Both were executed successfully for `0.2.0 → 0.3.0`.

Still required before the lane carries production traffic — operator work,

not engineering work:

   and re-upgrade, asserting each `release_handler` result, both state

   schemas, node readiness, and `ReleaseState` retention. It reads the versions

   and schemas from `package.json`, and it creates and drops a database, so it

   refuses to run unless `OPENAGENTS_RELUP_PROOF_DISPOSABLE=1`, the URL host is

   loopback, and the database name contains `proof`, `smoke`, or `test`.

What is not true yet, and is engineering work rather than operator work:

- **No code calls the lane.** `RelupDeployment.run/2` has no caller outside

  tests. Nothing classifies a candidate as a relup, builds the request, or

  triggers a fleet relup; an operator drives the two scripts by hand.

- **The release gate does not run either script.** `ops/ci/gate.sh` runs

  `ops/relup-proof/run.sh`, `version-chain.sh`, and `kill-during-install.sh`,

  which exercise the pinned `0.1.0 → 0.2.0` pair only. Neither

  `package-relup.sh` nor `install-proof.sh` is a gate stage, so no receipt

  binds a general pair to a candidate SHA.

- **Nothing binds a package to its gate receipt.** `package.json` records the

  revisions it was built from, but `RelupDeployment` verifies the receipt for

  `request.sha` without checking that the artifact came from that revision.

Operator work still required before the lane carries production traffic:

- Production approval recorded against

  [`docs/operations/release-deployment-fallbacks.md`](release-deployment-fallbacks.md),

  which remains the authority that relups are not production-approved.

- Wiring `package-relup.sh` into automation so a promoted SHA packages and

  deploys without manual steps. Until then the lane is driven by the same

  commands above, which are safe to rehearse on staging.

- A staging rehearsal of `package-relup.sh` followed by `install-proof.sh`

  against a disposable database, before any fleet use.

Note on scope: hot-load diffs and relups remain different artifact classes.

A BEAM-diff artifact cannot drive `release_handler`; only a full release

back to the filename consumed by `unpack_release/1` before every attempt. This

restaging step is mandatory after an interrupted install.

The node also records which artifact each unpacked version came from, in

`releases/.openagents-relup-cache/unpacked-<version>`. Reusing an already

unpacked version requires the recorded digest to match the request, so

re-cutting a version number from a different revision fails with

`unpacked_version_conflict` instead of installing the bytes the node unpacked

the first time. Recover by cutting a new version number for the new revision.

## Rolling replacement lane

Build a local immutable image only after the exact release gate passes:

  Generates a two-way `relup` in an explicit output directory.

      mix openagents.relup --target /path/to/0.2.0/openagents \

        --from /path/to/0.1.0/openagents --outdir /tmp/proof

        --from /path/to/0.1.0/openagents --outdir /tmp/proof \

        --from-ebin /path/to/0.1.0/_build/prod/lib/openagents/ebin \

        --to-ebin /path/to/0.2.0/_build/prod/lib/openagents/ebin \

        --from-state 1 --to-state 2

  The `ebin` and state arguments are required: after generating the relup the

  task checks it against `OpenAgents.Release.Appup`, so a relup that covers

  fewer modules than the two builds differ in fails here rather than

  half-upgrading a node.

  """

  use Mix.Task

  @shortdoc "Generates an explicit two-way OpenAgents relup"

  @switches [target: :string, from: :string, outdir: :string]

  alias OpenAgents.Release.Appup

  @shortdoc "Generates and verifies an explicit two-way OpenAgents relup"

  @switches [

    target: :string,

    from: :string,

    outdir: :string,

    from_ebin: :string,

    to_ebin: :string,

    from_state: :integer,

    to_state: :integer

  ]

  @impl true

  def run(arguments) do

    target = required_path!(options, :target)

    from = required_path!(options, :from)

    outdir = required_directory!(options, :outdir)

    outdir = created_directory!(options, :outdir)

    paths =

      [target, from]

           warnings_as_errors: true

         ) do

      {:ok, _relup, _module, []} ->

        Mix.shell().info("Generated #{Path.join(outdir, "relup")}")

        relup = Path.join(outdir, "relup")

        covered =

          Appup.verify_relup!(relup,

            from_ebin: required_directory!(options, :from_ebin),

            to_ebin: required_directory!(options, :to_ebin),

            from_state_version: Keyword.fetch!(options, :from_state),

            to_state_version: Keyword.fetch!(options, :to_state)

          )

        Mix.shell().info("Generated #{relup}, covering #{length(covered)} modules")

      {:ok, _relup, module, warnings} ->

        Mix.raise(

      else: Mix.raise("--#{key} must name a release resource without the .rel suffix")

  end

  defp required_directory!(options, key) do

  defp created_directory!(options, key) do

    path = options |> Keyword.fetch!(key) |> Path.expand()

    File.mkdir_p!(path)

    path

  end

  defp required_directory!(options, key) do

    path = options |> Keyword.fetch!(key) |> Path.expand()

    if File.dir?(path),

      do: path,

      else: Mix.raise("--#{String.replace(to_string(key), "_", "-")} must name a directory")

  end

  defp format(module, function, value) do

    module

    |> apply(function, [value])

    end

  end

  @doc "Restore the consumable tar and unpack it when needed."

  @doc """

  Restore the consumable tar and unpack it when needed.

  A release version already known to `release_handler` is only reused when it

  was unpacked from these exact artifact bytes, recorded next to the immutable

  cache. Re-cutting a version from a different revision therefore fails closed

  instead of installing whatever the node unpacked the first time.

  """

  def unpack(request, opts \\ []) do

    with :ok <- restage(request, opts) do

      if release_known?(request.to_version, opts) do

        {:ok, %{"phase" => "unpacked", "restaged" => true}}

      else

        expected_version = to_charlist(request.to_version)

      cond do

        not release_known?(request.to_version, opts) ->

          perform_unpack(request, opts)

        unpacked_digest(request, opts) == request.artifact_digest ->

          {:ok, %{"phase" => "unpacked", "restaged" => true}}

        true ->

          {:error, :unpacked_version_conflict}

      end

    end

  end

        case handler_call(opts, :unpack_release, [to_charlist(release_basename(request))]) do

          {:ok, ^expected_version} ->

            {:ok, %{"phase" => "unpacked", "restaged" => true}}

  defp perform_unpack(request, opts) do

    expected_version = to_charlist(request.to_version)

          {:error, reason} ->

            {:error, {:unpack_failed, safe_code(reason)}}

    # Record the intent before unpacking, so an interrupted unpack retries from

    # the same bytes instead of reading as a conflicting version.

    with :ok <- record_unpacked_digest(request, opts) do

      case handler_call(opts, :unpack_release, [to_charlist(release_basename(request))]) do

        {:ok, ^expected_version} ->

          {:ok, %{"phase" => "unpacked", "restaged" => true}}

          _other ->

            {:error, :unexpected_unpack_result}

        end

        {:error, reason} ->

          {:error, {:unpack_failed, safe_code(reason)}}

        _other ->

          {:error, :unexpected_unpack_result}

      end

    end

  end

  defp unpacked_digest(request, opts) do

    case File.read(unpacked_digest_path(request, opts)) do

      {:ok, digest} -> String.trim(digest)

      {:error, _reason} -> nil

    end

  end

  defp record_unpacked_digest(request, opts) do

    File.mkdir_p!(cache_dir(opts))

    File.write!(unpacked_digest_path(request, opts), request.artifact_digest)

    :ok

  rescue

    _error -> {:error, :release_directory_unavailable}

  end

  defp unpacked_digest_path(request, opts),

    do: Path.join(cache_dir(opts), "unpacked-#{request.to_version}")

  @doc "Generate runtime configuration and preflight the relup."

  def check_install(request, opts \\ []) do

    with :ok <- generate_config(request.to_version, opts) do

defmodule OpenAgents.Release.Appup do

  @moduledoc """

  Derives one release pair's appup instructions from the two builds' modules.

  `:systools.make_relup/4` consumes an appup verbatim. It never compares module

  contents, so a constant instruction list upgrades only the modules it names

  and leaves every other changed module running the old code while the node

  reports itself converged. This module reads both builds' compiled modules and

  emits one instruction per module that actually differs.

  Both directories must be application `ebin` directories from the same build

  stage — `_build/<env>/lib/openagents/ebin` on each side — so that protocol

  consolidation, which `mix release` performs later, cannot make the two sides

  disagree about modules neither revision changed.

  `OpenAgents.ReleaseState` always takes an advanced update carrying the target

  schema version, because the target schema belongs to the other release and a

  running module cannot infer it. `OpenAgents.ReleaseState.install_barrier/0`

  runs after the module instructions in both directions.

  Consolidated protocol modules live outside the application `ebin` directory

  and no appup covers them. A revision that changes protocol consolidation is a

  structural change and belongs on the rolling replacement path.

  """

  @state_module OpenAgents.ReleaseState

  @barrier {:apply, {OpenAgents.ReleaseState, :install_barrier, []}}

  @schema_versions [1, 2]

  @version_pattern ~r/\A\d+\.\d+\.\d+(?:[-+][0-9A-Za-z.-]+)?\z/

  @doc """

  Build the complete appup term for one concrete release pair.

  Returns `{to_version, [{from_version, up}], [{from_version, down}]}` in the

  charlist shape `:systools` expects.

  """

  def build(options) do

    from_version = version!(options, :from_version)

    to_version = version!(options, :to_version)

    if from_version == to_version do

      raise ArgumentError, "from_version and to_version must differ"

    end

    {up, down} = instructions(options)

    {String.to_charlist(to_version), [{String.to_charlist(from_version), up}],

     [{String.to_charlist(from_version), down}]}

  end

  @doc """

  Return `{up, down}` instruction lists covering every module that differs.

  Options:

    * `:from_ebin` — the older build's application `ebin` directory

    * `:to_ebin` — the newer build's application `ebin` directory

    * `:from_state_version` — the older build's `ReleaseState` schema

    * `:to_state_version` — the newer build's `ReleaseState` schema

  """

  def instructions(options) do

    from = read_modules(Keyword.fetch!(options, :from_ebin))

    to = read_modules(Keyword.fetch!(options, :to_ebin))

    from_schema = schema!(options, :from_state_version)

    to_schema = schema!(options, :to_state_version)

    added = Enum.sort(Map.keys(to) -- Map.keys(from))

    removed = Enum.sort(Map.keys(from) -- Map.keys(to))

    changed =

      for {module, entry} <- to,

          previous = Map.get(from, module),

          previous != nil,

          previous.digest != entry.digest,

          do: module

    migrated =

      changed

      |> with_state_module(from, to)

      |> Enum.sort()

    up =

      Enum.map(added, &{:add_module, &1}) ++

        Enum.map(migrated, &instruction(&1, to, to_schema)) ++

        [@barrier] ++

        Enum.map(removed, &{:delete_module, &1})

    down =

      Enum.map(removed, &{:add_module, &1}) ++

        Enum.map(migrated, &instruction(&1, from, from_schema)) ++

        [@barrier] ++

        Enum.map(added, &{:delete_module, &1})

    {up, down}

  end

  @doc """

  Raise unless the generated relup names every module the two builds differ in.

  `:systools` reads the appup from the candidate's `ebin` directory, so a stale

  appup, missing build environment, or dropped instruction produces a relup

  that installs fewer modules than the revisions changed. This recomputes the

  instruction list from the same two `ebin` directories and checks the relup

  against it in both directions. It proves the packaging pipeline carried the

  instructions through; it cannot prove that loading a module is the right

  instruction for it.

  Returns the sorted list of modules the upgrade direction covers.

  """

  def verify_relup!(path, options) do

    {up, down} = instructions(options)

    case :file.consult(String.to_charlist(path)) do

      {:ok, [{_to_version, up_entries, down_entries}]} ->

        check_direction!(path, "upgrade", up, up_entries)

        check_direction!(path, "downgrade", down, down_entries)

        up |> Enum.flat_map(&instruction_module/1) |> Enum.sort()

      other ->

        raise ArgumentError, "#{path} is not a relup file: #{inspect(other)}"

    end

  end

  @doc "Return every atom a relup or appup instruction term names."

  def named_atoms(term) when is_tuple(term), do: term |> Tuple.to_list() |> named_atoms()

  def named_atoms(term) when is_list(term), do: Enum.flat_map(term, &named_atoms/1)

  def named_atoms(term) when is_atom(term), do: [term]

  def named_atoms(_term), do: []

  defp check_direction!(path, direction, expected_instructions, entries) do

    expected = expected_instructions |> Enum.flat_map(&instruction_module/1) |> MapSet.new()

    named = entries |> named_atoms() |> MapSet.new()

    missing = expected |> MapSet.difference(named) |> Enum.sort()

    if missing != [] do

      raise "#{path} carries no #{direction} instruction for: #{Enum.map_join(missing, ", ", &inspect/1)}"

    end

  end

  defp instruction_module({:add_module, module}), do: [module]

  defp instruction_module({:delete_module, module}), do: [module]

  defp instruction_module({:load_module, module}), do: [module]

  defp instruction_module({:update, module, _change}), do: [module]

  defp instruction_module(_instruction), do: []

  defp with_state_module(changed, from, to) do

    if Map.has_key?(from, @state_module) and Map.has_key?(to, @state_module) do

      Enum.uniq([@state_module | changed])

    else

      changed

    end

  end

  defp instruction(@state_module, _modules, schema),

    do: {:update, @state_module, {:advanced, [schema_version: schema]}}

  defp instruction(module, modules, _schema) do

    if modules[module].supervisor?,

      do: {:update, module, :supervisor},

      else: {:load_module, module}

  end

  defp read_modules(directory) when is_binary(directory) do

    beams = directory |> Path.join("*.beam") |> Path.wildcard()

    if beams == [] do

      raise ArgumentError, "no compiled modules found in #{directory}"

    end

    Map.new(beams, &read_module/1)

  end

  defp read_module(path) do

    charlist = String.to_charlist(path)

    case :beam_lib.md5(charlist) do

      {:ok, {module, digest}} ->

        {module, %{digest: digest, supervisor?: supervisor?(charlist)}}

      {:error, :beam_lib, reason} ->

        raise ArgumentError, "cannot read #{path}: #{inspect(reason)}"

    end

  end

  # `use Supervisor` records the Elixir behaviour, but the process it starts is

  # an `:supervisor` process registered under the callback module, so the

  # supervisor instruction is the one that reaches it. Anything else takes

  # `load_module`, which is correct whenever the process state shape did not

  # change; a revision that changes a running process's state shape is a

  # structural change and belongs on the rolling replacement path.

  defp supervisor?(charlist) do

    case :beam_lib.chunks(charlist, [:attributes]) do

      {:ok, {_module, [attributes: attributes]}} ->

        attributes

        |> Enum.flat_map(fn

          {:behaviour, behaviours} when is_list(behaviours) -> behaviours

          _other -> []

        end)

        |> Enum.any?(&(&1 in [:supervisor, Supervisor]))

      _other ->

        false

    end

  end

  defp version!(options, key) do

    version = Keyword.fetch!(options, key)

    if is_binary(version) and Regex.match?(@version_pattern, version) do

      version

    else

      raise ArgumentError, "#{key} must be an X.Y.Z version, got: #{inspect(version)}"

    end

  end

  defp schema!(options, key) do

    schema = Keyword.fetch!(options, key)

    if schema in @schema_versions do

      schema

    else

      raise ArgumentError,

            "#{key} must be one of #{inspect(@schema_versions)}, got: #{inspect(schema)}"

    end

  end

end

  @moduledoc """

  Holds bounded runtime observations across supported release upgrades.

  The process uses an explicitly versioned state struct. The `0.1.0` release

  uses schema 1 and the `0.2.0` release uses schema 2. A two-way relup calls

  `code_change/3`, which preserves the PID and observations while adding or

  removing schema 2's integrity field.

  The process uses an explicitly versioned state struct whose schema is

  compiled into each release from `OPENAGENTS_RELUP_STATE_VERSION`. A two-way

  relup calls `code_change/3`, which preserves the PID and observations while

  adding or removing schema 2's integrity field.

  The target schema is explicit, not positional: `OpenAgents.Release.Appup`

  puts the installing release's schema in the appup's `extra` term for each

  direction. A pair whose schemas match on both sides therefore keeps its

  schema through a downgrade instead of being forced back to schema 1.

  """

  use GenServer

  end

  @impl true

  def code_change({:down, _from_version}, %State{} = state, _extra) do

    {:ok, state_for(1, state.observations)}

  def code_change({:down, _from_version}, %State{} = state, extra) do

    # A downgrade runs this clause in the new module before the old code is

    # loaded, so the target schema belongs to the release being installed and

    # this module cannot infer it. The appup carries it in `extra`; without it

    # the migration refuses rather than guessing a schema.

    case target_schema(extra) do

      {:ok, schema} -> {:ok, state_for(schema, state.observations)}

      :error -> {:error, :missing_downgrade_schema_version}

    end

  end

  def code_change(_from_version, %State{schema_version: 1} = state, _extra) do

    {:ok, state_for(2, state.observations)}

  def code_change(_from_version, %State{} = state, extra) do

    # An upgrade runs in the release being installed, so this release's own

    # compiled schema is the target when the appup names none.

    case target_schema(extra) do

      {:ok, schema} -> {:ok, state_for(schema, state.observations)}

      :error -> {:ok, state_for(@current_schema, state.observations)}

    end

  end

  def code_change(_from_version, %State{} = state, _extra) do

    {:ok, state_for(@current_schema, state.observations)}

  defp target_schema(extra) when is_list(extra) do

    Enum.find_value(extra, :error, fn

      {:schema_version, schema} when schema in [1, 2] -> {:ok, schema}

      _other -> nil

    end)

  end

  defp target_schema(_extra), do: :error

  defp state_for(1, observations) do

    %State{schema_version: 1, observations: observations, integrity: nil}

  end

        ]

      ],

      deps: deps(),

      compilers: [:appup, :phoenix_live_view] ++ Mix.compilers(),

      # `:appup` runs last because rel/openagents.appup.exs derives its

      # instruction list from this build's compiled modules through

      # OpenAgents.Release.Appup, which only exists once :elixir has run.

      compilers: [:phoenix_live_view] ++ Mix.compilers() ++ [:appup],

      appup: "rel/openagents.appup.exs",

      releases: releases(),

      listeners: [Phoenix.CodeReloader]

  OPENAGENTS_RELUP_STATE_VERSION="$from_state" \

  sh -c 'cd "$1" && mix do compile --force --warnings-as-errors + release --overwrite' sh "$build_root/from"

# The appup derives its instruction list from these two module sets, so every

# module that differs between the revisions is covered. Both sides must be

# _build ebin directories: protocol consolidation happens later, in `mix

# release`, and comparing a release ebin against a _build ebin would report

# modules as added or deleted that neither revision touched.

from_ebin="$build_root/from/_build/prod/lib/openagents/ebin"

[ -d "$from_ebin" ] || { echo "missing from-build ebin: $from_ebin" >&2; exit 1; }

echo "Building $to_version release resource"

env -u OPENAGENTS_RELUP_PATH \

  MIX_ENV=prod \

  OPENAGENTS_RELUP_STATE_VERSION="$to_state" \

  RELUP_FROM="$from_version" \

  RELUP_TO="$to_version" \

  RELUP_FROM_EBIN="$from_ebin" \

  RELUP_FROM_STATE="$from_state" \

  RELUP_TO_STATE="$to_state" \

  sh -c 'cd "$1" && mix do compile --force --warnings-as-errors + release --overwrite' sh "$build_root/to"

to_ebin="$build_root/to/_build/prod/lib/openagents/ebin"

[ -d "$to_ebin" ] || { echo "missing to-build ebin: $to_ebin" >&2; exit 1; }

echo "Generating the two-way relup"

(cd "$build_root/to" && MIX_ENV=prod mix openagents.relup \

  --target "$build_root/to/release/releases/$to_version/openagents" \

  --from "$build_root/from/release/releases/$from_version/openagents" \

  --outdir "$build_root")

  --outdir "$build_root" \

  --from-ebin "$from_ebin" \

  --to-ebin "$to_ebin" \

  --from-state "$from_state" \

  --to-state "$to_state")

echo "Reassembling $to_version with the embedded relup"

env \

  OPENAGENTS_RELUP_STATE_VERSION="$to_state" \

  RELUP_FROM="$from_version" \

  RELUP_TO="$to_version" \

  RELUP_FROM_EBIN="$from_ebin" \

  RELUP_FROM_STATE="$from_state" \

  RELUP_TO_STATE="$to_state" \

  sh -c 'cd "$1" && mix release --overwrite' sh "$build_root/to"

tar -tzf "$build_root/to/release/openagents-$to_version.tar.gz" |

#!/bin/sh

# Installs an already-packaged two-way relup against a live single-node

# release, proving forward upgrade, reverse rollback, and re-upgrade with

# ReleaseState retention. Consumes the output of ops/forge/package-relup.sh.

# ReleaseState retention. Consumes the output of ops/forge/package-relup.sh,

# including its versions and state schemas, so the assertions describe the

# package actually under test rather than a hardcoded pair.

#

# This script creates and drops a database. It refuses to run unless

# OPENAGENTS_RELUP_PROOF_DISPOSABLE=1, the URL host is loopback, and the

# database name contains `proof`, `smoke`, or `test`. It performs every one of

# those checks before it arms its cleanup trap, so a refused run destroys

# nothing.

#

# Required environment: OPENAGENTS_RELUP_PROOF_DISPOSABLE=1 and

# OPENAGENTS_RELUP_PROOF_DATABASE_URL (a disposable PostgreSQL URL),

# matching the contract of ops/relup-proof/run.sh.

# OPENAGENTS_RELUP_PROOF_DATABASE_URL.

#

# Optional: RELUP_PROOF_PACKAGE_DIR, RELUP_PROOF_NODE, RELUP_PROOF_PORT.

# Install proof for a generalized relup pair: 0.2.0 -> 0.3.0 -> reverse ->

# re-upgrade, using the tarballs produced by ops/forge/package-relup.sh.

#

# Assertions check observable effects (directories, running version, retained

# state) rather than release_handler return values relayed through the shell

# rpc channel, whose IO can drop mid-call while release_handler runs.

# Every release_handler call records its result to a file on the node and the

# assertion reads that file back. The rpc channel's IO can drop mid-call while

# release_handler suspends processes, so the shell cannot trust what it saw on

# stdout; it can trust what the node wrote. The reverse leg asserts exactly

# what OpenAgents.Forge.RelupNode.verify_reverse_health/2 asserts: the from

# release is current or permanent, the node reports ready, and the process

# state carries the packaged from_state_version.

set -eu

repo_root=$(CDPATH= cd -- "$(dirname -- "$0")/../.." && pwd)

pkg="${RELUP_PROOF_PACKAGE_DIR:-$repo_root/.git/openagents/relup-package}"

runtime_root=$(mktemp -d /tmp/openagents-relup-runtime.XXXXXX)

release_log="$runtime_root/release.log"

node_name="${RELUP_PROOF_NODE:-openagents-relup-proof@127.0.0.1}"

db="${RELUP_PROOF_DB:-openagents_relup_proof}"

cleanup() {

  if [ -n "${release_pid:-}" ]; then

    profile "$runtime_root/bin/openagents" stop >/dev/null 2>&1 || true

    kill -TERM "$release_pid" 2>/dev/null || true

  fi

  [ -d "$runtime_root" ] && find "$runtime_root" -depth -delete

  dropdb -h localhost --if-exists "$db" >/dev/null 2>&1 || true

}

trap cleanup EXIT INT TERM

if [ "${OPENAGENTS_RELUP_PROOF_DISPOSABLE:-}" != "1" ]; then

  echo "set OPENAGENTS_RELUP_PROOF_DISPOSABLE=1 for a disposable database" >&2

fi

proof_database_url=${OPENAGENTS_RELUP_PROOF_DATABASE_URL:-}

[ -n "$proof_database_url" ] || {

if [ -z "$proof_database_url" ]; then

  echo "OPENAGENTS_RELUP_PROOF_DATABASE_URL is required" >&2

  exit 1

fi

authority=${proof_database_url#*://}

db_path=${authority#*/}

authority=${authority%%/*}

host_port=${authority##*@}

db_host=${host_port%%:*}

db_name=${db_path%%\?*}

case "$host_port" in

  *:*) db_port=${host_port##*:} ;;

  *) db_port=5432 ;;

esac

case "$db_host" in

  localhost | 127.0.0.1) : ;;

  *)

    echo "refusing: the proof database host must be loopback, got $db_host" >&2

    exit 1

    ;;

esac

case "$db_name" in

  *proof* | *smoke* | *test*) : ;;

  *)

    echo "refusing: the proof database name must contain proof, smoke, or test" >&2

    exit 1

    ;;

esac

if [ ! -f "$pkg/package.json" ]; then

  echo "missing relup package: $pkg/package.json" >&2

  echo "run ops/forge/package-relup.sh --out-dir $pkg first" >&2

  exit 1

fi

package_field() {

  sed -n "s/^[[:space:]]*\"$1\"[[:space:]]*:[[:space:]]*\"\{0,1\}\([^\",]*\)\"\{0,1\},\{0,1\}[[:space:]]*$/\1/p" \

    "$pkg/package.json" | head -1

}

from_version=$(package_field from_version)

to_version=$(package_field to_version)

from_state=$(package_field from_state_version)

to_state=$(package_field to_state_version)

for field in "$from_version" "$to_version" "$from_state" "$to_state"; do

  [ -n "$field" ] || { echo "package.json is missing a required field" >&2; exit 1; }

done

runtime_root=$(mktemp -d /tmp/openagents-relup-runtime.XXXXXX)

release_log="$runtime_root/release.log"

result_file="$runtime_root/handler-result"

release_pid=

cleanup() {

  if [ -n "${release_pid:-}" ]; then

    profile "$runtime_root/bin/openagents" stop >/dev/null 2>&1 || true

    kill -TERM "$release_pid" 2>/dev/null || true

  fi

  [ -d "$runtime_root" ] && find "$runtime_root" -depth -delete

  dropdb -h "$db_host" -p "$db_port" --if-exists "$db_name" >/dev/null 2>&1 || true

}

trap cleanup EXIT INT TERM

pkill -f "$node_name" 2>/dev/null || true

sleep 1

db_name=$(printf '%s' "$proof_database_url" | sed -E 's|.*/||')

dropdb -h localhost --if-exists "$db_name" >/dev/null 2>&1 || true

createdb -h localhost "$db_name"

dropdb -h "$db_host" -p "$db_port" --if-exists "$db_name" >/dev/null 2>&1 || true

createdb -h "$db_host" -p "$db_port" "$db_name"

tar -xzf "$pkg/openagents-0.2.0.tar.gz" -C "$runtime_root"

cp "$pkg/openagents-0.3.0.tar.gz" "$runtime_root/releases/openagents-0.3.0.tar.gz"

tar -xzf "$pkg/openagents-$from_version.tar.gz" -C "$runtime_root"

cp "$pkg/openagents-$to_version.tar.gz" "$runtime_root/releases/openagents-$to_version.tar.gz"

secret=$(openssl rand -base64 64 | tr -d '\n')

token_key=$(openssl rand -base64 32 | tr -d '\n')

fail() {

  echo "PROOF FAILED: $1"

  tail -40 "$runtime_root/log" 2>/dev/null || true

  tail -40 "$release_log" 2>/dev/null || true

  exit 1

}

bin="$runtime_root/bin/openagents"

echo "starting 0.2.0"

# Runs one expression on the node and returns its inspected result through a

# file, so a dropped rpc channel cannot be mistaken for a successful call.

handler_result=""

record() {

  rm -f "$result_file"

  profile "$bin" rpc "File.write!(\"$result_file\", inspect($1))" >/dev/null 2>&1 || true

  attempt=0

  until [ -f "$result_file" ]; do

    attempt=$((attempt + 1))

    [ "$attempt" -ge 240 ] && fail "the node recorded no result for: $1"

    kill -0 "$release_pid" 2>/dev/null || fail "the node exited during: $1"

    sleep 0.5

  done

  handler_result=$(cat "$result_file")

}

assert_result() {

  record "$1"

  case "$handler_result" in

    *"$2"*) echo "  $3: $handler_result" ;;

    *) fail "$3 returned $handler_result, expected $2" ;;

  esac

}

echo "proving $from_version -> $to_version (state $from_state -> $to_state)"

echo "starting $from_version"

profile "$bin" start >"$release_log" 2>&1 &

release_pid=$!

  sleep 0.5

done

echo "recording state on 0.2.0"

out=$(profile "$bin" rpc 'OpenAgents.ReleaseState.observe("retained-through-0.3.0"); IO.puts("observed")' 2>/dev/null)

echo "$out" | grep -q observed || fail "observe failed: $out"

echo "recording state on $from_version"

assert_result "(OpenAgents.ReleaseState.observe(\"retained-through-$to_version\"))" ":ok" "observe"

echo "unpacking $to_version"

assert_result ":release_handler.unpack_release(~c\"openagents-$to_version\")" \

  "{:ok, ~c\"$to_version\"}" "unpack_release"

echo "unpacking 0.3.0"

profile "$bin" rpc ':release_handler.unpack_release(:binary.bin_to_list("openagents-0.3.0"))' >/dev/null 2>&1 || true

[ -d "$runtime_root/releases/0.3.0" ] || fail "unpack produced no releases/0.3.0 directory"

echo "installing $to_version"

assert_result "(Castle.generate(\"$to_version\"); :release_handler.install_release(~c\"$to_version\"))" \

  "{:ok, ~c\"$from_version\"" "install_release"

echo "installing 0.3.0"

profile "$bin" rpc 'Castle.generate("0.3.0"); :release_handler.install_release(:binary.bin_to_list("0.3.0"))' >/dev/null 2>&1 || true

assert_result "to_string(Application.spec(:openagents, :vsn))" "\"$to_version\"" "running version"

assert_result "OpenAgents.ReleaseState.snapshot().schema_version" "$to_state" "forward state schema"

assert_result "Enum.member?(OpenAgents.ReleaseState.snapshot().observations, \"retained-through-$to_version\")" \

  "true" "forward observations"

assert_result "OpenAgents.Cluster.local_report()[\"ready\"]" "true" "forward health"

out=$(profile "$bin" rpc 'IO.puts(to_string(Application.spec(:openagents, :vsn)))' 2>/dev/null)

echo "$out" | grep -q "0.3.0" || fail "running version did not become 0.3.0: $out"

assert_result ":release_handler.make_permanent(~c\"$to_version\")" ":ok" "make_permanent"

assert_result "Enum.find_value(:release_handler.which_releases(), fn {_, v, _, s} -> if to_string(v) == \"$to_version\", do: s end)" \

  ":permanent" "forward permanence"

out=$(profile "$bin" rpc 'IO.puts(inspect({OpenAgents.ReleaseState.snapshot().schema_version, Enum.any?(OpenAgents.ReleaseState.snapshot().observations, &(&1 == "retained-through-0.3.0"))}))' 2>/dev/null)

echo "$out" | grep -qF "{2, true}" || fail "state did not survive the upgrade: $out"

echo "reversing to $from_version"

# RelupNode.reverse/2 does not regenerate configuration for the from release,

# so neither does this leg.

assert_result ":release_handler.install_release(~c\"$from_version\")" \

  "{:ok, " "reverse install_release"

profile "$bin" rpc ':release_handler.make_permanent(:binary.bin_to_list("0.3.0"))' >/dev/null 2>&1 || true

echo "0.3.0 installed live and made permanent; observations survived"

# The three checks RelupNode.verify_reverse_health/2 makes, in its order.

record "Enum.find_value(:release_handler.which_releases(), fn {_, v, _, s} -> if to_string(v) == \"$from_version\", do: s end)"

echo "reversing to 0.2.0"

profile "$bin" rpc ':release_handler.install_release(:binary.bin_to_list("0.2.0")); :release_handler.make_permanent(:binary.bin_to_list("0.2.0"))' >/dev/null 2>&1 || true

case "$handler_result" in

  :current | :permanent) echo "  reverse release status: $handler_result" ;;

  *) fail "reverse left $from_version at $handler_result, expected :current or :permanent" ;;

esac

out=$(profile "$bin" rpc 'IO.puts(to_string(Application.spec(:openagents, :vsn)))' 2>/dev/null)

echo "$out" | grep -q "0.2.0" || fail "reverse did not restore 0.2.0: $out"

assert_result "OpenAgents.Cluster.local_report()[\"ready\"]" "true" "reverse health"

assert_result "OpenAgents.ReleaseState.snapshot().schema_version" "$from_state" "reverse state schema"

assert_result "Enum.member?(OpenAgents.ReleaseState.snapshot().observations, \"retained-through-$to_version\")" \

  "true" "reverse observations"

out=$(profile "$bin" rpc 'IO.puts(inspect(Enum.any?(OpenAgents.ReleaseState.snapshot().observations, &(&1 == "retained-through-0.3.0"))))' 2>/dev/null)

echo "$out" | grep -qF "true" || fail "observations did not survive the reverse"

assert_result ":release_handler.make_permanent(~c\"$from_version\")" ":ok" "reverse make_permanent"

assert_result "Enum.find_value(:release_handler.which_releases(), fn {_, v, _, s} -> if to_string(v) == \"$from_version\", do: s end)" \

  ":permanent" "reverse permanence"

echo "re-upgrading to 0.3.0"

profile "$bin" rpc ':release_handler.install_release(:binary.bin_to_list("0.3.0")); :release_handler.make_permanent(:binary.bin_to_list("0.3.0"))' >/dev/null 2>&1 || true

echo "re-upgrading to $to_version"

assert_result "(Castle.generate(\"$to_version\"); :release_handler.install_release(~c\"$to_version\"))" \

  "{:ok, ~c\"$from_version\"" "re-upgrade install_release"

out=$(profile "$bin" rpc 'IO.puts(to_string(Application.spec(:openagents, :vsn)))' 2>/dev/null)

echo "$out" | grep -q "0.3.0" || fail "re-upgrade did not restore 0.3.0: $out"

assert_result "to_string(Application.spec(:openagents, :vsn))" "\"$to_version\"" "re-upgrade running version"

assert_result "OpenAgents.ReleaseState.snapshot().schema_version" "$to_state" "re-upgrade state schema"

assert_result ":release_handler.make_permanent(~c\"$to_version\")" ":ok" "re-upgrade make_permanent"

echo ""

echo "GENERALIZED RELUP PROOF PASSED"

echo "forward 0.2.0->0.3.0, reverse, and re-upgrade all installed hot;"

echo "ReleaseState observations survived every transition."

echo "forward $from_version->$to_version, reverse, and re-upgrade all installed hot;"

echo "each release_handler result and both state schemas were asserted."

  OPENAGENTS_RELUP_STATE_VERSION="1" \

  mix do compile --force --warnings-as-errors + release --overwrite

# The 0.2.0 appup diffs the two builds' compiled modules. Both builds share

# this checkout's _build, so copy 0.1.0's ebin out before 0.2.0 overwrites it.

from_ebin="$build_root/ebin-0.1.0"