evidence before production promotion.

Read the [Forge hot loop runbook](docs/operations/forge-hot-loop.md) for the

current deployment contract and production evidence.

current deployment contract and production evidence. Read the

[Forge cache recovery runbook](docs/operations/forge-cache-recovery.md) when a

repository read differs between fleet nodes or returns `503`.

## Architecture

- **Refs live in exactly one place.** The WAL is authority, PostgreSQL receipts are derived from it and reconciled by sequence number (`lib/openagents/forge/pushes.ex:143`), and the bare repositories declare themselves cache (`lib/openagents/forge/repos.ex:3`). Almost every hard bug in a distributed Git host comes from having two ref authorities. This system has one.

- **The acknowledgement barrier is in the right place.** A push is not acknowledged until it is durable, and a WAL rejection rolls local refs back (`lib/openagents/forge/pushes.ex:57`, `:82`).

- **Reads degrade instead of failing.** An unreachable WAL logs and serves the local cache (`lib/openagents/forge/sync.ex:30`), and a damaged cache serves what it has rather than turning a node-local fault into a public outage (`lib/openagents/forge/browse.ex:49`).

- **Reads fail closed when the cache is not authoritative.** A repository-cache synchronization failure returns `503`, marks the node unready, and preserves the last complete local projection for recovery. It never turns missing cache objects into an authoritative `404` (`lib/openagents/forge/sync.ex`, `lib/openagents/forge/browse.ex`).

- **Git invocations are argv-only.** No request-derived value ever enters a shell string, including in the one place `sh` is used for stdin redirection (`lib/openagents/forge/repos.ex:7`, `lib/openagents/forge/git_http.ex:381`).

---

# Forge cache recovery

Date: 2026-08-22

Status: Active operational procedure.

Use this runbook when the same forge blob, tree, or clone request differs

between fleet nodes or returns `503`. The WAL is the durable push authority.

Each node's bare Git repository is a disposable projection of that authority.

## Expected failure behavior

`OpenAgents.Forge.Sync` serializes synchronization and pushes per repository.

When it detects missing objects, it builds a complete sibling repository,

verifies every authoritative ref, and atomically activates that repository.

Readers never observe the sibling while it is incomplete.

If WAL replay or activation fails, the node:

- preserves the last complete local repository cache;

- returns `503` for the affected forge read instead of a false `404`;

- reports `forge_cache_ready: false` through the local cluster health report;

- returns `503` from `/health` so the load balancer can stop admitting traffic;

- logs `forge_sync_unavailable` with the repository, operation, and typed cause.

A later successful synchronization clears that repository's failure and

restores readiness without a process restart.

## Diagnose a divergent node

1. Request the same blob or Git ref advertisement directly from every fleet

   node. Record the node name, HTTP status, response size, and revision.

2. Compare the affected repository's applied WAL sequence on every node. A

   sequence of `0` or a sequence behind healthy peers identifies a stale local

   projection; it does not identify a missing durable push.

3. Search the node log for `forge_sync_cache_rebuild` and

   `forge_sync_unavailable`. Record the `operation`, `code`, and repository.

4. Inspect ownership and write permissions from the repository root through

   the failing object's two-character fan-out directory. Run the check as the

   same user that runs the release.

5. Verify that the WAL index and every referenced WAL object remain readable.

   Stop if the authority is unavailable. Do not infer repository absence from

   a cache failure.

The 2026-08-22 incident followed this pattern. One of three production nodes

had a root-owned Git object fan-out directory under a cache otherwise owned by

the release user. WAL replay failed with `EACCES` on that node. Requests through

the load balancer alternated between successful responses from two nodes and

false `404` responses from the damaged node.

## Recover one node

1. Keep the node out of load-balancer admission while

   `forge_cache_ready` is `false`.

2. Confirm that the WAL is readable and that another node can materialize the

   same repository and authoritative refs.

3. Correct the state-directory ownership so the release user can create Git

   object fan-out directories and files. Apply the correction only to the

   affected state path.

4. Move the affected bare repository cache aside. Do not delete or modify WAL

   indexes or WAL objects. Keep the moved cache until verification finishes.

5. Trigger repository synchronization or restart the application so boot

   convergence replays the WAL into a new local cache.

6. Verify the applied WAL sequence, every authoritative ref, the reported blob,

   Git ref advertisement, and `/health` on that node.

7. Restore load-balancer admission only after `forge_cache_ready` and the

   aggregate `ready` value are both `true`.

8. Remove the moved cache after the node passes verification.

## Prevent recurrence

- Create and mount the forge state directory with the release user's numeric

  UID and GID before the application starts.

- Never run cache repair, import, or Git maintenance commands as a different

  user against the live state directory.

- Alert when `forge_cache_ready` becomes `false` or when

  `forge_sync_unavailable` appears.

- Include a multi-node blob and clone comparison in staging resilience tests.

- Treat a forge `503` as retryable. Treat `404` only as an authoritative

  repository, ref, tree, or blob absence after synchronization succeeds.

## Focused verification

Run the cache, Git HTTP, and health contracts together:

```sh

mix test test/openagents/forge/sync_test.exs \

  test/openagents/forge/git_http_test.exs \

  test/openagents_web/controllers/health_controller_test.exs

```

        OpenAgentsWeb.Telemetry,

        OpenAgents.Repo,

        OpenAgents.ReleaseState,

        OpenAgents.Forge.CacheReadiness,

        # Deployment identity and boot convergence must settle before cluster

        # discovery or the endpoint can make this node externally reachable.

        OpenAgents.Forge.DeploymentNode,

    boot_ready? = OpenAgents.Forge.BootConverge.ready?()

    deployment = OpenAgents.Forge.DeploymentNode.health()

    admission_ready? = OpenAgents.Cluster.Admission.ready?()

    forge_cache_ready? = OpenAgents.Forge.CacheReadiness.ready?()

    %{

      "schema" => "openagents.cluster_health.v1",

      "boot_converged" => boot_ready?,

      "deployment_ready" => deployment["participant_ready"],

      "admission_ready" => admission_ready?,

      "forge_cache_ready" => forge_cache_ready?,

      "uptime_ms" => uptime_ms(),

      "live" => true,

      "ready" => boot_ready? and deployment["ready"] == true and admission_ready?

      "ready" =>

        boot_ready? and deployment["ready"] == true and admission_ready? and forge_cache_ready?

    }

  end

  never stream an unbounded `git show` to an anonymous socket.

  """

  require Logger

  alias OpenAgents.Forge.{Repos, Sync}

  alias OpenAgents.Repositories.Repository

    end

  end

  # A node whose bare-repo cache is damaged (the 2026-08-19 incident, #134:

  # a torn loose object after a VM reset) makes WAL replay raise. That is a

  # node-local cache fault, not a reason to fail a public read: serve what

  # this node already has, and let the operator repair path fix the cache.

  defp freshen(repo) do

    Sync.ensure_fresh(storage_key(repo), default_branch(repo))

  rescue

    error ->

      Logger.warning(

        "forge_browse_sync_failed repo=#{storage_key(repo)} code=#{OpenAgents.OperationalLog.code(error)}"

      )

      :ok

  catch

    :exit, reason ->

      Logger.warning(

        "forge_browse_sync_exited repo=#{storage_key(repo)} code=#{OpenAgents.OperationalLog.code(reason)}"

      )

      :ok

    Sync.ensure_fresh!(storage_key(repo), default_branch(repo))

  end

  @doc "The default branch head, if the repo has one."

defmodule OpenAgents.Forge.CacheReadiness do

  @moduledoc """

  Tracks repository caches that this node cannot materialize from the WAL.

  A successful synchronization clears the repository's failure. Readiness

  returns only when no observed cache failure remains.

  """

  use GenServer

  def start_link(options \\ []) do

    GenServer.start_link(__MODULE__, %{}, Keyword.put_new(options, :name, __MODULE__))

  end

  def mark_unavailable(repo, code) do

    GenServer.call(__MODULE__, {:mark_unavailable, repo, code})

  end

  def mark_available(repo) do

    GenServer.call(__MODULE__, {:mark_available, repo})

  end

  def ready? do

    GenServer.call(__MODULE__, :ready?)

  end

  def report do

    GenServer.call(__MODULE__, :report)

  end

  @doc false

  def reset do

    GenServer.call(__MODULE__, :reset)

  end

  @impl true

  def init(failures), do: {:ok, failures}

  @impl true

  def handle_call({:mark_unavailable, repo, code}, _from, failures) do

    {:reply, :ok, Map.put(failures, repo, code)}

  end

  def handle_call({:mark_available, repo}, _from, failures),

    do: {:reply, :ok, Map.delete(failures, repo)}

  def handle_call(:ready?, _from, failures),

    do: {:reply, map_size(failures) == 0, failures}

  def handle_call(:report, _from, failures),

    do: {:reply, %{"ready" => map_size(failures) == 0, "failures" => failures}, failures}

  def handle_call(:reset, _from, _failures), do: {:reply, :ok, %{}}

end

    operation = if service == "git-upload-pack", do: :read, else: :write

    with {:ok, repository} <- resolve_repository(conn, owner, name),

         :ok <- authorize(conn, repository, operation) do

      Sync.ensure_fresh(repository.storage_key, repository.default_branch)

         :ok <- authorize(conn, repository, operation),

         :ok <- Sync.ensure_fresh(repository.storage_key, repository.default_branch) do

      command = String.trim_leading(service, "git-")

      path = Repos.ensure_repo!(repository.storage_key, repository.default_branch)

  defp upload_pack(conn, owner, name) do

    with {:ok, repository} <- resolve_repository(conn, owner, name),

         :ok <- authorize(conn, repository, :read),

         {:ok, body, conn} <- read_git_body(conn) do

      Sync.ensure_fresh(repository.storage_key, repository.default_branch)

         {:ok, body, conn} <- read_git_body(conn),

         :ok <- Sync.ensure_fresh(repository.storage_key, repository.default_branch) do

      path = Repos.ensure_repo!(repository.storage_key, repository.default_branch)

      {output, _status} = run_git_service("upload-pack", [path], body, git_protocol(conn))

        {:error, :wal_persist_failed} ->

          send_resp(conn, 503, "push not persisted; refs rolled back — retry") |> halt()

        {:error, %OpenAgents.Forge.SyncError{}} ->

          send_resp(conn, 503, "repository cache unavailable; retry") |> halt()

        {:error, _reason} ->

          send_resp(conn, 500, "push failed") |> halt()

      end

    conn |> send_resp(status, message) |> halt()

  end

  defp send_git_error(conn, {:error, %OpenAgents.Forge.SyncError{}}) do

    conn |> send_resp(503, "repository cache unavailable; retry") |> halt()

  end

  defp send_git_error(conn, {:error, status, message, headers}) do

    conn =

      Enum.reduce(headers, conn, fn {name, value}, acc -> put_resp_header(acc, name, value) end)

  back and the client sees a failed push; the cache never gets ahead of the

  authority.

  Pushes serialize per repo cluster-wide via `:global.trans`; the WAL index

  CAS remains the true serialization point (a conflict from a writer outside

  the cluster is re-synced and retried once).

  Pushes and local cache synchronization share one per-repository lock on each

  node. The WAL index CAS remains the cluster-wide serialization point (a

  conflict from another writer is re-synced and retried once).

  After the WAL accepts: a `forge_pushes` receipt row is derived (idempotent

  by WAL sequence — audit A7: receipts are derived from the WAL, never a

  only after WAL persist; `{:error, :wal_persist_failed}` after rollback.

  """

  def handle_receive_pack(repo, body, principal, git_protocol) do

    :global.trans({{:forge_push, repo}, self()}, fn ->

    Sync.with_repo_lock(repo, fn ->

      do_handle(repo, body, principal, git_protocol, false)

    end)

  end

  defp do_handle(repo, body, principal, git_protocol, retried?) do

    started_at = System.monotonic_time(:millisecond)

    Sync.ensure_fresh(repo)

    path = Repos.ensure_repo!(repo)

    refs_before = Repos.refs(repo)

    {output, status} = GitHTTP.run_git_service("receive-pack", [path], body, git_protocol)

    refs_after = Repos.refs(repo)

    cond do

      status != 0 ->

        {:error, :receive_pack_failed}

      refs_after == refs_before ->

        # Nothing changed (up to date, or all commands rejected by git);

        # the client's report-status in `output` says why. Nothing to persist.

        {:ok, output}

      true ->

        case persist(repo, body, refs_after, principal) do

          {:ok, seq} ->

            Repos.record_applied_seq!(repo, seq)

            record_repository_activity(repo)

            capture_push_received(

              repo,

              record_receipt(repo, seq, refs_before, refs_after, principal, started_at),

              refs_before,

              refs_after,

              started_at

            )

            broadcast(repo, seq, refs_after)

            mirror_async(repo)

            {:ok, output}

          {:error, :cas_conflict} when not retried? ->

            Sync.ensure_fresh(repo)

            do_handle(repo, body, principal, git_protocol, true)

          {:error, reason} ->

            Logger.error(

              "forge_push_wal_failed repo=#{repo} code=#{OpenAgents.OperationalLog.code(reason)}"

            )

            Repos.set_refs!(repo, refs_before)

            {:error, :wal_persist_failed}

        end

    with :ok <- Sync.ensure_fresh(repo) do

      path = Repos.ensure_repo!(repo)

      refs_before = Repos.refs(repo)

      {output, status} = GitHTTP.run_git_service("receive-pack", [path], body, git_protocol)

      refs_after = Repos.refs(repo)

      cond do

        status != 0 ->

          {:error, :receive_pack_failed}

        refs_after == refs_before ->

          # Nothing changed (up to date, or all commands rejected by git);

          # the client's report-status in `output` says why. Nothing to persist.

          {:ok, output}

        true ->

          case persist(repo, body, refs_after, principal) do

            {:ok, seq} ->

              Repos.record_applied_seq!(repo, seq)

              record_repository_activity(repo)

              capture_push_received(

                repo,

                record_receipt(repo, seq, refs_before, refs_after, principal, started_at),

                refs_before,

                refs_after,

                started_at

              )

              broadcast(repo, seq, refs_after)

              mirror_async(repo)

              {:ok, output}

            {:error, :cas_conflict} when not retried? ->

              with :ok <- Sync.ensure_fresh(repo) do

                do_handle(repo, body, principal, git_protocol, true)

              end

            {:error, reason} ->

              Logger.error(

                "forge_push_wal_failed repo=#{repo} code=#{OpenAgents.OperationalLog.code(reason)}"

              )

              Repos.set_refs!(repo, refs_before)

              {:error, :wal_persist_failed}

          end

      end

    end

  end

  @doc "Initialize the bare repository if absent. Returns the path."

  def ensure_repo!(repo, default_branch \\ "main") do

    path = bare_path(repo)

    repo |> bare_path() |> ensure_repo_at!(default_branch)

  end

  @doc false

  def ensure_repo_at!(path, default_branch \\ "main") do

    unless File.exists?(Path.join(path, "HEAD")) do

      File.mkdir_p!(path)

      {_, 0} = git(path, ["init", "--bare", "--initial-branch=#{default_branch}", path])

    end

    set_default_branch!(repo, default_branch)

    set_default_branch_at!(path, default_branch)

    path

  end

  def set_default_branch!(repo, default_branch) do

    path = bare_path(repo)

    repo |> bare_path() |> set_default_branch_at!(default_branch)

  end

  @doc false

  def set_default_branch_at!(path, default_branch) do

    {_, 0} = git(path, ["symbolic-ref", "HEAD", "refs/heads/#{default_branch}"])

    :ok

  end

  @doc "Current refs of the bare repo as a `%{name => sha}` map."

  def refs(repo) do

    path = bare_path(repo)

    repo |> bare_path() |> refs_at()

  end

  @doc false

  def refs_at(path) do

    case git(path, ["for-each-ref", "--format=%(objectname) %(refname)"]) do

      {output, 0} ->

        output

  in the target.

  """

  def set_refs!(repo, target_refs) when is_map(target_refs) do

    path = bare_path(repo)

    current = refs(repo)

    repo |> bare_path() |> set_refs_at!(target_refs)

  end

  @doc false

  def set_refs_at!(path, target_refs) when is_map(target_refs) do

    current = refs_at(path)

    Enum.each(current, fn {name, _sha} ->

      unless Map.has_key?(target_refs, name) do

  @doc "The WAL sequence this bare repo has applied (cache freshness marker)."

  def applied_seq(repo) do

    case File.read(Path.join(bare_path(repo), "openagents-wal-seq")) do

    repo |> bare_path() |> applied_seq_at()

  end

  @doc false

  def applied_seq_at(path) do

    case File.read(Path.join(path, "openagents-wal-seq")) do

      {:ok, contents} ->

        case Integer.parse(String.trim(contents)) do

          {seq, _} -> seq

  end

  def record_applied_seq!(repo, seq) when is_integer(seq) do

    File.write!(Path.join(bare_path(repo), "openagents-wal-seq"), Integer.to_string(seq))

    repo |> bare_path() |> record_applied_seq_at!(seq)

  end

  @doc false

  def record_applied_seq_at!(path, seq) when is_integer(seq) do

    File.write!(Path.join(path, "openagents-wal-seq"), Integer.to_string(seq))

  end

  @doc "Run git with `--git-dir` pinned to the bare repo. Returns {output, status}."

  require Logger

  alias OpenAgents.Cluster

  alias OpenAgents.Forge.{GitHTTP, Repos, WAL}

  alias OpenAgents.Forge.{CacheReadiness, GitHTTP, Repos, SyncError, WAL}

  @default_cluster_warm_timeout_ms 10 * 60 * 1_000

  @doc """

  Bring the local bare repo up to the WAL. Returns `:ok` (fresh, replayed,

  or nothing pushed yet) — read paths degrade to serving the local cache if

  the WAL is unreachable, logging honestly, rather than failing reads.

  Bring the local bare repo up to the WAL.

  Returns `:ok` when the projection is current and a typed error when the WAL

  or cache cannot produce an authoritative projection. Callers must not turn a

  synchronization error into a repository or object `404`.

  """

  def ensure_fresh(repo, default_branch \\ "main") do

    case WAL.read_index(repo) do

      {:error, :not_found} ->

        :ok

      {:ok, _generation, index} ->

        replay_missing(repo, index, default_branch)

      {:error, reason} ->

        Logger.warning(

          "forge_sync_wal_unreachable repo=#{repo} code=#{OpenAgents.OperationalLog.code(reason)}"

        )

    synchronize(repo, fn ->

      case WAL.read_index(repo) do

        {:error, :not_found} -> :ok

        {:ok, _generation, index} -> do_replay_missing(repo, index, default_branch)

        {:error, reason} -> raise_sync(repo, :read_wal, reason)

      end

    end)

  end

        :ok

  @doc "Bring the local bare repo up to the WAL or raise a `503`-typed error."

  def ensure_fresh!(repo, default_branch \\ "main") do

    case ensure_fresh(repo, default_branch) do

      :ok -> :ok

      {:error, %SyncError{} = error} -> raise error

    end

  end

  @doc "Replay WAL entries the local repo has not applied. Used by reads and boot."

  def replay_missing(repo, index, default_branch \\ "main") do

    Repos.ensure_repo!(repo, default_branch)

    applied = Repos.applied_seq(repo)

    synchronize(repo, fn -> do_replay_missing(repo, index, default_branch) end)

  end

  @doc false

  def with_repo_lock(repo, function) when is_function(function, 0) do

    lock_id = {{__MODULE__, repo}, self()}

    case :global.trans(lock_id, function, [node()]) do

      {:aborted, reason} -> raise_sync(repo, :acquire_lock, reason)

      result -> result

    end

  end

  defp do_replay_missing(repo, index, default_branch) do

    path = Repos.ensure_repo!(repo, default_branch)

    applied = Repos.applied_seq_at(path)

    index

    |> WAL.entries()

    |> Enum.filter(fn entry -> entry["seq"] > applied end)

    |> Enum.each(fn entry -> apply_entry!(repo, entry) end)

    |> Enum.each(fn entry -> apply_entry!(repo, path, entry) end)

    rebuild_if_objects_missing!(repo, index, default_branch)

    converge_refs(repo, index)

    Repos.set_default_branch!(repo, default_branch)

    path = Repos.bare_path(repo)

    converge_refs(path, index)

    Repos.set_default_branch_at!(path, default_branch)

    :ok

  end

  defp rebuild_if_objects_missing!(repo, index, default_branch) do

    unless refs_materialized?(repo, index) do

      Logger.warning("forge_sync_cache_rebuild repo=#{repo} code=missing_ref_object")

      :ok = Repos.delete_repo(repo)

      Repos.ensure_repo!(repo, default_branch)

      rebuild_at_sibling!(repo, index, default_branch)

    end

  end

  defp rebuild_at_sibling!(repo, index, default_branch) do

    live_path = Repos.bare_path(repo)

    suffix = System.unique_integer([:positive, :monotonic])

    rebuild_path = live_path <> ".rebuild-#{suffix}"

    previous_path = live_path <> ".previous-#{suffix}"

    try do

      Repos.ensure_repo_at!(rebuild_path, default_branch)

      index

      |> WAL.entries()

      |> Enum.each(fn entry -> apply_entry!(repo, entry) end)

      |> Enum.each(fn entry -> apply_entry!(repo, rebuild_path, entry) end)

      converge_refs(rebuild_path, index)

      Repos.set_default_branch_at!(rebuild_path, default_branch)

      unless refs_materialized?(repo, index) do

        raise "forge cache rebuild did not materialize every authoritative ref"

      unless refs_materialized_at?(rebuild_path, index) do

        raise_sync(repo, :verify_rebuild, :missing_ref_object)

      end

      swap_rebuild!(repo, live_path, rebuild_path, previous_path)

    after

      File.rm_rf(rebuild_path)

      # Keep the previous cache if activation and restoration both fail. It is

      # the last complete local projection an operator can recover.

      if File.exists?(live_path), do: File.rm_rf(previous_path)

    end

  end

  defp swap_rebuild!(repo, live_path, rebuild_path, previous_path) do

    live_exists? = File.exists?(live_path)

    if live_exists? do

      case File.rename(live_path, previous_path) do

        :ok -> :ok

        {:error, reason} -> raise_sync(repo, :stage_previous_cache, reason)

      end

    end

    case File.rename(rebuild_path, live_path) do

      :ok ->

        :ok

      {:error, reason} ->

        if live_exists?, do: File.rename(previous_path, live_path)

        raise_sync(repo, :activate_rebuild, reason)

    end

  end

  defp refs_materialized?(repo, index) do

    path = Repos.bare_path(repo)

    refs_materialized_at?(Repos.bare_path(repo), index)

  end

  defp refs_materialized_at?(path, index) do

    index

    |> WAL.refs()

    |> Map.values()

    end)

  end

  defp apply_entry!(repo, %{"seq" => seq, "object" => object} = entry) do

    path = Repos.bare_path(repo)

  defp apply_entry!(repo, path, %{"seq" => seq, "object" => object} = entry) do

    case entry["format"] || "receive_pack" do

      "receive_pack" ->

        {:ok, payload} = WAL.get_entry(repo, object)

        :ok

    end

    Repos.record_applied_seq!(repo, seq)

    Repos.record_applied_seq_at!(path, seq)

  end

  # Replay is exact in the common case; converge_refs makes the final state

  # authoritative even if an individual replayed request was non-idempotent

  # (e.g. a non-fast-forward the original push forced).

  defp converge_refs(repo, index) do

  defp converge_refs(path, index) do

    target = WAL.refs(index)

    if Repos.refs(repo) != target do

      Repos.set_refs!(repo, target)

    if Repos.refs_at(path) != target do

      Repos.set_refs_at!(path, target)

    end

  end

  catch

    :exit, reason -> {:error, reason}

  end

  defp synchronize(repo, function) do

    result = with_repo_lock(repo, function)

    CacheReadiness.mark_available(repo)

    result

  rescue

    error ->

      sync_error = normalize_error(repo, error)

      CacheReadiness.mark_unavailable(repo, sync_error.operation)

      Logger.error(

        "forge_sync_unavailable repo=#{repo} operation=#{sync_error.operation} " <>

          "code=#{OpenAgents.OperationalLog.code(sync_error.reason)} detail=#{inspect(sync_error.reason)}"

      )

      {:error, sync_error}

  catch

    kind, reason ->

      sync_error = %SyncError{repo: repo, operation: :materialize_cache, reason: {kind, reason}}

      CacheReadiness.mark_unavailable(repo, sync_error.operation)

      {:error, sync_error}

  end

  defp normalize_error(_repo, %SyncError{} = error), do: error

  defp normalize_error(repo, error) do

    %SyncError{repo: repo, operation: :materialize_cache, reason: error}

  end

  defp raise_sync(repo, operation, reason) do

    raise SyncError, repo: repo, operation: operation, reason: reason

  end

end

defmodule OpenAgents.Forge.SyncError do

  @moduledoc "A typed repository-cache synchronization failure."

  defexception [:repo, :operation, :reason, plug_status: 503]

  @impl true

  def message(%__MODULE__{repo: repo, operation: operation}) do

    "Forge repository #{repo} is temporarily unavailable during #{operation}"

  end

end

          reason: "runtime_not_ready",

          boot_converged: report["boot_converged"],

          deployment_ready: report["deployment_ready"],

          admission_ready: report["admission_ready"]

          admission_ready: report["admission_ready"],

          forge_cache_ready: report["forge_cache_ready"]

        })

      {{:error, _reason}, _report} ->

  import OpenAgents.AccountsFixtures

  alias OpenAgents.{AuditEvent, Forge, Machines, Repo, Repositories}

  alias OpenAgents.Forge.{Repos, WAL}

  alias OpenAgents.Forge.{CacheReadiness, Repos, WAL}

  defmodule TestPipeline do

    @moduledoc false

    previous_wal = Application.get_env(:openagents, :forge_wal_dir)

    Application.put_env(:openagents, :forge_data_dir, Path.join(base, "data"))

    Application.put_env(:openagents, :forge_wal_dir, Path.join(base, "wal"))

    CacheReadiness.reset()

    user = repository_user_fixture("git-http-owner")

    on_exit(fn ->

      Application.put_env(:openagents, :forge_data_dir, previous_data)

      Application.put_env(:openagents, :forge_wal_dir, previous_wal)

      CacheReadiness.reset()

      File.rm_rf(base)

    end)

    assert Repos.refs(repository.storage_key) == refs_before

  end

  test "an unavailable cache returns 503 instead of a false repository 404", %{

    repository: repository,

    token: token

  } do

    entry = %{

      "seq" => 0,

      "object" => "entries/missing",

      "format" => "git_bundle",

      "refs" => %{"refs/heads/main" => String.duplicate("a", 40)},

      "principal" => "test",

      "pushed_at" => DateTime.to_iso8601(DateTime.utc_now())

    }

    {:ok, _generation} =

      WAL.cas_index(

        repository.storage_key,

        :none,

        WAL.append_entry(WAL.new_index(), entry)

      )

    authorization = "Basic " <> Base.encode64("x:#{token}")

    response =

      :get

      |> Plug.Test.conn("/git-http-owner/demo.git/info/refs?service=git-upload-pack")

      |> Plug.Conn.put_req_header("authorization", authorization)

      |> TestPipeline.call([])

    assert response.status == 503

    assert response.resp_body == "repository cache unavailable; retry"

    refute CacheReadiness.ready?()

  end

  test "unauthenticated and wrong-token pushes are refused", %{

    base: base,

    port: port,

defmodule OpenAgents.Forge.SyncTest do

  use ExUnit.Case, async: false

  alias OpenAgents.Forge.{Browse, Repos, Sync, WAL}

  alias OpenAgents.Forge.{Browse, CacheReadiness, Repos, Sync, SyncError, WAL}

  alias OpenAgents.Repositories.Repository

  setup do

    Application.put_env(:openagents, :forge_data_dir, Path.join(root, "data"))

    Application.put_env(:openagents, :forge_wal_dir, Path.join(root, "wal"))

    Application.put_env(:openagents, :forge_wal_adapter, OpenAgents.Forge.WAL.Local)

    CacheReadiness.reset()

    on_exit(fn ->

      restore_env(:forge_data_dir, previous_data)

      restore_env(:forge_wal_dir, previous_wal)

      restore_env(:forge_wal_adapter, previous_adapter)

      CacheReadiness.reset()

      File.rm_rf!(root)

    end)

    assert Repos.refs("empty-authority") == %{}

  end

  test "a failed sibling rebuild preserves the last complete cache and fails with 503", %{

    root: root

  } do

    {valid_index, sha} = put_bundle_entry!(root, "unavailable-cache", "trunk")

    assert :ok = Sync.ensure_fresh("unavailable-cache", "trunk")

    bare_path = Repos.bare_path("unavailable-cache")

    Repos.record_applied_seq_at!(bare_path, 1)

    missing_entry = %{

      "seq" => 1,

      "object" => "entries/00000001-000000000000",

      "format" => "git_bundle",

      "refs" => %{"refs/heads/trunk" => String.duplicate("a", 40)},

      "principal" => "test",

      "pushed_at" => DateTime.to_iso8601(DateTime.utc_now())

    }

    {:ok, generation, _index} = WAL.read_index("unavailable-cache")

    {:ok, next_generation} =

      WAL.cas_index("unavailable-cache", generation, WAL.append_entry(valid_index, missing_entry))

    assert {:error, %SyncError{operation: :materialize_cache, plug_status: 503} = error} =

             Sync.ensure_fresh("unavailable-cache", "trunk")

    assert Plug.Exception.status(error) == 503

    refute CacheReadiness.ready?()

    assert CacheReadiness.report()["failures"] == %{"unavailable-cache" => :materialize_cache}

    # The rebuild failed before activation, so readers never receive a partial

    # repository and the last complete cache remains available for recovery.

    assert String.trim(git_bare!(bare_path, ["show", "trunk:README.md"])) == "durable import"

    assert Repos.refs("unavailable-cache") == %{"refs/heads/trunk" => sha}

    assert Path.wildcard(bare_path <> ".rebuild-*") == []

    assert Path.wildcard(bare_path <> ".previous-*") == []

    repository = %Repository{storage_key: "unavailable-cache", default_branch: "trunk"}

    assert_raise SyncError, fn -> Browse.blob(repository, "trunk", "README.md") end

    # Once the authority becomes materializable again, the next successful

    # synchronization restores this node's readiness without a restart.

    assert {:ok, _generation} =

             WAL.cas_index("unavailable-cache", next_generation, valid_index)

    assert :ok = Sync.ensure_fresh("unavailable-cache", "trunk")

    assert CacheReadiness.ready?()

  end

  test "concurrent readers serialize one repository cache replay", %{root: root} do

    {_index, sha} = put_bundle_entry!(root, "serialized-cache", "trunk")

    results =

      1..12

      |> Task.async_stream(

        fn _reader -> Sync.ensure_fresh("serialized-cache", "trunk") end,

        max_concurrency: 12,

        timeout: 10_000

      )

      |> Enum.to_list()

    assert Enum.all?(results, &(&1 == {:ok, :ok}))

    assert Repos.refs("serialized-cache") == %{"refs/heads/trunk" => sha}

    assert CacheReadiness.ready?()

  end

  test "cluster warming materializes the local cache and every connected peer" do

    test_process = self()

    peer = :"peer@127.0.0.1"

    output

  end

  defp put_bundle_entry!(root, repository, branch) do

    source = Path.join(root, "#{repository}-source")

    File.mkdir_p!(source)

    git!(source, ["init", "--initial-branch=#{branch}"])

    git!(source, ["config", "user.email", "test@example.com"])

    git!(source, ["config", "user.name", "Forge test"])

    File.write!(Path.join(source, "README.md"), "durable import\n")

    git!(source, ["add", "README.md"])

    git!(source, ["commit", "-m", "Imported commit"])

    sha = source |> git!(["rev-parse", "HEAD"]) |> String.trim()

    bundle = Path.join(root, "#{repository}.bundle")

    git!(source, ["bundle", "create", bundle, "--all"])

    refs = %{"refs/heads/#{branch}" => sha}

    {:ok, object} = WAL.put_entry_file(repository, 0, bundle)

    entry = %{

      "seq" => 0,

      "object" => object,

      "format" => "git_bundle",

      "refs" => refs,

      "principal" => "test",

      "pushed_at" => DateTime.to_iso8601(DateTime.utc_now())

    }

    index = WAL.append_entry(WAL.new_index(), entry)

    {:ok, _generation} = WAL.cas_index(repository, :none, index)

    {index, sha}

  end

  defp restore_env(key, nil), do: Application.delete_env(:openagents, key)

  defp restore_env(key, value), do: Application.put_env(:openagents, key, value)

end

defmodule OpenAgentsWeb.HealthControllerTest do

  use OpenAgentsWeb.ConnCase

  setup do

    OpenAgents.Forge.CacheReadiness.reset()

    on_exit(&OpenAgents.Forge.CacheReadiness.reset/0)

    :ok

  end

  test "reports healthy when PostgreSQL is reachable", %{conn: conn} do

    conn = get(conn, ~p"/health")

             "reason" => "runtime_not_ready",

             "boot_converged" => false,

             "deployment_ready" => true,

             "admission_ready" => true

             "admission_ready" => true,

             "forge_cache_ready" => true

           }

  end

             "reason" => "runtime_not_ready",

             "boot_converged" => true,

             "deployment_ready" => true,

             "admission_ready" => false

             "admission_ready" => false,

             "forge_cache_ready" => true

           }

  end

  test "refuses readiness after a repository cache cannot materialize", %{conn: conn} do

    :ok = OpenAgents.Forge.CacheReadiness.mark_unavailable("broken-cache", :materialize_cache)

    conn = get(conn, ~p"/healthz")

    assert json_response(conn, 503) == %{

             "status" => "unavailable",

             "reason" => "runtime_not_ready",

             "boot_converged" => true,

             "deployment_ready" => true,

             "admission_ready" => true,

             "forge_cache_ready" => false

           }

  end

end