# Repository storage

**Date:** 2026-08-21

**Commit measured:** `5116bfe5d8ef` (`openagents/main`, the forge)

**Commit measured:** `a82a5efe0bad` (`openagents/main`, the forge)

**Question:** How does this application store Git repositories today? Is the SCV's repository storage different from the way the main repository is stored and synchronized, and should it be? What should you build, in what order, to make the whole thing sound?

**Method:** direct reading of `lib/openagents/forge/` (`repos.ex`, `wal.ex`, `wal/gcs.ex`, `sync.ex`, `pushes.ex`, `git_http.ex`, `browse.ex`, `janitor.ex`, `mirror_watch.ex`, `supervisor.ex`, `boot_converge.ex`), `lib/openagents/repositories.ex` and `lib/openagents/repositories/`, `lib/openagents/scv/`, `lib/openagents/tools/repository.ex` and the repository tool family, `lib/openagents_web/router.ex`, `lib/openagents_web/plugs/forge_git_auth.ex`, `lib/openagents/runtime_config.ex`, `config/`, `Dockerfile`, `rel/`, `infra/staging/`, `ops/`, `docs/openagents-cli/`, `INVARIANTS.md` (REPOSITORY-001 and REPOSITORY-002), and `ops/ci/push-remote-check.sh`, plus an exhaustive search of `lib/` for every process that runs `git` or writes to disk. Every architectural claim cites a file and line current at the measured commit. Claims the repository cannot settle are collected in section 8 with the command that would settle them, rather than repeated as fact.

The owner-supplied assessment is right that imports run on live application nodes and that every node materializes a full cache. It is wrong or overstated on several checkable points, most importantly: the bundle upload does not need a third temporary copy, the import is a depth-1 snapshot rather than full history, `/var/lib/sarah-forge` is not the host path in this repository, and the `docker image prune` that "recovered 2.887 GB per node" was added on 2026-08-21 to a staging Terraform template that has never been applied. Section 4 itemizes all of it.

One failure mode outranks all of the disk findings, and it is not about disk. The GitHub mirror pushes with `--mirror`, a force push of every ref (`lib/openagents/forge/pushes.ex:240`), and the drift watcher that triggers it compares the two `main` values for equality with no ordering test, classifying a mirror that is *ahead* as lag and force-overwriting it (`lib/openagents/forge/mirror_watch.ex:73`, `:82`). It then reports "current" and records no incident. As of the measured commit, the forge and GitHub have diverged in exactly that direction. `INVARIANTS.md:1794` (REPOSITORY-002) now guards the cause by refusing pushes to any non-forge remote; the mechanism itself is unchanged. Section 3.1 traces it.

One failure mode outranks all of the disk findings, and it is not about disk. The GitHub mirror pushes with `--mirror`, a force push of every ref (`lib/openagents/forge/pushes.ex:240`), and the drift watcher that triggers it compares the two `main` values for equality with no ordering test, classifying a mirror that is *ahead* as lag and force-overwriting it (`lib/openagents/forge/mirror_watch.ex:73`, `:82`). It then reports "current" and records no incident. The mirror is off today, and `INVARIANTS.md:1794` (REPOSITORY-002) is explicit that turning it on overwrites whatever direct pushes left on GitHub. The risk is scheduled, not avoided. Section 3.1 traces it.

The first things to build are small, and none of them is sharding: teach the mirror to refuse an ahead mirror, move transient repository writes onto the durable volume, put a ceiling on the import retry loop, and check free space before an import starts instead of discovering `ENOSPC` from `git`'s stderr.

- **`--mirror` is a force push of every ref.** It updates non-fast-forward and deletes remote refs that the local bare repository does not have. Whatever is on the mirror is replaced by the forge's state, unconditionally.

- **The drift test cannot tell "behind" from "ahead".** `drift?/1` compares the two `refs/heads/main` values for equality and classifies every inequality as `:behind` (`lib/openagents/forge/mirror_watch.ex:73`). A mirror carrying a commit the forge has never seen is therefore treated as lag, and `check_repo/3` responds by calling `Pushes.mirror_now/1` immediately (`lib/openagents/forge/mirror_watch.ex:82`).

This is the enforcement context for `INVARIANTS.md:1794` (REPOSITORY-002), added at the measured commit: development pushes go to the forge, never to the mirror, because the forge is the authority and GitHub is a projection of it. `ops/ci/push-remote-check.sh` and `.githooks/pre-push` refuse a push to any non-forge remote. Section 3.1 covers what happens when a push reaches GitHub anyway.

This is the enforcement context for `INVARIANTS.md:1794` (REPOSITORY-002), added at the measured commit: development pushes go to the forge, never to the mirror, because the forge is the authority and GitHub is a projection of it. That invariant also records that the mirror is not running at all — `:forge_mirror_urls` is empty and set by no environment, so `MirrorWatch` reports `off` and GitHub holds only what was pushed to it directly. `ops/ci/push-remote-check.sh` and `.githooks/pre-push` refuse a push to any non-forge remote. Section 3.1 covers what happens when the mirror is switched on.

### 1.6 Paths E and F — working clones

Ranked by likelihood times blast radius, each traced to code.

### 3.1 The mirror silently force-overwrites a mirror that is ahead of the forge (conditional, but the largest blast radius here)

### 3.1 Turning the mirror on would destroy whatever the mirror holds (armed by one configuration value, largest blast radius here)

This is the only failure mode in this document that destroys data rather than consuming a resource, and as of the measured commit the forge and GitHub have genuinely diverged: GitHub carries a commit that was pushed directly to it and that the forge's WAL never recorded.

This is the only failure mode in this document that destroys data rather than consuming a resource. It is not firing today, and the reason it is not firing is the reason it is dangerous: the mirror has never been switched on, so nobody has met its behavior yet.

The mechanism is three lines. `drift?/1` compares the forge's `refs/heads/main` with the mirror's and classifies *any* inequality as `:behind`, with no ordering test (`lib/openagents/forge/mirror_watch.ex:71`–`:73`). `check_repo/3` responds to `:behind` by calling `Pushes.mirror_now/1` at once (`lib/openagents/forge/mirror_watch.ex:82`). `mirror_now/1` runs `git push --mirror` (`lib/openagents/forge/pushes.ex:240`), which force-updates every ref and deletes any the forge lacks. The GitHub-only commit is overwritten. The recheck on the next line then returns `:current`, so the watcher publishes "current" to the status page and records no incident (`lib/openagents/forge/mirror_watch.ex:84`–`:86`). The same tick that destroys the commit also reports that everything is fine.

The mechanism is three lines. `drift?/1` compares the forge's `refs/heads/main` with the mirror's and classifies *any* inequality as `:behind`, with no ordering test (`lib/openagents/forge/mirror_watch.ex:71`–`:73`). `check_repo/3` responds to `:behind` by calling `Pushes.mirror_now/1` at once (`lib/openagents/forge/mirror_watch.ex:82`). `mirror_now/1` runs `git push --mirror` (`lib/openagents/forge/pushes.ex:240`), which force-updates every ref and deletes any ref the forge lacks. A mirror that is *ahead* is therefore treated as lag and overwritten. The recheck on the next line then returns `:current`, so the watcher publishes "current" to the status page and records no incident (`lib/openagents/forge/mirror_watch.ex:84`–`:86`). The same tick that destroys the commit also reports that everything is fine.

Blast radius, stated honestly: any commit that reached the mirror but not the forge is lost from every surface this system controls. It was never in the WAL, so no replay recovers it; it is not in any bare repository, so no node holds it; there is no receipt, no incident, and no log line naming it. It survives only in GitHub's own retention of unreachable objects, recoverable by SHA by someone who knows to look, which is not a property this system provides.

Blast radius: any commit that reached the mirror but not the forge is lost from every surface this system controls. It was never in the WAL, so no replay recovers it; it is not in any bare repository, so no node holds it; there is no receipt, no incident, and no log line naming it. It survives only in GitHub's own retention of unreachable objects, recoverable by SHA by someone who knows to look, which is not a property this system provides.

Two honest qualifications. First, the whole path is gated on `:forge_mirror_urls`, which has no default (`lib/openagents/forge/pushes.ex:252`) and is set in no configuration file in this repository, so whether it is armed in a given environment is not determinable here — see section 8. Second, the divergence itself is now guarded at the source: `INVARIANTS.md:1794` and `ops/ci/push-remote-check.sh` refuse a push to any non-forge remote, which is a fix for the cause. It is not a fix for the mechanism, which will still overwrite an ahead mirror the next time one exists for any other reason, including a manual mirror push, a GitHub-side revert, or a restore.

The trigger is a single value. `mirror_url/1` reads `:forge_mirror_urls`, which is empty in `config/config.exs` and set by no environment, so `MirrorWatch` reports `off` and the whole path is inert (`lib/openagents/forge/pushes.ex:252`, `lib/openagents/forge/mirror_watch.ex:56`). `INVARIANTS.md:1794` (REPOSITORY-002) records this and is explicit that configuring the mirror "overwrites whatever direct pushes left on GitHub rather than merging with it". That is the correct disclosure, and it means the risk is scheduled rather than avoided: GitHub currently holds commits pushed to it directly, before the guard existed, and the act of completing the mirror contract is the act that deletes them.

The same invariant guards the cause going forward — `ops/ci/push-remote-check.sh` and `.githooks/pre-push` refuse a push to any non-forge remote. It does not guard the mechanism, which will still overwrite an ahead mirror whenever one exists for any other reason: the bounded manual-mirroring override the invariant itself permits, a GitHub-side revert, or a restore. Stage 1 makes turning the mirror on a safe operation instead of a destructive one.

### 3.2 A failing import becomes a self-amplifying disk consumer (high, node-wide)

**Costs.** One extra `git fetch` per watch tick per mirrored repository, one new incident code, and a decision about what an operator does with a diverged mirror. The push-lease change must keep the credential out of logs and out of the URL, which the current code is careful about (`lib/openagents/forge/pushes.ex:229`).

**Unblocks.** Trusting the mirror enough to leave it enabled. REPOSITORY-002 guards the common cause of divergence; this guards every other one, including manual mirroring during a forge outage, which the invariant explicitly permits as a bounded override (`INVARIANTS.md:1808`).

**Unblocks.** Switching the mirror on at all. Today that is a destructive act by design, so this stage is a prerequisite for completing the REPOSITORY-002 contract rather than an improvement to a running system. REPOSITORY-002 guards the common cause of divergence; this guards every other one, including manual mirroring during a forge outage, which the invariant explicitly permits as a bounded override (`INVARIANTS.md:1817`).

**Size.** Small. **Seams.** `lib/openagents/forge/mirror_watch.ex:68`–`:110`, `lib/openagents/forge/pushes.ex:232`–`:249`, `lib/openagents/incidents.ex`, `test/openagents/forge/` mirror coverage.

These could not be determined from the repository. Each is paired with the command that would settle it.

1. **Is the GitHub mirror armed in the running environment, and has it already overwritten anything?** `:forge_mirror_urls` has no default and appears in no configuration file here (`lib/openagents/forge/pushes.ex:253`), so section 3.1's failure mode is either live or inert depending on a value this repository does not carry. Settle by checking the deployed runtime configuration for that key, and by comparing GitHub's `main` with the forge's WAL head. If a commit reached GitHub and not the forge, recover it by SHA from GitHub before the next five-minute watch tick (`lib/openagents/forge/mirror_watch.ex:26`) rather than after.

1. **Which commits does GitHub hold that the forge's WAL never recorded, and are any of them wanted?** The mirror is off, so nothing has been overwritten yet, but section 3.1 shows that switching it on is what deletes them. Settle before configuring `:forge_mirror_urls`, by comparing `git ls-remote origin` against the forge's refs and inspecting anything GitHub has that the forge does not. This is a decision, not a lookup: either the extra commits get pushed to the forge first, or they get abandoned deliberately.

2. **What are the production nodes' actual disks?** `infra/` contains only `infra/staging/`, and no production Terraform has ever existed in this repository's history. Production is described only by assertion in `ops/production/preflight.sh:105`, which expects three instances matching `^sarah-fleet-` in project `openagentsgemini`. Settle with `gcloud compute instances describe <node> --project=<project> --zone=<zone> --format='json(disks)'`.

3. **Do the production nodes have a state disk at all, and is `/var/lib/openagents` a real mount there?** The bind-mount and the dedicated disk are declared only in the unapplied staging template. Settle with `findmnt -T /var/lib/openagents` and `docker inspect <container> --format '{{json .Mounts}}'` on a node.

4. **Where does Docker actually store its data in production?** Nothing in the repository configures a data root. Settle with `docker info --format '{{.DockerRootDir}}'` and `df -hT`.