Complete Forge loop activation

3479f124960b · AtlantisPleb · · parent 4bf3f0c9f2f6

Complete Forge loop activation

Deploy story

What this commit did to the running system — joined from the forge receipt chain, the part a commit page elsewhere cannot show.

built
111 modules in 99.7 s
deployed
live · 111 modules on 3 nodes · push→live —
deployed
needs_rolling_replace · 111 modules on 0 nodes · push→live —

Changed files

  • modified config/config.exs
  • modified docs/operations/forge-hot-loop.md
  • modified docs/runtime-configuration.md
  • modified infra/staging/templates/fleet-startup.sh.tftpl
  • modified lib/openagents/forge/mirror_watch.ex
  • modified test/openagents/forge/mirror_watch_test.exs

Diff

6 files changed, +59 -14

config/config.exs modified +1

@@ -252,6 +252,7 @@ config :openagents,

252 252
  forge_hot_load_allowlist: [
253 253
    "OpenAgentsWeb.",
254 254
    "OpenAgents.Forge.Browse",
255
    "OpenAgents.Forge.MirrorWatch",
255 256
    "OpenAgents.Changelog",
256 257
    "OpenAgents.Scratch.",
257 258
    "OpenAgents.BuildInfo"
docs/operations/forge-hot-loop.md modified +42 -8

@@ -8,8 +8,8 @@ for allowlisted code changes. Relup and rolling replacement remain fallbacks.

8 8
This is the operator procedure for the fast deployment lane: push to the owned
9 9
forge, promote, and watch a code-only change go live across the fleet in
10 10
seconds without an image build. It also describes the production relup path,
11
the independent GitHub mirror repair worker, and the remaining direct-load
12
enablement work.
11
the independent GitHub mirror repair worker, and the production activation
12
evidence.
13 13
14 14
## Verified state of the lanes
15 15

@@ -18,7 +18,7 @@ enablement work.

18 18
| Direct BEAM transaction on production | Works | `fa4b792` loaded across three nodes via the transaction protocol; `live` target and deployment receipt recorded; uptimes unbroken |
19 19
| Automated push → promote → build → hot-load loop | Active | The web role runs `Builder`, `HotLoader`, and `Janitor`; every fleet node runs the pinned builder sidecar; `/status` reports the lane as **Active** and exposes target, build, deploy, and timing receipts |
20 20
| General relup lane | Active in production | `RelupPackage` binds source and target revisions, release versions, state schemas, target system, and artifact digests before `RelupDeployment` upgrades one node at a time. Production upgraded `0.2.0@81e4c25` to `0.2.1@9763bf7` in 48.838 seconds without restarting the BEAM. |
21
| Forge-to-GitHub mirror | Active in production | The production mirror uses a write-enabled deploy key. `MirrorWatch` runs independently of the deploy lane, repairs drift every five minutes, and reports freshness. Forge and GitHub exposed 10 identical refs after the production drill. |
21
| Forge-to-GitHub mirror | Active in production | The production mirror uses a write-enabled deploy key. `MirrorWatch` checks immediately at process startup, repairs drift every five minutes, and reports freshness. Forge and GitHub exposed 10 identical refs after the production drill. |
22 22
| Rolling image replacement | Available for structural changes | Production requires an operator-directed rollout when the classifier returns `needs_rolling_replace`; staging can use the configured GCP provider |
23 23
24 24
Two consequences worth stating plainly:

@@ -90,8 +90,9 @@ The production fleet uses this configuration:

90 90
   regardless of this flag. Boot convergence is already proven on this fleet.
91 91
5. **Allowlist**: no change needed. Baked configuration already admits the
92 92
   whole `OpenAgentsWeb.` layer plus `OpenAgents.Changelog`,
93
   `OpenAgents.Forge.Browse`, `OpenAgents.BuildInfo`, and the scratch prefix,
94
   with boot-time classification self-tests.
93
   `OpenAgents.Forge.Browse`, `OpenAgents.Forge.MirrorWatch`,
94
   `OpenAgents.BuildInfo`, and the scratch prefix, with boot-time
95
   classification self-tests.
95 96
6. **Optional: Turn the mirror on** by configuring a mirror URL for
96 97
   `openagents.com`. This only affects the public status projection and GitHub
97 98
   mirroring, never deploys.

@@ -119,6 +120,11 @@ seeds the persistent cache. Subsequent web-layer diffs should land in seconds.

119 120
Receipts measure pipeline time from push acknowledgment to live, not operator
120 121
reaction time.
121 122
123
Keep the application version unchanged for direct BEAM transactions. Use the
124
next patch version only when a compatible full-release package needs a new
125
version. Reserve a minor-version change for a deliberate compatibility or
126
feature boundary.
127
122 128
If classification returns `needs_rolling_replace`, keep that receipt and use
123 129
this fallback order:
124 130

@@ -209,9 +215,37 @@ the URL or instance metadata.

209 215
`Pushes.mirror_storage_key/1` resolves the logical repository name to its
210 216
canonical storage UUID before reading refs. This distinction matters for
211 217
migrated repositories whose display name and storage key differ.
212
`MirrorWatch` runs even when `OPENAGENTS_FEATURE_FORGE_DEPLOY=false`, compares
213
the canonical `main` ref every five minutes, retries a full mirror push on
214
drift, and publishes `current` or `lagging` status.
218
`MirrorWatch` runs even when `OPENAGENTS_FEATURE_FORGE_DEPLOY=false`. It checks
219
the canonical `main` ref immediately when the process starts and every five
220
minutes afterward, retries a full mirror push on drift, and publishes
221
`current` or `lagging` status.
222
223
## Production activation evidence
224
225
The 2026-08-22 activation established this baseline:
226
227
- The exact-SHA release gate passed all 13 stages in 163 seconds, including
228
  2,040 tests, direct transaction, relup, rolling replacement, infrastructure
229
  contracts, and disposable PostgreSQL release smoke tests.
230
- All three production nodes run revision `4bf3f0c` from immutable application
231
  digest `sha256:345220949cf29ae652879624df45db2b871926418bd3bb6f8d085fc9f9e3d8ab`
232
  with builder digest
233
  `sha256:a55ba9a80b781d953c24a254fc755541dbb954297ef92f81c982a58d13aafa9f`.
234
- The one-time structural baseline built in about 127 seconds and settled as
235
  `live` with a 542-module receipt. Later compatible changes use that manifest
236
  for direct classification.
237
- Restarting a node reported boot convergence `ready: true`, reason
238
  `image_matches_live`, and the exact baseline SHA before admission.
239
- `/api/status` reported the Forge lane as `active`. The configured GitHub
240
  mirror accepted `git ls-remote` from the application identity and reported
241
  `current` after its freshness check.
242
- Production's small boot partitions required removing obsolete build
243
  containers before pruning immutable images. The startup metadata and the
244
  repository-owned fleet template now remove those retired containers and
245
  replace the disposable builder sidecar before pulling a new builder image.
246
247
The activation did not change the application version from `0.2.0`. Do not
248
increment a release version merely to record a source commit.
215 249
216 250
After a rollout or storage repair, force convergence before validating the
217 251
mirror:
docs/runtime-configuration.md modified +5 -4

@@ -251,10 +251,11 @@ whether those settings passed; it never prints their values.

251 251
252 252
Setting `OPENAGENTS_FORGE_MIRROR_URLS_JSON` turns on one-way mirroring: every
253 253
accepted forge push is followed by a best-effort `git push --mirror` to the
254
configured URL, and `OpenAgents.Forge.MirrorWatch` compares refs every five
255
minutes, retries drift, and raises one `forge_mirror_lagging` incident per lag
256
episode past fifteen minutes. Mirror freshness appears on the public status
257
page as `current` or `lagging`; with no URLs configured it reads `off`.
254
configured URL. `OpenAgents.Forge.MirrorWatch` compares refs immediately after
255
process startup and every five minutes afterward, retries drift, and raises
256
one `forge_mirror_lagging` incident per lag episode past fifteen minutes.
257
Mirror freshness appears on the public status page as `current` or `lagging`;
258
with no URLs configured it reads `off`.
258 259
259 260
Two rules are load-bearing:
260 261
infra/staging/templates/fleet-startup.sh.tftpl modified +5 -1

@@ -164,6 +164,7 @@ docker-credential-gcr configure-docker --registries=${region}-docker.pkg.dev

164 164
165 165
# Fleet images are immutable and recoverable from Artifact Registry. Reclaim
166 166
# only images that no running container uses before pulling the next candidate.
167
docker rm --force sarah-builder sarah-breakglass 2>/dev/null || true
167 168
docker image prune --all --force
168 169
169 170
cloud_sql_proxy_image=gcr.io/cloud-sql-connectors/cloud-sql-proxy@sha256:825d5e4ce70d38bd0006c9eea15a6a2e2983e87b31ac6924d33e2dba56eafc9f

@@ -239,8 +240,11 @@ if [ -n "$builder_image" ] || [ -n "$builder_digest" ]; then

239 240
    exit 1
240 241
  fi
241 242
  append_secret OPENAGENTS_FORGE_OPERATOR_TOKEN openagents-staging-forge-operator-token /run/openagents/builder.env
242
  docker pull "$builder_image"
243
  # The small COS boot disk cannot retain two builder images. The running
244
  # application remains available while this disposable sidecar is replaced.
243 245
  docker rm --force openagents-builder 2>/dev/null || true
246
  docker image prune --all --force
247
  docker pull "$builder_image"
244 248
  docker run --detach \
245 249
    --name openagents-builder \
246 250
    --network host \
lib/openagents/forge/mirror_watch.ex modified +1 -1

@@ -31,7 +31,7 @@ defmodule OpenAgents.Forge.MirrorWatch do

31 31
32 32
  @impl true
33 33
  def init(_opts) do
34
    schedule()
34
    send(self(), :tick)
35 35
    {:ok, %{lagging_since: %{}, incident_reported: MapSet.new()}}
36 36
  end
37 37
test/openagents/forge/mirror_watch_test.exs modified +5

@@ -88,6 +88,11 @@ defmodule OpenAgents.Forge.MirrorWatchTest do

88 88
89 89
  defp fresh_state, do: %{lagging_since: %{}, incident_reported: MapSet.new()}
90 90
91
  test "the watcher checks mirror freshness immediately after it starts" do
92
    assert {:ok, _state} = MirrorWatch.init([])
93
    assert_receive :tick
94
  end
95
91 96
  test "a behind mirror is retried immediately and becomes current", %{path: path, mirror: mirror} do
92 97
    # Forge has a commit the mirror lacks → behind → check retries → healed.
93 98
    state = MirrorWatch.check_all(fresh_state())

This page updates live while a promote is in flight · changelog