Make forge fleet deployments transactional

ef3f17d35f6e · Christopher David · 11d ago · parent 04021662930f

Make forge fleet deployments transactional

Deploy story

What this commit did to the running system — joined from the forge receipt chain, the part a commit page elsewhere cannot show.

Not deployed through the forge lane

No push, promotion, build, or deploy receipt references this commit (receipts are scanned over a bounded recent window). Changes shipped by full node replacement carry their proof in the release gate receipt instead.

Changed files

modified config/config.exs
modified config/runtime.exs
modified docs/2026-08-20-integration-hardening-and-staging-readiness-recommendations.md
modified docs/beam-hot-deployment-plan.md
modified docs/operations/forge-build-lane.md
added docs/operations/forge-transactional-deployment.md
modified docs/runtime-configuration.md
modified lib/openagents/application.ex
modified lib/openagents/cluster.ex
modified lib/openagents/forge/boot_converge.ex
modified lib/openagents/forge/build_artifact.ex
modified lib/openagents/forge/deploy_receipt.ex
added lib/openagents/forge/deployment.ex
added lib/openagents/forge/deployment_node.ex
modified lib/openagents/forge/hot_loader.ex
modified lib/openagents/forge/targets.ex
modified lib/openagents/network_status.ex
modified lib/openagents/runtime_config.ex
modified lib/openagents/runtime_supervisor.ex
modified lib/openagents_web/controllers/health_controller.ex
modified ops/staging/gate-5-profile.sh
added priv/repo/migrations/20260820140000_harden_forge_deployment_transactions.exs
modified test/openagents/forge/boot_converge_test.exs
added test/openagents/forge/deployment_cluster_test.exs
added test/openagents/forge/deployment_node_test.exs
modified test/openagents/forge/hot_loader_test.exs
modified test/openagents/forge/loop_v0_test.exs
modified test/openagents/forge/target_lifecycle_test.exs
modified test/openagents/network_status_test.exs
modified test/openagents_web/controllers/health_controller_test.exs

Diff

30 files changed, +3495 -325

  forge_artifact_dir: "/var/lib/openagents/artifacts",
  forge_build_timeout_ms: 300_000,
  forge_build_output_retention_ms: 604_800_000,
  forge_deploy_timeout_ms: 15_000,
  forge_deploy_token_ttl_ms: 120_000,
  forge_boot_retry_min_ms: 1_000,
  forge_boot_retry_max_ms: 30_000,
  forge_artifact_store: :local,
  forge_build_executor: OpenAgents.Forge.BuildExecutor.Sidecar,
  forge_expected_fleet_size: 1,

      parse_integer.("OPENAGENTS_FORGE_BUILD_TIMEOUT_MS", 30_000..1_800_000),
    forge_build_output_retention_ms:
      parse_integer.("OPENAGENTS_FORGE_BUILD_OUTPUT_RETENTION_MS", 86_400_000..2_592_000_000),
    forge_deploy_timeout_ms: parse_integer.("OPENAGENTS_FORGE_DEPLOY_TIMEOUT_MS", 1_000..120_000),
    forge_deploy_token_ttl_ms:
      parse_integer.("OPENAGENTS_FORGE_DEPLOY_TOKEN_TTL_MS", 30_000..1_800_000),
    forge_boot_retry_min_ms: parse_integer.("OPENAGENTS_FORGE_BOOT_RETRY_MIN_MS", 100..60_000),
    forge_boot_retry_max_ms: parse_integer.("OPENAGENTS_FORGE_BOOT_RETRY_MAX_MS", 1_000..300_000),
    forge_artifact_store: forge_artifact_store,
    forge_build_executor: forge_build_executor,
    forge_expected_fleet_size: parse_integer.("OPENAGENTS_FORGE_EXPECTED_FLEET_SIZE", 1..100),


Date: 2026-08-20

Status: In progress; Gates 0–5 and 7–9 complete, Gate 6 application controls locally verified
Status: In progress; Gates 0–5 and 7–10 complete, Gate 6 application controls locally verified

## Outcome


  disables terminal prompting.
- Added deterministic BEAM normalization and canonical artifact manifests with
  source, baseline, Elixir, OTP, ERTS, application version/spec, and dependency
  lock identities plus complete added, changed, and deleted module sets.
  lock identities plus complete added, changed, and deleted module sets. Fixed
  tar ownership and time metadata so identical inputs remain byte-for-byte
  reproducible across wall-clock boundaries.
- Addressed artifacts by the full tar SHA-256 in both the local cache and
  durable WAL store. A target cannot become `built` until independent
  verification, local publication, durable storage, and receipt completion all

  classification, verification, recovery, retention, and staging proof in the
  [forge build lane runbook](operations/forge-build-lane.md).

Gate 9 is complete. Keep the staging deploy lane disabled until Gate 10 makes
fleet application transactional. Gates 12–15 must still exercise the real
builder image against the exact staging SHA and retain the image, build,
artifact, and output-proof identities; local success is not staging admission.
Gate 9 is complete, and Gate 10 now provides transactional fleet application.
Keep the staging deploy lane disabled until Gate 11 adds the fallback lanes and
Gate 12 creates the isolated distributed staging environment. Gates 12–15 must
exercise the real builder image against the exact staging SHA and retain the
image, build, artifact, and output-proof identities; local success is not
staging admission.

## Gate 10: Make fleet deployment transactional


960	964		`timeout behavior, node replacement, cold-cache convergence, and refusal to`
961	965		`serve divergent code.`
962	966
	967	+	`### Gate 10 implementation status`
	968	+
	969	+	`Completed locally on 2026-08-20:`
	970	+
	971	+	`- Replaced the one-way remote load with an explicit fleet transaction that`
	972	+	`snapshots the exact healthy member set, requires the configured fleet size,`
	973	+	`selects one canary, and rechecks membership between every mutating phase.`
	974	+	`- Added a supervised participant on every node. It independently verifies the`
	975	+	`artifact, canonical manifest, source, build, runtime toolchain,`
	976	+	`classification, and allowlist before creating module atoms. It caches the`
	977	+	`digest-addressed bytes and captures exact prior object code before returning`
	978	+	`a random 256-bit token.`
	979	+	`- Bound each token to one deployment, target, build, source SHA, artifact,`
	980	+	`manifest, candidate, prior object set, and expected fleet. Tokens expire and`
	981	+	`admit at most four prepared transactions per node. Expired uncommitted tokens`
	982	+	`restore immediately. Expired committed tokens finalize only when the durable`
	983	+	`target and artifact identities match, defer while the database commit remains`
	984	+	`in progress, and restore after a durable refusal.`
	985	+	`- Persisted the bounded participant fence across supervised process restarts.`
	986	+	`A participant restart therefore recovers the token, exact prior object code,`
	987	+	`candidate identity, and readiness fence instead of exposing partially`
	988	+	`applied code.`
	989	+	`- Kept every participant out of external readiness from prepare through fleet`
	990	+	`commit. Candidate verification checks loaded BEAM MD5 identities,`
	991	+	`application version, candidate revision, and node-level deployment`
	992	+	`readiness before commit.`
	993	+	`- Made all errors and timeouts trigger rollback on every prepared node. A node`
	994	+	`reloads or removes every affected module and verifies the exact restored`
	995	+	BEAM identity. Only a fully verified fleet restoration records `reverted`;
	996	+	an unreachable or divergent node records `failed` and remains out of
	997	+	`readiness.`
	998	+	- Added a database transaction that advances the target to `live` only in the
	999	+	`same commit that inserts its terminal deployment receipt. The receipt binds`
	1000	+	`deployment, artifact, manifest, expected membership, bounded per-node`
	1001	+	`outcomes, rollback verification, and timing. PostgreSQL rejects receipt`
	1002	+	`updates and deletes.`
	1003	+	`- Rehearsed the receipt migration forward, backward, and forward again on a`
	1004	+	`disposable populated database. The migration backfilled legacy rows,`
	1005	+	`preserved them across rollback, and recreated the hardened schema without`
	1006	+	`data loss.`
	1007	+	`- Fenced deployment ownership to the newest promotion, so a late build for a`
	1008	+	`superseded target cannot change fleet code.`
	1009	+	`- Moved the deployment participant and synchronous boot convergence after`
	1010	+	`PostgreSQL but before cluster discovery, PubSub, runtime workers, and the`
	1011	+	`endpoint. A cold node verifies its local artifact or fetches the identical`
	1012	+	`durable blob, retains the current and immediate rollback artifacts, prunes`
	1013	+	`older digest-addressed cache entries, and retries failures with capped`
	1014	+	`exponential backoff.`
	1015	+	`- Added durable live-target checks to readiness and the periodic convergence`
	1016	+	`cycle. A late-joining or replaced node leaves readiness while a target is`
	1017	+	`deploying or as soon as a newer live target exists, then converges before it`
	1018	+	`serves that revision.`
	1019	+	- Made `/healthz` return `503` while boot code or a deployment participant is
	1020	+	`divergent. Added a bounded, content-free boot and deployment projection to`
	1021	+	`/status`, and made its quorum calculation honor the configured forge fleet
	1022	+	`size even when Ra is disabled.`
	1023	+	`- Added direct participant tests for token fencing, exact rollback, expiry,`
	1024	+	`commit authority, and supervised restart recovery; database tests for`
	1025	+	`duplicate delivery, superseded targets, atomic live completion, and receipt`
	1026	+	`immutability; boot tests for cold-cache fetch, cache retention, stale-target`
	1027	+	`refusal, and degraded retry; and real three-node tests for success, remote`
	1028	+	`failure with exact rollback, rollback refusal, timeout, and membership loss.`
	1029	+	`- Passed 107 default forge tests and five real three-node deployment tests.`
	1030	+	`The full precommit gate passed 17 browser tests and 1,318 default Elixir`
	1031	+	`tests, with 14 distributed tests excluded from the default lane. The pinned`
	1032	+	Elixir 1.20.3 and OTP 29.0.5 `forge-builder` target also built the production
	1033	+	`release without application warnings.`
	1034	+	`- Documented the operator contract in the`
	1035	+	`[transactional deployment runbook](operations/forge-transactional-deployment.md).`
	1036	+
	1037	+	`Gate 10 is complete locally. Keep the deploy and boot-convergence features`
	1038	+	`disabled until Gate 11 supplies relup and rolling-replacement fallbacks and`
	1039	+	`Gate 12 provides an isolated distributed staging lane. No staging or`
	1040	+	`production environment was changed.`
	1041	+
963	1042		`## Gate 11: Complete relup and rolling replacement`
964	1043
965	1044		`Direct loading is only one deployment class. Implement and test the two required`

- [x] Voice recording starts only after generation admission.
- [x] Build requests are structured, unique, bounded, and non-executable.
- [x] Artifacts are immutable, digested, manifest-checked, and durably stored.
- [ ] Fleet deployment is transactional and rolls back every affected node.
- [ ] Boot convergence controls readiness.
- [x] Fleet deployment is transactional and rolls back every affected node.
- [x] Boot convergence controls readiness.
- [ ] Relup and rolling replacement pass their staging drills.
- [ ] Owned local gates produce exact-SHA receipts.
- [ ] Web and distributed staging are isolated from production.


Implementation status: Gate 9 completed this phase on 2026-08-20. See the
[forge build lane runbook](operations/forge-build-lane.md) for the deployed
contract. Fleet application remains disabled pending the transactional work in
Phase 5 and Gate 10.
contract. Gate 10 completed transactional fleet application, but the lane
remains disabled until isolated distributed staging and the fallback classes
are ready.

## Phase 5: Add direct BEAM classification and local canary loading



**Exit criteria:** multi-node tests prove all-node success, remote timeout, remote load failure, exact fleet rollback, duplicate event handling, node membership changes during deployment, and refusal to mark a divergent fleet live.

Implementation status: Gate 10 completed this phase on 2026-08-20. The
[transactional deployment runbook](operations/forge-transactional-deployment.md)
documents the implemented node protocol, atomic receipt boundary, readiness
behavior, and three-node proofs.

## Phase 7: Converge replacement and restarted nodes

Add `OpenAgents.Forge.BootConverge` as a synchronous boot step after `OpenAgents.Repo` starts and before `DNSCluster`, `OpenAgents.PubSub`, and `OpenAgentsWeb.Endpoint` start.

2. If the target has a direct BEAM artifact, load it from the local cache or fetch it from the durable artifact store.
3. Verify the digest, allowlist, and module manifest before loading.
4. Record the outcome in `:persistent_term` for health and status reports.
5. Treat a live target with an empty artifact as converged.
5. Treat a live target with no artifact as converged only when the image
   revision exactly matches that target.
6. Start on image code if the target or artifact cannot load, but keep readiness false while the node differs from the live target.
7. Retry convergence with bounded backoff so an artifact-store interruption does not require another restart.
8. Prune old local artifacts while retaining the current target and immediate rollback artifacts. Never prune the durable copy or receipts.

**Exit criteria:** tests cover local convergence, durable fetch, no target, non-live target, empty artifact, missing artifact, off-allowlist content, corrupt digest, and readiness behavior.

Implementation status: Gate 10 completed readiness-bound convergence on
2026-08-20. It starts synchronously before discovery and the endpoint, fetches
from durable storage on a cold cache, retains the immediate predecessor, prunes
older digest-addressed cache entries, periodically checks the durable live
target, retries with bounded backoff, and keeps divergent code out of
readiness.

## Phase 8: Add the production relup lane

Use relups for changes that need OTP-coordinated module updates or `code_change/3` state migration but do not cross a structural boundary.


Date: 2026-08-20

Status: Implemented locally; keep staging deployment disabled until Gate 10 is complete
Status: Implemented locally; the transactional deployment lane now consumes its verified artifacts

## Purpose

The forge build lane turns one operator-promoted, fully qualified Git commit
into a reproducible BEAM artifact. Compilation runs in an isolated builder
container. The public release receives no compiler, Docker socket, Git
into a reproducible BEAM artifact with fixed tar ownership and time metadata.
Compilation runs in an isolated builder container. The public release receives
no compiler, Docker socket, Git
credential, or ability to execute queue contents.

This lane only produces and verifies artifacts. Gate 10 owns transactional
fleet deployment. Do not enable direct staging loads merely because a build
completes.
This lane only produces and verifies artifacts. The
[transactional deployment runbook](forge-transactional-deployment.md) owns
fleet application. Do not enable direct staging loads merely because a build
completes; the isolated distributed staging lane and fallback deployment
classes remain required.

## Runtime roles


1	+	`# Forge transactional deployment`
2	+
3	+	`Date: 2026-08-20`
4	+
5	+	`Status: Implemented locally; keep staging deployment disabled until the Gate 12 distributed staging lane exists`
6	+
7	+	`## Purpose`
8	+
9	+	`The direct-load lane applies one verified BEAM artifact to an exact expected`
10	+	fleet without exposing a mixed revision as `live`. It treats the database
11	+	`target, every node's loaded object code, external readiness, and the immutable`
12	+	`terminal receipt as one coordinated deployment outcome.`
13	+
14	+	Use this lane only for artifacts classified as `direct_candidate` whose every
15	+	`changed module matches the operator-owned allowlist. Send structural changes to`
16	+	`the relup or rolling-replacement lanes.`
17	+
18	+	`## Preconditions`
19	+
20	+	`The coordinator refuses deployment unless all of these conditions hold:`
21	+
22	+	- The target remains the newest promotion and has status `built`.
23	+	`- Connected membership exactly matches`
24	+	`OPENAGENTS_FORGE_EXPECTED_FLEET_SIZE`.
25	+	`- Every expected node reports successful boot convergence, no active or`
26	+	`divergent deployment, and the same current revision.`
27	+	`- Every node can verify the artifact digest, canonical manifest digest,`
28	+	`repository, full source SHA, build ID, runtime toolchain, classification, and`
29	+	`allowlist.`
30	+	`- Every node can cache the digest-addressed artifact and recover the exact`
31	+	`object code that the candidate would replace.`
32	+
33	+	`A missing, unexpected, unhealthy, or revision-divergent node blocks the`
34	+	`transaction before any candidate becomes live.`
35	+
36	+	Claiming deployment ownership changes the newest target from `built` to
37	+	`deploying` and removes the fleet from ordinary external readiness. The
38	+	`coordinator then uses a target-scoped internal health check that accepts only`
39	+	`nodes converged to the prior serving target for that exact deployment. This`
40	+	`separation prevents a late-joining node from serving old code without blocking`
41	+	`the prepared fleet from completing the transaction.`
42	+
43	+	`## Transaction protocol`
44	+
45	+	`The coordinator runs these phases in order:`
46	+
47	+	`1. Snapshot the sorted healthy member set and select the coordinator node as`
48	+	`the canary when it belongs to that set.`
49	+	`2. Send the immutable artifact and its expected identities to every node.`
50	+	`3. Have each node verify the complete artifact, cache it by SHA-256, capture`
51	+	`exact prior object code, and return a random 256-bit token.`
52	+	`4. Apply and verify the candidate on the canary.`
53	+	`5. Recheck exact membership.`
54	+	`6. Apply the candidate on every remaining prepared node.`
55	+	`7. Recheck exact membership, then verify loaded BEAM MD5 identities,`
56	+	`application version, candidate revision, and deployment readiness on every`
57	+	`node.`
58	+	`8. Recheck membership and commit every expiring token.`
59	+	9. In one PostgreSQL transaction, advance the target to `live` and insert its
60	+	`immutable terminal receipt.`
61	+	`10. Finalize every token and release the nodes into external readiness.`
62	+
63	+	`OpenAgents.Forge.DeploymentNode` keeps its node out of readiness from prepare
64	+	`through commit. The token binds one deployment ID to its target, build, source`
65	+	`SHA, artifact, manifest, expected member set, candidate binaries, and prior`
66	+	`object code. Another attempt cannot reuse it.`
67	+
68	+	`The participant bounds itself to four concurrent prepared transactions. It`
69	+	`automatically resolves expired transactions against their phase and durable`
70	+	`authority. Configure enough token lifetime for all phases and rollback; runtime`
71	+	`validation requires at least eight phase timeouts. It persists the bounded`
72	+	`transaction fence in node-local VM state, so a supervised participant restart`
73	+	`preserves the exact rollback data and keeps the node out of readiness. A VM`
74	+	`restart returns to image code and runs boot convergence before the endpoint`
75	+	`starts.`
76	+
77	+	`When a committed token expires, the participant asks PostgreSQL for the exact`
78	+	`target, deployment, and artifact authority. It finalizes the candidate only`
79	+	after the durable `live` transaction exists. It extends the readiness fence
80	+	while that database transaction remains `deploying`, and it restores the prior
81	+	`code after a durable terminal refusal. If PostgreSQL is unavailable, the node`
82	+	`stays out of readiness and retries the authority check.`
83	+
84	+	`## Failure and rollback behavior`
85	+
86	+	`Any prepare, apply, verification, commit, membership, or RPC failure triggers`
87	+	`rollback on every node that returned a token. Each participant:`
88	+
89	+	`1. Reloads the exact captured binary for a preexisting module or removes a`
90	+	`module that was absent before the transaction.`
91	+	`2. Verifies the restored module's embedded BEAM MD5 against the captured`
92	+	`object.`
93	+	3. Returns `restored` only after every affected module passes verification.
94	+
95	+	The coordinator records `reverted` only when at least one node applied the
96	+	candidate and every prepared node verified restoration. It records `failed`
97	+	`for an unverified rollback, timeout, or unreachable participant. A participant`
98	+	whose restoration fails keeps `ready=false` with reason
99	+	`rollback_unverified`; traffic must not return until boot convergence or
100	+	`rolling replacement repairs it.`
101	+
102	+	`If the PostgreSQL target-and-receipt transaction fails after node commit, the`
103	+	`coordinator rolls back the still-fenced tokens before it records the terminal`
104	+	outcome. It never advances the target to `live` based only on a canary result.
105	+
106	+	`## Immutable receipts`
107	+
108	+	Each `forge_deploys` row records bounded operational evidence:
109	+
110	+	`- deployment, target, build, repository, and full source identities;`
111	+	`- artifact and canonical manifest SHA-256 digests;`
112	+	`- the expected nodes and bounded per-node terminal outcomes;`
113	+	`- canary result, changed module names for the operator surface, and`
114	+	`push-to-live duration;`
115	+	`- failure code and whether rollback verification succeeded;`
116	+	`- transaction start and completion timestamps.`
117	+
118	+	PostgreSQL rejects every `UPDATE` and `DELETE` on this table. Public status
119	+	`projects only anonymous node positions, a short SHA, state, module count, and`
120	+	`timing. It does not expose node names, module names, tokens, or free-form`
121	+	`failure data.`
122	+
123	+	`## Boot convergence and readiness`
124	+
125	+	`Application supervision starts the deployment participant and synchronous`
126	+	`boot convergence after PostgreSQL and before cluster discovery, PubSub, the`
127	+	`runtime supervisor, or the endpoint.`
128	+
129	+	`On a cold or replaced node, boot convergence:`
130	+
131	+	1. Reads the newest immutable `live` target.
132	+	`2. Verifies a local digest-addressed artifact or fetches it from the durable`
133	+	`WAL artifact store.`
134	+	`3. Applies it through the same participant verifier and exact rollback logic.`
135	+	`4. Verifies and retains the current artifact and immediate live predecessor in`
136	+	`the local cache.`
137	+	`5. Prunes older digest-addressed cache entries without touching durable`
138	+	`artifacts or receipts.`
139	+	`6. Publishes a bounded convergence state and enters readiness.`
140	+
141	+	If image code does not match the live target and convergence fails, `/healthz`
142	+	returns `503` and `/status` reports a content-free degraded state. The worker
143	+	`retries indefinitely with exponential backoff capped by`
144	+	`OPENAGENTS_FORGE_BOOT_RETRY_MAX_MS`. A missing live target is the only normal
145	+	`image-only ready state. A live target without immutable artifact identity is a`
146	+	`readiness failure unless the image revision exactly matches that target. After`
147	+	`convergence, the worker periodically rechecks the durable target. Readiness`
148	+	also rejects an active `deploying` target and performs this identity check, so
149	+	`a late-joining node cannot continue to serve an older target during the next`
150	+	`convergence interval.`
151	+
152	+	`## Runtime settings`
153	+
154	+	`\| Setting \| Purpose \|`
155	+	`\| --- \| --- \|`
156	+	\| `OPENAGENTS_FORGE_EXPECTED_FLEET_SIZE` \| Exact member count required throughout a transaction \|
157	+	\| `OPENAGENTS_FORGE_DEPLOY_TIMEOUT_MS` \| Per-node RPC phase timeout \|
158	+	\| `OPENAGENTS_FORGE_DEPLOY_TOKEN_TTL_MS` \| Token lifetime; must cover at least eight phase timeouts \|
159	+	\| `OPENAGENTS_FORGE_BOOT_RETRY_MIN_MS` \| Initial convergence retry interval \|
160	+	\| `OPENAGENTS_FORGE_BOOT_RETRY_MAX_MS` \| Maximum convergence retry interval \|
161	+
162	+	Keep `OPENAGENTS_FEATURE_FORGE_DEPLOY=false` and
163	+	`OPENAGENTS_FEATURE_BOOT_CONVERGENCE=false` until the isolated distributed
164	+	`staging lane exists and Gate 11 adds the relup and rolling-replacement`
165	+	`fallbacks.`
166	+
167	+	`## Verification`
168	+
169	+	`Run the direct participant, coordinator, boot, and receipt proofs:`
170	+
171	+	```sh
172	+	`mix test test/openagents/forge/deployment_node_test.exs`
173	+	`mix test test/openagents/forge/hot_loader_test.exs`
174	+	`mix test test/openagents/forge/boot_converge_test.exs`
175	+	`mix test test/openagents/forge/target_lifecycle_test.exs`
176	+	`mix test test/openagents/forge/deployment_cluster_test.exs --only cluster`
177	+	`mix precommit`
178	+	```
179	+
180	+	`The cluster test starts two real peer nodes beside the test coordinator. It`
181	+	`proves a three-node commit, a remote apply failure with exact fleet rollback, a`
182	+	`unverified rollback that removes the affected node from readiness, a remote`
183	+	`timeout that cannot claim rollback, and membership loss before commit. The boot`
184	+	`suite proves cold-cache durable fetch, current-and-predecessor retention,`
185	+	`old-cache pruning, live-target freshness, and refusal to serve missing,`
186	+	`unidentified, or off-allowlist live code.`

| `OPENAGENTS_FORGE_ARTIFACT_DIR` | Absolute durable path outside `/tmp` |
| `OPENAGENTS_FORGE_BUILD_TIMEOUT_MS` | `300000`; admitted range 30 seconds to 30 minutes |
| `OPENAGENTS_FORGE_BUILD_OUTPUT_RETENTION_MS` | `604800000` (seven days); admitted range one to 30 days |
| `OPENAGENTS_FORGE_DEPLOY_TIMEOUT_MS` | `15000`; admitted range one to 120 seconds |
| `OPENAGENTS_FORGE_DEPLOY_TOKEN_TTL_MS` | `120000`; admitted range 30 seconds to 30 minutes and at least eight deployment timeouts |
| `OPENAGENTS_FORGE_BOOT_RETRY_MIN_MS` | `1000`; admitted range 100 milliseconds to one minute |
| `OPENAGENTS_FORGE_BOOT_RETRY_MAX_MS` | `30000`; admitted range one to 300 seconds and not less than the minimum |
| `OPENAGENTS_CODING_JOBS_DIR` | Absolute durable path outside `/tmp` when work or computers are enabled |
| `OPENAGENTS_RA_DATA_DIR` | Absolute durable path outside `/tmp` when Ra is enabled |
| `OPENAGENTS_RA_EXPECTED_SIZE` | At least `3` when Ra is enabled |

    children = [
      OpenAgentsWeb.Telemetry,
      OpenAgents.Repo,
      # Deployment identity and boot convergence must settle before cluster
      # discovery or the endpoint can make this node externally reachable.
      OpenAgents.Forge.DeploymentNode,
      OpenAgents.Forge.BootConverge,
      {DNSCluster, query: Application.get_env(:openagents, :dns_cluster_query) || :ignore},
      {Phoenix.PubSub, name: OpenAgents.PubSub},
      OpenAgents.RuntimeSupervisor,


  @doc """
  A bounded health report for this node. Liveness means the VM and BEAM are
  running; readiness is a runtime assertion that will be driven by boot
  convergence and application health checks in later phases.
  running. Readiness requires successful boot convergence and no active or
  divergent forge deployment.
  """
  @spec local_report() :: map()
  def local_report do
    boot = OpenAgents.Forge.BootConverge.state()
    boot_ready? = OpenAgents.Forge.BootConverge.ready?()
    deployment = OpenAgents.Forge.DeploymentNode.health()

    %{
      "schema" => "openagents.cluster_health.v1",
      "node" => to_string(Node.self()),
      "version" => to_string(Application.spec(:openagents, :vsn) || "unknown"),
      "revision" => OpenAgents.BuildInfo.revision(),
      "boot_converged" => nil,
      "revision" => deployment["revision"] || boot["sha"] || OpenAgents.BuildInfo.revision(),
      "boot_converged" => boot_ready?,
      "deployment_ready" => deployment["participant_ready"],
      "uptime_ms" => uptime_ms(),
      "live" => true,
      "ready" => true
      "ready" => boot_ready? and deployment["ready"] == true
    }
  end


defmodule OpenAgents.Forge.BootConverge do
  @moduledoc """
  Boot convergence (roadmap P6, issue #123): on node start — after the Repo,
  before Horde/Ra membership or the endpoint — load the current live fleet
  target's beam artifact, so **a node converges to the forge's target
  commit, not to the image**. Image rebakes become structural-only.

  Honesty rules:
  - If there is no live target, no local artifact (a freshly replaced node
    has an empty partition — artifacts are per-node cache), an off-allowlist
    module, or any error, the node boots on image code and records why —
    it NEVER refuses to start. The state is published in `state/0` and on
    the public status page's per-node report.
  - The same operator-owned allowlist as the hot-load lane gates what boot
    convergence will load: convergence is a replay of an approved deploy,
    not a second authority (SELF-EDIT-001: hot-loaded code is a projection
    of the promoted commit).

  Runs synchronously inside `start_link/1` and returns `:ignore` — it is a
  boot step, not a process.
  Gates readiness on convergence to the newest immutable live target.

  A cold node fetches the live artifact from durable storage when its local
  digest-addressed cache is empty, verifies the same artifact and manifest
  identities used during promotion, and installs it through the transactional
  node participant. A divergent node stays out of readiness and retries with
  bounded exponential backoff. The public state contains only identity hashes,
  counters, and stable reason codes.
  """

  use GenServer

  require Logger

  alias OpenAgents.Forge.BuildArtifact
  alias OpenAgents.Forge.BuildProtocol
  alias OpenAgents.Forge.HotLoader
  alias OpenAgents.Forge.DeploymentNode
  alias OpenAgents.Forge.Repos
  alias OpenAgents.Forge.Targets

  @state_key {__MODULE__, :state}
  @default_retry_min_ms 1_000
  @default_retry_max_ms 30_000

  def start_link(opts \\ []) do
    GenServer.start_link(__MODULE__, opts, name: Keyword.get(opts, :name, __MODULE__))
  end

  @doc "Return this node's bounded convergence state."
  def state do
    :persistent_term.get(@state_key, initial_state(false))
  end

  @doc "Return whether this node may enter external readiness."
  def ready?(repo \\ "openagents.com") do
    convergence = state()

  def child_spec(_opts) do
    %{id: __MODULE__, start: {__MODULE__, :start_link, []}, type: :worker, restart: :temporary}
    cond do
      convergence["ready"] != true -> false
      not Application.get_env(:openagents, :forge_boot_converge_enabled, false) -> true
      true -> durable_target_ready?(repo, convergence)
    end
  rescue
    _error -> false
  catch
    _kind, _reason -> false
  end

  def start_link do
    if Application.get_env(:openagents, :forge_boot_converge_enabled, false) do
      converge()
  @doc false
  def ready_for_deployment?(repo, target_id) do
    convergence = state()

    cond do
      convergence["ready"] != true -> false
      not Application.get_env(:openagents, :forge_boot_converge_enabled, false) -> true
      true -> deployment_target_ready?(repo, target_id, convergence)
    end
  rescue
    _error -> false
  catch
    _kind, _reason -> false
  end

    :ignore
  @doc "Run one synchronous convergence attempt. Tests and repair tools use this API."
  def converge(repo \\ "openagents.com") do
    outcome = safe_attempt(repo, state()["attempts"] + 1)
    publish(outcome)
    outcome
  end

  @doc "The boot-convergence outcome for this node (for /status honesty)."
  def state do
    :persistent_term.get(@state_key, %{"state" => "image", "reason" => "not_attempted"})
  @doc false
  def mark_converged(identity) when is_map(identity) do
    outcome = %{
      "schema" => "openagents.forge.boot-convergence.v2",
      "state" => "converged",
      "ready" => true,
      "reason" => "fleet_commit",
      "sha" => identity.sha,
      "artifact_digest" => identity.artifact_digest,
      "manifest_digest" => identity.manifest_digest,
      "modules" => identity.modules,
      "attempts" => state()["attempts"],
      "retry_in_ms" => nil
    }

    publish(outcome)
  end

  @doc false
  def converge(repo \\ "openagents.com") do
    outcome =
      try do
        attempt(repo)
      rescue
        error ->
          %{"state" => "image", "reason" => OpenAgents.OperationalLog.code(error)}
      catch
        _kind, reason ->
          %{"state" => "image", "reason" => OpenAgents.OperationalLog.code(reason)}
  def restore_state(%{"schema" => "openagents.forge.boot-convergence.v2"} = outcome),
    do: publish(outcome)

  @impl true
  def init(opts) do
    repo = Keyword.get(opts, :repo, "openagents.com")
    enabled? = Application.get_env(:openagents, :forge_boot_converge_enabled, false)

    if enabled? do
      outcome = safe_attempt(repo, 1)

      if outcome["ready"] do
        publish(outcome)
        Process.send_after(self(), :retry_convergence, retry_max_ms())
        {:ok, %{repo: repo, retry_ms: retry_min_ms()}}
      else
        retry_ms = retry_min_ms()
        publish(Map.put(outcome, "retry_in_ms", retry_ms))
        Process.send_after(self(), :retry_convergence, retry_ms)
        {:ok, %{repo: repo, retry_ms: min(retry_ms * 2, retry_max_ms())}}
      end
    else
      publish(initial_state(false))
      {:ok, %{repo: repo, retry_ms: retry_min_ms()}}
    end
  end

    :persistent_term.put(@state_key, outcome)
  @impl true
  def handle_info(:retry_convergence, server_state), do: run_convergence(server_state)
  def handle_info(_message, server_state), do: {:noreply, server_state}

    case outcome do
      %{"state" => "converged", "sha" => sha} ->
        Logger.info("forge boot convergence: loaded target #{sha}")
  defp run_convergence(server_state) do
    outcome = safe_attempt(server_state.repo, state()["attempts"] + 1)

      %{"reason" => reason} ->
        Logger.info("forge boot convergence: booting on image code (#{reason})")
    if outcome["ready"] do
      publish(outcome)
      Process.send_after(self(), :retry_convergence, retry_max_ms())
      {:noreply, %{server_state | retry_ms: retry_min_ms()}}
    else
      retry_ms = min(server_state.retry_ms, retry_max_ms())
      outcome = Map.put(outcome, "retry_in_ms", retry_ms)
      publish(outcome)
      Process.send_after(self(), :retry_convergence, retry_ms)
      {:noreply, %{server_state | retry_ms: min(retry_ms * 2, retry_max_ms())}}
    end
  end

    outcome
  defp safe_attempt(repo, attempts) do
    attempt(repo, attempts)
  rescue
    error -> degraded("convergence_exception", attempts, error)
  catch
    _kind, reason -> degraded("convergence_exit", attempts, reason)
  end

  defp durable_target_ready?(repo, convergence) do
    case Targets.current(repo) do
      %{status: "deploying"} -> false
      _not_deploying -> current_target_matches?(repo, convergence)
    end
  end

  defp attempt(repo) do
  defp deployment_target_ready?(repo, target_id, convergence) do
    case Targets.current(repo) do
      %{id: ^target_id, status: "deploying"} -> current_target_matches?(repo, convergence)
      _other_target -> false
    end
  end

  defp current_target_matches?(repo, convergence) do
    case {Targets.live(repo), convergence} do
      {nil, %{"state" => "image", "reason" => "no_live_target"}} ->
        true

      {%{sha: sha, details: details}, %{"sha" => sha, "artifact_digest" => nil}} ->
        (details || %{})["artifact_digest"] == nil and OpenAgents.BuildInfo.revision() == sha

      {%{sha: sha, details: details},
       %{
         "sha" => sha,
         "artifact_digest" => artifact_digest,
         "manifest_digest" => manifest_digest
       }} ->
        details = details || %{}

        details["artifact_digest"] == artifact_digest and
          details["manifest_digest"] == manifest_digest

      _divergent ->
        false
    end
  end

  defp attempt(repo, attempts) do
    case Targets.live(repo) do
      nil ->
        image_ready("no_live_target", attempts)

      %{sha: sha, details: details} = target ->
        converge_target(repo, target, sha, details || %{}, attempts)
    end
  end

  defp converge_target(repo, target, sha, details, attempts) do
    with {:ok, identity} <- target_identity(target, details),
         {:ok, bytes, cache_state} <- artifact_bytes(repo, identity),
         {:ok, response} <- DeploymentNode.install_artifact(install_request(identity, bytes)),
         :ok <- retain_artifacts(repo, target.id, identity.artifact_digest) do
      %{
        status: "live",
        sha: sha,
        details: %{
          "artifact" => relative,
          "artifact_digest" => digest,
          "build_id" => build_id
        }
        "schema" => "openagents.forge.boot-convergence.v2",
        "state" => "converged",
        "ready" => true,
        "reason" => cache_state,
        "sha" => sha,
        "artifact_digest" => identity.artifact_digest,
        "manifest_digest" => identity.manifest_digest,
        "modules" => response["modules"] || identity.modules,
        "attempts" => attempts,
        "retry_in_ms" => nil
      }
      when is_binary(relative) and is_binary(digest) and is_binary(build_id) ->
        load_artifact(repo, sha, digest, build_id, Path.join(Repos.data_dir(), relative))
    else
      {:image_matches, ^sha} ->
        image_ready("image_matches_live", attempts, sha)

      {:error, reason} ->
        degraded(OpenAgents.OperationalLog.code(reason), attempts, reason, sha)
    end
  end

      %{status: "live", details: %{"artifact" => _relative}} ->
        %{"state" => "image", "reason" => "artifact_identity_missing"}
  defp target_identity(target, details) do
    case details do
      %{
        "artifact_digest" => artifact_digest,
        "build_id" => build_id,
        "manifest" => manifest
      }
      when is_binary(artifact_digest) and is_binary(build_id) and is_map(manifest) ->
        manifest_digest =
          details["manifest_digest"] ||
            BuildArtifact.digest(BuildProtocol.canonical_json(manifest))

      %{status: "live", sha: sha} ->
        # A live target with no artifact recorded (a no-op deploy): the
        # image already is that code as far as this lane knows.
        %{"state" => "converged", "sha" => sha, "modules" => 0}
        changes = manifest["changes"] || %{}
        modules = (changes["added"] || []) ++ (changes["changed"] || [])

      %{status: status} ->
        %{"state" => "image", "reason" => "target_not_live:#{status}"}
        {:ok,
         %{
           repo: target.repo,
           sha: target.sha,
           target_id: target.id,
           build_id: build_id,
           artifact_digest: artifact_digest,
           manifest_digest: manifest_digest,
           modules: length(modules)
         }}

      nil ->
        %{"state" => "image", "reason" => "no_target"}
      _missing ->
        if OpenAgents.BuildInfo.revision() == target.sha,
          do: {:image_matches, target.sha},
          else: {:error, :live_artifact_identity_missing}
    end
  end

  defp load_artifact(repo, sha, digest, build_id, artifact) do
    cond do
      not File.exists?(artifact) ->
        # The local cache misses on a replaced node — fetch the blob the
        # builder uploaded next to the WAL, then converge from it. Only if
        # the store misses too does the node boot on image code.
        case OpenAgents.Forge.WAL.get_artifact(repo, digest) do
          {:ok, payload} ->
            with {:ok, verified} <-
                   BuildArtifact.verify(payload,
                     digest: digest,
                     repo: repo,
                     source_sha: sha,
                     build_id: build_id
                   ) do
              # Cache only bytes that passed the immutable identity check.
              _cache_result = BuildProtocol.atomic_write(artifact, payload)
              load_beams_from(sha, verified)
            else
              {:error, _reason} ->
                %{"state" => "image", "reason" => "artifact_verification_failed"}
            end

          {:error, _reason} ->
            %{"state" => "image", "reason" => "artifact_missing"}
  defp artifact_bytes(repo, identity) do
    path = cache_path(identity.artifact_digest)

    case File.read(path) do
      {:ok, bytes} ->
        with {:ok, _verified} <- verify_artifact(bytes, identity) do
          {:ok, bytes, "local_cache"}
        end

      true ->
        case BuildArtifact.verify_file(artifact,
               digest: digest,
               repo: repo,
               source_sha: sha,
               build_id: build_id
             ) do
          {:ok, verified} ->
            load_beams_from(sha, verified)

          {:error, _reason} ->
            %{"state" => "image", "reason" => "artifact_verification_failed"}
      {:error, :enoent} ->
        with {:ok, bytes} <- OpenAgents.Forge.WAL.get_artifact(repo, identity.artifact_digest),
             {:ok, _verified} <- verify_artifact(bytes, identity),
             :ok <- cache_verified(path, bytes, identity.artifact_digest) do
          {:ok, bytes, "durable_fetch"}
        end

      {:error, reason} ->
        {:error, {:artifact_cache_read_failed, reason}}
    end
  end

  defp load_beams_from(sha, verified) do
    allowlist = Application.get_env(:openagents, :forge_hot_load_allowlist, default_allowlist())
  defp verify_artifact(bytes, identity) do
    BuildArtifact.verify(bytes,
      digest: identity.artifact_digest,
      repo: identity.repo,
      source_sha: identity.sha,
      build_id: identity.build_id
    )
  end

    offenders =
      verified.modules
      |> Enum.reject(&HotLoader.allowlisted?(&1, allowlist))
  defp install_request(identity, bytes) do
    %{
      artifact_bytes: bytes,
      artifact_digest: identity.artifact_digest,
      build_id: identity.build_id,
      deployment_id: Ecto.UUID.generate(),
      expected_nodes: [to_string(Node.self())],
      manifest_digest: identity.manifest_digest,
      repo: identity.repo,
      sha: identity.sha,
      target_id: identity.target_id
    }
  end

    cond do
      verified.manifest["classification"] != "direct_candidate" ->
        %{"state" => "image", "reason" => "artifact_not_direct"}

      offenders != [] ->
        %{"state" => "image", "reason" => "off_allowlist:#{Enum.join(offenders, ",")}"}

      true ->
        # Atom creation follows complete verification and policy checks.
        beams =
          Enum.map(verified.beams, fn %{module: module, binary: binary} ->
            {BuildArtifact.module_atom(module), binary}
          end)

        failures =
          beams
          |> HotLoader.load_beams()
          |> Enum.reject(fn {_mod, result} -> result == :ok end)

        if failures == [] do
          %{"state" => "converged", "sha" => sha, "modules" => length(beams)}
        else
          %{"state" => "image", "reason" => "load_failed"}
        end
  defp retain_artifacts(repo, current_target_id, current_digest) do
    with {:ok, predecessor_digest} <- retain_predecessor(repo, current_target_id) do
      [current_digest, predecessor_digest]
      |> Enum.reject(&is_nil/1)
      |> prune_artifact_cache()
    end
  end

  defp retain_predecessor(repo, current_target_id) do
    predecessor =
      repo
      |> Targets.live_history(3)
      |> Enum.reject(&(&1.id == current_target_id))
      |> List.first()

    case predecessor do
      nil -> {:ok, nil}
      target -> ensure_target_cached(repo, target)
    end
  end

  defp ensure_target_cached(repo, target) do
    details = target.details || %{}

    with {:ok, identity} <- target_identity(target, details),
         {:ok, _bytes, _source} <- artifact_bytes(repo, identity) do
      {:ok, identity.artifact_digest}
    else
      {:image_matches, _sha} -> {:ok, nil}
      {:error, reason} -> {:error, {:rollback_artifact_unavailable, reason}}
    end
  end

  defp prune_artifact_cache(retained_digests) do
    directory = Path.join(Repos.data_dir(), "beams")
    retained = MapSet.new(retained_digests)

    case File.ls(directory) do
      {:ok, entries} ->
        Enum.reduce_while(entries, :ok, fn entry, :ok ->
          case Regex.run(~r/^([0-9a-f]{64})\.tar$/, entry) do
            [_, digest] ->
              if MapSet.member?(retained, digest) do
                {:cont, :ok}
              else
                case File.rm(Path.join(directory, entry)) do
                  :ok -> {:cont, :ok}
                  {:error, :enoent} -> {:cont, :ok}
                  {:error, reason} -> {:halt, {:error, {:artifact_prune_failed, reason}}}
                end
              end

            nil ->
              {:cont, :ok}
          end
        end)

      {:error, :enoent} ->
        :ok

      {:error, reason} ->
        {:error, {:artifact_cache_list_failed, reason}}
    end
  end

  defp cache_verified(path, bytes, digest) do
    with :ok <- File.mkdir_p(Path.dirname(path)) do
      case BuildProtocol.atomic_write(path, bytes) do
        :ok ->
          :ok

        {:error, :destination_exists} ->
          with {:ok, existing} <- File.read(path),
               true <- BuildArtifact.digest(existing) == digest or {:error, :digest_collision} do
            :ok
          end

        {:error, reason} ->
          {:error, {:artifact_cache_write_failed, reason}}
      end
    else
      {:error, reason} -> {:error, {:artifact_cache_directory_failed, reason}}
    end
  end

  defp default_allowlist, do: ["OpenAgents.Scratch.", "OpenAgents.BuildInfo"]
  defp cache_path(digest), do: Path.join([Repos.data_dir(), "beams", digest <> ".tar"])

  defp image_ready(reason, attempts, sha \\ nil) do
    %{
      "schema" => "openagents.forge.boot-convergence.v2",
      "state" => "image",
      "ready" => true,
      "reason" => reason,
      "sha" => sha || OpenAgents.BuildInfo.revision(),
      "artifact_digest" => nil,
      "manifest_digest" => nil,
      "modules" => 0,
      "attempts" => attempts,
      "retry_in_ms" => nil
    }
  end

  defp degraded(code, attempts, _reason, sha \\ nil) do
    Logger.warning("forge_boot_convergence_degraded code=#{code}")

    %{
      "schema" => "openagents.forge.boot-convergence.v2",
      "state" => "degraded",
      "ready" => false,
      "reason" => String.slice(code, 0, 128),
      "sha" => sha,
      "artifact_digest" => nil,
      "manifest_digest" => nil,
      "modules" => 0,
      "attempts" => attempts,
      "retry_in_ms" => nil
    }
  end

  defp initial_state(false) do
    %{
      "schema" => "openagents.forge.boot-convergence.v2",
      "state" => "disabled",
      "ready" => true,
      "reason" => "feature_disabled",
      "sha" => OpenAgents.BuildInfo.revision(),
      "artifact_digest" => nil,
      "manifest_digest" => nil,
      "modules" => 0,
      "attempts" => 0,
      "retry_in_ms" => nil
    }
  end

  defp publish(outcome) do
    :persistent_term.put(@state_key, outcome)
    outcome
  end

  defp retry_min_ms do
    Application.get_env(:openagents, :forge_boot_retry_min_ms, @default_retry_min_ms)
  end

  defp retry_max_ms do
    Application.get_env(:openagents, :forge_boot_retry_max_ms, @default_retry_max_ms)
  end
end

      )

    try do
      with :ok <- :erl_tar.create(String.to_charlist(path), entries),
      with :ok <- write_reproducible_tar(path, entries),
           {:ok, bytes} <- File.read(path),
           :ok <- validate_artifact_size(bytes) do
        {:ok, bytes}

299	299		`end`
300	300		`end`
301	301
	302	+	`defp write_reproducible_tar(path, entries) do`
	303	+	`case :erl_tar.open(String.to_charlist(path), [:write]) do`
	304	+	`{:ok, tar} ->`
	305	+	`write_result =`
	306	+	`Enum.reduce_while(entries, :ok, fn entry, :ok ->`
	307	+	`case :erl_tar.add(tar, entry,`
	308	+	`mtime: 0,`
	309	+	`atime: 0,`
	310	+	`ctime: 0,`
	311	+	`uid: 0,`
	312	+	`gid: 0`
	313	+	`) do`
	314	+	`:ok -> {:cont, :ok}`
	315	+	`{:error, reason} -> {:halt, {:error, reason}}`
	316	+	`end`
	317	+	`end)`
	318	+
	319	+	`close_result = :erl_tar.close(tar)`
	320	+
	321	+	`case {write_result, close_result} do`
	322	+	`{:ok, :ok} -> :ok`
	323	+	`{{:error, reason}, _close} -> {:error, reason}`
	324	+	`{:ok, {:error, reason}} -> {:error, reason}`
	325	+	`end`
	326	+
	327	+	`{:error, reason} ->`
	328	+	`{:error, reason}`
	329	+	`end`
	330	+	`end`
	331	+
302	332		`defp extract_entries(bytes) do`
303	333		`case :erl_tar.extract({:binary, bytes}, [:memory]) do`
304	334		`{:ok, entries} when length(entries) <= @max_modules + 1 ->`

defmodule OpenAgents.Forge.DeployReceipt do
  @moduledoc """
  Receipt for one hot-deploy attempt (`forge_deploys`, roadmap P4): what was
  loaded (or refused), on which nodes, with the canary outcome and the
  first-class **push→live duration** measured from the matching push receipt.
  Every outcome is receipted — `live`, `reverted`, `needs_rolling_replace`,
  and `failed` alike — so the deploy lane never fails silently.
  Immutable terminal receipt for one fleet deployment attempt. The receipt
  binds the candidate to its deployment, artifact, manifest, expected member
  set, bounded per-node outcomes, rollback verification, and push-to-live
  duration. PostgreSQL rejects updates and deletes after insertion.
  """

  use Ecto.Schema

    field :repo, :string
    field :sha, :string
    field :target_id, :binary_id
    field :deployment_id, :binary_id
    field :artifact_digest, :string
    field :manifest_digest, :string
    field :modules, {:array, :string}, default: []
    field :nodes, {:array, :string}, default: []
    field :expected_nodes, {:array, :string}, default: []
    field :node_results, :map, default: %{}
    field :result, :string
    field :canary, :string
    field :error_code, :string
    field :rollback_verified, :boolean
    field :push_to_live_ms, :integer
    field :started_at, :utc_datetime_usec
    field :completed_at, :utc_datetime_usec
    timestamps(updated_at: false)
  end


  def results, do: @results

  def changeset(receipt, attrs) do
    now = DateTime.utc_now()

    attrs =
      attrs
      |> default(:deployment_id, Ecto.UUID.generate())
      |> default(:started_at, now)
      |> default(:completed_at, now)

    receipt
    |> cast(attrs, [
      :repo,
      :sha,
      :target_id,
      :deployment_id,
      :artifact_digest,
      :manifest_digest,
      :modules,
      :nodes,
      :expected_nodes,
      :node_results,
      :result,
      :canary,
      :push_to_live_ms
      :error_code,
      :rollback_verified,
      :push_to_live_ms,
      :started_at,
      :completed_at
    ])
    |> validate_required([
      :repo,
      :sha,
      :target_id,
      :deployment_id,
      :result,
      :started_at,
      :completed_at
    ])
    |> validate_required([:repo, :sha, :target_id, :result])
    |> validate_inclusion(:result, @results)
    |> validate_format(:sha, ~r/^[0-9a-f]{40}$/)
    |> validate_format(:artifact_digest, ~r/^[0-9a-f]{64}$/)
    |> validate_format(:manifest_digest, ~r/^[0-9a-f]{64}$/)
    |> validate_length(:modules, max: 512)
    |> validate_length(:nodes, max: 100)
    |> validate_length(:expected_nodes, max: 100)
    |> validate_length(:canary, max: 255)
    |> validate_length(:error_code, max: 128)
    |> validate_node_results()
    |> unique_constraint(:deployment_id)
    |> check_constraint(:result, name: :forge_deploys_result)
    |> check_constraint(:artifact_digest, name: :forge_deploys_artifact_digest)
    |> check_constraint(:manifest_digest, name: :forge_deploys_manifest_digest)
    |> check_constraint(:node_results, name: :forge_deploys_node_bounds)
  end

  defp default(attrs, key, value) do
    cond do
      Map.has_key?(attrs, key) -> attrs
      Map.has_key?(attrs, to_string(key)) -> attrs
      true -> Map.put(attrs, key, value)
    end
  end

  defp validate_node_results(changeset) do
    validate_change(changeset, :node_results, fn :node_results, results ->
      valid? =
        is_map(results) and map_size(results) <= 100 and
          Enum.all?(results, fn {node, result} ->
            is_binary(node) and byte_size(node) in 1..255 and is_binary(result) and
              byte_size(result) in 1..255
          end)

      if valid?, do: [], else: [node_results: "must contain at most 100 bounded outcomes"]
    end)
  end
end

1	+	`defmodule OpenAgents.Forge.Deployment do`
2	+	`@moduledoc """`
3	+	`Coordinates one all-or-rollback direct deployment across an expected fleet.`
4	+
5	+	`The coordinator snapshots a healthy, revision-consistent membership set,`
6	+	`prepares every node, applies and verifies one canary, applies and verifies`
7	+	`the remaining nodes, rechecks exact membership between phases, and commits`
8	+	`only after every expected participant reports success. Any error triggers an`
9	+	`exact rollback on every participant that issued a token.`
10	+	`"""`
11	+
12	+	`alias OpenAgents.Forge.BuildArtifact`
13	+	`alias OpenAgents.Forge.BuildProtocol`
14	+	`alias OpenAgents.Forge.DeploymentNode`
15	+
16	+	`@default_timeout_ms 15_000`
17	+
18	+	@doc "Run through fleet commit, retaining tokens until `finalize/1`."
19	+	`def run(build, verified, artifact_bytes, opts \\ []) do`
20	+	`deployment_id = Ecto.UUID.generate()`
21	+	`manifest_digest = BuildArtifact.digest(BuildProtocol.canonical_json(verified.manifest))`
22	+
23	+	`base = %{`
24	+	`deployment_id: deployment_id,`
25	+	`target_id: build.target_id,`
26	+	`repo: build.repo,`
27	+	`sha: build.sha,`
28	+	`build_id: build.build_id,`
29	+	`artifact_digest: verified.digest,`
30	+	`manifest_digest: manifest_digest,`
31	+	`started_at: DateTime.utc_now(),`
32	+	`expected_nodes: [],`
33	+	`canary: nil,`
34	+	`tokens: %{},`
35	+	`applied_nodes: MapSet.new(),`
36	+	`node_results: %{}`
37	+	`}`
38	+
39	+	`result =`
40	+	`with {:ok, session} <- snapshot_fleet(base, opts),`
41	+	`{:ok, session} <- prepare_fleet(session, artifact_bytes, opts),`
42	+	`:ok <- stable_membership(session, opts),`
43	+	`{:ok, session} <- apply_canary(session, opts),`
44	+	`{:ok, session} <- verify_canary(session, opts),`
45	+	`:ok <- stable_membership(session, opts),`
46	+	`{:ok, session} <- apply_remainder(session, opts),`
47	+	`:ok <- stable_membership(session, opts),`
48	+	`{:ok, session} <- verify_fleet(session, opts),`
49	+	`:ok <- stable_membership(session, opts),`
50	+	`{:ok, session} <- commit_fleet(session, opts),`
51	+	`:ok <- stable_membership(session, opts) do`
52	+	`{:ok, session}`
53	+	`end`
54	+
55	+	`case result do`
56	+	`{:ok, session} ->`
57	+	`{:ok, public_session(session)}`
58	+
59	+	`{:error, reason, session} ->`
60	+	`rollback_failure(session, reason, opts)`
61	+	`end`
62	+	`end`
63	+
64	+	`@doc "Finalize every committed node and release it into external readiness."`
65	+	`def finalize(session, opts \\ []) do`
66	+	`results = token_fanout(session, :finalize, opts)`
67	+
68	+	`if all_stage_ok?(results),`
69	+	`do: :ok,`
70	+	`else: {:error, {:finalize_failed, result_codes(results)}}`
71	+	`end`
72	+
73	+	`@doc "Roll back a committed-but-not-finalized fleet transaction."`
74	+	`def rollback(session, opts \\ []) do`
75	+	`results = token_fanout(session, :rollback, opts)`
76	+
77	+	`if all_restored?(results),`
78	+	`do: {:ok, Map.new(results, fn {node, _result} -> {to_string(node), "restored"} end)},`
79	+	`else: {:error, result_codes(results)}`
80	+	`end`
81	+
82	+	`defp snapshot_fleet(session, opts) do`
83	+	`nodes = members(opts)`
84	+	`expected_size = Application.fetch_env!(:openagents, :forge_expected_fleet_size)`
85	+
86	+	`cond do`
87	+	`length(nodes) != expected_size ->`
88	+	`{:error, {:fleet_size_mismatch, expected_size, length(nodes)}, session}`
89	+
90	+	`nodes == [] ->`
91	+	`{:error, :empty_fleet, session}`
92	+
93	+	`true ->`
94	+	`results = fanout(nodes, :deployment_health, [session.repo, session.target_id], opts)`
95	+
96	+	`with true <- all_health_ready?(results) or {:error, :fleet_not_ready},`
97	+	`true <- consistent_revisions?(results) or {:error, :fleet_revision_divergent} do`
98	+	`canary = if Node.self() in nodes, do: Node.self(), else: hd(nodes)`
99	+
100	+	`{:ok,`
101	+	`%{`
102	+	`session`
103	+	`\| expected_nodes: nodes,`
104	+	`canary: canary,`
105	+	`node_results: health_results(results)`
106	+	`}}`
107	+	`else`
108	+	`{:error, reason} -> {:error, reason, session_with_results(session, results)}`
109	+	`end`
110	+	`end`
111	+	`end`
112	+
113	+	`defp prepare_fleet(session, artifact_bytes, opts) do`
114	+	`expected_strings = Enum.map(session.expected_nodes, &to_string/1)`
115	+
116	+	`request = %{`
117	+	`artifact_bytes: artifact_bytes,`
118	+	`artifact_digest: session.artifact_digest,`
119	+	`build_id: session.build_id,`
120	+	`deployment_id: session.deployment_id,`
121	+	`expected_nodes: expected_strings,`
122	+	`manifest_digest: session.manifest_digest,`
123	+	`repo: session.repo,`
124	+	`sha: session.sha,`
125	+	`target_id: session.target_id`
126	+	`}`
127	+
128	+	`results = fanout(session.expected_nodes, :prepare, [request], opts)`
129	+
130	+	`tokens =`
131	+	`Enum.reduce(results, %{}, fn`
132	+	`{node, {:ok, {:ok, %{"token" => token}}}}, acc -> Map.put(acc, node, token)`
133	+	`{_node, _result}, acc -> acc`
134	+	`end)`
135	+
136	+	`session = %{`
137	+	`session`
138	+	`\| tokens: tokens,`
139	+	`node_results: merge_results(session, results, "prepared")`
140	+	`}`
141	+
142	+	`if map_size(tokens) == length(session.expected_nodes),`
143	+	`do: {:ok, session},`
144	+	`else: {:error, {:prepare_failed, result_codes(results)}, session}`
145	+	`end`
146	+
147	+	`defp apply_canary(session, opts) do`
148	+	`case call_token(session, session.canary, :apply_candidate, opts) do`
149	+	`{:ok, {:ok, %{"phase" => "applied"}}} ->`
150	+	`{:ok, mark_applied(session, session.canary)}`
151	+
152	+	`result ->`
153	+	`{:error, {:canary_apply_failed, result_code(result)}, session}`
154	+	`end`
155	+	`end`
156	+
157	+	`defp verify_canary(session, opts) do`
158	+	`case call_token(session, session.canary, :verify_candidate, opts) do`
159	+	`{:ok, {:ok, %{"deployment_ready" => true, "revision" => revision}}}`
160	+	`when revision == session.sha ->`
161	+	`{:ok, put_result(session, session.canary, "verified")}`
162	+
163	+	`result ->`
164	+	`{:error, {:canary_verify_failed, result_code(result)}, session}`
165	+	`end`
166	+	`end`
167	+
168	+	`defp apply_remainder(session, opts) do`
169	+	`nodes = Enum.reject(session.expected_nodes, &(&1 == session.canary))`
170	+	`results = fanout_tokens(session, nodes, :apply_candidate, opts)`
171	+
172	+	`applied =`
173	+	`Enum.reduce(results, session.applied_nodes, fn`
174	+	`{node, {:ok, {:ok, %{"phase" => "applied"}}}}, acc -> MapSet.put(acc, node)`
175	+	`{_node, _result}, acc -> acc`
176	+	`end)`
177	+
178	+	`session = %{`
179	+	`session`
180	+	`\| applied_nodes: applied,`
181	+	`node_results: merge_results(session, results, "applied")`
182	+	`}`
183	+
184	+	`if Enum.all?(results, fn {_node, result} ->`
185	+	`match?({:ok, {:ok, %{"phase" => "applied"}}}, result)`
186	+	`end),`
187	+	`do: {:ok, session},`
188	+	`else: {:error, {:fleet_apply_failed, result_codes(results)}, session}`
189	+	`end`
190	+
191	+	`defp verify_fleet(session, opts) do`
192	+	`results = fanout_tokens(session, session.expected_nodes, :verify_candidate, opts)`
193	+
194	+	`verified? =`
195	+	`Enum.all?(results, fn`
196	+	`{_node, {:ok, {:ok, %{"deployment_ready" => true, "revision" => revision}}}} ->`
197	+	`revision == session.sha`
198	+
199	+	`_other ->`
200	+	`false`
201	+	`end)`
202	+
203	+	`session = %{session \| node_results: merge_results(session, results, "verified")}`
204	+
205	+	`if verified?,`
206	+	`do: {:ok, session},`
207	+	`else: {:error, {:fleet_verify_failed, result_codes(results)}, session}`
208	+	`end`
209	+
210	+	`defp commit_fleet(session, opts) do`
211	+	`results = fanout_tokens(session, session.expected_nodes, :commit, opts)`
212	+	`session = %{session \| node_results: merge_results(session, results, "committed")}`
213	+
214	+	`if Enum.all?(results, fn {_node, result} ->`
215	+	`match?({:ok, {:ok, %{"phase" => "committed"}}}, result)`
216	+	`end),`
217	+	`do: {:ok, session},`
218	+	`else: {:error, {:fleet_commit_failed, result_codes(results)}, session}`
219	+	`end`
220	+
221	+	`defp stable_membership(session, opts) do`
222	+	`current = members(opts)`
223	+
224	+	`if current == session.expected_nodes,`
225	+	`do: :ok,`
226	+	`else:`
227	+	`{:error, {:membership_changed, membership_delta(session.expected_nodes, current)},`
228	+	`session}`
229	+	`end`
230	+
231	+	`defp rollback_failure(session, reason, opts) do`
232	+	`results = token_fanout(session, :rollback, opts)`
233	+	`restored? = all_restored?(results)`
234	+
235	+	`node_results =`
236	+	`Map.new(results, fn`
237	+	`{node, {:ok, {:ok, %{"restored" => true}}}} -> {to_string(node), "restored"}`
238	+	`{node, result} -> {to_string(node), "rollback_failed:" <> result_code(result)}`
239	+	`end)`
240	+
241	+	`{:error, failure_outcome(session, reason, restored?, node_results)}`
242	+	`end`
243	+
244	+	`defp failure_outcome(session, reason, restored?, node_results) do`
245	+	`%{`
246	+	`deployment_id: session.deployment_id,`
247	+	`target_id: session.target_id,`
248	+	`repo: session.repo,`
249	+	`sha: session.sha,`
250	+	`build_id: session.build_id,`
251	+	`artifact_digest: session.artifact_digest,`
252	+	`manifest_digest: session.manifest_digest,`
253	+	`expected_nodes: Enum.map(session.expected_nodes, &to_string/1),`
254	+	`canary: if(session.canary, do: to_string(session.canary), else: nil),`
255	+	`nodes:`
256	+	`node_results \|> Enum.map(fn {node, status} -> "#{node}=#{status}" end) \|> Enum.sort(),`
257	+	`node_results: node_results,`
258	+	`result:`
259	+	`if(restored? and MapSet.size(session.applied_nodes) > 0, do: "reverted", else: "failed"),`
260	+	`error_code: safe_code(reason),`
261	+	`rollback_verified: restored?,`
262	+	`started_at: session.started_at`
263	+	`}`
264	+	`end`
265	+
266	+	`defp public_session(session) do`
267	+	`%{`
268	+	`deployment_id: session.deployment_id,`
269	+	`target_id: session.target_id,`
270	+	`repo: session.repo,`
271	+	`sha: session.sha,`
272	+	`build_id: session.build_id,`
273	+	`artifact_digest: session.artifact_digest,`
274	+	`manifest_digest: session.manifest_digest,`
275	+	`expected_nodes: Enum.map(session.expected_nodes, &to_string/1),`
276	+	`canary: to_string(session.canary),`
277	+	`nodes: Enum.map(session.expected_nodes, &(to_string(&1) <> "=committed")),`
278	+	`node_results: Map.new(session.expected_nodes, &{to_string(&1), "committed"}),`
279	+	`result: "live",`
280	+	`error_code: nil,`
281	+	`rollback_verified: nil,`
282	+	`started_at: session.started_at,`
283	+	`tokens: session.tokens,`
284	+	`internal_nodes: session.expected_nodes`
285	+	`}`
286	+	`end`
287	+
288	+	`defp token_fanout(session, phase, opts) do`
289	+	`nodes = Map.get(session, :internal_nodes, session.expected_nodes)`
290	+	`fanout_tokens(session, nodes, phase, opts)`
291	+	`end`
292	+
293	+	`defp fanout_tokens(session, nodes, phase, opts) do`
294	+	`Map.new(nodes, fn node -> {node, call_token(session, node, phase, opts)} end)`
295	+	`end`
296	+
297	+	`defp call_token(session, node, phase, opts) do`
298	+	`token = Map.fetch!(session.tokens, node)`
299	+	`rpc(node, phase, [session.deployment_id, token], opts)`
300	+	`end`
301	+
302	+	`defp fanout(nodes, function, arguments, opts) do`
303	+	`nodes`
304	+	`\|> Task.async_stream(&{&1, rpc(&1, function, arguments, opts)},`
305	+	`ordered: true,`
306	+	`timeout: timeout_ms(opts) + 1_000,`
307	+	`on_timeout: :kill_task,`
308	+	`max_concurrency: max(length(nodes), 1)`
309	+	`)`
310	+	`\|> Enum.zip(nodes)`
311	+	`\|> Map.new(fn`
312	+	`{{:ok, {node, result}}, _expected} -> {node, result}`
313	+	`{{:exit, reason}, node} -> {node, {:error, {:task_exit, reason}}}`
314	+	`end)`
315	+	`end`
316	+
317	+	`defp rpc(node, function, arguments, opts) do`
318	+	`timeout = timeout_ms(opts)`
319	+
320	+	`try do`
321	+	`result =`
322	+	`if node == Node.self() do`
323	+	`apply(DeploymentNode, function, arguments)`
324	+	`else`
325	+	`:erpc.call(node, DeploymentNode, function, arguments, timeout)`
326	+	`end`
327	+
328	+	`{:ok, result}`
329	+	`rescue`
330	+	`error -> {:error, {:exception, safe_code(error)}}`
331	+	`catch`
332	+	`:exit, reason -> {:error, {:exit, safe_code(reason)}}`
333	+	`kind, reason -> {:error, {kind, safe_code(reason)}}`
334	+	`end`
335	+	`end`
336	+
337	+	`defp members(opts) do`
338	+	`provider = Keyword.get(opts, :members, fn -> [Node.self() \| Node.list()] end)`
339	+	`provider.() \|> Enum.uniq() \|> Enum.sort()`
340	+	`end`
341	+
342	+	`defp all_health_ready?(results) do`
343	+	`Enum.all?(results, fn`
344	+	`{_node, {:ok, %{"ready" => true}}} -> true`
345	+	`_other -> false`
346	+	`end)`
347	+	`end`
348	+
349	+	`defp consistent_revisions?(results) do`
350	+	`revisions =`
351	+	`Enum.map(results, fn`
352	+	`{_node, {:ok, %{"revision" => revision}}} -> revision`
353	+	`_other -> nil`
354	+	`end)`
355	+
356	+	`nil not in revisions and length(Enum.uniq(revisions)) == 1`
357	+	`end`
358	+
359	+	`defp health_results(results) do`
360	+	`Map.new(results, fn`
361	+	`{node, {:ok, %{"ready" => true}}} -> {to_string(node), "healthy"}`
362	+	`{node, result} -> {to_string(node), "unhealthy:" <> result_code(result)}`
363	+	`end)`
364	+	`end`
365	+
366	+	`defp all_stage_ok?(results) do`
367	+	`Enum.all?(results, fn {_node, result} -> match?({:ok, {:ok, _response}}, result) end)`
368	+	`end`
369	+
370	+	`defp all_restored?(results) do`
371	+	`results != %{} and`
372	+	`Enum.all?(results, fn`
373	+	`{_node, {:ok, {:ok, %{"restored" => true}}}} -> true`
374	+	`_other -> false`
375	+	`end)`
376	+	`end`
377	+
378	+	`defp mark_applied(session, node) do`
379	+	`session`
380	+	`\|> Map.update!(:applied_nodes, &MapSet.put(&1, node))`
381	+	`\|> put_result(node, "applied")`
382	+	`end`
383	+
384	+	`defp put_result(session, node, result) do`
385	+	`put_in(session, [:node_results, to_string(node)], result)`
386	+	`end`
387	+
388	+	`defp merge_results(session, results, success) do`
389	+	`Enum.reduce(results, session.node_results, fn`
390	+	`{node, {:ok, {:ok, _response}}}, acc -> Map.put(acc, to_string(node), success)`
391	+	`{node, result}, acc -> Map.put(acc, to_string(node), result_code(result))`
392	+	`end)`
393	+	`end`
394	+
395	+	`defp session_with_results(session, results),`
396	+	`do: %{session \| node_results: merge_results(session, results, "ok")}`
397	+
398	+	`defp membership_delta(expected, current) do`
399	+	`%{`
400	+	`missing: Enum.map(expected -- current, &to_string/1),`
401	+	`unexpected: Enum.map(current -- expected, &to_string/1)`
402	+	`}`
403	+	`end`
404	+
405	+	`defp result_codes(results),`
406	+	`do: Map.new(results, fn {node, result} -> {to_string(node), result_code(result)} end)`
407	+
408	+	`defp result_code({:ok, {:ok, _response}}), do: "ok"`
409	+	`defp result_code({:ok, {:error, reason}}), do: safe_code(reason)`
410	+	`defp result_code({:error, reason}), do: safe_code(reason)`
411	+	`defp result_code(other), do: safe_code(other)`
412	+
413	+	`defp safe_code(reason), do: OpenAgents.OperationalLog.code(reason)`
414	+
415	+	`defp timeout_ms(opts),`
416	+	`do:`
417	+	`Keyword.get(`
418	+	`opts,`
419	+	`:timeout_ms,`
420	+	`Application.get_env(:openagents, :forge_deploy_timeout_ms, @default_timeout_ms)`
421	+	`)`
422	+	`end`

1	+	`defmodule OpenAgents.Forge.DeploymentNode do`
2	+	`@moduledoc """`
3	+	`Per-node participant in a transactional direct deployment.`
4	+
5	+	`A participant independently verifies the immutable artifact, snapshots the`
6	+	`exact object code it would replace, and returns an expiring opaque token.`
7	+	`Later phases must present that token and the deployment ID. The participant`
8	+	`keeps external readiness false from prepare through commit. It restores and`
9	+	`verifies the snapshot on any failed or expired transaction, and it remains`
10	+	`out of readiness when exact restoration fails.`
11	+	`"""`
12	+
13	+	`use GenServer`
14	+
15	+	`alias OpenAgents.Forge.BootConverge`
16	+	`alias OpenAgents.Forge.BuildArtifact`
17	+	`alias OpenAgents.Forge.BuildProtocol`
18	+	`alias OpenAgents.Forge.Repos`
19	+	`alias OpenAgents.Forge.Targets`
20	+
21	+	`@default_timeout_ms 15_000`
22	+	`@default_token_ttl_ms 120_000`
23	+	`@maximum_transactions 4`
24	+	`@sweep_interval_ms 1_000`
25	+	`@state_key {__MODULE__, :state}`
26	+	`@request_keys ~w(artifact_bytes artifact_digest build_id deployment_id expected_nodes manifest_digest repo sha target_id)a`
27	+
28	+	`def start_link(opts \\ []) do`
29	+	`GenServer.start_link(__MODULE__, opts, name: __MODULE__)`
30	+	`end`
31	+
32	+	`@doc false`
33	+	`def start(opts \\ []) do`
34	+	`GenServer.start(__MODULE__, opts, name: __MODULE__)`
35	+	`end`
36	+
37	+	`@doc "Prepare one verified deployment and return its expiring token."`
38	+	`def prepare(request), do: call({:prepare, request})`
39	+
40	+	`@doc "Apply the candidate associated with an expiring deployment token."`
41	+	`def apply_candidate(deployment_id, token),`
42	+	`do: call({:phase, :apply, deployment_id, token})`
43	+
44	+	`@doc "Verify candidate object-code identity and the application revision."`
45	+	`def verify_candidate(deployment_id, token),`
46	+	`do: call({:phase, :verify, deployment_id, token})`
47	+
48	+	`@doc "Commit a verified candidate while retaining the rollback fence."`
49	+	`def commit(deployment_id, token), do: call({:phase, :commit, deployment_id, token})`
50	+
51	+	`@doc "Release a fleet-committed candidate into external readiness."`
52	+	`def finalize(deployment_id, token), do: call({:phase, :finalize, deployment_id, token})`
53	+
54	+	`@doc "Restore and verify the exact predeployment object code."`
55	+	`def rollback(deployment_id, token), do: call({:phase, :rollback, deployment_id, token})`
56	+
57	+	`@doc "Install one boot-convergence artifact through the same local verifier."`
58	+	`def install_artifact(request), do: call({:install_artifact, request}, timeout_ms() * 2)`
59	+
60	+	`@doc "Return a bounded, content-free readiness report for this node."`
61	+	`def health do`
62	+	`case Process.whereis(__MODULE__) do`
63	+	`nil -> unavailable_health()`
64	+	`_pid -> call(:health)`
65	+	`end`
66	+	`catch`
67	+	`:exit, _reason -> unavailable_health()`
68	+	`end`
69	+
70	+	`@doc false`
71	+	`def deployment_health(repo, target_id), do: call({:deployment_health, repo, target_id})`
72	+
73	+	`@impl true`
74	+	`def init(opts) do`
75	+	`Process.send_after(self(), :sweep_expired, @sweep_interval_ms)`
76	+
77	+	`recovered =`
78	+	`case :persistent_term.get(@state_key, nil) do`
79	+	`%{schema: 1} = state -> state`
80	+	`_missing_or_old -> %{transactions: %{}, live: nil, divergence: nil}`
81	+	`end`
82	+
83	+	`state =`
84	+	`%{`
85	+	`transactions: recovered.transactions,`
86	+	`live: recovered.live,`
87	+	`divergence: recovered.divergence,`
88	+	`faults: Keyword.get(opts, :faults, %{}),`
89	+	`fault_timeout_ms: Keyword.get(opts, :fault_timeout_ms, timeout_ms() * 2),`
90	+	`notify: Keyword.get(opts, :notify)`
91	+	`}`
92	+
93	+	`{:ok, persist(state)}`
94	+	`end`
95	+
96	+	`@impl true`
97	+	`def handle_call(:health, _from, state), do: {:reply, health_report(state), state}`
98	+
99	+	`def handle_call({:deployment_health, repo, target_id}, _from, state) do`
100	+	`boot_ready? = BootConverge.ready_for_deployment?(repo, target_id)`
101	+	`{:reply, health_report(state, boot_ready?), state}`
102	+	`end`
103	+
104	+	`def handle_call({:prepare, request}, _from, state) do`
105	+	`state = expire_transactions(state)`
106	+
107	+	`with :ok <- run_fault(:prepare, state),`
108	+	`:ok <- capacity_available(state),`
109	+	`{:ok, transaction, response} <- prepare_transaction(request, state) do`
110	+	`notify(state, :prepared)`
111	+
112	+	`state = put_in(state, [:transactions, transaction.token], transaction)`
113	+	`{:reply, {:ok, response}, persist(state)}`
114	+	`else`
115	+	`{:error, reason} -> {:reply, {:error, reason}, persist(state)}`
116	+	`end`
117	+	`end`
118	+
119	+	`def handle_call({:phase, phase, deployment_id, token}, _from, state) do`
120	+	`state = expire_transactions(state)`
121	+
122	+	`with :ok <- run_fault(phase, state),`
123	+	`{:ok, transaction} <- fetch_transaction(state, deployment_id, token),`
124	+	`{:ok, response, state} <- run_phase(phase, transaction, state) do`
125	+	`notify(state, phase)`
126	+	`{:reply, {:ok, response}, persist(state)}`
127	+	`else`
128	+	`{:error, reason, updated_state} -> {:reply, {:error, reason}, persist(updated_state)}`
129	+	`{:error, reason} -> {:reply, {:error, reason}, persist(state)}`
130	+	`end`
131	+	`end`
132	+
133	+	`def handle_call({:install_artifact, request}, _from, state) do`
134	+	`state = expire_transactions(state)`
135	+
136	+	`result =`
137	+	`with :ok <- participant_idle(state),`
138	+	`{:ok, transaction, _prepared} <- prepare_transaction(request, state),`
139	+	`state = put_in(state, [:transactions, transaction.token], transaction),`
140	+	`{:ok, _applied, state} <- run_phase(:apply, transaction, state),`
141	+	`{:ok, _verified, state} <-`
142	+	`run_phase(:verify, state.transactions[transaction.token], state),`
143	+	`{:ok, _committed, state} <-`
144	+	`run_phase(:commit, state.transactions[transaction.token], state),`
145	+	`{:ok, finalized, state} <-`
146	+	`run_phase(:finalize, state.transactions[transaction.token], state) do`
147	+	`{:ok, finalized, state}`
148	+	`else`
149	+	`{:error, reason, failed_state} ->`
150	+	`{:error, reason, rollback_install(request, failed_state)}`
151	+
152	+	`{:error, reason} ->`
153	+	`{:error, reason, state}`
154	+	`end`
155	+
156	+	`case result do`
157	+	`{:ok, response, updated_state} -> {:reply, {:ok, response}, persist(updated_state)}`
158	+	`{:error, reason, updated_state} -> {:reply, {:error, reason}, persist(updated_state)}`
159	+	`end`
160	+	`end`
161	+
162	+	`@impl true`
163	+	`def handle_info(:sweep_expired, state) do`
164	+	`Process.send_after(self(), :sweep_expired, @sweep_interval_ms)`
165	+	`{:noreply, state \|> expire_transactions() \|> persist()}`
166	+	`end`
167	+
168	+	`def handle_info(_message, state), do: {:noreply, state}`
169	+
170	+	`defp call(message, timeout \\ nil) do`
171	+	`GenServer.call(__MODULE__, message, timeout \|\| timeout_ms())`
172	+	`end`
173	+
174	+	`defp prepare_transaction(request, state) do`
175	+	`with :ok <- validate_request(request),`
176	+	`{:ok, verified} <-`
177	+	`BuildArtifact.verify(request.artifact_bytes,`
178	+	`digest: request.artifact_digest,`
179	+	`repo: request.repo,`
180	+	`source_sha: request.sha,`
181	+	`build_id: request.build_id`
182	+	`),`
183	+	`:ok <- verify_manifest_identity(verified, request),`
184	+	`:ok <- verify_direct_candidate(verified),`
185	+	`:ok <- verify_runtime_toolchain(verified.manifest["toolchain"]),`
186	+	`:ok <- cache_artifact(request.artifact_bytes, request.artifact_digest),`
187	+	`candidates = candidates(verified.beams),`
188	+	`{:ok, prior} <- snapshot_prior(candidates, state.live) do`
189	+	`token = Base.url_encode64(:crypto.strong_rand_bytes(32), padding: false)`
190	+	`expires_in_ms = token_ttl_ms()`
191	+
192	+	`transaction = %{`
193	+	`deployment_id: request.deployment_id,`
194	+	`target_id: request.target_id,`
195	+	`repo: request.repo,`
196	+	`sha: request.sha,`
197	+	`artifact_digest: request.artifact_digest,`
198	+	`manifest_digest: request.manifest_digest,`
199	+	`manifest: verified.manifest,`
200	+	`expected_nodes: request.expected_nodes,`
201	+	`candidates: candidates,`
202	+	`prior: prior,`
203	+	`prior_live: state.live,`
204	+	`prior_boot: BootConverge.state(),`
205	+	`token: token,`
206	+	`expires_at: monotonic_ms() + expires_in_ms,`
207	+	`phase: :prepared`
208	+	`}`
209	+
210	+	`response = %{`
211	+	`"token" => token,`
212	+	`"artifact_digest" => request.artifact_digest,`
213	+	`"manifest_digest" => request.manifest_digest,`
214	+	`"prior" => prior_projection(prior),`
215	+	`"expires_in_ms" => expires_in_ms`
216	+	`}`
217	+
218	+	`{:ok, transaction, response}`
219	+	`end`
220	+	`end`
221	+
222	+	`defp run_phase(:apply, %{phase: :prepared} = transaction, state) do`
223	+	`case load_candidates(transaction.candidates) do`
224	+	`:ok ->`
225	+	`updated = %{transaction \| phase: :applied}`
226	+	`{:ok, %{"phase" => "applied"}, put_transaction(state, updated)}`
227	+
228	+	`{:error, reason} ->`
229	+	`rollback_after_failure(transaction, state, {:apply_failed, reason})`
230	+	`end`
231	+	`end`
232	+
233	+	`defp run_phase(:verify, %{phase: :applied} = transaction, state) do`
234	+	`case verify_candidates(transaction) do`
235	+	`:ok ->`
236	+	`updated = %{transaction \| phase: :verified}`
237	+
238	+	`{:ok,`
239	+	`%{`
240	+	`"phase" => "verified",`
241	+	`"revision" => transaction.sha,`
242	+	`"application_version" => transaction.manifest["toolchain"]["application_version"],`
243	+	`"deployment_ready" => true`
244	+	`}, put_transaction(state, updated)}`
245	+
246	+	`{:error, reason} ->`
247	+	`rollback_after_failure(transaction, state, {:verification_failed, reason})`
248	+	`end`
249	+	`end`
250	+
251	+	`defp run_phase(:verify, %{phase: :verified} = transaction, state) do`
252	+	`case verify_candidates(transaction) do`
253	+	`:ok ->`
254	+	`{:ok,`
255	+	`%{`
256	+	`"phase" => "verified",`
257	+	`"revision" => transaction.sha,`
258	+	`"application_version" => transaction.manifest["toolchain"]["application_version"],`
259	+	`"deployment_ready" => true`
260	+	`}, state}`
261	+
262	+	`{:error, reason} ->`
263	+	`rollback_after_failure(transaction, state, {:verification_failed, reason})`
264	+	`end`
265	+	`end`
266	+
267	+	`defp run_phase(:commit, %{phase: :verified} = transaction, state) do`
268	+	`updated = %{transaction \| phase: :committed}`
269	+	`state = state \|> put_transaction(updated) \|> Map.put(:live, live_identity(transaction))`
270	+	`{:ok, %{"phase" => "committed", "revision" => transaction.sha}, state}`
271	+	`end`
272	+
273	+	`defp run_phase(:finalize, %{phase: :committed} = transaction, state) do`
274	+	`state = %{`
275	+	`state`
276	+	`\| transactions: Map.delete(state.transactions, transaction.token),`
277	+	`divergence: nil`
278	+	`}`
279	+
280	+	`BootConverge.mark_converged(live_identity(transaction))`
281	+
282	+	`{:ok,`
283	+	`%{`
284	+	`"phase" => "live",`
285	+	`"revision" => transaction.sha,`
286	+	`"artifact_digest" => transaction.artifact_digest`
287	+	`}, state}`
288	+	`end`
289	+
290	+	`defp run_phase(:rollback, transaction, state) do`
291	+	`case restore_and_verify(transaction.prior) do`
292	+	`:ok ->`
293	+	`state = %{`
294	+	`state`
295	+	`\| transactions: Map.delete(state.transactions, transaction.token),`
296	+	`live: transaction.prior_live,`
297	+	`divergence: nil`
298	+	`}`
299	+
300	+	`BootConverge.restore_state(transaction.prior_boot)`
301	+
302	+	`{:ok, %{"phase" => "restored", "restored" => true}, state}`
303	+
304	+	`{:error, reason} ->`
305	+	`state = %{state \| divergence: "rollback_unverified"}`
306	+	`{:error, {:rollback_failed, reason}, state}`
307	+	`end`
308	+	`end`
309	+
310	+	`defp run_phase(phase, transaction, _state) do`
311	+	`{:error, {:invalid_phase, transaction.phase, phase}}`
312	+	`end`
313	+
314	+	`defp rollback_after_failure(transaction, state, reason) do`
315	+	`case restore_and_verify(transaction.prior) do`
316	+	`:ok ->`
317	+	`state = %{`
318	+	`state`
319	+	`\| transactions: Map.delete(state.transactions, transaction.token),`
320	+	`live: transaction.prior_live`
321	+	`}`
322	+
323	+	`{:error, reason, state}`
324	+
325	+	`{:error, rollback_reason} ->`
326	+	`state = %{state \| divergence: "rollback_unverified"}`
327	+	`{:error, {:rollback_failed, rollback_reason}, state}`
328	+	`end`
329	+	`end`
330	+
331	+	`defp rollback_install(request, state) do`
332	+	`case Enum.find(state.transactions, fn {_token, transaction} ->`
333	+	`transaction.deployment_id == request.deployment_id`
334	+	`end) do`
335	+	`{_token, transaction} ->`
336	+	`case run_phase(:rollback, transaction, state) do`
337	+	`{:ok, _response, updated_state} -> updated_state`
338	+	`{:error, _reason, updated_state} -> updated_state`
339	+	`end`
340	+
341	+	`nil ->`
342	+	`state`
343	+	`end`
344	+	`end`
345	+
346	+	`defp fetch_transaction(state, deployment_id, token) do`
347	+	`case state.transactions[token] do`
348	+	`%{deployment_id: ^deployment_id} = transaction -> {:ok, transaction}`
349	+	`nil -> {:error, :unknown_or_expired_token}`
350	+	`_other -> {:error, :deployment_token_mismatch}`
351	+	`end`
352	+	`end`
353	+
354	+	`defp capacity_available(state) do`
355	+	`if map_size(state.transactions) < @maximum_transactions,`
356	+	`do: :ok,`
357	+	`else: {:error, :deployment_capacity_reached}`
358	+	`end`
359	+
360	+	`defp participant_idle(%{transactions: transactions}) when map_size(transactions) == 0, do: :ok`
361	+	`defp participant_idle(_state), do: {:error, :deployment_in_progress}`
362	+
363	+	`defp validate_request(request) when is_map(request) do`
364	+	`with true <-`
365	+	`Map.keys(request) \|> Enum.sort() == Enum.sort(@request_keys) or`
366	+	`{:error, :unexpected_deployment_fields},`
367	+	`:ok <- uuid(request.deployment_id, :invalid_deployment_id),`
368	+	`:ok <- uuid(request.target_id, :invalid_target_id),`
369	+	`:ok <- uuid(request.build_id, :invalid_build_id),`
370	+	`true <- valid_sha?(request.sha) or {:error, :invalid_source_sha},`
371	+	`true <- valid_digest?(request.artifact_digest) or {:error, :invalid_artifact_digest},`
372	+	`true <- valid_digest?(request.manifest_digest) or {:error, :invalid_manifest_digest},`
373	+	`true <- is_binary(request.repo) or {:error, :invalid_repo},`
374	+	`true <- is_binary(request.artifact_bytes) or {:error, :invalid_artifact},`
375	+	`true <-`
376	+	`valid_expected_nodes?(request.expected_nodes) or {:error, :invalid_expected_nodes} do`
377	+	`:ok`
378	+	`else`
379	+	`{:error, _reason} = error -> error`
380	+	`end`
381	+	`end`
382	+
383	+	`defp validate_request(_request), do: {:error, :invalid_deployment_request}`
384	+
385	+	`defp verify_manifest_identity(verified, request) do`
386	+	`digest = BuildArtifact.digest(BuildProtocol.canonical_json(verified.manifest))`
387	+
388	+	`if digest == request.manifest_digest,`
389	+	`do: :ok,`
390	+	`else: {:error, :manifest_digest_mismatch}`
391	+	`end`
392	+
393	+	`defp verify_direct_candidate(verified) do`
394	+	`allowlist = Application.fetch_env!(:openagents, :forge_hot_load_allowlist)`
395	+
396	+	`offenders =`
397	+	`Enum.reject(verified.modules, &OpenAgents.Forge.HotLoader.allowlisted?(&1, allowlist))`
398	+
399	+	`cond do`
400	+	`verified.manifest["classification"] != "direct_candidate" ->`
401	+	`{:error, :artifact_not_direct}`
402	+
403	+	`offenders != [] ->`
404	+	`{:error, :module_not_allowlisted}`
405	+
406	+	`true ->`
407	+	`:ok`
408	+	`end`
409	+	`end`
410	+
411	+	`defp verify_runtime_toolchain(toolchain) do`
412	+	`expected = %{`
413	+	`"elixir" => System.version(),`
414	+	`"otp" => System.otp_release(),`
415	+	`"erts" => to_string(:erlang.system_info(:version)),`
416	+	`"application_version" => to_string(Application.spec(:openagents, :vsn) \|\| "unknown")`
417	+	`}`
418	+
419	+	`if Enum.all?(expected, fn {key, value} -> toolchain[key] == value end),`
420	+	`do: :ok,`
421	+	`else: {:error, :runtime_toolchain_mismatch}`
422	+	`end`
423	+
424	+	`defp candidates(beams) do`
425	+	`Enum.map(beams, fn %{module: module, binary: binary} ->`
426	+	`%{`
427	+	`module: BuildArtifact.module_atom(module),`
428	+	`name: module,`
429	+	`binary: binary,`
430	+	`digest: BuildArtifact.digest(binary),`
431	+	`md5: beam_md5(binary)`
432	+	`}`
433	+	`end)`
434	+	`end`
435	+
436	+	`defp snapshot_prior(candidates, live) do`
437	+	`Enum.reduce_while(candidates, {:ok, %{}}, fn %{module: module, name: name}, {:ok, acc} ->`
438	+	`case prior_object(module, name, live) do`
439	+	`{:ok, object} ->`
440	+	`{:cont, {:ok, Map.put(acc, name, %{module: module, object: object})}}`
441	+
442	+	`{:error, reason} ->`
443	+	`{:halt, {:error, reason}}`
444	+	`end`
445	+	`end)`
446	+	`end`
447	+
448	+	`defp prior_object(module, name, live) do`
449	+	`case :code.get_object_code(module) do`
450	+	`{^module, binary, file} ->`
451	+	`{:ok,`
452	+	`%{`
453	+	`binary: binary,`
454	+	`file: file,`
455	+	`digest: BuildArtifact.digest(binary),`
456	+	`md5: beam_md5(binary)`
457	+	`}}`
458	+
459	+	`:error ->`
460	+	`cond do`
461	+	`:code.is_loaded(module) == false ->`
462	+	`{:ok, :absent}`
463	+
464	+	`is_map(live) and is_map(live[:objects]) and is_map(live.objects[name]) ->`
465	+	`{:ok, live.objects[name]}`
466	+
467	+	`true ->`
468	+	`{:error, :prior_object_code_unavailable}`
469	+	`end`
470	+	`end`
471	+	`end`
472	+
473	+	`defp prior_projection(prior) do`
474	+	`prior`
475	+	`\|> Enum.map(fn`
476	+	`{name, %{object: :absent}} ->`
477	+	`%{"module" => name, "state" => "absent"}`
478	+
479	+	`{name, %{object: object}} ->`
480	+	`%{"module" => name, "state" => "present", "sha256" => object.digest}`
481	+	`end)`
482	+	`\|> Enum.sort_by(& &1["module"])`
483	+	`end`
484	+
485	+	`defp load_candidates(candidates) do`
486	+	`Enum.reduce_while(candidates, :ok, fn candidate, :ok ->`
487	+	`case :code.load_binary(candidate.module, ~c"forge-transaction", candidate.binary) do`
488	+	`{:module, module} when module == candidate.module -> {:cont, :ok}`
489	+	`{:error, reason} -> {:halt, {:error, reason}}`
490	+	`end`
491	+	`end)`
492	+	`end`
493	+
494	+	`defp verify_candidates(transaction) do`
495	+	`with true <-`
496	+	`Enum.all?(transaction.candidates, &candidate_loaded?/1) or`
497	+	`{:error, :candidate_object_code_mismatch},`
498	+	`true <-`
499	+	`Enum.all?(transaction.candidates, &smoke_ok?/1) or`
500	+	`{:error, :candidate_smoke_failed} do`
501	+	`:ok`
502	+	`else`
503	+	`{:error, _reason} = error -> error`
504	+	`end`
505	+	`end`
506	+
507	+	`defp candidate_loaded?(candidate) do`
508	+	`:code.is_loaded(candidate.module) != false and`
509	+	`apply(candidate.module, :module_info, [:md5]) == candidate.md5`
510	+	`rescue`
511	+	`_error -> false`
512	+	`end`
513	+
514	+	`defp smoke_ok?(candidate) do`
515	+	`Code.ensure_loaded?(candidate.module) and`
516	+	`(not function_exported?(candidate.module, :revision, 0) or`
517	+	`is_binary(candidate.module.revision()))`
518	+	`rescue`
519	+	`_error -> false`
520	+	`end`
521	+
522	+	`defp restore_and_verify(prior) do`
523	+	`with :ok <- restore_prior(prior), :ok <- verify_prior(prior), do: :ok`
524	+	`end`
525	+
526	+	`defp restore_prior(prior) do`
527	+	`Enum.reduce_while(prior, :ok, fn`
528	+	`{_name, %{module: module, object: :absent}}, :ok ->`
529	+	`:code.purge(module)`
530	+	`:code.delete(module)`
531	+	`:code.purge(module)`
532	+	`{:cont, :ok}`
533	+
534	+	`{_name, %{module: module, object: object}}, :ok ->`
535	+	`case :code.load_binary(module, object.file, object.binary) do`
536	+	`{:module, ^module} -> {:cont, :ok}`
537	+	`{:error, reason} -> {:halt, {:error, {:restore_load_failed, reason}}}`
538	+	`end`
539	+	`end)`
540	+	`end`
541	+
542	+	`defp verify_prior(prior) do`
543	+	`Enum.reduce_while(prior, :ok, fn`
544	+	`{_name, %{module: module, object: :absent}}, :ok ->`
545	+	`if :code.is_loaded(module) == false,`
546	+	`do: {:cont, :ok},`
547	+	`else: {:halt, {:error, :absent_module_remained_loaded}}`
548	+
549	+	`{_name, %{module: module, object: object}}, :ok ->`
550	+	`if :code.is_loaded(module) != false and`
551	+	`apply(module, :module_info, [:md5]) == object.md5,`
552	+	`do: {:cont, :ok},`
553	+	`else: {:halt, {:error, :prior_object_code_mismatch}}`
554	+	`end)`
555	+	`rescue`
556	+	`_error -> {:error, :prior_object_code_missing}`
557	+	`end`
558	+
559	+	`defp cache_artifact(bytes, digest) do`
560	+	`path = Path.join([Repos.data_dir(), "beams", digest <> ".tar"])`
561	+
562	+	`with :ok <- File.mkdir_p(Path.dirname(path)) do`
563	+	`case BuildProtocol.atomic_write(path, bytes) do`
564	+	`:ok ->`
565	+	`:ok`
566	+
567	+	`{:error, :destination_exists} ->`
568	+	`with {:ok, existing} <- File.read(path),`
569	+	`true <- BuildArtifact.digest(existing) == digest or {:error, :digest_collision} do`
570	+	`:ok`
571	+	`end`
572	+
573	+	`{:error, reason} ->`
574	+	`{:error, {:artifact_cache_write_failed, reason}}`
575	+	`end`
576	+	`else`
577	+	`{:error, reason} -> {:error, {:artifact_cache_directory_failed, reason}}`
578	+	`end`
579	+	`end`
580	+
581	+	`defp expire_transactions(state) do`
582	+	`now = monotonic_ms()`
583	+
584	+	`Enum.reduce(state.transactions, state, fn {_token, transaction}, acc ->`
585	+	`if transaction.expires_at <= now do`
586	+	`expire_transaction(transaction, acc)`
587	+	`else`
588	+	`acc`
589	+	`end`
590	+	`end)`
591	+	`end`
592	+
593	+	`defp expire_transaction(%{phase: :committed} = transaction, state) do`
594	+	`case committed_authority(transaction) do`
595	+	`:candidate_live ->`
596	+	`phase_state(:finalize, transaction, state)`
597	+
598	+	`:candidate_not_live ->`
599	+	`phase_state(:rollback, transaction, state)`
600	+
601	+	`:pending ->`
602	+	`deferred = %{transaction \| expires_at: monotonic_ms() + retry_delay_ms()}`
603	+	`put_transaction(state, deferred)`
604	+
605	+	`:authority_unavailable ->`
606	+	`deferred = %{transaction \| expires_at: monotonic_ms() + retry_delay_ms()}`
607	+
608	+	`state`
609	+	`\|> put_transaction(deferred)`
610	+	`\|> Map.put(:divergence, "commit_authority_unavailable")`
611	+	`end`
612	+	`end`
613	+
614	+	`defp expire_transaction(transaction, state), do: phase_state(:rollback, transaction, state)`
615	+
616	+	`defp phase_state(phase, transaction, state) do`
617	+	`case run_phase(phase, transaction, state) do`
618	+	`{:ok, _response, updated_state} -> updated_state`
619	+	`{:error, _reason, updated_state} -> updated_state`
620	+	`{:error, _reason} -> updated_state_without(state, transaction.token)`
621	+	`end`
622	+	`end`
623	+
624	+	`defp committed_authority(transaction) do`
625	+	`Targets.deployment_authority(`
626	+	`transaction.target_id,`
627	+	`transaction.deployment_id,`
628	+	`transaction.artifact_digest`
629	+	`)`
630	+	`rescue`
631	+	`_error -> :authority_unavailable`
632	+	`catch`
633	+	`_kind, _reason -> :authority_unavailable`
634	+	`end`
635	+
636	+	`defp health_report(state, boot_ready? \\ nil) do`
637	+	`participant =`
638	+	`cond do`
639	+	`state.divergence ->`
640	+	`default_health(%{"ready" => false, "reason" => state.divergence})`
641	+
642	+	`map_size(state.transactions) > 0 ->`
643	+	`phases =`
644	+	`state.transactions \|> Map.values() \|> Enum.map(&to_string(&1.phase)) \|> Enum.uniq()`
645	+
646	+	`default_health(%{`
647	+	`"ready" => false,`
648	+	`"reason" => "deployment_in_progress",`
649	+	`"phase" => phases \|> Enum.sort() \|> List.first()`
650	+	`})`
651	+
652	+	`state.live ->`
653	+	`default_health(%{`
654	+	`"ready" => true,`
655	+	`"reason" => "committed",`
656	+	`"revision" => state.live.sha,`
657	+	`"artifact_digest" => state.live.artifact_digest`
658	+	`})`
659	+
660	+	`true ->`
661	+	`default_health()`
662	+	`end`
663	+
664	+	`participant = Map.put(participant, "participant_ready", participant["ready"])`
665	+
666	+	`boot_ready? = if is_boolean(boot_ready?), do: boot_ready?, else: BootConverge.ready?()`
667	+
668	+	`if participant["ready"] and not boot_ready? do`
669	+	`%{participant \| "ready" => false, "reason" => "boot_not_converged"}`
670	+	`else`
671	+	`participant`
672	+	`end`
673	+	`end`
674	+
675	+	`defp default_health(overrides \\ %{}) do`
676	+	`Map.merge(`
677	+	`%{`
678	+	`"schema" => "openagents.forge.deployment-node.v1",`
679	+	`"ready" => true,`
680	+	`"participant_ready" => true,`
681	+	`"reason" => "idle",`
682	+	`"phase" => "idle",`
683	+	`"revision" => OpenAgents.BuildInfo.revision(),`
684	+	`"artifact_digest" => nil`
685	+	`},`
686	+	`overrides`
687	+	`)`
688	+	`end`
689	+
690	+	`defp unavailable_health do`
691	+	`default_health(%{`
692	+	`"ready" => false,`
693	+	`"participant_ready" => false,`
694	+	`"reason" => "participant_unavailable"`
695	+	`})`
696	+	`end`
697	+
698	+	`defp live_identity(transaction) do`
699	+	`%{`
700	+	`repo: transaction.repo,`
701	+	`sha: transaction.sha,`
702	+	`target_id: transaction.target_id,`
703	+	`build_id: transaction.manifest["build_id"],`
704	+	`artifact_digest: transaction.artifact_digest,`
705	+	`manifest_digest: transaction.manifest_digest,`
706	+	`modules: length(transaction.candidates),`
707	+	`objects:`
708	+	`Map.new(transaction.candidates, fn candidate ->`
709	+	`{candidate.name,`
710	+	`%{`
711	+	`binary: candidate.binary,`
712	+	`file: ~c"forge-transaction",`
713	+	`digest: candidate.digest,`
714	+	`md5: candidate.md5`
715	+	`}}`
716	+	`end)`
717	+	`}`
718	+	`end`
719	+
720	+	`defp beam_md5(binary) do`
721	+	`{:ok, {_module, md5}} = :beam_lib.md5(binary)`
722	+	`md5`
723	+	`end`
724	+
725	+	`defp put_transaction(state, transaction),`
726	+	`do: put_in(state, [:transactions, transaction.token], transaction)`
727	+
728	+	`defp updated_state_without(state, token),`
729	+	`do: %{state \| transactions: Map.delete(state.transactions, token)}`
730	+
731	+	`defp run_fault(stage, state) do`
732	+	`case Map.get(state.faults, stage) do`
733	+	`nil ->`
734	+	`:ok`
735	+
736	+	`:error ->`
737	+	`{:error, :injected_failure}`
738	+
739	+	`:timeout ->`
740	+	`receive do`
741	+	`:release_injected_timeout -> :ok`
742	+	`after`
743	+	`state.fault_timeout_ms -> {:error, :injected_timeout}`
744	+	`end`
745	+
746	+	`other ->`
747	+	`{:error, {:invalid_injected_fault, other}}`
748	+	`end`
749	+	`end`
750	+
751	+	`defp notify(%{notify: pid}, stage) when is_pid(pid),`
752	+	`do: send(pid, {:forge_deployment_node, node(), stage})`
753	+
754	+	`defp notify(_state, _stage), do: :ok`
755	+
756	+	`defp uuid(value, reason) do`
757	+	`case Ecto.UUID.cast(value) do`
758	+	`{:ok, _uuid} -> :ok`
759	+	`:error -> {:error, reason}`
760	+	`end`
761	+	`end`
762	+
763	+	`defp valid_sha?(value), do: is_binary(value) and Regex.match?(~r/^[0-9a-f]{40}$/, value)`
764	+
765	+	`defp valid_digest?(value),`
766	+	`do: is_binary(value) and Regex.match?(~r/^[0-9a-f]{64}$/, value)`
767	+
768	+	`defp valid_expected_nodes?(nodes) when is_list(nodes) and length(nodes) in 1..100 do`
769	+	`nodes == Enum.sort(Enum.uniq(nodes)) and`
770	+	`Enum.all?(nodes, &(is_binary(&1) and byte_size(&1) in 1..255))`
771	+	`end`
772	+
773	+	`defp valid_expected_nodes?(_nodes), do: false`
774	+
775	+	`defp monotonic_ms, do: System.monotonic_time(:millisecond)`
776	+
777	+	`defp timeout_ms,`
778	+	`do: Application.get_env(:openagents, :forge_deploy_timeout_ms, @default_timeout_ms)`
779	+
780	+	`defp token_ttl_ms,`
781	+	`do: Application.get_env(:openagents, :forge_deploy_token_ttl_ms, @default_token_ttl_ms)`
782	+
783	+	`defp retry_delay_ms,`
784	+	`do: Application.get_env(:openagents, :forge_boot_retry_min_ms, 1_000)`
785	+
786	+	`defp persist(state) do`
787	+	`:persistent_term.put(@state_key, %{`
788	+	`schema: 1,`
789	+	`transactions: state.transactions,`
790	+	`live: state.live,`
791	+	`divergence: state.divergence`
792	+	`})`
793	+
794	+	`state`
795	+	`end`
796	+	`end`

defmodule OpenAgents.Forge.HotLoader do
  @moduledoc """
  Hot-load lane (roadmap P4): applies built beam artifacts to the running
  fleet without a restart.

  Listens on `forge:builds` for build-ready broadcasts and, serially per
  build: refuses any artifact touching a module off the operator-owned
  hot-load allowlist (honest `needs_rolling_replace`, never a partial load),
  canaries the load on the local node with revert-on-failure, then
  `:erpc.multicall`s the rest of the fleet. Every outcome — `live`,
  `reverted`, `needs_rolling_replace`, `failed` — lands as a
  `forge_deploys` receipt including the measured push→live duration, and
  the target row advances honestly at each step.
  Coordinates immutable build artifacts through the transactional direct-load
  lane. The coordinator refuses structural or off-allowlist changes before
  deployment, then delegates prepare, canary, fleet apply, verification,
  commit, and exact rollback to `OpenAgents.Forge.Deployment`.
  """

  use GenServer

  require Logger

  alias OpenAgents.Forge.BuildArtifact
  alias OpenAgents.Forge.Deployment
  alias OpenAgents.Forge.DeployReceipt
  alias OpenAgents.Forge.PushReceipt
  alias OpenAgents.Forge.Targets

  @builds_topic "forge:builds"
  @deploys_topic "forge:deploys"
  @default_allowlist ["OpenAgents.Scratch.", "OpenAgents.BuildInfo"]
  @fleet_timeout_ms 15_000

  # ── api ──────────────────────────────────────────────────────────────────


    GenServer.start_link(__MODULE__, opts, name: __MODULE__)
  end

  @doc """
  Load a set of `{module, beam_binary}` pairs on this node. Public because
  the canary node calls it on the rest of the fleet via `:erpc.multicall`
  (no remote revert logic in v0 — per-node results are recorded instead).
  """
  def load_beams(beams) do
    Enum.map(beams, fn {mod, binary} ->
      case :code.load_binary(mod, ~c"forge-hot-load", binary) do
        {:module, ^mod} -> {mod, :ok}
        {:error, reason} -> {mod, {:error, reason}}
      end
    end)
  end

  @doc """
  Whether `module_name` may be hot-loaded under `allowlist`. An entry ending
  in `.` is a prefix (`"OpenAgents.Scratch."`); any other entry is an exact


  defp handle_build(%{repo: repo, sha: sha, target_id: target_id, modules: modules} = build) do
    if File.exists?(build.artifact) do
      with {:ok, verified} <-
             BuildArtifact.verify_file(build.artifact,
      with {:ok, artifact_bytes} <- File.read(build.artifact),
           {:ok, verified} <-
             BuildArtifact.verify(artifact_bytes,
               digest: Map.get(build, :artifact_digest),
               repo: repo,
               source_sha: sha,

           true <-
             is_nil(Map.get(build, :manifest)) or Map.get(build, :manifest) == verified.manifest or
               {:error, :declared_manifest_mismatch} do
        route_verified(build, verified)
        route_verified(build, verified, artifact_bytes)
      else
        {:error, reason} -> fail_verified_build(build, reason)
      end

    :refused -> :ok
  end

  defp route_verified(build, verified) do
  defp route_verified(build, verified, artifact_bytes) do
    allowlist = Application.get_env(:openagents, :forge_hot_load_allowlist, @default_allowlist)
    offending = Enum.reject(verified.modules, &allowlisted?(&1, allowlist))


        route_rolling(build, Enum.map(offending, &"off_allowlist:#{&1}"))

      true ->
        deploy(build, verified)
        build =
          build
          |> Map.put(:artifact_digest, verified.digest)
          |> Map.put(:build_id, verified.manifest["build_id"])
          |> Map.put(:manifest, verified.manifest)

        deploy(build, verified, artifact_bytes)
    end
  end


    broadcast_deploy(repo, sha, "failed")
  end

  defp deploy(%{repo: repo, sha: sha, target_id: target_id, modules: modules}, verified) do
    case advance(target_id, "deploying") do
      :ok -> :ok
      :error -> throw(:refused)
  defp deploy(%{target_id: target_id} = build, verified, bytes) do
    case Targets.begin_deployment(target_id) do
      {:ok, _target} ->
        run_transaction(build, verified, bytes)

      {:error, reason} ->
        Logger.warning("forge_deployment_refused code=#{OpenAgents.OperationalLog.code(reason)}")
    end
  rescue
    error ->
      fail_started_deployment(build, %{error_code: OpenAgents.OperationalLog.code(error)})
  catch
    kind, reason ->
      fail_started_deployment(build, %{error_code: OpenAgents.OperationalLog.code({kind, reason})})
  end

  defp run_transaction(build, verified, bytes) do
    case Deployment.run(build, verified, bytes) do
      {:ok, session} -> commit_live(build, session)
      {:error, outcome} -> finish_failed_transaction(build, outcome)
    end
  end

    # Atom creation happens only here, after the full tar and manifest have
    # passed every bounded identity and classification check.
    beams =
      Enum.map(verified.beams, fn %{module: module, binary: binary} ->
        {BuildArtifact.module_atom(module), binary}
      end)
  defp commit_live(build, session) do
    receipt = receipt_attributes(build, session, "ok", push_to_live_ms(build.repo, build.sha))

    case canary_load(beams) do
      :ok ->
        nodes = fleet_load(beams)
        advance(target_id, "live", %{"modules" => modules})
        push_ms = push_to_live_ms(repo, sha)
        insert_receipt(repo, sha, target_id, modules, nodes, "live", "ok", push_ms)
        broadcast_deploy(repo, sha, "live")
    case Targets.finish_deployment(
           build.target_id,
           "live",
           live_details(build, session),
           receipt
         ) do
      {:ok, _committed} ->
        case Deployment.finalize(session) do
          :ok ->
            :ok

          {:error, reason} ->
            Logger.error(
              "forge_deployment_finalize_failed code=#{OpenAgents.OperationalLog.code(reason)}"
            )
        end

        broadcast_deploy(build.repo, build.sha, "live")

      {:error, reason} ->
        advance(target_id, "reverted", %{"error" => bounded(reason)})
        insert_receipt(repo, sha, target_id, modules, [], "reverted", bounded(reason), nil)
        broadcast_deploy(repo, sha, "reverted")
        rollback_after_database_failure(build, session, reason)
    end
  end

  defp rollback_after_database_failure(build, session, database_reason) do
    case Deployment.rollback(session) do
      {:ok, node_results} ->
        outcome =
          session
          |> Map.merge(%{
            result: "reverted",
            rollback_verified: true,
            error_code: OpenAgents.OperationalLog.code(database_reason),
            node_results: node_results,
            nodes: node_lines(node_results)
          })

        finish_failed_transaction(build, outcome)

      {:error, node_results} ->
        outcome =
          session
          |> Map.merge(%{
            result: "failed",
            rollback_verified: false,
            error_code: "database_commit_and_rollback_failed",
            node_results: node_results,
            nodes: node_lines(node_results)
          })

        finish_failed_transaction(build, outcome)
    end
  end

  defp finish_failed_transaction(build, outcome) do
    status = if outcome.result == "reverted", do: "reverted", else: "failed"
    details = %{"error_code" => outcome.error_code, "modules" => build.modules}
    receipt = receipt_attributes(build, outcome, bounded(outcome.error_code), nil)

    case Targets.finish_deployment(build.target_id, status, details, receipt) do
      {:ok, _committed} ->
        broadcast_deploy(build.repo, build.sha, status)

      {:error, reason} ->
        Logger.error(
          "forge_deployment_receipt_failed code=#{OpenAgents.OperationalLog.code(reason)}"
        )
    end
  end

  defp fail_started_deployment(build, outcome) do
    defaults = %{
      deployment_id: Ecto.UUID.generate(),
      artifact_digest: Map.get(build, :artifact_digest),
      manifest_digest: manifest_digest(Map.get(build, :manifest)),
      expected_nodes: [],
      node_results: %{},
      nodes: [],
      canary: nil,
      rollback_verified: false,
      started_at: DateTime.utc_now(),
      result: "failed"
    }

    finish_failed_transaction(build, Map.merge(defaults, outcome))
  end

  # Every terminal outcome is announced on the deploys topic — live,
  # reverted, needs_rolling_replace, and failed alike.
  defp broadcast_deploy(repo, sha, result) do

    end
  end

  # ── canary (local node, revert on any failure) ───────────────────────────

  defp canary_load(beams), do: canary_load(beams, [])

  defp canary_load([], loaded) do
    case smoke_check(loaded) do
      :ok ->
        :ok

      {:error, reason} ->
        revert(loaded)
        {:error, reason}
    end
  end

  defp canary_load([{mod, binary} | rest], loaded) do
    # Keep the currently loaded object code so a later failure can revert.
    previous = :code.get_object_code(mod)

    case :code.load_binary(mod, ~c"forge-hot-load", binary) do
      {:module, ^mod} ->
        canary_load(rest, [{mod, previous} | loaded])

      {:error, reason} ->
        revert(loaded)
        {:error, "canary load of #{inspect(mod)} failed: #{inspect(reason)}"}
    end
  end

  defp smoke_check(loaded) do
    Enum.reduce_while(loaded, :ok, fn {mod, _previous}, :ok ->
      cond do
        not Code.ensure_loaded?(mod) ->
          {:halt, {:error, "smoke check: #{inspect(mod)} not loaded"}}

        function_exported?(mod, :revision, 0) and not revision_binary?(mod) ->
          {:halt, {:error, "smoke check: #{inspect(mod)}.revision/0 did not return a binary"}}

        true ->
          {:cont, :ok}
      end
    end)
  end

  defp revision_binary?(mod) do
    is_binary(mod.revision())
  rescue
    _ -> false
  end

  defp revert(loaded) do
    Enum.each(loaded, fn
      {mod, {_mod, binary, file}} ->
        :code.load_binary(mod, file, binary)

      {mod, :error} ->
        # Was not loaded before this deploy: remove it entirely.
        :code.purge(mod)
        :code.delete(mod)
    end)
  end

  # ── fleet ────────────────────────────────────────────────────────────────

  defp fleet_load(beams) do
    remote =
      case Node.list() do
        [] ->
          []

        nodes ->
          nodes
          |> Enum.zip(:erpc.multicall(nodes, __MODULE__, :load_beams, [beams], @fleet_timeout_ms))
          |> Enum.map(fn
            {node, {:ok, results}} ->
              if Enum.all?(results, &match?({_mod, :ok}, &1)),
                do: "#{node}=ok",
                else: "#{node}=error"

            {node, _failure} ->
              "#{node}=error"
          end)
      end

    ["#{Node.self()}=ok" | remote]
  end

  # ── receipts ─────────────────────────────────────────────────────────────

  # Push→live duration: the push receipt whose refs advanced some ref to

345	329		`:error`
346	330		`end`
347	331
348		-	`defp advance(target_id, status, details \\ %{}) do`
	332	+	`defp receipt_attributes(build, outcome, canary, push_ms) do`
	333	+	`%{`
	334	+	`deployment_id: outcome.deployment_id,`
	335	+	`artifact_digest: outcome.artifact_digest,`
	336	+	`manifest_digest: outcome.manifest_digest,`
	337	+	`modules: build.modules,`
	338	+	`nodes: outcome.nodes,`
	339	+	`expected_nodes: outcome.expected_nodes,`
	340	+	`node_results: outcome.node_results,`
	341	+	`canary: canary \|\| outcome.canary,`
	342	+	`error_code: outcome.error_code,`
	343	+	`rollback_verified: outcome.rollback_verified,`
	344	+	`started_at: outcome.started_at,`
	345	+	`push_to_live_ms: push_ms`
	346	+	`}`
	347	+	`end`
	348	+
	349	+	`defp live_details(build, session) do`
	350	+	`%{`
	351	+	`"artifact" => Map.get(build, :artifact) \|> relative_artifact(),`
	352	+	`"artifact_digest" => session.artifact_digest,`
	353	+	`"build_id" => build.build_id,`
	354	+	`"deployment_id" => session.deployment_id,`
	355	+	`"manifest" => build.manifest,`
	356	+	`"manifest_digest" => session.manifest_digest,`
	357	+	`"modules" => build.modules,`
	358	+	`"nodes" => length(session.expected_nodes)`
	359	+	`}`
	360	+	`end`
	361	+
	362	+	`defp relative_artifact(nil), do: nil`
	363	+
	364	+	`defp relative_artifact(path) do`
	365	+	`Path.relative_to(path, OpenAgents.Forge.Repos.data_dir())`
	366	+	`end`
	367	+
	368	+	`defp manifest_digest(nil), do: nil`
	369	+
	370	+	`defp manifest_digest(manifest) do`
	371	+	`manifest`
	372	+	`\|> OpenAgents.Forge.BuildProtocol.canonical_json()`
	373	+	`\|> BuildArtifact.digest()`
	374	+	`end`
	375	+
	376	+	`defp node_lines(results),`
	377	+	`do: results \|> Enum.map(fn {node, result} -> "#{node}=#{result}" end) \|> Enum.sort()`
	378	+
	379	+	`defp advance(target_id, status, details) do`
349	380		`case Targets.advance(target_id, status, details) do`
350	381		`{:ok, _target} ->`
351	382		`:ok`

  import Ecto.Query

  alias OpenAgents.Forge.Target
  alias OpenAgents.Forge.DeployReceipt
  alias OpenAgents.Repo

  @statuses ~w(promoted building built deploying live failed reverted needs_rolling_replace)

86	87		`\|> Repo.one()`
87	88		`end`
88	89
	90	+	@doc "Newest immutable live targets for `repo`, bounded and newest first."
	91	+	`def live_history(repo, limit \\ 2) do`
	92	+	`Target`
	93	+	`\|> where([t], t.repo == ^repo and t.status == "live")`
	94	+	`\|> order_by([t], desc: t.updated_at, desc: t.inserted_at)`
	95	+	`\|> limit(^limit)`
	96	+	`\|> Repo.all()`
	97	+	`end`
	98	+
	99	+	`@doc "Resolve the durable authority for one committed node token."`
	100	+	`def deployment_authority(target_id, deployment_id, artifact_digest) do`
	101	+	`case Repo.get(Target, target_id) do`
	102	+	`%Target{`
	103	+	`status: "live",`
	104	+	`details: %{`
	105	+	`"deployment_id" => ^deployment_id,`
	106	+	`"artifact_digest" => ^artifact_digest`
	107	+	`}`
	108	+	`} ->`
	109	+	`:candidate_live`
	110	+
	111	+	`%Target{status: "deploying"} ->`
	112	+	`:pending`
	113	+
	114	+	`%Target{} ->`
	115	+	`:candidate_not_live`
	116	+
	117	+	`nil ->`
	118	+	`:candidate_not_live`
	119	+	`end`
	120	+	`end`
	121	+
89	122		`@doc "Recent targets for a repo, newest first, bounded."`
90	123		`def recent(repo, limit \\ 10) do`
91	124		`Target`

136	169		`end`
137	170		`end`
138	171
	172	+	`@doc "Fence deployment ownership to the newest promoted target."`
	173	+	`def begin_deployment(target_id) do`
	174	+	`result =`
	175	+	`:global.trans({{:forge_target_deploy, target_id}, self()}, fn ->`
	176	+	`Repo.transaction(fn ->`
	177	+	`target = Repo.get(Target, target_id, lock: "FOR UPDATE") \|\| Repo.rollback(:not_found)`
	178	+
	179	+	`current_id =`
	180	+	`Target`
	181	+	`\|> where([t], t.repo == ^target.repo)`
	182	+	`\|> order_by([t], desc: t.inserted_at)`
	183	+	`\|> limit(1)`
	184	+	`\|> select([t], t.id)`
	185	+	`\|> Repo.one()`
	186	+
	187	+	`if current_id != target.id, do: Repo.rollback(:superseded_target)`
	188	+
	189	+	`unless target.status == "built" do`
	190	+	`Repo.rollback({:invalid_transition, target.status, "deploying"})`
	191	+	`end`
	192	+
	193	+	`target`
	194	+	`\|> Target.status_changeset("deploying", %{})`
	195	+	`\|> Repo.update!()`
	196	+	`end)`
	197	+	`end)`
	198	+
	199	+	`with {:ok, target} <- result do`
	200	+	`broadcast_status(target)`
	201	+	`{:ok, target}`
	202	+	`end`
	203	+	`end`
	204	+
	205	+	`@doc "Atomically write a terminal deployment receipt and target status."`
	206	+	`def finish_deployment(target_id, status, details, receipt_attrs)`
	207	+	`when status in ~w(live reverted failed) and is_map(details) and is_map(receipt_attrs) do`
	208	+	`result =`
	209	+	`:global.trans({{:forge_target_deploy, target_id}, self()}, fn ->`
	210	+	`Repo.transaction(fn ->`
	211	+	`target = Repo.get(Target, target_id, lock: "FOR UPDATE") \|\| Repo.rollback(:not_found)`
	212	+
	213	+	`unless target.status == "deploying" do`
	214	+	`Repo.rollback({:invalid_transition, target.status, status})`
	215	+	`end`
	216	+
	217	+	`target =`
	218	+	`target`
	219	+	`\|> Target.status_changeset(status, bounded_details(details))`
	220	+	`\|> Repo.update!()`
	221	+
	222	+	`receipt_attrs =`
	223	+	`receipt_attrs`
	224	+	`\|> Map.put(:target_id, target.id)`
	225	+	`\|> Map.put(:repo, target.repo)`
	226	+	`\|> Map.put(:sha, target.sha)`
	227	+	`\|> Map.put(:result, status)`
	228	+
	229	+	`receipt =`
	230	+	`%DeployReceipt{}`
	231	+	`\|> DeployReceipt.changeset(receipt_attrs)`
	232	+	`\|> Repo.insert()`
	233	+	`\|> case do`
	234	+	`{:ok, receipt} -> receipt`
	235	+	`{:error, changeset} -> Repo.rollback({:invalid_receipt, changeset})`
	236	+	`end`
	237	+
	238	+	`%{target: target, receipt: receipt}`
	239	+	`end)`
	240	+	`end)`
	241	+
	242	+	`with {:ok, %{target: target} = committed} <- result do`
	243	+	`broadcast_status(target)`
	244	+	`{:ok, committed}`
	245	+	`end`
	246	+	`end`
	247	+
139	248		`defp bounded_details(details) do`
140	249		`details`
141	250		`\|> Enum.map(fn {key, value} -> {to_string(key), bound_value(value)} end)`

    # not report itself healthy). Where Ra is off (single-node Cloud Run) the
    # expected size is honestly 1.
    expected =
      if Application.get_env(:openagents, :ra_enabled, false),
        do: Application.get_env(:openagents, :ra_expected_size, 3),
        else: 1
      cond do
        Application.get_env(:openagents, :ra_enabled, false) ->
          Application.get_env(:openagents, :ra_expected_size, 3)

        Application.get_env(:openagents, :forge_deploy_lane_enabled, false) ->
          Application.get_env(:openagents, :forge_expected_fleet_size, 1)

        true ->
          1
      end

    quorum = safely(fn -> OpenAgents.Cluster.quorum?(max(expected, 1)) end) || false


        Map.put(report, "label", "node #{index}")
      end)

    nodes_ready? = Enum.all?(nodes, &(&1["reachable"] == true and &1["ready"] == true))

    %{
      "schema" => @schema,
      # Legacy /status compatibility keys — pollers migrating from the old
      # payload find them here unchanged.
      "status" => if(quorum or length(beam_nodes) == 1, do: "ok", else: "degraded"),
      "revision" => safely(fn -> OpenAgents.BuildInfo.revision() end),
      "status" => if(quorum and nodes_ready?, do: "ok", else: "degraded"),
      "revision" => safely(fn -> OpenAgents.Forge.DeploymentNode.health()["revision"] end),
      "cluster" => %{
        "distributed" => Map.get(cluster, "distributed", false),
        "beam" => length(beam_nodes),

  @doc false
  def local_report do
    {wall_ms, _} = :erlang.statistics(:wall_clock)
    boot = OpenAgents.Forge.BootConverge.state()
    boot_ready? = OpenAgents.Forge.BootConverge.ready?()
    deployment = OpenAgents.Forge.DeploymentNode.health()

    %{
      "reachable" => true,
      "ready" => boot_ready? and deployment["ready"] == true,
      "release" => safely(fn -> permanent_release_version() end),
      "revision" => safely(fn -> OpenAgents.BuildInfo.revision() end),
      "revision" => deployment["revision"] || boot["sha"] || OpenAgents.BuildInfo.revision(),
      "hot_loaded_at" => safely(fn -> OpenAgents.BuildInfo.loaded_at() end),
      "marker" => safely(fn -> OpenAgents.Cluster.relup_marker() end),
      "boot" => safely(fn -> OpenAgents.Forge.BootConverge.state()["state"] end),
      "boot" => %{
        "state" => boot["state"],
        "ready" => boot_ready?,
        "reason" => boot["reason"],
        "attempts" => boot["attempts"],
        "retry_in_ms" => boot["retry_in_ms"]
      },
      "deployment" => %{
        "ready" => deployment["participant_ready"],
        "phase" => deployment["phase"],
        "reason" => deployment["reason"]
      },
      "uptime_seconds" => div(wall_ms, 1_000),
      "beam_seen" => 1 + length(Node.list()),
      "raft_seen" => safely(fn -> length(OpenAgents.Cluster.Ra.members()) end) || 0

    expected_fleet_size = Map.get(settings, :forge_expected_fleet_size)
    artifact_store = Map.get(settings, :forge_artifact_store)
    build_executor = Map.get(settings, :forge_build_executor)
    deploy_timeout_ms = Map.get(settings, :forge_deploy_timeout_ms)
    deploy_token_ttl_ms = Map.get(settings, :forge_deploy_token_ttl_ms)
    boot_retry_min_ms = Map.get(settings, :forge_boot_retry_min_ms)
    boot_retry_max_ms = Map.get(settings, :forge_boot_retry_max_ms)
    operator_token = Map.get(settings, :forge_operator_token)
    mirror_urls = Map.get(settings, :forge_mirror_urls)
    durable_required? = environment in [:staging, :production] and features.forge

             :forge_build_executor,
             "must be a module"
           ),
         :ok <-
           ensure(
             is_integer(deploy_timeout_ms) and deploy_timeout_ms in 1_000..120_000,
             :forge_deploy_timeout_ms,
             "must be between 1 and 120 seconds"
           ),
         :ok <-
           ensure(
             is_integer(deploy_token_ttl_ms) and deploy_token_ttl_ms in 30_000..1_800_000 and
               deploy_token_ttl_ms >= deploy_timeout_ms * 8,
             :forge_deploy_token_ttl_ms,
             "must cover eight deployment phase timeouts"
           ),
         :ok <-
           ensure(
             is_integer(boot_retry_min_ms) and is_integer(boot_retry_max_ms) and
               boot_retry_min_ms in 100..60_000 and boot_retry_max_ms in 1_000..300_000 and
               boot_retry_min_ms <= boot_retry_max_ms,
             :forge_boot_retry_bounds,
             "must define an ordered bounded backoff"
           ),
         :ok <-
           ensure(
             not features.forge_deploy or expected_fleet_size >= 2,

        OpenAgents.Leaderboard.Server,
        {Task.Supervisor, name: OpenAgents.ProviderTaskSupervisor},
        {Task.Supervisor, name: OpenAgents.ToolTaskSupervisor},
        {Task.Supervisor, name: OpenAgents.ShadowProgramTaskSupervisor},
        OpenAgents.Forge.BootConverge
        {Task.Supervisor, name: OpenAgents.ShadowProgramTaskSupervisor}
      ] ++
        maybe_forge() ++
        maybe_semantic_worker() ++

  use OpenAgentsWeb, :controller

  def show(conn, _params) do
    case OpenAgents.Repo.query("SELECT 1") do
      {:ok, _result} ->
        json(conn, %{status: "ok", revision: OpenAgents.BuildInfo.revision()})
    case {OpenAgents.Repo.query("SELECT 1"), OpenAgents.Cluster.local_report()} do
      {{:ok, _result}, %{"ready" => true, "revision" => revision}} ->
        json(conn, %{status: "ok", revision: revision})

      {:error, _reason} ->
        conn |> put_status(:service_unavailable) |> json(%{status: "unavailable"})
      {{:ok, _result}, report} ->
        conn
        |> put_status(:service_unavailable)
        |> json(%{
          status: "unavailable",
          reason: "runtime_not_ready",
          boot_converged: report["boot_converged"],
          deployment_ready: report["deployment_ready"]
        })

      {{:error, _reason}, _report} ->
        conn
        |> put_status(:service_unavailable)
        |> json(%{status: "unavailable", reason: "database_unavailable"})
    end
  end
end

export OPENAGENTS_FORGE_BUILD_QUEUE_DIR="/var/lib/openagents/workspace/build-queue"
export OPENAGENTS_FORGE_BUILD_TIMEOUT_MS="300000"
export OPENAGENTS_FORGE_BUILD_OUTPUT_RETENTION_MS="604800000"
export OPENAGENTS_FORGE_DEPLOY_TIMEOUT_MS="15000"
export OPENAGENTS_FORGE_DEPLOY_TOKEN_TTL_MS="120000"
export OPENAGENTS_FORGE_BOOT_RETRY_MIN_MS="1000"
export OPENAGENTS_FORGE_BOOT_RETRY_MAX_MS="30000"
export OPENAGENTS_FORGE_DATA_DIR="/var/lib/openagents/forge"
export OPENAGENTS_FORGE_EXPECTED_FLEET_SIZE="1"
export OPENAGENTS_FORGE_INTERNAL_GIT_URL="http://127.0.0.1:8080/git"

1	+	`defmodule OpenAgents.Repo.Migrations.HardenForgeDeploymentTransactions do`
2	+	`use Ecto.Migration`
3	+
4	+	`def up do`
5	+	`alter table(:forge_deploys) do`
6	+	`add :deployment_id, :uuid`
7	+	`add :artifact_digest, :string`
8	+	`add :manifest_digest, :string`
9	+	`add :expected_nodes, {:array, :string}, null: false, default: []`
10	+	`add :node_results, :map, null: false, default: %{}`
11	+	`add :error_code, :string`
12	+	`add :rollback_verified, :boolean`
13	+	`add :started_at, :utc_datetime_usec`
14	+	`add :completed_at, :utc_datetime_usec`
15	+	`end`
16	+
17	+	`execute("""`
18	+	`UPDATE forge_deploys`
19	+	`SET deployment_id = gen_random_uuid(),`
20	+	`expected_nodes = COALESCE(nodes, ARRAY[]::varchar[]),`
21	+	`node_results = '{}'::jsonb,`
22	+	`started_at = inserted_at,`
23	+	`completed_at = inserted_at`
24	+	`""")`
25	+
26	+	`alter table(:forge_deploys) do`
27	+	`modify :deployment_id, :uuid, null: false`
28	+	`modify :started_at, :utc_datetime_usec, null: false`
29	+	`modify :completed_at, :utc_datetime_usec, null: false`
30	+	`end`
31	+
32	+	`create unique_index(:forge_deploys, [:deployment_id])`
33	+	`create index(:forge_deploys, [:target_id, :completed_at])`
34	+
35	+	`create constraint(:forge_deploys, :forge_deploys_artifact_digest,`
36	+	`check: "artifact_digest IS NULL OR artifact_digest ~ '^[0-9a-f]{64}$'"`
37	+	`)`
38	+
39	+	`create constraint(:forge_deploys, :forge_deploys_manifest_digest,`
40	+	`check: "manifest_digest IS NULL OR manifest_digest ~ '^[0-9a-f]{64}$'"`
41	+	`)`
42	+
43	+	`create constraint(:forge_deploys, :forge_deploys_node_bounds,`
44	+	`check:`
45	+	`"cardinality(nodes) <= 100 AND cardinality(expected_nodes) <= 100 AND jsonb_typeof(node_results) = 'object' AND octet_length(node_results::text) <= 32768"`
46	+	`)`
47	+
48	+	`execute("""`
49	+	`CREATE FUNCTION reject_forge_deploy_receipt_mutation()`
50	+	`RETURNS trigger`
51	+	`LANGUAGE plpgsql`
52	+	`AS $$`
53	+	`BEGIN`
54	+	`RAISE EXCEPTION 'forge deployment receipts are immutable';`
55	+	`END;`
56	+	`$$`
57	+	`""")`
58	+
59	+	`execute("""`
60	+	`CREATE TRIGGER forge_deploy_receipts_immutable`
61	+	`BEFORE UPDATE OR DELETE ON forge_deploys`
62	+	`FOR EACH ROW EXECUTE FUNCTION reject_forge_deploy_receipt_mutation()`
63	+	`""")`
64	+	`end`
65	+
66	+	`def down do`
67	+	`execute("DROP TRIGGER IF EXISTS forge_deploy_receipts_immutable ON forge_deploys")`
68	+	`execute("DROP FUNCTION IF EXISTS reject_forge_deploy_receipt_mutation()")`
69	+
70	+	`drop constraint(:forge_deploys, :forge_deploys_node_bounds)`
71	+	`drop constraint(:forge_deploys, :forge_deploys_manifest_digest)`
72	+	`drop constraint(:forge_deploys, :forge_deploys_artifact_digest)`
73	+	`drop index(:forge_deploys, [:target_id, :completed_at])`
74	+	`drop index(:forge_deploys, [:deployment_id])`
75	+
76	+	`alter table(:forge_deploys) do`
77	+	`remove :deployment_id`
78	+	`remove :artifact_digest`
79	+	`remove :manifest_digest`
80	+	`remove :expected_nodes`
81	+	`remove :node_results`
82	+	`remove :error_code`
83	+	`remove :rollback_verified`
84	+	`remove :started_at`
85	+	`remove :completed_at`
86	+	`end`
87	+	`end`
88	+	`end`

  use OpenAgents.DataCase, async: false
  alias OpenAgents.Forge.ArtifactFixtures
  alias OpenAgents.Forge.BootConverge
  alias OpenAgents.Forge.DeploymentNode
  alias OpenAgents.Forge.Repos
  alias OpenAgents.Forge.Target
  alias OpenAgents.Repo

        {key, Application.get_env(:openagents, key)}
      end

    previous_node_state = :sys.get_state(DeploymentNode)
    previous_persisted = :persistent_term.get({DeploymentNode, :state}, :missing)
    :persistent_term.erase({DeploymentNode, :state})

    :sys.replace_state(DeploymentNode, fn state ->
      %{state | transactions: %{}, live: nil, divergence: nil, faults: %{}, notify: nil}
    end)

    Application.put_env(:openagents, :forge_data_dir, Path.join(base, "data"))
    Application.put_env(:openagents, :forge_wal_dir, Path.join(base, "wal"))



      File.rm_rf(base)
      :persistent_term.erase({BootConverge, :state})
      :sys.replace_state(DeploymentNode, fn _state -> previous_node_state end)
      restore_persistent(previous_persisted)
    end)

    %{base: base}

        "artifact" => "beams/#{built.digest}.tar",
        "artifact_digest" => built.digest,
        "build_id" => built.build_id,
        "manifest" => built.manifest,
        "manifest_digest" =>
          built.manifest
          |> OpenAgents.Forge.BuildProtocol.canonical_json()
          |> OpenAgents.Forge.BuildArtifact.digest(),
        "modules" => Enum.map(built.beams, & &1.module)
      }
    }


  test "boots on image code, honestly, for every degraded path" do
    # No target at all.
    assert %{"state" => "image", "reason" => "no_target"} = BootConverge.converge(@repo)
    assert %{"state" => "image", "reason" => "no_live_target", "ready" => true} =
             BootConverge.converge(@repo)

    # A target that is not live.
    insert_target!("failed", %{})

    assert %{"state" => "image", "reason" => "target_not_live:failed"} =
    assert %{"state" => "image", "reason" => "no_live_target", "ready" => true} =
             BootConverge.converge(@repo)

    # A live target whose artifact this node does not have (replaced node).
    {missing_module, missing_binary} = scratch_beam(OpenAgents.Scratch.BootConvergeMissing)
    missing = artifact(missing_module, missing_binary)
    insert_target!("live", missing.details)
    assert %{"state" => "image", "reason" => "artifact_missing"} = BootConverge.converge(@repo)
    assert %{"state" => "degraded", "ready" => false} = BootConverge.converge(@repo)

    # A live target with an off-allowlist module in the tar.
    {module, binary} = scratch_beam(OpenAgents.NotAllowed.BootConvergeOffLimits)

    File.write!(artifact_abs, off_limit.built.bytes)
    insert_target!("live", off_limit.details)

    assert %{"state" => "image", "reason" => "off_allowlist:" <> _rest} =
    assert %{"state" => "degraded", "ready" => false} =
             BootConverge.converge(@repo)

    refute Code.ensure_loaded?(OpenAgents.NotAllowed.BootConvergeOffLimits)

150	167		`:code.delete(module)`
151	168		`end`
152	169
153		-	`test "a live no-op target (no artifact recorded) counts as converged" do`
	170	+	`test "convergence retains current and predecessor artifacts and prunes older cache entries" do`
	171	+	`{predecessor_module, predecessor_binary} =`
	172	+	`scratch_beam(OpenAgents.Scratch.BootConvergePredecessor)`
	173	+
	174	+	`predecessor = artifact(predecessor_module, predecessor_binary)`
	175	+
	176	+	`File.write!(`
	177	+	`Path.join(Repos.data_dir(), predecessor.details["artifact"]),`
	178	+	`predecessor.built.bytes`
	179	+	`)`
	180	+
	181	+	`insert_target!("live", predecessor.details)`
	182	+
	183	+	`{current_module, current_binary} = scratch_beam(OpenAgents.Scratch.BootConvergeCurrent)`
	184	+	`current = artifact(current_module, current_binary)`
	185	+	`File.write!(Path.join(Repos.data_dir(), current.details["artifact"]), current.built.bytes)`
	186	+	`insert_target!("live", current.details)`
	187	+
	188	+	`orphan_digest = String.duplicate("a", 64)`
	189	+	`orphan_path = Path.join([Repos.data_dir(), "beams", orphan_digest <> ".tar"])`
	190	+	`File.write!(orphan_path, "obsolete-cache-entry")`
	191	+
	192	+	`assert %{"state" => "converged"} = BootConverge.converge(@repo)`
	193	+	`assert File.exists?(Path.join(Repos.data_dir(), current.details["artifact"]))`
	194	+	`assert File.exists?(Path.join(Repos.data_dir(), predecessor.details["artifact"]))`
	195	+	`refute File.exists?(orphan_path)`
	196	+
	197	+	`:code.purge(current_module)`
	198	+	`:code.delete(current_module)`
	199	+	`end`
	200	+
	201	+	`test "a live target without artifact identity stays out of readiness" do`
154	202		`insert_target!("live", %{})`
155		-	`assert %{"state" => "converged", "modules" => 0} = BootConverge.converge(@repo)`
	203	+
	204	+	`assert %{`
	205	+	`"state" => "degraded",`
	206	+	`"ready" => false,`
	207	+	`"reason" => "live_artifact_identity_missing"`
	208	+	`} = BootConverge.converge(@repo)`
156	209		`end`
	210	+
	211	+	`test "readiness fails closed when a newer live target appears" do`
	212	+	`{module, binary} = scratch_beam(OpenAgents.Scratch.BootConvergeFreshness)`
	213	+	`artifact = artifact(module, binary)`
	214	+	`File.write!(Path.join(Repos.data_dir(), artifact.details["artifact"]), artifact.built.bytes)`
	215	+	`insert_target!("live", artifact.details)`
	216	+
	217	+	`assert %{"state" => "converged", "ready" => true} = BootConverge.converge(@repo)`
	218	+
	219	+	`previous_enabled = Application.get_env(:openagents, :forge_boot_converge_enabled)`
	220	+	`Application.put_env(:openagents, :forge_boot_converge_enabled, true)`
	221	+
	222	+	`on_exit(fn -> restore_env(:forge_boot_converge_enabled, previous_enabled) end)`
	223	+
	224	+	`assert BootConverge.ready?(@repo)`
	225	+
	226	+	`newer =`
	227	+	`%Target{}`
	228	+	`\|> Target.changeset(%{`
	229	+	`repo: @repo,`
	230	+	`sha: String.duplicate("e", 40),`
	231	+	`promoted_by: "operator:t",`
	232	+	`status: "promoted"`
	233	+	`})`
	234	+	`\|> Repo.insert!()`
	235	+	`\|> Ecto.Changeset.change(%{status: "deploying", details: %{}})`
	236	+	`\|> Repo.update!()`
	237	+
	238	+	`refute BootConverge.ready?(@repo)`
	239	+	`assert BootConverge.ready_for_deployment?(@repo, newer.id)`
	240	+
	241	+	`newer`
	242	+	`\|> Ecto.Changeset.change(%{status: "live"})`
	243	+	`\|> Repo.update!()`
	244	+
	245	+	`refute BootConverge.ready?(@repo)`
	246	+
	247	+	`:code.purge(module)`
	248	+	`:code.delete(module)`
	249	+	`end`
	250	+
	251	+	`test "the supervised worker retains degraded readiness and caps retry backoff" do`
	252	+	`insert_target!("live", %{})`
	253	+	`previous_enabled = Application.get_env(:openagents, :forge_boot_converge_enabled)`
	254	+	`previous_min = Application.get_env(:openagents, :forge_boot_retry_min_ms)`
	255	+	`previous_max = Application.get_env(:openagents, :forge_boot_retry_max_ms)`
	256	+
	257	+	`Application.put_env(:openagents, :forge_boot_converge_enabled, true)`
	258	+	`Application.put_env(:openagents, :forge_boot_retry_min_ms, 10)`
	259	+	`Application.put_env(:openagents, :forge_boot_retry_max_ms, 20)`
	260	+
	261	+	`name = Module.concat(__MODULE__, "Retry#{System.unique_integer([:positive])}")`
	262	+
	263	+	`start_supervised!(`
	264	+	`Supervisor.child_spec(`
	265	+	`{BootConverge, name: name, repo: @repo},`
	266	+	`id: name`
	267	+	`)`
	268	+	`)`
	269	+
	270	+	`server_state = :sys.get_state(name)`
	271	+	`convergence = BootConverge.state()`
	272	+
	273	+	`refute convergence["ready"]`
	274	+	`assert convergence["state"] == "degraded"`
	275	+	`assert convergence["retry_in_ms"] == 10`
	276	+	`assert server_state.retry_ms == 20`
	277	+
	278	+	`on_exit(fn ->`
	279	+	`restore_env(:forge_boot_converge_enabled, previous_enabled)`
	280	+	`restore_env(:forge_boot_retry_min_ms, previous_min)`
	281	+	`restore_env(:forge_boot_retry_max_ms, previous_max)`
	282	+	`end)`
	283	+	`end`
	284	+
	285	+	`defp restore_env(key, nil), do: Application.delete_env(:openagents, key)`
	286	+	`defp restore_env(key, value), do: Application.put_env(:openagents, key, value)`
	287	+
	288	+	`defp restore_persistent(:missing), do: :persistent_term.erase({DeploymentNode, :state})`
	289	+
	290	+	`defp restore_persistent(state),`
	291	+	`do: :persistent_term.put({DeploymentNode, :state}, state)`
157	292		`end`

1	+	`defmodule OpenAgents.Forge.DeploymentClusterTest do`
2	+	`use ExUnit.Case, async: false`
3	+
4	+	`alias OpenAgents.Forge.ArtifactFixtures`
5	+	`alias OpenAgents.Forge.BuildArtifact`
6	+	`alias OpenAgents.Forge.Deployment`
7	+	`alias OpenAgents.Forge.DeploymentNode`
8	+
9	+	`@moduletag :cluster`
10	+
11	+	`setup do`
12	+	`ensure_distributed!()`
13	+
14	+	`base =`
15	+	`Path.join(System.tmp_dir!(), "deployment-cluster-#{System.unique_integer([:positive])}")`
16	+
17	+	`File.mkdir_p!(base)`
18	+
19	+	`previous = %{`
20	+	`data: Application.get_env(:openagents, :forge_data_dir),`
21	+	`expected: Application.get_env(:openagents, :forge_expected_fleet_size),`
22	+	`allowlist: Application.get_env(:openagents, :forge_hot_load_allowlist)`
23	+	`}`
24	+
25	+	`Application.put_env(:openagents, :forge_data_dir, Path.join(base, "local"))`
26	+	`Application.put_env(:openagents, :forge_expected_fleet_size, 3)`
27	+	`Application.put_env(:openagents, :forge_hot_load_allowlist, ["OpenAgents.Scratch."])`
28	+	`reset_local_participant()`
29	+
30	+	`on_exit(fn ->`
31	+	`reset_local_participant()`
32	+	`restore_env(:forge_data_dir, previous.data)`
33	+	`restore_env(:forge_expected_fleet_size, previous.expected)`
34	+	`restore_env(:forge_hot_load_allowlist, previous.allowlist)`
35	+	`:persistent_term.erase({OpenAgents.Forge.BootConverge, :state})`
36	+	`File.rm_rf(base)`
37	+	`end)`
38	+
39	+	`%{base: base}`
40	+	`end`
41	+
42	+	`test "three nodes prepare, canary, verify, commit, and finalize one exact candidate", %{`
43	+	`base: base`
44	+	`} do`
45	+	`peers = start_peers(base, [%{}, %{}])`
46	+	`fixture = versions("FleetSuccess", peers)`
47	+
48	+	`try do`
49	+	`assert {:ok, session} =`
50	+	`Deployment.run(build(fixture), fixture.verified, fixture.built.bytes)`
51	+
52	+	`assert length(session.expected_nodes) == 3`
53	+	`assert session.canary == to_string(Node.self())`
54	+
55	+	`for node <- nodes(peers) do`
56	+	`assert revision(node, fixture.module) == "candidate"`
57	+	`refute health(node)["ready"]`
58	+	`end`
59	+
60	+	`assert :ok = Deployment.finalize(session)`
61	+
62	+	`for node <- nodes(peers) do`
63	+	`assert health(node)["ready"]`
64	+	`assert health(node)["revision"] == fixture.sha`
65	+	`end`
66	+	`after`
67	+	`cleanup_fixture(fixture, peers)`
68	+	`stop_peers(peers)`
69	+	`end`
70	+	`end`
71	+
72	+	`test "one remote apply error restores exact prior code on every prepared node", %{base: base} do`
73	+	`peers = start_peers(base, [%{}, %{apply: :error}])`
74	+	`fixture = versions("FleetRollback", peers)`
75	+
76	+	`try do`
77	+	`assert {:error, outcome} =`
78	+	`Deployment.run(build(fixture), fixture.verified, fixture.built.bytes)`
79	+
80	+	`assert outcome.result == "reverted"`
81	+	`assert outcome.rollback_verified`
82	+	`assert length(outcome.expected_nodes) == 3`
83	+
84	+	`for node <- nodes(peers) do`
85	+	`assert revision(node, fixture.module) == "prior"`
86	+	`assert health(node)["ready"]`
87	+	`end`
88	+	`after`
89	+	`cleanup_fixture(fixture, peers)`
90	+	`stop_peers(peers)`
91	+	`end`
92	+	`end`
93	+
94	+	`test "an unverified rollback leaves the affected node out of readiness", %{base: base} do`
95	+	`peers = start_peers(base, [%{rollback: :error}, %{apply: :error}])`
96	+	`[{_rollback_peer, rollback_node}, _failing_peer] = peers`
97	+	`fixture = versions("FleetDivergence", peers)`
98	+
99	+	`try do`
100	+	`assert {:error, outcome} =`
101	+	`Deployment.run(build(fixture), fixture.verified, fixture.built.bytes)`
102	+
103	+	`assert outcome.result == "failed"`
104	+	`refute outcome.rollback_verified`
105	+	`assert revision(rollback_node, fixture.module) == "candidate"`
106	+	`refute health(rollback_node)["ready"]`
107	+	`after`
108	+	`cleanup_fixture(fixture, peers)`
109	+	`stop_peers(peers)`
110	+	`end`
111	+	`end`
112	+
113	+	`test "a timed-out participant blocks live and cannot be mistaken for rollback", %{base: base} do`
114	+	`peers = start_peers(base, [%{}, %{apply: :timeout}], fault_timeout_ms: 1_000)`
115	+	`fixture = versions("FleetTimeout", peers)`
116	+
117	+	`try do`
118	+	`assert {:error, outcome} =`
119	+	`Deployment.run(build(fixture), fixture.verified, fixture.built.bytes,`
120	+	`timeout_ms: 100`
121	+	`)`
122	+
123	+	`assert outcome.result == "failed"`
124	+	`refute outcome.rollback_verified`
125	+	`assert outcome.error_code == "fleet_apply_failed"`
126	+
127	+	`assert Enum.any?(outcome.node_results, fn {_node, result} ->`
128	+	`String.contains?(result, "rollback_failed")`
129	+	`end)`
130	+	`after`
131	+	`cleanup_fixture(fixture, peers)`
132	+	`stop_peers(peers)`
133	+	`end`
134	+	`end`
135	+
136	+	`test "membership loss after prepare aborts before fleet commit", %{base: base} do`
137	+	`peers = start_peers(base, [%{}, %{}])`
138	+	`[{peer_to_stop, _node_to_stop}, _survivor] = peers`
139	+	`fixture = versions("MembershipLoss", peers)`
140	+	`calls = :counters.new(1, [])`
141	+
142	+	`membership = fn ->`
143	+	`:counters.add(calls, 1, 1)`
144	+	`call = :counters.get(calls, 1)`
145	+	`if call == 3, do: safe_stop_peer(peer_to_stop)`
146	+	`[Node.self() \| Node.list()] \|> Enum.uniq() \|> Enum.sort()`
147	+	`end`
148	+
149	+	`try do`
150	+	`assert {:error, outcome} =`
151	+	`Deployment.run(build(fixture), fixture.verified, fixture.built.bytes,`
152	+	`members: membership,`
153	+	`timeout_ms: 500`
154	+	`)`
155	+
156	+	`assert outcome.result == "failed"`
157	+	`refute outcome.rollback_verified`
158	+	`assert outcome.error_code == "membership_changed"`
159	+	`assert revision(Node.self(), fixture.module) == "prior"`
160	+	`after`
161	+	`cleanup_fixture(fixture, peers)`
162	+	`stop_peers(peers)`
163	+	`end`
164	+	`end`
165	+
166	+	`defp start_peers(base, faults, opts \\ []) do`
167	+	`cookie = Node.get_cookie()`
168	+	`paths = :code.get_path()`
169	+
170	+	`faults`
171	+	`\|> Enum.with_index(1)`
172	+	`\|> Enum.map(fn {node_faults, index} ->`
173	+	`suffix = System.unique_integer([:positive])`
174	+	`name = String.to_atom("forge_deploy_peer#{index}_#{suffix}")`
175	+
176	+	`{:ok, peer, node} =`
177	+	`:peer.start_link(%{`
178	+	`name: name,`
179	+	`host: ~c"127.0.0.1",`
180	+	`user: %{},`
181	+	`shutdown: OpenAgents.Test.RemoteCover.shutdown(),`
182	+	`args: [~c"-setcookie", to_charlist(Atom.to_string(cookie))]`
183	+	`})`
184	+
185	+	`:ok = :erpc.call(node, :code, :add_paths, [paths])`
186	+	`:ok = :erpc.call(node, Application, :load, [:openagents])`
187	+
188	+	`:ok =`
189	+	`:erpc.call(node, Application, :put_env, [`
190	+	`:openagents,`
191	+	`:forge_data_dir,`
192	+	`Path.join(base, "peer-#{index}")`
193	+	`])`
194	+
195	+	`:ok =`
196	+	`:erpc.call(node, Application, :put_env, [`
197	+	`:openagents,`
198	+	`:forge_hot_load_allowlist,`
199	+	`["OpenAgents.Scratch."]`
200	+	`])`
201	+
202	+	`start_opts =`
203	+	`[faults: node_faults]`
204	+	`\|> Keyword.put(:fault_timeout_ms, Keyword.get(opts, :fault_timeout_ms, 30_000))`
205	+
206	+	`{:ok, _participant} = :erpc.call(node, DeploymentNode, :start, [start_opts])`
207	+	`{peer, node}`
208	+	`end)`
209	+	`end`
210	+
211	+	`defp versions(suffix, peers) do`
212	+	`name = "OpenAgents.Scratch.#{suffix}#{System.unique_integer([:positive])}"`
213	+	`module = Module.concat([name])`
214	+	`prior_binary = compile(name, "prior")`
215	+	`unload(module)`
216	+	`candidate_binary = compile(name, "candidate")`
217	+	`unload(module)`
218	+
219	+	`prior_dir =`
220	+	`Path.join(`
221	+	`System.tmp_dir!(),`
222	+	`"deployment-cluster-prior-#{System.unique_integer([:positive])}"`
223	+	`)`
224	+
225	+	`File.mkdir_p!(prior_dir)`
226	+	`File.write!(Path.join(prior_dir, Atom.to_string(module) <> ".beam"), prior_binary)`
227	+
228	+	`for node <- nodes(peers) do`
229	+	`assert true = call(node, :code, :add_patha, [to_charlist(prior_dir)])`
230	+	`_purged = call(node, :code, :purge, [module])`
231	+	`_deleted = call(node, :code, :delete, [module])`
232	+	`assert {:module, ^module} = call(node, :code, :load_file, [module])`
233	+	`end`
234	+
235	+	`sha = random_sha()`
236	+	`built = ArtifactFixtures.create!("openagents.com", sha, [{name, candidate_binary}])`
237	+
238	+	`{:ok, verified} =`
239	+	`BuildArtifact.verify(built.bytes,`
240	+	`digest: built.digest,`
241	+	`repo: "openagents.com",`
242	+	`source_sha: sha,`
243	+	`build_id: built.build_id`
244	+	`)`
245	+
246	+	`%{`
247	+	`module: module,`
248	+	`built: built,`
249	+	`verified: verified,`
250	+	`sha: sha,`
251	+	`prior_dir: prior_dir`
252	+	`}`
253	+	`end`
254	+
255	+	`defp build(fixture) do`
256	+	`%{`
257	+	`repo: "openagents.com",`
258	+	`sha: fixture.sha,`
259	+	`target_id: Ecto.UUID.generate(),`
260	+	`build_id: fixture.built.build_id,`
261	+	`modules: fixture.verified.modules,`
262	+	`manifest: fixture.built.manifest`
263	+	`}`
264	+	`end`
265	+
266	+	`defp compile(name, revision) do`
267	+	`[{_module, binary}] =`
268	+	`Code.compile_string("defmodule #{name} do\n def revision, do: #{inspect(revision)}\nend")`
269	+
270	+	`binary`
271	+	`end`
272	+
273	+	`defp revision(target_node, module) do`
274	+	`if target_node == Node.self(),`
275	+	`do: module.revision(),`
276	+	`else: :erpc.call(target_node, module, :revision, [])`
277	+	`end`
278	+
279	+	`defp health(target_node) do`
280	+	`if target_node == Node.self(),`
281	+	`do: DeploymentNode.health(),`
282	+	`else: :erpc.call(target_node, DeploymentNode, :health, [])`
283	+	`end`
284	+
285	+	`defp nodes(peers), do: [Node.self() \| Enum.map(peers, &elem(&1, 1))]`
286	+
287	+	`defp cleanup_fixture(fixture, peers) do`
288	+	`Enum.each(nodes(peers), &safe_cleanup_node(&1, fixture))`
289	+
290	+	`File.rm_rf(fixture.prior_dir)`
291	+	`end`
292	+
293	+	`defp safe_cleanup_node(target_node, fixture) do`
294	+	`if target_node == Node.self() do`
295	+	`unload(fixture.module)`
296	+	`:code.del_path(to_charlist(fixture.prior_dir))`
297	+	`else`
298	+	`_result = :erpc.call(target_node, :code, :purge, [fixture.module])`
299	+	`_result = :erpc.call(target_node, :code, :delete, [fixture.module])`
300	+	`_result = :erpc.call(target_node, :code, :del_path, [to_charlist(fixture.prior_dir)])`
301	+	`end`
302	+	`rescue`
303	+	`_error -> :ok`
304	+	`catch`
305	+	`_kind, _reason -> :ok`
306	+	`end`
307	+
308	+	`defp stop_peers(peers) do`
309	+	`Enum.each(peers, fn {peer, _node} -> safe_stop_peer(peer) end)`
310	+	`end`
311	+
312	+	`defp safe_stop_peer(peer) do`
313	+	`:peer.stop(peer)`
314	+	`catch`
315	+	`:exit, _reason -> :ok`
316	+	`end`
317	+
318	+	`defp unload(module) do`
319	+	`:code.purge(module)`
320	+	`:code.delete(module)`
321	+	`:code.purge(module)`
322	+	`end`
323	+
324	+	`defp call(target_node, module, function, arguments) do`
325	+	`if target_node == Node.self(),`
326	+	`do: apply(module, function, arguments),`
327	+	`else: :erpc.call(target_node, module, function, arguments)`
328	+	`end`
329	+
330	+	`defp reset_local_participant do`
331	+	`:persistent_term.erase({DeploymentNode, :state})`
332	+
333	+	`:sys.replace_state(DeploymentNode, fn state ->`
334	+	`%{state \| transactions: %{}, live: nil, divergence: nil, faults: %{}, notify: nil}`
335	+	`end)`
336	+	`end`
337	+
338	+	`defp ensure_distributed! do`
339	+	`if Node.self() == :nonode@nohost do`
340	+	`suffix = System.unique_integer([:positive])`
341	+	`name = String.to_atom("forge_deploy_test_#{suffix}@127.0.0.1")`
342	+
343	+	`case :net_kernel.start([name, :longnames]) do`
344	+	`:ok -> :ok`
345	+	`{:ok, _pid} -> :ok`
346	+	`{:error, {:already_started, _pid}} -> :ok`
347	+	`end`
348	+
349	+	`Node.set_cookie(:openagents_test_cookie)`
350	+	`end`
351	+
352	+	`:ok`
353	+	`end`
354	+
355	+	`defp random_sha, do: 20 \|> :crypto.strong_rand_bytes() \|> Base.encode16(case: :lower)`
356	+	`defp restore_env(key, nil), do: Application.delete_env(:openagents, key)`
357	+	`defp restore_env(key, value), do: Application.put_env(:openagents, key, value)`
358	+	`end`

1	+	`defmodule OpenAgents.Forge.DeploymentNodeTest do`
2	+	`use OpenAgents.DataCase, async: false`
3	+
4	+	`alias OpenAgents.Forge.ArtifactFixtures`
5	+	`alias OpenAgents.Forge.BuildArtifact`
6	+	`alias OpenAgents.Forge.BuildProtocol`
7	+	`alias OpenAgents.Forge.DeploymentNode`
8	+	`alias OpenAgents.Forge.Target`
9	+
10	+	`setup do`
11	+	`base = Path.join(System.tmp_dir!(), "deployment-node-#{System.unique_integer([:positive])}")`
12	+	`previous_data = Application.get_env(:openagents, :forge_data_dir)`
13	+	`previous_allowlist = Application.get_env(:openagents, :forge_hot_load_allowlist)`
14	+	`previous_state = :sys.get_state(DeploymentNode)`
15	+	`previous_persisted = :persistent_term.get({DeploymentNode, :state}, :missing)`
16	+
17	+	`Application.put_env(:openagents, :forge_data_dir, base)`
18	+	`Application.put_env(:openagents, :forge_hot_load_allowlist, ["OpenAgents.Scratch."])`
19	+	`reset_participant()`
20	+
21	+	`on_exit(fn ->`
22	+	`:sys.replace_state(DeploymentNode, fn _state -> previous_state end)`
23	+	`restore_persistent(previous_persisted)`
24	+
25	+	`restore_env(:forge_data_dir, previous_data)`
26	+	`restore_env(:forge_hot_load_allowlist, previous_allowlist)`
27	+	`:persistent_term.erase({OpenAgents.Forge.BootConverge, :state})`
28	+	`File.rm_rf(base)`
29	+	`end)`
30	+
31	+	`:ok`
32	+	`end`
33	+
34	+	`test "tokens fence prepare, apply, verify, commit, and readiness" do`
35	+	`fixture = versions("TokenFence")`
36	+	`request = request(fixture)`
37	+
38	+	`assert {:ok, %{"token" => token, "prior" => [prior]}} =`
39	+	`DeploymentNode.prepare(request)`
40	+
41	+	`assert prior["sha256"] == BuildArtifact.digest(fixture.prior_binary)`
42	+	`refute DeploymentNode.health()["ready"]`
43	+
44	+	`assert {:error, :unknown_or_expired_token} =`
45	+	`DeploymentNode.apply_candidate(request.deployment_id, "wrong-token")`
46	+
47	+	`assert {:ok, %{"phase" => "applied"}} =`
48	+	`DeploymentNode.apply_candidate(request.deployment_id, token)`
49	+
50	+	`assert fixture.module.revision() == "candidate"`
51	+
52	+	`assert {:ok, %{"deployment_ready" => true, "revision" => revision}} =`
53	+	`DeploymentNode.verify_candidate(request.deployment_id, token)`
54	+
55	+	`assert revision == request.sha`
56	+	`assert {:ok, %{"phase" => "committed"}} = DeploymentNode.commit(request.deployment_id, token)`
57	+	`refute DeploymentNode.health()["ready"]`
58	+
59	+	`assert {:ok, %{"phase" => "live"}} =`
60	+	`DeploymentNode.finalize(request.deployment_id, token)`
61	+
62	+	`assert DeploymentNode.health()["ready"]`
63	+	`assert DeploymentNode.health()["revision"] == request.sha`
64	+	`end`
65	+
66	+	`test "rollback restores and verifies exact prior object code" do`
67	+	`fixture = versions("ExactRollback")`
68	+	`request = request(fixture)`
69	+	`prior_digest = BuildArtifact.digest(fixture.prior_binary)`
70	+
71	+	`assert {:ok, %{"token" => token}} = DeploymentNode.prepare(request)`
72	+	`assert {:ok, _response} = DeploymentNode.apply_candidate(request.deployment_id, token)`
73	+	`assert {:ok, _response} = DeploymentNode.verify_candidate(request.deployment_id, token)`
74	+	`assert fixture.module.revision() == "candidate"`
75	+
76	+	`assert {:ok, %{"restored" => true}} = DeploymentNode.rollback(request.deployment_id, token)`
77	+	`assert fixture.module.revision() == "prior"`
78	+
79	+	`assert {module, binary, _file} = :code.get_object_code(fixture.module)`
80	+	`assert module == fixture.module`
81	+	`assert BuildArtifact.digest(binary) == prior_digest`
82	+	`assert DeploymentNode.health()["ready"]`
83	+	`end`
84	+
85	+	`test "an expired token restores applied code before removing the transaction" do`
86	+	`fixture = versions("ExpiredToken")`
87	+	`request = request(fixture)`
88	+
89	+	`assert {:ok, %{"token" => token}} = DeploymentNode.prepare(request)`
90	+	`assert {:ok, _response} = DeploymentNode.apply_candidate(request.deployment_id, token)`
91	+	`assert fixture.module.revision() == "candidate"`
92	+
93	+	`expire_token(token)`
94	+
95	+	`send(DeploymentNode, :sweep_expired)`
96	+	`_state = :sys.get_state(DeploymentNode)`
97	+
98	+	`assert fixture.module.revision() == "prior"`
99	+	`assert DeploymentNode.health()["ready"]`
100	+
101	+	`assert {:error, :unknown_or_expired_token} =`
102	+	`DeploymentNode.verify_candidate(request.deployment_id, token)`
103	+	`end`
104	+
105	+	`test "an expired committed token follows immutable live database authority" do`
106	+	`fixture = versions("CommittedToken")`
107	+	`request = request(fixture)`
108	+
109	+	`%Target{id: request.target_id}`
110	+	`\|> Target.changeset(%{`
111	+	`repo: request.repo,`
112	+	`sha: request.sha,`
113	+	`promoted_by: "operator:test",`
114	+	`status: "live",`
115	+	`details: %{`
116	+	`"deployment_id" => request.deployment_id,`
117	+	`"artifact_digest" => request.artifact_digest`
118	+	`}`
119	+	`})`
120	+	`\|> Repo.insert!()`
121	+
122	+	`assert {:ok, %{"token" => token}} = DeploymentNode.prepare(request)`
123	+	`assert {:ok, _response} = DeploymentNode.apply_candidate(request.deployment_id, token)`
124	+	`assert {:ok, _response} = DeploymentNode.verify_candidate(request.deployment_id, token)`
125	+	`assert {:ok, _response} = DeploymentNode.commit(request.deployment_id, token)`
126	+
127	+	`expire_token(token)`
128	+	`send(DeploymentNode, :sweep_expired)`
129	+	`_state = :sys.get_state(DeploymentNode)`
130	+
131	+	`assert fixture.module.revision() == "candidate"`
132	+	`assert DeploymentNode.health()["ready"]`
133	+	`assert DeploymentNode.health()["revision"] == request.sha`
134	+
135	+	`assert {:error, :unknown_or_expired_token} =`
136	+	`DeploymentNode.rollback(request.deployment_id, token)`
137	+	`end`
138	+
139	+	`test "an expired committed token rolls back when durable authority refused it" do`
140	+	`fixture = versions("RejectedCommit")`
141	+	`request = request(fixture)`
142	+
143	+	`assert {:ok, %{"token" => token}} = DeploymentNode.prepare(request)`
144	+	`assert {:ok, _response} = DeploymentNode.apply_candidate(request.deployment_id, token)`
145	+	`assert {:ok, _response} = DeploymentNode.verify_candidate(request.deployment_id, token)`
146	+	`assert {:ok, _response} = DeploymentNode.commit(request.deployment_id, token)`
147	+
148	+	`expire_token(token)`
149	+	`send(DeploymentNode, :sweep_expired)`
150	+	`_state = :sys.get_state(DeploymentNode)`
151	+
152	+	`assert fixture.module.revision() == "prior"`
153	+	`assert DeploymentNode.health()["ready"]`
154	+	`end`
155	+
156	+	`test "an in-flight database commit extends a committed token without serving" do`
157	+	`fixture = versions("PendingCommit")`
158	+	`request = request(fixture)`
159	+
160	+	`target =`
161	+	`%Target{id: request.target_id}`
162	+	`\|> Target.changeset(%{`
163	+	`repo: request.repo,`
164	+	`sha: request.sha,`
165	+	`promoted_by: "operator:test",`
166	+	`status: "deploying"`
167	+	`})`
168	+	`\|> Repo.insert!()`
169	+
170	+	`assert {:ok, %{"token" => token}} = DeploymentNode.prepare(request)`
171	+	`assert {:ok, _response} = DeploymentNode.apply_candidate(request.deployment_id, token)`
172	+	`assert {:ok, _response} = DeploymentNode.verify_candidate(request.deployment_id, token)`
173	+	`assert {:ok, _response} = DeploymentNode.commit(request.deployment_id, token)`
174	+
175	+	`expire_token(token)`
176	+	`send(DeploymentNode, :sweep_expired)`
177	+	`_state = :sys.get_state(DeploymentNode)`
178	+
179	+	`assert fixture.module.revision() == "candidate"`
180	+	`refute DeploymentNode.health()["ready"]`
181	+
182	+	`target`
183	+	`\|> Target.status_changeset("live", %{`
184	+	`"deployment_id" => request.deployment_id,`
185	+	`"artifact_digest" => request.artifact_digest`
186	+	`})`
187	+	`\|> Repo.update!()`
188	+
189	+	`expire_token(token)`
190	+	`send(DeploymentNode, :sweep_expired)`
191	+	`_state = :sys.get_state(DeploymentNode)`
192	+
193	+	`assert DeploymentNode.health()["ready"]`
194	+	`assert fixture.module.revision() == "candidate"`
195	+	`end`
196	+
197	+	`test "a supervised participant restart preserves the rollback fence" do`
198	+	`fixture = versions("ParticipantRestart")`
199	+	`request = request(fixture)`
200	+
201	+	`assert {:ok, %{"token" => token}} = DeploymentNode.prepare(request)`
202	+	`assert {:ok, _response} = DeploymentNode.apply_candidate(request.deployment_id, token)`
203	+	`assert fixture.module.revision() == "candidate"`
204	+
205	+	`assert :ok = Supervisor.terminate_child(OpenAgents.Supervisor, DeploymentNode)`
206	+	`refute DeploymentNode.health()["ready"]`
207	+	`refute DeploymentNode.health()["participant_ready"]`
208	+	`assert {:ok, _pid} = Supervisor.restart_child(OpenAgents.Supervisor, DeploymentNode)`
209	+	`refute DeploymentNode.health()["ready"]`
210	+
211	+	`assert {:ok, %{"restored" => true}} =`
212	+	`DeploymentNode.rollback(request.deployment_id, token)`
213	+
214	+	`assert fixture.module.revision() == "prior"`
215	+	`assert DeploymentNode.health()["ready"]`
216	+	`end`
217	+
218	+	`defp versions(suffix) do`
219	+	`name = "OpenAgents.Scratch.#{suffix}#{System.unique_integer([:positive])}"`
220	+	`module = Module.concat([name])`
221	+	`prior_binary = compile(name, "prior")`
222	+	`unload(module)`
223	+	`candidate_binary = compile(name, "candidate")`
224	+
225	+	`prior_dir =`
226	+	`Path.join(System.tmp_dir!(), "deployment-prior-#{System.unique_integer([:positive])}")`
227	+
228	+	`File.mkdir_p!(prior_dir)`
229	+	`File.write!(Path.join(prior_dir, Atom.to_string(module) <> ".beam"), prior_binary)`
230	+	`true = :code.add_patha(to_charlist(prior_dir))`
231	+	`unload(module)`
232	+	`assert {:module, ^module} = :code.load_file(module)`
233	+
234	+	`on_exit(fn ->`
235	+	`unload(module)`
236	+	`:code.del_path(to_charlist(prior_dir))`
237	+	`File.rm_rf(prior_dir)`
238	+	`end)`
239	+
240	+	`sha = random_sha()`
241	+	`built = ArtifactFixtures.create!("openagents.com", sha, [{name, candidate_binary}])`
242	+
243	+	`%{`
244	+	`module: module,`
245	+	`prior_binary: prior_binary,`
246	+	`built: built,`
247	+	`sha: sha`
248	+	`}`
249	+	`end`
250	+
251	+	`defp request(fixture) do`
252	+	`manifest_digest =`
253	+	`fixture.built.manifest`
254	+	`\|> BuildProtocol.canonical_json()`
255	+	`\|> BuildArtifact.digest()`
256	+
257	+	`%{`
258	+	`artifact_bytes: fixture.built.bytes,`
259	+	`artifact_digest: fixture.built.digest,`
260	+	`build_id: fixture.built.build_id,`
261	+	`deployment_id: Ecto.UUID.generate(),`
262	+	`expected_nodes: [to_string(Node.self())],`
263	+	`manifest_digest: manifest_digest,`
264	+	`repo: "openagents.com",`
265	+	`sha: fixture.sha,`
266	+	`target_id: Ecto.UUID.generate()`
267	+	`}`
268	+	`end`
269	+
270	+	`defp compile(name, revision) do`
271	+	`[{_module, binary}] =`
272	+	`Code.compile_string("defmodule #{name} do\n def revision, do: #{inspect(revision)}\nend")`
273	+
274	+	`binary`
275	+	`end`
276	+
277	+	`defp random_sha, do: 20 \|> :crypto.strong_rand_bytes() \|> Base.encode16(case: :lower)`
278	+
279	+	`defp unload(module) do`
280	+	`:code.purge(module)`
281	+	`:code.delete(module)`
282	+	`:code.purge(module)`
283	+	`end`
284	+
285	+	`defp reset_participant do`
286	+	`:persistent_term.erase({DeploymentNode, :state})`
287	+
288	+	`:sys.replace_state(DeploymentNode, fn state ->`
289	+	`%{state \| transactions: %{}, live: nil, divergence: nil, faults: %{}, notify: nil}`
290	+	`end)`
291	+	`end`
292	+
293	+	`defp expire_token(token) do`
294	+	`:sys.replace_state(DeploymentNode, fn state ->`
295	+	`update_in(`
296	+	`state,`
297	+	`[:transactions, token],`
298	+	`&%{&1 \| expires_at: System.monotonic_time(:millisecond) - 1}`
299	+	`)`
300	+	`end)`
301	+	`end`
302	+
303	+	`defp restore_env(key, nil), do: Application.delete_env(:openagents, key)`
304	+	`defp restore_env(key, value), do: Application.put_env(:openagents, key, value)`
305	+
306	+	`defp restore_persistent(:missing), do: :persistent_term.erase({DeploymentNode, :state})`
307	+
308	+	`defp restore_persistent(state),`
309	+	`do: :persistent_term.put({DeploymentNode, :state}, state)`
310	+	`end`


  alias OpenAgents.Forge.ArtifactFixtures
  alias OpenAgents.Forge.DeployReceipt
  alias OpenAgents.Forge.DeploymentNode
  alias OpenAgents.Forge.HotLoader
  alias OpenAgents.Forge.PushReceipt
  alias OpenAgents.Forge.Target

  # async: false tests (start_owner!(shared: true)), so the HotLoader
  # GenServer shares the test connection without an explicit mode call.
  setup do
    base = Path.join(System.tmp_dir!(), "hot-loader-#{System.unique_integer([:positive])}")
    previous_data = Application.get_env(:openagents, :forge_data_dir)
    previous_node_state = :sys.get_state(DeploymentNode)
    previous_persisted = :persistent_term.get({DeploymentNode, :state}, :missing)
    Application.put_env(:openagents, :forge_data_dir, base)
    :persistent_term.erase({DeploymentNode, :state})

    :sys.replace_state(DeploymentNode, fn state ->
      %{state | transactions: %{}, live: nil, divergence: nil, faults: %{}, notify: nil}
    end)

    pid =
      case Process.whereis(HotLoader) do
        nil -> start_supervised!(HotLoader)
        pid -> pid
      end

    on_exit(fn ->
      if previous_data,
        do: Application.put_env(:openagents, :forge_data_dir, previous_data),
        else: Application.delete_env(:openagents, :forge_data_dir)

      File.rm_rf(base)
      :sys.replace_state(DeploymentNode, fn _state -> previous_node_state end)
      restore_persistent(previous_persisted)
    end)

    %{loader: pid}
  end


    :code.purge(mod)
  end

  defp restore_persistent(:missing), do: :persistent_term.erase({DeploymentNode, :state})

  defp restore_persistent(state),
    do: :persistent_term.put({DeploymentNode, :state}, state)

  defp artifact(entries, sha) do
    built = ArtifactFixtures.create!("openagents.com", sha, entries)
    path = ArtifactFixtures.write!(built)


    broadcast_build_ready(loader, build_payload(target, sha, artifact))

    receipt = deploy_receipt(sha)
    assert Code.ensure_loaded?(mod)
    assert mod.revision() == revision

    assert Repo.get!(Target, target.id).status == "live"

    receipt = deploy_receipt(sha)
    assert receipt.result == "live"
    assert receipt.repo == "openagents.com"
    assert receipt.target_id == target.id
    assert receipt.modules == [name]
    assert receipt.canary == "ok"
    assert receipt.nodes == ["#{Node.self()}=ok"]
    assert receipt.nodes == ["#{Node.self()}=committed"]
    assert receipt.expected_nodes == [to_string(Node.self())]
    assert receipt.rollback_verified == nil
    assert is_binary(receipt.artifact_digest)
    assert is_binary(receipt.manifest_digest)

    assert_receive {:forge_deploy, %{repo: "openagents.com", sha: ^sha, result: "live"}}
  end

  """

  use OpenAgents.DataCase, async: false
  alias OpenAgents.Forge.{FakeBuildExecutor, Targets}
  alias OpenAgents.Forge.{DeploymentNode, FakeBuildExecutor, Targets}

  defmodule TestPipeline do
    @moduledoc false

      allowlist: Application.get_env(:openagents, :forge_hot_load_allowlist)
    }

    previous_node_state = :sys.get_state(DeploymentNode)
    previous_persisted = :persistent_term.get({DeploymentNode, :state}, :missing)
    :persistent_term.erase({DeploymentNode, :state})

    :sys.replace_state(DeploymentNode, fn state ->
      %{state | transactions: %{}, live: nil, divergence: nil, faults: %{}, notify: nil}
    end)

    Application.put_env(:openagents, :forge_data_dir, Path.join(base, "data"))
    Application.put_env(:openagents, :forge_wal_dir, Path.join(base, "wal"))
    Application.put_env(:openagents, :forge_build_executor, FakeBuildExecutor)

      restore(:forge_wal_dir, previous.wal)
      restore(:forge_build_executor, previous.executor)
      restore(:forge_hot_load_allowlist, previous.allowlist)
      :sys.replace_state(DeploymentNode, fn _state -> previous_node_state end)
      restore_persistent(previous_persisted)
      File.rm_rf(base)
    end)


  defp restore(key, nil), do: Application.delete_env(:openagents, key)
  defp restore(key, value), do: Application.put_env(:openagents, key, value)

  defp restore_persistent(:missing), do: :persistent_term.erase({DeploymentNode, :state})

  defp restore_persistent(state),
    do: :persistent_term.put({DeploymentNode, :state}, state)

  defp purge!(mod) do
    :code.purge(mod)
    :code.delete(mod)


    # The module is actually live in this runtime.
    module = Module.concat([module_name])
    on_exit(fn -> purge!(module) end)
    assert module.revision() == "loop-v0"

    # Target walked the full lifecycle.

defmodule OpenAgents.Forge.TargetLifecycleTest do
  use OpenAgents.DataCase, async: false
  alias OpenAgents.Forge.{Repos, Targets}
  alias OpenAgents.Forge.{DeployReceipt, Repos, Targets}

  setup do
    base = Path.join(System.tmp_dir!(), "forge-targets-#{System.unique_integer([:positive])}")

106	106		`assert Targets.current("demo").id == second.id`
107	107		`assert length(Targets.recent("demo")) == 2`
108	108		`end`
	109	+
	110	+	`test "deployment ownership refuses a superseded built target", %{sha: sha} do`
	111	+	`{:ok, first} = Targets.promote("demo", sha, "operator:first")`
	112	+	`{:ok, _building} = Targets.advance(first.id, "building")`
	113	+	`{:ok, _built} = Targets.advance(first.id, "built")`
	114	+	`{:ok, second} = Targets.promote("demo", sha, "operator:second")`
	115	+
	116	+	`assert {:error, :superseded_target} = Targets.begin_deployment(first.id)`
	117	+	`assert Targets.current("demo").id == second.id`
	118	+	`assert Repo.get!(OpenAgents.Forge.Target, first.id).status == "built"`
	119	+	`end`
	120	+
	121	+	`test "duplicate deployment delivery cannot claim an active target twice", %{sha: sha} do`
	122	+	`{:ok, target} = Targets.promote("demo", sha, "operator:test")`
	123	+	`{:ok, _building} = Targets.advance(target.id, "building")`
	124	+	`{:ok, _built} = Targets.advance(target.id, "built")`
	125	+
	126	+	`assert {:ok, %{status: "deploying"}} = Targets.begin_deployment(target.id)`
	127	+
	128	+	`assert {:error, {:invalid_transition, "deploying", "deploying"}} =`
	129	+	`Targets.begin_deployment(target.id)`
	130	+
	131	+	`assert Repo.get!(OpenAgents.Forge.Target, target.id).status == "deploying"`
	132	+	`end`
	133	+
	134	+	`test "fleet commit writes the live target and terminal receipt atomically", %{sha: sha} do`
	135	+	`{:ok, target} = Targets.promote("demo", sha, "operator:test")`
	136	+	`{:ok, _building} = Targets.advance(target.id, "building")`
	137	+	`{:ok, _built} = Targets.advance(target.id, "built")`
	138	+	`{:ok, _deploying} = Targets.begin_deployment(target.id)`
	139	+
	140	+	`deployment_id = Ecto.UUID.generate()`
	141	+	`digest = String.duplicate("a", 64)`
	142	+	`manifest_digest = String.duplicate("b", 64)`
	143	+
	144	+	`assert {:ok, %{target: live, receipt: receipt}} =`
	145	+	`Targets.finish_deployment(`
	146	+	`target.id,`
	147	+	`"live",`
	148	+	`%{"deployment_id" => deployment_id},`
	149	+	`%{`
	150	+	`deployment_id: deployment_id,`
	151	+	`artifact_digest: digest,`
	152	+	`manifest_digest: manifest_digest,`
	153	+	`expected_nodes: ["one", "two", "three"],`
	154	+	`nodes: ["one=committed", "two=committed", "three=committed"],`
	155	+	`node_results: %{`
	156	+	`"one" => "committed",`
	157	+	`"two" => "committed",`
	158	+	`"three" => "committed"`
	159	+	`},`
	160	+	`rollback_verified: nil`
	161	+	`}`
	162	+	`)`
	163	+
	164	+	`assert live.status == "live"`
	165	+	`assert receipt.result == "live"`
	166	+	`assert receipt.deployment_id == deployment_id`
	167	+	`assert receipt.artifact_digest == digest`
	168	+	`assert receipt.manifest_digest == manifest_digest`
	169	+	`end`
	170	+
	171	+	`test "PostgreSQL rejects deployment receipt mutation", %{sha: sha} do`
	172	+	`target =`
	173	+	`%OpenAgents.Forge.Target{}`
	174	+	`\|> OpenAgents.Forge.Target.changeset(%{`
	175	+	`repo: "demo",`
	176	+	`sha: sha,`
	177	+	`promoted_by: "operator:test",`
	178	+	`status: "failed"`
	179	+	`})`
	180	+	`\|> Repo.insert!()`
	181	+
	182	+	`receipt =`
	183	+	`%DeployReceipt{}`
	184	+	`\|> DeployReceipt.changeset(%{`
	185	+	`repo: "demo",`
	186	+	`sha: sha,`
	187	+	`target_id: target.id,`
	188	+	`result: "failed"`
	189	+	`})`
	190	+	`\|> Repo.insert!()`
	191	+
	192	+	`assert_raise Postgrex.Error, ~r/forge deployment receipts are immutable/, fn ->`
	193	+	`Repo.transaction(`
	194	+	`fn ->`
	195	+	`Repo.query!("UPDATE forge_deploys SET result = 'live' WHERE id = $1", [`
	196	+	`Ecto.UUID.dump!(receipt.id)`
	197	+	`])`
	198	+	`end,`
	199	+	`mode: :savepoint`
	200	+	`)`
	201	+	`end`
	202	+	`end`
109	203		`end`

    assert [first | _rest] = projection["nodes"]
    assert first["label"] =~ ~r/^node \d+$/
    assert first["reachable"] == true
    assert is_boolean(first["ready"])
    assert Map.keys(first["boot"]) == ["attempts", "ready", "reason", "retry_in_ms", "state"]
    refute inspect(first["boot"]) =~ ~r/[0-9a-f]{64}/

    assert %{"phase" => _phase, "ready" => _ready, "reason" => _reason} =
             first["deployment"]

    assert is_integer(first["uptime_seconds"])

    # Counts only, never content.

    assert refreshed["schema"] == first["schema"]
  end

  test "a single forge node cannot report configured fleet quorum" do
    previous_lane = Application.get_env(:openagents, :forge_deploy_lane_enabled)
    previous_size = Application.get_env(:openagents, :forge_expected_fleet_size)
    Application.put_env(:openagents, :forge_deploy_lane_enabled, true)
    Application.put_env(:openagents, :forge_expected_fleet_size, 3)

    on_exit(fn ->
      restore_env(:forge_deploy_lane_enabled, previous_lane)
      restore_env(:forge_expected_fleet_size, previous_size)
    end)

    projection = NetworkStatus.projection(refresh: true)
    refute projection["cluster"]["quorum"]
    assert projection["status"] == "degraded"
  end

  describe "forge section (#126)" do
    alias OpenAgents.Forge.{DeployReceipt, Target}


143	166		`assert loop["median_ms"] == 20_000`
144	167		`end`
145	168		`end`
	169	+
	170	+	`defp restore_env(key, nil), do: Application.delete_env(:openagents, key)`
	171	+	`defp restore_env(key, value), do: Application.put_env(:openagents, key, value)`
146	172		`end`

9	9		`"revision" => OpenAgents.BuildInfo.revision()`
10	10		`}`
11	11		`end`
	12	+
	13	+	`test "refuses readiness while boot code diverges from the live target", %{conn: conn} do`
	14	+	`key = {OpenAgents.Forge.BootConverge, :state}`
	15	+	`previous = :persistent_term.get(key, :missing)`
	16	+
	17	+	`:persistent_term.put(key, %{`
	18	+	`"state" => "degraded",`
	19	+	`"ready" => false,`
	20	+	`"reason" => "artifact_missing",`
	21	+	`"sha" => String.duplicate("a", 40),`
	22	+	`"attempts" => 2,`
	23	+	`"retry_in_ms" => 1_000`
	24	+	`})`
	25	+
	26	+	`on_exit(fn ->`
	27	+	`if previous == :missing,`
	28	+	`do: :persistent_term.erase(key),`
	29	+	`else: :persistent_term.put(key, previous)`
	30	+	`end)`
	31	+
	32	+	`conn = get(conn, ~p"/healthz")`
	33	+
	34	+	`assert json_response(conn, 503) == %{`
	35	+	`"status" => "unavailable",`
	36	+	`"reason" => "runtime_not_ready",`
	37	+	`"boot_converged" => false,`
	38	+	`"deployment_ready" => true`
	39	+	`}`
	40	+	`end`
12	41		`end`

2	2
3	3		`Date: 2026-08-20`
4	4
5		-	`Status: In progress; Gates 0–5 and 7–9 complete, Gate 6 application controls locally verified`
	5	+	`Status: In progress; Gates 0–5 and 7–10 complete, Gate 6 application controls locally verified`
6	6
7	7		`## Outcome`
8	8

893	893		`disables terminal prompting.`
894	894		`- Added deterministic BEAM normalization and canonical artifact manifests with`
895	895		`source, baseline, Elixir, OTP, ERTS, application version/spec, and dependency`
896		-	`lock identities plus complete added, changed, and deleted module sets.`
	896	+	`lock identities plus complete added, changed, and deleted module sets. Fixed`
	897	+	`tar ownership and time metadata so identical inputs remain byte-for-byte`
	898	+	`reproducible across wall-clock boundaries.`
897	899		`- Addressed artifacts by the full tar SHA-256 in both the local cache and`
898	900		durable WAL store. A target cannot become `built` until independent
899	901		`verification, local publication, durable storage, and receipt completion all`

921	923		`classification, verification, recovery, retention, and staging proof in the`
922	924		`[forge build lane runbook](operations/forge-build-lane.md).`
923	925
924		-	`Gate 9 is complete. Keep the staging deploy lane disabled until Gate 10 makes`
925		-	`fleet application transactional. Gates 12–15 must still exercise the real`
926		-	`builder image against the exact staging SHA and retain the image, build,`
927		-	`artifact, and output-proof identities; local success is not staging admission.`
	926	+	`Gate 9 is complete, and Gate 10 now provides transactional fleet application.`
	927	+	`Keep the staging deploy lane disabled until Gate 11 adds the fallback lanes and`
	928	+	`Gate 12 creates the isolated distributed staging environment. Gates 12–15 must`
	929	+	`exercise the real builder image against the exact staging SHA and retain the`
	930	+	`image, build, artifact, and output-proof identities; local success is not`
	931	+	`staging admission.`
928	932
929	933		`## Gate 10: Make fleet deployment transactional`
930	934

1381	1460		`- [x] Voice recording starts only after generation admission.`
1382	1461		`- [x] Build requests are structured, unique, bounded, and non-executable.`
1383	1462		`- [x] Artifacts are immutable, digested, manifest-checked, and durably stored.`
1384		-	`- [ ] Fleet deployment is transactional and rolls back every affected node.`
1385		-	`- [ ] Boot convergence controls readiness.`
	1463	+	`- [x] Fleet deployment is transactional and rolls back every affected node.`
	1464	+	`- [x] Boot convergence controls readiness.`
1386	1465		`- [ ] Relup and rolling replacement pass their staging drills.`
1387	1466		`- [ ] Owned local gates produce exact-SHA receipts.`
1388	1467		`- [ ] Web and distributed staging are isolated from production.`

1	1		`defmodule OpenAgents.Forge.DeployReceipt do`
2	2		`@moduledoc """`
3		-	Receipt for one hot-deploy attempt (`forge_deploys`, roadmap P4): what was
4		-	`loaded (or refused), on which nodes, with the canary outcome and the`
5		-	`first-class push→live duration measured from the matching push receipt.`
6		-	Every outcome is receipted — `live`, `reverted`, `needs_rolling_replace`,
7		-	and `failed` alike — so the deploy lane never fails silently.
	3	+	`Immutable terminal receipt for one fleet deployment attempt. The receipt`
	4	+	`binds the candidate to its deployment, artifact, manifest, expected member`
	5	+	`set, bounded per-node outcomes, rollback verification, and push-to-live`
	6	+	`duration. PostgreSQL rejects updates and deletes after insertion.`
8	7		`"""`
9	8
10	9		`use Ecto.Schema`

194	194		`forge_artifact_dir: "/var/lib/openagents/artifacts",`
195	195		`forge_build_timeout_ms: 300_000,`
196	196		`forge_build_output_retention_ms: 604_800_000,`
	197	+	`forge_deploy_timeout_ms: 15_000,`
	198	+	`forge_deploy_token_ttl_ms: 120_000,`
	199	+	`forge_boot_retry_min_ms: 1_000,`
	200	+	`forge_boot_retry_max_ms: 30_000,`
197	201		`forge_artifact_store: :local,`
198	202		`forge_build_executor: OpenAgents.Forge.BuildExecutor.Sidecar,`
199	203		`forge_expected_fleet_size: 1,`

284	284		`parse_integer.("OPENAGENTS_FORGE_BUILD_TIMEOUT_MS", 30_000..1_800_000),`
285	285		`forge_build_output_retention_ms:`
286	286		`parse_integer.("OPENAGENTS_FORGE_BUILD_OUTPUT_RETENTION_MS", 86_400_000..2_592_000_000),`
	287	+	`forge_deploy_timeout_ms: parse_integer.("OPENAGENTS_FORGE_DEPLOY_TIMEOUT_MS", 1_000..120_000),`
	288	+	`forge_deploy_token_ttl_ms:`
	289	+	`parse_integer.("OPENAGENTS_FORGE_DEPLOY_TOKEN_TTL_MS", 30_000..1_800_000),`
	290	+	`forge_boot_retry_min_ms: parse_integer.("OPENAGENTS_FORGE_BOOT_RETRY_MIN_MS", 100..60_000),`
	291	+	`forge_boot_retry_max_ms: parse_integer.("OPENAGENTS_FORGE_BOOT_RETRY_MAX_MS", 1_000..300_000),`
287	292		`forge_artifact_store: forge_artifact_store,`
288	293		`forge_build_executor: forge_build_executor,`
289	294		`forge_expected_fleet_size: parse_integer.("OPENAGENTS_FORGE_EXPECTED_FLEET_SIZE", 1..100),`

220	220
221	221		`Implementation status: Gate 9 completed this phase on 2026-08-20. See the`
222	222		`[forge build lane runbook](operations/forge-build-lane.md) for the deployed`
223		-	`contract. Fleet application remains disabled pending the transactional work in`
224		-	`Phase 5 and Gate 10.`
	223	+	`contract. Gate 10 completed transactional fleet application, but the lane`
	224	+	`remains disabled until isolated distributed staging and the fallback classes`
	225	+	`are ready.`
225	226
226	227		`## Phase 5: Add direct BEAM classification and local canary loading`
227	228

261	262
262	263		`Exit criteria: multi-node tests prove all-node success, remote timeout, remote load failure, exact fleet rollback, duplicate event handling, node membership changes during deployment, and refusal to mark a divergent fleet live.`
263	264
	265	+	`Implementation status: Gate 10 completed this phase on 2026-08-20. The`
	266	+	`[transactional deployment runbook](operations/forge-transactional-deployment.md)`
	267	+	`documents the implemented node protocol, atomic receipt boundary, readiness`
	268	+	`behavior, and three-node proofs.`
	269	+
264	270		`## Phase 7: Converge replacement and restarted nodes`
265	271
266	272		Add `OpenAgents.Forge.BootConverge` as a synchronous boot step after `OpenAgents.Repo` starts and before `DNSCluster`, `OpenAgents.PubSub`, and `OpenAgentsWeb.Endpoint` start.

269	275		`2. If the target has a direct BEAM artifact, load it from the local cache or fetch it from the durable artifact store.`
270	276		`3. Verify the digest, allowlist, and module manifest before loading.`
271	277		4. Record the outcome in `:persistent_term` for health and status reports.
272		-	`5. Treat a live target with an empty artifact as converged.`
	278	+	`5. Treat a live target with no artifact as converged only when the image`
	279	+	`revision exactly matches that target.`
273	280		`6. Start on image code if the target or artifact cannot load, but keep readiness false while the node differs from the live target.`
274	281		`7. Retry convergence with bounded backoff so an artifact-store interruption does not require another restart.`
275	282		`8. Prune old local artifacts while retaining the current target and immediate rollback artifacts. Never prune the durable copy or receipts.`
276	283
277	284		`Exit criteria: tests cover local convergence, durable fetch, no target, non-live target, empty artifact, missing artifact, off-allowlist content, corrupt digest, and readiness behavior.`
278	285
	286	+	`Implementation status: Gate 10 completed readiness-bound convergence on`
	287	+	`2026-08-20. It starts synchronously before discovery and the endpoint, fetches`
	288	+	`from durable storage on a cold cache, retains the immediate predecessor, prunes`
	289	+	`older digest-addressed cache entries, periodically checks the durable live`
	290	+	`target, retries with bounded backoff, and keeps divergent code out of`
	291	+	`readiness.`
	292	+
279	293		`## Phase 8: Add the production relup lane`
280	294
281	295		Use relups for changes that need OTP-coordinated module updates or `code_change/3` state migration but do not cross a structural boundary.

2	2
3	3		`Date: 2026-08-20`
4	4
5		-	`Status: Implemented locally; keep staging deployment disabled until Gate 10 is complete`
	5	+	`Status: Implemented locally; the transactional deployment lane now consumes its verified artifacts`
6	6
7	7		`## Purpose`
8	8
9	9		`The forge build lane turns one operator-promoted, fully qualified Git commit`
10		-	`into a reproducible BEAM artifact. Compilation runs in an isolated builder`
11		-	`container. The public release receives no compiler, Docker socket, Git`
	10	+	`into a reproducible BEAM artifact with fixed tar ownership and time metadata.`
	11	+	`Compilation runs in an isolated builder container. The public release receives`
	12	+	`no compiler, Docker socket, Git`
12	13		`credential, or ability to execute queue contents.`
13	14
14		-	`This lane only produces and verifies artifacts. Gate 10 owns transactional`
15		-	`fleet deployment. Do not enable direct staging loads merely because a build`
16		-	`completes.`
	15	+	`This lane only produces and verifies artifacts. The`
	16	+	`[transactional deployment runbook](forge-transactional-deployment.md) owns`
	17	+	`fleet application. Do not enable direct staging loads merely because a build`
	18	+	`completes; the isolated distributed staging lane and fallback deployment`
	19	+	`classes remain required.`
17	20
18	21		`## Runtime roles`
19	22

136	136		\| `OPENAGENTS_FORGE_ARTIFACT_DIR` \| Absolute durable path outside `/tmp` \|
137	137		\| `OPENAGENTS_FORGE_BUILD_TIMEOUT_MS` \| `300000`; admitted range 30 seconds to 30 minutes \|
138	138		\| `OPENAGENTS_FORGE_BUILD_OUTPUT_RETENTION_MS` \| `604800000` (seven days); admitted range one to 30 days \|
	139	+	\| `OPENAGENTS_FORGE_DEPLOY_TIMEOUT_MS` \| `15000`; admitted range one to 120 seconds \|
	140	+	\| `OPENAGENTS_FORGE_DEPLOY_TOKEN_TTL_MS` \| `120000`; admitted range 30 seconds to 30 minutes and at least eight deployment timeouts \|
	141	+	\| `OPENAGENTS_FORGE_BOOT_RETRY_MIN_MS` \| `1000`; admitted range 100 milliseconds to one minute \|
	142	+	\| `OPENAGENTS_FORGE_BOOT_RETRY_MAX_MS` \| `30000`; admitted range one to 300 seconds and not less than the minimum \|
139	143		\| `OPENAGENTS_CODING_JOBS_DIR` \| Absolute durable path outside `/tmp` when work or computers are enabled \|
140	144		\| `OPENAGENTS_RA_DATA_DIR` \| Absolute durable path outside `/tmp` when Ra is enabled \|
141	145		\| `OPENAGENTS_RA_EXPECTED_SIZE` \| At least `3` when Ra is enabled \|

37	37		`children = [`
38	38		`OpenAgentsWeb.Telemetry,`
39	39		`OpenAgents.Repo,`
	40	+	`# Deployment identity and boot convergence must settle before cluster`
	41	+	`# discovery or the endpoint can make this node externally reachable.`
	42	+	`OpenAgents.Forge.DeploymentNode,`
	43	+	`OpenAgents.Forge.BootConverge,`
40	44		`{DNSCluster, query: Application.get_env(:openagents, :dns_cluster_query) \|\| :ignore},`
41	45		`{Phoenix.PubSub, name: OpenAgents.PubSub},`
42	46		`OpenAgents.RuntimeSupervisor,`

58	58
59	59		`@doc """`
60	60		`A bounded health report for this node. Liveness means the VM and BEAM are`
61		-	`running; readiness is a runtime assertion that will be driven by boot`
62		-	`convergence and application health checks in later phases.`
	61	+	`running. Readiness requires successful boot convergence and no active or`
	62	+	`divergent forge deployment.`
63	63		`"""`
64	64		`@spec local_report() :: map()`
65	65		`def local_report do`
	66	+	`boot = OpenAgents.Forge.BootConverge.state()`
	67	+	`boot_ready? = OpenAgents.Forge.BootConverge.ready?()`
	68	+	`deployment = OpenAgents.Forge.DeploymentNode.health()`
	69	+
66	70		`%{`
67	71		`"schema" => "openagents.cluster_health.v1",`
68	72		`"node" => to_string(Node.self()),`
69	73		`"version" => to_string(Application.spec(:openagents, :vsn) \|\| "unknown"),`
70		-	`"revision" => OpenAgents.BuildInfo.revision(),`
71		-	`"boot_converged" => nil,`
	74	+	`"revision" => deployment["revision"] \|\| boot["sha"] \|\| OpenAgents.BuildInfo.revision(),`
	75	+	`"boot_converged" => boot_ready?,`
	76	+	`"deployment_ready" => deployment["participant_ready"],`
72	77		`"uptime_ms" => uptime_ms(),`
73	78		`"live" => true,`
74		-	`"ready" => true`
	79	+	`"ready" => boot_ready? and deployment["ready"] == true`
75	80		`}`
76	81		`end`
77	82

1	1		`defmodule OpenAgents.Forge.BootConverge do`
2	2		`@moduledoc """`
3		-	`Boot convergence (roadmap P6, issue #123): on node start — after the Repo,`
4		-	`before Horde/Ra membership or the endpoint — load the current live fleet`
5		-	`target's beam artifact, so **a node converges to the forge's target`
6		-	`commit, not to the image**. Image rebakes become structural-only.`
7		-
8		-	`Honesty rules:`
9		-	`- If there is no live target, no local artifact (a freshly replaced node`
10		-	`has an empty partition — artifacts are per-node cache), an off-allowlist`
11		-	`module, or any error, the node boots on image code and records why —`
12		-	it NEVER refuses to start. The state is published in `state/0` and on
13		-	`the public status page's per-node report.`
14		-	`- The same operator-owned allowlist as the hot-load lane gates what boot`
15		-	`convergence will load: convergence is a replay of an approved deploy,`
16		-	`not a second authority (SELF-EDIT-001: hot-loaded code is a projection`
17		-	`of the promoted commit).`
18		-
19		-	Runs synchronously inside `start_link/1` and returns `:ignore` — it is a
20		-	`boot step, not a process.`
	3	+	`Gates readiness on convergence to the newest immutable live target.`
	4	+
	5	+	`A cold node fetches the live artifact from durable storage when its local`
	6	+	`digest-addressed cache is empty, verifies the same artifact and manifest`
	7	+	`identities used during promotion, and installs it through the transactional`
	8	+	`node participant. A divergent node stays out of readiness and retries with`
	9	+	`bounded exponential backoff. The public state contains only identity hashes,`
	10	+	`counters, and stable reason codes.`
21	11		`"""`
22	12
	13	+	`use GenServer`
	14	+
23	15		`require Logger`
24	16
25	17		`alias OpenAgents.Forge.BuildArtifact`
26	18		`alias OpenAgents.Forge.BuildProtocol`
27		-	`alias OpenAgents.Forge.HotLoader`
	19	+	`alias OpenAgents.Forge.DeploymentNode`
28	20		`alias OpenAgents.Forge.Repos`
29	21		`alias OpenAgents.Forge.Targets`
30	22
31	23		`@state_key {__MODULE__, :state}`
	24	+	`@default_retry_min_ms 1_000`
	25	+	`@default_retry_max_ms 30_000`
	26	+
	27	+	`def start_link(opts \\ []) do`
	28	+	`GenServer.start_link(__MODULE__, opts, name: Keyword.get(opts, :name, __MODULE__))`
	29	+	`end`
	30	+
	31	+	`@doc "Return this node's bounded convergence state."`
	32	+	`def state do`
	33	+	`:persistent_term.get(@state_key, initial_state(false))`
	34	+	`end`
	35	+
	36	+	`@doc "Return whether this node may enter external readiness."`
	37	+	`def ready?(repo \\ "openagents.com") do`
	38	+	`convergence = state()`
32	39
33		-	`def child_spec(_opts) do`
34		-	`%{id: __MODULE__, start: {__MODULE__, :start_link, []}, type: :worker, restart: :temporary}`
	40	+	`cond do`
	41	+	`convergence["ready"] != true -> false`
	42	+	`not Application.get_env(:openagents, :forge_boot_converge_enabled, false) -> true`
	43	+	`true -> durable_target_ready?(repo, convergence)`
	44	+	`end`
	45	+	`rescue`
	46	+	`_error -> false`
	47	+	`catch`
	48	+	`_kind, _reason -> false`
35	49		`end`
36	50
37		-	`def start_link do`
38		-	`if Application.get_env(:openagents, :forge_boot_converge_enabled, false) do`
39		-	`converge()`
	51	+	`@doc false`
	52	+	`def ready_for_deployment?(repo, target_id) do`
	53	+	`convergence = state()`
	54	+
	55	+	`cond do`
	56	+	`convergence["ready"] != true -> false`
	57	+	`not Application.get_env(:openagents, :forge_boot_converge_enabled, false) -> true`
	58	+	`true -> deployment_target_ready?(repo, target_id, convergence)`
40	59		`end`
	60	+	`rescue`
	61	+	`_error -> false`
	62	+	`catch`
	63	+	`_kind, _reason -> false`
	64	+	`end`
41	65
42		-	`:ignore`
	66	+	`@doc "Run one synchronous convergence attempt. Tests and repair tools use this API."`
	67	+	`def converge(repo \\ "openagents.com") do`
	68	+	`outcome = safe_attempt(repo, state()["attempts"] + 1)`
	69	+	`publish(outcome)`
	70	+	`outcome`
43	71		`end`
44	72
45		-	`@doc "The boot-convergence outcome for this node (for /status honesty)."`
46		-	`def state do`
47		-	`:persistent_term.get(@state_key, %{"state" => "image", "reason" => "not_attempted"})`
	73	+	`@doc false`
	74	+	`def mark_converged(identity) when is_map(identity) do`
	75	+	`outcome = %{`
	76	+	`"schema" => "openagents.forge.boot-convergence.v2",`
	77	+	`"state" => "converged",`
	78	+	`"ready" => true,`
	79	+	`"reason" => "fleet_commit",`
	80	+	`"sha" => identity.sha,`
	81	+	`"artifact_digest" => identity.artifact_digest,`
	82	+	`"manifest_digest" => identity.manifest_digest,`
	83	+	`"modules" => identity.modules,`
	84	+	`"attempts" => state()["attempts"],`
	85	+	`"retry_in_ms" => nil`
	86	+	`}`
	87	+
	88	+	`publish(outcome)`
48	89		`end`
49	90
50	91		`@doc false`
51		-	`def converge(repo \\ "openagents.com") do`
52		-	`outcome =`
53		-	`try do`
54		-	`attempt(repo)`
55		-	`rescue`
56		-	`error ->`
57		-	`%{"state" => "image", "reason" => OpenAgents.OperationalLog.code(error)}`
58		-	`catch`
59		-	`_kind, reason ->`
60		-	`%{"state" => "image", "reason" => OpenAgents.OperationalLog.code(reason)}`
	92	+	`def restore_state(%{"schema" => "openagents.forge.boot-convergence.v2"} = outcome),`
	93	+	`do: publish(outcome)`
	94	+
	95	+	`@impl true`
	96	+	`def init(opts) do`
	97	+	`repo = Keyword.get(opts, :repo, "openagents.com")`
	98	+	`enabled? = Application.get_env(:openagents, :forge_boot_converge_enabled, false)`
	99	+
	100	+	`if enabled? do`
	101	+	`outcome = safe_attempt(repo, 1)`
	102	+
	103	+	`if outcome["ready"] do`
	104	+	`publish(outcome)`
	105	+	`Process.send_after(self(), :retry_convergence, retry_max_ms())`
	106	+	`{:ok, %{repo: repo, retry_ms: retry_min_ms()}}`
	107	+	`else`
	108	+	`retry_ms = retry_min_ms()`
	109	+	`publish(Map.put(outcome, "retry_in_ms", retry_ms))`
	110	+	`Process.send_after(self(), :retry_convergence, retry_ms)`
	111	+	`{:ok, %{repo: repo, retry_ms: min(retry_ms * 2, retry_max_ms())}}`
61	112		`end`
	113	+	`else`
	114	+	`publish(initial_state(false))`
	115	+	`{:ok, %{repo: repo, retry_ms: retry_min_ms()}}`
	116	+	`end`
	117	+	`end`
62	118
63		-	`:persistent_term.put(@state_key, outcome)`
	119	+	`@impl true`
	120	+	`def handle_info(:retry_convergence, server_state), do: run_convergence(server_state)`
	121	+	`def handle_info(_message, server_state), do: {:noreply, server_state}`
64	122
65		-	`case outcome do`
66		-	`%{"state" => "converged", "sha" => sha} ->`
67		-	`Logger.info("forge boot convergence: loaded target #{sha}")`
	123	+	`defp run_convergence(server_state) do`
	124	+	`outcome = safe_attempt(server_state.repo, state()["attempts"] + 1)`
68	125
69		-	`%{"reason" => reason} ->`
70		-	`Logger.info("forge boot convergence: booting on image code (#{reason})")`
	126	+	`if outcome["ready"] do`
	127	+	`publish(outcome)`
	128	+	`Process.send_after(self(), :retry_convergence, retry_max_ms())`
	129	+	`{:noreply, %{server_state \| retry_ms: retry_min_ms()}}`
	130	+	`else`
	131	+	`retry_ms = min(server_state.retry_ms, retry_max_ms())`
	132	+	`outcome = Map.put(outcome, "retry_in_ms", retry_ms)`
	133	+	`publish(outcome)`
	134	+	`Process.send_after(self(), :retry_convergence, retry_ms)`
	135	+	`{:noreply, %{server_state \| retry_ms: min(retry_ms * 2, retry_max_ms())}}`
71	136		`end`
	137	+	`end`
72	138
73		-	`outcome`
	139	+	`defp safe_attempt(repo, attempts) do`
	140	+	`attempt(repo, attempts)`
	141	+	`rescue`
	142	+	`error -> degraded("convergence_exception", attempts, error)`
	143	+	`catch`
	144	+	`_kind, reason -> degraded("convergence_exit", attempts, reason)`
	145	+	`end`
	146	+
	147	+	`defp durable_target_ready?(repo, convergence) do`
	148	+	`case Targets.current(repo) do`
	149	+	`%{status: "deploying"} -> false`
	150	+	`_not_deploying -> current_target_matches?(repo, convergence)`
	151	+	`end`
74	152		`end`
75	153
76		-	`defp attempt(repo) do`
	154	+	`defp deployment_target_ready?(repo, target_id, convergence) do`
77	155		`case Targets.current(repo) do`
	156	+	`%{id: ^target_id, status: "deploying"} -> current_target_matches?(repo, convergence)`
	157	+	`_other_target -> false`
	158	+	`end`
	159	+	`end`
	160	+
	161	+	`defp current_target_matches?(repo, convergence) do`
	162	+	`case {Targets.live(repo), convergence} do`
	163	+	`{nil, %{"state" => "image", "reason" => "no_live_target"}} ->`
	164	+	`true`
	165	+
	166	+	`{%{sha: sha, details: details}, %{"sha" => sha, "artifact_digest" => nil}} ->`
	167	+	`(details \|\| %{})["artifact_digest"] == nil and OpenAgents.BuildInfo.revision() == sha`
	168	+
	169	+	`{%{sha: sha, details: details},`
	170	+	`%{`
	171	+	`"sha" => sha,`
	172	+	`"artifact_digest" => artifact_digest,`
	173	+	`"manifest_digest" => manifest_digest`
	174	+	`}} ->`
	175	+	`details = details \|\| %{}`
	176	+
	177	+	`details["artifact_digest"] == artifact_digest and`
	178	+	`details["manifest_digest"] == manifest_digest`
	179	+
	180	+	`_divergent ->`
	181	+	`false`
	182	+	`end`
	183	+	`end`
	184	+
	185	+	`defp attempt(repo, attempts) do`
	186	+	`case Targets.live(repo) do`
	187	+	`nil ->`
	188	+	`image_ready("no_live_target", attempts)`
	189	+
	190	+	`%{sha: sha, details: details} = target ->`
	191	+	`converge_target(repo, target, sha, details \|\| %{}, attempts)`
	192	+	`end`
	193	+	`end`
	194	+
	195	+	`defp converge_target(repo, target, sha, details, attempts) do`
	196	+	`with {:ok, identity} <- target_identity(target, details),`
	197	+	`{:ok, bytes, cache_state} <- artifact_bytes(repo, identity),`
	198	+	`{:ok, response} <- DeploymentNode.install_artifact(install_request(identity, bytes)),`
	199	+	`:ok <- retain_artifacts(repo, target.id, identity.artifact_digest) do`
78	200		`%{`
79		-	`status: "live",`
80		-	`sha: sha,`
81		-	`details: %{`
82		-	`"artifact" => relative,`
83		-	`"artifact_digest" => digest,`
84		-	`"build_id" => build_id`
85		-	`}`
	201	+	`"schema" => "openagents.forge.boot-convergence.v2",`
	202	+	`"state" => "converged",`
	203	+	`"ready" => true,`
	204	+	`"reason" => cache_state,`
	205	+	`"sha" => sha,`
	206	+	`"artifact_digest" => identity.artifact_digest,`
	207	+	`"manifest_digest" => identity.manifest_digest,`
	208	+	`"modules" => response["modules"] \|\| identity.modules,`
	209	+	`"attempts" => attempts,`
	210	+	`"retry_in_ms" => nil`
86	211		`}`
87		-	`when is_binary(relative) and is_binary(digest) and is_binary(build_id) ->`
88		-	`load_artifact(repo, sha, digest, build_id, Path.join(Repos.data_dir(), relative))`
	212	+	`else`
	213	+	`{:image_matches, ^sha} ->`
	214	+	`image_ready("image_matches_live", attempts, sha)`
	215	+
	216	+	`{:error, reason} ->`
	217	+	`degraded(OpenAgents.OperationalLog.code(reason), attempts, reason, sha)`
	218	+	`end`
	219	+	`end`
89	220
90		-	`%{status: "live", details: %{"artifact" => _relative}} ->`
91		-	`%{"state" => "image", "reason" => "artifact_identity_missing"}`
	221	+	`defp target_identity(target, details) do`
	222	+	`case details do`
	223	+	`%{`
	224	+	`"artifact_digest" => artifact_digest,`
	225	+	`"build_id" => build_id,`
	226	+	`"manifest" => manifest`
	227	+	`}`
	228	+	`when is_binary(artifact_digest) and is_binary(build_id) and is_map(manifest) ->`
	229	+	`manifest_digest =`
	230	+	`details["manifest_digest"] \|\|`
	231	+	`BuildArtifact.digest(BuildProtocol.canonical_json(manifest))`
92	232
93		-	`%{status: "live", sha: sha} ->`
94		-	`# A live target with no artifact recorded (a no-op deploy): the`
95		-	`# image already is that code as far as this lane knows.`
96		-	`%{"state" => "converged", "sha" => sha, "modules" => 0}`
	233	+	`changes = manifest["changes"] \|\| %{}`
	234	+	`modules = (changes["added"] \|\| []) ++ (changes["changed"] \|\| [])`
97	235
98		-	`%{status: status} ->`
99		-	`%{"state" => "image", "reason" => "target_not_live:#{status}"}`
	236	+	`{:ok,`
	237	+	`%{`
	238	+	`repo: target.repo,`
	239	+	`sha: target.sha,`
	240	+	`target_id: target.id,`
	241	+	`build_id: build_id,`
	242	+	`artifact_digest: artifact_digest,`
	243	+	`manifest_digest: manifest_digest,`
	244	+	`modules: length(modules)`
	245	+	`}}`
100	246
101		-	`nil ->`
102		-	`%{"state" => "image", "reason" => "no_target"}`
	247	+	`_missing ->`
	248	+	`if OpenAgents.BuildInfo.revision() == target.sha,`
	249	+	`do: {:image_matches, target.sha},`
	250	+	`else: {:error, :live_artifact_identity_missing}`
103	251		`end`
104	252		`end`
105	253
106		-	`defp load_artifact(repo, sha, digest, build_id, artifact) do`
107		-	`cond do`
108		-	`not File.exists?(artifact) ->`
109		-	`# The local cache misses on a replaced node — fetch the blob the`
110		-	`# builder uploaded next to the WAL, then converge from it. Only if`
111		-	`# the store misses too does the node boot on image code.`
112		-	`case OpenAgents.Forge.WAL.get_artifact(repo, digest) do`
113		-	`{:ok, payload} ->`
114		-	`with {:ok, verified} <-`
115		-	`BuildArtifact.verify(payload,`
116		-	`digest: digest,`
117		-	`repo: repo,`
118		-	`source_sha: sha,`
119		-	`build_id: build_id`
120		-	`) do`
121		-	`# Cache only bytes that passed the immutable identity check.`
122		-	`_cache_result = BuildProtocol.atomic_write(artifact, payload)`
123		-	`load_beams_from(sha, verified)`
124		-	`else`
125		-	`{:error, _reason} ->`
126		-	`%{"state" => "image", "reason" => "artifact_verification_failed"}`
127		-	`end`
128		-
129		-	`{:error, _reason} ->`
130		-	`%{"state" => "image", "reason" => "artifact_missing"}`
	254	+	`defp artifact_bytes(repo, identity) do`
	255	+	`path = cache_path(identity.artifact_digest)`
	256	+
	257	+	`case File.read(path) do`
	258	+	`{:ok, bytes} ->`
	259	+	`with {:ok, _verified} <- verify_artifact(bytes, identity) do`
	260	+	`{:ok, bytes, "local_cache"}`
131	261		`end`
132	262
133		-	`true ->`
134		-	`case BuildArtifact.verify_file(artifact,`
135		-	`digest: digest,`
136		-	`repo: repo,`
137		-	`source_sha: sha,`
138		-	`build_id: build_id`
139		-	`) do`
140		-	`{:ok, verified} ->`
141		-	`load_beams_from(sha, verified)`
142		-
143		-	`{:error, _reason} ->`
144		-	`%{"state" => "image", "reason" => "artifact_verification_failed"}`
	263	+	`{:error, :enoent} ->`
	264	+	`with {:ok, bytes} <- OpenAgents.Forge.WAL.get_artifact(repo, identity.artifact_digest),`
	265	+	`{:ok, _verified} <- verify_artifact(bytes, identity),`
	266	+	`:ok <- cache_verified(path, bytes, identity.artifact_digest) do`
	267	+	`{:ok, bytes, "durable_fetch"}`
145	268		`end`
	269	+
	270	+	`{:error, reason} ->`
	271	+	`{:error, {:artifact_cache_read_failed, reason}}`
146	272		`end`
147	273		`end`
148	274
149		-	`defp load_beams_from(sha, verified) do`
150		-	`allowlist = Application.get_env(:openagents, :forge_hot_load_allowlist, default_allowlist())`
	275	+	`defp verify_artifact(bytes, identity) do`
	276	+	`BuildArtifact.verify(bytes,`
	277	+	`digest: identity.artifact_digest,`
	278	+	`repo: identity.repo,`
	279	+	`source_sha: identity.sha,`
	280	+	`build_id: identity.build_id`
	281	+	`)`
	282	+	`end`
151	283
152		-	`offenders =`
153		-	`verified.modules`
154		-	`\|> Enum.reject(&HotLoader.allowlisted?(&1, allowlist))`
	284	+	`defp install_request(identity, bytes) do`
	285	+	`%{`
	286	+	`artifact_bytes: bytes,`
	287	+	`artifact_digest: identity.artifact_digest,`
	288	+	`build_id: identity.build_id,`
	289	+	`deployment_id: Ecto.UUID.generate(),`
	290	+	`expected_nodes: [to_string(Node.self())],`
	291	+	`manifest_digest: identity.manifest_digest,`
	292	+	`repo: identity.repo,`
	293	+	`sha: identity.sha,`
	294	+	`target_id: identity.target_id`
	295	+	`}`
	296	+	`end`
155	297
156		-	`cond do`
157		-	`verified.manifest["classification"] != "direct_candidate" ->`
158		-	`%{"state" => "image", "reason" => "artifact_not_direct"}`
159		-
160		-	`offenders != [] ->`
161		-	`%{"state" => "image", "reason" => "off_allowlist:#{Enum.join(offenders, ",")}"}`
162		-
163		-	`true ->`
164		-	`# Atom creation follows complete verification and policy checks.`
165		-	`beams =`
166		-	`Enum.map(verified.beams, fn %{module: module, binary: binary} ->`
167		-	`{BuildArtifact.module_atom(module), binary}`
168		-	`end)`
169		-
170		-	`failures =`
171		-	`beams`
172		-	`\|> HotLoader.load_beams()`
173		-	`\|> Enum.reject(fn {_mod, result} -> result == :ok end)`
174		-
175		-	`if failures == [] do`
176		-	`%{"state" => "converged", "sha" => sha, "modules" => length(beams)}`
177		-	`else`
178		-	`%{"state" => "image", "reason" => "load_failed"}`
179		-	`end`
	298	+	`defp retain_artifacts(repo, current_target_id, current_digest) do`
	299	+	`with {:ok, predecessor_digest} <- retain_predecessor(repo, current_target_id) do`
	300	+	`[current_digest, predecessor_digest]`
	301	+	`\|> Enum.reject(&is_nil/1)`
	302	+	`\|> prune_artifact_cache()`
	303	+	`end`
	304	+	`end`
	305	+
	306	+	`defp retain_predecessor(repo, current_target_id) do`
	307	+	`predecessor =`
	308	+	`repo`
	309	+	`\|> Targets.live_history(3)`
	310	+	`\|> Enum.reject(&(&1.id == current_target_id))`
	311	+	`\|> List.first()`
	312	+
	313	+	`case predecessor do`
	314	+	`nil -> {:ok, nil}`
	315	+	`target -> ensure_target_cached(repo, target)`
	316	+	`end`
	317	+	`end`
	318	+
	319	+	`defp ensure_target_cached(repo, target) do`
	320	+	`details = target.details \|\| %{}`
	321	+
	322	+	`with {:ok, identity} <- target_identity(target, details),`
	323	+	`{:ok, _bytes, _source} <- artifact_bytes(repo, identity) do`
	324	+	`{:ok, identity.artifact_digest}`
	325	+	`else`
	326	+	`{:image_matches, _sha} -> {:ok, nil}`
	327	+	`{:error, reason} -> {:error, {:rollback_artifact_unavailable, reason}}`
	328	+	`end`
	329	+	`end`
	330	+
	331	+	`defp prune_artifact_cache(retained_digests) do`
	332	+	`directory = Path.join(Repos.data_dir(), "beams")`
	333	+	`retained = MapSet.new(retained_digests)`
	334	+
	335	+	`case File.ls(directory) do`
	336	+	`{:ok, entries} ->`
	337	+	`Enum.reduce_while(entries, :ok, fn entry, :ok ->`
	338	+	`case Regex.run(~r/^([0-9a-f]{64})\.tar$/, entry) do`
	339	+	`[_, digest] ->`
	340	+	`if MapSet.member?(retained, digest) do`
	341	+	`{:cont, :ok}`
	342	+	`else`
	343	+	`case File.rm(Path.join(directory, entry)) do`
	344	+	`:ok -> {:cont, :ok}`
	345	+	`{:error, :enoent} -> {:cont, :ok}`
	346	+	`{:error, reason} -> {:halt, {:error, {:artifact_prune_failed, reason}}}`
	347	+	`end`
	348	+	`end`
	349	+
	350	+	`nil ->`
	351	+	`{:cont, :ok}`
	352	+	`end`
	353	+	`end)`
	354	+
	355	+	`{:error, :enoent} ->`
	356	+	`:ok`
	357	+
	358	+	`{:error, reason} ->`
	359	+	`{:error, {:artifact_cache_list_failed, reason}}`
	360	+	`end`
	361	+	`end`
	362	+
	363	+	`defp cache_verified(path, bytes, digest) do`
	364	+	`with :ok <- File.mkdir_p(Path.dirname(path)) do`
	365	+	`case BuildProtocol.atomic_write(path, bytes) do`
	366	+	`:ok ->`
	367	+	`:ok`
	368	+
	369	+	`{:error, :destination_exists} ->`
	370	+	`with {:ok, existing} <- File.read(path),`
	371	+	`true <- BuildArtifact.digest(existing) == digest or {:error, :digest_collision} do`
	372	+	`:ok`
	373	+	`end`
	374	+
	375	+	`{:error, reason} ->`
	376	+	`{:error, {:artifact_cache_write_failed, reason}}`
	377	+	`end`
	378	+	`else`
	379	+	`{:error, reason} -> {:error, {:artifact_cache_directory_failed, reason}}`
180	380		`end`
181	381		`end`
182	382
183		-	`defp default_allowlist, do: ["OpenAgents.Scratch.", "OpenAgents.BuildInfo"]`
	383	+	`defp cache_path(digest), do: Path.join([Repos.data_dir(), "beams", digest <> ".tar"])`
	384	+
	385	+	`defp image_ready(reason, attempts, sha \\ nil) do`
	386	+	`%{`
	387	+	`"schema" => "openagents.forge.boot-convergence.v2",`
	388	+	`"state" => "image",`
	389	+	`"ready" => true,`
	390	+	`"reason" => reason,`
	391	+	`"sha" => sha \|\| OpenAgents.BuildInfo.revision(),`
	392	+	`"artifact_digest" => nil,`
	393	+	`"manifest_digest" => nil,`
	394	+	`"modules" => 0,`
	395	+	`"attempts" => attempts,`
	396	+	`"retry_in_ms" => nil`
	397	+	`}`
	398	+	`end`
	399	+
	400	+	`defp degraded(code, attempts, _reason, sha \\ nil) do`
	401	+	`Logger.warning("forge_boot_convergence_degraded code=#{code}")`
	402	+
	403	+	`%{`
	404	+	`"schema" => "openagents.forge.boot-convergence.v2",`
	405	+	`"state" => "degraded",`
	406	+	`"ready" => false,`
	407	+	`"reason" => String.slice(code, 0, 128),`
	408	+	`"sha" => sha,`
	409	+	`"artifact_digest" => nil,`
	410	+	`"manifest_digest" => nil,`
	411	+	`"modules" => 0,`
	412	+	`"attempts" => attempts,`
	413	+	`"retry_in_ms" => nil`
	414	+	`}`
	415	+	`end`
	416	+
	417	+	`defp initial_state(false) do`
	418	+	`%{`
	419	+	`"schema" => "openagents.forge.boot-convergence.v2",`
	420	+	`"state" => "disabled",`
	421	+	`"ready" => true,`
	422	+	`"reason" => "feature_disabled",`
	423	+	`"sha" => OpenAgents.BuildInfo.revision(),`
	424	+	`"artifact_digest" => nil,`
	425	+	`"manifest_digest" => nil,`
	426	+	`"modules" => 0,`
	427	+	`"attempts" => 0,`
	428	+	`"retry_in_ms" => nil`
	429	+	`}`
	430	+	`end`
	431	+
	432	+	`defp publish(outcome) do`
	433	+	`:persistent_term.put(@state_key, outcome)`
	434	+	`outcome`
	435	+	`end`
	436	+
	437	+	`defp retry_min_ms do`
	438	+	`Application.get_env(:openagents, :forge_boot_retry_min_ms, @default_retry_min_ms)`
	439	+	`end`
	440	+
	441	+	`defp retry_max_ms do`
	442	+	`Application.get_env(:openagents, :forge_boot_retry_max_ms, @default_retry_max_ms)`
	443	+	`end`
184	444		`end`

28	22		`@builds_topic "forge:builds"`
29	23		`@deploys_topic "forge:deploys"`
30	24		`@default_allowlist ["OpenAgents.Scratch.", "OpenAgents.BuildInfo"]`
31		-	`@fleet_timeout_ms 15_000`
32	25
33	26		`# ── api ──────────────────────────────────────────────────────────────────`
34	27

289	289		`)`
290	290
291	291		`try do`
292		-	`with :ok <- :erl_tar.create(String.to_charlist(path), entries),`
	292	+	`with :ok <- write_reproducible_tar(path, entries),`
293	293		`{:ok, bytes} <- File.read(path),`
294	294		`:ok <- validate_artifact_size(bytes) do`
295	295		`{:ok, bytes}`

19	18		`field :repo, :string`
20	19		`field :sha, :string`
21	20		`field :target_id, :binary_id`
	21	+	`field :deployment_id, :binary_id`
	22	+	`field :artifact_digest, :string`
	23	+	`field :manifest_digest, :string`
22	24		`field :modules, {:array, :string}, default: []`
23	25		`field :nodes, {:array, :string}, default: []`
	26	+	`field :expected_nodes, {:array, :string}, default: []`
	27	+	`field :node_results, :map, default: %{}`
24	28		`field :result, :string`
25	29		`field :canary, :string`
	30	+	`field :error_code, :string`
	31	+	`field :rollback_verified, :boolean`
26	32		`field :push_to_live_ms, :integer`
	33	+	`field :started_at, :utc_datetime_usec`
	34	+	`field :completed_at, :utc_datetime_usec`
27	35		`timestamps(updated_at: false)`
28	36		`end`
29	37

31	39		`def results, do: @results`
32	40
33	41		`def changeset(receipt, attrs) do`
	42	+	`now = DateTime.utc_now()`
	43	+
	44	+	`attrs =`
	45	+	`attrs`
	46	+	`\|> default(:deployment_id, Ecto.UUID.generate())`
	47	+	`\|> default(:started_at, now)`
	48	+	`\|> default(:completed_at, now)`
	49	+
34	50		`receipt`
35	51		`\|> cast(attrs, [`
36	52		`:repo,`
37	53		`:sha,`
38	54		`:target_id,`
	55	+	`:deployment_id,`
	56	+	`:artifact_digest,`
	57	+	`:manifest_digest,`
39	58		`:modules,`
40	59		`:nodes,`
	60	+	`:expected_nodes,`
	61	+	`:node_results,`
41	62		`:result,`
42	63		`:canary,`
43		-	`:push_to_live_ms`
	64	+	`:error_code,`
	65	+	`:rollback_verified,`
	66	+	`:push_to_live_ms,`
	67	+	`:started_at,`
	68	+	`:completed_at`
	69	+	`])`
	70	+	`\|> validate_required([`
	71	+	`:repo,`
	72	+	`:sha,`
	73	+	`:target_id,`
	74	+	`:deployment_id,`
	75	+	`:result,`
	76	+	`:started_at,`
	77	+	`:completed_at`
44	78		`])`
45		-	`\|> validate_required([:repo, :sha, :target_id, :result])`
46	79		`\|> validate_inclusion(:result, @results)`
	80	+	`\|> validate_format(:sha, ~r/^[0-9a-f]{40}$/)`
	81	+	`\|> validate_format(:artifact_digest, ~r/^[0-9a-f]{64}$/)`
	82	+	`\|> validate_format(:manifest_digest, ~r/^[0-9a-f]{64}$/)`
	83	+	`\|> validate_length(:modules, max: 512)`
	84	+	`\|> validate_length(:nodes, max: 100)`
	85	+	`\|> validate_length(:expected_nodes, max: 100)`
	86	+	`\|> validate_length(:canary, max: 255)`
	87	+	`\|> validate_length(:error_code, max: 128)`
	88	+	`\|> validate_node_results()`
	89	+	`\|> unique_constraint(:deployment_id)`
47	90		`\|> check_constraint(:result, name: :forge_deploys_result)`
	91	+	`\|> check_constraint(:artifact_digest, name: :forge_deploys_artifact_digest)`
	92	+	`\|> check_constraint(:manifest_digest, name: :forge_deploys_manifest_digest)`
	93	+	`\|> check_constraint(:node_results, name: :forge_deploys_node_bounds)`
	94	+	`end`
	95	+
	96	+	`defp default(attrs, key, value) do`
	97	+	`cond do`
	98	+	`Map.has_key?(attrs, key) -> attrs`
	99	+	`Map.has_key?(attrs, to_string(key)) -> attrs`
	100	+	`true -> Map.put(attrs, key, value)`
	101	+	`end`
	102	+	`end`
	103	+
	104	+	`defp validate_node_results(changeset) do`
	105	+	`validate_change(changeset, :node_results, fn :node_results, results ->`
	106	+	`valid? =`
	107	+	`is_map(results) and map_size(results) <= 100 and`
	108	+	`Enum.all?(results, fn {node, result} ->`
	109	+	`is_binary(node) and byte_size(node) in 1..255 and is_binary(result) and`
	110	+	`byte_size(result) in 1..255`
	111	+	`end)`
	112	+
	113	+	`if valid?, do: [], else: [node_results: "must contain at most 100 bounded outcomes"]`
	114	+	`end)`
48	115		`end`
49	116		`end`

1	1		`defmodule OpenAgents.Forge.HotLoader do`
2	2		`@moduledoc """`
3		-	`Hot-load lane (roadmap P4): applies built beam artifacts to the running`
4		-	`fleet without a restart.`
5		-
6		-	Listens on `forge:builds` for build-ready broadcasts and, serially per
7		-	`build: refuses any artifact touching a module off the operator-owned`
8		-	hot-load allowlist (honest `needs_rolling_replace`, never a partial load),
9		-	`canaries the load on the local node with revert-on-failure, then`
10		-	`:erpc.multicall`s the rest of the fleet. Every outcome — `live`,
11		-	`reverted`, `needs_rolling_replace`, `failed` — lands as a
12		-	`forge_deploys` receipt including the measured push→live duration, and
13		-	`the target row advances honestly at each step.`
	3	+	`Coordinates immutable build artifacts through the transactional direct-load`
	4	+	`lane. The coordinator refuses structural or off-allowlist changes before`
	5	+	`deployment, then delegates prepare, canary, fleet apply, verification,`
	6	+	commit, and exact rollback to `OpenAgents.Forge.Deployment`.
14	7		`"""`
15	8
16	9		`use GenServer`

98	77
99	78		`defp handle_build(%{repo: repo, sha: sha, target_id: target_id, modules: modules} = build) do`
100	79		`if File.exists?(build.artifact) do`
101		-	`with {:ok, verified} <-`
102		-	`BuildArtifact.verify_file(build.artifact,`
	80	+	`with {:ok, artifact_bytes} <- File.read(build.artifact),`
	81	+	`{:ok, verified} <-`
	82	+	`BuildArtifact.verify(artifact_bytes,`
103	83		`digest: Map.get(build, :artifact_digest),`
104	84		`repo: repo,`
105	85		`source_sha: sha,`

109	89		`true <-`
110	90		`is_nil(Map.get(build, :manifest)) or Map.get(build, :manifest) == verified.manifest or`
111	91		`{:error, :declared_manifest_mismatch} do`
112		-	`route_verified(build, verified)`
	92	+	`route_verified(build, verified, artifact_bytes)`
113	93		`else`
114	94		`{:error, reason} -> fail_verified_build(build, reason)`
115	95		`end`

125	105		`:refused -> :ok`
126	106		`end`
127	107
128		-	`defp route_verified(build, verified) do`
	108	+	`defp route_verified(build, verified, artifact_bytes) do`
129	109		`allowlist = Application.get_env(:openagents, :forge_hot_load_allowlist, @default_allowlist)`
130	110		`offending = Enum.reject(verified.modules, &allowlisted?(&1, allowlist))`
131	111

137	117		`route_rolling(build, Enum.map(offending, &"off_allowlist:#{&1}"))`
138	118
139	119		`true ->`
140		-	`deploy(build, verified)`
	120	+	`build =`
	121	+	`build`
	122	+	`\|> Map.put(:artifact_digest, verified.digest)`
	123	+	`\|> Map.put(:build_id, verified.manifest["build_id"])`
	124	+	`\|> Map.put(:manifest, verified.manifest)`
	125	+
	126	+	`deploy(build, verified, artifact_bytes)`
141	127		`end`
142	128		`end`
143	129

166	152		`broadcast_deploy(repo, sha, "failed")`
167	153		`end`
168	154
169		-	`defp deploy(%{repo: repo, sha: sha, target_id: target_id, modules: modules}, verified) do`
170		-	`case advance(target_id, "deploying") do`
171		-	`:ok -> :ok`
172		-	`:error -> throw(:refused)`
	155	+	`defp deploy(%{target_id: target_id} = build, verified, bytes) do`
	156	+	`case Targets.begin_deployment(target_id) do`
	157	+	`{:ok, _target} ->`
	158	+	`run_transaction(build, verified, bytes)`
	159	+
	160	+	`{:error, reason} ->`
	161	+	`Logger.warning("forge_deployment_refused code=#{OpenAgents.OperationalLog.code(reason)}")`
	162	+	`end`
	163	+	`rescue`
	164	+	`error ->`
	165	+	`fail_started_deployment(build, %{error_code: OpenAgents.OperationalLog.code(error)})`
	166	+	`catch`
	167	+	`kind, reason ->`
	168	+	`fail_started_deployment(build, %{error_code: OpenAgents.OperationalLog.code({kind, reason})})`
	169	+	`end`
	170	+
	171	+	`defp run_transaction(build, verified, bytes) do`
	172	+	`case Deployment.run(build, verified, bytes) do`
	173	+	`{:ok, session} -> commit_live(build, session)`
	174	+	`{:error, outcome} -> finish_failed_transaction(build, outcome)`
173	175		`end`
	176	+	`end`
174	177
175		-	`# Atom creation happens only here, after the full tar and manifest have`
176		-	`# passed every bounded identity and classification check.`
177		-	`beams =`
178		-	`Enum.map(verified.beams, fn %{module: module, binary: binary} ->`
179		-	`{BuildArtifact.module_atom(module), binary}`
180		-	`end)`
	178	+	`defp commit_live(build, session) do`
	179	+	`receipt = receipt_attributes(build, session, "ok", push_to_live_ms(build.repo, build.sha))`
181	180
182		-	`case canary_load(beams) do`
183		-	`:ok ->`
184		-	`nodes = fleet_load(beams)`
185		-	`advance(target_id, "live", %{"modules" => modules})`
186		-	`push_ms = push_to_live_ms(repo, sha)`
187		-	`insert_receipt(repo, sha, target_id, modules, nodes, "live", "ok", push_ms)`
188		-	`broadcast_deploy(repo, sha, "live")`
	181	+	`case Targets.finish_deployment(`
	182	+	`build.target_id,`
	183	+	`"live",`
	184	+	`live_details(build, session),`
	185	+	`receipt`
	186	+	`) do`
	187	+	`{:ok, _committed} ->`
	188	+	`case Deployment.finalize(session) do`
	189	+	`:ok ->`
	190	+	`:ok`
	191	+
	192	+	`{:error, reason} ->`
	193	+	`Logger.error(`
	194	+	`"forge_deployment_finalize_failed code=#{OpenAgents.OperationalLog.code(reason)}"`
	195	+	`)`
	196	+	`end`
	197	+
	198	+	`broadcast_deploy(build.repo, build.sha, "live")`
189	199
190	200		`{:error, reason} ->`
191		-	`advance(target_id, "reverted", %{"error" => bounded(reason)})`
192		-	`insert_receipt(repo, sha, target_id, modules, [], "reverted", bounded(reason), nil)`
193		-	`broadcast_deploy(repo, sha, "reverted")`
	201	+	`rollback_after_database_failure(build, session, reason)`
194	202		`end`
195	203		`end`
196	204
	205	+	`defp rollback_after_database_failure(build, session, database_reason) do`
	206	+	`case Deployment.rollback(session) do`
	207	+	`{:ok, node_results} ->`
	208	+	`outcome =`
	209	+	`session`
	210	+	`\|> Map.merge(%{`
	211	+	`result: "reverted",`
	212	+	`rollback_verified: true,`
	213	+	`error_code: OpenAgents.OperationalLog.code(database_reason),`
	214	+	`node_results: node_results,`
	215	+	`nodes: node_lines(node_results)`
	216	+	`})`
	217	+
	218	+	`finish_failed_transaction(build, outcome)`
	219	+
	220	+	`{:error, node_results} ->`
	221	+	`outcome =`
	222	+	`session`
	223	+	`\|> Map.merge(%{`
	224	+	`result: "failed",`
	225	+	`rollback_verified: false,`
	226	+	`error_code: "database_commit_and_rollback_failed",`
	227	+	`node_results: node_results,`
	228	+	`nodes: node_lines(node_results)`
	229	+	`})`
	230	+
	231	+	`finish_failed_transaction(build, outcome)`
	232	+	`end`
	233	+	`end`
	234	+
	235	+	`defp finish_failed_transaction(build, outcome) do`
	236	+	`status = if outcome.result == "reverted", do: "reverted", else: "failed"`
	237	+	`details = %{"error_code" => outcome.error_code, "modules" => build.modules}`
	238	+	`receipt = receipt_attributes(build, outcome, bounded(outcome.error_code), nil)`
	239	+
	240	+	`case Targets.finish_deployment(build.target_id, status, details, receipt) do`
	241	+	`{:ok, _committed} ->`
	242	+	`broadcast_deploy(build.repo, build.sha, status)`
	243	+
	244	+	`{:error, reason} ->`
	245	+	`Logger.error(`
	246	+	`"forge_deployment_receipt_failed code=#{OpenAgents.OperationalLog.code(reason)}"`
	247	+	`)`
	248	+	`end`
	249	+	`end`
	250	+
	251	+	`defp fail_started_deployment(build, outcome) do`
	252	+	`defaults = %{`
	253	+	`deployment_id: Ecto.UUID.generate(),`
	254	+	`artifact_digest: Map.get(build, :artifact_digest),`
	255	+	`manifest_digest: manifest_digest(Map.get(build, :manifest)),`
	256	+	`expected_nodes: [],`
	257	+	`node_results: %{},`
	258	+	`nodes: [],`
	259	+	`canary: nil,`
	260	+	`rollback_verified: false,`
	261	+	`started_at: DateTime.utc_now(),`
	262	+	`result: "failed"`
	263	+	`}`
	264	+
	265	+	`finish_failed_transaction(build, Map.merge(defaults, outcome))`
	266	+	`end`
	267	+
197	268		`# Every terminal outcome is announced on the deploys topic — live,`
198	269		`# reverted, needs_rolling_replace, and failed alike.`
199	270		`defp broadcast_deploy(repo, sha, result) do`

217	288		`end`
218	289		`end`
219	290
220		-	`# ── canary (local node, revert on any failure) ───────────────────────────`
221		-
222		-	`defp canary_load(beams), do: canary_load(beams, [])`
223		-
224		-	`defp canary_load([], loaded) do`
225		-	`case smoke_check(loaded) do`
226		-	`:ok ->`
227		-	`:ok`
228		-
229		-	`{:error, reason} ->`
230		-	`revert(loaded)`
231		-	`{:error, reason}`
232		-	`end`
233		-	`end`
234		-
235		-	`defp canary_load([{mod, binary} \| rest], loaded) do`
236		-	`# Keep the currently loaded object code so a later failure can revert.`
237		-	`previous = :code.get_object_code(mod)`
238		-
239		-	`case :code.load_binary(mod, ~c"forge-hot-load", binary) do`
240		-	`{:module, ^mod} ->`
241		-	`canary_load(rest, [{mod, previous} \| loaded])`
242		-
243		-	`{:error, reason} ->`
244		-	`revert(loaded)`
245		-	`{:error, "canary load of #{inspect(mod)} failed: #{inspect(reason)}"}`
246		-	`end`
247		-	`end`
248		-
249		-	`defp smoke_check(loaded) do`
250		-	`Enum.reduce_while(loaded, :ok, fn {mod, _previous}, :ok ->`
251		-	`cond do`
252		-	`not Code.ensure_loaded?(mod) ->`
253		-	`{:halt, {:error, "smoke check: #{inspect(mod)} not loaded"}}`
254		-
255		-	`function_exported?(mod, :revision, 0) and not revision_binary?(mod) ->`
256		-	`{:halt, {:error, "smoke check: #{inspect(mod)}.revision/0 did not return a binary"}}`
257		-
258		-	`true ->`
259		-	`{:cont, :ok}`
260		-	`end`
261		-	`end)`
262		-	`end`
263		-
264		-	`defp revision_binary?(mod) do`
265		-	`is_binary(mod.revision())`
266		-	`rescue`
267		-	`_ -> false`
268		-	`end`
269		-
270		-	`defp revert(loaded) do`
271		-	`Enum.each(loaded, fn`
272		-	`{mod, {_mod, binary, file}} ->`
273		-	`:code.load_binary(mod, file, binary)`
274		-
275		-	`{mod, :error} ->`
276		-	`# Was not loaded before this deploy: remove it entirely.`
277		-	`:code.purge(mod)`
278		-	`:code.delete(mod)`
279		-	`end)`
280		-	`end`
281		-
282		-	`# ── fleet ────────────────────────────────────────────────────────────────`
283		-
284		-	`defp fleet_load(beams) do`
285		-	`remote =`
286		-	`case Node.list() do`
287		-	`[] ->`
288		-	`[]`
289		-
290		-	`nodes ->`
291		-	`nodes`
292		-	`\|> Enum.zip(:erpc.multicall(nodes, __MODULE__, :load_beams, [beams], @fleet_timeout_ms))`
293		-	`\|> Enum.map(fn`
294		-	`{node, {:ok, results}} ->`
295		-	`if Enum.all?(results, &match?({_mod, :ok}, &1)),`
296		-	`do: "#{node}=ok",`
297		-	`else: "#{node}=error"`
298		-
299		-	`{node, _failure} ->`
300		-	`"#{node}=error"`
301		-	`end)`
302		-	`end`
303		-
304		-	`["#{Node.self()}=ok" \| remote]`
305		-	`end`
306		-
307	291		`# ── receipts ─────────────────────────────────────────────────────────────`
308	292
309	293		`# Push→live duration: the push receipt whose refs advanced some ref to`

14	14		`import Ecto.Query`
15	15
16	16		`alias OpenAgents.Forge.Target`
	17	+	`alias OpenAgents.Forge.DeployReceipt`
17	18		`alias OpenAgents.Repo`
18	19
19	20		`@statuses ~w(promoted building built deploying live failed reverted needs_rolling_replace)`

62	62		`# not report itself healthy). Where Ra is off (single-node Cloud Run) the`
63	63		`# expected size is honestly 1.`
64	64		`expected =`
65		-	`if Application.get_env(:openagents, :ra_enabled, false),`
66		-	`do: Application.get_env(:openagents, :ra_expected_size, 3),`
67		-	`else: 1`
	65	+	`cond do`
	66	+	`Application.get_env(:openagents, :ra_enabled, false) ->`
	67	+	`Application.get_env(:openagents, :ra_expected_size, 3)`
	68	+
	69	+	`Application.get_env(:openagents, :forge_deploy_lane_enabled, false) ->`
	70	+	`Application.get_env(:openagents, :forge_expected_fleet_size, 1)`
	71	+
	72	+	`true ->`
	73	+	`1`
	74	+	`end`
68	75
69	76		`quorum = safely(fn -> OpenAgents.Cluster.quorum?(max(expected, 1)) end) \|\| false`
70	77

89	96		`Map.put(report, "label", "node #{index}")`
90	97		`end)`
91	98
	99	+	`nodes_ready? = Enum.all?(nodes, &(&1["reachable"] == true and &1["ready"] == true))`
	100	+
92	101		`%{`
93	102		`"schema" => @schema,`
94	103		`# Legacy /status compatibility keys — pollers migrating from the old`
95	104		`# payload find them here unchanged.`
96		-	`"status" => if(quorum or length(beam_nodes) == 1, do: "ok", else: "degraded"),`
97		-	`"revision" => safely(fn -> OpenAgents.BuildInfo.revision() end),`
	105	+	`"status" => if(quorum and nodes_ready?, do: "ok", else: "degraded"),`
	106	+	`"revision" => safely(fn -> OpenAgents.Forge.DeploymentNode.health()["revision"] end),`
98	107		`"cluster" => %{`
99	108		`"distributed" => Map.get(cluster, "distributed", false),`
100	109		`"beam" => length(beam_nodes),`

201	210		`@doc false`
202	211		`def local_report do`
203	212		`{wall_ms, _} = :erlang.statistics(:wall_clock)`
	213	+	`boot = OpenAgents.Forge.BootConverge.state()`
	214	+	`boot_ready? = OpenAgents.Forge.BootConverge.ready?()`
	215	+	`deployment = OpenAgents.Forge.DeploymentNode.health()`
204	216
205	217		`%{`
206	218		`"reachable" => true,`
	219	+	`"ready" => boot_ready? and deployment["ready"] == true,`
207	220		`"release" => safely(fn -> permanent_release_version() end),`
208		-	`"revision" => safely(fn -> OpenAgents.BuildInfo.revision() end),`
	221	+	`"revision" => deployment["revision"] \|\| boot["sha"] \|\| OpenAgents.BuildInfo.revision(),`
209	222		`"hot_loaded_at" => safely(fn -> OpenAgents.BuildInfo.loaded_at() end),`
210	223		`"marker" => safely(fn -> OpenAgents.Cluster.relup_marker() end),`
211		-	`"boot" => safely(fn -> OpenAgents.Forge.BootConverge.state()["state"] end),`
	224	+	`"boot" => %{`
	225	+	`"state" => boot["state"],`
	226	+	`"ready" => boot_ready?,`
	227	+	`"reason" => boot["reason"],`
	228	+	`"attempts" => boot["attempts"],`
	229	+	`"retry_in_ms" => boot["retry_in_ms"]`
	230	+	`},`
	231	+	`"deployment" => %{`
	232	+	`"ready" => deployment["participant_ready"],`
	233	+	`"phase" => deployment["phase"],`
	234	+	`"reason" => deployment["reason"]`
	235	+	`},`
212	236		`"uptime_seconds" => div(wall_ms, 1_000),`
213	237		`"beam_seen" => 1 + length(Node.list()),`
214	238		`"raft_seen" => safely(fn -> length(OpenAgents.Cluster.Ra.members()) end) \|\| 0`

449	449		`expected_fleet_size = Map.get(settings, :forge_expected_fleet_size)`
450	450		`artifact_store = Map.get(settings, :forge_artifact_store)`
451	451		`build_executor = Map.get(settings, :forge_build_executor)`
	452	+	`deploy_timeout_ms = Map.get(settings, :forge_deploy_timeout_ms)`
	453	+	`deploy_token_ttl_ms = Map.get(settings, :forge_deploy_token_ttl_ms)`
	454	+	`boot_retry_min_ms = Map.get(settings, :forge_boot_retry_min_ms)`
	455	+	`boot_retry_max_ms = Map.get(settings, :forge_boot_retry_max_ms)`
452	456		`operator_token = Map.get(settings, :forge_operator_token)`
453	457		`mirror_urls = Map.get(settings, :forge_mirror_urls)`
454	458		`durable_required? = environment in [:staging, :production] and features.forge`

503	507		`:forge_build_executor,`
504	508		`"must be a module"`
505	509		`),`
	510	+	`:ok <-`
	511	+	`ensure(`
	512	+	`is_integer(deploy_timeout_ms) and deploy_timeout_ms in 1_000..120_000,`
	513	+	`:forge_deploy_timeout_ms,`
	514	+	`"must be between 1 and 120 seconds"`
	515	+	`),`
	516	+	`:ok <-`
	517	+	`ensure(`
	518	+	`is_integer(deploy_token_ttl_ms) and deploy_token_ttl_ms in 30_000..1_800_000 and`
	519	+	`deploy_token_ttl_ms >= deploy_timeout_ms * 8,`
	520	+	`:forge_deploy_token_ttl_ms,`
	521	+	`"must cover eight deployment phase timeouts"`
	522	+	`),`
	523	+	`:ok <-`
	524	+	`ensure(`
	525	+	`is_integer(boot_retry_min_ms) and is_integer(boot_retry_max_ms) and`
	526	+	`boot_retry_min_ms in 100..60_000 and boot_retry_max_ms in 1_000..300_000 and`
	527	+	`boot_retry_min_ms <= boot_retry_max_ms,`
	528	+	`:forge_boot_retry_bounds,`
	529	+	`"must define an ordered bounded backoff"`
	530	+	`),`
506	531		`:ok <-`
507	532		`ensure(`
508	533		`not features.forge_deploy or expected_fleet_size >= 2,`

35	35		`OpenAgents.Leaderboard.Server,`
36	36		`{Task.Supervisor, name: OpenAgents.ProviderTaskSupervisor},`
37	37		`{Task.Supervisor, name: OpenAgents.ToolTaskSupervisor},`
38		-	`{Task.Supervisor, name: OpenAgents.ShadowProgramTaskSupervisor},`
39		-	`OpenAgents.Forge.BootConverge`
	38	+	`{Task.Supervisor, name: OpenAgents.ShadowProgramTaskSupervisor}`
40	39		`] ++`
41	40		`maybe_forge() ++`
42	41		`maybe_semantic_worker() ++`

2	2		`use OpenAgentsWeb, :controller`
3	3
4	4		`def show(conn, _params) do`
5		-	`case OpenAgents.Repo.query("SELECT 1") do`
6		-	`{:ok, _result} ->`
7		-	`json(conn, %{status: "ok", revision: OpenAgents.BuildInfo.revision()})`
	5	+	`case {OpenAgents.Repo.query("SELECT 1"), OpenAgents.Cluster.local_report()} do`
	6	+	`{{:ok, _result}, %{"ready" => true, "revision" => revision}} ->`
	7	+	`json(conn, %{status: "ok", revision: revision})`
8	8
9		-	`{:error, _reason} ->`
10		-	`conn \|> put_status(:service_unavailable) \|> json(%{status: "unavailable"})`
	9	+	`{{:ok, _result}, report} ->`
	10	+	`conn`
	11	+	`\|> put_status(:service_unavailable)`
	12	+	`\|> json(%{`
	13	+	`status: "unavailable",`
	14	+	`reason: "runtime_not_ready",`
	15	+	`boot_converged: report["boot_converged"],`
	16	+	`deployment_ready: report["deployment_ready"]`
	17	+	`})`
	18	+
	19	+	`{{:error, _reason}, _report} ->`
	20	+	`conn`
	21	+	`\|> put_status(:service_unavailable)`
	22	+	`\|> json(%{status: "unavailable", reason: "database_unavailable"})`
11	23		`end`
12	24		`end`
13	25		`end`

48	48		`export OPENAGENTS_FORGE_BUILD_QUEUE_DIR="/var/lib/openagents/workspace/build-queue"`
49	49		`export OPENAGENTS_FORGE_BUILD_TIMEOUT_MS="300000"`
50	50		`export OPENAGENTS_FORGE_BUILD_OUTPUT_RETENTION_MS="604800000"`
	51	+	`export OPENAGENTS_FORGE_DEPLOY_TIMEOUT_MS="15000"`
	52	+	`export OPENAGENTS_FORGE_DEPLOY_TOKEN_TTL_MS="120000"`
	53	+	`export OPENAGENTS_FORGE_BOOT_RETRY_MIN_MS="1000"`
	54	+	`export OPENAGENTS_FORGE_BOOT_RETRY_MAX_MS="30000"`
51	55		`export OPENAGENTS_FORGE_DATA_DIR="/var/lib/openagents/forge"`
52	56		`export OPENAGENTS_FORGE_EXPECTED_FLEET_SIZE="1"`
53	57		`export OPENAGENTS_FORGE_INTERNAL_GIT_URL="http://127.0.0.1:8080/git"`

22	23		`{key, Application.get_env(:openagents, key)}`
23	24		`end`
24	25
	26	+	`previous_node_state = :sys.get_state(DeploymentNode)`
	27	+	`previous_persisted = :persistent_term.get({DeploymentNode, :state}, :missing)`
	28	+	`:persistent_term.erase({DeploymentNode, :state})`
	29	+
	30	+	`:sys.replace_state(DeploymentNode, fn state ->`
	31	+	`%{state \| transactions: %{}, live: nil, divergence: nil, faults: %{}, notify: nil}`
	32	+	`end)`
	33	+
25	34		`Application.put_env(:openagents, :forge_data_dir, Path.join(base, "data"))`
26	35		`Application.put_env(:openagents, :forge_wal_dir, Path.join(base, "wal"))`
27	36