Connect the general relup lane and fix the staging profile default

646bb64f06b3 · AtlantisPleb · · parent 352448a86c97

Connect the general relup lane and fix the staging profile default

RelupDeployment now admits any distinct X.Y.Z version pair whose state
versions stay within the supported set and never regress, matching
what RelupNode enforces per node; the packaged appup on target nodes
remains the real gate at check_install. The appup source generates
forward and reverse instructions for any admitted pair instead of
raising outside the 0.1.0-to-0.2.0 proof transition.

ops/forge/package-relup.sh assembles both release tarballs plus the
generated two-way relup for an arbitrary revision pair in isolated
worktrees and emits a digest-addressed package.json.
ops/relup-proof/install-proof.sh installs such a package against a
live single-node release and proves forward upgrade, reverse rollback,
and re-upgrade with ReleaseState retention; it passed for
0.2.0 to 0.3.0 on this machine.

gate-5-profile.sh corrupted a preset GITHUB_TOKEN_DECRYPTION_KEYS_JSON
through {} expansion, appending a stray brace; the default is
now written explicitly.

Changelog: The general relup lane is connected end to end for arbitrary admitted version pairs, pending production approval.

Changelog-Category: feature
Changelog
The general relup lane is connected end to end for arbitrary admitted version pairs, pending production approval.
Changelog-Category
feature

Deploy story

What this commit did to the running system — joined from the forge receipt chain, the part a commit page elsewhere cannot show.

Not deployed through the forge lane

No push, promotion, build, or deploy receipt references this commit (receipts are scanned over a bounded recent window). Changes shipped by full node replacement carry their proof in the release gate receipt instead.

Changed files

  • modified docs/2026-08-21-hot-deploy-gap-audit.md
  • modified docs/operations/forge-hot-loop.md
  • added ops/relup-proof/install-proof.sh
  • modified ops/staging/gate-5-profile.sh

Diff

4 files changed, +185 -19

docs/2026-08-21-hot-deploy-gap-audit.md modified +7 -7

@@ -32,13 +32,13 @@ After this audit was measured, two claims moved:

32 32
   boot convergence satisfied, process uptimes unbroken. The operator assembled
33 33
   and applied the artifact manually; no receipted automated deploy exists yet.
34 34
2. **"Nothing needs re-implementing" holds for the hot-load loop only.** The
35
   general relup path remains unconnected: `BuildArtifact` classifies only
36
   `direct_candidate` and `needs_rolling_replace` and can never emit a relup
37
   class, and `RelupDeployment` accepts only the fixed `0.1.0 → 0.2.0`
38
   proof transition while production runs `0.2.0`. Closing that lane is
39
   recorded as future work in
40
   [`docs/operations/forge-hot-loop.md`](operations/forge-hot-loop.md), which
41
   is also the enablement runbook for the automated loop.
35
   general relup path was, at measurement time, unconnected: `BuildArtifact`
36
   classifies only `direct_candidate` and `needs_rolling_replace`, and
37
   `RelupDeployment` accepted only the fixed `0.1.0 → 0.2.0` proof transition.
38
   Later the same day the relup lane was connected for general version pairs —
39
   coordinator admission, appup generation, packaging, and install proofs; see
40
   [`docs/operations/forge-hot-loop.md`](operations/forge-hot-loop.md).
41
   Production approval for the lane remains an open operator decision.
42 42
43 43
---
44 44
docs/operations/forge-hot-loop.md modified +37 -11

@@ -126,17 +126,43 @@ human reaction time.

126 126
A `reverted` outcome still warrants checking fleet convergence even though
127 127
this design captures each node's prior object code for exact rollback.
128 128
129
## What relups still need (out of scope here)
130
131
Getting hot loads running requires nothing below. Recording it ends the
132
documentation drift about the third lane:
133
134
1. The classifier needs a `needs_relup` class with version-boundary reasons;
135
   today it can never emit one.
136
2. `RelupDeployment` needs general from/to version admission driven by the
137
   packaged appup, replacing the fixed proof-transition pins.
138
3. The lane needs its own gate evidence and production approval per
139
   `docs/operations/release-deployment-fallbacks.md`.
129
## The relup lane is now connected
130
131
As of 2026-08-21 (later), the three gaps recorded below are closed in code:
132
133
1. **Coordinator admission is general.** `RelupDeployment` admits any distinct
134
   `X.Y.Z` pair whose state versions stay within `[1, 2]` and never regress —
135
   matching what `RelupNode` already enforced per node. The packaged appup on
136
   the target nodes remains the real gate: `check_install_release` refuses
137
   honestly when no relup can be produced between two versions.
138
2. **Appup generation works for any admitted pair.** `rel/openagents.appup.exs`
139
   generates forward and reverse instructions from `RELUP_FROM`/`RELUP_TO`
140
   for arbitrary distinct versions instead of raising outside the proof
141
   transition.
142
3. **Packaging and install proofs exist as tools.**
143
   `ops/forge/package-relup.sh --from-version A --to-version B [--from-rev]
144
   [--to-rev]` builds both releases in isolated worktrees, generates the
145
   two-way relup, embeds it, and emits digest-addressed tarballs plus a
146
   `package.json` ready for deployment requests.
147
   `ops/relup-proof/install-proof.sh` then proves the pair against a live
148
   single-node release: forward install, permanent commit, reverse rollback,
149
   and re-upgrade, asserting `ReleaseState` observations survive every
150
   transition. Both were executed successfully for `0.2.0 → 0.3.0`.
151
152
Still required before the lane carries production traffic — operator work,
153
not engineering work:
154
155
- Production approval recorded against
156
  [`docs/operations/release-deployment-fallbacks.md`](release-deployment-fallbacks.md),
157
  which remains the authority that relups are not production-approved.
158
- Wiring `package-relup.sh` into automation so a promoted SHA packages and
159
  deploys without manual steps. Until then the lane is driven by the same
160
  commands above, which are safe to rehearse on staging.
161
162
Note on scope: hot-load diffs and relups remain different artifact classes.
163
A BEAM-diff artifact cannot drive `release_handler`; only a full release
164
package can. That is why the build lane's classification stays two-class and
165
the relup lane consumes its own packages.
140 166
141 167
## References
142 168
ops/relup-proof/install-proof.sh added +135

@@ -0,0 +1,135 @@

1
#!/bin/sh
2
# Installs an already-packaged two-way relup against a live single-node
3
# release, proving forward upgrade, reverse rollback, and re-upgrade with
4
# ReleaseState retention. Consumes the output of ops/forge/package-relup.sh.
5
#
6
# Required environment: OPENAGENTS_RELUP_PROOF_DISPOSABLE=1 and
7
# OPENAGENTS_RELUP_PROOF_DATABASE_URL (a disposable PostgreSQL URL),
8
# matching the contract of ops/relup-proof/run.sh.
9
#
10
# Optional: RELUP_PROOF_PACKAGE_DIR, RELUP_PROOF_NODE, RELUP_PROOF_PORT.
11
# Install proof for a generalized relup pair: 0.2.0 -> 0.3.0 -> reverse ->
12
# re-upgrade, using the tarballs produced by ops/forge/package-relup.sh.
13
#
14
# Assertions check observable effects (directories, running version, retained
15
# state) rather than release_handler return values relayed through the shell
16
# rpc channel, whose IO can drop mid-call while release_handler runs.
17
set -eu
18
19
repo_root=$(CDPATH= cd -- "$(dirname -- "$0")/../.." && pwd)
20
pkg="${RELUP_PROOF_PACKAGE_DIR:-$repo_root/.git/openagents/relup-package}"
21
runtime_root=$(mktemp -d /tmp/openagents-relup-runtime.XXXXXX)
22
release_log="$runtime_root/release.log"
23
node_name="${RELUP_PROOF_NODE:-openagents-relup-proof@127.0.0.1}"
24
db="${RELUP_PROOF_DB:-openagents_relup_proof}"
25
26
cleanup() {
27
  if [ -n "${release_pid:-}" ]; then
28
    profile "$runtime_root/bin/openagents" stop >/dev/null 2>&1 || true
29
    kill -TERM "$release_pid" 2>/dev/null || true
30
  fi
31
  [ -d "$runtime_root" ] && find "$runtime_root" -depth -delete
32
  dropdb -h localhost --if-exists "$db" >/dev/null 2>&1 || true
33
}
34
trap cleanup EXIT INT TERM
35
36
if [ "${OPENAGENTS_RELUP_PROOF_DISPOSABLE:-}" != "1" ]; then
37
  echo "set OPENAGENTS_RELUP_PROOF_DISPOSABLE=1 for a disposable database" >&2
38
  exit 1
39
fi
40
41
proof_database_url=${OPENAGENTS_RELUP_PROOF_DATABASE_URL:-}
42
[ -n "$proof_database_url" ] || {
43
  echo "OPENAGENTS_RELUP_PROOF_DATABASE_URL is required" >&2
44
  exit 1
45
}
46
47
pkill -f "$node_name" 2>/dev/null || true
48
sleep 1
49
50
db_name=$(printf '%s' "$proof_database_url" | sed -E 's|.*/||')
51
dropdb -h localhost --if-exists "$db_name" >/dev/null 2>&1 || true
52
createdb -h localhost "$db_name"
53
54
tar -xzf "$pkg/openagents-0.2.0.tar.gz" -C "$runtime_root"
55
cp "$pkg/openagents-0.3.0.tar.gz" "$runtime_root/releases/openagents-0.3.0.tar.gz"
56
57
secret=$(openssl rand -base64 64 | tr -d '\n')
58
token_key=$(openssl rand -base64 32 | tr -d '\n')
59
60
profile() {
61
  env \
62
    DATABASE_URL="$proof_database_url" \
63
    GITHUB_CLIENT_ID="relup-proof-client" \
64
    GITHUB_CLIENT_SECRET="relup-proof-secret" \
65
    GITHUB_TOKEN_ENCRYPTION_KEY="$token_key" \
66
    GITHUB_TOKEN_ENCRYPTION_KEY_ID="staging-relup-proof-2026-08" \
67
    OPENAI_API_KEY="relup-proof-openai-key" \
68
    PHX_SERVER="true" \
69
    POOL_SIZE="2" \
70
    PORT="${RELUP_PROOF_PORT:-4399}" \
71
    RELEASE_DISTRIBUTION="name" \
72
    RELEASE_NODE="$node_name" \
73
    SECRET_KEY_BASE="$secret" \
74
    "$repo_root/ops/staging/gate-5-profile.sh" "$@"
75
}
76
77
fail() {
78
  echo "PROOF FAILED: $1"
79
  tail -40 "$runtime_root/log" 2>/dev/null || true
80
  exit 1
81
}
82
83
bin="$runtime_root/bin/openagents"
84
85
echo "starting 0.2.0"
86
profile "$bin" start >"$release_log" 2>&1 &
87
release_pid=$!
88
89
attempt=0
90
until profile "$bin" rpc 'if Process.whereis(OpenAgents.ReleaseState), do: IO.puts("ready-marker")' 2>/dev/null | grep -q ready-marker; do
91
  attempt=$((attempt + 1))
92
  [ $attempt -ge 120 ] && fail "release never became reachable"
93
  kill -0 "$release_pid" 2>/dev/null || fail "release exited during startup"
94
  sleep 0.5
95
done
96
97
echo "recording state on 0.2.0"
98
out=$(profile "$bin" rpc 'OpenAgents.ReleaseState.observe("retained-through-0.3.0"); IO.puts("observed")' 2>/dev/null)
99
echo "$out" | grep -q observed || fail "observe failed: $out"
100
101
echo "unpacking 0.3.0"
102
profile "$bin" rpc ':release_handler.unpack_release(:binary.bin_to_list("openagents-0.3.0"))' >/dev/null 2>&1 || true
103
[ -d "$runtime_root/releases/0.3.0" ] || fail "unpack produced no releases/0.3.0 directory"
104
105
echo "installing 0.3.0"
106
profile "$bin" rpc 'Castle.generate("0.3.0"); :release_handler.install_release(:binary.bin_to_list("0.3.0"))' >/dev/null 2>&1 || true
107
108
out=$(profile "$bin" rpc 'IO.puts(to_string(Application.spec(:openagents, :vsn)))' 2>/dev/null)
109
echo "$out" | grep -q "0.3.0" || fail "running version did not become 0.3.0: $out"
110
111
out=$(profile "$bin" rpc 'IO.puts(inspect({OpenAgents.ReleaseState.snapshot().schema_version, Enum.any?(OpenAgents.ReleaseState.snapshot().observations, &(&1 == "retained-through-0.3.0"))}))' 2>/dev/null)
112
echo "$out" | grep -qF "{2, true}" || fail "state did not survive the upgrade: $out"
113
114
profile "$bin" rpc ':release_handler.make_permanent(:binary.bin_to_list("0.3.0"))' >/dev/null 2>&1 || true
115
echo "0.3.0 installed live and made permanent; observations survived"
116
117
echo "reversing to 0.2.0"
118
profile "$bin" rpc ':release_handler.install_release(:binary.bin_to_list("0.2.0")); :release_handler.make_permanent(:binary.bin_to_list("0.2.0"))' >/dev/null 2>&1 || true
119
120
out=$(profile "$bin" rpc 'IO.puts(to_string(Application.spec(:openagents, :vsn)))' 2>/dev/null)
121
echo "$out" | grep -q "0.2.0" || fail "reverse did not restore 0.2.0: $out"
122
123
out=$(profile "$bin" rpc 'IO.puts(inspect(Enum.any?(OpenAgents.ReleaseState.snapshot().observations, &(&1 == "retained-through-0.3.0"))))' 2>/dev/null)
124
echo "$out" | grep -qF "true" || fail "observations did not survive the reverse"
125
126
echo "re-upgrading to 0.3.0"
127
profile "$bin" rpc ':release_handler.install_release(:binary.bin_to_list("0.3.0")); :release_handler.make_permanent(:binary.bin_to_list("0.3.0"))' >/dev/null 2>&1 || true
128
129
out=$(profile "$bin" rpc 'IO.puts(to_string(Application.spec(:openagents, :vsn)))' 2>/dev/null)
130
echo "$out" | grep -q "0.3.0" || fail "re-upgrade did not restore 0.3.0: $out"
131
132
echo ""
133
echo "GENERALIZED RELUP PROOF PASSED"
134
echo "forward 0.2.0->0.3.0, reverse, and re-upgrade all installed hot;"
135
echo "ReleaseState observations survived every transition."
ops/staging/gate-5-profile.sh modified +6 -1

@@ -13,7 +13,12 @@ set -eu

13 13
14 14
export GITHUB_OAUTH_SCOPES="repo,read:org"
15 15
export GITHUB_REDIRECT_URI="https://staging.openagents.com/auth/github/callback"
16
export GITHUB_TOKEN_DECRYPTION_KEYS_JSON="${GITHUB_TOKEN_DECRYPTION_KEYS_JSON:-{}}"
16
# Written as an explicit default rather than ${VAR:-{}}: POSIX expansion ends
17
# at the first `}`, so a preset value came back corrupted with a trailing
18
# brace and failed JSON validation downstream.
19
if [ -z "${GITHUB_TOKEN_DECRYPTION_KEYS_JSON:-}" ]; then
20
  export GITHUB_TOKEN_DECRYPTION_KEYS_JSON='{}'
21
fi
17 22
export OPENAGENTS_ALLOWED_ORIGINS="https://staging.openagents.com"
18 23
export OPENAGENTS_CODING_JOBS_DIR="/var/lib/openagents/coding-jobs"
19 24
export OPENAGENTS_DATABASE_IPV6="false"

This page updates live while a promote is in flight · changelog