OpenAgents

Commit

Add the staging resilience evidence gate

fa9cad5a96fb · Christopher David · · parent ab247b5146ed 

Add the staging resilience evidence gate

Deploy story

What this commit did to the running system — joined from the forge receipt chain, the part a commit page elsewhere cannot show.

Not deployed through the forge lane

No push, promotion, build, or deploy receipt references this commit (receipts are scanned over a bounded recent window). Changes shipped by full node replacement carry their proof in the release gate receipt instead.

Changed files

  • modified docs/2026-08-20-integration-hardening-and-staging-readiness-recommendations.md
  • modified docs/operations/staging-regression.md
  • modified docs/operations/staging-report-template.md
  • added docs/operations/staging-resilience-report-template.md
  • added docs/operations/staging-resilience.md
  • modified infra/staging/README.md
  • modified ops/ci/contracts.sh
  • modified ops/staging/finalize-report.sh
  • added ops/staging/new-resilience-report.sh
  • modified ops/staging/record-result.sh
  • added ops/staging/resilience-matrix.json
  • added ops/staging/resilience.sh
  • modified ops/staging/validate-report.sh
  • added ops/staging/validate-resilience-report.sh
  • modified test/openagents/staging_regression_contract_test.exs
  • added test/openagents/staging_resilience_contract_test.exs

Diff

16 files changed, +1221 -33

docs/2026-08-20-integration-hardening-and-staging-readiness-recommendations.md modified +32 -1

@@ -3,7 +3,7 @@

3 3 
Date: 2026-08-20
4 4
5 5 
Status: In progress; Gates 0–11 complete locally, Gate 12 cloud proof pending,
6 
Gate 13 live deployment pending, and the Gate 14 harness complete locally
6 
Gate 13 live deployment pending, and the Gate 14–15 harnesses complete locally
7 7
8 8 
## Outcome
9 9

@@ -1562,6 +1562,35 @@ After failure injection, run a staging soak:

1562 1562 
data loss, authority expansion, fleet divergence, secret leakage, or unexplained
1563 1563 
error accumulation.
1564 1564
1565 
### Gate 15 implementation status
1566
1567 
Implemented locally on 2026-08-20:
1568
1569 
- Added a versioned resilience matrix covering all 11 controlled-failure
1570 
  scenarios and six required soak canaries.
1571 
- Added an exact-candidate resilience report generator. A real report requires
1572 
  the same candidate manifest and a fully passed Gate 14 report before it can
1573 
  be created.
1574 
- Extended the attempt recorder and atomic finalizer to handle both regression
1575 
  and resilience reports while retaining failed attempts and refusing
1576 
  not-applicable controlled failures.
1577 
- Added a strict resilience validator. Completion requires all controlled
1578 
  failures passed, at least 172,800 measured seconds without a redeploy, stable
1579 
  candidate identity, 576 metric samples, the versioned status, typed, memory,
1580 
  tracker, Git, and voice canary minimums, a post-soak smoke, and zero
1581 
  unexplained harm counts.
1582 
- Bound the main staging report's final state to one nested, complete resilience
1583 
  report with the same candidate SHA and application image digest. The nested
1584 
  evidence tree is checksum-, path-, permission-, and content-scan validated.
1585 
- Added the [controlled-failure and soak runbook](operations/staging-resilience.md)
1586 
  and [resilience evidence contract](operations/staging-resilience-report-template.md).
1587 
  The network-free dry run proves the 11-case and 48-hour contracts fail closed.
1588
1589 
The local Gate 15 harness and contract tests pass. Live controlled failures and
1590 
the measured 48-hour soak remain blocked until Gates 12–14 complete on the
1591 
isolated staging target. No cloud or production resource changed while adding
1592 
the harness.
1593
1565 1594 
## Required staging evidence
1566 1595
1567 1596 
Store one staging report per candidate. Include:

@@ -1661,6 +1690,8 @@ each handoff.

1661 1690 
- [x] Owned local gates produce exact-SHA receipts.
1662 1691 
- [x] The versioned staging matrix, private evidence report, scanner, recorder,
1663 1692 
      validator, and network-free harness dry run exist.
1693 
- [x] The versioned controlled-failure matrix and fail-closed 48-hour soak
1694 
      evidence contract exist.
1664 1695 
- [ ] Web and distributed staging are isolated from production.
1665 1696 
- [ ] Staging has a separate database instance and failure domain.
1666 1697 
- [ ] The migration lineage is mapped and rehearsed for every nonempty target.
docs/operations/staging-regression.md modified +4 -2

@@ -311,8 +311,10 @@ complete, seal the regression result:

311 311 
ops/staging/finalize-report.sh --regression "$report"
312 312 
```
313 313
314 
Do not edit a regression-passed report. Gate 15 adds failure-injection timeline,
315 
soak, and known-issue evidence to a reviewed copy, then seals it with:
314 
Do not edit a regression-passed report until Gate 15 produces one complete,
315 
self-contained report through the
316 
[staging resilience runbook](staging-resilience.md). Attach that report as both
317 
the failure-injection timeline and soak receipt, then seal the main report with:
316 318
317 319 
```sh
318 320 
ops/staging/finalize-report.sh --final "$report"
docs/operations/staging-report-template.md modified +12 -5

@@ -77,11 +77,18 @@ under `scan-evidence.sh`.

77 77
78 78 
## Gate 15 evidence
79 79
80 
Final completion adds:
81
82 
- One or more evidence references in `failure_injection_timeline`.
83 
- A 48-hour `soak_receipt` that includes candidate identity, start, end,
84 
  scheduled canary counts, bounded resource summaries, and post-soak smoke.
80 
Final completion nests one complete report produced by the
81 
[staging resilience runbook](staging-resilience.md). Add the same
82 
`resilience-report` evidence reference to the failure-injection timeline and
83 
soak receipt. The main validator reruns the nested final validator and compares
84 
its candidate SHA and image digest with Gate 14.
85
86 
The resilience report proves:
87
88 
- All 11 controlled-failure cases and their retry histories.
89 
- At least 48 hours without a redeploy.
90 
- Scheduled typed, memory, voice, tracker, Git, and status canaries.
91 
- Five-minute resource samples and a post-soak smoke.
85 92 
- Every known issue with ID, owner, severity, and disposition.
86 93
87 94 
An empty known-issue list is valid only when review found no issues. A failure,
docs/operations/staging-resilience-report-template.md added +66

@@ -0,0 +1,66 @@

1 
# Staging resilience evidence contract
2
3 
Date: 2026-08-20
4
5 
Status: Current Gate 15 evidence contract
6
7 
`new-resilience-report.sh` generates one report for the candidate that already
8 
passed Gate 14. The report binds the candidate manifest, application image,
9 
release, and Gate 14 report checksum to 11 controlled-failure results and one
10 
48-hour soak.
11
12 
## Failure results
13
14 
Every failure result starts pending. A passed, failed, or blocked result requires
15 
an attempt and sanitized evidence. Ordinals remain contiguous, and a retry
16 
appends rather than replaces the first observation. Controlled-failure cases
17 
cannot be not applicable.
18
19 
Use `record-result.sh` to update results. It accepts both the Gate 14 and Gate 15
20 
report schemas and selects the correct strict validator.
21
22 
## Soak fields
23
24 
Completion requires:
25
26 
- Start and completion UTC timestamps separated by at least 172,800 seconds.
27 
- `candidate_identity_stable` set to true and `redeploy_count` set to zero.
28 
- At least 576 five-minute metric samples and a sanitized metric receipt.
29 
- A sanitized soak timeline receipt.
30 
- Exact canary IDs, cadences, minimums, completed counts, passed counts, and
31 
  aggregate receipts.
32 
- A full post-soak smoke receipt from the same candidate without a redeploy.
33 
- Zero unexplained errors, data loss, authority expansion, fleet divergence,
34 
  secret leakage, and unexplained restarts.
35
36 
The canary minimums are 576 status, 96 typed, 96 memory, 96 tracker, 96 Git, and
37 
24 fake-media voice passes.
38
39 
## Known issues
40
41 
Each known issue records a bounded ID, owner, severity, and disposition. Allowed
42 
dispositions are `resolved` and `accepted_non_blocking`. High and critical
43 
issues must be resolved. Do not put user content, raw logs, or credentials into
44 
an issue field.
45
46 
## Evidence references
47
48 
Evidence references use the same contract as the Gate 14 report: relative path
49 
under `evidence/`, lowercase SHA-256, and bounded kind. Files must be regular,
50 
unlinked, at most 50 MiB, mode `0400` or `0600`, checksum-correct, and safe under
51 
`scan-evidence.sh`.
52
53 
The final resilience directory is self-contained. When nested under the Gate 14
54 
report, its own `evidence/` paths continue to resolve relative to the resilience
55 
report.
56
57 
## States
58
59 
- `draft` permits pending and failed controlled-failure results.
60 
- `recorded` proves every failure case has an outcome but can retain a failure
61 
  or block.
62 
- `complete` requires every recovery, the full soak, all scheduled canaries,
63 
  operational evidence, and zero unexplained harm.
64
65 
Only `finalize-report.sh` changes report state. A synthetic dry-run report can
66 
never become recorded or complete.
docs/operations/staging-resilience.md added +279

@@ -0,0 +1,279 @@

1 
# Run controlled failures and the 48-hour staging soak
2
3 
Date: 2026-08-20
4
5 
Status: Harness implemented and locally proven; live Gate 15 execution pending
6
7 
Use this runbook after one exact candidate has passed Gate 14. It records all
8 
controlled-failure attempts, requires 48 measured hours without a redeploy,
9 
enforces scheduled canary and metric minimums, and refuses completion when any
10 
unexplained harm remains.
11
12 
This is a staging-only destructive test plan. Never run it against production,
13 
a shared database, a shared connection budget, a shared fleet, or credentials
14 
that authorize another environment.
15
16 
## Prove the safety fence
17
18 
Before injecting a failure, confirm all of these conditions:
19
20 
- The Gate 14 report is `regression_passed` for the selected candidate.
21 
- Staging has its own project, VPC, private database instance, web lane,
22 
  distributed fleet, deployer, identities, buckets, registry, and credentials.
23 
- Every cloud target contains the reviewed staging project number. No command
24 
  uses an implicit CLI project.
25 
- Database backups, snapshots, rollback artifacts, and last-known-good release
26 
  identities are current and restore-tested.
27 
- A manifest-scoped disposable run owns every test account, repository,
28 
  recording, and product machine.
29 
- The operator has a stop condition, a recovery command, and an observation
30 
  window for the single failure being injected.
31 
- No unrelated staging user or test is active.
32
33 
Keep production identifiers only as comparison fences. Do not run two failures
34 
at once. Restore the healthy baseline and run a bounded smoke between cases.
35
36 
## Prove the local harness
37
38 
Run the network-free check:
39
40 
```sh
41 
ops/staging/resilience.sh check
42 
```
43
44 
Expected result:
45
46 
```text
47 
Staging resilience harness dry run passed (11 failures; 48-hour soak; no network requests sent).
48 
```
49
50 
The check validates the versioned matrix, report generator, evidence scanner,
51 
attempt recorder, final validator, exact 48-hour duration, six canary schedules,
52 
and fail-closed finalization.
53
54 
## Create the resilience report
55
56 
Use the same candidate directory and passed Gate 14 report:
57
58 
```sh
59 
candidate_sha=$(git rev-parse HEAD)
60 
candidate_dir=".git/openagents/staging-candidates/$candidate_sha"
61 
gate14_run="gate14-20260820-0001"
62 
gate14_report=".git/openagents/staging-reports/$candidate_sha/$gate14_run/report.json"
63 
resilience_run="gate15-20260820-0001"
64
65 
ops/staging/new-resilience-report.sh \
66 
  "$candidate_dir" "$gate14_report" "$resilience_run"
67 
```
68
69 
The generator verifies the candidate manifest and Gate 14 report, compares the
70 
candidate SHA, and creates a mode-`0600` report at:
71
72 
```text
73 
.git/openagents/staging-resilience/<candidate-sha>/<run-id>/report.json
74 
```
75
76 
Validate the draft:
77
78 
```sh
79 
resilience_report=".git/openagents/staging-resilience/$candidate_sha/$resilience_run/report.json"
80 
ops/staging/validate-resilience-report.sh --draft "$resilience_report"
81 
```
82
83 
## Record a controlled-failure attempt
84
85 
Create a content-free receipt with the candidate identity, failure ID, UTC
86 
window, injected condition, stop condition, observed bounded states, recovery
87 
action, recovery time, post-recovery smoke hash, and zero-or-bounded error
88 
counts. Never retain request bodies, product content, credentials, private IPs,
89 
or raw database rows.
90
91 
Record a pass:
92
93 
```sh
94 
ops/staging/record-result.sh \
95 
  "$resilience_report" failure-001 passed controlled-failure \
96 
  /path/to/sanitized-recovery-receipt.json
97 
```
98
99 
Record a failure or block with a short mode-`0600` reason file:
100
101 
```sh
102 
ops/staging/record-result.sh \
103 
  "$resilience_report" failure-001 failed controlled-failure \
104 
  /path/to/sanitized-recovery-receipt.json /path/to/reason.txt
105 
```
106
107 
After correction, append a passed retry. The recorder preserves the failed
108 
attempt. All 11 scenarios are part of the architecture and cannot be marked not
109 
applicable.
110
111 
## Inject one failure at a time
112
113 
Use `ops/staging/resilience-matrix.json` as the canonical inventory. For every
114 
case, capture baseline identity and health, inject only the named condition,
115 
observe the documented bound, restore service, and run the full public smoke
116 
plus the affected authenticated flow.
117
118 
### Provider stream failures
119
120 
Inject a timeout, one malformed provider event, and a closed stream without a
121 
completion event in separate sub-attempts. Prove that the turn becomes an honest
122 
failed or incomplete terminal state, no false completion is persisted, budgets
123 
stop, partial output remains bounded, and a later turn succeeds.
124
125 
### PostgreSQL restart and exhaustion
126
127 
Snapshot first. Restart only the isolated staging database, then separately
128 
consume the staging-only connection reserve with a bounded test client. Prove
129 
that writes fail or retry according to their contracts, no committed row is
130 
lost or duplicated, readiness reflects the outage, connections return below
131 
budget, and the last-known-good application remains compatible.
132
133 
Stop immediately if any production instance, connection, or service account
134 
appears in the target inventory.
135
136 
### PubSub interruption and LiveView reconnect
137
138 
Interrupt the staging PubSub path while preserving PostgreSQL. Prove that UI
139 
projections may become stale but durable truth remains correct, reconnect
140 
restores the projection, and no action executes twice.
141
142 
### Supervised process termination
143
144 
Terminate one staging process at a time for turn, voice, work, semantic recall,
145 
builder, and deployer paths. Use the supervisor-visible PID, not a broad system
146 
kill. Prove each durable terminal state, restart behavior, idempotency fence,
147 
receipt, and user-visible recovery contract.
148
149 
### Machine disconnect
150
151 
Disconnect a disposable machine during a harmless committed step. Prove that
152 
the lease or step cannot execute twice, committed evidence survives, the job
153 
reaches a bounded state, reconnection does not expand authority, and revoke plus
154 
cleanup still work.
155
156 
### Artifact-store and cache faults
157
158 
Refuse the staging artifact store, then separately present a corrupt cache
159 
entry. Prove checksum refusal, readiness fencing, bounded retry, no unverified
160 
load, and convergence to the durable candidate after restoration.
161
162 
### Unreachable node and membership change
163
164 
Make one staging fleet node unreachable. Prove that a deployment cannot become
165 
live with a partial fleet and that restored nodes converge before readiness.
166 
In a separate attempt, change membership during a deployment and prove that the
167 
transaction uses one reviewed participant set or aborts and rolls every affected
168 
node back.
169
170 
### Builder sidecar failure
171
172 
Crash the isolated builder sidecar and inject a stale response with the wrong
173 
request identity. Prove that the request remains bounded, stale output cannot
174 
attach to a new build, no artifact becomes eligible, and a clean retry produces
175 
one immutable manifest.
176
177 
### Browser and recording failures
178
179 
Navigate away and destroy the active tab during microphone use. Prove tracks,
180 
audio elements, peer connections, recorder graphs, and server sessions close
181 
within their bounds. Separately fail a recording upload and send a late final
182 
chunk. Prove generation fencing, honest recording state, retention behavior,
183 
and continued typed and live-voice usability.
184
185 
## Seal the controlled-failure pass
186
187 
When every scenario has an outcome, a report with failures can be sealed for
188 
review:
189
190 
```sh
191 
ops/staging/finalize-report.sh --recorded "$resilience_report"
192 
```
193
194 
Corrective attempts return it to draft. Do not start the soak until all 11 last
195 
attempts pass, the candidate is healthy, and the test data is reconciled.
196
197 
## Run 48 hours without a redeploy
198
199 
Record a UTC start time after the final failure-recovery smoke. Keep the web and
200 
distributed staging lanes on the exact candidate for at least 172,800 seconds.
201 
Any application, configuration, image, release, migration, or fleet redeploy
202 
invalidates the soak; set `redeploy_count` honestly and start a new 48-hour run.
203
204 
Use these minimum schedules:
205
206 
| Canary or sample | Cadence | Minimum over 48 hours |
207 
| --- | ---: | ---: |
208 
| Status and candidate identity | 5 minutes | 576 passes |
209 
| Resource metrics | 5 minutes | 576 complete samples |
210 
| Typed chat | 30 minutes | 96 passes |
211 
| Memory write/read/forget | 30 minutes | 96 passes |
212 
| Tracker read/write/cleanup | 30 minutes | 96 passes |
213 
| Git clone/fetch/push/cleanup | 30 minutes | 96 passes |
214 
| Fake-media voice lifecycle | 2 hours | 24 passes |
215
216 
Each canary receipt must bind the candidate SHA, image digest, scheduled and
217 
actual UTC time, bounded outcome, attempt count, and cleanup result. A retry does
218 
not remove the initial failed observation. The final aggregate canary count must
219 
contain only completed attempts, and every completed attempt must pass before
220 
the report can complete.
221
222 
Each metric sample records bounded values for database connections, queue depth,
223 
mailbox growth, process count, memory, CPU, restart count, artifact cache, Ra
224 
state, and node convergence. Keep exact revision labels and counts; omit log
225 
content, host addresses, database values, and credentials.
226
227 
Investigate every crash, unexplained retry, stale active row, divergent node,
228 
leaked process, and content-bearing log entry during the soak. A high or critical
229 
known issue must be resolved. A low or medium issue can be accepted only with a
230 
named owner and an explicit non-blocking disposition.
231
232 
After 48 hours, run the full Gate 14 public and authenticated smoke again without
233 
redeploying. Record the exact end time after that smoke.
234
235 
## Complete and validate the resilience report
236
237 
Attach sanitized timeline, metric, canary, and post-soak-smoke receipts under
238 
the resilience report's `evidence/` directory. Populate the soak counts and
239 
booleans described in the
240 
[resilience evidence contract](staging-resilience-report-template.md).
241
242 
Finalization requires all failure cases passed, at least 48 measured hours,
243 
zero redeploys, stable candidate identity, every canary minimum, at least 576
244 
metric samples, and zero unexplained error, data-loss, authority-expansion,
245 
fleet-divergence, secret-leak, or restart counts:
246
247 
```sh
248 
ops/staging/finalize-report.sh --final "$resilience_report"
249 
```
250
251 
The command validates every nested evidence path, checksum, owner-only mode,
252 
and safety scan before replacing the report.
253
254 
## Attach Gate 15 to the staging report
255
256 
Copy the completed resilience report and its `evidence/` directory beneath the
257 
Gate 14 report as one self-contained tree, for example:
258
259 
```text
260 
<gate14-report-directory>/evidence/gate15/
261 
  report.json
262 
  report.sha256
263 
  evidence/
264 
    ...
265 
```
266
267 
Scan the copied tree and verify every checksum. Add one reference with kind
268 
`resilience-report` to both `failure_injection_timeline` and `soak_receipt` in
269 
the main staging report. The main validator reruns the full resilience
270 
validator and requires its candidate SHA and image digest to match Gate 14.
271
272 
Then finalize the main report:
273
274 
```sh
275 
ops/staging/finalize-report.sh --final "$gate14_report"
276 
```
277
278 
Retain only the sanitized, checksummed tree in staging evidence storage. Gate 15
279 
completion does not authorize production.
infra/staging/README.md modified +3 -1

@@ -39,7 +39,9 @@ to publish the exact application image, builder image, release archive, SBOM,

39 39 
and candidate manifest for Gate 13. Use the
40 40 
[staging regression runbook](../../docs/operations/staging-regression.md) only
41 41 
after the candidate and selected migration path are proven on this isolated
42 
target.
42 
target. The later
43 
[controlled-failure and soak runbook](../../docs/operations/staging-resilience.md)
44 
must use the same boundary and must not begin before the regression passes.
43 45
44 46 
## Prerequisites
45 47
ops/ci/contracts.sh modified +4

@@ -11,11 +11,14 @@ sh -n rel/overlays/bin/migration-lineage

11 11 
sh -n ops/staging/cleanup-run.sh
12 12 
sh -n ops/staging/finalize-report.sh
13 13 
sh -n ops/staging/new-report.sh
14 
sh -n ops/staging/new-resilience-report.sh
14 15 
sh -n ops/staging/publish-candidate.sh
15 16 
sh -n ops/staging/record-result.sh
16 17 
sh -n ops/staging/regression.sh
18 
sh -n ops/staging/resilience.sh
17 19 
sh -n ops/staging/run-public-smoke.sh
18 20 
sh -n ops/staging/scan-evidence.sh
21 
sh -n ops/staging/validate-resilience-report.sh
19 22 
sh -n ops/staging/validate-report.sh
20 23 
elixir ops/ci/docs-check.exs
21 24 
MIX_ENV=test mix test --warnings-as-errors \

@@ -25,6 +28,7 @@ MIX_ENV=test mix test --warnings-as-errors \

25 28 
  test/openagents/staging_cleanup_test.exs \
26 29 
  test/openagents/staging_candidate_contract_test.exs \
27 30 
  test/openagents/staging_regression_contract_test.exs \
31 
  test/openagents/staging_resilience_contract_test.exs \
28 32 
  test/openagents_web/icon_affordances_test.exs \
29 33 
  test/openagents_web/icons_test.exs \
30 34 
  test/openagents_web/ui_test.exs
ops/staging/finalize-report.sh modified +19 -1

@@ -27,6 +27,24 @@ for command_name in jq sha256sum; do

27 27 
  fi
28 28 
done
29 29
30 
report_schema=$(jq -r '.schema // empty' "$report")
31 
case "$report_schema" in
32 
  openagents.staging-report.v1)
33 
    validator="$script_dir/validate-report.sh"
34 
    ;;
35 
  openagents.staging-resilience-report.v1)
36 
    validator="$script_dir/validate-resilience-report.sh"
37 
    if [ "$mode" = --regression ]; then
38 
      echo "a resilience report supports only --recorded and --final" >&2
39 
      exit 1
40 
    fi
41 
    ;;
42 
  *)
43 
    echo "REPORT has an unsupported schema" >&2
44 
    exit 1
45 
    ;;
46 
esac
47
30 48 
report_dir=$(CDPATH= cd -- "$(dirname -- "$report")" && pwd)
31 49 
report_temp=$(mktemp "$report_dir/.report.XXXXXX")
32 50 
cleanup() {

@@ -41,7 +59,7 @@ jq --arg state "$state" --arg completed_at "$completed_at" '

41 59 
' "$report" >"$report_temp"
42 60 
chmod 600 "$report_temp"
43 61
44 
"$script_dir/validate-report.sh" "$mode" "$report_temp" >/dev/null || {
62 
"$validator" "$mode" "$report_temp" >/dev/null || {
45 63 
  echo "report remains unchanged because $mode validation failed" >&2
46 64 
  exit 1
47 65 
}
ops/staging/new-resilience-report.sh added +251

@@ -0,0 +1,251 @@

1 
#!/bin/sh
2 
set -eu
3
4 
script_dir=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)
5 
repo_root=$(CDPATH= cd -- "$script_dir/../.." && pwd)
6 
matrix="$script_dir/resilience-matrix.json"
7 
mode=${1:-}
8 
umask 077
9
10 
usage() {
11 
  echo "usage: ops/staging/new-resilience-report.sh CANDIDATE_DIRECTORY GATE14_REPORT RUN_ID" >&2
12 
  echo "       ops/staging/new-resilience-report.sh --dry-run OUTPUT" >&2
13 
  exit 64
14 
}
15
16 
for command_name in git jq sha256sum; do
17 
  if ! command -v "$command_name" >/dev/null 2>&1; then
18 
    echo "$command_name is required to create a staging resilience report" >&2
19 
    exit 1
20 
  fi
21 
done
22
23 
generated_at=$(date -u +%Y-%m-%dT%H:%M:%SZ)
24
25 
write_report() {
26 
  candidate_manifest=$1
27 
  candidate_manifest_sha256=$2
28 
  gate14_sha256=$3
29 
  run_id=$4
30 
  synthetic=$5
31 
  output=$6
32
33 
  jq -n \
34 
    --slurpfile matrix "$matrix" \
35 
    --slurpfile candidate "$candidate_manifest" \
36 
    --arg candidate_manifest_sha256 "$candidate_manifest_sha256" \
37 
    --arg gate14_sha256 "$gate14_sha256" \
38 
    --arg run_id "$run_id" \
39 
    --arg created_at "$generated_at" \
40 
    --argjson synthetic "$synthetic" '
41 
    ($matrix[0]) as $matrix |
42 
    ($candidate[0]) as $candidate |
43 
    {
44 
      schema: "openagents.staging-resilience-report.v1",
45 
      matrix_revision: $matrix.revision,
46 
      state: "draft",
47 
      synthetic: $synthetic,
48 
      run_id: $run_id,
49 
      created_at: $created_at,
50 
      completed_at: null,
51 
      target: {
52 
        environment: "staging",
53 
        base_url: "https://staging.openagents.com",
54 
        project: $candidate.target.project,
55 
        region: $candidate.target.region
56 
      },
57 
      candidate: {
58 
        git_sha: $candidate.git_sha,
59 
        candidate_manifest_sha256: $candidate_manifest_sha256,
60 
        application_image: $candidate.images.application.reference,
61 
        application_manifest_digest: $candidate.images.application.manifest_digest,
62 
        release_version: $candidate.release.version,
63 
        release_sha256: $candidate.release.sha256,
64 
        gate14_report_sha256: $gate14_sha256
65 
      },
66 
      failure_injections: [
67 
        $matrix.failure_injections[] |
68 
        {
69 
          id: .id,
70 
          title: .title,
71 
          status: "pending",
72 
          reason: null,
73 
          attempts: [],
74 
          evidence: []
75 
        }
76 
      ],
77 
      soak: {
78 
        required_duration_seconds: $matrix.soak.required_duration_seconds,
79 
        started_at: null,
80 
        completed_at: null,
81 
        candidate_identity_stable: null,
82 
        redeploy_count: null,
83 
        metric_sample_cadence_seconds: $matrix.soak.metric_sample_cadence_seconds,
84 
        metric_sample_count: null,
85 
        timeline_receipt: null,
86 
        metrics_receipt: null,
87 
        canaries: [
88 
          $matrix.soak.canaries[] |
89 
          {
90 
            id: .id,
91 
            cadence_seconds: .cadence_seconds,
92 
            minimum_passes: .minimum_passes,
93 
            completed_count: null,
94 
            passed_count: null,
95 
            receipt: null
96 
          }
97 
        ],
98 
        post_soak_smoke_receipt: null,
99 
        unexplained_error_count: null,
100 
        data_loss_count: null,
101 
        authority_expansion_count: null,
102 
        fleet_divergence_count: null,
103 
        secret_leak_count: null,
104 
        unexplained_restart_count: null
105 
      },
106 
      known_issues: []
107 
    }
108 
  ' >"$output"
109 
}
110
111 
if [ "$mode" = "--dry-run" ]; then
112 
  [ "$#" -eq 2 ] || usage
113 
  output=$2
114
115 
  if [ -e "$output" ]; then
116 
    echo "dry-run resilience report output already exists" >&2
117 
    exit 1
118 
  fi
119
120 
  dry_root=$(mktemp -d /tmp/openagents-resilience-report-dry-run.XXXXXX)
121 
  cleanup() {
122 
    find "$dry_root" -depth -delete 2>/dev/null || true
123 
  }
124 
  trap cleanup EXIT INT TERM
125
126 
  git_sha=$(git -C "$repo_root" rev-parse --verify HEAD)
127 
  zero_digest=0000000000000000000000000000000000000000000000000000000000000000
128 
  candidate_manifest="$dry_root/candidate-manifest.json"
129
130 
  jq -n \
131 
    --arg git_sha "$git_sha" \
132 
    --arg digest "sha256:$zero_digest" \
133 
    --arg sha256 "$zero_digest" '
134 
    {
135 
      schema: "openagents.staging-candidate.v1",
136 
      git_sha: $git_sha,
137 
      branch: "main",
138 
      target: {environment: "staging", project: "openagents-staging-dry-run", region: "us-central1"},
139 
      images: {application: {
140 
        reference: ("us-central1-docker.pkg.dev/openagents-staging-dry-run/openagents/openagents@" + $digest),
141 
        manifest_digest: $digest
142 
      }},
143 
      release: {version: "dry-run", sha256: $sha256}
144 
    }
145 
  ' >"$candidate_manifest"
146
147 
  write_report "$candidate_manifest" "$zero_digest" "$zero_digest" dry-run true "$output"
148 
  chmod 600 "$output"
149 
  exit 0
150 
fi
151
152 
[ "$#" -eq 3 ] || usage
153 
candidate_dir=$1
154 
gate14_report=$2
155 
run_id=$3
156
157 
case "$run_id" in
158 
  "" | *[!a-z0-9-]* | -* | *- | *--*)
159 
    echo "RUN_ID must use lowercase letters, digits, and single interior hyphens" >&2
160 
    exit 1
161 
    ;;
162 
esac
163
164 
if [ "${#run_id}" -gt 63 ]; then
165 
  echo "RUN_ID must contain at most 63 characters" >&2
166 
  exit 1
167 
fi
168
169 
candidate_manifest="$candidate_dir/candidate-manifest.json"
170 
candidate_checksum="$candidate_dir/candidate-manifest.sha256"
171
172 
if [ ! -f "$candidate_manifest" ] || [ ! -f "$candidate_checksum" ]; then
173 
  echo "candidate directory must contain the manifest and its checksum" >&2
174 
  exit 1
175 
fi
176
177 
(cd "$candidate_dir" && sha256sum --check --strict candidate-manifest.sha256 >/dev/null)
178 
gate14_dir=$(CDPATH= cd -- "$(dirname -- "$gate14_report")" && pwd)
179
180 
if [ ! -f "$gate14_dir/report.sha256" ]; then
181 
  echo "Gate 14 report checksum is missing" >&2
182 
  exit 1
183 
fi
184
185 
(cd "$gate14_dir" && sha256sum --check --strict report.sha256 >/dev/null)
186 
"$script_dir/validate-report.sh" --regression "$gate14_report" >/dev/null
187
188 
candidate_sha=$(jq -r '.git_sha' "$candidate_manifest")
189 
gate14_candidate_sha=$(jq -r '.candidate.git_sha' "$gate14_report")
190 
candidate_image_digest=$(jq -r '.images.application.manifest_digest' "$candidate_manifest")
191 
gate14_image_digest=$(jq -r '.candidate.application_manifest_digest' "$gate14_report")
192 
candidate_release_sha256=$(jq -r '.release.sha256' "$candidate_manifest")
193 
gate14_release_sha256=$(jq -r '.candidate.release_sha256' "$gate14_report")
194
195 
if [ "$candidate_sha" != "$gate14_candidate_sha" ] ||
196 
   [ "$candidate_image_digest" != "$gate14_image_digest" ] ||
197 
   [ "$candidate_release_sha256" != "$gate14_release_sha256" ]; then
198 
  echo "Gate 14 report does not identify the selected candidate" >&2
199 
  exit 1
200 
fi
201
202 
jq -e '
203 
  . as $manifest |
204 
  .schema == "openagents.staging-candidate.v1" and
205 
  (.git_sha | test("^[0-9a-f]{40}$")) and
206 
  .branch == "main" and
207 
  .target.environment == "staging" and
208 
  (.target.project | test("stag"; "i")) and
209 
  (.images.application.manifest_digest | test("^sha256:[0-9a-f]{64}$")) and
210 
  (.images.application.reference | endswith("@" + $manifest.images.application.manifest_digest)) and
211 
  (.release.sha256 | test("^[0-9a-f]{64}$"))
212 
' "$candidate_manifest" >/dev/null || {
213 
  echo "candidate manifest does not satisfy the resilience report contract" >&2
214 
  exit 1
215 
}
216
217 
candidate_manifest_sha256=$(sha256sum "$candidate_manifest" | cut -d ' ' -f 1)
218 
gate14_sha256=$(sha256sum "$gate14_report" | cut -d ' ' -f 1)
219 
report_root="$repo_root/.git/openagents/staging-resilience/$candidate_sha/$run_id"
220 
report_parent=$(dirname -- "$report_root")
221 
report_temp=
222
223 
if [ -e "$report_root" ]; then
224 
  echo "resilience report already exists for this candidate and run ID" >&2
225 
  exit 1
226 
fi
227
228 
mkdir -p "$report_parent"
229 
report_temp=$(mktemp -d "$report_parent/.report.$run_id.XXXXXX")
230 
cleanup() {
231 
  if [ -n "$report_temp" ] && [ -d "$report_temp" ]; then
232 
    find "$report_temp" -depth -delete 2>/dev/null || true
233 
  fi
234 
}
235 
trap cleanup EXIT INT TERM
236
237 
write_report \
238 
  "$candidate_manifest" \
239 
  "$candidate_manifest_sha256" \
240 
  "$gate14_sha256" \
241 
  "$run_id" \
242 
  false \
243 
  "$report_temp/report.json"
244
245 
report_sha256=$(sha256sum "$report_temp/report.json" | cut -d ' ' -f 1)
246 
printf '%s  report.json\n' "$report_sha256" >"$report_temp/report.sha256"
247 
mv "$report_temp" "$report_root"
248 
report_temp=
249
250 
echo "Created staging resilience report for $candidate_sha"
251 
echo "Report: .git/openagents/staging-resilience/$candidate_sha/$run_id/report.json"
ops/staging/record-result.sh modified +34 -9

@@ -28,6 +28,24 @@ for command_name in jq sha256sum; do

28 28 
  fi
29 29 
done
30 30
31 
report_schema=$(jq -r '.schema // empty' "$report")
32 
case "$report_schema" in
33 
  openagents.staging-report.v1)
34 
    validator="$script_dir/validate-report.sh"
35 
    result_field=results
36 
    matrix_name="staging regression matrix"
37 
    ;;
38 
  openagents.staging-resilience-report.v1)
39 
    validator="$script_dir/validate-resilience-report.sh"
40 
    result_field=failure_injections
41 
    matrix_name="staging resilience matrix"
42 
    ;;
43 
  *)
44 
    echo "REPORT has an unsupported schema" >&2
45 
    exit 1
46 
    ;;
47 
esac
48
31 49 
case "$case_id" in
32 50 
  [a-z]*-[0-9][0-9][0-9]) ;;
33 51 
  *) echo "CASE_ID has an invalid shape" >&2; exit 1 ;;

@@ -56,20 +74,26 @@ case "$outcome" in

56 74 
  *) usage ;;
57 75 
esac
58 76
77 
if [ "$report_schema" = openagents.staging-resilience-report.v1 ] &&
78 
   [ "$outcome" = not_applicable ]; then
79 
  echo "every controlled-failure case is applicable to the staging resilience gate" >&2
80 
  exit 1
81 
fi
82
59 83 
report_state=$(jq -r '.state // empty' "$report")
60 84 
case "$report_state" in
61 
  draft) "$script_dir/validate-report.sh" --draft "$report" >/dev/null ;;
62 
  recorded) "$script_dir/validate-report.sh" --recorded "$report" >/dev/null ;;
85 
  draft) "$validator" --draft "$report" >/dev/null ;;
86 
  recorded) "$validator" --recorded "$report" >/dev/null ;;
63 87 
  *)
64 88 
    echo "only draft or recorded reports accept result changes" >&2
65 89 
    exit 1
66 90 
    ;;
67 91 
esac
68 92
69 
if ! jq -e --arg case_id "$case_id" '
70 
  any(.results[]; .id == $case_id)
93 
if ! jq -e --arg case_id "$case_id" --arg result_field "$result_field" '
94 
  any(.[$result_field][]; .id == $case_id)
71 95 
' "$report" >/dev/null; then
72 
  echo "CASE_ID is not present in the staging matrix" >&2
96 
  echo "CASE_ID is not present in the $matrix_name" >&2
73 97 
  exit 1
74 98 
fi
75 99

@@ -110,8 +134,8 @@ trap cleanup EXIT INT TERM

110 134 
umask 077
111 135 
mkdir -p "$evidence_dir"
112 136
113 
ordinal=$(jq -r --arg case_id "$case_id" '
114 
  (.results[] | select(.id == $case_id) | .attempts | length) + 1
137 
ordinal=$(jq -r --arg case_id "$case_id" --arg result_field "$result_field" '
138 
  (.[$result_field][] | select(.id == $case_id) | .attempts | length) + 1
115 139 
' "$report")
116 140 
started_at=$(date -u +%Y-%m-%dT%H:%M:%SZ)
117 141 
completed_at=$started_at

@@ -160,11 +184,12 @@ jq \

160 184 
  --arg evidence_kind "$evidence_kind" \
161 185 
  --arg evidence_relative "$evidence_relative" \
162 186 
  --arg evidence_sha256 "$evidence_sha256" \
187 
  --arg result_field "$result_field" \
163 188 
  --rawfile reason "$reason_file" '
164 189 
  ($reason | rtrimstr("\n")) as $reason |
165 190 
  .state = "draft" |
166 191 
  .completed_at = null |
167 
  .results |= map(
192 
  .[$result_field] |= map(
168 193 
    if .id != $case_id then .
169 194 
    elif $outcome == "not_applicable" then
170 195 
      .status = "not_applicable" |

@@ -188,7 +213,7 @@ jq \

188 213 
' "$report" >"$report_temp"
189 214 
chmod 600 "$report_temp"
190 215
191 
"$script_dir/validate-report.sh" --draft "$report_temp" >/dev/null
216 
"$validator" --draft "$report_temp" >/dev/null
192 217 
mv "$report_temp" "$report"
193 218 
report_temp=
194 219
ops/staging/resilience-matrix.json added +30

@@ -0,0 +1,30 @@

1 
{
2 
  "schema": "openagents.staging-resilience-matrix.v1",
3 
  "revision": 1,
4 
  "failure_injections": [
5 
    {"id": "failure-001", "title": "Provider timeout, malformed event, and premature stream closure recover without a false completion"},
6 
    {"id": "failure-002", "title": "PostgreSQL restart and temporary connection exhaustion recover without data loss"},
7 
    {"id": "failure-003", "title": "PubSub interruption preserves durable truth and LiveView reconnects"},
8 
    {"id": "failure-004", "title": "Turn, voice, work, semantic, builder, and deployer process termination follows each recovery contract"},
9 
    {"id": "failure-005", "title": "Machine disconnect during a job preserves committed evidence and permits bounded recovery"},
10 
    {"id": "failure-006", "title": "Artifact-store unavailability and corrupt cache fail closed and reconverge"},
11 
    {"id": "failure-007", "title": "One unreachable fleet node does not expand authority or produce a partial live deployment"},
12 
    {"id": "failure-008", "title": "Node membership change during deployment preserves transaction and readiness invariants"},
13 
    {"id": "failure-009", "title": "Build sidecar crash and stale response cannot publish an invalid artifact"},
14 
    {"id": "failure-010", "title": "Browser navigation and tab destruction release microphone and voice resources"},
15 
    {"id": "failure-011", "title": "Recording upload failure and late final chunk remain generation-fenced and recoverable"}
16 
  ],
17 
  "soak": {
18 
    "required_duration_seconds": 172800,
19 
    "metric_sample_cadence_seconds": 300,
20 
    "minimum_metric_samples": 576,
21 
    "canaries": [
22 
      {"id": "status", "cadence_seconds": 300, "minimum_passes": 576},
23 
      {"id": "typed", "cadence_seconds": 1800, "minimum_passes": 96},
24 
      {"id": "memory", "cadence_seconds": 1800, "minimum_passes": 96},
25 
      {"id": "tracker", "cadence_seconds": 1800, "minimum_passes": 96},
26 
      {"id": "git", "cadence_seconds": 1800, "minimum_passes": 96},
27 
      {"id": "voice", "cadence_seconds": 7200, "minimum_passes": 24}
28 
    ]
29 
  }
30 
}
ops/staging/resilience.sh added +61

@@ -0,0 +1,61 @@

1 
#!/bin/sh
2 
set -eu
3
4 
script_dir=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)
5 
matrix="$script_dir/resilience-matrix.json"
6 
mode=${1:-}
7
8 
if [ "$mode" != check ] || [ "$#" -ne 1 ]; then
9 
  echo "usage: ops/staging/resilience.sh check" >&2
10 
  exit 64
11 
fi
12
13 
if ! command -v jq >/dev/null 2>&1; then
14 
  echo "jq is required to check the staging resilience harness" >&2
15 
  exit 1
16 
fi
17
18 
jq -e '
19 
  .schema == "openagents.staging-resilience-matrix.v1" and
20 
  .revision == 1 and
21 
  (.failure_injections | type == "array" and length == 11) and
22 
  ([.failure_injections[].id] | length == (unique | length)) and
23 
  all(.failure_injections[];
24 
    (.id | test("^failure-[0-9]{3}$")) and
25 
    (.title | type == "string" and length > 0)) and
26 
  .soak.required_duration_seconds == 172800 and
27 
  .soak.metric_sample_cadence_seconds == 300 and
28 
  .soak.minimum_metric_samples == 576 and
29 
  (.soak.canaries | type == "array" and length == 6) and
30 
  ([.soak.canaries[].id] | sort) == ["git", "memory", "status", "tracker", "typed", "voice"] and
31 
  all(.soak.canaries[]; . as $canary |
32 
    ($canary.cadence_seconds | type == "number" and . > 0 and floor == .) and
33 
    ($canary.minimum_passes | type == "number" and . >= (172800 / $canary.cadence_seconds) and floor == .))
34 
' "$matrix" >/dev/null || {
35 
  echo "staging resilience matrix contract failed" >&2
36 
  exit 1
37 
}
38
39 
check_root=$(mktemp -d /tmp/openagents-resilience-check.XXXXXX)
40 
cleanup() {
41 
  find "$check_root" -depth -delete 2>/dev/null || true
42 
}
43 
trap cleanup EXIT INT TERM
44
45 
report="$check_root/report.json"
46 
evidence="$check_root/evidence.json"
47
48 
"$script_dir/new-resilience-report.sh" --dry-run "$report"
49 
"$script_dir/validate-resilience-report.sh" --draft "$report" >/dev/null
50
51 
if "$script_dir/finalize-report.sh" --final "$report" >/dev/null 2>&1; then
52 
  echo "draft resilience report unexpectedly passed final validation" >&2
53 
  exit 1
54 
fi
55
56 
printf '%s\n' '{"schema":"openagents.resilience-check.v1","status":"recovered"}' >"$evidence"
57 
chmod 600 "$evidence"
58 
"$script_dir/record-result.sh" "$report" failure-001 passed dry-run "$evidence" >/dev/null
59 
"$script_dir/validate-resilience-report.sh" --draft "$report" >/dev/null
60
61 
echo "Staging resilience harness dry run passed (11 failures; 48-hour soak; no network requests sent)."
ops/staging/validate-report.sh modified +22

@@ -141,6 +141,11 @@ jq -e \

141 141 
     common_staging_evidence_complete and
142 142 
     (.staging_evidence.failure_injection_timeline | type == "array" and length > 0 and all(.[]; evidence_ref)) and
143 143 
     (.staging_evidence.soak_receipt | evidence_ref) and
144 
     .staging_evidence.soak_receipt.kind == "resilience-report" and
145 
     any(.staging_evidence.failure_injection_timeline[];
146 
       .path == $report.staging_evidence.soak_receipt.path and
147 
       .sha256 == $report.staging_evidence.soak_receipt.sha256 and
148 
       .kind == "resilience-report") and
144 149 
     (.staging_evidence.known_issues | type == "array" and all(.[];
145 150 
       (.id | nonempty) and (.owner | nonempty) and
146 151 
       (.severity | IN("low", "medium", "high", "critical")) and

@@ -207,4 +212,21 @@ while IFS="$tab" read -r relative_path expected_sha256; do

207 212 
  "$script_dir/scan-evidence.sh" "$evidence_path" >/dev/null
208 213 
done <"$refs"
209 214
215 
if [ "$mode" = --final ]; then
216 
  resilience_relative=$(jq -r '.staging_evidence.soak_receipt.path' "$report")
217 
  resilience_report="$report_dir/$resilience_relative"
218 
  "$script_dir/validate-resilience-report.sh" --final "$resilience_report" >/dev/null
219
220 
  main_candidate_sha=$(jq -r '.candidate.git_sha' "$report")
221 
  resilience_candidate_sha=$(jq -r '.candidate.git_sha' "$resilience_report")
222 
  main_image_digest=$(jq -r '.candidate.application_manifest_digest' "$report")
223 
  resilience_image_digest=$(jq -r '.candidate.application_manifest_digest' "$resilience_report")
224
225 
  if [ "$main_candidate_sha" != "$resilience_candidate_sha" ] ||
226 
     [ "$main_image_digest" != "$resilience_image_digest" ]; then
227 
    echo "resilience report does not identify the Gate 14 candidate" >&2
228 
    exit 1
229 
  fi
230 
fi
231
210 232 
echo "Staging report $mode validation passed."
ops/staging/validate-resilience-report.sh added +205

@@ -0,0 +1,205 @@

1 
#!/bin/sh
2 
set -eu
3
4 
script_dir=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)
5 
matrix="$script_dir/resilience-matrix.json"
6 
mode=${1:-}
7 
report=${2:-}
8
9 
case "$mode" in
10 
  --draft | --recorded | --final) ;;
11 
  *)
12 
    echo "usage: ops/staging/validate-resilience-report.sh [--draft|--recorded|--final] REPORT" >&2
13 
    exit 64
14 
    ;;
15 
esac
16
17 
if [ "$#" -ne 2 ] || [ ! -f "$report" ]; then
18 
  echo "REPORT must be a regular file" >&2
19 
  exit 1
20 
fi
21
22 
for command_name in jq realpath sha256sum stat; do
23 
  if ! command -v "$command_name" >/dev/null 2>&1; then
24 
    echo "$command_name is required to validate a staging resilience report" >&2
25 
    exit 1
26 
  fi
27 
done
28
29 
"$script_dir/scan-evidence.sh" "$report" >/dev/null
30
31 
jq -e \
32 
  --arg mode "$mode" \
33 
  --slurpfile matrix "$matrix" '
34 
  def digest: type == "string" and test("^[0-9a-f]{64}$");
35 
  def manifest_digest: type == "string" and test("^sha256:[0-9a-f]{64}$");
36 
  def timestamp: type == "string" and test("^[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}Z$");
37 
  def nonempty: type == "string" and length > 0 and length <= 500;
38 
  def evidence_ref:
39 
    type == "object" and
40 
    (.path | type == "string" and test("^evidence/[A-Za-z0-9._/-]+$") and (contains("..") | not)) and
41 
    (.sha256 | digest) and
42 
    (.kind | nonempty);
43 
  def valid_attempt:
44 
    type == "object" and
45 
    (.ordinal | type == "number" and . >= 1 and floor == .) and
46 
    (.outcome | IN("passed", "failed", "blocked")) and
47 
    (.started_at | timestamp) and
48 
    (.completed_at | timestamp) and
49 
    (.automatic_retry | type == "boolean") and
50 
    (.explanation == null or (.explanation | nonempty)) and
51 
    (.evidence | type == "array" and length > 0 and all(.[]; evidence_ref));
52 
  def valid_result($expected):
53 
    . as $result |
54 
    ($expected | map(select(.id == $result.id)) | .[0]) as $case |
55 
    $case != null and
56 
    $result.title == $case.title and
57 
    ($result.status | IN("pending", "passed", "failed", "blocked")) and
58 
    ($result.reason == null or ($result.reason | nonempty)) and
59 
    ($result.attempts | type == "array" and all(.[]; valid_attempt)) and
60 
    ($result.evidence | type == "array" and all(.[]; evidence_ref)) and
61 
    ($result.attempts as $attempts |
62 
      [$attempts[].ordinal] == [range(1; ($attempts | length) + 1)]) and
63 
    (if $result.status == "pending" then
64 
       ($result.attempts | length) == 0 and $result.reason == null
65 
     else
66 
       ($result.attempts | length) > 0 and
67 
       ($result.attempts[-1].outcome == $result.status) and
68 
       ($result.evidence | length) > 0 and
69 
       (if $result.status == "passed" then true else ($result.reason | nonempty) end)
70 
     end);
71 
  def issue_valid:
72 
    (.id | nonempty) and
73 
    (.owner | nonempty) and
74 
    (.severity | IN("low", "medium", "high", "critical")) and
75 
    (.disposition | IN("resolved", "accepted_non_blocking")) and
76 
    (if .severity | IN("high", "critical") then .disposition == "resolved" else true end);
77
78 
  $matrix[0].failure_injections as $expected_failures |
79 
  $matrix[0].soak.canaries as $expected_canaries |
80 
  . as $report |
81
82 
  .schema == "openagents.staging-resilience-report.v1" and
83 
  .matrix_revision == $matrix[0].revision and
84 
  (.synthetic | type == "boolean") and
85 
  (.run_id | type == "string" and test("^[a-z0-9](?:[a-z0-9-]{0,61}[a-z0-9])?$")) and
86 
  (.created_at | timestamp) and
87 
  (.completed_at == null or (.completed_at | timestamp)) and
88 
  .target.environment == "staging" and
89 
  .target.base_url == "https://staging.openagents.com" and
90 
  (.target.project | type == "string" and test("stag"; "i")) and
91 
  (.target.region | nonempty) and
92 
  (.candidate.git_sha | type == "string" and test("^[0-9a-f]{40}$")) and
93 
  (.candidate.candidate_manifest_sha256 | digest) and
94 
  (.candidate.application_manifest_digest | manifest_digest) and
95 
  (.candidate.application_image | endswith("@" + $report.candidate.application_manifest_digest)) and
96 
  (.candidate.release_version | nonempty) and
97 
  (.candidate.release_sha256 | digest) and
98 
  (.candidate.gate14_report_sha256 | digest) and
99 
  (.failure_injections | type == "array" and length == ($expected_failures | length)) and
100 
  ([.failure_injections[].id] | sort) == ([$expected_failures[].id] | sort) and
101 
  all(.failure_injections[]; valid_result($expected_failures)) and
102 
  .soak.required_duration_seconds == $matrix[0].soak.required_duration_seconds and
103 
  .soak.metric_sample_cadence_seconds == $matrix[0].soak.metric_sample_cadence_seconds and
104 
  (.soak.canaries | type == "array" and length == ($expected_canaries | length)) and
105 
  ([.soak.canaries[].id] | sort) == ([$expected_canaries[].id] | sort) and
106 
  all(.soak.canaries[]; . as $canary |
107 
    ($expected_canaries | map(select(.id == $canary.id)) | .[0]) as $expected |
108 
    $canary.cadence_seconds == $expected.cadence_seconds and
109 
    $canary.minimum_passes == $expected.minimum_passes and
110 
    ($canary.completed_count == null or ($canary.completed_count | type == "number" and . >= 0 and floor == .)) and
111 
    ($canary.passed_count == null or ($canary.passed_count | type == "number" and . >= 0 and floor == .)) and
112 
    ($canary.receipt == null or ($canary.receipt | evidence_ref))) and
113 
  (.known_issues | type == "array" and all(.[]; issue_valid)) and
114 
  ([.. | objects | select(has("path") or has("sha256"))] | all(.[]; evidence_ref)) and
115 
  (if $mode == "--draft" then
116 
     .state == "draft"
117 
   elif $mode == "--recorded" then
118 
     .state == "recorded" and .synthetic == false and (.completed_at | timestamp) and
119 
     all(.failure_injections[]; .status != "pending")
120 
   else
121 
     .state == "complete" and .synthetic == false and (.completed_at | timestamp) and
122 
     all(.failure_injections[]; .status == "passed") and
123 
     (.soak.started_at | timestamp) and
124 
     (.soak.completed_at | timestamp) and
125 
     ((.soak.completed_at | fromdateiso8601) - (.soak.started_at | fromdateiso8601) >= .soak.required_duration_seconds) and
126 
     ((.completed_at | fromdateiso8601) >= (.soak.completed_at | fromdateiso8601)) and
127 
     .soak.candidate_identity_stable == true and
128 
     .soak.redeploy_count == 0 and
129 
     (.soak.metric_sample_count | type == "number" and . >= $matrix[0].soak.minimum_metric_samples and floor == .) and
130 
     (.soak.timeline_receipt | evidence_ref) and
131 
     (.soak.metrics_receipt | evidence_ref) and
132 
     all(.soak.canaries[]; . as $canary |
133 
       ($canary.completed_count | type == "number" and . >= $canary.minimum_passes and floor == .) and
134 
       $canary.passed_count == $canary.completed_count and
135 
       ($canary.receipt | evidence_ref)) and
136 
     (.soak.post_soak_smoke_receipt | evidence_ref) and
137 
     .soak.unexplained_error_count == 0 and
138 
     .soak.data_loss_count == 0 and
139 
     .soak.authority_expansion_count == 0 and
140 
     .soak.fleet_divergence_count == 0 and
141 
     .soak.secret_leak_count == 0 and
142 
     .soak.unexplained_restart_count == 0
143 
   end)
144 
' "$report" >/dev/null || {
145 
  echo "staging resilience report does not satisfy $mode validation" >&2
146 
  exit 1
147 
}
148
149 
report_dir=$(realpath "$(dirname -- "$report")")
150 
refs=$(mktemp /tmp/openagents-resilience-report-refs.XXXXXX)
151 
cleanup() {
152 
  unlink "$refs" 2>/dev/null || true
153 
}
154 
trap cleanup EXIT INT TERM
155
156 
jq -r '
157 
  .. | objects |
158 
  select(has("path") and has("sha256")) |
159 
  [.path, .sha256] | @tsv
160 
' "$report" >"$refs"
161
162 
tab=$(printf '\t')
163 
while IFS="$tab" read -r relative_path expected_sha256; do
164 
  [ -n "$relative_path" ] || continue
165
166 
  case "$relative_path" in
167 
    evidence/*) ;;
168 
    *) echo "evidence reference must stay under evidence/: $relative_path" >&2; exit 1 ;;
169 
  esac
170
171 
  case "$relative_path" in
172 
    *..* | /* | *[!A-Za-z0-9._/-]*)
173 
      echo "unsafe evidence reference: $relative_path" >&2
174 
      exit 1
175 
      ;;
176 
  esac
177
178 
  evidence_path="$report_dir/$relative_path"
179 
  if [ ! -f "$evidence_path" ] || [ -L "$evidence_path" ]; then
180 
    echo "missing or linked evidence file: $relative_path" >&2
181 
    exit 1
182 
  fi
183
184 
  resolved=$(realpath "$evidence_path")
185 
  case "$resolved" in
186 
    "$report_dir"/evidence/*) ;;
187 
    *) echo "evidence path escapes the report directory: $relative_path" >&2; exit 1 ;;
188 
  esac
189
190 
  actual_sha256=$(sha256sum "$evidence_path" | cut -d ' ' -f 1)
191 
  if [ "$actual_sha256" != "$expected_sha256" ]; then
192 
    echo "evidence checksum mismatch: $relative_path" >&2
193 
    exit 1
194 
  fi
195
196 
  mode_bits=$(stat -c '%a' "$evidence_path" 2>/dev/null || stat -f '%Lp' "$evidence_path")
197 
  case "$mode_bits" in
198 
    400 | 600) ;;
199 
    *) echo "evidence file must use mode 0400 or 0600: $relative_path" >&2; exit 1 ;;
200 
  esac
201
202 
  "$script_dir/scan-evidence.sh" "$evidence_path" >/dev/null
203 
done <"$refs"
204
205 
echo "Staging resilience report $mode validation passed."
test/openagents/staging_regression_contract_test.exs modified +6 -14

@@ -100,7 +100,9 @@ defmodule OpenAgents.StagingRegressionContractTest do

100 100 
    refute output =~ credential
101 101 
  end
102 102
103 
  test "regression and final states require their complete evidence sets", %{test_root: test_root} do
103 
  test "regression passes with common evidence but final requires a resilience report", %{
104 
    test_root: test_root
105 
  } do
104 106 
    report = Path.join(test_root, "report.json")
105 107 
    evidence_dir = Path.join(test_root, "evidence")
106 108 
    evidence = Path.join(evidence_dir, "bounded-receipt.json")

@@ -174,19 +176,9 @@ defmodule OpenAgents.StagingRegressionContractTest do

174 176 
    assert {_, 0} = command("finalize-report.sh", ["--regression", report])
175 177 
    assert Jason.decode!(File.read!(report))["state"] == "regression_passed"
176 178
177 
    decoded = Jason.decode!(File.read!(report))
178
179 
    staging_evidence =
180 
      decoded["staging_evidence"]
181 
      |> Map.put("failure_injection_timeline", [reference])
182 
      |> Map.put("soak_receipt", reference)
183
184 
    decoded = Map.put(decoded, "staging_evidence", staging_evidence)
185 
    File.write!(report, Jason.encode!(decoded))
186 
    File.chmod!(report, 0o600)
187
188 
    assert {_, 0} = command("finalize-report.sh", ["--final", report])
189 
    assert Jason.decode!(File.read!(report))["state"] == "complete"
179 
    assert {output, 1} = command("finalize-report.sh", ["--final", report])
180 
    assert output =~ "report remains unchanged"
181 
    assert Jason.decode!(File.read!(report))["state"] == "regression_passed"
190 182
191 183 
    [checksum, "report.json"] =
192 184 
      test_root |> Path.join("report.sha256") |> File.read!() |> String.split()
test/openagents/staging_resilience_contract_test.exs added +193

@@ -0,0 +1,193 @@

1 
defmodule OpenAgents.StagingResilienceContractTest do
2 
  use ExUnit.Case, async: false
3
4 
  @scripts_root Path.expand("ops/staging")
5
6 
  setup do
7 
    test_root =
8 
      Path.join(
9 
        System.tmp_dir!(),
10 
        "openagents-staging-resilience-#{System.unique_integer([:positive])}"
11 
      )
12
13 
    File.mkdir_p!(test_root)
14 
    on_exit(fn -> File.rm_rf!(test_root) end)
15
16 
    %{test_root: test_root}
17 
  end
18
19 
  test "the resilience matrix covers every failure and soak canary" do
20 
    matrix =
21 
      @scripts_root |> Path.join("resilience-matrix.json") |> File.read!() |> Jason.decode!()
22
23 
    assert matrix["schema"] == "openagents.staging-resilience-matrix.v1"
24 
    assert length(matrix["failure_injections"]) == 11
25
26 
    assert Enum.uniq_by(matrix["failure_injections"], & &1["id"]) ==
27 
             matrix["failure_injections"]
28
29 
    assert matrix["soak"]["required_duration_seconds"] == 48 * 60 * 60
30 
    assert matrix["soak"]["minimum_metric_samples"] == 576
31
32 
    assert matrix["soak"]["canaries"] |> Enum.map(& &1["id"]) |> Enum.sort() ==
33 
             ~w(git memory status tracker typed voice)
34 
  end
35
36 
  test "the resilience dry run is local and fail-closed" do
37 
    {output, 0} = command("resilience.sh", ["check"])
38
39 
    assert output =~ "11 failures"
40 
    assert output =~ "48-hour soak"
41 
    assert output =~ "no network requests sent"
42 
  end
43
44 
  test "evidence permission checks support GNU and BSD stat" do
45 
    validator = File.read!(Path.join(@scripts_root, "validate-resilience-report.sh"))
46
47 
    assert validator =~ "stat -c '%a'"
48 
    assert validator =~ "stat -f '%Lp'"
49 
    assert validator =~ "report_dir=$(realpath"
50 
  end
51
52 
  test "finalization requires every recovery, 48 measured hours, canaries, and zero unexplained harm",
53 
       %{test_root: test_root} do
54 
    report = Path.join(test_root, "report.json")
55 
    evidence = Path.join(test_root, "recovery.json")
56
57 
    assert {_, 0} = command("new-resilience-report.sh", ["--dry-run", report])
58 
    File.write!(evidence, Jason.encode!(%{"schema" => "openagents.recovery-proof.v1"}))
59 
    File.chmod!(evidence, 0o600)
60
61 
    for ordinal <- 1..11 do
62 
      case_id = "failure-#{String.pad_leading(Integer.to_string(ordinal), 3, "0")}"
63
64 
      assert {_, 0} =
65 
               command("record-result.sh", [report, case_id, "passed", "recovery-proof", evidence])
66 
    end
67
68 
    decoded = report |> File.read!() |> Jason.decode!()
69 
    [reference | _] = hd(decoded["failure_injections"])["evidence"]
70 
    completed_at = DateTime.utc_now() |> DateTime.add(-1, :second) |> DateTime.truncate(:second)
71 
    started_at = DateTime.add(completed_at, -(48 * 60 * 60), :second)
72
73 
    soak = decoded["soak"]
74
75 
    canaries =
76 
      Enum.map(soak["canaries"], fn canary ->
77 
        canary
78 
        |> Map.put("completed_count", canary["minimum_passes"])
79 
        |> Map.put("passed_count", canary["minimum_passes"])
80 
        |> Map.put("receipt", reference)
81 
      end)
82
83 
    soak =
84 
      soak
85 
      |> Map.put("started_at", DateTime.to_iso8601(started_at))
86 
      |> Map.put("completed_at", DateTime.to_iso8601(completed_at))
87 
      |> Map.put("candidate_identity_stable", true)
88 
      |> Map.put("redeploy_count", 0)
89 
      |> Map.put("metric_sample_count", 576)
90 
      |> Map.put("timeline_receipt", reference)
91 
      |> Map.put("metrics_receipt", reference)
92 
      |> Map.put("canaries", canaries)
93 
      |> Map.put("post_soak_smoke_receipt", reference)
94 
      |> Map.put("unexplained_error_count", 0)
95 
      |> Map.put("data_loss_count", 0)
96 
      |> Map.put("authority_expansion_count", 0)
97 
      |> Map.put("fleet_divergence_count", 0)
98 
      |> Map.put("secret_leak_count", 0)
99 
      |> Map.put("unexplained_restart_count", 0)
100
101 
    decoded =
102 
      decoded
103 
      |> Map.put("synthetic", false)
104 
      |> Map.put("soak", soak)
105
106 
    File.write!(report, Jason.encode!(decoded))
107 
    File.chmod!(report, 0o600)
108
109 
    assert {_, 0} = command("finalize-report.sh", ["--final", report])
110
111 
    completed = report |> File.read!() |> Jason.decode!()
112 
    assert completed["state"] == "complete"
113 
    assert Enum.all?(completed["failure_injections"], &(&1["status"] == "passed"))
114 
    assert completed["soak"]["redeploy_count"] == 0
115
116 
    main_dir = Path.join(test_root, "main")
117 
    nested_dir = Path.join([main_dir, "evidence", "gate15"])
118 
    main_report = Path.join(main_dir, "report.json")
119 
    File.mkdir_p!(nested_dir)
120 
    File.cp!(report, Path.join(nested_dir, "report.json"))
121 
    File.cp!(Path.join(test_root, "report.sha256"), Path.join(nested_dir, "report.sha256"))
122 
    File.cp_r!(Path.join(test_root, "evidence"), Path.join(nested_dir, "evidence"))
123
124 
    assert {_, 0} = command("new-report.sh", ["--dry-run", main_report])
125 
    main = main_report |> File.read!() |> Jason.decode!()
126
127 
    resilience_reference = %{
128 
      "path" => "evidence/gate15/report.json",
129 
      "sha256" => sha256(Path.join(nested_dir, "report.json")),
130 
      "kind" => "resilience-report"
131 
    }
132
133 
    main_results =
134 
      Enum.map(main["results"], fn result ->
135 
        result
136 
        |> Map.put("status", "not_applicable")
137 
        |> Map.put("reason", "Not applicable in the synthetic nested-report proof.")
138 
      end)
139
140 
    staging_evidence = %{
141 
      "migration" => %{
142 
        "classification" => "empty_current",
143 
        "snapshot_receipt" => resilience_reference,
144 
        "rehearsal_receipt" => resilience_reference,
145 
        "migration_versions_receipt" => resilience_reference,
146 
        "rollback_compatibility_receipt" => resilience_reference
147 
      },
148 
      "configuration_readiness_receipt" => resilience_reference,
149 
      "local_gate" => %{
150 
        "default_test_count" => 1,
151 
        "cluster_test_count" => 1,
152 
        "javascript_test_count" => 1,
153 
        "coverage_summary_receipt" => resilience_reference
154 
      },
155 
      "deployment" => %{
156 
        "web_revision" => "openagents-staging-resilience-proof",
157 
        "web_image_digest" => main["candidate"]["application_manifest_digest"],
158 
        "distributed_node_release_receipt" => resilience_reference
159 
      },
160 
      "forge" => %{
161 
        "build_receipt" => resilience_reference,
162 
        "deployment_receipt" => resilience_reference,
163 
        "rollback_receipt" => resilience_reference,
164 
        "relup_receipt" => resilience_reference,
165 
        "rolling_replacement_receipt" => resilience_reference
166 
      },
167 
      "sanitized_artifacts" => [resilience_reference],
168 
      "failure_injection_timeline" => [resilience_reference],
169 
      "soak_receipt" => resilience_reference,
170 
      "known_issues" => []
171 
    }
172
173 
    main =
174 
      main
175 
      |> Map.put("synthetic", false)
176 
      |> Map.put("results", main_results)
177 
      |> Map.put("staging_evidence", staging_evidence)
178
179 
    File.write!(main_report, Jason.encode!(main))
180 
    File.chmod!(main_report, 0o600)
181
182 
    assert {_, 0} = command("finalize-report.sh", ["--final", main_report])
183 
    assert Jason.decode!(File.read!(main_report))["state"] == "complete"
184 
  end
185
186 
  defp command(script, arguments) do
187 
    System.cmd(Path.join(@scripts_root, script), arguments, stderr_to_stdout: true)
188 
  end
189
190 
  defp sha256(path) do
191 
    path |> File.read!() |> then(&:crypto.hash(:sha256, &1)) |> Base.encode16(case: :lower)
192 
  end
193 
end

This page updates live while a promote is in flight · changelog