After failure injection, run a staging soak:

- Keep the web and distributed staging lanes active for at least 48 hours.

- Keep the pinned web and distributed release-candidate lanes active for at

  least 15 minutes. Deploy later commits to a separate staging service.

- Exercise scheduled typed, memory, voice, tracker, Git, and status canaries.

- Watch database connections, queue depth, mailbox growth, process count,

  memory, CPU, restart count, artifact cache, Ra state, and node convergence.

  and resilience reports while retaining failed attempts and refusing

  not-applicable controlled failures.

- Added a strict resilience validator. Completion requires all controlled

  failures passed, at least 172,800 measured seconds without a redeploy, stable

  candidate identity, 576 metric samples, the versioned status, typed, memory,

  failures passed, at least 900 measured seconds on the pinned candidate track,

  stable candidate identity, 15 metric samples, the versioned status, typed, memory,

  tracker, Git, and voice canary minimums, a post-soak smoke, and zero

  unexplained harm counts.

- Bound the main staging report's final state to one nested, complete resilience

  evidence tree is checksum-, path-, permission-, and content-scan validated.

- Added the [controlled-failure and soak runbook](operations/staging-resilience.md)

  and [resilience evidence contract](operations/staging-resilience-report-template.md).

  The network-free dry run proves the 11-case and 48-hour contracts fail closed.

  The network-free dry run proves the 11-case and 15-minute contracts fail closed.

The local Gate 15 harness and contract tests pass. Live controlled failures and

the measured 48-hour soak remain blocked until Gates 12–14 complete on the

the measured 15-minute soak remain blocked until Gates 12–14 complete on the

isolated staging target. No cloud or production resource changed while adding

the harness.

| 13 | Provision isolated web and distributed staging lanes, including a separate database instance | Isolation and configuration review pass |

| 14 | Add staging harnesses and the evidence report template | Regression harness dry run passes |

| 15 | Deploy one staging candidate and complete the full matrix | Staging report is complete |

| 16 | Complete failure injection and the 48-hour soak | No unexplained blocking issues remain |

| 16 | Complete failure injection and the 15-minute soak | No unexplained blocking issues remain |

## Final staging readiness checklist

- [x] Owned local gates produce exact-SHA receipts.

- [x] The versioned staging matrix, private evidence report, scanner, recorder,

      validator, and network-free harness dry run exist.

- [x] The versioned controlled-failure matrix and fail-closed 48-hour soak

- [x] The versioned controlled-failure matrix and fail-closed 15-minute soak

      evidence contract exist.

- [ ] Web and distributed staging are isolated from production.

- [ ] Staging has a separate database instance and failure domain.

- [ ] The migration lineage is mapped and rehearsed for every nonempty target.

- [ ] The complete regression matrix passes on one SHA.

- [ ] Failure injection and the 48-hour soak pass.

- [ ] Failure injection and the 15-minute soak pass.

- [ ] The staging evidence report contains no secrets or private content.

- [ ] No production action has occurred.

## A2. Blocker: staging is not isolated from production today

Gate 12 lists staging isolation as a requirement and Gate 15 calls for failure

injection and a 48-hour soak against staging. **Both are unsafe as currently

injection and a 15-minute soak against staging. **Both are unsafe as currently

provisioned**, and the reason is not a missing control — it is the existing

topology:

- Staging and production share one shared-core instance with a small connection

  budget. Staging load consumes production's connection headroom.

- Running Gate 15 failure injection or a 48-hour soak against staging is

- Running Gate 15 failure injection or a 15-minute soak against staging is

  therefore **a load test against production's database instance.**

- A single-zone instance means one zonal event takes staging and production

  together, so staging cannot serve as evidence of production resilience.

Both lanes use staging-only hosts, credentials, buckets, repositories, service

accounts, and a PostgreSQL instance that does not share a production failure

domain. Production is out of scope until all hardening gates, the complete

staging matrix, failure injection, and the 48-hour soak pass.

staging matrix, failure injection, and the 15-minute pinned-candidate soak pass.

`OpenAgents.RuntimeConfig` validates the complete behavior-changing settings

boundary before migrations or traffic. The

Give both lanes staging-only hosts, secrets, service accounts, buckets,

repositories, and a PostgreSQL instance that is separate from production.

Disable production promotion until both lanes pass the complete matrix,

failure injection, and a 48-hour soak on one exact candidate.

failure injection, and a 15-minute soak on one pinned exact candidate.

## Consequences

  bucket, or secret.

- The candidate passes all Gate 14 cases on staging.

- The same candidate passes controlled failure injection and a measured

  48-hour soak without a redeploy.

  15-minute soak on the pinned `openagents-staging-release` service. Later

  commits can deploy to another staging service without replacing this release

  candidate or resetting its clock.

- A fresh production backup restores into a disposable Cloud SQL instance.

  The migration-lineage bridge, all remaining migrations, candidate startup,

  last-known-good startup, row-count checks, and integrity checks pass there.

The resilience report proves:

- All 11 controlled-failure cases and their retry histories.

- At least 48 hours without a redeploy.

- At least 15 minutes on the pinned release-candidate track.

- Scheduled typed, memory, voice, tracker, Git, and status canaries.

- Five-minute resource samples and a post-soak smoke.

- One-minute resource samples and a post-soak smoke.

- Every known issue with ID, owner, severity, and disposition.

An empty known-issue list is valid only when review found no issues. A failure,

| `draft` | `--draft` | Work is in progress; pending and failed cases may exist |

| `recorded` | `--recorded` | Every case has an outcome; failures or blocks may remain |

| `regression_passed` | `--regression` | Every applicable Gate 14 case passed and common evidence is complete |

| `complete` | `--final` | Gate 14, failure injection, and the 48-hour soak all passed |

| `complete` | `--final` | Gate 14, failure injection, and the 15-minute soak all passed |

Only `finalize-report.sh` should change the state and completion timestamp. It

validates a temporary copy before replacing the report and recomputes

`new-resilience-report.sh` generates one report for the candidate that already

passed Gate 14. The report binds the candidate manifest, application image,

release, and Gate 14 report checksum to 11 controlled-failure results and one

48-hour soak.

15-minute soak on the pinned release-candidate track.

## Failure results

Completion requires:

- Start and completion UTC timestamps separated by at least 172,800 seconds.

- `candidate_identity_stable` set to true and `redeploy_count` set to zero.

- At least 576 five-minute metric samples and a sanitized metric receipt.

- Start and completion UTC timestamps separated by at least 900 seconds.

- A target track of `release-candidate` and service of

  `openagents-staging-release`.

- `candidate_identity_stable` set to true and `redeploy_count` set to zero for

  that pinned service. Deployments to other staging services do not count.

- At least 15 one-minute metric samples and a sanitized metric receipt.

- A sanitized soak timeline receipt.

- Exact canary IDs, cadences, minimums, completed counts, passed counts, and

  aggregate receipts.

- A full post-soak smoke receipt from the same candidate without a redeploy.

- A full post-soak smoke receipt from the same pinned candidate.

- Zero unexplained errors, data loss, authority expansion, fleet divergence,

  secret leakage, and unexplained restarts.

The canary minimums are 576 status, 96 typed, 96 memory, 96 tracker, 96 Git, and

24 fake-media voice passes.

The canary minimums are 15 status, 3 typed, 3 memory, 3 tracker, 3 Git, and 1

fake-media voice pass.

## Known issues

# Run controlled failures and the 48-hour staging soak

# Run controlled failures and the 15-minute staging soak

Date: 2026-08-20

Status: Harness implemented and locally proven; live Gate 15 execution pending

Use this runbook after one exact candidate has passed Gate 14. It records all

controlled-failure attempts, requires 48 measured hours without a redeploy,

controlled-failure attempts, requires 15 measured minutes on a pinned release

track,

enforces scheduled canary and metric minimums, and refuses completion when any

unexplained harm remains.

Expected result:

```text

Staging resilience harness dry run passed (11 failures; 48-hour soak; no network requests sent).

Staging resilience harness dry run passed (11 failures; 15-minute soak; no network requests sent).

```

The check validates the versioned matrix, report generator, evidence scanner,

attempt recorder, final validator, exact 48-hour duration, six canary schedules,

attempt recorder, final validator, exact 15-minute duration, six canary schedules,

and fail-closed finalization.

## Create the resilience report

Corrective attempts return it to draft. Do not start the soak until all 11 last

attempts pass, the candidate is healthy, and the test data is reconciled.

## Run 48 hours without a redeploy

## Run 15 minutes on the pinned release track

Record a UTC start time after the final failure-recovery smoke. Keep the web and

distributed staging lanes on the exact candidate for at least 172,800 seconds.

Any application, configuration, image, release, migration, or fleet redeploy

invalidates the soak; set `redeploy_count` honestly and start a new 48-hour run.

distributed release-candidate lanes on the exact candidate for at least 900

seconds. Deploy later development commits to the ordinary staging service. Do

not replace or reconfigure `openagents-staging-release` while its candidate is

soaking.

The report's `redeploy_count` applies only to the pinned release-candidate

service and distributed lane. Deployments to another staging service do not

reset the clock. A deployment that changes the candidate track still fails the

soak because the report must prove one stable SHA and image digest.

Use these minimum schedules:

| Canary or sample | Cadence | Minimum over 48 hours |

| Canary or sample | Cadence | Minimum over 15 minutes |

| --- | ---: | ---: |

| Status and candidate identity | 5 minutes | 576 passes |

| Resource metrics | 5 minutes | 576 complete samples |

| Typed chat | 30 minutes | 96 passes |

| Memory write/read/forget | 30 minutes | 96 passes |

| Tracker read/write/cleanup | 30 minutes | 96 passes |

| Git clone/fetch/push/cleanup | 30 minutes | 96 passes |

| Fake-media voice lifecycle | 2 hours | 24 passes |

| Status and candidate identity | 1 minute | 15 passes |

| Resource metrics | 1 minute | 15 complete samples |

| Typed chat | 5 minutes | 3 passes |

| Memory write/read/forget | 5 minutes | 3 passes |

| Tracker read/write/cleanup | 5 minutes | 3 passes |

| Git clone/fetch/push/cleanup | 5 minutes | 3 passes |

| Fake-media voice lifecycle | 15 minutes | 1 pass |

Each canary receipt must bind the candidate SHA, image digest, scheduled and

actual UTC time, bounded outcome, attempt count, and cleanup result. A retry does

known issue must be resolved. A low or medium issue can be accepted only with a

named owner and an explicit non-blocking disposition.

After 48 hours, run the full Gate 14 public and authenticated smoke again without

redeploying. Record the exact end time after that smoke.

After 15 minutes, run the full Gate 14 public and authenticated smoke again

against the pinned candidate. Record the exact end time after that smoke.

## Complete and validate the resilience report

booleans described in the

[resilience evidence contract](staging-resilience-report-template.md).

Finalization requires all failure cases passed, at least 48 measured hours,

zero redeploys, stable candidate identity, every canary minimum, at least 576

Finalization requires all failure cases passed, at least 15 measured minutes,

zero candidate-track redeploys, stable candidate identity, every canary minimum, at least 15

metric samples, and zero unexplained error, data-loss, authority-expansion,

fleet-divergence, secret-leak, or restart counts:

ready. Documentation reconciliation, dependency and component consolidation,

route authority, token lifecycle, tenant scoping, recovery, build isolation,

transactional fleet deployment, staging isolation, failure injection, and the

48-hour soak remain governed by the current hardening plan.

15-minute pinned-candidate soak remain governed by the current hardening plan.

      target: {

        environment: "staging",

        base_url: "https://staging.openagents.com",

        track: "release-candidate",

        service: "openagents-staging-release",

        project: $candidate.target.project,

        region: $candidate.target.region

      },

{

  "schema": "openagents.staging-resilience-matrix.v1",

  "revision": 1,

  "revision": 2,

  "failure_injections": [

    {"id": "failure-001", "title": "Provider timeout, malformed event, and premature stream closure recover without a false completion"},

    {"id": "failure-002", "title": "PostgreSQL restart and temporary connection exhaustion recover without data loss"},

    {"id": "failure-011", "title": "Recording upload failure and late final chunk remain generation-fenced and recoverable"}

  ],

  "soak": {

    "required_duration_seconds": 172800,

    "metric_sample_cadence_seconds": 300,

    "minimum_metric_samples": 576,

    "required_duration_seconds": 900,

    "metric_sample_cadence_seconds": 60,

    "minimum_metric_samples": 15,

    "canaries": [

      {"id": "status", "cadence_seconds": 300, "minimum_passes": 576},

      {"id": "typed", "cadence_seconds": 1800, "minimum_passes": 96},

      {"id": "memory", "cadence_seconds": 1800, "minimum_passes": 96},

      {"id": "tracker", "cadence_seconds": 1800, "minimum_passes": 96},

      {"id": "git", "cadence_seconds": 1800, "minimum_passes": 96},

      {"id": "voice", "cadence_seconds": 7200, "minimum_passes": 24}

      {"id": "status", "cadence_seconds": 60, "minimum_passes": 15},

      {"id": "typed", "cadence_seconds": 300, "minimum_passes": 3},

      {"id": "memory", "cadence_seconds": 300, "minimum_passes": 3},

      {"id": "tracker", "cadence_seconds": 300, "minimum_passes": 3},

      {"id": "git", "cadence_seconds": 300, "minimum_passes": 3},

      {"id": "voice", "cadence_seconds": 900, "minimum_passes": 1}

    ]

  }

}

jq -e '

  .schema == "openagents.staging-resilience-matrix.v1" and

  .revision == 1 and

  .revision == 2 and

  (.failure_injections | type == "array" and length == 11) and

  ([.failure_injections[].id] | length == (unique | length)) and

  all(.failure_injections[];

    (.id | test("^failure-[0-9]{3}$")) and

    (.title | type == "string" and length > 0)) and

  .soak.required_duration_seconds == 172800 and

  .soak.metric_sample_cadence_seconds == 300 and

  .soak.minimum_metric_samples == 576 and

  .soak.required_duration_seconds == 900 and

  .soak.metric_sample_cadence_seconds == 60 and

  .soak.minimum_metric_samples == 15 and

  (.soak.canaries | type == "array" and length == 6) and

  ([.soak.canaries[].id] | sort) == ["git", "memory", "status", "tracker", "typed", "voice"] and

  all(.soak.canaries[]; . as $canary |

    ($canary.cadence_seconds | type == "number" and . > 0 and floor == .) and

    ($canary.minimum_passes | type == "number" and . >= (172800 / $canary.cadence_seconds) and floor == .))

    ($canary.minimum_passes | type == "number" and . >= (900 / $canary.cadence_seconds) and floor == .))

' "$matrix" >/dev/null || {

  echo "staging resilience matrix contract failed" >&2

  exit 1

"$script_dir/record-result.sh" "$report" failure-001 passed dry-run "$evidence" >/dev/null

"$script_dir/validate-resilience-report.sh" --draft "$report" >/dev/null

echo "Staging resilience harness dry run passed (11 failures; 48-hour soak; no network requests sent)."

echo "Staging resilience harness dry run passed (11 failures; 15-minute soak; no network requests sent)."

  (.completed_at == null or (.completed_at | timestamp)) and

  .target.environment == "staging" and

  .target.base_url == "https://staging.openagents.com" and

  .target.track == "release-candidate" and

  .target.service == "openagents-staging-release" and

  (.target.project | type == "string" and test("stag"; "i")) and

  (.target.region | nonempty) and

  (.candidate.git_sha | type == "string" and test("^[0-9a-f]{40}$")) and

    assert Enum.uniq_by(matrix["failure_injections"], & &1["id"]) ==

             matrix["failure_injections"]

    assert matrix["soak"]["required_duration_seconds"] == 48 * 60 * 60

    assert matrix["soak"]["minimum_metric_samples"] == 576

    assert matrix["revision"] == 2

    assert matrix["soak"]["required_duration_seconds"] == 15 * 60

    assert matrix["soak"]["minimum_metric_samples"] == 15

    assert matrix["soak"]["canaries"] |> Enum.map(& &1["id"]) |> Enum.sort() ==

             ~w(git memory status tracker typed voice)

    report =

      Path.join(

        System.tmp_dir!(),

        "openagents-resilience-target-#{System.unique_integer([:positive])}.json"

      )

    on_exit(fn -> File.rm(report) end)

    assert {_, 0} = command("new-resilience-report.sh", ["--dry-run", report])

    target = report |> File.read!() |> Jason.decode!() |> Map.fetch!("target")

    assert target["track"] == "release-candidate"

    assert target["service"] == "openagents-staging-release"

  end

  test "the resilience dry run is local and fail-closed" do

    {output, 0} = command("resilience.sh", ["check"])

    assert output =~ "11 failures"

    assert output =~ "48-hour soak"

    assert output =~ "15-minute soak"

    assert output =~ "no network requests sent"

  end

    assert validator =~ "report_dir=$(realpath"

  end

  test "finalization requires every recovery, 48 measured hours, canaries, and zero unexplained harm",

  test "finalization requires every recovery, 15 measured minutes, canaries, and zero unexplained harm",

       %{test_root: test_root} do

    report = Path.join(test_root, "report.json")

    evidence = Path.join(test_root, "recovery.json")

    decoded = report |> File.read!() |> Jason.decode!()

    [reference | _] = hd(decoded["failure_injections"])["evidence"]

    completed_at = DateTime.utc_now() |> DateTime.add(-1, :second) |> DateTime.truncate(:second)

    started_at = DateTime.add(completed_at, -(48 * 60 * 60), :second)

    started_at = DateTime.add(completed_at, -(15 * 60), :second)

    soak = decoded["soak"]

      |> Map.put("completed_at", DateTime.to_iso8601(completed_at))

      |> Map.put("candidate_identity_stable", true)

      |> Map.put("redeploy_count", 0)

      |> Map.put("metric_sample_count", 576)

      |> Map.put("metric_sample_count", 15)

      |> Map.put("timeline_receipt", reference)

      |> Map.put("metrics_receipt", reference)

      |> Map.put("canaries", canaries)