Integrate measured blockers into the hardening plan

a82d9465c997 · Christopher David · · parent 9b29b1764e57

Integrate measured blockers into the hardening plan

Make missing JavaScript coverage, staging database isolation, migration lineage, UI policy decisions, typed configuration, and known authorization gaps explicit release gates.

Deploy story

What this commit did to the running system — joined from the forge receipt chain, the part a commit page elsewhere cannot show.

Not deployed through the forge lane

No push, promotion, build, or deploy receipt references this commit (receipts are scanned over a bounded recent window). Changes shipped by full node replacement carry their proof in the release gate receipt instead.

Changed files

  • modified docs/2026-08-20-integration-hardening-and-staging-readiness-recommendations.md

Diff

1 file changed, +136 -54

docs/2026-08-20-integration-hardening-and-staging-readiness-recommendations.md modified +136 -54

@@ -2,7 +2,7 @@

2 2
3 3
Date: 2026-08-20
4 4
5
Status: Proposed
5
Status: Proposed, amended with measured findings
6 6
7 7
## Outcome
8 8

@@ -65,11 +65,20 @@ Create a reliable starting point before changing architecture or infrastructure.

65 65
1. Record the current Git SHA and confirm the worktree is clean.
66 66
2. Run `mix precommit` on an owned test machine.
67 67
3. Run `mix test --only cluster` as a separate stage.
68
4. Run the JavaScript tests for voice state, recording, and browser hooks.
69
5. Run `mix test --cover` and merge its result with the separate cluster stage.
70
6. Record compile warnings, test exclusions, flaky tests, and test duration.
71
7. Build a release and run its startup path against a disposable database.
72
8. Save the result as a local, content-free gate receipt tied to the SHA.
68
4. Port or recreate the missing Node tests for voice state, recording, and
69
   browser hooks, and add an explicit package test command.
70
5. Run those JavaScript tests as a required stage.
71
6. Run `mix test --cover` and merge its result with the separate cluster stage.
72
7. Record compile warnings, test exclusions, flaky tests, and test duration.
73
8. Build a release and run its startup path against a disposable database.
74
9. Save the result as a local, content-free gate receipt tied to the SHA.
75
76
The baseline at `d5679e8` recorded 1,218 default tests passing with 9 cluster
77
tests excluded, all 9 cluster tests passing separately, no compile warnings, no
78
hidden skips, and 83.14% line coverage before merging cluster coverage. Keep
79
that result as historical evidence, but rerun the complete gate for each
80
candidate. Gate 0 remains blocked until the missing JavaScript suite exists and
81
runs.
73 82
74 83
Do not use the current green suite as evidence for untested code. The updated
75 84
coverage audit records strong Issues and Projects coverage and the defects it

@@ -255,11 +264,20 @@ Remove the unused parser and update every related document in the same commit.

255 264
  removal list.
256 265
- Remove DaisyUI aliases and compatibility tokens after every surface has
257 266
  migrated.
258
- Use one application icon path. Remove Heroicons and its dependency after the
259
  remaining issue and layout surfaces use vendored icons.
267
- Use one documented icon policy: Apps SDK icons are preferred, and Heroicons
268
  remains the deliberate fallback when the preferred set has no suitable
269
  glyph. Do not add a third source. Inventory fallback uses so they stay
270
  exceptional rather than becoming the default.
260 271
- Keep Basecoat imports per component and do not load Basecoat JavaScript.
261
- Rename the style pack after generic application components no longer depend on
262
  the Sarah name.
272
- Reject any component library that emits generic component selectors into the
273
  Tailwind utilities layer or otherwise outranks the application component
274
  layer. Verify cascade behavior, not only dependency names.
275
- Keep the product deliberately dark-only for staging. Hide or remove the theme
276
  toggle while it has no visual effect; treat a light palette as a separate,
277
  owner-approved design project.
278
- Treat any style-pack rename as a controlled palette migration. Preserve the
279
  primitive token contract, run visual regression checks, and do not mistake a
280
  load-bearing palette file for a cosmetic filename.
263 281
- Make the component catalog the executable inventory of supported primitives.
264 282
265 283
### Audit dependencies and licenses

@@ -272,7 +290,8 @@ Remove the unused parser and update every related document in the same commit.

272 290
- Build an SBOM for the staging image and retain it with the staging evidence.
273 291
274 292
**Exit criteria:** The application has one Markdown parser, one component
275
system, one icon path, no unexplained dependency, and complete license records.
293
system, one documented two-tier icon policy, no nonfunctional theme control, no
294
unexplained dependency, and complete license records.
276 295
277 296
## Gate 5: Make runtime configuration explicit and fail closed
278 297

@@ -293,6 +312,10 @@ Validate these groups at boot:

293 312
294 313
Apply these rules:
295 314
315
- Prefer `Application.fetch_env!/2` for required settings whose absence changes
316
  behavior. Permit a default only for a documented, sanctioned degraded mode.
317
- Validate value kinds and domains, not only presence. Repository paths are not
318
  module names, and a structurally valid allowlist can still be unusable.
296 319
- Use explicit staging values instead of relying on development defaults.
297 320
- Refuse invalid or incomplete feature combinations. For example, refuse an
298 321
  enabled recording feature without a recording key.

@@ -310,6 +333,11 @@ Add a command that prints a content-free configuration readiness report. It

310 333
should show enabled features and validation status without printing secrets,
311 334
URLs with credentials, internal node names, or tokens.
312 335
336
Add startup self-tests for behavior-changing registries and policies. At a
337
minimum, prove the tool catalog is nonempty when tools are enabled and prove
338
that configured hot-load examples classify as intended. A node that silently
339
boots without its tools or rejects every direct load is not ready.
340
313 341
**Exit criteria:** A staging release either starts with a valid, reviewed
314 342
configuration or exits before serving traffic with a redacted diagnostic.
315 343

@@ -392,6 +420,17 @@ Build on the completed controller, LiveView, and domain coverage. Add

392 420
multi-repository isolation cases without duplicating the existing mount,
393 421
interaction, and JSON-contract cases.
394 422
423
Treat these measured behaviors as known blockers, not hypothetical risks:
424
425
- `OpenAgentsWeb.ProjectController` ignores the requested username in five
426
  project actions, allowing a project to be read or changed through another
427
  user's path.
428
- `AssigneeController` reports no assignable users while issue mutations accept
429
  an arbitrary login.
430
431
Replace both behaviors with explicit repository and user authorization rules,
432
then update the tests that currently pin them as existing behavior.
433
395 434
**Exit criteria:** An owner or repository path can never read or mutate a row
396 435
owned by another repository, and PostgreSQL enforces the boundary.
397 436

@@ -425,6 +464,11 @@ owned by another repository, and PostgreSQL enforces the boundary.

425 464
426 465
### Recovery workers
427 466
467
Write a behavior specification from the durable-state and recovery invariants
468
before writing tests. There is no inherited recovery-worker suite to translate,
469
so this work must resolve intended behavior rather than guess from the current
470
implementation.
471
428 472
Add direct tests for:
429 473
430 474
- `OpenAgents.TurnRecovery` after a turn process dies mid-stream.

@@ -551,6 +595,11 @@ fallbacks before broadening the direct-load allowlist.

551 595
- Prove that a stateful process keeps its PID and data through upgrade,
552 596
  downgrade, and re-upgrade.
553 597
598
Before broadening the direct-load allowlist, prove one complete
599
push-to-build-to-canary-to-fleet-to-live path with an allowlisted module. A
600
configuration review alone cannot detect an allowlist whose values are the
601
wrong kind.
602
554 603
### Rolling replacement lane
555 604
556 605
- Build an immutable image identified by digest.

@@ -587,6 +636,11 @@ LiveView, chat, memory, provider, and voice behavior. It cannot by itself prove

587 636
three-node BEAM hot loading, stable node identity, relup installation, or
588 637
node-by-node rolling replacement.
589 638
639
The current staging service and three-node fleet share the production Cloud SQL
640
instance. Provision a staging-only database instance before claiming this gate
641
or running load, restart, connection-exhaustion, destructive, or soak tests.
642
A separate database on the same instance is not an isolation boundary.
643
590 644
Use two staging lanes until the intended fleet replaces the web-only lane:
591 645
592 646
### Web acceptance lane

@@ -628,8 +682,8 @@ as evidence for an OTP relup.

628 682
629 683
- Use a separate Google Cloud project or a strictly isolated staging boundary.
630 684
- Use staging-specific service accounts with minimum permissions.
631
- Use separate Secret Manager secrets, database, buckets, DNS records, OAuth
632
  client, and machine tokens.
685
- Use a separate database instance and role, Secret Manager secrets, buckets,
686
  DNS records, OAuth client, and machine tokens.
633 687
- Deny access to production secrets and production databases.
634 688
- Mark every staging banner, status response, log entry, and receipt as staging.
635 689
- Take a database snapshot before migration and destructive data-rights drills.

@@ -637,7 +691,8 @@ as evidence for an OTP relup.

637 691
  and test accounts after the run.
638 692
639 693
**Exit criteria:** Staging can test both the user-facing product and the complete
640
distributed deployment mechanism without touching production state.
694
distributed deployment mechanism without touching production state or sharing
695
production's database capacity and failure domain.
641 696
642 697
## Gate 13: Deploy to staging reproducibly
643 698

@@ -647,23 +702,28 @@ Use this sequence for every staging candidate:

647 702
2. Require its local gate receipt.
648 703
3. Build and retain the image, release, SBOM, build manifest, and artifact
649 704
   digests.
650
4. Restore a recent sanitized staging snapshot into a disposable rehearsal
651
   database and run every migration.
652
5. Run startup, rollback-compatible schema, and data-integrity checks against
653
   the rehearsal database.
654
6. Snapshot the actual staging database.
655
7. Deploy the candidate to the web acceptance lane with high-risk features
705
4. Classify the target as an empty current-lineage database or an existing
706
   database created by the prior migration lineage.
707
5. For an empty current-lineage database, run every current migration. For an
708
   existing prior-lineage database, first produce a schema diff and a written
709
   baseline map that identifies already-satisfied versions, required
710
   `schema_migrations` entries, and genuinely new changes. Never replay the
711
   consolidated create migrations onto existing tables.
712
6. Rehearse the selected migration path on a disposable copy and run startup,
713
   rollback-compatible schema, and data-integrity checks.
714
7. Snapshot the actual staging database.
715
8. Deploy the candidate to the web acceptance lane with high-risk features
656 716
   disabled.
657
8. Confirm migration completion, `/healthz`, `/status`, database connectivity,
717
9. Confirm migration completion, `/healthz`, `/status`, database connectivity,
658 718
   LiveView connection, and revision identity.
659
9. Hard-reload persistent browser sessions so they connect to the new revision.
660
10. Enable one gated subsystem at a time and run its regression group.
661
11. Deploy the same candidate to the distributed lane.
662
12. Run direct-load, rollback, boot-convergence, relup, and rolling-replacement
719
10. Hard-reload persistent browser sessions so they connect to the new revision.
720
11. Enable one gated subsystem at a time and run its regression group.
721
12. Deploy the same candidate to the distributed lane.
722
13. Run direct-load, rollback, boot-convergence, relup, and rolling-replacement
663 723
   drills.
664
13. Collect sanitized logs, database truth checks, receipts, screenshots, and
724
14. Collect sanitized logs, database truth checks, receipts, screenshots, and
665 725
   timing evidence.
666
14. Roll back staging if any blocking check fails.
726
15. Roll back staging if any blocking check fails.
667 727
668 728
Do not combine an application change, schema contraction, infrastructure
669 729
change, and first-time feature enablement in one staging candidate.

@@ -816,6 +876,8 @@ verification.

816 876
## Gate 15: Run failure injection and soak staging
817 877
818 878
After functional regression passes, test the system under controlled failure.
879
Do not begin this gate while staging shares a database instance, connection
880
budget, or failure domain with production.
819 881
820 882
Inject these failures one at a time:
821 883

@@ -872,6 +934,8 @@ Production remains blocked while any of these conditions is true:

872 934
- Documentation and `INVARIANTS.md` disagree with the implementation.
873 935
- A current invariant points to missing evidence.
874 936
- Any test is silently skipped or a cluster test is not run.
937
- The browser-side voice, recording, and hook suite does not exist or is not
938
  part of the owned release gate.
875 939
- A critical route lacks an explicit authority class.
876 940
- Repository data is not scoped and constrained by repository ID.
877 941
- GitHub token retention is undocumented or cannot be rotated and revoked.

@@ -879,7 +943,10 @@ Production remains blocked while any of these conditions is true:

879 943
- The forge can mark a partial fleet deployment live.
880 944
- Artifact identity, digest, rollback, boot convergence, relup, or rolling
881 945
  replacement lacks staging proof.
882
- Staging depends on production credentials or data.
946
- Staging depends on production credentials, data, database capacity, or a
947
  shared database failure domain.
948
- The migration lineage for any existing database has not been mapped and
949
  rehearsed on a disposable copy.
883 950
- A staging regression, failure-injection case, or soak issue remains
884 951
  unexplained.
885 952
- Logs contain secrets or private product content.

@@ -900,15 +967,15 @@ each handoff.

900 967
| 2 | Reconcile plans, component docs, and invariant evidence | No broken evidence links or duplicate invariant IDs |
901 968
| 3 | Rename generic Sarah infrastructure and configure `openagents.com` targets | Allowed-reference check passes |
902 969
| 4 | Centralize and validate runtime configuration | Invalid staging configurations fail before traffic |
903
| 5 | Resolve Markdown, UI, icon, dependency, and license consolidation | Parser security and component contract tests pass |
970
| 5 | Resolve Markdown, UI, icon-policy, palette, dependency, and license consolidation | Parser security, cascade, theme, and component contract tests pass |
904 971
| 6 | Resolve GitHub token policy and classify route authority | Auth, CSRF, replay, and secret-redaction tests pass |
905 972
| 7 | Add repository entities and tenant-scoped tracker data | Cross-repository isolation tests pass |
906 973
| 8 | Close chat, recovery, memory, voice, work, and machine hardening gaps | Async failure and recovery tests pass |
907 974
| 9 | Replace the forge build queue and add immutable artifact manifests | Build reproducibility and corruption tests pass |
908 975
| 10 | Add transactional fleet deployment and readiness-bound boot convergence | Three-node rollback and cold-boot tests pass |
909 976
| 11 | Complete relup, reverse-relup, and rolling replacement | Upgrade and replacement drills pass |
910
| 12 | Add the owned release gate and content-free receipts | Exact-SHA refusal and full local gate pass |
911
| 13 | Provision isolated web and distributed staging lanes | Isolation and configuration review pass |
977
| 12 | Add missing JavaScript coverage, the owned release gate, and content-free receipts | Exact-SHA refusal and full local gate pass |
978
| 13 | Provision isolated web and distributed staging lanes, including a separate database instance | Isolation and configuration review pass |
912 979
| 14 | Add staging harnesses and the evidence report template | Regression harness dry run passes |
913 980
| 15 | Deploy one staging candidate and complete the full matrix | Staging report is complete |
914 981
| 16 | Complete failure injection and the 48-hour soak | No unexplained blocking issues remain |

@@ -918,12 +985,15 @@ each handoff.

918 985
- [ ] The repository has one accurate architecture narrative.
919 986
- [ ] Every remaining Sarah reference is intentional and specific.
920 987
- [ ] All documentation links and invariant evidence resolve.
921
- [ ] The application has one Markdown parser, component system, and icon path.
988
- [ ] The application has one Markdown parser, component system, and documented
989
      two-tier icon policy.
990
- [ ] The dark-only palette has no nonfunctional theme control.
922 991
- [ ] Runtime configuration is typed, redacted, and staging-specific.
923 992
- [ ] Every route has an explicit authority class.
924 993
- [ ] GitHub token behavior matches code, UI disclosure, and data rights.
925 994
- [ ] Issues and Projects are scoped by repository in code and PostgreSQL.
926 995
- [ ] Every asynchronous recovery path has direct tests.
996
- [ ] Browser-side voice, recording, and hook tests run in the owned gate.
927 997
- [ ] Voice recording starts only after generation admission.
928 998
- [ ] Build requests are structured, unique, bounded, and non-executable.
929 999
- [ ] Artifacts are immutable, digested, manifest-checked, and durably stored.

@@ -932,6 +1002,8 @@ each handoff.

932 1002
- [ ] Relup and rolling replacement pass their staging drills.
933 1003
- [ ] Owned local gates produce exact-SHA receipts.
934 1004
- [ ] Web and distributed staging are isolated from production.
1005
- [ ] Staging has a separate database instance and failure domain.
1006
- [ ] The migration lineage is mapped and rehearsed for every nonempty target.
935 1007
- [ ] The complete regression matrix passes on one SHA.
936 1008
- [ ] Failure injection and the 48-hour soak pass.
937 1009
- [ ] The staging evidence report contains no secrets or private content.

@@ -939,18 +1011,20 @@ each handoff.

939 1011
940 1012
---
941 1013
942
# Addendum: measured state and three unnamed blockers
1014
# Addendum: measured state and blocker evidence
943 1015
944 1016
Date: 2026-08-20
945 1017
Author: the agent that ran the DaisyUI consolidation, the port-gap fan-out, and
946 1018
the coverage audit
947
Status: notes on the plan above, not a revision of it
1019
Status: accepted and incorporated into the plan above
948 1020
949
The plan above is sound and I am not proposing changes to its gate structure.
950
This addendum supplies measured values for Gate 0, records three blockers the
951
plan does not name, and flags four places where the plan and a prior owner
952
decision disagree. Everything here was verified directly at the SHA given, not
953
inferred.
1021
The gate structure remains intact. The main plan now incorporates the measured
1022
Gate 0 baseline, the missing JavaScript suite, database isolation, migration
1023
lineage, configuration and authorization evidence, and recovery-test scope.
1024
It also adopts the owner's two-tier icon decision, makes the current dark-only
1025
palette explicit, and treats the style-pack rename as a palette migration.
1026
This addendum remains as the measurement record. Everything here was verified
1027
directly at the SHA given, not inferred.
954 1028
955 1029
## A1. Gate 0 baseline, already measured
956 1030

@@ -972,13 +1046,13 @@ Two qualifiers on that coverage number, both already in the coverage audit:

972 1046
  is not a nicety; without it those two modules are misreported as untested.
973 1047
- Line coverage maps where nothing is looking. It is not a quality score.
974 1048
975
**Gate 0 cannot pass as written.** Item 4 asks to "run the JavaScript tests for
976
voice state, recording, and browser hooks." **There are no JavaScript tests in
977
this repository.** `assets/` contains no test files and `assets/package.json`
978
declares no test script. Sarah has two Node test files (~329 lines) covering
979
voice state and recording; they were not carried across in the port. That is a
980
porting gap, not a step someone forgot to run, and Gate 0 should name it as work
981
rather than as a checkbox.
1049
**Gate 0 could not pass when this was measured.** Its item 4 asked to "run the
1050
JavaScript tests for voice state, recording, and browser hooks," but **there are
1051
no JavaScript tests in this repository.** `assets/` contains no test files and
1052
`assets/package.json` declares no test script. Sarah has two Node test files
1053
(~329 lines) covering voice state and recording; they were not carried across
1054
in the port. That is a porting gap, not a step someone forgot to run. Gate 0 now
1055
names creation of the missing suite as blocking work.
982 1056
983 1057
## A2. Blocker: staging is not isolated from production today
984 1058

@@ -1012,6 +1086,7 @@ Consequences the plan should state explicitly:

1012 1086
1013 1087
This does not require solving production HA. It requires staging to get its own
1014 1088
instance before Gate 12 is claimed, and Gate 15 must not run until it does.
1089
Gates 12 and 15 now state that hold explicitly.
1015 1090
1016 1091
## A3. Blocker: openagents.com migrations cannot replay onto a Sarah database
1017 1092

@@ -1039,11 +1114,16 @@ That fix works for staging precisely because staging has no data worth keeping.

1039 1114
needs a written lineage plan first: which of this repo's migrations are already
1040 1115
satisfied by the existing schema, how `schema_migrations` gets baselined so those
1041 1116
are marked run rather than replayed, and which are genuinely new. Treat the
1042
staging crash as the cheap rehearsal it was.
1117
staging crash as the cheap rehearsal it was. Gate 13 now separates empty
1118
current-lineage targets from nonempty prior-lineage targets and requires the
1119
baseline map and rehearsal before deployment.
1043 1120
1044 1121
## A4. Four places the plan disagrees with a prior decision or current state
1045 1122
1046
These are flagged for a deliberate choice, not corrected unilaterally.
1123
These were originally flagged for deliberate choices. Gate 4 now records their
1124
disposition: retain the owner-approved Heroicons fallback, keep staging
1125
deliberately dark-only without a nonfunctional toggle, control the palette-file
1126
rename as a visual migration, and reject conflicting cascade-layer behavior.
1047 1127
1048 1128
**Heroicons.** Gate 4 says "Remove Heroicons and its dependency after the
1049 1129
remaining issue and layout surfaces use vendored icons." The owner's ruling on

@@ -1097,10 +1177,10 @@ Separately, `Tools.Registry.install!` reads

1097 1177
upstream uses `fetch_env!`. With the key unset the node booted with an **empty
1098 1178
tool catalog**: 5,874 lines of tool code compiled and unreachable, no error.
1099 1179
1100
Three of four failures were invisible. That is the argument for Gate 5, and it
1101
suggests one addition to it: **prefer `fetch_env!` to `get_env/3` for any setting
1102
whose absence changes behaviour.** A default that silently degrades is worse than
1103
a crash at boot, and `DEGRADE-001` already forbids undeclared degradation.
1180
Three of four failures were invisible. That is the argument for Gate 5. It now
1181
requires `fetch_env!` instead of `get_env/3` for settings whose absence changes
1182
behavior. A default that silently degrades is worse than a crash at boot, and
1183
`DEGRADE-001` already forbids undeclared degradation.
1104 1184
1105 1185
The same class of bug hid a fourth: `forge_hot_load_allowlist` was configured
1106 1186
with repo *paths* (`"lib/openagents"`, `"config"`, `"mix.exs"`) while

@@ -1108,7 +1188,8 @@ with repo *paths* (`"lib/openagents"`, `"config"`, `"mix.exs"`) while

1108 1188
`needs_rolling_replace` and nothing could reach `live`. **Hot loading did not
1109 1189
work at all in this repository** and no error said so. Gate 11 should require a
1110 1190
functional push→live proof, not a configuration review — a well-formed config
1111
value of the wrong *kind* passes review and fails silently.
1191
value of the wrong *kind* passes review and fails silently. Gate 11 now requires
1192
that functional proof.
1112 1193
1113 1194
## A6. Gate 7 has a concrete existing violation
1114 1195

@@ -1126,7 +1207,8 @@ is an authorization gap and not merely a GitHub-shape mismatch.

1126 1207
Also relevant to Gate 7: `AssigneeController` is a hardcoded stub whose `index`
1127 1208
always returns `%{assignees: []}` and whose `show` always 404s, while
1128 1209
`POST .../issues/:n/assignees` accepts any login. No user is ever reported
1129
assignable, yet anyone can be assigned.
1210
assignable, yet anyone can be assigned. Gate 7 now names both behaviors as
1211
blocking authorization work.
1130 1212
1131 1213
## A7. The recovery-worker gap is inherited, not created by the port
1132 1214

This page updates live while a promote is in flight · changelog