Addendum to the hardening plan: measured state and three blockers

9b29b1764e57 · AtlantisPleb · · parent 67614b5b7dc2

Addendum to the hardening plan: measured state and three blockers

The plan is sound; this adds measured values for Gate 0 and records what
direct verification turned up that the gates do not name.

Three blockers:

- Gate 0 cannot pass as written. It asks to run the JavaScript tests for
  voice state, recording, and browser hooks. There are none in this repo --
  assets/ has no test files and no test script. Sarah's two Node test files
  were not carried across. That is porting work, not a checkbox.
- Staging is not isolated from production. Verified: Cloud Run `sarah`
  (DB_NAME=sarah) and `openagents-staging` (DB_NAME=openagents_staging) both
  mount the same db-f1-micro ZONAL instance, which also serves the fleet.
  So Gate 15's failure injection and 48-hour soak would be a load test
  against production's database instance, and a zonal event takes both down
  together. Staging needs its own instance before Gate 12 is claimed.
- Migrations cannot replay onto a Sarah-created database. Enabling
  migrate-on-boot crash-looped staging with 42P07 duplicate_table on `users`
  because the two repos have different migration lineages. A fresh empty
  database fixed staging precisely because staging has no data; that does
  not generalise, and Gate 13 needs a written baselining plan first.

Also flags four disagreements with prior decisions or current state --
heroicons (Gate 4 says remove, the owner ruled keep as backup), the palette
being dark-only since DaisyUI shipped the only light theme, sarah.css now
defining the palette primitives rather than aliasing them, and why a library
emitting into @layer utilities can never coexist beneath a design system.

Adds evidence for Gate 5 from four real config failures (three silent), a
concrete Gate 7 violation (ProjectController ignores :username across five
actions), and the fact that the recovery-worker gap is inherited from Sarah
rather than created by the port -- which means there is no upstream suite to
consult and the tests must be derived from invariants.

Deploy story

What this commit did to the running system — joined from the forge receipt chain, the part a commit page elsewhere cannot show.

Not deployed through the forge lane

No push, promotion, build, or deploy receipt references this commit (receipts are scanned over a bounded recent window). Changes shipped by full node replacement carry their proof in the release gate receipt instead.

Changed files

  • modified docs/2026-08-20-integration-hardening-and-staging-readiness-recommendations.md

Diff

1 file changed, +207 -0

docs/2026-08-20-integration-hardening-and-staging-readiness-recommendations.md modified +207

@@ -936,3 +936,210 @@ each handoff.

936 936
- [ ] Failure injection and the 48-hour soak pass.
937 937
- [ ] The staging evidence report contains no secrets or private content.
938 938
- [ ] No production action has occurred.
939
940
---
941
942
# Addendum: measured state and three unnamed blockers
943
944
Date: 2026-08-20
945
Author: the agent that ran the DaisyUI consolidation, the port-gap fan-out, and
946
the coverage audit
947
Status: notes on the plan above, not a revision of it
948
949
The plan above is sound and I am not proposing changes to its gate structure.
950
This addendum supplies measured values for Gate 0, records three blockers the
951
plan does not name, and flags four places where the plan and a prior owner
952
decision disagree. Everything here was verified directly at the SHA given, not
953
inferred.
954
955
## A1. Gate 0 baseline, already measured
956
957
Gate 0 asks for a reproducible baseline. Most of it exists as of `d5679e8`:
958
959
| measurement | value |
960
| --- | --- |
961
| `mix test` | 1218 passed, 0 failed, 9 excluded |
962
| `mix test --only cluster` | 9 passed |
963
| `mix precommit` | exit 0 |
964
| `mix compile --warnings-as-errors` | exit 0, 0 warnings |
965
| `mix test --cover` total | 83.14% (415 modules) |
966
| hidden test filters | none — every `@moduletag :skip` was deleted and `:skip` removed from `test_helper.exs`; only `:cluster` is excluded |
967
968
Two qualifiers on that coverage number, both already in the coverage audit:
969
970
- `Cluster.Drain` and `Cluster.RaBootstrap` read as 0% **only** because the
971
  coverage run excludes `:cluster`. The plan's instruction to merge the two runs
972
  is not a nicety; without it those two modules are misreported as untested.
973
- Line coverage maps where nothing is looking. It is not a quality score.
974
975
**Gate 0 cannot pass as written.** Item 4 asks to "run the JavaScript tests for
976
voice state, recording, and browser hooks." **There are no JavaScript tests in
977
this repository.** `assets/` contains no test files and `assets/package.json`
978
declares no test script. Sarah has two Node test files (~329 lines) covering
979
voice state and recording; they were not carried across in the port. That is a
980
porting gap, not a step someone forgot to run, and Gate 0 should name it as work
981
rather than as a checkbox.
982
983
## A2. Blocker: staging is not isolated from production today
984
985
Gate 12 lists staging isolation as a requirement and Gate 15 calls for failure
986
injection and a 48-hour soak against staging. **Both are unsafe as currently
987
provisioned**, and the reason is not a missing control — it is the existing
988
topology:
989
990
```
991
Cloud SQL instance  openagentsgemini:us-central1:sarah-postgres
992
  tier              db-f1-micro   (shared core, ~25 max connections)
993
  availability      ZONAL         (no HA)
994
  databases         sarah              <- PRODUCTION
995
                    sarah_staging
996
                    openagents_staging <- STAGING
997
```
998
999
Verified: Cloud Run service `sarah` (`DB_NAME=sarah`, `POOL_SIZE=5`) and service
1000
`openagents-staging` (`DB_NAME=openagents_staging`, `POOL_SIZE=5`) both mount
1001
`/cloudsql/openagentsgemini:us-central1:sarah-postgres`. The 3-node fleet
1002
connects to the same instance as well.
1003
1004
Consequences the plan should state explicitly:
1005
1006
- Staging and production share one shared-core instance with a small connection
1007
  budget. Staging load consumes production's connection headroom.
1008
- Running Gate 15 failure injection or a 48-hour soak against staging is
1009
  therefore **a load test against production's database instance.**
1010
- A single-zone instance means one zonal event takes staging and production
1011
  together, so staging cannot serve as evidence of production resilience.
1012
1013
This does not require solving production HA. It requires staging to get its own
1014
instance before Gate 12 is claimed, and Gate 15 must not run until it does.
1015
1016
## A3. Blocker: openagents.com migrations cannot replay onto a Sarah database
1017
1018
The ground rule "preserve historical migrations, correct live schema problems
1019
with new migrations" is right, but there is a specific landmine underneath it
1020
that Gate 13 (deploy to staging reproducibly) will hit again on any real data.
1021
1022
On 2026-08-19, migrate-on-boot was enabled and immediately crash-looped the
1023
staging container:
1024
1025
```
1026
== Running 20260816214000 OpenAgents.Repo.Migrations.CreateUsers.change/0 forward
1027
   create table users
1028
** (Postgrex.Error) ERROR 42P07 (duplicate_table) relation "users" already exists
1029
```
1030
1031
The service pointed at `sarah_staging`, which already held Sarah's tables, while
1032
openagents.com's `schema_migrations` had no record of its own versions — its
1033
migration history is a **different lineage** (Sarah's 57 files versus this
1034
repo's, with 18 of Sarah's consolidated into one). Staging was unblocked by
1035
pointing it at a brand-new empty `openagents_staging` database.
1036
1037
That fix works for staging precisely because staging has no data worth keeping.
1038
**It does not generalise.** Any future cutover onto a database Sarah created
1039
needs a written lineage plan first: which of this repo's migrations are already
1040
satisfied by the existing schema, how `schema_migrations` gets baselined so those
1041
are marked run rather than replayed, and which are genuinely new. Treat the
1042
staging crash as the cheap rehearsal it was.
1043
1044
## A4. Four places the plan disagrees with a prior decision or current state
1045
1046
These are flagged for a deliberate choice, not corrected unilaterally.
1047
1048
**Heroicons.** Gate 4 says "Remove Heroicons and its dependency after the
1049
remaining issue and layout surfaces use vendored icons." The owner's ruling on
1050
2026-08-19 was Apps SDK icons **preferred, heroicons as backup** — i.e. retained
1051
deliberately. `{:heroicons, ...}` is still in `mix.exs`. One of the two should
1052
move; the plan should not quietly overwrite a stated decision.
1053
1054
**The palette is dark-only, today.** Gate 4's UI consolidation does not mention
1055
it, but removing DaisyUI removed the only light theme in the application. Sarah's
1056
palette has no light variant. `data-theme` still runs and the toggle still
1057
switches, but it no longer repaints anything. This is a live user-visible state,
1058
not a pending task, and `theme_toggle/1`'s docstring now says so. Inventing a
1059
light Sarah palette is a design decision that needs an owner.
1060
1061
**Basecoat is load-bearing for the palette, not just components.** Gate 4 says to
1062
"rename the style pack after generic application components no longer depend on
1063
the Sarah name." Worth knowing before that rename: `sarah.css` now defines the
1064
palette *primitives* (`--accent`, `--ink-void`, `--text-primary`, `--line-strong`,
1065
…), not merely aliases. Prior to the DaisyUI removal it only aliased them and
1066
nothing defined them — the entire Sarah palette silently resolved to nothing and
1067
DaisyUI's theme was carrying every colour. Renaming that file is therefore a
1068
palette migration, not a cosmetic rename.
1069
1070
**Why DaisyUI had to be removed rather than layered under.** Recording this so a
1071
similar library is not reintroduced under a different name: DaisyUI emits
1072
component styles — a flat `.btn` setting `background-color`, `color`, and
1073
`border-color` — into `@layer utilities`. That layer must remain last or every
1074
Tailwind utility breaks. A later-declared layer beats an earlier one *regardless
1075
of specificity*, so DaisyUI's `.btn` outranked every `.btn[data-variant=…]` rule
1076
in `sarah.css` and all eight button variants rendered identically on staging.
1077
There is no layer ordering that fixes this. **Any component library that emits
1078
into `utilities` cannot coexist beneath a design system** — that is the
1079
acceptance test for Gate 4's "one component system", not merely counting the
1080
libraries in `mix.exs`.
1081
1082
## A5. Evidence for Gate 5, from failures already observed
1083
1084
Gate 5's "fail closed" requirement is correct and this is what its absence
1085
actually looked like in this codebase. Four config reads survived the
1086
re-namespacing and kept reading the `:sarah` OTP app:
1087
1088
| module | effect |
1089
| --- | --- |
1090
| `inference_proxy_controller.ex:104` | raised — the only one that was visible |
1091
| `plugs/forge_git_auth.ex:52` | **silent** — operator token always `nil`, every forge push 401'd |
1092
| `tools/selector.ex:153` | **silent** — returned config default |
1093
| `tools/embeddings.ex:127` | **silent** — returned config default |
1094
1095
Separately, `Tools.Registry.install!` reads
1096
`Application.get_env(:openagents, :tools, [])` — note the `[]` default where the
1097
upstream uses `fetch_env!`. With the key unset the node booted with an **empty
1098
tool catalog**: 5,874 lines of tool code compiled and unreachable, no error.
1099
1100
Three of four failures were invisible. That is the argument for Gate 5, and it
1101
suggests one addition to it: **prefer `fetch_env!` to `get_env/3` for any setting
1102
whose absence changes behaviour.** A default that silently degrades is worse than
1103
a crash at boot, and `DEGRADE-001` already forbids undeclared degradation.
1104
1105
The same class of bug hid a fourth: `forge_hot_load_allowlist` was configured
1106
with repo *paths* (`"lib/openagents"`, `"config"`, `"mix.exs"`) while
1107
`HotLoader.allowlisted?/2` matches *module names*. Every hot-load was refused as
1108
`needs_rolling_replace` and nothing could reach `live`. **Hot loading did not
1109
work at all in this repository** and no error said so. Gate 11 should require a
1110
functional push→live proof, not a configuration review — a well-formed config
1111
value of the wrong *kind* passes review and fails silently.
1112
1113
## A6. Gate 7 has a concrete existing violation
1114
1115
Gate 7 asks for repository and tenant scoping. One instance is already known and
1116
is pinned in tests as current behaviour rather than fixed, because changing
1117
authorization semantics needs an owner:
1118
1119
`OpenAgentsWeb.ProjectController` destructures `"username" => _username` in
1120
`show`, `items`, `create_item`, `update_item`, and `fields` — all five ignore it.
1121
Only `index` filters by owner. **A project owned by `alice` is readable and
1122
writable at `/users/bob/projectsV2/:n`.** It is consistent across all five
1123
actions, so it reads as deliberate simplification rather than oversight, but it
1124
is an authorization gap and not merely a GitHub-shape mismatch.
1125
1126
Also relevant to Gate 7: `AssigneeController` is a hardcoded stub whose `index`
1127
always returns `%{assignees: []}` and whose `show` always 404s, while
1128
`POST .../issues/:n/assignees` accepts any login. No user is ever reported
1129
assignable, yet anyone can be assigned.
1130
1131
## A7. The recovery-worker gap is inherited, not created by the port
1132
1133
Gate 8 correctly requires direct tests for `TurnRecovery`, `VoiceRecovery`,
1134
`WorkRecovery`, and `Memory.SemanticWorker`, and the production hold conditions
1135
correctly block on it. One fact changes who can answer the design questions:
1136
1137
**Sarah has no tests for any of those four modules either.** Verified directly.
1138
The port faithfully carried across a gap that already existed upstream.
1139
1140
Two implications. First, this is a longer-standing risk than the Issues and
1141
Projects gap was, not a lesser one — it predates the merge. Second, there is no
1142
upstream test suite to port or consult, so writing these means deriving intended
1143
behaviour from the invariants (`TURN-005`, `WORK-001`, and `VOICE-009` all
1144
describe recovery behaviour that should be assertable) rather than translating
1145
existing assertions. Budget accordingly; this is design work, not porting.

This page updates live while a promote is in flight · changelog