Invariant proof audit: the second pass, at the test files
docs/2026-08-23-invariant-proof-audit.md classified every contract in
INVARIANTS.md as specific or universal and repaired nineteen proofs that
could not fail for their claims. It also recorded its own limit: most verdicts
"were judged from the mechanism the contract names rather than re-derived. A
second pass over those would be worth doing and is not what this one did."
This document is that second pass. Every contract's cited proof was read at the file — the test, the script, the migration, and the production module it claims to cover — and checked against issue #166's mechanical test: name a violation the proof would not catch. Line numbers refer to the working tree on 2026-08-24.
The first pass asked whether a universal claim's population is closed. This pass asks a different question about the same proofs: when the claim is violated, does the named proof actually go red? Five defect classes recur:
- Fixture-path assertions. The test asserts on a fixture, a hand-built map, or a JSON file's own fields rather than the production path.
- Subset coverage. The proof examines some members of the claimed population — two adapters of four, one extension object of four, one route of a class.
- Shared-code tautologies. The expected value is computed by the code under test, so a bug moves both sides together and the assertion holds.
- Vacuous assertions. The assertion is true by construction — an element ID that exists nowhere, a literal read back from the module that hard-codes it, an immutability property every BEAM value has.
- Empty-input passes. An enumeration or scan that matched nothing passes as if it had checked everything, with no positive control forcing it to find what it should find.
Result
130 contracts, one verdict each.
| Verdict | Count |
|---|---|
| Can fail — the proof goes red when the claim is violated | 61 |
| Partial — the core clause bites, but named clauses have no proof that can | 57 |
| Cannot fail — a load-bearing clause has no proof that can go red | 12 |
One proof is red right now, and that is the mechanism working: see the next
section. The verdicts here are about the cited proofs as they exist; "partial"
is not a defect report against the code, and several "cannot fail" entries
cover code that is currently correct. The finding in each case is that nothing
executable would notice when it stops being correct — which is the exact
condition INVARIANTS.md lines 13-25 names as the companion to taxonomy
rule 7.
The proof that is failing right now
ADMIN-001's enumerating proof, test/openagents_web/operator_surface_test.exs,
is statically red at HEAD. Commit 310dac1 ("Open the Gym", an ancestor of
HEAD) added:
/gym, classified:operatorwith scopegym:readatlib/openagents_web/route_authority.exlines 192-193, absent from the declared operator-route table attest/openagents_web/operator_surface_test.exslines 33-72;OpenAgentsWeb.GymLive(lines 21 and 30) andOpenAgentsWeb.GymRunController(line 46), both callingOpenAgents.Accounts.admin?/1, absent from the declared authority-module table at lines 76-104.
ADMIN-001 does not name the Gym anywhere. The proof is doing exactly what
#146 built it to do — a new operator surface fails until the contract is
amended to name it — and the failure reached main anyway. That is a process
finding, not a proof finding: the commit's own message reports a scoped test
run, and the section below on the gate shows how a candidate can carry a
passed receipt without this file ever running. The fix is two-sided: amend
ADMIN-001 to name the Gym read surface, and close the gate path that let a
red enumeration land.
Proofs that cannot fail for a load-bearing clause
Each entry names the claim, the cited proof, the violation the proof cannot catch, and the smallest change that would let it fail.
MEMORY-005 — the tests assert the violating behavior
Claim: memory_remember.v1 accepts only a claim directly authorized by
the current user message, a host-recorded confirmation, or a first-party UI
action; "model arguments cannot substitute."
Reality: lib/openagents/tools/memory_remember.ex lines 130-135 swallow
every OpenAgents.Memory.Consent failure and proceed with
%{kind: "conversation_context"} — the model's arguments are the write
authority. The cited proof encodes this as correct:
test/openagents/tools/profile_memory_tools_test.exs lines 240-262 store a
model-paraphrased claim under consent_kind == "conversation_context", and
lines 292-307 store a claim that differs from the exact confirmation on
record ("I prefer concise answers" confirmed, "I prefer detailed answers"
stored) and assert success. The Consent unit tests (lines 22-119) prove the
strict module in isolation; no test runs the tool with a fabricated claim and
asserts refusal.
Smallest change: delete the {:error, _} -> %{kind: "conversation_context"}
fallback so the tool returns the consent error, or amend MEMORY-005 to say
what the code does — a conversation-context write authorized by topical
relevance, not by exact consent. Either way the current sentence and the
current code cannot both stand.
RELEASE-003 — the cited proof tests dead code
Claim: invalid, insecure, or path-bearing origins fail startup.
Reality: OpenAgentsWeb.AllowedOrigins has no production caller — the
only references in the repository are its own file and its test. Production
origin checking is config/runtime.exs line 190 feeding check_origin at
line 545, validated by lib/openagents/runtime_config.ex lines 823-840.
test/openagents_web/allowed_origins_test.exs therefore tests a module you
can delete without changing production behavior, and ops/ci/release-smoke.sh
never opens a WebSocket or sends an Origin header (lines 99-114 curl
/health over plain loopback HTTP), so the "production WebSocket read-back"
in the evidence line has no executable artifact.
Smallest change: either route config/runtime.exs through
OpenAgentsWeb.AllowedOrigins.for_production/2 so the tested module is the
production path, or retarget the proof at OpenAgents.RuntimeConfig's origin
validation and delete the dead module.
VOICE-005 — the disclosure assertion checks an element that exists nowhere
Claim: while recording is on, the surface that carries START VOICE
states before the microphone opens that calls are recorded; a visible marker
announces capture while running.
Reality: test/openagents_web/live/chat_live_test.exs line 206 asserts
refute has_element?(view, "#voice-recording-disclosure") — an ID that
appears nowhere else in the repository, so the assertion cannot fail under any
change. The positive disclosure is proven only on the memory surface
(test/openagents_web/controllers/data_controller_test.exs lines 243-247
against lib/openagents_web/live/memory_live.ex), while the surface that
carries START VOICE (lib/openagents_web/live/chat_live.ex line 2069)
renders no disclosure text. The capture marker at line 2063 is
visually-hidden and referenced by no test. The teardown half is also
unclosed: assets/test/voice_state_test.mjs lines 55-71 call the teardown
function directly with fakes; nothing proves destroyed() or reconnect
actually calls it (assets/js/voice_controller.js line 501 is the one call
site).
Smallest change: assert the disclosure element that the chat surface actually renders — which first requires the chat surface to render one — and delete the impossible-ID refutation.
MODULE-004 — the proof is a JSON fixture identity check
Claim: every capability surface preserves the same authority boundary; external effects need receipts; oversized catalogs degrade to discovery; missing executors fail honestly.
Reality: test/openagents/surface_eval_test.exs is 28 lines. Lines 13-18
assert that the case IDs in priv/sarah/evals/surfaces/identity-authority.v1.json
equal a hard-coded list. Three of the four corpus cases — external effect
under read authority, oversized catalog, unavailable executor — declare
expectations that nothing executes. The file never touches
OpenAgents.Modules.SurfacePolicy, OpenAgents.Tools.Registry, or
OpenAgents.Tools.Runner — the three modules the invariant names as
evidence. Renaming a corpus case turns the test red; an external effect
executing under read-only authority does not.
Smallest change: drive the three unexecuted corpus cases through
OpenAgents.Tools.Runner, or narrow MODULE-004's proof index row to the
surface-vocabulary and revalidation tests in
test/openagents/modules/router_test.exs (lines 107-134), which do bite.
COLLECTIVE-002 — the leak scanner has zero coverage
Claim: generalizer output "is scanned again for secrets, contacts, identifiers, paths, exact source fragments, authority-bearing fields, and size before storage."
Reality: the scanner exists (lib/openagents/collective/generalizer.ex
lines 267-299, five reject branches including the four-gram quote detector)
and no test reaches any reject branch — a grep for the rejection reasons
across test/ returns nothing. The leak assertions in
test/openagents/collective_generalizer_test.exs lines 38-48 run against a
payload built from fixed-vocabulary maps that no input text ever flows into,
so they are true by construction.
Smallest change: construct a payload carrying a source fragment and
assert validate_payload/2 rejects it — one test per reject branch.
IDENTITY-006 — the branch discipline is enforced but never tested
Claim: Git receive-pack authorizes every requested ref update; the assignment credential can update only its assignment branch, never a default or protected branch.
Reality: nothing in test/ exercises authorize_receive_pack/3 or
allowed_assignment_ref?/3 (lib/openagents/forge/git_http.ex lines
303-329). Deleting the default-branch and protected-branch checks fails no
test. test/openagents/forge/assignment_test.exs lines 133-147 test pkt-line
parsing — the input to the decision, not the decision. The non-Git-API
refusal in the auth plug is likewise untested, and the "stores only a digest"
assertion at lines 96-97 is tautological (the schema has no plaintext field
to leak).
Smallest change: a receive-pack push with an assignment credential
against the default branch, over the same real-HTTP harness
test/openagents/forge/push_closes_issues_test.exs already stands up,
asserting refusal — plus one against the assignment branch asserting success.
DEPLOYPLANE-004 — "the only definition" has no only-ness proof
Claim: OpenAgents.Deployments.Lifecycle is the only definition of legal
states and transitions, enforced transactionally.
Reality: the transitions themselves are well proven
(test/openagents/deployments/lifecycle_test.exs lines 12-18 quantify over
the transition table), but nothing enumerates writers of run state — no
compiled-import-table assertion like FLEETPROMOTE-001's, and no PostgreSQL
transition trigger on the runs table (contrast work_jobs, whose trigger
test/openagents/work_job_test.exs line 232 exercises). The suite itself
demonstrates the bypass: test/openagents/issues/completion_claims_test.exs
lines 565-571 force a run to succeeded with a raw update.
Smallest change: either a transition trigger on the runs table with a raw-update test, or a dependency-boundary assertion that the lifecycle module is the sole caller of the state-writing function.
OBSERVABILITY-001 — the read-back is proven against an empty database
Claim: release read-back recomputes zero-tolerance leakage, consent, provenance, executor-disclosure, and attribution checks from authoritative records without selecting private content.
Reality: test/openagents/observability_test.exs line 52 runs
OpenAgents.Observability.Readback against a freshly sandboxed database with
no conversation, message, or memory row seeded. The
"contains no private content" refutations (lines 65-66) are vacuous — there
is no content to leak — and the all-planes-healthy loop (lines 59-63) passes
over empty status maps. The blocked case is manufactured with put_in rather
than by seeding a violation, so neither direction (detects a real violation;
excludes real content) is exercised. The emit half does bite, but only 2 of 6
refusal branches are tested (lines 22-48).
Smallest change: seed one conversation with a known sentinel string and one zero-tolerance violation, then assert the read-back reports the violation and the sentinel appears nowhere in the snapshot.
RELEASE-001 — nothing can fail when migrations stop preceding traffic
Claim: the production image runs all pending migrations before starting the HTTP server.
Reality: the evidence line names the Docker CMD, but the CMD contains
no migration step — the ordering lives in lib/openagents/application.ex
lines 22-24 behind migrate_on_boot, which
lib/openagents/runtime_config.ex lines 267-268 require in staging and
production. No test asserts that requirement (the only migrate_on_boot hit
in test/ is a fixture setting it true), and
test/openagents_web/controllers/health_controller_test.exs runs against an
already-migrated test database. The smoke boots the release, but health is
SELECT 1 (lib/openagents_web/controllers/health_controller.ex line 5),
which succeeds on an unmigrated database, and the smoke never verifies the
database is fresh. Remove the migration call and every cited proof stays
green.
Smallest change: a test/openagents/runtime_config_test.exs case
asserting staging and production refuse migrate_on_boot: false, and a
migration-sensitive probe in the smoke (query a table the newest migration
creates). Also correct the evidence line: the mechanism is
OpenAgents.Release invoked from the application start, not the CMD.
RELEASE-002 — the absence half has no detector
Claim: secrets are absent from source, the build context, and image build arguments.
Reality: ops/ci/reference-check.sh is a naming check, not a secret
scan, and it cannot fail even at its own job when rg is missing: lines 37
and 39 end in || true, and ops/ci/gate.sh line 71 checks for jq mix npm
but never rg, so on a builder without ripgrep the check silently passes.
Nothing reads .gitignore or .dockerignore in any test; nothing scans
config/ or the Dockerfile for credential material; the Cloud Logging
exclusion the contract names exists only as prose in INVARIANTS.md. The
GitHub-config half (test/openagents/github_oauth/runtime_config_test.exs)
does bite, narrowly.
Smallest change: fail ops/ci/reference-check.sh when rg is absent
(and add it to the gate's tool check), and add a repository-tree scan test
for high-signal secret shapes in config/ and the Docker build context —
the same read-the-repo shape test/openagents/hosted_ci_absence_test.exs
already uses.
UI-003 — no population of surfaces exists to enumerate
Claim: product surfaces render only through the sanctioned component library.
Reality: test/openagents_web/component_catalog_test.exs proves the
catalog page documents the components — the arrow points the wrong way — and
test/openagents_web/ui_test.exs proves the primitives behave. Nothing scans
surfaces: a new LiveView rendering hand-rolled markup with its own palette
passes every cited proof. The narrow template scan that exists
(test/openagents_web/ui_contracts_test.exs) checks inert-control rules
only, by design.
Smallest change: narrow the claim to what the proofs cover (the primitive
contracts and the catalog), or add a template scan over
lib/openagents_web/** asserting variant-bearing markup uses the sanctioned
class-plus-data-attribute shapes.
DATA-001 — the projection-loss clause has no test
Claim: losing PubSub or the LiveView stream must not lose accepted data; rows are persisted before their state is presented as accepted.
Reality: the cited tests prove create_turn/2 persists and a happy-path
turn renders (test/openagents/conversations_test.exs lines 28-41,
test/openagents_web/live/chat_live_test.exs lines 536-589). Nothing kills
a subscription, drops a stream, or asserts persist-before-broadcast ordering,
so a regression to broadcast-then-persist passes.
Smallest change: narrow the claim to what the tests prove (durable rows for accepted turns), or add an ordering assertion at the delta-persistence seam.
Claims the code or the tests contradict
These are prose defects: the proof is fine or repairable, but the sentence in the ledger is not what the system does.
- VOICE-006 and VOICE-009 both state "sending typed input while voice is
active ends the voice generation first." The cited test proves the
opposite by design:
test/openagents_web/live/chat_live_test.exslines 282-342 is titled "typing during a live voice call keeps the call open and hands voice the message" and asserts the session stayslistening. The no-parallel-responses property may still hold through injection (no text turn opens), but the stated mechanism is a superseded design. - DATA-002 credits "unique indexes and identity-source constraint in
create_github_users." No migration by that name exists — the indexes live inpriv/repo/migrations/20260816214200_create_sarah_conversations.exs— and no identity-source constraint exists at all: nothing prevents a visitors row carrying both a browser key and a user ID. No test inserts a duplicate or races two creates, so dropping the unique indexes also fails nothing. - ADMIN-001 says the
/adminpanel shows recording completeness metadata;test/openagents_web/live/admin_live_test.exslines 110-119 refute recording strings with a stale comment claiming the product does not capture call audio, and line 139's model-ID refutation is vacuous because no call is seeded. - IDENTITY-007's proof index row cites
test/openagents/agents_test.exs, which contains no grant test; the lifecycle is actually proven intest/openagents/forge/assignment_test.exslines 105-131 and the two controller suites. The index row should follow the tests that exist. - STACK-001 cites
ops/ci/stack-contracts.shas evidence; nothing invokes it — it appears in no gate stage and wraps the same test file the suite already runs. - RELEASE-001's "Evidence: Docker
CMD" points at a command with no migration step (see above). - MEMORY-003's evidence line lists
test/openagents/profile_memory_test.exstwice — cosmetic.
Tautologies, wrong chokepoints, and unexercised constraints
The most consequential "partial" verdicts, each with the violation the proof misses.
THREAD-001 — the race proof does not race
test/openagents/threads/credit_race_test.exs line 20 is
use OpenAgents.DataCase, async: false, and test/support/data_case.ex
line 40 starts the sandbox owner in shared mode for non-async tests. Every
Task.async_stream worker therefore serializes on one shared connection —
the concurrent opens and mints are savepoints on a single transaction, not
competing transactions. Deleting both FOR UPDATE locks in
lib/openagents/threads.ex (the admission-cap lock and the mint-side lock)
leaves all four tests green, and THREAD-001's amended text cites this file as
proving "the cap holds under concurrency." The token-reach half of THREAD-001
(test/openagents/threads/grant_token_reach_test.exs) is the strongest
enumeration in the repository — four derived sets, both directions, explicit
non-emptiness guards, route population from the router. The race clause needs
a test that checks out two real connections (unsandboxed or with explicit
allowances) before it proves anything.
REPUTATION-001 — verification shares the producer's canonicalization
Every signature and digest check verifies with the same
OpenAgents.Provenance.Canonical encoding that produced the claim
(test/openagents/reputation_test.exs lines 38 and 45-49;
lib/openagents/reputation/claim.ex lines 103 and 112). A canonicalization
bug — a dropped field, an unstable ordering — verifies fine on both sides.
The tamper test (lines 116-124) catches a signature that ignores content, not
an encoding that ignores a field. "No function returns a ranking" is one
function_exported? check on one name (line 93) with no enumeration of the
module's exports. The key-rotation half
(test/openagents/forge/key_rotation_test.exs) bites hard and is the model
to follow. Smallest change: pin one full claim to a hard-coded expected byte
encoding and digest, the way test/openagents/forge/wal_test.exs lines
282-298 pin the WAL encoding.
TOOL-005 — the boundary is enumerated at the wrong chokepoint
test/openagents/dependency_boundary_test.exs lines 152-178 enumerate
callers of OpenAgents.Tools.Selector and require each to name
OpenAgents.Tools.Reach. But OpenAgents.Tools.Registry is itself a public
un-narrowed catalog builder — lib/openagents/tools/registry.ex line 135
forwards whatever options it was handed — and the real catalog builders sit
one hop out (lib/openagents/turns/turn_server.ex line 113,
lib/openagents/work/job_server.ex line 138), outside the enumeration. A new
caller of the registry's prompt-definitions functions that omits :reach
offers the whole catalog and stays green — the exact violation the contract
says this test catches. The "names Reach" check is also satisfied by the
registry's unrelated boot-time Reach.requirements() call
(lib/openagents/tools/registry.ex line 366), so it cannot distinguish
resolving a caller from mentioning the module. Smallest change: enumerate
callers of the registry's catalog-building functions and require the reach
option at those sites.
SELF-EDIT-001 — the population is a namespace prefix and the assertion is vacuous on empty
test/openagents/dependency_boundary_test.exs lines 196-198 filter to
modules named under the tools namespace, but the contract quantifies over
"every tool module." A tool-shaped module elsewhere — the coding work module,
a future plugin namespace — can name the hot-loader and stay green. The
assertion at line 93 is offenders == [], which passes if the module
enumeration breaks and returns nothing (unreadable BEAMs are swallowed at
line 210), and the fleet-release module list at lines 43-50 is hand-declared
with no existence check, so a renamed release module silently empties the
check forever. Smallest change: assert the enumerated population and the
release-module list are both non-empty, and derive the tool population from
the shipped-plus-fixture catalogs rather than a name prefix.
SETTLEMENT-001 — the uniqueness ladder is never climbed
Every "pays once" mechanism is a constraint no test reaches: the payment-hash
unique index (priv/repo/migrations/20260823060000_create_bounty_settlement.exs
line 147) cannot collide because the test gateway returns a constant hash and
no second payment ever succeeds; the single-paid-intent partial index and
the one-receipt-per-intent index are shadowed by earlier context refusals.
The invariant calls its schemas "seven append-only schemas," and no
append-only trigger exists on any settlement table — the refund test re-reads
a receipt and finds it unchanged, but nothing would have refused a rewrite.
Smallest change: raw-insert tests against each uniqueness constraint, and
either add the append-only triggers or strike the phrase.
COMPENSATION-001 — the payout check reads back a literal
"Never creates payout authority" is proven by
refute function_exported?(..., :payout, 2) and by asserting
payout_authority == false in a projection that hard-codes the literal
(lib/openagents/compensation.ex lines 30, 205, 239). The real defense — the
database CHECK on the policy rules
(priv/repo/migrations/20260817010500_create_compensation_accounting.exs
line 162) — is never bitten, and only one of seven append-only triggers is
tested. Smallest change: insert a policy with payout_authority: true raw
and assert PostgreSQL refuses.
EFFECT-001 — the production call site's transaction is untested
The outbox mechanics bite (test/openagents/effects_test.exs: real rollback,
real payload-digest conflict, real reclaim, genuinely concurrent claims). But
test/openagents/effects/work_launch_test.exs lines 27-41, titled "commits
its launch in the same transaction," asserts only that the effect exists
afterward, and the refusal case fails before the enqueue runs — so moving the
enqueue outside the job transaction, reintroducing the exact crash window
EFFECT-001 exists to close, keeps both tests green. The two schema
constraints the contract leans on (effects_lease_pair_check,
effects_status_shape_check in
priv/repo/migrations/20260824204740_create_effects.exs) are never reached
by any test. EFFECT-002's milestone separation therefore also rests on
untested constraints. Smallest change: a raw insert against each constraint,
and a work-launch test that crashes between the job insert and the launch and
asserts the effect row exists.
MODULE-001 — the pre-invocation byte check is real and unproven
The runner verifies the loaded executor before invocation
(lib/openagents/tools/registry.ex line 190), but the only tamper test
mutates an in-memory struct and calls the verify helper directly
(test/openagents/modules/registry_test.exs lines 171-184). Delete the
runner's check and the suite passes; no test unloads or changes module bytes
and drives the runner. Smallest change: purge a tool module's code in a test
and assert the runner returns the unavailable outcome.
EXIT-005 — the sole-writer clause and the publisher
The chain, the rewrite mutation, and the anchor checks are excellent — the rewrite is a real byte-level rewrite with recomputed keys, and the stranger's anchor is taken from served bytes. Two clauses still have no proof:
- "
append_entry/2is the one function every writer reaches the log through" — unchanged since the 2026-08-23 audit recorded it (issue #151). No test enumerates WAL writers, and the escape is real:append_entry/2is pure; the durable write is the CAS, and the test suite itself writes indexes without the chain in four places intest/openagents/forge/independence_test.exs. On a log with a pre-contract prefix, a bypassing writer'schain_link_missingis classified as history (lib/openagents/forge/verification.exlines 330-341), so the bypass can be absorbed silently. The current writer population is three modules (lib/openagents/forge/pushes.exline 175,lib/openagents/forge/git_plane.exline 532,lib/openagents/repositories/importer.exline 450); an import-table enumeration in the shape oftest/openagents/threads/grant_token_reach_test.exswould close it. OpenAgents.Forge.AnchorPublisherhas zero test coverage. Every anchor test calls the publish function by hand; nothing proves the scheduled child runs, reschedules, or thatpublished_atadvances — the clause the contract says lets a reader see that publication has stopped. Removing the supervisor child fails nothing.
Also: the encoding is verified by the same module that produces it;
test/openagents/forge/wal_test.exs lines 282-298 mitigate this with a
pinned hard-coded digest, which is the right shape and worth extending to a
full entry-plus-link vector.
EXIT-002 and EXIT-003 — depth-1 import tables and a hand-listed function set
The independence reads are strict about missing modules (a MatchError, not
an empty pass), but they read one module's own import table only: a helper
module between the verifier and the database, or an apply/3, is invisible.
The mirror-function refusal at test/openagents/forge/independence_test.exs
line 564 names three functions as a literal with no completeness assertion
against the pushes module's exports — a new mirror-reading export evades it.
EXIT-001 — six probes accept 200 with an empty body
read_probe/2 (test/openagents/data_rights/export_inventory_test.exs
lines 529-534) treats any 200 as portable without checking the body, so the
six families the contract narrates as previously blocked would still read
portable if their routes regressed to returning empty lists for the owner's
own records. The issue, project, and repository probes check for the seeded
record; these six should too.
WORK-001 — two of three bounds and the resume claim
The tool-call ceiling bites; the continuation ceiling and the ten-minute wall
clock have no test (no hit for either reason code in test/). The
non-advertisement clauses (no deep_work in a job's provider request, no
work.delegate or memory.write authority) are structurally invisible to
the scripted provider, which ignores the tools field entirely. Recovery
"resumes and continues" is asserted only as "not interrupted, generation
advanced" — a worker that adopts and immediately fails passes. SCV-001
inherits the same shape: "no job may deploy one" (scv.deploy never in job
authorities) has no test.
Route classification — the catch-alls and the operator class
test/openagents_web/authenticated_route_gate_test.exs is the real thing —
derived population, non-empty guard, actual anonymous dispatch. Two
structural gaps sit beside it. First, the classifier's catch-alls
(lib/openagents_web/route_authority.ex line 588 for API GETs, lines 695-724
for browser path shapes) classify unknown reads as :public_read silently —
a private GET added in one of those shapes leaves the gate population without
failing anything; the box fan-out status route already demonstrates the
mislabel (harmless today only because the API inventory independently
dispatches it). Second, no test dispatches :operator routes anonymously as
a class — the gate test filters them out, and
test/openagents_web/operator_surface_test.exs proves membership, not
behavior, checking pipe_through for exactly one route. A route correctly
classified and correctly listed, but wired without the operator pipeline and
without an in-handler recheck, passes everything. FLEETPROMOTE-001's
/admin/forge is one path-keyed classification away from that shape, saved
today by its own LiveView test. Smallest change: dispatch the operator class
anonymously the way the authenticated class is dispatched.
Scope-boundary scans — one backend is invisible and one guard is missing
test/openagents/memory/scope_boundary_test.exs is two proofs of different
strength. The entry-point enumeration bites. The query scan does not guard
against emptiness on the recall side (line 196 iterates whatever the AST scan
returns), and OpenAgents.Memory.HybridRecall reaches the database only
through raw SQL the AST scan cannot see — deleting the conversation predicate
from lib/openagents/memory/hybrid_recall.ex line 70 fails nothing. The scan
also accepts a scope column named anywhere in the query, not only in a
where. Separately, OpenAgents.Memory.SemanticIndex.rebuild/1
(lib/openagents/memory/semantic_index.ex lines 120-126) reads messages
across every conversation and sits outside both MEMORY-001's and MEMORY-004's
populations. And PRIVACY-001's projector enumeration labels
OpenAgents.Tools.MemoryContract as reading the category list only, while
its memory_output/1 returns the raw stored claim to the model with no
redaction pass (lib/openagents/tools/memory_contract.ex) — the one-projector
assertion at line 264 compares the declaration to itself, and the redaction
check at line 270 is satisfied by any mention of the redaction module.
UI-002 — the AST matcher's syntax blind spot
test/openagents_web/tool_activity_projection_test.exs has real anti-vacuity
guards and both-direction key sets, but its matcher recognizes only
from ... select: keyword queries; a projection written as a pipeline
(select/3, select_merge, a Map.take after the query) lands in neither
set and the exactness still passes.
PERSONA-001 — two adapters have no wire probe
The behaviour and atom-table enumerations bite in both directions. But of the
three outbound HTTP adapters, only one is driven against a capturing plug;
OpenAgents.Providers.OpenRouter and OpenAgents.Providers.VercelGateway
have no wire probe in the boundary file, the configured-provider backstop
reads test-double config rather than production selections, and the
gateway's config key is not in the checked set. An OpenRouter payload builder
that appended its own system text would pass. (The payload test that exists
covers the messages field only, and shows instructions are trimmed — so
"byte for byte" is already not literal there.)
PROVIDER-002 — lane routing is unverifiable under test config
The refusal paths bite against the production catalog list. But all three
provider lanes are the same test module under config/test.exs, so the
adapter-lane assertions are Test == Test, and the catalog endpoint tests
compare the response to the same function that renders it.
The gate itself
Findings that are about whether proofs run, not whether they can fail.
- A Markdown-only candidate runs no tests.
ops/dev/precommit.shlines 61-81 skip the test suite when every changed path is documentation, andops/ci/gate.shline 116 runs precommit as a stage — the receipt still records the stage as passed. Tests with no dedicated gate stage are only reached through precommit, includingtest/openagents/hosted_ci_absence_test.exs,test/openagents/network_status_test.exs,test/openagents_web/transparency_surface_test.exs,test/openagents/transparency/work_disclosure_test.exs,test/openagents/forge/deployment_lane_test.exs,test/openagents/capacity_test.exs, and both token-vault suites. This is the most likely mechanism by which the red operator-surface enumeration reachedmain. - The gate receipt is forgeable and short-circuits.
ops/ci/gate.shlines 15-30 and 51-54: an unsigned JSON at a known path skips the gate entirely.test/openagents/forge/gate_receipt_test.exsproves shape and SHA-binding, not authenticity — RELEASE-004 does not claim authenticity, so this is a bound to record rather than a defect, but the stage list is declared in three unlinked places and nothing ties the receipt's list to the script's. ops/ci/reference-check.shpasses whenrgis missing (see RELEASE-002 above).- RELEASE-004's absence test covers hosted providers by list, and the
list omits
cloudbuild.yaml— notable because the deployment target is Google Cloud — plus a handful of others (CodeBuild, Bitbucket, Gitea and Forgejo workflow paths). - VAULT-001's third vault has no test.
OpenAgents.Voice.RecordingVaulthas no test file; nothing proves it does not borrow the GitHub key, and no enumeration closes the vault population, so a fourth vault that borrows a key is invisible.
Every contract, one verdict
Verdicts: can fail (the cited proof goes red when the claim is violated), partial (the core clause can fail; the named clause cannot), cannot fail (a load-bearing clause has no proof that can go red). Details above where a section exists; one line here for everything else.
Identity and canon
| Contract | Verdict | Sharpest gap |
|---|---|---|
| CANON-001 | partial | Content SHA-256 is shape-checked, never compared to any file's bytes (lib/openagents/persona/source_manifest.ex line 256) |
| CANON-002 | can fail | Strongest enumeration in the set; pattern covers the public schema only, and the audit-kind check is one-directional |
| PERSONA-001 | partial | Two of three outbound adapters have no wire probe; config backstop reads test doubles |
| PERSONA-002 | can fail | Voice-session role receipt persistence has no cited proof |
| PERSONA-003 | partial | The incomplete-results branch is untested; corpus digest computed on both sides, so a weakened corpus moves both |
| BLUEPRINT-001 | partial | Deletion of admitted rows and the facts table never attempted; explicit-none pinned only at the composer layer |
| PROGRAM-001 | partial | No-effect-authority is four atom names on one module; the in-flight-capture test asserts BEAM immutability |
| DEGRADE-001 | can fail | Real trigger tests, default path covered |
| PROGRAM-002 | partial | "The live turn never consumes their result" has no test; the synthetic-corpus claim is the fixture describing itself |
| PROGRAM-003 | can fail | Generation-pointer proof is genuine |
Identity and authorization
| Contract | Verdict | Sharpest gap |
|---|---|---|
| IDENTITY-001 | can fail | One-time state, replay, ban, token-never-in-browser all driven |
| IDENTITY-002 | partial | Classifier catch-alls mislabel silently; Repo-reach scan has no positive control; public-read LiveViews exempt from the event hook |
| IDENTITY-003 | partial | Non-persistence is a schema field-name check; KDF envelope parameters unpinned |
| IDENTITY-004 | partial | One-active-link constraint and attribution-across-unlink untested |
| IDENTITY-005 | can fail | Route list is 5 of 12 box-control paths |
| IDENTITY-006 | cannot fail | See section above |
| IDENTITY-007 | partial | Cited file has no grant test; real coverage lives elsewhere |
| IDENTITY-008 | can fail | Lock asserted from the live database function; contention window honestly declared unproven |
| IDENTITY-009 | can fail | Planted-secret redaction is a real control |
| IDENTITY-010 | can fail | Catalog scan with positive control; nested-value and derived-copy residues |
| IDENTITY-011 | can fail | Query plan asserted with sequential scans priced out |
| IDENTITY-012 | can fail | Residue: a future non-changeset writer of guarded columns |
| CAPACITY-002 | can fail | Concurrency tests would fail without the locks |
| CAPACITY-003 | partial | Interval test asserts the literal it passed in; 15-second timeout and cost parser untested |
| WORK-002 | can fail | Single-probe and cancellation-stamp ordering proven |
Data, memory, and privacy
| Contract | Verdict | Sharpest gap |
|---|---|---|
| DATA-001 | cannot fail | See section above |
| DATA-002 | partial | Cited migration name and identity-source constraint do not exist; no duplicate-insert or race test |
| DATA-003 | can fail | Partition test is complete; timestamp-tie determinism unasserted |
| MEMORY-001 | can fail | The semantic-index rebuild reads cross-conversation and sits outside the population |
| MEMORY-002 | partial | Model-side clauses vacuous against a scripted provider; host-side halves bite |
| MEMORY-003 | can fail | Real trigger tests via raw SQL |
| MEMORY-004 | partial | Query scan vacuous for the hybrid backend's raw SQL; no non-empty guard on the recall side |
| MEMORY-005 | cannot fail | See section above |
| MEMORY-006 | partial | Synonym-recall metric measures the test's own regex provider |
| MEMORY-007 | can fail | Gate driven end to end with each bypass refused |
| MEMORY-008 | partial | Default-off asserted from the eval JSON, not the shipped config |
| MEMORY-009 | partial | Same default-off defect; build determinism genuinely proven |
| PRIVACY-001 | partial | The claim-projecting tool is mislabeled and skips redaction; projector assertion compares the declaration to itself |
Turn, provider, and effects
| Contract | Verdict | Sharpest gap |
|---|---|---|
| TURN-001 | can fail | Real second insert against the real index; no concurrent case |
| TURN-002 | partial | Single-transaction claim untestable as written — moving inserts out of the transaction passes |
| TURN-003 | partial | Non-blocking proven; persist-before-broadcast ordering not |
| TURN-004 | can fail | Idempotency claimed, never called twice |
| TURN-005 | partial | Parallel-call refusal and continuation ceiling have zero tests |
| PROVENANCE-001 | can fail | Five raw-SQL trigger tests; the strongest turn proof |
| PROVIDER-001 | partial | Contract test iterates two of four adapters |
| PROVIDER-002 | partial | Lane assertions are the same module compared to itself under test config |
| EFFECT-001 | partial | See section above |
| EFFECT-002 | partial | The status-shape constraint that carries the claim is untested |
| TOOL-001 | partial | No registry rebuild mid-turn; digest compared to the live registry |
| TOOL-002 | partial | Execution contexts hand-built rather than application-created |
| TOOL-003 | can fail | Claim atomicity proven sequentially only |
| TOOL-004 | partial | The UI disclosure test runs entirely on literal maps |
| TOOL-005 | partial | See section above — wrong chokepoint |
| TOOL-006 | can fail | Reads shipped config through the prod reader, not the test fixture; both directions |
| DEGRADE-002 | can fail | Honest-degradation asserted on the rendered answer |
Collective, settlement, and modules
| Contract | Verdict | Sharpest gap |
|---|---|---|
| COLLECTIVE-001 | partial | Cited database constraints never exercised; raw-quote check is six hand-listed fields |
| COLLECTIVE-002 | cannot fail | See section above |
| COLLECTIVE-003 | can fail | Append-only via real constraint errors |
| COMPENSATION-001 | partial | See section above |
| REPUTATION-001 | partial | See section above |
| SETTLEMENT-001 | partial | See section above |
| MODULE-001 | partial | See section above |
| MODULE-002 | can fail | Projection key allowlist would strengthen it |
| MODULE-003 | can fail | Nine-axis filter matrix is a real enumeration |
| MODULE-004 | cannot fail | See section above |
Delegated work, threads, and deployment plane
| Contract | Verdict | Sharpest gap |
|---|---|---|
| WORK-001 | partial | See section above |
| SELF-EDIT-001 | partial | See section above |
| SCV-001 | partial | The no-job-may-deploy clause has no test |
| OUTCOME-001 | partial | Four of five false-green classes never pass through either path; the pure-grader suite proves fixtures |
| THREAD-001 | partial | Token reach is exemplary; the race proof does not race (see above) |
| THREAD-002 | can fail | Write routes dispatched as a reader; one subset check is one-directional |
| DEPLOYPLANE-001 | partial | Artifact-digest format never validated |
| DEPLOYPLANE-002 | can fail | Every bound dimension widened in turn |
| DEPLOYPLANE-003 | partial | The digest half of "same SHA and same digest" never varied on the admission path |
| DEPLOYPLANE-004 | cannot fail | See section above |
| DEPLOYPLANE-005 | partial | Supervisor gating untested; all provider claims carried by the one fake |
| FLEETPROMOTE-001 | can fail | Best-proven contract in the ledger; the browser route's pipeline membership rests on its own LiveView test |
Voice, admin, and interface
| Contract | Verdict | Sharpest gap |
|---|---|---|
| VOICE-001 | can fail | Architecture and provider values never given a bad case |
| VOICE-002 | partial | CSRF never omitted; SDP size ceiling untested; header check greps the variable name, not the value |
| VOICE-003 | partial | The database-constraint half is unproven; the one-active index has no test and the code path does not use it |
| VOICE-004 | can fail | Hostile provider frames through the real decoder |
| VOICE-005 | cannot fail | See section above |
| VOICE-006 | can fail | But the prose contradicts the cited test (see above) |
| VOICE-007 | can fail | Changed-identity refusal has no dedicated test |
| VOICE-008 | can fail | Exact durable terminal set asserted |
| VOICE-009 | can fail | Real chronology plus a real constraint error; same stale sentence as VOICE-006 |
| VOICE-010 | partial | The advisory-lock serialization claim has no concurrent test |
| VOICE-011 | partial | The event-kind field is a free-text channel no test constrains; the 64-event cap untested |
| VOICE-012 | can fail | The no-opt-out never-claim has no closing mechanism |
| ADMIN-001 | can fail | Red right now (see above); membership proven, class-wide behavior not |
| DATA-004 | can fail | Cascade seeded broadly with direct-delete refusals first; the voice-summary export branch is vacuous |
| UI-001 | partial | The operator class is never dispatched anonymously |
| UI-002 | can fail | Pipeline-syntax queries evade the matcher |
| UI-003 | cannot fail | See section above |
| LEADERBOARD-001 | can fail | Two named exclusions never seeded, so excluded vacuously |
| OBSERVABILITY-001 | cannot fail | See section above |
Release, status, and transparency
| Contract | Verdict | Sharpest gap |
|---|---|---|
| RELEASE-001 | cannot fail | See section above |
| RELEASE-002 | cannot fail | See section above |
| VAULT-001 | partial | The recording vault has no test; no enumeration over the vault population |
| RELEASE-003 | cannot fail | See section above |
| RELEASE-004 | partial | Provider path list omits the deployment target's own CI config file |
| RELEASE-005 | can fail | Empty-relup refusal is a real positive control |
| RELEASE-006 | can fail | Both directions of refusal at target and readiness |
| RELEASE-007 | can fail | Parses the real Dockerfile; one silent-skip seam when a variable is undeclared in a stage |
| RELEASE-008 | can fail | Runs the real OTP walk against the real running application |
| RELEASE-009 | can fail | Fail-closed in every direction; the file has no dedicated gate stage |
| STATUS-001 | can fail | Structs contribute one opaque key; degradation enumerated for one section only |
| CAPACITY-001 | partial | Redaction proven over nine hand-listed fields, no exact key set on the projection |
| TRANSPARENCY-001 | partial | The route population filter misses the anonymous JSON API entirely; the gate check proves a static reference, not a gated path |
Repository, exit, and issues
| Contract | Verdict | Sharpest gap |
|---|---|---|
| REPOSITORY-001 | can fail | The past insert-the-grant-yourself defect is genuinely closed; regex-evading join forms remain the residue |
| API-001 | partial | Governance covers one of four extension objects; one enum is written twice with nothing binding the copies |
| CONTRIBUTION-001 | can fail | Cross-binds the document to the router, both authorities, and the real push guard |
| REPOSITORY-002 | can fail | Executes the real script against real hostile remotes |
| REPOSITORY-003 | can fail | Real pushes, real cache deletion, real replay |
| EXIT-001 | can fail | Six probes accept 200 with an empty body (see above) |
| EXIT-002 | partial | Depth-1 import tables; helper-module indirection invisible |
| EXIT-003 | partial | Mirror-function list is a literal with no completeness check |
| EXIT-004 | can fail | Rebuilds the exact #179 failing shape and clones for real |
| EXIT-005 | partial | Sole-writer clause still unproven (#151); the anchor publisher has zero coverage (see above) |
| EXIT-006 | can fail | Gap list vacuous today in one direction, by honest construction |
| STACK-001 | can fail | The cited CI script is invoked by nothing |
| ISSUE-001 | can fail | Real pushes over real HTTP through the real auth plug |
| FORUM-001 | can fail | The source scan carries its own anti-rot control |
| ISSUE-002 | can fail | The lock-based no-clobber clause and the 200-body bound are untested |
| ISSUE-003 | can fail | Ambiguous-name and unsettled-name cases covered |
| ISSUE-004 | can fail | Crafted events from readers asserted to write nothing |
| PROMISE-001 | partial | The LIVE gate is proven on create, not on update into LIVE |
| PROMISE-002 | can fail | Proposed status; the append-only trigger is already exercised |
| NOTIFY-001 | partial | The "reveals nothing" clause has no proof — no schema enumeration, no content refutation anywhere in the cited tests |
| FORGEAPI-001 | can fail | Dispatches every envelope-classified route and pins the exact key set |
What to fix first
- Amend ADMIN-001 for the Gym and re-run the enumeration — the ledger is wrong today and the proof already says so.
- MEMORY-005 — decide whether the code or the sentence is right; the two cannot coexist. If the fallback stays, the invariant must be rewritten and the consent claim withdrawn.
- Close the gate's Markdown-only skip or give the enumeration tests (operator surface, transparency surface, hosted-CI absence, work disclosure) a dedicated gate stage, so a red enumeration cannot land with a passing receipt again.
- RELEASE-003 — wire the tested module into production or retarget the proof; today the claim is carried by code nothing cited tests, and the cited test covers code nothing runs.
- THREAD-001's race — a concurrency proof in shared sandbox mode proves serialization by the test harness, not by the lock. Two real connections or an honest narrowing.
- The unexercised constraint set — settlement uniqueness and append-only,
the effects lease-pair and status-shape checks, compensation's payout
CHECK, the voice one-active index: each is one raw-SQL test in the shape
test/openagents/threads/grant_fence_test.exsalready uses everywhere. - EXIT-005's writer set and publisher — the one remaining row from the 2026-08-23 audit's closing table, still open, plus a supervised child with no test.
- IDENTITY-006 — the credential's branch discipline is the whole point of the contract and is one real receive-pack test away from proven.
A closing observation the counts understate: the proofs this repository builds when it decides to enumerate — the operator surface, the grant-token reach, the visibility joins, the WAL rewrite, the fleet-promotion caller sets — are genuinely strong, and three of them caught real defects during this audit (the Gym, the mislabeled fan-out route, the stale admin-panel comment). The recurring weakness is not the enumerations; it is the sentence beside them that names a mechanism ("in one transaction", "under a lock", "append-only", "the only definition", "before the supervision tree starts") that no test has ever watched fail. Where a constraint or an ordering is load-bearing, the proof should reach it raw — the way the fence, provenance, and profile-memory tests already do — or the ledger should stop crediting it.