OpenAgents

Commit

Audit: test coverage, measured

78d4cb06cf48 · AtlantisPleb · · parent dcbe8a46684c 

Audit: test coverage, measured

The prompting question assumed issues/projects was the well-tested part and
the rest might not be. `mix test --cover` at dcbe8a4 says the opposite:

  issues/projects (native)   37 modules   33.0% mean   62% at zero
  everything else (ported)  372 modules   83.8% mean    3% at zero
  total                     415 modules   79.14%

Its domain schemas are genuinely at 100% -- Issue, Label, Milestone,
Project. What was never written is the web layer: all 8 LiveViews, 6 of 7
REST controllers, and 3 of 4 JSON views are at zero, plus the ProjectFields
and ProjectItems contexts. Two fixtures are themselves at 0%, meaning they
have never been executed by any test.

The gap maps exactly onto AGENTS.md's own nine-step TDD order: steps 1-5
(issues, comments) are covered, steps 6-9 (assignees, labels, milestones,
projectsV2) are at zero. The discipline held for five endpoints and was
dropped for four.

Records two honest qualifiers. Cluster.Drain and Cluster.RaBootstrap read
as 0% only because the coverage run excludes :cluster -- they are covered by
the separate stage, so that is a measurement artifact rather than a gap. And
the one real gap outside issues/projects, the recovery workers (TurnRecovery,
VoiceRecovery, WorkRecovery, SemanticWorker), is INHERITED: Sarah has no
tests for them either, so the port carried across an upstream gap rather
than creating one.

Deploy story

What this commit did to the running system — joined from the forge receipt chain, the part a commit page elsewhere cannot show.

Not deployed through the forge lane

No push, promotion, build, or deploy receipt references this commit (receipts are scanned over a bounded recent window). Changes shipped by full node replacement carry their proof in the release gate receipt instead.

Changed files

  • added docs/2026-08-20-test-coverage-audit.md

Diff

1 file changed, +155 -0

docs/2026-08-20-test-coverage-audit.md added +155

@@ -0,0 +1,155 @@

1 
# Test coverage audit
2
3 
**Date:** 2026-08-20
4 
**Commit measured:** `dcbe8a4`
5 
**Method:** `mix test --cover` (Elixir's built-in line coverage), 415 modules measured
6 
**Suite state at measurement:** 939 passed, 0 failed, 9 excluded (`:cluster`, run separately)
7
8 
---
9
10 
## 0. Verdict
11
12 
The question that prompted this audit was: *"we added a bunch of tests for issues/projects, do we have similar coverage for the rest of our codebase?"*
13
14 
**The premise is inverted. The issues/projects layer is the least-tested code in the
15 
repository — by a wide margin — and everything else is in good shape.**
16
17 
| | modules | mean coverage | modules at 0% |
18 
|---|---|---|---|
19 
| **issues/projects (openagents.com-native)** | 37 | **33.0%** | **23 (62%)** |
20 
| **everything else (ported from Sarah)** | 372 | **83.8%** | 13 (3%) |
21 
| **Total** | 415 | **79.14%** | 41 (10%) |
22
23 
The intuition behind the question is understandable: issues/projects is the surface that
24 
was built here, test-first, with a written TDD workflow — so it *feels* like the well-tested
25 
part. And its **domain schemas** genuinely are: `Issues.Issue`, `Labels.Label`,
26 
`Milestones.Milestone`, `Projects.Project` are all at 100%.
27
28 
What never got written is the **web layer**. Every LiveView, most controllers, and most JSON
29 
views in that surface are at zero.
30
31 
---
32
33 
## 1. The gap maps exactly onto an abandoned plan
34
35 
`AGENTS.md` §"Test-driven development workflow" defines a nine-step build order for this
36 
surface and names `docs/github-api-issues-projects-assessment.md` as *"the source of truth
37 
for paths, status codes, and JSON shape."* It states the discipline plainly:
38
39 
> One endpoint at a time. Each new endpoint starts with a failing test before the route,
40 
> controller, context, or schema exists.
41
42 
Measured against that list:
43
44 
| step | endpoint group | state |
45 
|---|---|---|
46 
| 1–4 | issues list / get / create / update | **covered** — `IssueController` 88.9%, `IssueJSON` 96.2% |
47 
| 5 | comments | **covered** |
48 
| 6 | assignees | **0%** |
49 
| 7 | labels | **0%** |
50 
| 8 | milestones | **0%** |
51 
| 9 | projectsV2 | **0%** |
52
53 
The discipline held for the first five steps and was dropped for the last four. This is not
54 
a mystery gap; it is a plan that stopped being followed, and the coverage numbers mark the
55 
exact line where it stopped.
56
57 
## 2. The issues/projects layer, module by module
58
59 
**Zero coverage (23 modules):**
60
61 
```
62 
OpenAgents.ProjectFields                 OpenAgentsWeb.IssueIndexLive
63 
OpenAgents.ProjectFields.ProjectField    OpenAgentsWeb.IssueNewLive
64 
OpenAgents.ProjectItems                  OpenAgentsWeb.IssueShowLive
65 
OpenAgents.ProjectItems.ProjectItem      OpenAgentsWeb.LabelIndexLive
66 
OpenAgents.ProjectFieldsFixtures         OpenAgentsWeb.MilestoneIndexLive
67 
OpenAgents.ProjectItemsFixtures          OpenAgentsWeb.ProjectIndexLive
68 
OpenAgentsWeb.AssigneeController         OpenAgentsWeb.ProjectShowLive
69 
OpenAgentsWeb.IssueAssigneeController    OpenAgentsWeb.AssigneeIndexLive
70 
OpenAgentsWeb.IssueLabelController       OpenAgentsWeb.LabelJSON
71 
OpenAgentsWeb.LabelController            OpenAgentsWeb.MilestoneJSON
72 
OpenAgentsWeb.MilestoneController        OpenAgentsWeb.ProjectJSON
73 
OpenAgentsWeb.ProjectController
74 
```
75
76 
**Partial:** `Projects` 30.6%, `Issues` 37.8%, `Milestones` 78.6%, `Labels` 87.5%.
77
78 
`OpenAgents.Issues` at 37.8% is the one to worry about: it backs the entire `/api/v3` issues
79 
surface, which is the compatibility promise the GitHub-clone plan rests on.
80
81 
**Two fixtures are themselves at 0%.** `ProjectFieldsFixtures` and `ProjectItemsFixtures`
82 
exist in `test/support/fixtures/` but are referenced by no test. Fixtures that have never
83 
been executed are not a head start — they are unverified code that looks like a head start.
84
85 
**All eight LiveViews are at zero.** Not "thinly covered" — never executed by a test. Nothing
86 
asserts that `/OpenAgentsInc/openagents.com/issues` renders at all.
87
88 
## 3. Everything else is in good shape, with one honest exception
89
90 
The ported subsystems average 83.8% with 198 modules at or above 90%. That coverage came
91 
across with the code: the Sarah suite was lifted wholesale, and as of `dcbe8a4` every one of
92 
those tests runs (all `@moduletag :skip` tags were deleted and `test_helper.exs` excludes
93 
only `:cluster`).
94
95 
The 13 non-issues/projects modules at 0% break down as:
96
97 
- **Four derived `Inspect` implementations** and **two test-support modules**
98 
  (`OpenAgents.Test.HordePeer`, `OpenAgentsWeb.SarahChannelCase`) — not production code.
99 
- **`OpenAgents.Cluster.Drain` and `Cluster.RaBootstrap`** — these are a **measurement
100 
  artifact, not a gap.** The coverage run excludes `:cluster`, and both are exercised by the
101 
  9 tests in `mix test --only cluster`. Their true coverage is not zero; this run simply
102 
  could not see it.
103 
- **`Mix.Tasks.OpenAgents.BackfillVisitors`** — a one-shot backfill task.
104 
- **Provider adapters:** `Memory.OpenAIEmbeddings`, `Voice.OpenAI.Sideband`,
105 
  `Voice.Operations.LoadProbe`. These call out to OpenAI. The suite runs with no network by
106 
  design (fakes are wired in `config/test.exs`), so these are covered through their
107 
  behaviours rather than directly. Defensible.
108 
- **`OpenAgents.Release`** — release-time migration entry point, only meaningful in a release.
109
110 
**The one real gap: the recovery workers.** `TurnRecovery`, `VoiceRecovery`, `WorkRecovery`,
111 
and `Memory.SemanticWorker` are at 0%. These are the code paths that run *after* something
112 
has already gone wrong — a node died mid-turn, a voice session dropped, a durable job was
113 
orphaned. They are exactly the code you cannot afford to have wrong, and nothing exercises
114 
them.
115
116 
**Important qualifier: this is inherited, not a port omission.** Verified directly — Sarah
117 
has no tests for `TurnRecovery`, `VoiceRecovery`, `WorkRecovery`, or `SemanticWorker` either.
118 
The port faithfully carried across a gap that already existed upstream. That makes it a
119 
longer-standing risk than the issues/projects gap, not a lesser one, and it is not something
120 
the port can be blamed for or expected to have fixed.
121
122 
## 4. What this audit does not tell you
123
124 
- **Line coverage is not test quality.** A module at 90% can still have every assertion
125 
  checking the wrong thing. Treat these numbers as a map of where *nothing* is looking,
126 
  not as a quality score.
127 
- **The `:cluster` exclusion skews two modules** (see §3). Any future coverage run intended
128 
  as a whole-repo number should union `mix test --cover` with
129 
  `mix test --only cluster --cover`.
130 
- **Coverage of a LiveView through a route** counts, so some modules that look covered are
131 
  only incidentally exercised by a test aimed at something else.
132
133 
## 5. Actions
134
135 
In progress at the time of writing, in response to this audit. This section states intent,
136 
not completed work — check the commits following `dcbe8a4` for what actually landed:
137
138 
1. Tests written for the six untested `/api/v3` controllers and three JSON views, against
139 
   `docs/github-api-issues-projects-assessment.md` as the contract — including error paths
140 
   (404, 422), since status codes are part of a compatibility promise.
141 
2. Tests written for all eight issues/projects LiveViews: mount, seeded records, empty
142 
   state, and at least one interaction each. Assertions target ids, `aria-*`, and visible
143 
   text rather than CSS classes, since this repo just migrated off DaisyUI onto basecoat.
144 
3. `ProjectFields` and `ProjectItems` contexts and schemas covered; `Projects` and `Issues`
145 
   raised from 30.6% and 37.8%. The two never-executed fixtures verified rather than assumed.
146
147 
Left open, deliberately:
148
149 
- **The recovery workers.** Worth its own task with the invariants in hand
150 
  (`TURN-005`, `WORK-001`, `VOICE-009` all describe recovery behaviour that should be
151 
  assertable). It needs upstream Sarah's design intent, not just a coverage number.
152 
- **A coverage floor in CI.** `mix test --cover` currently exits non-zero against the default
153 
  threshold but nothing gates on it. Once the issues/projects work above lands, setting a
154 
  floor in `mix.exs` `test_coverage` would stop this recurring — the gap here did not appear
155 
  suddenly, it accumulated unmeasured.

This page updates live while a promote is in flight · changelog