OpenAgents

Verification reports lag as tampering, and the three nodes disagree about the same repository #251

Closed AtlantisPleb opened this 4d ago 2 comments

Evidence

Shipped in the release at 0bf2706, promoted 2026-08-27.

1 pushes receipt

What happened

#180's rehearsal 2 was performed against the live forge on 2026-08-25, on revision 46cf8a5aea3791936c22e82c145a9a8dd734374d. Run on one node it reports findings: [] for this repository, which is the result the rehearsal asks for. Run on all three nodes at the same time it reports three different answers.

Two samples, minutes apart, each one a single sweep across sarah-fleet-1, sarah-fleet-2, and sarah-fleet-3:

Time (UTC) Repository fleet-1 fleet-2 fleet-3
14:38–14:40 OpenAgentsInc/openagents.com clean, 382 entries served_refs_diverged, object_missing, 383 clean, 383
14:38–14:40 OpenAgentsInc/openagents served_refs_diverged, object_missing clean served_refs_diverged, object_missing
14:57–14:58 OpenAgentsInc/openagents.com clean, 391 clean, 392 served_refs_diverged, object_missing, 393
14:57–14:58 OpenAgentsInc/openagents served_refs_diverged, object_missing clean served_refs_diverged, object_missing, object_missing

These are the two findings that mean the served state disagrees with the record. OpenAgents.Forge.Verification's own moduledoc calls object_missing "the repository cannot produce an object the WAL says a push introduced", and #179 treats one of them as a corruption serious enough to file. On a healthy forge, under ordinary push traffic, every node reports them about some repository most of the time.

Why the nodes disagree

The WAL is shared and the projection is not. forge_wal_adapter is OpenAgents.Forge.WAL.Gcs against the bucket gs://sarah-forge-wal, so all three nodes read one log. The bare repositories live on each node's own disk — /var/lib/openagents/forge/repos is on /dev/sda1, not a shared mount — so a push accepted on one node reaches the other two only when they replay it.

verify/2 compares a shared log against a local projection and has no way to tell a node that has not replayed yet from a node whose projection was altered. Both produce the same two findings.

The window is minutes rather than seconds. Entry 136 of OpenAgentsInc/openagents was written at 14:32:33Z; sarah-fleet-1 still reported the divergence at 14:38:25Z and was clean by 14:42:35Z. It does converge on its own, and by 14:42 the node's WAL head and its refs/heads/main on disk both read 4773472f.

Why it matters

#179's third acceptance criterion cannot be met as written. It asks for "a periodic verification pass over the configured repositories whose findings reach the status disclosure EXIT-006 publishes". A pass built today would publish served_refs_diverged and object_missing for a forge with nothing wrong with it, on whichever node ran it, most of the time. The finding that would have caught #179 is the finding that fires constantly, so it would be tuned out.

A verifier that cannot distinguish lag from divergence weakens EXIT-002. The contract's value is that a reader can be told whether to trust what the forge serves. An answer that depends on which of three nodes replies, and that is wrong for several minutes after every push, is not that.

It is also visible to clients, in principle. A clone is answered by whichever node the load balancer picks. Eight consecutive git ls-remote calls against OpenAgentsInc/openagents returned the same sha, so no flapping was observed from outside, but nothing in the design prevents one node answering with refs another node has already advanced past.

Reproduction

cat > /tmp/probe.exs <<'PROBE'
alias OpenAgents.Forge.Verification
lines =
  for r <- ["OpenAgentsInc/openagents.com", "OpenAgentsInc/openagents"] do
    {st, rep} = Verification.verify(r)
    "#{r}: #{st} entries=#{rep.entries} findings=#{inspect(Enum.map(rep.findings, & &1.code))}"
  end
File.write!("/tmp/probe.out", "node=#{node()} at=#{DateTime.utc_now()}\n" <> Enum.join(lines, "\n"))
PROBE

Copy it into the container on each of sarah-fleet-1, sarah-fleet-2, and sarah-fleet-3 and run bin/openagents rpc 'Code.eval_file("/tmp/probe.exs")', then read /tmp/probe.out. Under any push traffic the three answers differ.

Acceptance criteria

  • verify/2 distinguishes a projection that has not replayed an entry yet from one that disagrees with an entry it has applied. The applied-sequence marker OpenAgents.Forge.Sync.ensure_fresh/1 already trusts is the obvious input: an entry above the marker is lag, and an entry at or below it that the projection cannot produce is a finding.
  • A verification report says which node produced it and what its applied sequence was, so a reader can tell one node's answer from the forge's.
  • #179's scheduled pass, whenever it lands, does not publish a finding for a node that is merely behind.

Found by: #180's rehearsal 2, docs/forge-exit-rehearsals.md.

  1. AtlantisPleb opened this issue 4d ago
  2. A AtlantisPleb Author 4d ago

    Sweep verdict: stays open. This is a real design task, not a remainder, and it has no owner.

    Nothing here is a small finish. The first acceptance criterion changes what verify/2 means: it has to take the applied-sequence marker as an input and split "this node has not replayed entry N yet" from "this node applied entry N and cannot produce what it says". That is a contract change to a verifier EXIT-002 depends on, and it needs the design pass the issue's own analysis sets up rather than a patch.

    What blocks it: nobody is assigned, and it should not be started casually — a verifier that is wrong in the other direction (silently treating real divergence as lag) is worse than the false positives being reported now. The lag window is minutes and self-healing, so this is not urgent in the way a red build is.

    One coupling worth recording so it is not discovered twice. #179's third acceptance criterion asks for a periodic verification pass whose findings reach the EXIT-006 status disclosure. As this issue shows, a pass built on today's verify/2 would publish served_refs_diverged and object_missing for a healthy forge most of the time, on whichever node ran it — so the scheduled pass should not ship before this lands, or it ships pre-tuned-out. #179 is in flight; that ordering is the thing to carry across.

    No code changed in this sweep.

  3. AtlantisPleb closed this as completed in 6bef922 4d ago
  4. A AtlantisPleb Author 4d ago

    Fixed on main in 6bef9222b8cc31fbe4a8ddf6419b342354cfea4f.

    Reproduced first. Three data directories against one WAL, which is the fleet's actual shape (shared object storage, per-node /var/lib/openagents/forge/repos). A node one entry behind — applied_seq=1, head seq=2 — returned {:error, ...} with served_refs_diverged (recorded = the new sha, served = the old one) and object_missing, byte for byte the shape a tampered projection produces. A node that had never replayed returned the same two codes, three times over. The issue does not overstate it.

    How lag and tampering are told apart now. verify/2 locates the projection on the log instead of comparing it to the head. The greatest sequence whose recorded post-state refs the repository serves exactly is its :position, and head_seq - position is :behind. A projection sitting at some sequence on the log carries no finding and reports status: :behind with {:ok, report}. A projection sitting at no sequence reports served_refs_diverged per ref and status: :diverged.

    The boundary is tight because lag runs one way only. A node that has not replayed an entry is missing what that entry introduced; it can never serve a ref the log has no record of, a ref at a value the log never recorded, or a value the log recorded before the state it is serving. Each of those is still a finding however far behind the node is, and object_missing is bounded by the position rather than waived — every object an entry at or below it introduced must still be there.

    The applied-sequence marker bounds the search rather than answering it: only sequences at or above it are candidates. Rolling the marker forward to the head makes the check stricter, not quieter, because the head's refs still have to be present. Rolling it back to -1 only widens the search to older states the log itself records. A marker past the end of the log is a new finding, applied_seq_beyond_log — which is also how a WAL truncated at the tail surfaces, since 0..n-1 stays contiguous afterward and every other check misses it.

    Two limits are named rather than hidden. A projection rolled back to a state the log passed through, with its marker rolled back to match, is reported as behind: from the WAL and the repository alone that is the same observation as a node mid-replay. And a deleted cache is an empty projection at sequence -1, which is what a node that has never replayed looks like — the report still says how far from the log it is, and REPOSITORY-003 is what brings it back. The independence test that asserted a lost cache was divergence now asserts it is behind by the whole log, which is still not clean.

    The cluster answer. verify_cluster/2 asks every member and combines the answers into a verdict lag cannot move: :verified when every member answered and every one is at the head; :converging when nothing contradicts the log and some member is behind or silent; :diverged only when a member contradicts the log, when a member cannot see a repository the others verified, or when two members report different chain links at the same sequence; :unavailable when nothing was checked. A replaying node moves the fleet between :verified and :converging, both of which clear themselves. Every report names its node, applied sequence, position, and distance from the head, and cluster findings carry the node that produced them. Passing :anchor through to the members is how you check they are reading one log rather than only each other.

    Acceptance criteria.

    1. An entry above the marker is lag, an entry at or below it that the projection cannot produce is a finding. Done, and made stricter: the marker is a claim that bounds the search, not a tolerance.
    2. A report says which node produced it and at what applied sequence: :node, :applied_seq, :position, :behind, :head_seq.
    3. #179's scheduled pass can publish findings without publishing the replay window — a behind node returns {:ok, report} with findings: [].

    Tests. test/openagents/forge/verification_test.exs, 15 tests over three real projections replayed from one real WAL of genuine receive-pack entries: a lagging node, a never-replayed node, a tampered current node, a node that is both behind and tampered, a smuggled ref on a lagging node, a missing object, a forged marker in each direction, and the fleet at three sequences, converging, verified, diverged, partly silent, and wholly silent.

    EXIT-002 is amended in the same commit with the lag/divergence rule, the marker's role, both named limits, and the cluster verdict; docs/taxonomy.md gains behind and diverged so the two words stop being interchangeable.

Sign in with GitHub to comment on this issue.