Check the exit runbook's commands against the compiled code

30d71aacf968 · AtlantisPleb · · parent 4a578fce1235

Check the exit runbook's commands against the compiled code

#189 found rehearsal 3, step 3, of docs/forge-exit-rehearsals.md telling an
operator to run OpenAgents.Forge.Sync.rebuild/1 before that function existed.
1f22414 closed the first half by adding rebuild/2 with tests, so the command
now runs and the doc's description of it is accurate. What remained open was
the issue's third criterion: nothing kept the runbook honest, because
invariants read compiled modules and the procedure was a string in a Markdown
file.

The new test reads the strings. Every `bin/openagents rpc '…'` command in any
document under docs/ is parsed as Elixir — pipes rewritten so a piped call is
checked at its real arity — and every OpenAgents function it calls must be
exported at the called arity. Qualified Module.function/arity references must
resolve, every module the runbook names must exist, and a bare `function/arity`
span must be exported by some module the document names. Renaming the doc's
rebuild call to a function that does not exist turns the test red, which was
verified before committing.

The rehearsal doc now says the check exists and why, and the cache-recovery
runbook names rebuild/1 as the direct discard-and-replay entry point, with the
honest caveat that a successful rebuild deletes the previous cache rather than
keeping it for verification.

Rehearsal 3 itself remains unperformed against the live forge; the status
table still says so, and #189's second criterion stays open until someone runs
it and records the output.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GoYpb8FEmdxVErsv7ABCYi
Co-Authored-By
Claude Fable 5 <noreply@anthropic.com>

Deploy story

What this commit did to the running system — joined from the forge receipt chain, the part a commit page elsewhere cannot show.

pushed
by user · WAL seq 298 · 2026-08-24T19:44:43.459087Z

Changed files

  • modified docs/forge-exit-rehearsals.md
  • modified docs/operations/forge-cache-recovery.md
  • added test/openagents/forge/exit_rehearsal_runbook_test.exs

Diff

3 files changed, +167 -1

docs/forge-exit-rehearsals.md modified +8

@@ -15,6 +15,14 @@ This document defines six rehearsals. Each one says what it proves, what it

15 15
cannot prove, and whether it has been performed. **Performed** means someone
16 16
ran it and recorded the result here. Anything else says so.
17 17
18
`test/openagents/forge/exit_rehearsal_runbook_test.exs` checks every
19
`OpenAgents` module and function this document names against the compiled
20
code. Rehearsal 3's step 3 once named `OpenAgents.Forge.Sync.rebuild/1`
21
before that function existed (#189), and no invariant could notice, because
22
invariants read compiled modules and the step was a string in this file. A
23
renamed or removed function now turns a test red instead of leaving a step
24
that reads as executable.
25
18 26
## Rehearsal status
19 27
20 28
| Rehearsal | Executable proof | Performed against the live forge |
docs/operations/forge-cache-recovery.md modified +7 -1

@@ -68,7 +68,13 @@ false `404` responses from the damaged node.

68 68
4. Move the affected bare repository cache aside. Do not delete or modify WAL
69 69
   indexes or WAL objects. Keep the moved cache until verification finishes.
70 70
5. Trigger repository synchronization or restart the application so boot
71
   convergence replays the WAL into a new local cache.
71
   convergence replays the WAL into a new local cache. To force a discard and
72
   replay in one call, run
73
   `bin/openagents rpc 'OpenAgents.Forge.Sync.rebuild("{storage_key}")'`. It
74
   rebuilds the projection from sequence zero in a sibling directory, verifies
75
   every authoritative ref, and swaps the result in atomically. On success it
76
   deletes the previous cache, so complete step 4 first when you want to keep
77
   the old cache until verification finishes.
72 78
6. Verify the applied WAL sequence, every authoritative ref, the reported blob,
73 79
   Git ref advertisement, and `/health` on that node.
74 80
7. Restore load-balancer admission only after `forge_cache_ready` and the
test/openagents/forge/exit_rehearsal_runbook_test.exs added +152

@@ -0,0 +1,152 @@

1
defmodule OpenAgents.Forge.ExitRehearsalRunbookTest do
2
  @moduledoc """
3
  #189's third acceptance criterion. Rehearsal 3's step 3 told an operator to
4
  run `OpenAgents.Forge.Sync.rebuild/1` before that function existed, and no
5
  invariant could notice: invariants read compiled modules, and the procedure
6
  was a string in a Markdown file. This test reads the strings.
7
8
  Three shapes of reference in `docs/forge-exit-rehearsals.md` are checked
9
  against the compiled code:
10
11
  - every `bin/openagents rpc '…'` command in any document under `docs/`
12
    parses as Elixir, and every `OpenAgents` function it calls is exported at
13
    the called arity;
14
  - every qualified `OpenAgents.Module.function/arity` reference resolves;
15
  - every bare `function/arity` reference in code font is exported by at
16
    least one module the document names.
17
18
  A renamed or removed function now turns this test red instead of leaving a
19
  runbook step that reads as executable but is not.
20
  """
21
22
  use ExUnit.Case, async: true
23
24
  @runbook "docs/forge-exit-rehearsals.md"
25
26
  test "every rpc command in the documentation calls functions that exist" do
27
    commands =
28
      for path <- Path.wildcard("docs/**/*.md"),
29
          [command] <-
30
            Regex.scan(~r{bin/openagents rpc '([^']+)'}, File.read!(path),
31
              capture: :all_but_first
32
            ),
33
          do: {path, command}
34
35
    assert Enum.any?(commands, fn {path, _command} -> path == @runbook end),
36
           "#{@runbook} no longer contains rpc commands; retire this test"
37
38
    for {path, command} <- commands do
39
      calls =
40
        command
41
        |> Code.string_to_quoted!()
42
        |> expand_pipes()
43
        |> collect_openagents_calls()
44
45
      assert calls != [], "rpc command in #{path} calls no OpenAgents function: #{command}"
46
47
      for {module, function, arity} <- calls do
48
        assert Code.ensure_loaded?(module),
49
               "#{inspect(module)} is named in #{path} but does not exist"
50
51
        assert function_exported?(module, function, arity),
52
               "#{inspect(module)}.#{function}/#{arity} is named in #{path} " <>
53
                 "but is not exported at that arity"
54
      end
55
    end
56
  end
57
58
  test "every qualified function/arity reference in the runbook resolves" do
59
    references =
60
      Regex.scan(
61
        ~r/(OpenAgents(?:\.[A-Z][A-Za-z0-9_]*)+)\.([a-z_][a-z0-9_]*[?!]?)\/(\d+)/,
62
        runbook(),
63
        capture: :all_but_first
64
      )
65
66
    for [module_name, function, arity] <- references do
67
      module = Module.concat([module_name])
68
      function = String.to_atom(function)
69
      arity = String.to_integer(arity)
70
71
      assert Code.ensure_loaded?(module),
72
             "#{module_name} is named in #{@runbook} but does not exist"
73
74
      assert function_exported?(module, function, arity),
75
             "#{module_name}.#{function}/#{arity} is named in #{@runbook} but is not exported"
76
    end
77
  end
78
79
  test "every module the runbook names exists" do
80
    modules = named_modules()
81
82
    assert OpenAgents.Forge.Sync in modules
83
84
    for module <- modules do
85
      assert Code.ensure_loaded?(module),
86
             "#{inspect(module)} is named in #{@runbook} but does not exist"
87
    end
88
  end
89
90
  test "every bare function/arity reference is exported by a module the runbook names" do
91
    # `function_exported?/3` does not load a module, so load them first. The
92
    # modules-exist test owns the assertion that loading never fails.
93
    modules = Enum.filter(named_modules(), &Code.ensure_loaded?/1)
94
95
    references =
96
      Regex.scan(~r/`([a-z_][a-z0-9_]*[?!]?)\/(\d+)`/, runbook(), capture: :all_but_first)
97
98
    # The reference this test exists for. Its absence means the document
99
    # changed shape, not that the check passed.
100
    assert ["rebuild", "1"] in references
101
102
    for [function, arity] <- references do
103
      # Document content is repository-controlled, so minting the atom here
104
      # is bounded by what review admits into the runbook.
105
      function = String.to_atom(function)
106
      arity = String.to_integer(arity)
107
108
      assert Enum.any?(modules, &function_exported?(&1, function, arity)),
109
             "`#{function}/#{arity}` is named in #{@runbook} but no module " <>
110
               "the document names exports it"
111
    end
112
  end
113
114
  defp runbook, do: File.read!(@runbook)
115
116
  defp named_modules do
117
    Regex.scan(~r/OpenAgents(?:\.[A-Z][A-Za-z0-9_]*)+/, runbook())
118
    |> List.flatten()
119
    |> Enum.uniq()
120
    |> Enum.map(&Module.concat([&1]))
121
  end
122
123
  # `verify(…) |> IO.inspect()` calls `IO.inspect/1`, not `IO.inspect/0`, so
124
  # pipes are rewritten into ordinary calls before arities are read.
125
  defp expand_pipes(ast) do
126
    Macro.prewalk(ast, fn
127
      {:|>, _, [_, _]} = pipe ->
128
        [{first, _} | rest] = Macro.unpipe(pipe)
129
130
        Enum.reduce(rest, first, fn {call, position}, piped ->
131
          Macro.pipe(piped, call, position)
132
        end)
133
134
      node ->
135
        node
136
    end)
137
  end
138
139
  defp collect_openagents_calls(ast) do
140
    {_, calls} =
141
      Macro.prewalk(ast, [], fn
142
        {{:., _, [{:__aliases__, _, [:OpenAgents | _] = parts}, function]}, _, args} = node, acc
143
        when is_list(args) ->
144
          {node, [{Module.concat(parts), function, length(args)} | acc]}
145
146
        node, acc ->
147
          {node, acc}
148
      end)
149
150
    Enum.reverse(calls)
151
  end
152
end

This page updates live while a promote is in flight · changelog