Tell a repository's name apart from the key it is stored under

bc446048d1ba · AtlantisPleb · · parent 639f024fd8f0

Tell a repository's name apart from the key it is stored under

The forge keys durable state by `Repository.storage_key` and hands people a
name. Both are strings, and a name is a legal path segment, so a name passed
where a key belongs built a path instead of failing. The verifier is where that
cost the most: the first documented step of exit rehearsal 2 says to run
`Verification.verify/1` on the repository the configuration admits, and
`Repos.allowed_repos/0` returns `openagents.com` while the live repository is
stored under an opaque key. The name went into the path, the path held a bare
repository projecting no log, and the report was `wal_unreadable` — the finding
that means the write-ahead log is gone — about a log of 275 entries that was
intact under the other key.

`OpenAgents.Forge.RepoRef` is now the one place a name becomes a key, and it
answers in a fixed order. A string the WAL holds a log under is a key and
resolves to itself, reaching no database, so a verifier handed a key stays
independent of PostgreSQL and a log older than the repositories table still
resolves. Anything else is a name, resolved through the same mapping the
serving path uses, aliases included. A name that settles on no repository, or
on two, stops there with `repository_not_found`, `repository_name_ambiguous`,
or `repository_lookup_unavailable` — never a path built out of the string, so
an absent repository can no longer look half-alive.

`Repos` now says in its types and its documentation that everything reaching
the disk takes a key and that `allowed_repos/0` and `valid_name?/1` are the
only two functions that speak in names. The coding lane's workspace resolved
its name the same wrong way and cloned from that shadow directory; it resolves
first now. `Pushes.mirror_storage_key/1` keeps its name and delegates, so there
is one resolver rather than two orders.

Nothing here deletes a directory. The stale `openagents.com.git` bare
repository on the live node is operator state, and the new test seeds one to
prove the verifier is not fooled by it and leaves it standing.

Closes #190.
Closes
#190

Deploy story

What this commit did to the running system — joined from the forge receipt chain, the part a commit page elsewhere cannot show.

pushed
by user · WAL seq 386 · 2026-08-25T14:50:18.286398Z

Changed files

  • modified INVARIANTS.md
  • modified docs/forge-exit-rehearsals.md
  • modified docs/taxonomy.md
  • modified lib/openagents/forge/pushes.ex
  • added lib/openagents/forge/repo_ref.ex
  • modified lib/openagents/forge/repos.ex
  • modified lib/openagents/forge/verification.ex
  • modified lib/openagents/tools/repository.ex
  • added test/openagents/forge/repo_ref_test.exs

Diff

9 files changed, +620 -92

INVARIANTS.md modified +31 -7

@@ -4487,11 +4487,33 @@ projection of that record. Whether the projection still matches is therefore a

4487 4487
question with an answer, and `OpenAgents.Forge.Verification` computes it from
4488 4488
the WAL and the repository alone.
4489 4489
4490
The verifier accepts either a `Repository.storage_key` or an `owner/name` path.
4491
An `owner/name` path is resolved to a storage key through the same repository
4492
mapping the serving path uses, and only the resolved key is checked against the
4493
WAL. The report names both the requested path and the resolved key so an operator
4494
sees the mapping.
4490
The verifier accepts a repository *name* — `openagents.com`, or the
4491
`owner/name` path a person clones — or a `Repository.storage_key`.
4492
`OpenAgents.Forge.RepoRef` is the one place the two are told apart, and it
4493
answers in a fixed order: a string under which the WAL holds a log is a storage
4494
key and resolves to itself, reaching no database; anything else is a name and
4495
is resolved through the same repository mapping the serving path uses,
4496
namespace aliases included. Only the resolved key is checked against the WAL,
4497
and the report names both the requested reference and the resolved key so an
4498
operator sees the mapping.
4499
4500
A reference that settles on no single repository stops there, and stops with a
4501
finding that says so: `repository_not_found`, `repository_name_ambiguous` for a
4502
name two repositories answer to, or `repository_lookup_unavailable` when the
4503
lookup a name needs could not be made. None of them turns the string into a
4504
path. That is the difference this rule now carries: before it, a name went
4505
straight into a path segment, and a name whose repository was stored under a
4506
different key reported `wal_unreadable` — the finding that means the log is
4507
gone — about a log that was intact. `Repos.allowed_repos/0` returns names, so
4508
the list an operator reads before verifying is a list of references the
4509
verifier accepts.
4510
4511
(Amended 2026-08-25, issue #190: the live forge's first repository is stored
4512
under an opaque key while its configured name is `openagents.com`, so the
4513
documented first step of rehearsal 2 verified nothing and said the log was
4514
missing. `REPOSITORY-001`'s sentence about that repository keeping the
4515
historical `openagents.com` key describes the migration's seed and no longer
4516
describes the live row.)
4495 4517
4496 4518
Independence here is structural, not a promise: the WAL-and-repository check
4497 4519
itself reaches no database, because a verifier that queried PostgreSQL for the

@@ -4529,8 +4551,10 @@ cache. This proves that divergence between the WAL and what is served is

4529 4551
detectable. The first is about replay; the second is about detection, and
4530 4552
neither substitutes for the other.
4531 4553
4532
Evidence: `OpenAgents.Forge.Verification`, `OpenAgents.Forge.WAL`,
4533
`OpenAgents.Forge.Repos`, and `test/openagents/forge/independence_test.exs`.
4554
Evidence: `OpenAgents.Forge.Verification`, `OpenAgents.Forge.RepoRef`,
4555
`OpenAgents.Forge.WAL`, `OpenAgents.Forge.Repos`,
4556
`test/openagents/forge/independence_test.exs`, and
4557
`test/openagents/forge/repo_ref_test.exs`.
4534 4558
4535 4559
### EXIT-003 — Recovery comes from the WAL, and the mirror is strictly lossy
4536 4560
docs/forge-exit-rehearsals.md modified +27 -3

@@ -200,12 +200,23 @@ any document, so it refutes a rewrite only for a reader who kept a copy.

200 200
201 201
### Steps
202 202
203
1. On the forge host, with no database available:
203
1. On the forge host:
204 204
205 205
   ```sh
206
   bin/openagents rpc 'OpenAgents.Forge.Verification.verify("{storage_key}") |> IO.inspect()'
206
   bin/openagents rpc 'OpenAgents.Forge.Verification.verify("OpenAgentsInc/openagents.com") |> IO.inspect()'
207 207
   ```
208 208
209
   The argument is the repository you clone — an `owner/name` path, or the
210
   name `OpenAgents.Forge.Repos.allowed_repos/0` lists.
211
   `OpenAgents.Forge.RepoRef` resolves it to the storage key the log is kept
212
   under, and the report names both, so you can see which repository was
213
   checked. A storage key still works and is the reference to use with no
214
   database available: resolving a *name* is the one step that reads the
215
   `repositories` table, and a key resolves against the WAL alone. A name that
216
   names no repository here, or two, is reported as `repository_not_found` or
217
   `repository_name_ambiguous` rather than checked as if it were a key
218
   (issue #190).
219
209 220
2. Expect `findings: []`. Each non-empty finding names one disagreement:
210 221
   `entry_object_missing`, `entry_digest_mismatch`, `entry_sequence_broken`,
211 222
   `served_refs_diverged`, `object_missing`, `chain_link_mismatch`, or

@@ -214,7 +225,7 @@ any document, so it refutes a rewrite only for a reader who kept a copy.

214 225
3. Anchor the check against a link you remember from an earlier run:
215 226
216 227
   ```sh
217
   bin/openagents rpc 'OpenAgents.Forge.Verification.verify("{storage_key}", anchor: %{seq: 41, link: "…"}) |> IO.inspect()'
228
   bin/openagents rpc 'OpenAgents.Forge.Verification.verify("OpenAgentsInc/openagents.com", anchor: %{seq: 41, link: "…"}) |> IO.inspect()'
218 229
   ```
219 230
220 231
   A rewritten prefix reports `anchor_mismatch`. Without the anchor argument

@@ -247,6 +258,19 @@ assumed: `OpenAgents.Forge.WAL` exports `chain_link/2` and `entry_link/1`, and

247 258
`GET /api/v1/repos/OpenAgentsInc/openagents.com/pushes` answers `200` and
248 259
serves the same head link and `chained_from` the verifier reports.
249 260
261
**Amended 2026-08-25 (issue #190).** Step 1 above used to say `{storage_key}`,
262
and the obvious value to substitute — the name
263
`OpenAgents.Forge.Repos.allowed_repos/0` returns, `openagents.com` — was not
264
one. It went into a path, the path held a bare repository that projects no log,
265
and the report was `wal_unreadable`: the finding that means "your write-ahead
266
log is missing", for a repository whose log is intact under the key
267
`ecd89cf6-f602-479f-9f47-266307345aaa`. `OpenAgents.Forge.RepoRef` now resolves
268
a name to a key before anything is read, so the step works with the name an
269
operator has, and a name that settles on no repository is
270
`repository_not_found` rather than an empty repository that looks half-alive.
271
The stale `openagents.com.git` bare repository on the live node is untouched by
272
that change: it is operator state, and removing it is an operator's decision.
273
250 274
The bound in this rehearsal's own preamble still holds, with one correction:
251 275
an anchor **is** published now, at
252 276
`/.well-known/openagents-forge-anchor.json`, and `GET /api/status` reports
docs/taxonomy.md modified +11

@@ -59,6 +59,17 @@ set of content-addressed, chained entry objects per repository, and it is the

59 59
only ref authority. PostgreSQL holds projections of it — push receipts — never
60 60
refs. A push is a receipt, not a deployment.
61 61
62
**Repository name and storage key** — two strings for one repository, and never
63
interchangeable. The **name** is what a person has: `openagents.com`, or the
64
`owner/name` path they clone, which is what `OpenAgents.Forge.Repos.allowed_repos/0`
65
lists and what a build or deploy receipt records. It is unique only inside a
66
namespace. The **storage key** is `Repository.storage_key`: opaque, unique
67
across the forge, and the single path segment the WAL keeps a log under and a
68
node keeps a bare repository under. Say which one you mean, and resolve a name
69
with `OpenAgents.Forge.RepoRef` before it reaches a path — a name is a legal
70
path segment, so using one as a key silently builds a directory that projects
71
nothing, which is what issue #190 found on the live node.
72
62 73
**MirrorWatch** — the component that exports accepted `main` commits from the
63 74
forge to GitHub. GitHub is a mirror only; nothing on GitHub can affect what
64 75
the forge serves.
lib/openagents/forge/pushes.ex modified +12 -25

@@ -29,7 +29,7 @@ defmodule OpenAgents.Forge.Pushes do

29 29
30 30
  alias OpenAgents.{Analytics, Repositories}
31 31
  alias OpenAgents.Accounts.User
32
  alias OpenAgents.Forge.{GitHTTP, PushReceipt, Repos, Sync, WAL}
32
  alias OpenAgents.Forge.{GitHTTP, PushReceipt, RepoRef, Repos, Sync, WAL}
33 33
  alias OpenAgents.Issues.ClosingReferences
34 34
  alias OpenAgents.Repo
35 35
  alias OpenAgents.Repositories.Repository

@@ -463,14 +463,17 @@ defmodule OpenAgents.Forge.Pushes do

463 463
    end
464 464
  end
465 465
466
  @doc "Resolve a configured repository name or storage key to its bare-cache storage key."
467
  def mirror_storage_key(repo) when is_binary(repo) do
468
    storage_key_for_storage_key(repo) || storage_key_for_name(repo) || repo
469
  rescue
470
    _database_unavailable -> repo
471
  end
472
473
  def mirror_storage_key(repo), do: repo
466
  @doc """
467
  Resolve a configured repository name or storage key to its bare-cache
468
  storage key.
469
470
  One name for the resolution every caller needs, so a name is turned into a
471
  key in one place: `OpenAgents.Forge.RepoRef.storage_key_or_ref/1`. A string
472
  that settles on no repository is returned unchanged, which is what a mirror
473
  push and a cache warm need — both act on a bare repository that may be keyed
474
  by the string itself.
475
  """
476
  def mirror_storage_key(repo), do: RepoRef.storage_key_or_ref(repo)
474 477
475 478
  @doc "Return the logical and canonical repository keys used by derived push receipts."
476 479
  def receipt_repo_keys(repo) when is_binary(repo) do

@@ -480,22 +483,6 @@ defmodule OpenAgents.Forge.Pushes do

480 483
481 484
  def receipt_repo_keys(repo), do: [repo]
482 485
483
  defp storage_key_for_storage_key(storage_key) do
484
    Repo.one(
485
      from repository in Repository,
486
        where: repository.storage_key == ^storage_key,
487
        select: repository.storage_key
488
    )
489
  end
490
491
  defp storage_key_for_name(name) do
492
    Repo.one(
493
      from repository in Repository,
494
        where: repository.name == ^name,
495
        select: repository.storage_key
496
    )
497
  end
498
499 486
  defp repository_name(storage_key) when is_binary(storage_key) do
500 487
    Repo.one(
501 488
      from repository in Repository,
lib/openagents/forge/repo_ref.ex added +162

@@ -0,0 +1,162 @@

1
defmodule OpenAgents.Forge.RepoRef do
2
  @moduledoc """
3
  The one place a repository *name* becomes a repository *storage key*.
4
5
  Two different strings identify a repository here, and confusing them is what
6
  issue #190 was:
7
8
    * A **name** is what a person has. It is `openagents.com`, or the
9
      `owner/name` path they clone — `OpenAgentsInc/openagents.com`. It is what
10
      `OpenAgents.Forge.Repos.allowed_repos/0` lists, what a Git URL carries,
11
      and what `OpenAgents.Forge.Targets` records on a receipt. A name is not
12
      unique on its own: two namespaces can each own a repository called
13
      `docs`.
14
    * A **storage key** is what the forge keys durable state with. It is
15
      `Repository.storage_key`, it is unique, it is opaque — a UUID for every
16
      repository created since `REPOSITORY-001` — and it is the single path
17
      segment under which the WAL keeps a log
18
      (`OpenAgents.Forge.WAL`) and a node keeps its bare repository
19
      (`OpenAgents.Forge.Repos.bare_path/1`).
20
21
  Passing a name where a storage key belongs does not fail loudly. Both are
22
  strings, and a name is a legal path segment, so the forge builds a path out
23
  of it and finds an empty directory or nothing at all. That is how
24
  `Verification.verify("openagents.com")` came to report `wal_unreadable` for a
25
  repository whose log was intact, and how a bare repository under the *name*
26
  came to sit beside the one under the key.
27
28
  ## Resolution order
29
30
  `storage_key/1` answers in this order, and the order is the point:
31
32
    1. If the string is a legal storage key *and* the WAL holds a log under it,
33
       it is a storage key and it resolves to itself. This step reaches no
34
       database, so a verifier handed a key stays independent of PostgreSQL —
35
       `EXIT-002` — and a repository whose log predates the `repositories`
36
       table still resolves.
37
    2. Otherwise the string is treated as a name and looked up in
38
       `repositories`, by storage key, by `name`, or by the `owner/name` path
39
       the serving path resolves (`OpenAgents.Forge.GitHTTP`), namespace
40
       aliases included. Exactly one repository is an answer; two is
41
       `:ambiguous_name`, because guessing which of two repositories an
42
       operator meant is worse than saying the name is not enough.
43
    3. Otherwise `:repository_not_found`.
44
45
  A database that cannot be reached during step 2 is `:repository_lookup_unavailable`
46
  rather than "not found": a name whose answer is unknown must not be reported
47
  as a name with no answer.
48
  """
49
50
  import Ecto.Query
51
52
  alias OpenAgents.Forge.{Repos, WAL}
53
  alias OpenAgents.Repo
54
  alias OpenAgents.Repositories.{NamespaceAlias, Repository}
55
56
  @typedoc "What a person has: a repository `name`, or an `owner/name` path."
57
  @type name :: String.t()
58
59
  @typedoc """
60
  What the forge keys durable state with: one opaque, unique path segment.
61
  """
62
  @type storage_key :: String.t()
63
64
  @typedoc "Either of the two, before anyone has decided which it is."
65
  @type ref :: name() | storage_key()
66
67
  @typedoc "Why a reference names no single repository."
68
  @type resolution_error ::
69
          :repository_not_found | :ambiguous_name | :repository_lookup_unavailable
70
71
  @doc """
72
  The storage key `ref` names.
73
74
  Returns `{:ok, storage_key}`, or `{:error, reason}` where `reason` is one of
75
  the `t:resolution_error/0` values. See the module documentation for the
76
  order in which the two interpretations are tried.
77
  """
78
  @spec storage_key(term()) :: {:ok, storage_key()} | {:error, resolution_error()}
79
  def storage_key(ref) when is_binary(ref) do
80
    if wal_key?(ref) do
81
      {:ok, ref}
82
    else
83
      resolve_name(ref)
84
    end
85
  end
86
87
  def storage_key(_ref), do: {:error, :repository_not_found}
88
89
  @doc """
90
  The storage key `ref` names, or `ref` itself when nothing settles it.
91
92
  For the callers whose honest fallback is the string they were given: a mirror
93
  push and a cache warm both act on a bare repository that may well be keyed by
94
  the string itself, and neither is in a position to refuse. A caller that
95
  reports to a person should use `storage_key/1` instead, so an unresolved name
96
  is named as one rather than turned into a path.
97
  """
98
  @spec storage_key_or_ref(term()) :: term()
99
  def storage_key_or_ref(ref) when is_binary(ref) do
100
    case storage_key(ref) do
101
      {:ok, resolved} -> resolved
102
      {:error, _reason} -> ref
103
    end
104
  end
105
106
  def storage_key_or_ref(ref), do: ref
107
108
  # A log under this exact key is the only evidence that settles a string as a
109
  # storage key without asking the database anything.
110
  defp wal_key?(ref) do
111
    Repos.valid_storage_key?(ref) and match?({:ok, _generation, _index}, WAL.read_index(ref))
112
  end
113
114
  defp resolve_name(ref) do
115
    case candidate_keys(ref) do
116
      [storage_key] -> {:ok, storage_key}
117
      [] -> {:error, :repository_not_found}
118
      _two_or_more -> {:error, :ambiguous_name}
119
    end
120
  rescue
121
    _database_unavailable -> {:error, :repository_lookup_unavailable}
122
  end
123
124
  defp candidate_keys(ref) do
125
    case String.split(ref, "/") do
126
      [owner, name] -> owner |> path_query(name) |> keys()
127
      [_bare_name] -> ref |> bare_query() |> keys()
128
      _not_a_reference -> []
129
    end
130
  end
131
132
  defp keys(query) do
133
    query
134
    |> select([repository], repository.storage_key)
135
    |> limit(2)
136
    |> Repo.all()
137
    |> Enum.uniq()
138
  end
139
140
  # The same mapping the serving path resolves a clone URL through
141
  # (`OpenAgents.Forge.GitHTTP`), including renamed namespaces, so an operator
142
  # verifies the repository their `git clone` reached and not a different one.
143
  defp path_query(owner, name) do
144
    owner_key = String.downcase(owner)
145
    name_key = String.downcase(name)
146
147
    from repository in Repository,
148
      join: namespace in assoc(repository, :namespace),
149
      left_join: namespace_alias in NamespaceAlias,
150
      on: namespace_alias.namespace_id == namespace.id and namespace_alias.slug_key == ^owner_key,
151
      where:
152
        repository.name_key == ^name_key and
153
          (namespace.slug_key == ^owner_key or not is_nil(namespace_alias.id))
154
  end
155
156
  defp bare_query(ref) do
157
    name_key = String.downcase(ref)
158
159
    from repository in Repository,
160
      where: repository.storage_key == ^ref or repository.name_key == ^name_key
161
  end
162
end
lib/openagents/forge/repos.ex modified +59 -16

@@ -6,10 +6,30 @@ defmodule OpenAgents.Forge.Repos do

6 6
7 7
  All git invocations are argv-only against `--git-dir`; no shell strings
8 8
  ever carry request data.
9
10
  ## Names are not storage keys
11
12
  Every function here that reaches the disk takes a **storage key** — the
13
  opaque, unique segment `Repository.storage_key` holds — and never a
14
  repository **name**. `allowed_repos/0` and `valid_name?/1` are the only two
15
  that speak in names, and a name is what a person has: `openagents.com`, or
16
  the `owner/name` path they clone.
17
18
  The two are both strings and a name is a legal path segment, so passing one
19
  where the other belongs builds a path rather than failing. `bare_path/1` on a
20
  name produces a directory beside the real repository that projects nothing —
21
  which is what issue #190 found on the live node. `OpenAgents.Forge.RepoRef`
22
  is the one place a name becomes a key; call it before calling anything here.
9 23
  """
10 24
11 25
  require Logger
12 26
27
  @typedoc "What a person has: a repository name, or an `owner/name` path."
28
  @type name :: String.t()
29
30
  @typedoc "What this module keys every path with: see `OpenAgents.Forge.RepoRef`."
31
  @type storage_key :: String.t()
32
13 33
  @name_pattern ~r/^[a-z0-9](?:[a-z0-9_-]|\.(?=[a-z0-9])){0,63}$/
14 34
  @storage_key_pattern ~r/\A[A-Za-z0-9][A-Za-z0-9._-]{0,127}\z/
15 35

@@ -19,27 +39,44 @@ defmodule OpenAgents.Forge.Repos do

19 39
      "/var/lib/openagents/forge"
20 40
  end
21 41
22
  @doc "Repositories this forge serves. Bounded, config-owned."
42
  @doc """
43
  Repository *names* this forge serves operationally. Bounded, config-owned.
44
45
  These are names, not storage keys: `["openagents.com"]` is the name of the
46
  repository whose storage key is a UUID. Resolve one with
47
  `OpenAgents.Forge.RepoRef.storage_key/1` before using it as a path segment.
48
  """
49
  @spec allowed_repos() :: [name()]
23 50
  def allowed_repos do
24 51
    Application.get_env(:openagents, :forge_repos, ["openagents.com"])
25 52
  end
26 53
54
  @doc "Whether `name` is a well-formed name this forge serves operationally."
55
  @spec valid_name?(term()) :: boolean()
27 56
  def valid_name?(name) when is_binary(name) do
28 57
    Regex.match?(@name_pattern, name) and name in allowed_repos()
29 58
  end
30 59
31 60
  def valid_name?(_), do: false
32 61
33
  @doc "Whether an opaque repository storage key is safe as one path segment."
62
  @doc """
63
  Whether an opaque repository storage key is safe as one path segment.
64
65
  Shape only. A repository name is shaped like a storage key, so this admits
66
  one; only `OpenAgents.Forge.RepoRef` can tell you which you are holding.
67
  """
68
  @spec valid_storage_key?(term()) :: boolean()
34 69
  def valid_storage_key?(storage_key) when is_binary(storage_key),
35 70
    do: Regex.match?(@storage_key_pattern, storage_key)
36 71
37 72
  def valid_storage_key?(_storage_key), do: false
38 73
39
  @doc "Absolute path of the bare repository for `repo`."
40
  def bare_path(repo), do: Path.join([data_dir(), "repos", repo <> ".git"])
74
  @doc "Absolute path of the bare repository for `storage_key`."
75
  @spec bare_path(storage_key()) :: String.t()
76
  def bare_path(storage_key), do: Path.join([data_dir(), "repos", storage_key <> ".git"])
41 77
42 78
  @doc "Delete one repository's disposable local bare-repository cache."
79
  @spec delete_repo(storage_key()) :: :ok | {:error, term()}
43 80
  def delete_repo(storage_key) do
44 81
    if valid_storage_key?(storage_key) do
45 82
      case File.rm_rf(bare_path(storage_key)) do

@@ -52,8 +89,9 @@ defmodule OpenAgents.Forge.Repos do

52 89
  end
53 90
54 91
  @doc "Initialize the bare repository if absent. Returns the path."
55
  def ensure_repo!(repo, default_branch \\ "main") do
56
    repo |> bare_path() |> ensure_repo_at!(default_branch)
92
  @spec ensure_repo!(storage_key(), String.t()) :: String.t()
93
  def ensure_repo!(storage_key, default_branch \\ "main") do
94
    storage_key |> bare_path() |> ensure_repo_at!(default_branch)
57 95
  end
58 96
59 97
  @doc false

@@ -101,8 +139,9 @@ defmodule OpenAgents.Forge.Repos do

101 139
    match?({"true" <> _rest, 0}, git(path, ["rev-parse", "--is-bare-repository"]))
102 140
  end
103 141
104
  def set_default_branch!(repo, default_branch) do
105
    repo |> bare_path() |> set_default_branch_at!(default_branch)
142
  @spec set_default_branch!(storage_key(), String.t()) :: :ok
143
  def set_default_branch!(storage_key, default_branch) do
144
    storage_key |> bare_path() |> set_default_branch_at!(default_branch)
106 145
  end
107 146
108 147
  @doc false

@@ -125,8 +164,9 @@ defmodule OpenAgents.Forge.Repos do

125 164
  end
126 165
127 166
  @doc "Current refs of the bare repo as a `%{name => sha}` map."
128
  def refs(repo) do
129
    repo |> bare_path() |> refs_at()
167
  @spec refs(storage_key()) :: %{String.t() => String.t()}
168
  def refs(storage_key) do
169
    storage_key |> bare_path() |> refs_at()
130 170
  end
131 171
132 172
  @doc false

@@ -150,8 +190,9 @@ defmodule OpenAgents.Forge.Repos do

150 190
  rollback and WAL materialization convergence). Deletes refs not present
151 191
  in the target.
152 192
  """
153
  def set_refs!(repo, target_refs) when is_map(target_refs) do
154
    repo |> bare_path() |> set_refs_at!(target_refs)
193
  @spec set_refs!(storage_key(), map()) :: :ok
194
  def set_refs!(storage_key, target_refs) when is_map(target_refs) do
195
    storage_key |> bare_path() |> set_refs_at!(target_refs)
155 196
  end
156 197
157 198
  @doc false

@@ -177,8 +218,9 @@ defmodule OpenAgents.Forge.Repos do

177 218
  end
178 219
179 220
  @doc "The WAL sequence this bare repo has applied (cache freshness marker)."
180
  def applied_seq(repo) do
181
    repo |> bare_path() |> applied_seq_at()
221
  @spec applied_seq(storage_key()) :: integer()
222
  def applied_seq(storage_key) do
223
    storage_key |> bare_path() |> applied_seq_at()
182 224
  end
183 225
184 226
  @doc false

@@ -195,8 +237,9 @@ defmodule OpenAgents.Forge.Repos do

195 237
    end
196 238
  end
197 239
198
  def record_applied_seq!(repo, seq) when is_integer(seq) do
199
    repo |> bare_path() |> record_applied_seq_at!(seq)
240
  @spec record_applied_seq!(storage_key(), integer()) :: :ok
241
  def record_applied_seq!(storage_key, seq) when is_integer(seq) do
242
    storage_key |> bare_path() |> record_applied_seq_at!(seq)
200 243
  end
201 244
202 245
  @doc false
lib/openagents/forge/verification.ex modified +56 -38

@@ -37,6 +37,16 @@ defmodule OpenAgents.Forge.Verification do

37 37
  * `anchor_unreachable` — the caller supplied an anchor for a sequence this
38 38
    log does not have, or for an entry that carries no link.
39 39
40
  Three more findings come before any of that, because a check of the wrong
41
  repository is worse than no check. The caller names a repository by storage
42
  key or by name, `OpenAgents.Forge.RepoRef` resolves the two apart, and a
43
  reference that settles on no single repository stops there:
44
  `repository_not_found`, `repository_name_ambiguous` for a name two
45
  repositories answer to, and `repository_lookup_unavailable` when the lookup
46
  a name needs could not be made. None of them builds a path out of the string,
47
  which is what made an absent repository report `wal_unreadable` — the finding
48
  that means "your log is gone" — for a log that was never there (issue #190).
49
40 50
  What this cannot do is stated as plainly as what it can. Content addressing
41 51
  and the chain make tampering *evident*, not *impossible*. An operator who
42 52
  rewrites an entry, its key, the index, and every link after it produces a

@@ -53,7 +63,7 @@ defmodule OpenAgents.Forge.Verification do

53 63
  only an operator who serves something other than what was pushed.
54 64
  """
55 65
56
  alias OpenAgents.Forge.{ReceiptRepository, Repos, WAL}
66
  alias OpenAgents.Forge.{RepoRef, Repos, WAL}
57 67
58 68
  @internal_ref_prefix "refs/internal/"
59 69

@@ -65,8 +75,8 @@ defmodule OpenAgents.Forge.Verification do

65 75
66 76
  @typedoc "The verification outcome for one repository."
67 77
  @type report :: %{
68
          repo: String.t(),
69
          storage_key: String.t() | nil,
78
          repo: RepoRef.ref(),
79
          storage_key: RepoRef.storage_key() | nil,
70 80
          entries: non_neg_integer(),
71 81
          findings: [finding()],
72 82
          head: anchor() | nil,

@@ -76,6 +86,15 @@ defmodule OpenAgents.Forge.Verification do

76 86
  @doc """
77 87
  Verify one repository's served state against its WAL.
78 88
89
  `repo_ref` is either a storage key or a name — the `openagents.com` or
90
  `OpenAgentsInc/openagents.com` a person actually has. `OpenAgents.Forge.RepoRef`
91
  resolves the two apart, and the report names both what was asked for
92
  (`:repo`) and the key that was checked (`:storage_key`), so the mapping is
93
  visible rather than assumed. A name that names no repository, or two, is a
94
  finding — `repository_not_found` or `repository_name_ambiguous` — and never a
95
  path built out of the name, which is what made an absent repository look
96
  half-alive (issue #190).
97
79 98
  Returns `{:ok, report}` when the two agree and `{:error, report}` when they
80 99
  do not. The report always carries the findings list so a caller can render
81 100
  every disagreement, not only the first.

@@ -92,47 +111,46 @@ defmodule OpenAgents.Forge.Verification do

92 111
  `:chained_from`, the first sequence that carries a link. Entries before that
93 112
  sequence predate the chain and are not covered by it.
94 113
  """
95
  @spec verify(String.t(), keyword()) :: {:ok, report()} | {:error, report()}
96
  def verify(repo_or_path, opts \\ []) when is_binary(repo_or_path) and is_list(opts) do
97
    {repo, storage_key} = resolve_repo_or_path(repo_or_path)
98
99
    if is_nil(storage_key) do
100
      report(repo, nil, [], [finding("repository_not_found", %{"repo" => repo})])
101
    else
102
      case WAL.read_index(storage_key) do
103
        {:ok, _generation, index} ->
104
          entries = WAL.entries(index)
105
106
          findings =
107
            sequence_findings(entries) ++
108
              entry_findings(storage_key, entries) ++
109
              ref_findings(storage_key, index) ++
110
              object_findings(storage_key, entries) ++
111
              reachability_findings(storage_key, index) ++
112
              chain_findings(entries) ++
113
              anchor_findings(entries, normalize_anchor(opts[:anchor]))
114
115
          report(repo, storage_key, entries, findings)
116
117
        {:error, reason} ->
118
          report(repo, storage_key, [], [
119
            finding("wal_unreadable", %{"reason" => inspect(reason)})
120
          ])
121
      end
114
  @spec verify(RepoRef.ref(), keyword()) :: {:ok, report()} | {:error, report()}
115
  def verify(repo_ref, opts \\ []) when is_binary(repo_ref) and is_list(opts) do
116
    case RepoRef.storage_key(repo_ref) do
117
      {:ok, storage_key} -> verify_storage_key(repo_ref, storage_key, opts)
118
      {:error, reason} -> report(repo_ref, nil, [], [resolution_finding(repo_ref, reason)])
122 119
    end
123 120
  end
124 121
125
  defp resolve_repo_or_path(repo) when is_binary(repo) do
126
    if String.contains?(repo, "/") do
127
      case ReceiptRepository.resolve(repo) do
128
        %{storage_key: storage_key} -> {repo, storage_key}
129
        nil -> {repo, nil}
130
      end
131
    else
132
      {repo, repo}
122
  defp verify_storage_key(repo_ref, storage_key, opts) do
123
    case WAL.read_index(storage_key) do
124
      {:ok, _generation, index} ->
125
        entries = WAL.entries(index)
126
127
        findings =
128
          sequence_findings(entries) ++
129
            entry_findings(storage_key, entries) ++
130
            ref_findings(storage_key, index) ++
131
            object_findings(storage_key, entries) ++
132
            reachability_findings(storage_key, index) ++
133
            chain_findings(entries) ++
134
            anchor_findings(entries, normalize_anchor(opts[:anchor]))
135
136
        report(repo_ref, storage_key, entries, findings)
137
138
      {:error, reason} ->
139
        report(repo_ref, storage_key, [], [
140
          finding("wal_unreadable", %{"reason" => inspect(reason)})
141
        ])
133 142
    end
134 143
  end
135 144
145
  defp resolution_finding(repo_ref, :ambiguous_name),
146
    do: finding("repository_name_ambiguous", %{"repo" => repo_ref})
147
148
  defp resolution_finding(repo_ref, :repository_lookup_unavailable),
149
    do: finding("repository_lookup_unavailable", %{"repo" => repo_ref})
150
151
  defp resolution_finding(repo_ref, _not_found),
152
    do: finding("repository_not_found", %{"repo" => repo_ref})
153
136 154
  defp report(repo, storage_key, entries, findings) do
137 155
    report = %{
138 156
      repo: repo,
lib/openagents/tools/repository.ex modified +21 -3

@@ -16,6 +16,7 @@ defmodule OpenAgents.Tools.Repository do

16 16
  inside its root before any filesystem call.
17 17
  """
18 18
19
  alias OpenAgents.Forge.RepoRef
19 20
  alias OpenAgents.Forge.Repos
20 21
  alias OpenAgents.Forge.Sync
21 22

@@ -35,9 +36,25 @@ defmodule OpenAgents.Tools.Repository do

35 36
    )
36 37
  end
37 38
38
  @doc "The forge repo the coding lane edits."
39
  @doc """
40
  The *name* of the forge repository the coding lane edits.
41
42
  A name, not a storage key: receipts and targets record this string, and
43
  anything that reaches the disk resolves it first with `storage_key/0`.
44
  """
39 45
  def repo, do: @repo
40 46
47
  @doc """
48
  The storage key the coding lane's repository is stored under.
49
50
  `repo/0` is a name and the forge keys durable state by
51
  `Repository.storage_key`, so the two are only the same string for a
52
  repository old enough to predate the split. Cloning from the name's path
53
  instead of the key's is how a workspace came to be cut from a bare
54
  repository projecting nothing (issue #190).
55
  """
56
  def storage_key, do: RepoRef.storage_key_or_ref(@repo)
57
41 58
  @doc "This job's clone directory (may not exist yet)."
42 59
  def workspace_dir("work-job:" <> job_id), do: Path.join(jobs_dir(), "job-" <> job_id)
43 60

@@ -54,8 +71,9 @@ defmodule OpenAgents.Tools.Repository do

54 71
    if File.dir?(Path.join(dir, ".git")) do
55 72
      {:ok, dir}
56 73
    else
57
      Sync.ensure_fresh(@repo)
58
      bare = Repos.bare_path(@repo)
74
      storage_key = storage_key()
75
      Sync.ensure_fresh(storage_key)
76
      bare = Repos.bare_path(storage_key)
59 77
      File.mkdir_p!(jobs_dir())
60 78
61 79
      case System.cmd("git", ["clone", "--quiet", bare, dir], stderr_to_stdout: true) do
test/openagents/forge/repo_ref_test.exs added +241

@@ -0,0 +1,241 @@

1
defmodule OpenAgents.Forge.RepoRefTest do
2
  @moduledoc """
3
  Issue #190: the name a person is told to use and the key the forge stores
4
  under are different strings, and the verifier is the surface where confusing
5
  them does the most damage.
6
7
  A rehearsal reads `OpenAgents.Forge.Repos.allowed_repos/0`, gets a name, and
8
  runs the verifier on it. Before this, the name went straight into a path, the
9
  path held a bare repository projecting nothing, and the report said
10
  `wal_unreadable` — "your write-ahead log is gone" — about a log that was
11
  intact under a different key. These tests hold the name-to-key step in place
12
  and hold the shadow directory harmless.
13
  """
14
15
  use OpenAgents.DataCase, async: false
16
17
  alias OpenAgents.Forge.{CacheReadiness, RepoRef, Repos, Sync, Verification, WAL}
18
19
  @name "openagents.com"
20
  @owner "OpenAgentsInc"
21
22
  setup do
23
    Ecto.Adapters.SQL.Sandbox.mode(OpenAgents.Repo, {:shared, self()})
24
25
    root =
26
      Path.join(
27
        System.tmp_dir!(),
28
        "forge-repo-ref-#{System.unique_integer([:positive, :monotonic])}"
29
      )
30
31
    previous_data = Application.get_env(:openagents, :forge_data_dir)
32
    previous_wal = Application.get_env(:openagents, :forge_wal_dir)
33
    previous_adapter = Application.get_env(:openagents, :forge_wal_adapter)
34
    previous_repos = Application.get_env(:openagents, :forge_repos)
35
36
    Application.put_env(:openagents, :forge_data_dir, Path.join(root, "data"))
37
    Application.put_env(:openagents, :forge_wal_dir, Path.join(root, "wal"))
38
    Application.put_env(:openagents, :forge_wal_adapter, OpenAgents.Forge.WAL.Local)
39
    Application.put_env(:openagents, :forge_repos, [@name])
40
    CacheReadiness.reset()
41
42
    on_exit(fn ->
43
      restore(:forge_data_dir, previous_data)
44
      restore(:forge_wal_dir, previous_wal)
45
      restore(:forge_wal_adapter, previous_adapter)
46
      restore(:forge_repos, previous_repos)
47
      CacheReadiness.reset()
48
      File.rm_rf!(root)
49
    end)
50
51
    # Production's shape, exactly: the first repository's row has moved off the
52
    # historical name-as-key and onto an opaque one, while the name it is
53
    # cloned and configured by has not moved at all.
54
    storage_key = Ecto.UUID.generate()
55
56
    repository =
57
      OpenAgents.Repo.get_by!(OpenAgents.Repositories.Repository,
58
        owner_key: String.downcase(@owner),
59
        name_key: @name
60
      )
61
62
    repository
63
    |> Ecto.Changeset.change(storage_key: storage_key)
64
    |> OpenAgents.Repo.update!()
65
66
    sha = seed_wal!(root, storage_key)
67
68
    %{root: root, storage_key: storage_key, sha: sha}
69
  end
70
71
  describe "the name a rehearsal is handed" do
72
    test "verifies the repository the forge actually serves", context do
73
      assert {:ok, report} = Verification.verify(@name)
74
75
      assert report.repo == @name
76
      assert report.storage_key == context.storage_key
77
      assert report.entries == 1
78
      assert report.findings == []
79
    end
80
81
    test "is the name the configuration admits", context do
82
      # `allowed_repos/0` is the list an operator reads before verifying, so
83
      # what it returns has to be a reference the verifier accepts.
84
      assert [name] = Repos.allowed_repos()
85
      assert {:ok, %{storage_key: storage_key}} = Verification.verify(name)
86
      assert storage_key == context.storage_key
87
    end
88
89
    test "resolves the same through the owner/name path, in any case", context do
90
      assert {:ok, %{storage_key: key}} = Verification.verify("#{@owner}/#{@name}")
91
      assert key == context.storage_key
92
93
      assert {:ok, %{storage_key: downcased}} =
94
               Verification.verify("openagentsinc/#{@name}")
95
96
      assert downcased == context.storage_key
97
    end
98
99
    test "is not shadowed by a bare repository standing under it", context do
100
      # The live node carries exactly this: a bare repository under the *name*,
101
      # holding one ref, beside the one under the key. It is a projection of no
102
      # log, so it must not be able to answer for the repository.
103
      shadow = seed_shadow!(context.root, @name)
104
      refute Repos.refs_at(shadow)["refs/heads/main"] == context.sha
105
106
      assert {:ok, report} = Verification.verify(@name)
107
      assert report.storage_key == context.storage_key
108
      assert report.findings == []
109
      assert report.entries == 1
110
111
      # And the shadow is still there afterwards: cleaning up production state
112
      # is an operator's decision, not a side effect of reading it.
113
      assert File.dir?(shadow)
114
    end
115
  end
116
117
  describe "a reference that names no repository" do
118
    test "is a typed finding rather than an empty repository", _context do
119
      assert {:error, report} = Verification.verify("no-such-repository")
120
121
      assert report.storage_key == nil
122
      assert report.entries == 0
123
124
      assert [%{code: "repository_not_found", detail: %{"repo" => "no-such-repository"}}] =
125
               report.findings
126
127
      # The old answer. `wal_unreadable` means the log of a known repository
128
      # could not be read, which is a very different report from "that name is
129
      # not a repository here".
130
      refute Enum.any?(report.findings, &(&1.code == "wal_unreadable"))
131
    end
132
133
    test "reports an unknown owner/name path the same way", _context do
134
      assert {:error, report} = Verification.verify("#{@owner}/no-such-repository")
135
      assert report.storage_key == nil
136
      assert [%{code: "repository_not_found"}] = report.findings
137
    end
138
139
    test "reports a name two repositories answer to as ambiguous", _context do
140
      for owner <- ["FirstOwner", "SecondOwner"] do
141
        {:ok, _repository} =
142
          OpenAgents.Repositories.create_repository(%{
143
            owner: owner,
144
            name: "shared",
145
            visibility: "public",
146
            default_branch: "main",
147
            storage_key: Ecto.UUID.generate()
148
          })
149
      end
150
151
      assert {:error, %{storage_key: nil, findings: findings}} = Verification.verify("shared")
152
      assert [%{code: "repository_name_ambiguous", detail: %{"repo" => "shared"}}] = findings
153
154
      # Naming the owner settles it.
155
      assert {:error, %{storage_key: storage_key, findings: settled}} =
156
               Verification.verify("FirstOwner/shared")
157
158
      refute is_nil(storage_key)
159
      assert [%{code: "wal_unreadable"}] = settled
160
    end
161
  end
162
163
  describe "a storage key" do
164
    test "resolves to itself without consulting the repositories table", context do
165
      assert RepoRef.storage_key(context.storage_key) == {:ok, context.storage_key}
166
      assert {:ok, %{repo: repo, storage_key: key}} = Verification.verify(context.storage_key)
167
      assert repo == context.storage_key
168
      assert key == context.storage_key
169
    end
170
171
    test "resolves to itself when its log predates the repositories table", context do
172
      # A repository whose key is its own name and which has no row at all —
173
      # the shape of the forge's oldest logs. The WAL is what settles it.
174
      _sha = seed_wal!(context.root, "legacy-key")
175
176
      assert RepoRef.storage_key("legacy-key") == {:ok, "legacy-key"}
177
      assert {:ok, %{findings: [], storage_key: "legacy-key"}} = Verification.verify("legacy-key")
178
    end
179
  end
180
181
  # The live node's shadow: a bare repository under the *name*, holding one ref
182
  # at a commit the served repository never had, and no WAL of its own.
183
  defp seed_shadow!(root, name) do
184
    stale = Path.join(root, "stale-source")
185
    File.mkdir_p!(stale)
186
    git!(stale, ["init", "--initial-branch=main"])
187
    git!(stale, ["config", "user.email", "test@example.com"])
188
    git!(stale, ["config", "user.name", "Forge test"])
189
    File.write!(Path.join(stale, "README.md"), "left behind\n")
190
    git!(stale, ["add", "README.md"])
191
    git!(stale, ["commit", "-m", "A commit the served repository passed long ago"])
192
193
    shadow = Repos.ensure_repo!(name)
194
    {_output, 0} = Repos.git(shadow, ["fetch", stale, "main:refs/heads/main"])
195
196
    shadow
197
  end
198
199
  defp restore(key, nil), do: Application.delete_env(:openagents, key)
200
  defp restore(key, value), do: Application.put_env(:openagents, key, value)
201
202
  # One real commit, bundled, recorded as one WAL entry, materialized into the
203
  # bare repository the way a push would leave it.
204
  defp seed_wal!(root, storage_key) do
205
    source = Path.join(root, "source-#{storage_key}")
206
    File.mkdir_p!(source)
207
    git!(source, ["init", "--initial-branch=main"])
208
    git!(source, ["config", "user.email", "test@example.com"])
209
    git!(source, ["config", "user.name", "Forge test"])
210
    File.write!(Path.join(source, "README.md"), "served\n")
211
    git!(source, ["add", "README.md"])
212
    git!(source, ["commit", "-m", "Served commit"])
213
214
    sha = source |> git!(["rev-parse", "HEAD"]) |> String.trim()
215
    bundle = Path.join(root, "#{storage_key}.bundle")
216
    git!(source, ["bundle", "create", bundle, "--all"])
217
218
    {:ok, object} = WAL.put_entry_file(storage_key, 0, bundle)
219
220
    entry = %{
221
      "seq" => 0,
222
      "object" => object,
223
      "format" => "git_bundle",
224
      "refs" => %{"refs/heads/main" => sha},
225
      "principal" => "test:repo-ref",
226
      "pushed_at" => DateTime.to_iso8601(DateTime.utc_now())
227
    }
228
229
    {:ok, _generation} =
230
      WAL.cas_index(storage_key, :none, WAL.append_entry(WAL.new_index(), entry))
231
232
    :ok = Sync.ensure_fresh(storage_key)
233
234
    sha
235
  end
236
237
  defp git!(dir, args) do
238
    {output, 0} = System.cmd("git", ["-C", dir | args], stderr_to_stdout: true)
239
    output
240
  end
241
end

This page updates live while a promote is in flight · changelog