fix: refuse a staging or production boot on a borrowed vault key

342720ea8bb4 · AtlantisPleb · · parent 6fd8b921949f

fix: refuse a staging or production boot on a borrowed vault key

VAULT-001 says each vault seals under its own key, and until now nothing
checked the "own" half. `config/runtime.exs` bridges an unset
`MACHINE_TOKEN_ENCRYPTION_KEY` to the GitHub vault's active key so the deploy
that introduced the variable could boot before an operator provisioned the
secret. A bridged boot presents a valid 32-byte key that belongs to another
vault, so the existing presence check passes and a GitHub rotation quietly
moves the pairing vault with it. That is correct on the day it ships and wrong
a week later, which is exactly what #253 measured.

`RuntimeConfig.validate/1` now compares every configured vault key and refuses
a staging or production boot on any duplicate, naming the borrowing setting and
the one it borrowed from. Distinctness is the predicate rather than "did the
bridge fire", so it also refuses a dedicated secret provisioned by hand with
another vault's value. The bridge stays in `config/runtime.exs`: configuration
still assembles everywhere, and development and test stay exempt because both
already configure distinct keys and a shared key there strands nothing.

Production is already correct. A read of all three fleet nodes on 2026-08-25,
without printing key material, answers `machine_set=true` and
`same_as_github=false`, with all four vault keys set and pairwise distinct. So
the docs that still described the bridge as production's configuration are
corrected here too: rehearsal 4 of `docs/forge-exit-rehearsals.md`, VAULT-001,
the pairing vault's moduledoc, the secrets table, and the dated rehearsal
inventory, which gets a follow-up note rather than a rewritten reading.

Closes #253.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SoZMfWRSGnf6FZX2Ar9rQ2
Co-Authored-By
Claude Fable 5 <noreply@anthropic.com>
Closes
#253

Deploy story

What this commit did to the running system — joined from the forge receipt chain, the part a commit page elsewhere cannot show.

Not deployed through the forge lane

No push, promotion, build, or deploy receipt references this commit (receipts are scanned over a bounded recent window). Changes shipped by full node replacement carry their proof in the release gate receipt instead.

Changed files

  • modified INVARIANTS.md
  • modified docs/2026-08-25-forge-exit-rehearsals-2-to-6.md
  • modified docs/forge-exit-rehearsals.md
  • modified docs/security/secrets-and-log-handling.md
  • modified lib/openagents/machines/token_vault.ex
  • modified lib/openagents/runtime_config.ex
  • modified test/openagents/runtime_config_test.exs

Diff

7 files changed, +156 -11

INVARIANTS.md modified +20 -4

@@ -3532,10 +3532,17 @@ documented GitHub rotation would have unread every outstanding pairing.

3532 3532
Decrypt-side compatibility is narrower than sealing and stays explicit: the
3533 3533
pairing vault opens with its dedicated key first and falls back to the GitHub
3534 3534
keyring — active key plus `:github_token_decryption_keys` — because that is
3535
the only key material its historical records were sealed under, and because
3536
`config/runtime.exs` deliberately bridges an unset
3537
`MACHINE_TOKEN_ENCRYPTION_KEY` to the GitHub key until the operator
3538
provisions the dedicated secret. The fallback never rewraps: the only reader
3535
the only key material its historical records were sealed under, including
3536
every record sealed while `config/runtime.exs` bridged an unset
3537
`MACHINE_TOKEN_ENCRYPTION_KEY` to the GitHub key. That bridge is no longer the
3538
production configuration: the dedicated secret is provisioned, and the bridge
3539
is now only the fallback for a node that comes up without it. It cannot stay
3540
load-bearing unnoticed, because `RuntimeConfig.validate/1` refuses a staging or
3541
production boot in which any two vault keys are equal — presence alone never
3542
distinguished a provisioned key from a bridged one, which is how the bridge
3543
outlived by a week the single deploy it was written for (#253). The bridge
3544
survives in `config/runtime.exs` so configuration still assembles; what it can
3545
no longer do is serve. The fallback never rewraps: the only reader
3539 3546
nulls `token_ciphertext` in the same transaction as a successful open
3540 3547
(IDENTITY-011), so no record survives a read, and every fallback-sealed
3541 3548
record is claimed or expired within one ten-minute pairing lifetime.

@@ -3577,6 +3584,15 @@ dropping a live column mid-roll breaks the nodes still writing into it; the

3577 3584
contract migration removes them the way `machine_pairings.user_id` was removed
3578 3585
a release after its last reader.
3579 3586
3587
Amended 2026-08-25 (issue #253). "Its own key" is now checked rather than
3588
described. `RuntimeConfig.validate/1` compares every configured vault key and
3589
refuses a staging or production boot on any duplicate, naming the borrowing
3590
setting and the setting it borrowed from. Distinctness is the predicate rather
3591
than "did the boot bridge fire", so it also refuses a dedicated secret
3592
provisioned by hand with another vault's value. Development and test are
3593
exempt: both already configure distinct keys, and a shared key there strands
3594
nothing a person would retry.
3595
3580 3596
Evidence: `OpenAgents.Machines.TokenVault`, `OpenAgents.Accounts.TokenVault`,
3581 3597
`OpenAgents.Voice.RecordingVault`, `OpenAgents.ContentVault`,
3582 3598
`OpenAgents.Forge.AtRest`,
docs/2026-08-25-forge-exit-rehearsals-2-to-6.md modified +9

@@ -298,6 +298,15 @@ the old key to the keyring, activate the new one, rewrap with

298 298
production. That is a fact rather than a defect, and it is the reason this
299 299
rehearsal's live half is an inventory instead of a rotation.
300 300
301
> **Later the same day (2026-08-25):** the pairing-vault row above is no longer
302
> current. `MACHINE_TOKEN_ENCRYPTION_KEY` is provisioned in Secret Manager,
303
> `ops/deploy/fleet-startup.template.sh` exports it, and a re-read of all three
304
> fleet nodes answers `machine_set=true` and `same_as_github=false`, with all
305
> four vault keys set and pairwise distinct. `RuntimeConfig.validate/1` now
306
> refuses a staging or production boot on a duplicate vault key, so the reading
307
> above cannot recur unnoticed. #253 records the gap and its close. The rest of
308
> this inventory stands: no key has been rotated here.
309
301 310
**No attestation has been signed here.** `OpenAgents.Reputation.keys()` returns
302 311
none on the live node, so the issuer rotation path has no production
303 312
population. The ordering rule is proven in the test process and has never been
docs/forge-exit-rehearsals.md modified +18 -4

@@ -413,10 +413,13 @@ The forge holds several key-like secrets and they rotate differently:

413 413
    ten-minute window of unclaimed pairings becomes unreadable, and a person
414 414
    retries the pairing. There is no keyring because no record outlives the
415 415
    window. The code no longer reaches the GitHub key, which is #192's fix,
416
    but production has not provisioned `MACHINE_TOKEN_ENCRYPTION_KEY`, so the
417
    boot bridge in `config/runtime.exs` still configures this vault with the
418
    GitHub vault's active key. Until that secret is set, a GitHub key rotation
419
    does move this vault's key. #253 carries it.
416
    and since 2026-08-25 production provisions `MACHINE_TOKEN_ENCRYPTION_KEY`
417
    with its own value on every fleet node, so a GitHub key rotation no longer
418
    moves this vault's key. The boot bridge in `config/runtime.exs` survives as
419
    a fallback for a node that comes up without the secret, and
420
    `OpenAgents.RuntimeConfig.validate/1` refuses a staging or production boot
421
    whose pairing key equals the GitHub key, so the bridge can no longer be
422
    load-bearing without something failing. #253 closed on that.
420 423
  - `OpenAgents.Voice.RecordingVault` (call audio,
421 424
    `VOICE_RECORDING_ENCRYPTION_KEY`) rotates with permanent loss: one key,
422 425
    no key id, no keyring, so recordings sealed under the retired key never

@@ -446,6 +449,17 @@ GitHub retired keyring is empty and no reputation issuer key is admitted, so

446 449
neither rotation has ever run here.
447 450
`docs/2026-08-25-forge-exit-rehearsals-2-to-6.md` records the inventory.
448 451
452
**The gap is closed, 2026-08-25.** A later read the same day, again without
453
printing key material, answers `machine_set=true` and `same_as_github=false` on
454
all three fleet nodes, and all four vault keys are set and pairwise distinct.
455
`MACHINE_TOKEN_ENCRYPTION_KEY` is provisioned in Secret Manager and exported by
456
`ops/deploy/fleet-startup.template.sh`, and the fleet has rolled onto a revision
457
that carries it. `OpenAgents.RuntimeConfig.validate/1` now refuses a staging or
458
production boot on a borrowed vault key, so the next reader does not have to
459
compare two configured values by hand to learn whether the bridge is
460
load-bearing. The retired GitHub keyring is still empty and no reputation
461
issuer key is admitted; neither rotation has run.
462
449 463
## 5. Operator loss
450 464
451 465
**Proves:** that the source and the receipts come back from the WAL alone.
docs/security/secrets-and-log-handling.md modified +1 -1

@@ -35,7 +35,7 @@ use distinct names and values and remains locked.

35 35
| `GITHUB_CLIENT_SECRET` | `openagents-staging-github-client-secret` | web, fleet | OAuth app rotation |
36 36
| `GITHUB_TOKEN_ENCRYPTION_KEY` | `openagents-staging-github-vault-active` | web, fleet, migrator | Scheduled vault rotation or suspected exposure |
37 37
| `GITHUB_TOKEN_DECRYPTION_KEYS_JSON` | `openagents-staging-github-vault-previous` | web, fleet, migrator, only during rewrap | Delete after every row uses the active key ID |
38
| `MACHINE_TOKEN_ENCRYPTION_KEY` | `openagents-staging-machine-vault-active` | web, fleet | Scheduled vault rotation or suspected exposure; rotation strands at most one ten-minute window of unclaimed pairings (VAULT-001). Until this secret exists, `config/runtime.exs` bridges the machine pairing vault to `GITHUB_TOKEN_ENCRYPTION_KEY` |
38
| `MACHINE_TOKEN_ENCRYPTION_KEY` | `openagents-staging-machine-vault-active` | web, fleet | Scheduled vault rotation or suspected exposure; rotation strands at most one ten-minute window of unclaimed pairings (VAULT-001). `config/runtime.exs` bridges an unset value to `GITHUB_TOKEN_ENCRYPTION_KEY`, and a staging or production boot then refuses on the duplicate rather than serving two vaults from one key (#253) |
39 39
| `OPENAI_API_KEY` | `openagents-staging-openai-api-key` | web, fleet | Provider rotation or suspected prompt/log exposure |
40 40
| `VOICE_RECORDING_ENCRYPTION_KEY` | `openagents-staging-voice-recording-key` | web and fleet when recording is admitted | Scheduled recording-key procedure or suspected exposure |
41 41
| `CONTENT_ENCRYPTION_KEY` | `openagents-staging-content-vault-active` | web, fleet, migrator | Scheduled vault rotation or suspected exposure; the content vault has no keyring, so rotation strands every sealed voice transcript, compaction summary, preference observation, and project note (VAULT-001) |
lib/openagents/machines/token_vault.ex modified +5 -2

@@ -34,8 +34,11 @@ defmodule OpenAgents.Machines.TokenVault do

34 34
  rotation procedure moves a retired key into that keyring rather than
35 35
  deleting it. The fallback covers two bounded populations: pairings sealed by
36 36
  the previous release during a deploy, and pairings sealed while
37
  `config/runtime.exs` still bridges `:machine_token_encryption_key` to the
38
  GitHub key pending `MACHINE_TOKEN_ENCRYPTION_KEY` provisioning.
37
  `config/runtime.exs` bridged `:machine_token_encryption_key` to the GitHub
38
  key. That bridge is no longer production's configuration — the dedicated
39
  secret is provisioned — and `OpenAgents.RuntimeConfig.validate/1` refuses a
40
  staging or production boot whose vault keys are not distinct, so the bridge
41
  cannot become load-bearing again without failing (#253).
39 42
40 43
  The fallback deliberately never rewraps. The only reader is
41 44
  `OpenAgents.Machines.claim_locked_pairing/1`, which nulls
lib/openagents/runtime_config.ex modified +60

@@ -66,6 +66,7 @@ defmodule OpenAgents.RuntimeConfig do

66 66
         :ok <- validate_endpoint(settings, environment),
67 67
         :ok <- validate_database(settings, environment),
68 68
         :ok <- validate_github(settings, environment),
69
         :ok <- validate_vault_separation(settings, environment),
69 70
         {:ok, features} <- validate_features(settings, environment, staging_gate),
70 71
         :ok <- validate_staging_cleanup(settings, environment, staging_gate),
71 72
         :ok <- validate_providers(settings, features),

@@ -315,6 +316,8 @@ defmodule OpenAgents.RuntimeConfig do

315 316
         # it historically borrowed from. `config/runtime.exs` bridges an
316 317
         # unset `MACHINE_TOKEN_ENCRYPTION_KEY` to the GitHub key, so an
317 318
         # absent value here means both are missing (VAULT-001, #192).
319
         # Presence alone cannot tell a provisioned key from a bridged one;
320
         # `validate_vault_separation/2` is what does (#253).
318 321
         :ok <-
319 322
           ensure(
320 323
             encryption_key?(machine_token_key),

@@ -325,6 +328,63 @@ defmodule OpenAgents.RuntimeConfig do

325 328
    end
326 329
  end
327 330
331
  # The vaults VAULT-001 keeps apart, in the order a collision is reported.
332
  # The GitHub vault comes first because it holds the key the others
333
  # historically borrowed, so a duplicate is named against the borrower.
334
  @vault_key_settings [
335
    :github_token_encryption_key,
336
    :machine_token_encryption_key,
337
    :voice_recording_encryption_key,
338
    :content_encryption_key
339
  ]
340
341
  # Each vault seals under its own key (VAULT-001), and until this check
342
  # nothing enforced the "own" half. `config/runtime.exs` bridges an unset
343
  # `MACHINE_TOKEN_ENCRYPTION_KEY` to the GitHub vault's active key so the
344
  # deploy that introduced the variable could boot before an operator
345
  # provisioned the secret. A bridged boot presents a perfectly valid
346
  # 32-byte key that belongs to another vault, so the presence check above
347
  # passes and a GitHub rotation quietly moves the pairing vault with it —
348
  # correct on the day it shipped, wrong a week later, and indistinguishable
349
  # from correct without reading two values by hand (#192, #253).
350
  #
351
  # Distinctness is the predicate rather than "did the bridge fire", because
352
  # it also catches an operator who provisions the dedicated secret with the
353
  # GitHub vault's value, which is the same defect by a different route.
354
  #
355
  # This lives here rather than in `config/runtime.exs` on purpose. The
356
  # bridge is deliberate and still assembles a complete configuration, so
357
  # configuration evaluation keeps working everywhere; what changes is that
358
  # a staging or production node refuses to *serve* on a borrowed key, at
359
  # the declared fail-closed boundary, with the setting named. Development
360
  # and test are exempt: both already configure distinct keys, and a shared
361
  # key there strands nothing a person would retry. That is the same posture
362
  # the content vault's key already takes in `validate_features/3`.
363
  #
364
  # Only configured keys are compared. An absent key is another check's
365
  # error, and a vault whose feature is off has none.
366
  defp validate_vault_separation(settings, environment)
367
       when environment in [:staging, :production] do
368
    @vault_key_settings
369
    |> Enum.map(&{&1, Map.get(settings, &1)})
370
    |> Enum.filter(fn {_setting, key} -> encryption_key?(key) end)
371
    |> Enum.reduce_while(%{}, fn {setting, key}, seen ->
372
      case Map.fetch(seen, key) do
373
        {:ok, owner} -> {:halt, {setting, owner}}
374
        :error -> {:cont, Map.put(seen, key, setting)}
375
      end
376
    end)
377
    |> case do
378
      {setting, owner} ->
379
        error(setting, "must differ from #{owner}: no vault seals under another vault's key")
380
381
      seen when is_map(seen) ->
382
        :ok
383
    end
384
  end
385
386
  defp validate_vault_separation(_settings, _environment), do: :ok
387
328 388
  defp token_key_id?(key_id) when is_binary(key_id),
329 389
    do: String.match?(key_id, ~r/\A[a-zA-Z0-9][a-zA-Z0-9._-]{0,63}\z/)
330 390
test/openagents/runtime_config_test.exs modified +43

@@ -308,6 +308,49 @@ defmodule OpenAgents.RuntimeConfigTest do

308 308
             RuntimeConfig.validate(settings)
309 309
  end
310 310
311
  test "a staging or production release refuses to serve on a borrowed vault key" do
312
    # VAULT-001, issues #192 and #253. The presence check above passes for a
313
    # bridged boot, because `config/runtime.exs` hands the pairing vault the
314
    # GitHub vault's key, which is a valid 32-byte key. Distinctness is what
315
    # separates a provisioned key from a load-bearing bridge.
316
    settings = staging_settings()
317
    github_key = Map.fetch!(settings, :github_token_encryption_key)
318
319
    assert {:ok, _config} = RuntimeConfig.validate(settings)
320
321
    bridged = Map.put(settings, :machine_token_encryption_key, github_key)
322
323
    assert {:error, %{setting: :machine_token_encryption_key, reason: reason}} =
324
             RuntimeConfig.validate(bridged)
325
326
    assert reason =~ "github_token_encryption_key"
327
328
    # The same rule holds for every other vault, and names the borrower.
329
    assert {:error, %{setting: :content_encryption_key}} =
330
             settings
331
             |> Map.put(:content_encryption_key, github_key)
332
             |> RuntimeConfig.validate()
333
334
    assert {:error, %{setting: :voice_recording_encryption_key}} =
335
             settings
336
             |> Map.put(
337
               :voice_recording_encryption_key,
338
               Map.fetch!(settings, :machine_token_encryption_key)
339
             )
340
             |> RuntimeConfig.validate()
341
342
    # Development boots the bridge without complaint: a shared key there
343
    # strands nothing a person would retry.
344
    assert {:ok, _config} =
345
             bridged
346
             |> Map.put(:runtime_environment, :development)
347
             |> Map.put(:migrate_on_boot, false)
348
             |> Map.put(:secure_cookies, false)
349
             |> update_oauth(:redirect_uri, "http://localhost:4000/auth/github/callback")
350
             |> Map.put(:github_token_encryption_key_id, "development-2026-08")
351
             |> RuntimeConfig.validate()
352
  end
353
311 354
  test "forge mirror remotes refuse credential-bearing URLs" do
312 355
    for url <- [
313 356
          "https://operator:secret@mirror.example/openagents.com.git",

This page updates live while a promote is in flight · changelog