The pairing vault still runs on the GitHub vault's key in production, so VAULT-001's separation is not deployed #253

Closed AtlantisPleb opened this 4d ago 2 comments

Evidence

Shipped in the release at 0bf2706, promoted 2026-08-27.

What happened

#180's rehearsal 4 was performed against the live forge on 2026-08-25, on revision 46cf8a5aea3791936c22e82c145a9a8dd734374d. It read the deployed key configuration without printing any key material:

Secret Live state
GITHUB_TOKEN_ENCRYPTION_KEY set, key id production-legacy-v1
GITHUB_TOKEN_DECRYPTION_KEYS_JSON empty — the retired keyring holds nothing
MACHINE_TOKEN_ENCRYPTION_KEY unset
VOICE_RECORDING_ENCRYPTION_KEY set, and a different key

With MACHINE_TOKEN_ENCRYPTION_KEY unset, config/runtime.exs falls through to the boot bridge and configures machine_token_encryption_key as the GitHub vault's active key. Checked on the node: the two configured values hash to the same SHA-256 prefix, so the pairing vault and the GitHub vault seal under one key in production right now.

Why it matters

#192 is closed and its fix is deployed. The fix gave OpenAgents.Machines.TokenVault its own key and left a deliberate bridge so the deploy that introduced the variable would boot before an operator provisioned the secret. The comment in config/runtime.exs says the bridge "ends the moment MACHINE_TOKEN_ENCRYPTION_KEY is set". Nobody has set it, so the bridge that was meant to last one deploy is the production configuration.

Two published statements are false against production while it is:

  • docs/forge-exit-rehearsals.md, rehearsal 4: "Rotating the GitHub key no longer touches this vault; #192 records that it once did, silently." It does touch it today.
  • INVARIANTS.md, VAULT-001: each vault seals under its own key. Two of the three currently do not.

The blast radius is the one #192 established and is unchanged: at most one ten-minute window of unclaimed pairings, which a person retries. The defect is that a contract and a rehearsal both describe a separation the deployment does not have, which is the same class of disagreement as #188.

This is an owner action rather than a code change. The code is already correct and already prefers the dedicated key.

Reproduction

On any fleet node, with no key material printed:

active = Application.get_env(:openagents, :github_token_encryption_key)
machine = Application.get_env(:openagents, :machine_token_encryption_key)
{map_size(Application.get_env(:openagents, :github_token_decryption_keys) || %{}), machine == active}
#=> {0, true}

Acceptance criteria

  • MACHINE_TOKEN_ENCRYPTION_KEY is provisioned with its own base64 32-byte value on every fleet node, and the check above answers false for the second element.
  • Something fails when the bridge is load-bearing in production, rather than a person reading two configuration values by hand. The bridge is correct at boot and wrong a week later, and nothing today can tell the difference.
  • Until it is provisioned, docs/forge-exit-rehearsals.md says the bridge is active rather than saying the rotation no longer touches the pairing vault.

Found by: #180's rehearsal 4, docs/forge-exit-rehearsals.md.

  1. AtlantisPleb opened this issue 4d ago
  2. A AtlantisPleb Author 4d ago

    Sweep verdict: stays open. The issue is right, and its owner action is already queued where the owner reads it — recording that so nobody files it a third time.

    The ask is live. NEEDS_OWNER.md at the AtlantisPleb/workspace root carries Provision the machine pairing vault's own key (2026-08-24), written when #192 landed. It names the same three steps this issue's first acceptance criterion asks for: create the secret with a fresh base64 32-byte value, mount it as MACHINE_TOKEN_ENCRYPTION_KEY on the web and fleet identities, redeploy, and keep the retired GitHub key in GITHUB_TOKEN_DECRYPTION_KEYS_JSON for at least ten minutes on the next rotation. That file is committed and pushed, so the owner has it.

    So this issue's contribution is not a new ask — it is the measurement that the bridge is still load-bearing a day later, against a named revision. That is worth keeping open on its own.

    Who has to act: the owner, in Google Secret Manager. No agent can provision it, and the code is already correct — config/runtime.exs prefers the dedicated key and only falls through when the variable is unset.

    On the second and third criteria

    Criterion 2 — "something fails when the bridge is load-bearing in production, rather than a person reading two configuration values by hand" — is the durable half of this issue and the part that outlives the provisioning. It is unwritten and unassigned. It is also the same shape as #251 and #188: a contract that describes a deployment nobody re-checks. A boot-time or status-surface check that the two vault keys differ would close it; nothing today can tell a correct bridge at boot from a wrong one a week later.

    Criterion 3 — correcting docs/forge-exit-rehearsals.md rehearsal 4, which currently says "Rotating the GitHub key no longer touches this vault" when it does — is a small edit, but that file is #180's live artifact and is being written right now. Left for whoever finishes #180 rather than edited underneath them.

    No code changed in this sweep.

  3. A AtlantisPleb Author 3d ago

    Closing. All three acceptance criteria are met, and the third one had drifted in
    the opposite direction since this issue was filed.

    1. MACHINE_TOKEN_ENCRYPTION_KEY is provisioned with its own value

    Verified against the live fleet on 2026-08-25, on all three nodes, without
    printing any key material:

    sarah-fleet-1 (us-central1-a): machine_set=true same_as_github=false keyring=0
    sarah-fleet-2 (us-central1-b): machine_set=true same_as_github=false keyring=0
    sarah-fleet-3 (us-central1-c): machine_set=true same_as_github=false keyring=0
    

    The reproduction in this issue asked for false in the second element, and it
    answers false. A wider read on sarah-fleet-1 reports all four vault keys set
    and pairwise distinct. The secret exists in Secret Manager,
    ops/deploy/fleet-startup.template.sh exports it, and the fleet has rolled onto
    a revision that carries it.

    2. Something fails when the bridge is load-bearing

    This was the durable half and it is now written.
    OpenAgents.RuntimeConfig.validate/1 compares every configured vault key and
    refuses a staging or production boot on any duplicate, naming the borrowing
    setting and the setting it borrowed from.

    The predicate is distinctness rather than "did the boot bridge fire", because a
    bridged boot presents a perfectly valid 32-byte key — the existing presence
    check passes on it — and because an operator who provisions the dedicated secret
    by hand with the GitHub vault's value produces the same defect by a different
    route.

    The check lives in RuntimeConfig rather than in config/runtime.exs. Three
    shapes were considered:

    • A raise in config/runtime.exs. Rejected: it removes the bridge
      outright, and configuration assembly is the wrong layer. RuntimeConfig is
      the declared fail-closed boundary and already refuses a staging or production
      boot without the content vault's key, on exactly this reasoning (VAULT-001,
      #193).
    • A reported status. Rejected: this issue's own complaint is that a person
      reading two configuration values by hand is what failed for a week. A
      published boolean nobody re-reads is another value nobody re-reads, which is
      #188's failure as well as this one's.
    • A validation error, scoped to staging and production. Chosen. The bridge
      stays and still assembles a complete configuration, so nothing loses the
      ability to boot unattended; what changes is that a staging or production node
      refuses to serve on a borrowed key. Development and test are exempt: both
      already configure distinct keys, and a shared key there strands nothing a
      person would retry.

    Turning it on is a no-op against the current fleet — the live read above is what
    makes that safe to assert rather than hope.

    Covered by test/openagents/runtime_config_test.exs, "a staging or production
    release refuses to serve on a borrowed vault key", which asserts the bridged
    shape is refused, that the rule holds for the other vaults and names the
    borrower, and that development still boots.

    3. The documents say what is true

    They had gone false in the other direction: rehearsal 4 of
    docs/forge-exit-rehearsals.md said production had not provisioned the secret
    and that the bridge was the production configuration. Corrected, dated, and the
    rehearsal-4 gap is recorded as closed. Also corrected: VAULT-001's decrypt-side
    paragraph, which described the bridge as the current production configuration
    and now describes it as the fallback it is, without weakening the invariant;
    OpenAgents.Machines.TokenVault's moduledoc; the
    MACHINE_TOKEN_ENCRYPTION_KEY row in docs/security/secrets-and-log-handling.md;
    and docs/2026-08-25-forge-exit-rehearsals-2-to-6.md, which gets a dated
    follow-up note rather than a rewritten reading, because it is a record of what
    was read that day.

    VAULT-001 carries an amendment: "its own key" is now checked rather than
    described.

  4. closed this as completed 3d ago
  5. AtlantisPleb referenced this in 342720e 3d ago
Sign in with GitHub to comment on this issue.