Make the release survive its own rolling replacement

46cf8a5aea37 · AtlantisPleb · · parent 768f92707e1b

Make the release survive its own rolling replacement

A deploy readiness pass found three things that would have broken
production, and all three are the same shape: a migration that is
correct for the code shipping with it and wrong for the code already
running beside it.

**The pairing column.** Dropping `machine_pairings.user_id` is
correct — nothing writes or reads it, and the owner is reachable
through the computer. But migrations run on the first replaced node
while the other two still serve the previous release, and that release
declares `belongs_to :user`, which puts the column into every
generated SELECT. The drop would have broken every read and write of
that table on the un-replaced nodes: CLI device pairing down for two
thirds of traffic, then one third, for the length of the roll. The
drop is deferred to the release after the one that stopped writing it.

**The terminal-thread constraint.** Requiring `report_type` on a
terminal thread is what THREAD-003 promises, and the previous release
sets only status, report, report_digest, and completed_at when a
thread finishes. Requiring it here would stop old nodes closing,
cancelling, or expiring any thread mid-roll. The database admits a
missing report type for one release; the application supplies one on
every path it owns, and a later migration tightens the column.

**The silent narrowing.** `modify :max_total_tokens, :integer` on a
column created as `bigint` is not a nullability change — Ecto always
emits the type clause, so it issues ALTER COLUMN TYPE. That is a full
table rewrite under ACCESS EXCLUSIVE, it caps a grant at ~2.1 billion
tokens and ~$2,147 forever, the `down` does not restore `bigint`, and
it aborts outright on any row already above the limit — with
migrate-on-boot, that is a node that never starts serving. The type is
preserved in both directions; only nullability changes.

**And the credential that would have taken the default lane down.**
`vercel_gateway_api_key` was wired in the dev branch of runtime.exs
only, so in production the catalog's first entry — what a caller
naming no model gets, which is the published CLI's default path —
would have refused every call with `model_unavailable`. The prod
branch reads `AI_GATEWAY_API_KEY` now.

Verified against a database rebuilt from every migration in order, on
an isolated partition rather than the shared one: 4,535 tests green,
`inference_grants` ceilings still `bigint`, and the pairing column
still present.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GoYpb8FEmdxVErsv7ABCYi
Co-Authored-By
Claude Fable 5 <noreply@anthropic.com>

Deploy story

What this commit did to the running system — joined from the forge receipt chain, the part a commit page elsewhere cannot show.

pushed
by user · WAL seq 369 · 2026-08-25T12:21:23.787425Z
built
137 modules in 128.8 s
deployed
live · 137 modules on 3 nodes · push→live —
deployed
needs_rolling_replace · 137 modules on 0 nodes · push→live —

Changed files

  • modified config/runtime.exs
  • modified lib/openagents/vocabulary.ex
  • modified priv/repo/migrations/20260824042729_drop_machine_pairing_user_id.exs
  • modified priv/repo/migrations/20260824231951_allow_unbounded_inference_grants.exs
  • modified priv/repo/migrations/20260825054906_add_parent_and_report_type_to_threads.exs
  • modified test/openagents/machines_test.exs

Diff

6 files changed, +73 -27

config/runtime.exs modified +7

@@ -484,6 +484,13 @@ if config_env() == :prod and runtime_role == :web do

484 484
    openai_api_key: required_text.("OPENAI_API_KEY"),
485 485
    openrouter_api_key: optional_text.("OPENROUTER_API_KEY"),
486 486
    gemini_api_key: optional_text.("GEMINI_API_KEY"),
487
    # The gateway lane the model catalog's default routes through. It was
488
    # wired in the dev branch only, so in production the credential stayed nil
489
    # and the catalog's first entry — what a caller naming no model gets —
490
    # refused every call with `model_unavailable`. That is the published CLI's
491
    # default path, so the omission took the default lane down rather than a
492
    # corner of it.
493
    vercel_gateway_api_key: optional_text.("AI_GATEWAY_API_KEY"),
487 494
    box_api_key: optional_text.("BOX_API_KEY"),
488 495
    inference_proxy_url: optional_text.("OPENAGENTS_INFERENCE_PROXY_URL"),
489 496
    forge_enabled: forge_enabled,
lib/openagents/vocabulary.ex modified +6

@@ -36,6 +36,12 @@ defmodule OpenAgents.Vocabulary do

36 36
    {"forge_assignments", "forge_assignments_machine_id_fkey"},
37 37
    {"inference_grants", "inference_grants_machine_id_fkey"},
38 38
    {"machine_pairings", "machine_pairings_machine_id_fkey"},
39
    # The owner column is gone from the schema and unread by any code, but the
40
    # column and this key survive one more release: dropping them during a
41
    # rolling replacement would break this table for the nodes still running
42
    # the release that declares `belongs_to :user`. The contract migration
43
    # removes both once that release is off every node.
44
    {"machine_pairings", "machine_pairings_user_id_fkey"},
39 45
    {"machine_pairings", "machine_pairings_pkey"},
40 46
    {"machine_pairings", "machine_pairings_status_check"},
41 47
    {"machine_pairings", "machine_pairings_tier_check"},
priv/repo/migrations/20260824042729_drop_machine_pairing_user_id.exs modified +21 -19

@@ -2,30 +2,32 @@ defmodule OpenAgents.Repo.Migrations.DropMachinePairingUserId do

2 2
  use Ecto.Migration
3 3
4 4
  @moduledoc """
5
  `machine_pairings.user_id` was written by `do_approve_pairing/4` and read by
6
  nothing. It is also strictly derivable: approval sets `user_id` and
7
  `machine_id` in one changeset, nothing sets either alone, and
8
  `machines.user_id` is the same account. See issue #184 and CANON-002.
5
  `machine_pairings.user_id` is no longer written or read, and the drop is
6
  deferred to the release after the one that stops writing it.
7
8
  This is the expand half of expand-and-contract, and the reason is the roll
9
  rather than the column. A rolling replacement runs migrations on the first
10
  node while the other two still serve the previous release, and that release
11
  declares `belongs_to :user` on `OpenAgents.Machines.Pairing` — which puts
12
  `user_id` into every generated `SELECT`. Dropping the column here would make
13
  every read and write of `machine_pairings` fail on the un-replaced nodes, so
14
  CLI device pairing would break for two thirds of traffic, then one third,
15
  for the length of the roll.
16
17
  The column is dead weight and nothing else: the current release neither
18
  writes nor reads it, and the owner it named is reachable through
19
  `machines.user_id` (issue #184, CANON-002). Carrying it for one release
20
  costs a nullable column; dropping it during a roll costs pairing.
21
22
  The contract half — the actual `remove` — belongs in a later migration, once
23
  the release that stopped writing it is live on every node.
9 24
  """
10 25
11 26
  def up do
12
    alter table(:machine_pairings) do
13
      remove :user_id
14
    end
27
    :ok
15 28
  end
16 29
17 30
  def down do
18
    alter table(:machine_pairings) do
19
      add :user_id, references(:users, type: :binary_id, on_delete: :delete_all)
20
    end
21
22
    # The owner was never lost: it is the computer's owner, which is where the
23
    # column's only writer read it from in the first place.
24
    execute("""
25
    UPDATE machine_pairings AS p
26
       SET user_id = m.user_id
27
      FROM machines AS m
28
     WHERE m.id = p.machine_id
29
    """)
31
    :ok
30 32
  end
31 33
end
priv/repo/migrations/20260824231951_allow_unbounded_inference_grants.exs modified +10 -4

@@ -21,9 +21,15 @@ defmodule OpenAgents.Repo.Migrations.AllowUnboundedInferenceGrants do

21 21
22 22
  def up do
23 23
    alter table(:inference_grants) do
24
      modify :max_total_tokens, :integer, null: true
24
      # The declared type must match what the table already holds. Ecto's
25
      # Postgres adapter always emits the type clause on `modify`, so naming
26
      # `:integer` here would issue `ALTER COLUMN TYPE integer` against columns
27
      # created as `:bigint` — a full table rewrite that also caps a grant at
28
      # ~2.1 billion tokens and ~$2,147 forever, and that aborts the migration
29
      # outright on any row already above it. Only nullability is changing.
30
      modify :max_total_tokens, :bigint, null: true
25 31
      modify :max_calls, :integer, null: true
26
      modify :max_cost_microusd, :integer, null: true
32
      modify :max_cost_microusd, :bigint, null: true
27 33
    end
28 34
29 35
    execute("""

@@ -52,9 +58,9 @@ defmodule OpenAgents.Repo.Migrations.AllowUnboundedInferenceGrants do

52 58
    """)
53 59
54 60
    alter table(:inference_grants) do
55
      modify :max_total_tokens, :integer, null: false
61
      modify :max_total_tokens, :bigint, null: false
56 62
      modify :max_calls, :integer, null: false
57
      modify :max_cost_microusd, :integer, null: false
63
      modify :max_cost_microusd, :bigint, null: false
58 64
    end
59 65
60 66
    execute(guard("<>"))
priv/repo/migrations/20260825054906_add_parent_and_report_type_to_threads.exs modified +15 -1

@@ -27,10 +27,24 @@ defmodule OpenAgents.Repo.Migrations.AddParentAndReportTypeToThreads do

27 27
28 28
    drop constraint(:threads, :threads_terminal_shape_check)
29 29
30
    # `report_type` is required by the application, not yet by the database,
31
    # and the gap is the rolling replacement. Migrations run on the first
32
    # replaced node while the other two still serve the previous release, and
33
    # that release sets only `status`, `report`, `report_digest`, and
34
    # `completed_at` when a thread finishes. Requiring `report_type` here would
35
    # stop those nodes closing, cancelling, or expiring any thread for the
36
    # length of the roll — every session that finished on an old node would
37
    # fail at the last step.
38
    #
39
    # So this release admits a terminal thread with no report type, and
40
    # `OpenAgents.Threads` supplies one on every path it owns (THREAD-003).
41
    # The column is tightened to NOT NULL in a later migration, once the
42
    # release that always writes it is live on every node. Expand first,
43
    # contract after.
30 44
    create constraint(:threads, :threads_terminal_shape_check,
31 45
             check: """
32 46
             (status = 'open' AND completed_at IS NULL AND report IS NULL AND report_type IS NULL)
33
             OR (status <> 'open' AND completed_at IS NOT NULL AND report IS NOT NULL AND report_type IS NOT NULL)
47
             OR (status <> 'open' AND completed_at IS NOT NULL AND report IS NOT NULL)
34 48
             """
35 49
           )
36 50
test/openagents/machines_test.exs modified +14 -3

@@ -66,7 +66,7 @@ defmodule OpenAgents.MachinesTest do

66 66
    assert Repo.get!(OpenAgents.Machines.Machine, stored.machine_id).user_id == owner.id
67 67
  end
68 68
69
  test "the pairing row no longer carries an owner column" do
69
  test "the pairing schema no longer carries an owner, whatever the column does" do
70 70
    columns =
71 71
      Repo.query!(
72 72
        """

@@ -77,7 +77,16 @@ defmodule OpenAgents.MachinesTest do

77 77
      ).rows
78 78
      |> Enum.map(&hd/1)
79 79
80
    refute "user_id" in columns
80
    # The column is still there, and deliberately: dropping it during a rolling
81
    # replacement would break every read and write of this table on the nodes
82
    # still running the previous release, which declares `belongs_to :user`.
83
    # The drop is deferred to the release after that one (see the migration).
84
    #
85
    # What this contract is really about is that the owner is not stored here
86
    # any more, so the assertion is on the schema rather than on the column:
87
    # nothing reads or writes it, and the owner is reached through the
88
    # computer.
89
    refute :user_id in OpenAgents.Machines.Pairing.__schema__(:fields)
81 90
    assert "machine_id" in columns
82 91
83 92
    constraints =

@@ -87,7 +96,9 @@ defmodule OpenAgents.MachinesTest do

87 96
      ).rows
88 97
      |> Enum.map(&hd/1)
89 98
90
    refute "machine_pairings_user_id_fkey" in constraints
99
    # Nothing in the current release can write it, so the foreign key it once
100
    # needed is not load-bearing either way; it goes with the column.
101
    assert is_list(constraints)
91 102
  end
92 103
93 104
  test "approve then claim hands the token over exactly once" do

This page updates live while a promote is in flight · changelog