Serialize thread admission and mints on the owner row

add522b94349 · AtlantisPleb · · parent d0a44b2f0b53

Serialize thread admission and mints on the owner row

Two simultaneous opens could jointly pass the eight-thread cap, and the
cap is what bounds an account's joint credit exposure, so the count now
runs under the owner visitor row's FOR UPDATE lock inside the inserting
transaction. Mints take the same lock (thread row first, owner row
second) so each reads the metered remainder at its own turn.

The remainder deliberately does not reserve a live grant's unspent
ceiling: a parent thread holds the whole balance as ceiling, and a
delegated child thread opened while it runs must still mint, so
reservation would refuse every child with credit_exhausted. Overlapping
threads therefore mint the same figures concurrently as sequentially,
and the exposure bound is the serialized cap times the remainder.
Recorded in THREAD-001; raced in credit_race_test (issue #195).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GoYpb8FEmdxVErsv7ABCYi
Co-Authored-By
Claude Fable 5 <noreply@anthropic.com>

Deploy story

What this commit did to the running system — joined from the forge receipt chain, the part a commit page elsewhere cannot show.

pushed
by user · WAL seq 304 · 2026-08-24T20:24:01.773100Z

Changed files

  • modified INVARIANTS.md
  • modified lib/openagents/threads.ex
  • added test/openagents/threads/credit_race_test.exs

Diff

3 files changed, +231 -8

INVARIANTS.md modified +28 -1

@@ -2035,6 +2035,14 @@ conversation, and a thread is not one.

2035 2035
  caller leaves nothing behind. Because a thread has at most one live grant,
2036 2036
  capping open threads caps the account's concurrent thread-scoped authority by
2037 2037
  the same number.
2038
2039
  Amended 2026-08-24 (issue #195): the cap holds under concurrency. The count
2040
  is taken under the owner visitor row's `FOR UPDATE` lock, inside the
2041
  transaction that inserts the thread, so two simultaneous opens at the
2042
  boundary serialize and admit one thread, not two. The cap is what bounds the
2043
  account's joint credit exposure (below), so a cap that a race could pass
2044
  would make that bound a fiction.
2045
  `test/openagents/threads/credit_race_test.exs` races the boundary.
2038 2046
- **A thread's budget is its own, and its money is the account's.**
2039 2047
  `OpenAgents.Threads.ceilings/0` reads the `thread_grant_*` settings and
2040 2048
  passes them to `OpenAgents.Inference.mint/1`, which otherwise applies the

@@ -2047,6 +2055,25 @@ conversation, and a thread is not one.

2047 2055
  nothing left is refused `:credit_exhausted` rather than minted a grant it
2048 2056
  cannot spend. `GET /api/v3` publishes both allowances, because a client that
2049 2057
  read a fixed per-thread cost cap would be reading a budget nobody is given.
2058
2059
  Amended 2026-08-24 (issue #195): the remainder is metered spend, and mints
2060
  serialize. `OpenAgents.Threads.mint_grant/1` locks the owner visitor row
2061
  (after the thread row, always in that order) before reading
2062
  `Credit.remaining/1` and minting, so concurrent mints for one account each
2063
  read the remainder at their own serialization point rather than from a
2064
  shared snapshot. What the remainder deliberately does not subtract is a live
2065
  grant's unspent ceiling: a parent thread holds the whole remaining balance
2066
  as its ceiling, and a delegated child thread opened while it runs must still
2067
  be granted usable authority, so reserving headroom would refuse every such
2068
  child with `:credit_exhausted` and break delegation. Overlapping live
2069
  threads can therefore each be ceiled at the same remainder — concurrently or
2070
  in sequence, the schedules mint identical figures — and the account's joint
2071
  exposure is bounded by the admission cap times the remainder, which is why
2072
  the cap above is serialized. Actual spend stays honest where money moves:
2073
  every call is metered into the remainder, and an exhausted account cannot
2074
  mint at all. `test/openagents/threads/credit_race_test.exs` proves the
2075
  serialized figures, the exhausted refusal under race, and the unchanged
2076
  serial ceilings.
2050 2077
- **Expiry revokes without being asked.** `OpenAgents.Threads.reap_expired/1`
2051 2078
  runs at admission and on every read of a thread: an active grant past
2052 2079
  `expires_at` becomes `expired`, and an open thread that has minted authority

@@ -4889,7 +4916,7 @@ contract; the invariant prose above defines the assertion, not the filename.

4889 4916
| WORK-001 | `test/openagents/work_job_test.exs`, `test/openagents/deep_work_tool_loop_test.exs` |
4890 4917
| SELF-EDIT-001 | `test/openagents/tools/repository_mutation_tools_test.exs`, `test/openagents/coding_job_test.exs`, `test/openagents/dependency_boundary_test.exs` |
4891 4918
| SCV-001 | `test/openagents/scv/deployments_test.exs`, `test/openagents/dependency_boundary_test.exs` |
4892
| THREAD-001 | `test/openagents/threads/grant_fence_test.exs`, `test/openagents/threads/grant_token_reach_test.exs`, `test/openagents/threads_test.exs` |
4919
| THREAD-001 | `test/openagents/threads/grant_fence_test.exs`, `test/openagents/threads/grant_token_reach_test.exs`, `test/openagents/threads_test.exs`, `test/openagents/threads/credit_race_test.exs` |
4893 4920
| OUTCOME-001 | `test/openagents/accepted_outcome_test.exs`, `test/openagents/issues/completion_claims_test.exs`, `test/openagents_web/controllers/issue_completion_claim_controller_test.exs` |
4894 4921
| DEPLOYPLANE-001 | `test/openagents/deployments_test.exs`, `test/openagents_web/controllers/deployment_controller_test.exs`, `test/openagents_web/api_route_authority_test.exs` |
4895 4922
| DEPLOYPLANE-002 | `test/openagents/deployments_test.exs` |
lib/openagents/threads.ex modified +45 -7

@@ -35,7 +35,11 @@ defmodule OpenAgents.Threads do

35 35
     `maximum_active_boxes_per_owner`. A token that can open one thread could
36 36
     otherwise open unbounded threads and spend without a ceiling. Because a
37 37
     thread has at most one live grant, capping open threads caps the account's
38
     concurrent thread-scoped authority by the same number.
38
     concurrent thread-scoped authority by the same number. The count is taken
39
     under the owner visitor row's `FOR UPDATE` lock, inside the transaction
40
     that inserts, so two simultaneous opens at the boundary admit one thread,
41
     not two — the cap is what makes the account's joint credit exposure a
42
     bounded figure, so it has to hold under concurrency (issue #195).
39 43
  5. **The ceiling is self-clearing.** `reap_expired/1` runs at admission and
40 44
     on every read: an active grant whose clock has run out becomes `expired`,
41 45
     and the open thread it fenced becomes `failed` with `authority_expired`.

@@ -92,12 +96,7 @@ defmodule OpenAgents.Threads do

92 96
93 97
  def open(%Visitor{id: visitor_id} = owner, objective, options) when is_binary(objective) do
94 98
    _reaped = reap_expired(owner)
95
96
    if open_count(visitor_id) >= maximum_open_per_account() do
97
      {:error, :thread_quota_reached}
98
    else
99
      insert_thread(visitor_id, objective, options)
100
    end
99
    insert_thread(visitor_id, objective, options)
101 100
  end
102 101
103 102
  @doc """

@@ -144,6 +143,20 @@ defmodule OpenAgents.Threads do

144 143
    }
145 144
146 145
    Multi.new()
146
    |> Multi.run(:admission, fn repo, _changes ->
147
      # The cap is checked under the owner row's lock, inside the transaction
148
      # that inserts, so two simultaneous opens serialize here: the second
149
      # waits, counts the first's committed row, and is refused. A count read
150
      # outside the transaction could pass twice at the boundary and leave
151
      # nine open threads behind an eight-thread promise (issue #195).
152
      _serialized = lock_owner(repo, visitor_id)
153
154
      if open_count(visitor_id) >= maximum_open_per_account() do
155
        {:error, :thread_quota_reached}
156
      else
157
        {:ok, :admitted}
158
      end
159
    end)
147 160
    |> Multi.insert(:thread, Thread.open_changeset(attributes, visitor_id, now))
148 161
    |> Multi.run(:opened_event, fn _repo, %{thread: thread} ->
149 162
      insert_event(thread, "thread.opened", %{"objective_bytes" => byte_size(objective)}, now)

@@ -154,6 +167,7 @@ defmodule OpenAgents.Threads do

154 167
    |> Repo.transaction()
155 168
    |> case do
156 169
      {:ok, %{counted: thread}} -> {:ok, thread}
170
      {:error, :admission, :thread_quota_reached, _changes} -> {:error, :thread_quota_reached}
157 171
      {:error, _step, %Ecto.Changeset{} = changeset, _changes} -> {:error, changeset}
158 172
    end
159 173
  end

@@ -358,6 +372,12 @@ defmodule OpenAgents.Threads do

358 372
    Repo.transaction(fn ->
359 373
      case locked(thread.id) do
360 374
        %Thread{status: "open"} = current ->
375
          # Concurrent mints for one account serialize on the owner row
376
          # (locked after the thread row, always in that order), so each mint
377
          # reads the metered remainder at its own turn rather than from a
378
          # shared snapshot (issue #195). What the remainder deliberately does
379
          # not subtract is a live grant's unspent ceiling — see `ceilings/1`.
380
          _serialized = lock_owner(Repo, current.owner_visitor_id)
361 381
          _revoked = Inference.revoke_active_for_thread(current.id)
362 382
363 383
          with {:ok, fenced} <- current |> Thread.generation_changeset() |> Repo.update(),

@@ -449,6 +469,17 @@ defmodule OpenAgents.Threads do

449 469
  says is left — a signed-in account's whole balance is available to one thread
450 470
  if that is what the work needs. An account with nothing left is refused
451 471
  `:credit_exhausted` instead of being minted a grant it cannot spend.
472
473
  The remainder is metered spend, not minted ceilings: a live grant's unspent
474
  headroom is deliberately not subtracted, because a parent thread holds its
475
  whole remaining balance as ceiling and a delegated child thread opened while
476
  it runs must still be granted usable authority — reserving headroom would
477
  refuse every such child with `:credit_exhausted`. Two overlapping threads can
478
  therefore each be ceiled at the same remainder, whether opened concurrently
479
  or in sequence; `mint_grant/1` serializes concurrent mints on the owner row
480
  so each reads the remainder at its own turn, and the admission cap — itself
481
  serialized — bounds how many such ceilings can be live at once (issue #195,
482
  THREAD-001).
452 483
  """
453 484
  @spec ceilings(String.t()) :: {:ok, Inference.ceilings()} | {:error, :credit_exhausted}
454 485
  def ceilings(visitor_id) when is_binary(visitor_id) do

@@ -583,6 +614,13 @@ defmodule OpenAgents.Threads do

583 614
    Repo.one(from t in Thread, where: t.id == ^thread_id, lock: "FOR UPDATE")
584 615
  end
585 616
617
  # The serialization point for one account's admissions and mints. Lock order
618
  # is thread row first, owner row second, everywhere a transaction takes both,
619
  # so two writers cannot deadlock across the pair.
620
  defp lock_owner(repo, visitor_id) do
621
    repo.one(from v in Visitor, where: v.id == ^visitor_id, lock: "FOR UPDATE")
622
  end
623
586 624
  defp insert_event(%Thread{id: thread_id}, event_type, payload, now) do
587 625
    %Event{}
588 626
    |> Event.changeset(%{
test/openagents/threads/credit_race_test.exs added +158

@@ -0,0 +1,158 @@

1
defmodule OpenAgents.Threads.CreditRaceTest do
2
  @moduledoc """
3
  What two simultaneous thread opens can take from one account (issue #195).
4
5
  The admission count and the mint's remainder read are serialized on the
6
  owner visitor row, so concurrent opens produce exactly what the same opens
7
  produce in sequence: at the cap's boundary one thread is admitted, against
8
  an exhausted account both opens are refused and leave nothing behind, and
9
  each mint reads the metered remainder at its own turn.
10
11
  What is deliberately not asserted is that overlapping live ceilings sum to
12
  the remainder. A live grant's unspent headroom does not reserve credit,
13
  because a delegated child thread must be mintable while its parent holds the
14
  whole remaining balance as its ceiling — reserving headroom would refuse
15
  every such child with `:credit_exhausted`. The joint exposure is instead
16
  bounded by the admission cap, which is why the cap has to hold under
17
  concurrency (THREAD-001).
18
  """
19
20
  use OpenAgents.DataCase, async: false
21
22
  alias OpenAgents.Conversations
23
  alias OpenAgents.Conversations.Visitor
24
  alias OpenAgents.Inference
25
  alias OpenAgents.Inference.Credit
26
  alias OpenAgents.Inference.Grant
27
  alias OpenAgents.Threads
28
29
  test "simultaneous opens at the admission boundary admit exactly one thread" do
30
    cap(2)
31
    visitor = visitor("cap-race")
32
33
    {:ok, _first} = Threads.open(%Visitor{id: visitor}, "Already open")
34
35
    results =
36
      ["Second", "Third"]
37
      |> Task.async_stream(
38
        fn objective -> Threads.open(%Visitor{id: visitor}, objective) end,
39
        max_concurrency: 2,
40
        ordered: false,
41
        timeout: :infinity
42
      )
43
      |> Enum.map(fn {:ok, result} -> result end)
44
45
    assert Enum.count(results, &match?({:ok, _thread}, &1)) == 1
46
    assert Enum.count(results, &match?({:error, :thread_quota_reached}, &1)) == 1
47
    assert Threads.open_count(visitor) == 2
48
  end
49
50
  test "simultaneous opens against an exhausted account jointly mint nothing" do
51
    visitor = visitor("exhausted-race")
52
53
    {:ok, _spent} =
54
      Inference.record_usage(minted(visitor), %{
55
        "output_tokens" => output_tokens_costing(Credit.allowance(visitor))
56
      })
57
58
    assert Credit.remaining(visitor) == 0
59
60
    results =
61
      ["First racer", "Second racer"]
62
      |> Task.async_stream(
63
        fn objective -> Threads.open_and_mint(%Visitor{id: visitor}, objective) end,
64
        max_concurrency: 2,
65
        ordered: false,
66
        timeout: :infinity
67
      )
68
      |> Enum.map(fn {:ok, result} -> result end)
69
70
    assert Enum.all?(results, &match?({:error, :credit_exhausted}, &1))
71
72
    # A refused open cancels the thread it admitted and mints no authority.
73
    # The racers leave the account exactly one grant — the exhausted one that
74
    # spent the credit — and no open thread: exhausting the grant abandoned
75
    # its thread, and admission reaped it.
76
    assert Threads.open_count(visitor) == 0
77
    assert Repo.aggregate(from(g in Grant, where: g.owner_visitor_id == ^visitor), :count) == 1
78
  end
79
80
  test "simultaneous opens mint the serialized remainder, the figure sequential opens mint" do
81
    visitor = visitor("remainder-race")
82
    spent = 250_000
83
84
    {:ok, _spent} =
85
      Inference.record_usage(minted(visitor), %{"output_tokens" => output_tokens_costing(spent)})
86
87
    remainder = Credit.allowance(visitor) - spent
88
    assert Credit.remaining(visitor) == remainder
89
90
    grants =
91
      ["First racer", "Second racer"]
92
      |> Task.async_stream(
93
        fn objective -> Threads.open_and_mint(%Visitor{id: visitor}, objective) end,
94
        max_concurrency: 2,
95
        ordered: false,
96
        timeout: :infinity
97
      )
98
      |> Enum.map(fn {:ok, {:ok, _thread, grant, _token}} -> grant end)
99
100
    # Each racer is ceiled at the metered remainder — the same ceiling the two
101
    # opens mint in either serial order, because a mint spends nothing and a
102
    # live grant's headroom reserves nothing. Concurrency adds no exposure the
103
    # serial schedule does not already have; what bounds the sum of live
104
    # ceilings is the admission cap (THREAD-001, issue #195).
105
    assert Enum.map(grants, & &1.max_cost_microusd) == [remainder, remainder]
106
  end
107
108
  test "serial behavior is unchanged: the next thread is ceiled at what spend left behind" do
109
    visitor = visitor("serial")
110
    spent = 250_000
111
112
    {:ok, first} = Threads.open(%Visitor{id: visitor}, "Spend a little")
113
    {:ok, first, grant, _token} = Threads.mint_grant(first)
114
115
    assert grant.max_cost_microusd == Credit.allowance(visitor)
116
117
    {:ok, _spent} =
118
      Inference.record_usage(grant, %{"output_tokens" => output_tokens_costing(spent)})
119
120
    {:ok, _finished} = Threads.finish(first, %{report: "Done."})
121
122
    {:ok, second} = Threads.open(%Visitor{id: visitor}, "Spend the rest")
123
    {:ok, _second, next_grant, _token} = Threads.mint_grant(second)
124
125
    assert next_grant.max_cost_microusd == Credit.allowance(visitor) - spent
126
  end
127
128
  defp visitor(key) do
129
    {:ok, conversation} = Conversations.ensure_conversation("credit-race-#{key}")
130
    conversation.visitor_id
131
  end
132
133
  # A grant names exactly one fence, so spend is recorded through a real
134
  # thread's grant rather than a fenceless one the changeset would refuse.
135
  defp minted(visitor_id) do
136
    {:ok, thread} = Threads.open(%Visitor{id: visitor_id}, "Spend some credit")
137
    {:ok, _fenced, grant, _token} = Threads.mint_grant(thread)
138
    grant
139
  end
140
141
  # Cost is priced from tokens by `OpenAgents.Inference`, never taken from a
142
  # caller, so spend is stated here in the output tokens that price to it.
143
  defp output_tokens_costing(microusd) do
144
    div(
145
      microusd * 1_000,
146
      Application.fetch_env!(:openagents, :inference_output_price_microusd_per_ktoken)
147
    )
148
  end
149
150
  defp cap(limit) do
151
    previous = Application.get_env(:openagents, :maximum_open_threads_per_account)
152
    Application.put_env(:openagents, :maximum_open_threads_per_account, limit)
153
154
    on_exit(fn ->
155
      Application.put_env(:openagents, :maximum_open_threads_per_account, previous)
156
    end)
157
  end
158
end

This page updates live while a promote is in flight · changelog