unscoped fallback. GitHub identity establishes account continuity but does not

turn conversation evidence into verified facts about a person.

**One exception, and it is named here rather than discovered later: the

`system` memory bucket.** An admitted system memory (MEMORY-011) is written by

one account and read into every account's turn. That is cross-account recall by

construction, it cannot be reconciled with the rule above, and it is the point

of the bucket rather than a side effect of one, so it is admitted as an

amendment to this invariant instead of being smuggled past it. Four bounds make

it an exception rather than a hole:

* **It lives in one module.** `OpenAgents.Memories.SystemRecall` and the one

  shared ranking query in `OpenAgents.Memories.Retrieval.Lexical` are the only

  reads in either memory plane that name no account. Both are enumerated by

  name, so a second unscoped query fails the enumeration until somebody adds it

  there on purpose.

* **It is off unless an operator turns it on.**

  `OPENAGENTS_FEATURE_SYSTEM_MEMORY_RECALL` sets

  `:memory_recall, :system_bucket_enabled`, which is `false` in

  `config/config.exs` and declared `false` in both the production

  (`ops/deploy/fleet-startup.template.sh`) and staging

  (`ops/staging/gate-5-profile.sh`) profiles. With it off no query in that

  module runs, and `user` and `learned` recall is byte for byte what it was

  before the bucket had a recall path at all.

* **An eligibility filter replaces the scope predicate, and it is a predicate.**

  MEMORY-004's discipline holds through the exception rather than around it:

  what stands in for `user_id` is `bucket = 'system'`, `superseded_by_id IS

  NULL`, `tier IN ('ledger','glass')`, and membership of the id set

  `OpenAgents.Memories.Admissions.statuses/0` derived as `admitted`. A

  candidate, a rejected row, a suspended row, and a sub-`ledger` row are absent

  from the read rather than filtered out of its result, and a row's own

  `admission` column — the author's claim about their own row — is read by

  nothing (system-memory specification section 7.1).

* **Per-source caps bound it at ranking time.** At most 25% of one message's

  ranked candidate pool comes from a single account, by a stable round-robin

  over accounts in rank order; at most one memory per account and two in total

  attach to a note; and system rows spend the same per-turn character budget as

  the account's own, so a note that quotes the network is no longer than one

  that does not (section 7.2). What the caps exclude is not counted into the

  note's `dropped` line, because that count would tell every account how large

  the shared store is.

Nothing else gains an unscoped read from this. Every other query in both planes

still names `conversation_id` or `user_id`, and the enumerations below are what

prove it.

The recall APIs are enumerable rather than remembered. Recall reaches

PostgreSQL only through the backend `:recall_search_backend` names in

`config/config.exs`, so the backends are the population, and each backend's

Evidence: `OpenAgents.Memory.RecallSnapshot`, `OpenAgents.Memory.LexicalRecall`,

`OpenAgents.Memory.HybridRecall`, cross-scope tests in

`OpenAgents.Memory.LexicalRecallTest`, and the entry-point enumeration in

`OpenAgents.Memory.ScopeBoundaryTest`.

`OpenAgents.Memory.LexicalRecallTest`, the entry-point enumeration in

`OpenAgents.Memory.ScopeBoundaryTest`, and, for the amendment,

`OpenAgents.Memories.SystemRecall`, `test/openagents/memories/system_recall_test.exs`,

and the unscoped-query enumeration in `test/openagents/memories_test.exs`.

### MEMORY-002 — Recalled history is classified evidence, not current profile truth

rooted at `OpenAgents.Memories.Memory` names `user_id`, in the context and in

every retrieval backend that reads PostgreSQL, and the queries are read from

each module's own source AST rather than remembered, so a query added beside

the scoped ones fails until it carries its scope. No read offers a cross-account

or unscoped fallback, and a memory of another account is refused as absent

rather than as forbidden.

the scoped ones fails until it carries its scope. A memory of another account

is refused as absent rather than as forbidden.

The `system` bucket is the one exception, amended into MEMORY-001 rather than

assumed here. Its two queries — the eligibility read in

`OpenAgents.Memories.SystemRecall` and the shared ranking query in

`OpenAgents.Memories.Retrieval.Lexical` — name no account, and the enumeration

admits exactly those two by module and by count: each must name the `system`

bucket in place of `user_id`, and a third unscoped query fails until somebody

declares it. The `user` and `learned` buckets keep the unamended rule, and they

keep it whether the system flag is on or off.

Writes are explicit. Nothing infers a memory from what a turn contained: a row

exists because a caller asked for it through `POST /api/v1/memories`.

`OpenAgents.Memories.Memory` names `user_id` holds unchanged, and the AST proof

reads this module too.

Nothing surfaces. `OpenAgents.Memories.recall/3` reads the `user` and `learned`

buckets, named as a predicate in the query rather than filtered out of its

result, and no session sees a system memory — not another account's, and not

its own author's, and not one of any derived status: admitted, suspended, and

challenged rows all reach the same number of turns. A challenged memory nobody

can see is still a coherent state; the point of recording the challenge is that

a wrong admitted claim has a path other than editing someone else's row, and

that a contested claim is marked before an eligibility filter exists to read

the mark. That is deliberate and it is the whole reason this invariant

can stand beside MEMORY-001 and MEMORY-010 rather than amending them: an

admitted row read into every account's turn is cross-account recall by

construction, so surfacing the bucket is a privacy decision with an eligibility

filter of its own, not a ranking change. A system memory that is stored,

evidenced, and admitted but recalled by nobody is a coherent state; a quietly

widened recall predicate is not.

Surfacing is a decision an operator makes, not a state the store drifts into.

`OpenAgents.Memories.recall/3` reads the `user` and `learned` buckets, named as

a predicate in the query rather than filtered out of its result, and it reads

the `system` bucket only when `:memory_recall, :system_bucket_enabled` is on.

That switch is `false` in `config/config.exs` and declared `false` in the

production and staging profiles, so on every deployment that has not made the

decision no session sees a system memory — not another account's, not its own

author's, and not one of any derived status. This is the decision MEMORY-001's

amendment records, and the reason it is a decision is that an admitted row read

into every account's turn is cross-account recall by construction rather than a

ranking change.

When the switch is on, `OpenAgents.Memories.SystemRecall` is the whole of what

surfaces. Only rows the admission records derived as `admitted` are eligible:

candidates, rejected rows, and rows suspended by an open evidenced challenge

reach no turn, and neither does a superseded row or one below the `ledger`

tier. The filter is the query rather than a pass over its result. Per-source

caps bound the rest — 25% of one message's ranked pool per account by stable

round-robin, one memory per account and two in total per note — so an account

that wrote most of the admitted store still does not own a note. Every step is

deterministic: a total read order, the ranking module's stable tie-breaks, and

a round-robin that consults nothing else, so equal inputs give equal notes.

A note names what it is quoting. A system line reads

`[From memory: (system, as of <date>, <status>)] <body>`: the bucket, the

`as_of` date that dates the claim rather than the insert time that orders the

chain, and the status the admission records derived — never the `admission`

column, which is what the author claimed about their own row. A reader can tell

a network claim from their own, and can see how old it is.

Recall degrades to silence, never to an error. An empty store, an unreadable

one, and an unavailable ranking backend each surface nothing and fail no turn.

Evidence: `OpenAgents.Memories.Admissions`, `OpenAgents.Memories.Admission`,

`OpenAgents.Memories.Memory`, `OpenAgents.Memories.Evidence`, the

`OpenAgents.Memories.Memory`, `OpenAgents.Memories.SystemRecall`,

`OpenAgents.Memories.Note`, `OpenAgents.Memories.Evidence`, the

`memories_system_shape` and `memory_admissions_shape` constraints and the

composite foreign keys `memory_admissions_memory_fkey` and

`memory_admissions_challenge_fkey`,

`test/openagents/memories/system_memory_test.exs`, and

`test/openagents/memories/challenge_test.exs`.

`test/openagents/memories/system_memory_test.exs`,

`test/openagents/memories/challenge_test.exs`, and

`test/openagents/memories/system_recall_test.exs`.

### PRIVACY-001 — Secret-bearing profile memory is rejected, never scrub-stored

| DATA-001 | `test/openagents/conversations_test.exs` |

| DATA-002 | `test/openagents/accounts_test.exs`, `test/openagents/conversations_test.exs` |

| DATA-003 | `test/openagents/conversations_test.exs` |

| MEMORY-001 | `test/openagents/memory/lexical_recall_test.exs`, `test/openagents/memory/scope_boundary_test.exs` |

| MEMORY-001 | `test/openagents/memory/lexical_recall_test.exs`, `test/openagents/memory/scope_boundary_test.exs`, `test/openagents/memories/system_recall_test.exs` |

| MEMORY-002 | `test/openagents/memory/evidence_test.exs`, `test/openagents/turn_memory_evidence_journeys_test.exs` |

| MEMORY-003 | `test/openagents/profile_memory_test.exs` |

| MEMORY-004 | `test/openagents/memory/lexical_recall_test.exs`, `test/openagents/tools/conversation_recall_tools_test.exs`, `test/openagents/memory/scope_boundary_test.exs` |

| MEMORY-008 | `test/openagents/experience_memory_test.exs` |

| MEMORY-009 | `test/openagents/graph_memory_test.exs` |

| MEMORY-010 | `test/openagents/memories_test.exs`, `test/openagents_web/controllers/memory_controller_test.exs`, `test/openagents_web/controllers/responses_controller_test.exs` |

| MEMORY-011 | `test/openagents/memories/system_memory_test.exs`, `test/openagents/memories/challenge_test.exs` |

| MEMORY-011 | `test/openagents/memories/system_memory_test.exs`, `test/openagents/memories/challenge_test.exs`, `test/openagents/memories/system_recall_test.exs` |

| PRIVACY-001 | `test/openagents/memory/policy_and_redaction_test.exs`, `test/openagents/memory/scope_boundary_test.exs` |

| TURN-001 | `test/openagents/conversations_test.exs` |

| TURN-002 | `test/openagents/conversations_test.exs` |

  # target retrieval backend; with it off, recall runs on the lexical stand-in.

  # The three bounds are the store's ceiling per account and the per-turn

  # ceilings on how much memory may reach the model.

  #

  # `system_bucket_enabled` is the cross-account switch, and it is off here and

  # in production. With it off, recall reads the `user` and `learned` buckets

  # and nothing else, exactly as it did before the system bucket existed. With

  # it on, an admitted system memory written by one account reaches every

  # account's turn under the eligibility filter and the caps in

  # `OpenAgents.Memories.SystemRecall` (MEMORY-001, MEMORY-011).

  # `maximum_system_pool` bounds the ranked candidate pool the per-source cap

  # is a share of.

  memory_recall: [

    embeddings_enabled: false,

    system_bucket_enabled: false,

    maximum_system_pool: 40,

    provider: OpenAgents.Memory.OpenAIEmbeddings,

    model_id: "text-embedding-3-small",

    model_version: "2024-01",

  shadow_enabled = feature.("SHADOW_PROGRAMS")

  tool_embeddings_enabled = feature.("TOOL_EMBEDDINGS")

  memory_embeddings_enabled = feature.("MEMORY_EMBEDDINGS")

  system_memory_recall_enabled = feature.("SYSTEM_MEMORY_RECALL")

  computers_enabled = feature.("COMPUTERS")

  conversation_reset_enabled = feature.("CONVERSATION_RESET")

  incident_fixer_enabled = feature.("INCIDENT_FIXER")

    |> Application.fetch_env!(:tool_discovery)

    |> Keyword.put(:embeddings_enabled, tool_embeddings_enabled)

  # `system_bucket_enabled` is the one switch in this file that widens who a

  # row can reach rather than how well it is ranked. It is declared `false` in

  # `ops/deploy/fleet-startup.template.sh`, so production reads the `user` and

  # `learned` buckets and nothing else until an operator turns it on.

  memory_recall =

    :openagents

    |> Application.fetch_env!(:memory_recall)

    |> Keyword.put(:embeddings_enabled, memory_embeddings_enabled)

    |> Keyword.put(:system_bucket_enabled, system_memory_recall_enabled)

  forge_repos = parse_csv.("OPENAGENTS_FORGE_REPOSITORIES")

  forge_owner = required_text.("OPENAGENTS_FORGE_OWNER")

  alias Ecto.Multi

  alias OpenAgents.Accounts.User

  alias OpenAgents.Memories.{Memory, Recall, Retrieval}

  alias OpenAgents.Memories.{Memory, Recall, Retrieval, SystemRecall}

  alias OpenAgents.Memories.Retrieval.Semantic

  alias OpenAgents.Repo

  account holds is ranked against it; `user` memories are kept regardless of

  score and `learned` ones only above the backend's floor; the result is cut to

  `maximum_attached` memories and `maximum_attached_characters`, and what the

  cut excluded is counted rather than dropped in silence.

  The `system` bucket is not read here, by anyone, including its own author.

  A system memory is stored, admitted, and contestable

  (`OpenAgents.Memories.Admissions`) and surfaced to nobody, whatever status

  its records derive: an admitted row reaches every account's turn or none, and

  the first is cross-account recall, which MEMORY-001 and MEMORY-010 forbid.

  The bucket list is a predicate in the query rather than a filter applied to

  its results, so widening it is a deliberate edit to the recall issue's

  eligibility filter and not something a ranking change can do by accident.

  cut excluded is counted rather than dropped in silence. That read names

  `user_id`, and it is the whole of what this function does unless an operator

  says otherwise.

  The `system` bucket is the one exception, and it is off by default. When

  `:memory_recall, :system_bucket_enabled` is on, `OpenAgents.Memories.SystemRecall`

  appends what the network has learned: rows a steward's records derived as

  `admitted`, at or above the `ledger` tier, live, and unsuspended, capped at

  one per writing account and two per note, spending whatever the account's own

  memories left of the character budget. Those rows are cross-account by

  design — that is the point of the bucket, and it is why the switch exists.

  With it off, this function does what it did before the bucket had a recall

  path at all, byte for byte.

  System rows never consume the account's count budget and are never counted

  into `dropped`. `dropped` tells a reader their own store was larger than the

  turn; a count of network claims that did not fit would tell them how large

  the shared store is, which is a fact about other accounts.

  Never raises. An unreadable store or an unavailable backend recalls nothing.

  """

        if memory.bucket == "user" or score > floor, do: [memory], else: []

      end)

    {kept, dropped} = bound(eligible, opts)

    {kept, dropped, remaining} = bound(eligible, opts)

    %Recall{memories: kept, dropped: dropped, backend: backend}

    %Recall{memories: kept ++ shared(query, remaining), dropped: dropped, backend: backend}

  rescue

    _error -> %Recall{memories: [], dropped: 0, backend: :lexical}

  end

  # Count first, size second, and the count of what neither admitted. Taking

  # the highest-ranked memories until the character budget is spent keeps the

  # note about this turn rather than about whichever memory is longest.

  #

  # What is left of the character budget travels out with the result, because

  # the system bucket spends the same budget rather than one of its own: a note

  # that quotes the network is no longer than a note that does not.

  defp bound(memories, opts) do

    count = Keyword.get(opts, :maximum_attached, maximum_attached())

    characters = Keyword.get(opts, :maximum_attached_characters, maximum_attached_characters())

    {kept, _left} =

    {kept, left} =

      memories

      |> Enum.take(count)

      |> Enum.reduce({[], characters}, fn memory, {kept, remaining} ->

      end)

    kept = Enum.reverse(kept)

    {kept, length(memories) - length(kept)}

    {kept, length(memories) - length(kept), left}

  end

  # The cross-account half, and nothing at all unless an operator turned it on.

  # `SystemRecall.pool/1` answers `[]` with the flag off without issuing a

  # query, so this is a no-op on every deployment that has not made the

  # decision (MEMORY-001).

  defp shared(query, remaining) do

    query

    |> SystemRecall.pool()

    |> SystemRecall.attachable(remaining)

  end

  defp scope(%User{id: user_id}, opts) do

    # The generated `tsvector` the lexical stand-in ranks over. PostgreSQL

    # writes it; nothing here reads it back, so it never rides a select.

    field :search_vector, :string, load_in_query: false

    # What `OpenAgents.Memories.Admissions` derived for this row, carried so a

    # note can print the status a steward's receipts produce rather than the

    # `admission` field the author claimed. Virtual on purpose: a derived

    # status has no column, because a column is exactly the thing an author

    # could write for themselves.

    field :derived_status, :string, virtual: true

    belongs_to :superseded_by, __MODULE__, foreign_key: :superseded_by_id

    timestamps()

  end

  def buckets, do: @buckets

  @doc """

  The buckets recall reads.

  `system` is stored and admitted but surfaced to nobody. Reading an admitted

  system row into every account's turn is cross-account recall by construction,

  which MEMORY-001 and MEMORY-010 forbid, so widening this list is a privacy

  decision rather than a ranking change. It belongs to the recall issue that

  owns the eligibility filter, not to the store.

  The buckets account-scoped recall reads.

  `system` is not one of them and never becomes one. Reading an admitted system

  row into every account's turn is cross-account recall by construction, so it

  cannot ride the query that names `user_id`; it has a plane of its own in

  `OpenAgents.Memories.SystemRecall`, under an eligibility filter that replaces

  the scope predicate and a feature flag that is off by default (MEMORY-001).

  Widening this list would surface the bucket without either one, so it stays

  two buckets long.

  """

  @spec recallable_buckets() :: [String.t()]

  def recallable_buckets, do: @recallable_buckets

  When the bounds excluded something, the block says so in its last line. A

  note that trailed off would leave the model believing it had been told

  everything the account remembers.

  ## The system bucket reads differently on purpose

  A `user` or `learned` line is labelled with its bucket and its age. A

  `system` line is labelled `(system, as of <date>, admitted)`, because a claim

  the network makes has to be weighed against a claim the reader made, and the

  three things that decide that weight are which bucket it came from, what date

  it was observed true, and whether a steward admitted it. The date is the

  row's `as_of` rather than its age, since `as_of` dates the claim while the

  insert time only orders the chain, and a stale truth should read as dated.

  The status is the one the admission records derived

  (`OpenAgents.Memories.Admissions`), carried on `derived_status`. It is never

  the `admission` column, which is the author's own claim about their own row.

  Specification section 7.1 draws the example with the body inside the

  brackets. This repository's rails put the body after them, and one note

  carrying two bracket shapes would be harder to read than either, so the label

  is the specification's verbatim and the placement is the rails'.

  """

  alias OpenAgents.Memories.{Memory, Recall}

    |> Enum.join("\n")

  end

  defp line(%Memory{bucket: "system"} = memory) do

    "[From memory: (system, as of #{as_of(memory)}, #{status(memory)})] #{memory.body}"

  end

  defp line(%Memory{} = memory) do

    "[From memory: #{memory.bucket}, #{age(memory)}] #{memory.body}"

  end

  defp as_of(%Memory{as_of: %Date{} = date}), do: Date.to_iso8601(date)

  defp as_of(_undated), do: "date unknown"

  # Derived, never claimed. A row that reached a note without a derived status

  # is a bug in the eligibility filter rather than a candidate to describe as

  # one, so it says what it knows and nothing more.

  defp status(%Memory{derived_status: derived}) when is_binary(derived), do: derived

  defp status(_underived), do: "status unknown"

  defp omission(0), do: []

  defp omission(dropped) do

  @callback score(user_id :: String.t(), query :: String.t(), candidates :: [Memory.t()]) ::

              {:ok, %{optional(String.t()) => float()}} | :error

  @doc """

  Scores `candidates` from the shared `system` bucket, which no account owns.

  This is the one scoring path that takes no `user_id`, and the reason it can

  is that its caller has already narrowed the candidates with the eligibility

  filter in `OpenAgents.Memories.SystemRecall` — the predicate MEMORY-001's

  amendment puts in place of the scope predicate for this bucket. A backend

  that reads PostgreSQL still names `bucket` here, so the query cannot reach an

  account-scoped row even if a caller assembled the candidate list wrongly.

  Never reachable from the account's own recall: `rank/3` calls `score/3`.

  """

  @callback score_shared(query :: String.t(), candidates :: [Memory.t()]) ::

              {:ok, %{optional(String.t()) => float()}} | :error

  @doc "The score a `learned` memory must clear before it interrupts a turn."

  @callback floor() :: float()

  def rank(user_id, query, candidates, module) do

    case module.score(user_id, query, candidates) do

      {:ok, scores} ->

        {name(module), ordered(candidates, scores), module.floor()}

        {name(module), order(candidates, scores), module.floor()}

      :error when module != Lexical ->

        Logger.info("memory_retrieval_fell_back backend=#{name(module)}")

        rank(user_id, query, candidates, Lexical)

      :error ->

        {name(module), ordered(candidates, %{}), module.floor()}

        {name(module), order(candidates, %{}), module.floor()}

    end

  end

  @doc """

  Ranks shared `system` candidates against `query`, highest first.

  Returns `{backend, ranked}`. There is no floor in the answer: the floor is

  what decides whether an unasked-for `learned` memory earns a turn, and every

  candidate reaching here has already cleared a steward's admission. What

  bounds this pool is the caps in `OpenAgents.Memories.SystemRecall`, not a

  score threshold.

  Falls back to the stand-in the way `rank/3` does, and a stand-in that cannot

  answer either scores nothing rather than failing the turn.

  """

  @spec rank_shared(String.t(), [Memory.t()]) :: {backend(), [ranked()]}

  def rank_shared(query, candidates) when is_binary(query) and is_list(candidates) do

    rank_shared(query, candidates, backend())

  end

  @doc "Ranks shared candidates with a named backend."

  @spec rank_shared(String.t(), [Memory.t()], module()) :: {backend(), [ranked()]}

  def rank_shared(_query, [], module), do: {name(module), []}

  def rank_shared(query, candidates, module) do

    case module.score_shared(query, candidates) do

      {:ok, scores} ->

        {name(module), order(candidates, scores)}

      :error when module != Lexical ->

        Logger.info("memory_retrieval_fell_back backend=#{name(module)} pool=system")

        rank_shared(query, candidates, Lexical)

      :error ->

        {name(module), order(candidates, %{})}

    end

  end

  def name(Semantic), do: :semantic

  def name(_lexical), do: :lexical

  # Highest score first, then newest first, so an account whose memories all

  # score zero still gets a stable, meaningful order rather than table order.

  defp ordered(candidates, scores) do

  @doc """

  Pairs `candidates` with `scores` and orders them, highest first.

  Highest score first, then newest first, so a set of memories that all score

  zero still gets a stable, meaningful order rather than table order.

  `Enum.sort_by/2` is stable, so candidates that tie on both keys keep the

  order the caller read them in — which is why every caller reads them under a

  total order (`desc: inserted_at, desc: id`) and equal inputs give equal

  output.

  This is public because the system bucket ranks a pool no account owns

  (`OpenAgents.Memories.SystemRecall`) and must break its ties the same way

  this module breaks the account's. One tie-break rule, one place.

  """

  @spec order([Memory.t()], %{optional(String.t()) => float()}) :: [ranked()]

  def order(candidates, scores) when is_list(candidates) and is_map(scores) do

    candidates

    |> Enum.map(&{&1, Map.get(scores, &1.id, 0.0)})

    |> Enum.sort_by(fn {memory, score} -> {-score, -stamp(memory)} end)

    _error -> :error

  end

  @impl true

  def score_shared(query, candidates) do

    text = prepare(query)

    ids = Enum.map(candidates, & &1.id)

    if text == "" or ids == [] do

      {:ok, %{}}

    else

      {:ok, shared(text, ids)}

    end

  rescue

    _error -> :error

  end

  # `user_id` is the scope predicate, and it is written here rather than

  # inherited from the candidate ids (MEMORY-010). The id list narrows the

  # read; it does not bound it, and a caller that assembled that list wrongly

    |> Map.new(fn {id, rank} -> {id, rank / 1} end)

  end

  # The shared bucket's ranking query, and the one query in this module that

  # names no account. `bucket` is what stands in its place, which is

  # MEMORY-001's amendment written as a predicate rather than as a filter over

  # the result: a caller who assembled the candidate ids wrongly still cannot

  # reach an account-scoped row through this read. The caller narrowed those

  # ids to admitted, live, `ledger`-or-above rows before it got here

  # (`OpenAgents.Memories.SystemRecall`); this query re-states the two

  # predicates it can state cheaply rather than trusting the list alone.

  defp shared(text, ids) do

    from(memory in Memory,

      where: memory.bucket == "system",

      where: is_nil(memory.superseded_by_id),

      where: memory.id in ^ids,

      where: fragment("? @@ websearch_to_tsquery('english', ?)", memory.search_vector, ^text),

      select: {

        memory.id,

        fragment(

          "ts_rank_cd(?, websearch_to_tsquery('english', ?), 32)",

          memory.search_vector,

          ^text

        )

      }

    )

    |> Repo.all()

    |> Map.new(fn {id, rank} -> {id, rank / 1} end)

  end

  # `websearch_to_tsquery` reads `-` as negation, quotes as phrases, and `or`

  # as disjunction. A turn carrying punctuation would ask for something the

  # reader did not, so everything but letters and digits goes first; then the

    _error -> :error

  end

  # This backend issues no query at all, so the shared pool costs it nothing

  # but a predicate on the candidates: only `system` rows are scored, which

  # keeps an account-scoped row out of the shared ranking even if a caller

  # assembled the list wrongly. The eligibility filter narrowed the list before

  # it arrived (`OpenAgents.Memories.SystemRecall`).

  @impl true

  def score_shared(query, candidates) do

    with true <- available?(),

         {:ok, vector} <- embed(query) do

      {:ok, cosines(vector, Enum.filter(candidates, &(&1.bucket == "system")))}

    else

      _unavailable -> :error

    end

  rescue

    _error -> :error

  end

  @doc """

  The embedding to store on a new memory, or `nil` when the rail is off.

defmodule OpenAgents.Memories.SystemRecall do

  @moduledoc """

  The one recall path that crosses an account boundary, and the two things that

  make crossing it defensible: an eligibility filter written as a database

  predicate, and per-source caps that bound how much of a note any one writer

  can own.

  ## What this widens, and why it is a decision rather than a detail

  Every other read in `OpenAgents.Memories` names `user_id`. That is

  MEMORY-001 and MEMORY-010: recall is confined to the acting account, with no

  unscoped fallback anywhere. The `system` bucket is the deliberate exception —

  a row one account wrote, admitted by a steward's receipt, read into every

  account's turn. That is the point of the bucket and it is also the sharpest

  thing about it, so this module exists to hold the exception in one place

  where it can be read, enumerated, and tested, rather than as a widened

  bucket list somewhere in the account's query.

  Two properties keep it bounded:

  * **The unscoped reads are enumerable.** The eligibility read below and the

    shared ranking query in `OpenAgents.Memories.Retrieval.Lexical` are the

    only two reads in either memory plane that name no account, both name the

    `system` bucket in place of one, and both are declared by module and by

    count in `test/openagents/memories_test.exs`. A third fails that test until

    somebody declares it on purpose.

  * **It is off unless an operator turns it on.** `enabled?/0` reads

    `:memory_recall, :system_bucket_enabled`, which is `false` in

    `config/config.exs` and declared `false` in the production and staging

    profiles. With it off nothing here issues a query, so `recall/3` behaves

    byte for byte as it did before this module existed.

  * **The eligibility filter replaces the scope predicate.** It is not an

    application-side filter over rows this module read anyway. What reaches

    `memories` is a query whose predicates are the bucket, liveness, the tier

    floor, and the set of ids a steward's records derived as `admitted`.

  ## The eligibility filter (specification section 7.1)

  A row surfaces only when all of these hold, and every one of them is a

  predicate in `pool_query/1`:

  * `bucket = 'system'` — the whole namespace, one query.

  * `superseded_by_id IS NULL` — a corrected claim is the correction, not both.

  * `tier IN ('ledger','glass')` — at or above `ledger`. A `dark` or `pulse`

    claim cannot ship its content, so it is not a system memory; the table

    refuses one outright, and the predicate is here as well because a floor

    stated in exactly one place is a floor that moves when that place does.

  * `id IN (…)` — the ids `OpenAgents.Memories.Admissions.statuses/0` derived

    as `admitted`. Candidates, rejected rows, and rows suspended by an open

    evidenced challenge are all absent from that set, so none of them reaches

    the query at all.

  The derived status is bound into the query rather than compared after it.

  Deriving it is a fold over `memory_admissions` — a network-level table with

  no account column — and the fold cannot be pushed into the `memories` query

  because the challenge-flood cap is a property of the whole record set rather

  than of one row. So the honest description is: one read derives the admitted

  set, and the `memories` read then names that set as a predicate. Nothing is

  filtered out of a result.

  A row's own `admission` column is read by nothing here. An author who writes

  `admitted` on their own row has claimed something, and the claim reaches no

  turn.

  ## Per-source caps (specification section 7.2)

  Admission is the gate on truth; the caps are the gate on volume. A writer who

  passes the first still cannot win a note by flooding.

  * **Per-pool:** at most 25% of the ranked candidate pool's slots come from

    one account, enforced by a stable round-robin over accounts in rank order.

    Accounts enter the rotation in the order of their best-ranked memory, and

    each contributes its own memories in rank order, so the pool is a spread of

    the store rather than the top of whichever account writes most.

  * **Per-note:** at most 1 memory per account and at most 2 in total actually

    attach, matching the knowledge-base note limit. Past two, more network

    claims are noise rather than context.

  This is the pool cap the store-level cap in `Admissions` deliberately did not

  imply. That one bounds how much of the admitted **store** one account's

  challenges can suspend; this one bounds how much of one message's **pool**

  one account's claims can occupy, and a share of the store does not imply the

  same share of a pool, because an account's writing can concentrate on one

  subject.

  ## Determinism

  Equal inputs give equal notes. The eligibility read is ordered

  `desc: inserted_at, desc: id`, which is total; ranking is

  `OpenAgents.Memories.Retrieval.order/2`, the same stable sort the account's

  own recall uses; and the round-robin walks accounts and memories in that

  order without consulting anything else. No step reads insertion order from

  the database, a map's traversal order, or the wall clock.

  ## It degrades to silence

  An empty store, an unreadable one, an unavailable ranking backend: each one

  recalls nothing. `pool/1` never raises, because a turn that cannot reach the

  network's memory is a turn without it, not a failed turn.

  """

  import Ecto.Query

  alias OpenAgents.Memories.{Admissions, Memory, Retrieval}

  alias OpenAgents.Repo

  # Specification section 7.2's numbers. They are constants rather than

  # configuration: they are the poisoning posture the design argues for, not a

  # dial an operator tunes per deployment.

  @pool_share 25

  @per_account_per_note 1

  @per_note 2

  # The tier floor from section 5.2. `ledger` means the body and metadata are

  # readable by every account, which is what recall does with them.

  @tiers ~w(ledger glass)

  @doc """

  Whether this deployment surfaces the system bucket at all.

  `false` in `config/config.exs` and in the production profile. Read it before

  anything else: with it off, no query in this module runs and recall is the

  account-scoped read it has always been.

  """

  @spec enabled?() :: boolean()

  def enabled?, do: setting(:system_bucket_enabled, false) == true

  @doc "The tiers a system memory must carry to surface. The floor is `ledger`."

  @spec tiers() :: [String.t()]

  def tiers, do: @tiers

  @doc "The share of one message's ranked pool a single account may fill, as a percentage."

  @spec pool_share() :: pos_integer()

  def pool_share, do: @pool_share

  @doc """

  How many of `slots` one account may fill, rounded up.

  Rounded up so a pool of one or two candidates is not emptied by its own cap,

  and so the bound reads the same way `Admissions.challenge_cap/1` reads.

  """

  @spec pool_cap(non_neg_integer()) :: non_neg_integer()

  def pool_cap(slots) when is_integer(slots) and slots >= 0 do

    div(slots * @pool_share + 99, 100)

  end

  @doc "The most memories one account may put in a single note."

  @spec per_account_per_note() :: pos_integer()

  def per_account_per_note, do: @per_account_per_note

  @doc "The most system memories one note may carry in total."

  @spec per_note() :: pos_integer()

  def per_note, do: @per_note

  @doc """

  The ranked, capped candidate pool for `query`, highest first.

  Empty when the flag is off, when `query` is empty, and when nothing in the

  store is eligible. Each memory carries the status the admission records

  derived, on `derived_status`, so a note prints what a steward's receipts say

  rather than what the row claims about itself.

  """

  @spec pool(String.t()) :: [Memory.t()]

  def pool(query) when is_binary(query) and query != "" do

    if enabled?() do

      case eligible() do

        [] -> []

        candidates -> candidates |> rank(query) |> round_robin()

      end

    else

      []

    end

  rescue

    _error -> []

  end

  def pool(_query), do: []

  @doc """

  What one note attaches from `pool`, within `characters` of body text.

  The two caps of section 7.2's per-note half: one memory per account, two in

  total. The character budget is whatever the account's own memories left of

  MEMORY-010's per-turn ceiling, so the system bucket widens who a note can

  quote without widening how long a note can be.

  What the caps and the budget exclude is not reported. The account's own

  `dropped` count exists so a reader can see their store was larger than the

  turn; a count of network claims that did not fit would tell every account how

  big the shared store is, which is a fact about other accounts.

  """

  @spec attachable([Memory.t()], non_neg_integer()) :: [Memory.t()]

  def attachable(pool, characters) when is_list(pool) and is_integer(characters) do

    {kept, _seen, _left} =

      Enum.reduce(pool, {[], %{}, characters}, fn memory, {kept, seen, remaining} ->

        written = Map.get(seen, memory.user_id, 0)

        cost = String.length(memory.body)

        if length(kept) < @per_note and written < @per_account_per_note and cost <= remaining do

          {[memory | kept], Map.put(seen, memory.user_id, written + 1), remaining - cost}

        else

          {kept, seen, remaining}

        end

      end)

    Enum.reverse(kept)

  end

  # ── internal ───────────────────────────────────────────────────────────────

  # The eligibility filter, as one query. `admitted` is derived first because

  # the derivation is a fold over a table with no account column; what happens

  # here is that the derived set is named as a predicate rather than compared

  # to rows this read returned anyway.

  #

  # MEMORY-001: one of the two queries in the memory plane that name no

  # account, and these predicates are what stand in its place. The other is the

  # shared ranking query in `OpenAgents.Memories.Retrieval.Lexical`, which

  # names the bucket too and reads only the ids this one already narrowed.

  defp eligible do

    admitted = admitted_ids()

    if admitted == [] do

      []

    else

      admitted

      |> pool_query()

      |> Repo.all()

      |> Enum.map(&%{&1 | derived_status: "admitted"})

    end

  end

  # Named rather than inlined so the MEMORY-001 amendment has one query to

  # point at, and so a test can read this module's source and prove that the

  # predicates below are the ones standing in for `user_id`.

  defp pool_query(admitted) when is_list(admitted) do

    from(memory in Memory,

      where: memory.bucket == "system",

      where: is_nil(memory.superseded_by_id),

      where: memory.tier in ^@tiers,

      where: memory.id in ^admitted,

      order_by: [desc: memory.inserted_at, desc: memory.id],

      limit: ^maximum_pool()

    )

  end

  # Only `admitted`. `candidate`, `rejected`, and `suspended` are every other

  # value the derivation produces, and none of them surfaces.

  defp admitted_ids do

    for {id, "admitted"} <- Admissions.statuses(), do: id

  end

  # The same backend the account's recall runs on, and the same tie-breaks,

  # scored over a pool no account owns.

  defp rank(candidates, query) do

    {_backend, ranked} = Retrieval.rank_shared(query, candidates)

    ranked

  end

  # Section 7.2's per-pool cap. Accounts enter in the order of their

  # best-ranked memory and each contributes in its own rank order, one per

  # round, until the pool is full or every account is spent. A prolific account

  # therefore holds its share of the slots and no more, and an account with one

  # good memory is not pushed out by an account with forty mediocre ones.

  defp round_robin(ranked) do

    memories = Enum.map(ranked, fn {memory, _score} -> memory end)

    slots = min(length(memories), maximum_pool())

    cap = pool_cap(slots)

    accounts = memories |> Enum.map(& &1.user_id) |> Enum.uniq()

    by_account = Enum.group_by(memories, & &1.user_id)

    0..max(cap - 1, 0)

    |> Enum.flat_map(fn round ->

      Enum.flat_map(accounts, fn account ->

        case by_account |> Map.fetch!(account) |> Enum.at(round) do

          nil -> []

          memory -> [memory]

        end

      end)

    end)

    |> Enum.take(slots)

  end

  defp maximum_pool, do: setting(:maximum_system_pool, 40)

  # `|| []` rather than a `get_env/3` default: the key can be present and nil,

  # and a nil there would reach `Keyword.get/3` as a hard crash on a path whose

  # whole contract is to degrade.

  defp setting(key, fallback) do

    (Application.get_env(:openagents, :memory_recall) || [])

    |> Keyword.get(key, fallback)

  end

end

export OPENAGENTS_FEATURE_SCV_DEPLOY="true"

export OPENAGENTS_FEATURE_SEMANTIC_MEMORY="true"

export OPENAGENTS_FEATURE_SHADOW_PROGRAMS="true"

# Cross-account recall of the system bucket (MEMORY-001, MEMORY-011). Off in

# production: an admitted system memory is written by one account and read by

# every account, which is the one place recall crosses the account boundary,

# and it ships dark until an operator turns it on deliberately. With it off,

# recall reads the `user` and `learned` buckets and nothing else.

export OPENAGENTS_FEATURE_SYSTEM_MEMORY_RECALL="false"

export OPENAGENTS_FEATURE_TOOL_EMBEDDINGS="true"

export OPENAGENTS_FEATURE_TOOLS="true"

export OPENAGENTS_FEATURE_TURN_RECOVERY="true"

  OPENAGENTS_FEATURE_SCV_DEPLOY OPENAGENTS_SCV_DEPLOY_OUTPUT_ROOT

  OPENAGENTS_SCV_TEMPORARY_ROOT

  OPENAGENTS_FEATURE_SEMANTIC_MEMORY OPENAGENTS_FEATURE_SHADOW_PROGRAMS

  OPENAGENTS_FEATURE_SYSTEM_MEMORY_RECALL

  OPENAGENTS_FEATURE_TOOL_EMBEDDINGS OPENAGENTS_FEATURE_TOOLS

  OPENAGENTS_FEATURE_TURN_RECOVERY OPENAGENTS_FEATURE_VOICE

  OPENAGENTS_FEATURE_VOICE_RECORDING OPENAGENTS_FEATURE_VOICE_RETENTION

export OPENAGENTS_FEATURE_RA="false"

export OPENAGENTS_FEATURE_SEMANTIC_MEMORY="false"

export OPENAGENTS_FEATURE_SHADOW_PROGRAMS="false"

export OPENAGENTS_FEATURE_SYSTEM_MEMORY_RECALL="false"

export OPENAGENTS_FEATURE_TOOL_EMBEDDINGS="false"

export OPENAGENTS_FEATURE_TOOLS="true"

export OPENAGENTS_FEATURE_TURN_RECOVERY="false"

defmodule OpenAgents.Memories.SystemRecallTest do

  @moduledoc """

  The one recall path that crosses an account boundary: what it takes to turn

  it on, what it lets through when it is on, and what it changes when it is

  off.

  Four properties carry the weight.

  **Off is off, and off is the default.** The flag is `false` in

  `config/config.exs` and declared `false` in the production and staging

  profiles, and with it off an account's own recall is not merely similar to

  what it was before this bucket had a recall path — it is byte for byte the

  same. That is proved by comparison here rather than asserted: the same

  account's recall is captured with no system store behind it, an admitted

  system store is then written, and the two results are compared whole.

  **Eligibility is derived, never claimed.** A row surfaces only when the

  admission records derive `admitted` for it. A candidate, a rejected row, a

  suspended row, a superseded row, and a row whose own `admission` column says

  `admitted` with no steward behind it all reach nobody.

  **The caps bound volume, not truth.** One account that wrote most of the

  admitted store still fills at most its quarter of a message's pool and at

  most one line of a note.

  **It is deterministic.** Equal inputs give equal notes, every time.

  """

  use OpenAgents.DataCase, async: false

  alias OpenAgents.Memories

  alias OpenAgents.Memories.{Admissions, Memory, Note, Recall, SystemRecall}

  # The owner account is an operator by definition, so a steward needs no

  # configuration change.

  @owner_github_id 14_167_547

  setup do

    original = Application.get_env(:openagents, :memory_recall)

    on_exit(fn -> Application.put_env(:openagents, :memory_recall, original) end)

    :ok

  end

  defp surfacing(enabled?) do

    settings =

      :openagents

      |> Application.get_env(:memory_recall, [])

      |> Keyword.put(:system_bucket_enabled, enabled?)

    Application.put_env(:openagents, :memory_recall, settings)

  end

  defp account(key) do

    digest = :crypto.hash(:sha256, key)

    github_id = digest |> binary_part(0, 7) |> :binary.decode_unsigned()

    upsert(github_id, "sysrec-" <> (digest |> Base.encode16(case: :lower) |> binary_part(0, 12)))

  end

  defp steward, do: upsert(@owner_github_id, "AtlantisPleb")

  defp upsert(github_id, login) do

    {:ok, user} =

      OpenAgents.Accounts.upsert_github_user(%{

        github_id: github_id,

        github_login: login,

        github_avatar_url: "https://avatars.githubusercontent.com/u/#{github_id}?v=4"

      })

    user

  end

  defp evidence do

    [%{"kind" => "receipt", "ref" => "receipt:4f1c", "digest" => "sha256:9ab3"}]

  end

  defp candidate(overrides \\ %{}) do

    Map.merge(

      %{

        "bucket" => "system",

        "slug" => "sys:gateway-402-retired-model",

        "body" =>

          "A 402 from the inference gateway usually means the default model was " <>

            "retired upstream. Check gateway status before bisecting local lanes.",

        "entity" => "inference-gateway",

        "tier" => "ledger",

        "as_of" => ~D[2026-08-25],

        "admission" => "candidate",

        "evidence_refs" => evidence()

      },