database guards, `test/openagents/graph_memory_test.exs`, the committed graph comparison,

and `OpenAgents.GraphMemoryTest`.

### MEMORY-010 — Cloud memories are account-scoped, explicit, and bounded at recall

Status: Current

Cloud memories (`OpenAgents.Memories`) are a plane of their own, distinct from

the visitor-scoped memory planes MEMORY-001 through MEMORY-009 govern. They are

account-scoped rather than conversation-scoped, thread-sourced rather than

message-sourced, and authoritative rather than derived: dropping the table

loses what an account asked to have remembered, which is the property that

separates this store from every projection beside it. It is a separate plane

because the existing ones cannot hold it — profile-memory records admit only a

same-owner conversation-message source or host assertion (MEMORY-003), a thread

is not a conversation (THREAD-001), and MEMORY-002 forbids consolidation-derived

material from entering the profile plane at all.

Scope is a database predicate, never an application filter. Every Ecto query

rooted at `OpenAgents.Memories.Memory` names `user_id`, in the context and in

every retrieval backend that reads PostgreSQL, and the queries are read from

each module's own source AST rather than remembered, so a query added beside

the scoped ones fails until it carries its scope. No read offers a cross-account

or unscoped fallback, and a memory of another account is refused as absent

rather than as forbidden.

Writes are explicit. Nothing infers a memory from what a turn contained: a row

exists because a caller asked for it through `POST /api/v1/memories`.

Corrections supersede rather than edit — the replacement is a new row, the old

row points at it, and recall reads live rows only — so a wrong memory is traced

and replaced rather than overwritten. Bodies are immutable for the life of a

row.

Recall is bounded three ways and reports what it excluded. The store caps live

memories per account at write; each turn caps how many memories attach and how

many characters they spend; and the `[From memory: …]` note states the count

that did not fit rather than truncating. A `user` memory attaches whenever the

account holds one, because the reader asked for it and it need not share

vocabulary with the turn; a `learned` memory must clear the retrieval backend's

floor. Recall never fails a turn: an empty store, an unavailable embedding

provider, and an unreadable backend each recall nothing.

Retrieval is a swappable interface with two backends. The embedding backend is

the target; the PostgreSQL full-text backend is a marked stand-in, and it is

what a deployment runs unless `OPENAGENTS_FEATURE_MEMORY_EMBEDDINGS` and an

embedding credential are both configured. No pgvector index exists for this

plane — `message_semantic_embeddings` is bound to `message_id` — so the

embedding backend compares vectors stored on the memory rows in process, the

way the tool catalog does.

Recognition on `POST /api/v1/responses` grants no authority. The ambient plug

refuses nobody, so an anonymous caller, an unreadable credential, and a

credential scoped elsewhere all reach the route exactly as before recall

existed, and the account it recognizes only widens the context of the answer.

Product-data deletion removes them. Memories key on the retained account row

rather than the visitor root, so the DATA-004 visitor cascade does not reach

them and `OpenAgents.DataRights.delete/3` removes them explicitly in the same

transaction.

Evidence: `OpenAgents.Memories`, `OpenAgents.Memories.Retrieval`,

`OpenAgentsWeb.MemoryController`, `OpenAgentsWeb.Plugs.AmbientApiTokenAuth`,

the `memories` table's shape constraint and partial indexes,

`test/openagents/memories_test.exs`,

`test/openagents_web/controllers/memory_controller_test.exs`, and

`test/openagents_web/controllers/responses_controller_test.exs`.

### PRIVACY-001 — Secret-bearing profile memory is rejected, never scrub-stored

Status: Current

memberships, outbox events, cascade plans, and operation receipts are likewise

removed in that transaction only after deletion of the visitor root.

Cloud memories (MEMORY-010) are the one product-data family the visitor cascade

cannot reach, because they key on the retained account row rather than on the

visitor root. Deletion removes them by `user_id` explicitly, in the same

transaction, so an account that asked for everything to be removed does not keep

what it asked to have remembered.

The minimal local account record (GitHub numeric ID, current login/avatar,

access status, authentication timestamps, and currently the encrypted GitHub

token ciphertext) is retained so deletion cannot erase a ban or bypass

| MEMORY-007 | `test/openagents/preferences_test.exs` |

| MEMORY-008 | `test/openagents/experience_memory_test.exs` |

| MEMORY-009 | `test/openagents/graph_memory_test.exs` |

| MEMORY-010 | `test/openagents/memories_test.exs`, `test/openagents_web/controllers/memory_controller_test.exs`, `test/openagents_web/controllers/responses_controller_test.exs` |

| PRIVACY-001 | `test/openagents/memory/policy_and_redaction_test.exs`, `test/openagents/memory/scope_boundary_test.exs` |

| TURN-001 | `test/openagents/conversations_test.exs` |

| TURN-002 | `test/openagents/conversations_test.exs` |

    maximum_export_artifacts: 500

  ],

  memory_portability: [enabled: false],

  # Cloud memories (`OpenAgents.Memories`). `embeddings_enabled` chooses the

  # target retrieval backend; with it off, recall runs on the lexical stand-in.

  # The three bounds are the store's ceiling per account and the per-turn

  # ceilings on how much memory may reach the model.

  memory_recall: [

    embeddings_enabled: false,

    provider: OpenAgents.Memory.OpenAIEmbeddings,

    model_id: "text-embedding-3-small",

    model_version: "2024-01",

    dimensions: 64,

    floor: 0.3,

    maximum_live_memories: 200,

    maximum_attached: 8,

    maximum_attached_characters: 2_000

  ],

  tool_discovery: [

    embeddings_enabled: false,

    provider: OpenAgents.Memory.OpenAIEmbeddings,

  portability_enabled = feature.("MEMORY_PORTABILITY")

  shadow_enabled = feature.("SHADOW_PROGRAMS")

  tool_embeddings_enabled = feature.("TOOL_EMBEDDINGS")

  memory_embeddings_enabled = feature.("MEMORY_EMBEDDINGS")

  computers_enabled = feature.("COMPUTERS")

  conversation_reset_enabled = feature.("CONVERSATION_RESET")

  incident_fixer_enabled = feature.("INCIDENT_FIXER")

    |> Application.fetch_env!(:tool_discovery)

    |> Keyword.put(:embeddings_enabled, tool_embeddings_enabled)

  memory_recall =

    :openagents

    |> Application.fetch_env!(:memory_recall)

    |> Keyword.put(:embeddings_enabled, memory_embeddings_enabled)

  forge_repos = parse_csv.("OPENAGENTS_FORGE_REPOSITORIES")

  forge_owner = required_text.("OPENAGENTS_FORGE_OWNER")

  github_oauth_scopes = parse_csv.("GITHUB_OAUTH_SCOPES")

    memory_portability: memory_portability,

    shadow_programs: shadow_programs,

    tool_discovery: tool_discovery,

    memory_recall: memory_recall,

    computer_controller_enabled: computers_enabled,

    machine_token_ttl_seconds:

      parse_integer.("OPENAGENTS_MACHINE_TOKEN_TTL_SECONDS", 300..2_592_000),

inform Sarah's expression. It informs; it never grants capability or

authority.

**Memory planes** — account-scoped, consent-gated projections: conversation

**Memory planes** — visitor-scoped, consent-gated projections: conversation

recall (hybrid lexical + semantic), profile memory, learned preferences,

experience memory, graph memory. All disposable except the authoritative

messages and tool steps underneath them. *Search over thread history

(checkpoints, trailers, and receipts) is proposed and unclaimed as a use of

these planes, not a second index.*

**Memories** — `OpenAgents.Memories`, and a different thing from the memory

planes above. A memory is account-scoped (`memories.user_id`, not a visitor),

thread-sourced, and authoritative rather than derived: nothing can rebuild it,

because the sentence a reader asked to have remembered is the only copy. Two

buckets, `user` and `learned`. Recall runs server-side inside `POST

/api/v1/responses`, so every client gets it without implementing retrieval.

Use "memories" for this store and "memory planes" for the projections; a

durable fact a reader states in the web conversation is still profile memory.

### Threads

A **thread** is the unit of agent work. Everything in this section describes a

    SourceMembership

  }

  alias OpenAgents.Memories.Memory

  alias OpenAgents.Memory.SemanticDerivativeReceipt

  alias OpenAgents.{Accounts, ApiTokens, Conversations, ProfileMemory, Repo}

  alias OpenAgents.Voice.{ResponseContext, ResponseReceipt, Session, TranscriptItem}

          from(record in ProfileMemory.Record, where: record.owner_visitor_id == ^visitor_id)

        )

      # Cloud memories are keyed on the account row, not on the visitor root,

      # and the account row is deliberately retained (DATA-004) so deletion

      # cannot erase a ban. The visitor cascade therefore does not reach them,

      # and leaving that to the cascade would quietly keep what an account

      # asked to have remembered after it asked for everything to be removed.

      # They are removed here, explicitly, in the same transaction.

      {_deleted_memories, nil} =

        Repo.delete_all(from(memory in Memory, where: memory.user_id == ^user_id))

      Repo.delete!(owner)

      {_deleted_receipts, nil} =

          "carries the objective, the terminal report, usage, and the transcript, " <>

          "so the export ledger now reaches what the deletion cascade always did."

    },

    %{

      family: :memory,

      api?: true,

      status: :portable,

      mechanism: "GET /api/v1/memories",

      proof: {:test, "test/openagents_web/controllers/memory_controller_test.exs"},

      issue: nil,

      note:

        "The list route is the export: it returns every memory the account " <>

          "wrote, live and superseded both, to that account and to nobody " <>

          "else. Unlike the recall planes this store is authoritative — the " <>

          "sentence a reader typed once is the only copy — so it leaves " <>

          "through a route of its own rather than as a projection of messages."

    },

    %{

      family: :pull_request,

      api?: true,

defmodule OpenAgents.Memories do

  @moduledoc """

  The account's memories: what it asked to have remembered, and what the server

  learned on its behalf.

  Memory lives where the account lives. This store is a table in this database,

  not a file on whichever machine happened to run the session, because the CLI,

  the web app, and the API are three clients of one account and a memory

  written through any of them belongs to all three.

  ## Why this is a plane of its own

  This repository already has memory planes, and a fourth one needs a reason

  rather than a preference. The reason is that both existing planes are scoped

  and sourced on axes this store cannot use.

  * **Scope.** `OpenAgents.ProfileMemory` and `OpenAgents.ExperienceMemory` are

    scoped to `OpenAgents.Conversations.Visitor` — a signed browser under the

    account's one canonical conversation (DATA-002). This lane authenticates an

    `OpenAgents.Accounts.User` over the API, and a CLI session has no browser to

    be a visitor of. Scoping here is `memories.user_id`, and MEMORY-010 makes

    that a database predicate rather than a filter someone remembered to write.

  * **Source.** A profile-memory record is only active with a same-owner

    complete user-message source or a host-recorded owner assertion

    (MEMORY-003), and its sources are `messages` rows in that one conversation.

    A memory written from a coding session comes out of a **thread**, and a

    thread is explicitly not a conversation (THREAD-001). There is no

    admissible profile-memory source for it.

  * **The `learned` bucket cannot live there at all.** MEMORY-002 says

    conversation evidence never enters the profile-memory plane through

    repetition, model confidence, or recall classification, and MEMORY-003 says

    candidates never activate through repetition or model confidence.

    Consolidation-derived memory is exactly that, so admitting it to profile

    memory would mean weakening two current invariants to fit a design.

  What this store is **not** is a second home for browser profile claims. A

  durable fact a reader states in the web conversation still belongs in

  `OpenAgents.ProfileMemory`, under its consent and correction contract. The

  two planes are expected to be reconciled once one scope can express the

  other; that is a decision with its own issue, not a thing to assume here.

  ## What it holds

  Two buckets, described on `OpenAgents.Memories.Memory`. `user` memories are

  explicit: a reader said "remember that I prefer X" and something called

  `create/2`. Nothing here infers a memory from what a turn contained, and

  nothing should — a store that fills itself is a store nobody trusts.

  `learned` memories come from server-side consolidation over thread events.

  ## Corrections supersede

  `create/2` accepts `supersedes`, and the replacement points the old row at

  itself rather than editing it. Recall reads live rows only, so the correction

  takes effect immediately, and the row it corrected stays readable through

  `list/2` with `include_superseded: true`. A wrong `learned` memory is traced

  through `source_ref` to the work that taught it and superseded from there.

  ## Recall is bounded three ways

  `recall/3` is what `POST /api/v1/responses` calls before the provider. It is

  bounded by the store (`maximum_live_memories` per account, enforced at

  write), by count (`maximum_attached` per turn), and by size

  (`maximum_attached_characters` per turn). Whatever the bounds exclude is

  counted into `OpenAgents.Memories.Recall`'s `dropped`, and the note says so —

  a memory that did not fit is reported, never truncated into a half sentence.

  The two buckets clear the bar differently, and the difference is the whole

  point of the feature:

  * A `user` memory attaches whenever the account has one. The reader asked for

    it; "remember I use pnpm, not npm" has to reach "install the deps", and

    those two sentences share no word.

  * A `learned` memory must clear the retrieval backend's floor. Consolidation

    writes these without being asked, so they earn attention by being about

    this turn rather than by existing.

  Retrieval itself is `OpenAgents.Memories.Retrieval`, which chooses between an

  embedding backend and a lexical stand-in.

  """

  import Ecto.Query

  alias Ecto.Multi

  alias OpenAgents.Accounts.User

  alias OpenAgents.Memories.{Memory, Recall, Retrieval}

  alias OpenAgents.Memories.Retrieval.Semantic

  alias OpenAgents.Repo

  @maximum_listed 200

  @doc """

  Writes one memory for `user`.

  Attributes: `body` (required), `bucket` (`user` by default), `source_ref`,

  and `supersedes` — the id of a memory this one replaces, which must belong to

  the same account and must still be live.

  The owner is set on the struct and never cast, so a request body cannot name

  whose memory it is writing.

  """

  @spec create(User.t(), map()) ::

          {:ok, Memory.t()}

          | {:error, Ecto.Changeset.t()}

          | {:error, :quota_reached}

          | {:error, :supersedes_not_found}

  def create(%User{} = user, attrs) when is_map(attrs) do

    attrs = normalize(attrs)

    body = Map.get(attrs, "body")

    embedding =

      case body do

        text when is_binary(text) and text != "" -> Semantic.embedding_for(text)

        _absent -> nil

      end

    attrs =

      case embedding do

        {vector, model} -> Map.merge(attrs, %{"embedding" => vector, "embedding_model" => model})

        nil -> attrs

      end

    changeset = Memory.changeset(%Memory{user_id: user.id}, attrs)

    Multi.new()

    |> Multi.run(:supersedes, fn _repo, _changes -> superseded(user, attrs) end)

    |> Multi.run(:quota, fn _repo, changes -> quota(user, changes.supersedes) end)

    |> Multi.insert(:memory, changeset)

    |> Multi.run(:supersede, fn repo, changes -> link(repo, changes) end)

    |> Repo.transaction()

    |> case do

      {:ok, %{memory: memory}} -> {:ok, memory}

      {:error, :memory, changeset, _changes} -> {:error, changeset}

      {:error, _step, reason, _changes} -> {:error, reason}

    end

  end

  @doc """

  The account's memories, newest first.

  Live only unless `include_superseded: true`. Options: `bucket` to narrow to

  one bucket, and `limit`, capped at #{@maximum_listed}.

  """

  @spec list(User.t(), keyword()) :: [Memory.t()]

  def list(%User{} = user, opts \\ []) do

    user

    |> scope(opts)

    |> order_by([memory], desc: memory.inserted_at, desc: memory.id)

    |> limit(^limit(opts))

    |> Repo.all()

  end

  @doc "One of the account's memories by id, live or superseded."

  @spec fetch(User.t(), String.t()) :: {:ok, Memory.t()} | {:error, :not_found}

  def fetch(%User{id: user_id}, id) when is_binary(id) do

    with {:ok, memory_id} <- Ecto.UUID.cast(id),

         %Memory{} = memory <- Repo.get_by(Memory, id: memory_id, user_id: user_id) do

      {:ok, memory}

    else

      _absent -> {:error, :not_found}

    end

  end

  def fetch(%User{}, _id), do: {:error, :not_found}

  @doc """

  Points `memory` at the memory that replaced it.

  Both must belong to the same account, and a memory cannot supersede itself.

  """

  @spec supersede(Memory.t(), Memory.t()) :: {:ok, Memory.t()} | {:error, :not_supersedable}

  def supersede(%Memory{user_id: owner} = memory, %Memory{user_id: owner} = replacement)

      when memory.id != replacement.id do

    memory

    |> Memory.supersede_changeset(replacement)

    |> Repo.update()

    |> case do

      {:ok, updated} -> {:ok, updated}

      {:error, _changeset} -> {:error, :not_supersedable}

    end

  end

  def supersede(%Memory{}, %Memory{}), do: {:error, :not_supersedable}

  @doc """

  Removes one memory outright.

  Deletion is deletion: unlike a correction, nothing is kept. A memory that

  points at this one as its replacement keeps its own row and loses the

  pointer, so removing a correction never removes the history behind it.

  """

  @spec delete(User.t(), String.t()) :: {:ok, Memory.t()} | {:error, :not_found}

  def delete(%User{} = user, id) do

    with {:ok, memory} <- fetch(user, id),

         {:ok, deleted} <- Repo.delete(memory) do

      {:ok, deleted}

    else

      _absent -> {:error, :not_found}

    end

  end

  @doc """

  What this turn should be told, bounded.

  `query` is the incoming input. Every live memory the account holds is ranked

  against it; `user` memories are kept regardless of score and `learned` ones

  only above the backend's floor; the result is cut to `maximum_attached`

  memories and `maximum_attached_characters`, and what the cut excluded is

  counted rather than dropped in silence.

  Never raises. An unreadable store or an unavailable backend recalls nothing.

  """

  @spec recall(User.t(), String.t(), keyword()) :: Recall.t()

  def recall(user, query, opts \\ [])

  def recall(%User{} = user, query, opts) when is_binary(query) and query != "" do

    candidates = list(user, limit: maximum_live_memories())

    {backend, ranked, floor} = Retrieval.rank(user.id, query, candidates)

    eligible =

      Enum.flat_map(ranked, fn {memory, score} ->

        if memory.bucket == "user" or score > floor, do: [memory], else: []

      end)

    {kept, dropped} = bound(eligible, opts)

    %Recall{memories: kept, dropped: dropped, backend: backend}

  rescue

    _error -> %Recall{memories: [], dropped: 0, backend: :lexical}

  end

  def recall(%User{}, _query, _opts),

    do: %Recall{memories: [], dropped: 0, backend: :lexical}

  @doc "The most live memories one account may hold."

  @spec maximum_live_memories() :: pos_integer()

  def maximum_live_memories, do: setting(:maximum_live_memories, 200)

  @doc "The most memories one turn may attach."

  @spec maximum_attached() :: pos_integer()

  def maximum_attached, do: setting(:maximum_attached, 8)

  @doc "The most characters of memory bodies one turn may attach."

  @spec maximum_attached_characters() :: pos_integer()

  def maximum_attached_characters, do: setting(:maximum_attached_characters, 2_000)

  # ── internal ───────────────────────────────────────────────────────────────

  # Count first, size second, and the count of what neither admitted. Taking

  # the highest-ranked memories until the character budget is spent keeps the

  # note about this turn rather than about whichever memory is longest.

  defp bound(memories, opts) do

    count = Keyword.get(opts, :maximum_attached, maximum_attached())

    characters = Keyword.get(opts, :maximum_attached_characters, maximum_attached_characters())

    {kept, _left} =

      memories

      |> Enum.take(count)

      |> Enum.reduce({[], characters}, fn memory, {kept, remaining} ->

        cost = String.length(memory.body)

        if cost <= remaining, do: {[memory | kept], remaining - cost}, else: {kept, remaining}

      end)

    kept = Enum.reverse(kept)

    {kept, length(memories) - length(kept)}

  end

  defp scope(%User{id: user_id}, opts) do

    query = from(memory in Memory, where: memory.user_id == ^user_id)

    query =

      if Keyword.get(opts, :include_superseded, false) do

        query

      else

        where(query, [memory], is_nil(memory.superseded_by_id))

      end

    case Keyword.get(opts, :bucket) do

      bucket when bucket in ["user", "learned"] -> where(query, [m], m.bucket == ^bucket)

      _all -> query

    end

  end

  defp limit(opts) do

    case Keyword.get(opts, :limit) do

      value when is_integer(value) and value > 0 -> min(value, @maximum_listed)

      _absent -> @maximum_listed

    end

  end

  # A correction replaces one live row with another, so it is admitted at the

  # ceiling. Refusing it there would leave an account that has filled its store

  # unable to fix anything already in it.

  defp quota(_user, %Memory{}), do: {:ok, :superseding}

  defp quota(user, nil) do

    live =

      Repo.aggregate(

        from(memory in Memory,

          where: memory.user_id == ^user.id and is_nil(memory.superseded_by_id)

        ),

        :count

      )

    if live < maximum_live_memories(), do: {:ok, live}, else: {:error, :quota_reached}

  end

  defp superseded(user, attrs) do

    case Map.get(attrs, "supersedes") do

      nil ->

        {:ok, nil}

      id when is_binary(id) ->

        case fetch(user, id) do

          {:ok, %Memory{superseded_by_id: nil} = memory} -> {:ok, memory}

          _absent_or_already_superseded -> {:error, :supersedes_not_found}

        end

      _invalid ->

        {:error, :supersedes_not_found}

    end

  end

  defp link(_repo, %{supersedes: nil}), do: {:ok, nil}

  defp link(repo, %{supersedes: previous, memory: memory}) do

    previous

    |> Memory.supersede_changeset(memory)

    |> repo.update()

  end

  defp normalize(attrs) do

    Map.new(attrs, fn {key, value} -> {to_string(key), value} end)

  end

  # `|| []` rather than a `get_env/3` default: the key can be present and nil,

  # and a nil there would reach `Keyword.get/3` as a hard crash rather than as

  # the configured fallback.

  defp setting(key, fallback) do

    (Application.get_env(:openagents, :memory_recall) || [])

    |> Keyword.get(key, fallback)

  end

end

defmodule OpenAgents.Memories.Memory do

  @moduledoc """

  One thing the account asked to have remembered, or one thing the server

  learned on its behalf.

  A memory is account-scoped and authoritative. That is what separates it from

  the memory planes in `OpenAgents.Memory` (`docs/taxonomy.md`), which are

  disposable projections of messages that still exist underneath them: delete

  a memory and nothing can rebuild it, because the sentence a reader typed once

  is the only copy.

  Two buckets, kept distinct because they earn attention differently:

  * `user` — the reader said "remember that I prefer X". Explicit only, never

    inferred from what a turn happened to contain.

  * `learned` — server-side consolidation over thread events produced it. It

    carries `source_ref` so a wrong learning is traced back to the work that

    taught it.

  `superseded_by_id` is how a correction lands. The replacement is a new row

  and the old row points at it, so the store keeps the chain rather than

  overwriting the mistake. Nothing here updates `body`: a memory's text is

  fixed for the life of the row.

  `user_id` is set on the struct and never cast, so a request body cannot name

  whose memory it is writing.

  """

  use Ecto.Schema

  import Ecto.Changeset

  alias OpenAgents.Accounts.User

  @primary_key {:id, :binary_id, autogenerate: true}

  @foreign_key_type :binary_id

  @timestamps_opts [type: :utc_datetime_usec]

  @buckets ~w(user learned)

  @default_bucket "user"

  @body_characters 2_000

  @source_ref_characters 200

  schema "memories" do

    belongs_to :user, User

    field :bucket, :string, default: "user"

    field :body, :string, redact: true

    field :source_ref, :string

    field :embedding, {:array, :float}

    field :embedding_model, :string

    # The generated `tsvector` the lexical stand-in ranks over. PostgreSQL

    # writes it; nothing here reads it back, so it never rides a select.

    field :search_vector, :string, load_in_query: false

    belongs_to :superseded_by, __MODULE__, foreign_key: :superseded_by_id

    timestamps()

  end

  @type t :: %__MODULE__{}

  @doc "The buckets a memory may be written into."

  @spec buckets() :: [String.t()]

  def buckets, do: @buckets

  @doc "The bucket a write lands in when it names none."

  @spec default_bucket() :: String.t()

  def default_bucket, do: @default_bucket

  @doc "The longest body the store accepts, in characters."

  @spec body_characters() :: pos_integer()

  def body_characters, do: @body_characters

  @doc """

  Validates one new memory. The owner is not cast: pass it on the struct.

  """

  @spec changeset(t(), map()) :: Ecto.Changeset.t()

  def changeset(memory, attrs) do

    memory

    |> cast(attrs, [:bucket, :body, :source_ref, :embedding, :embedding_model])

    |> update_change(:body, &trim/1)

    |> update_change(:source_ref, &trim/1)

    |> validate_required([:bucket, :body])

    |> validate_inclusion(:bucket, @buckets)

    |> validate_length(:body, min: 1, max: @body_characters, count: :graphemes)

    |> validate_length(:source_ref, min: 1, max: @source_ref_characters, count: :graphemes)

    |> foreign_key_constraint(:user_id)

    |> check_constraint(:body, name: :memories_shape)

  end

  @doc "Points a memory at the memory that replaced it."

  @spec supersede_changeset(t(), t()) :: Ecto.Changeset.t()

  def supersede_changeset(memory, replacement) do

    memory

    |> change(superseded_by_id: replacement.id)

    |> check_constraint(:superseded_by_id, name: :memories_shape)

  end

  defp trim(value) when is_binary(value), do: String.trim(value)

  defp trim(value), do: value

end

defmodule OpenAgents.Memories.Note do

  @moduledoc """

  The `[From memory: …]` block a recalled turn carries.

  The convention is the retrieval rails': the capability catalog and the

  knowledge base each attach what they found as a bracketed note the model

  reads as context rather than as a tool result, and memory reads the same way.

  One line per memory, each labelled with its bucket and its age, because a

  model shown "you use pnpm" without knowing whether the reader said so

  yesterday or eighteen months ago cannot weigh it.

  When the bounds excluded something, the block says so in its last line. A

  note that trailed off would leave the model believing it had been told

  everything the account remembers.

  """

  alias OpenAgents.Memories.{Memory, Recall}

  alias OpenAgentsWeb.RelativeTime

  @doc """

  Renders one recall as a note, or `nil` when there is nothing to say.

  An empty recall renders `nil` rather than an empty block, so a turn with no

  memories behind it is exactly the turn it was before recall existed.

  """

  @spec render(Recall.t()) :: String.t() | nil

  def render(%Recall{memories: []}), do: nil

  def render(%Recall{memories: memories, dropped: dropped}) do

    (Enum.map(memories, &line/1) ++ omission(dropped))

    |> Enum.join("\n")

  end

  defp line(%Memory{} = memory) do

    "[From memory: #{memory.bucket}, #{age(memory)}] #{memory.body}"

  end

  defp omission(0), do: []

  defp omission(dropped) do

    [

      "[From memory: #{dropped} more #{noun(dropped)} not attached; " <>

        "this turn's recall is bounded to #{OpenAgents.Memories.maximum_attached()} entries " <>

        "and #{OpenAgents.Memories.maximum_attached_characters()} characters.]"

    ]

  end

  defp noun(1), do: "memory was"

  defp noun(_many), do: "memories were"

  defp age(%Memory{inserted_at: at}) do

    case RelativeTime.ago(at) do

      nil -> "age unknown"

      ago -> ago

    end

  end

end

defmodule OpenAgents.Memories.Recall do

  @moduledoc """

  What one turn's recall found, and what it left behind.

  `dropped` is the point of the struct. Recall is bounded — by how many

  memories may attach and by how many characters they may spend — and a bound

  that silently discards is a bound nobody can debug. So the count of what did

  not fit travels with what did, and the note states it in words rather than

  trailing off.

  """

  alias OpenAgents.Memories.Memory

  alias OpenAgents.Memories.Retrieval

  @enforce_keys [:memories, :dropped, :backend]

  defstruct [:memories, :dropped, :backend]

  @type t :: %__MODULE__{

          memories: [Memory.t()],

          dropped: non_neg_integer(),

          backend: Retrieval.backend()

        }

end

defmodule OpenAgents.Memories.Retrieval do

  @moduledoc """

  The swappable boundary that decides which memories a turn is about.

  Two implementations sit behind one behaviour:

  * `OpenAgents.Memories.Retrieval.Semantic` — **the target.** It embeds the

    incoming input and ranks the account's live memories by cosine similarity

    over the embeddings stored on the rows.

  * `OpenAgents.Memories.Retrieval.Lexical` — **the stand-in.** PostgreSQL

    full-text scoring, marked as a stand-in in the same way the capability

    rail's lexical scorer is (`OpenAgents.Tools.Selector`). It is here so

    recall works on a deployment with no embedding credential, not because

    word overlap is the right way to decide what a sentence is about.

  `rank/3` chooses: the semantic backend when the embedding rail is configured

  and answers, the lexical stand-in otherwise, and the stand-in again when the

  provider errors mid-turn. A turn is never starved of memory because the

  embedding provider is cold, and it is never failed by one either — an

  unreachable backend recalls nothing rather than raising.

  Scores are comparable only within a backend. What crosses the boundary is the

  ordering and the backend's own `floor/0`, never a number a caller interprets.

  """

  alias OpenAgents.Memories.Memory

  alias OpenAgents.Memories.Retrieval.{Lexical, Semantic}

  require Logger

  @typedoc "One ranked memory and the backend's score for it."

  @type ranked :: {Memory.t(), float()}

  @typedoc "Which backend produced a ranking."

  @type backend :: :semantic | :lexical

  @doc """

  Scores `candidates` against `query`, as a map of memory id to score.

  `user_id` is passed rather than inferred from the candidates because

  MEMORY-010 requires the account boundary to be a predicate in every query a

  backend issues. A backend that reads PostgreSQL names the column; the list of

  candidate ids narrows the read, it does not scope it.

  """

  @callback score(user_id :: String.t(), query :: String.t(), candidates :: [Memory.t()]) ::

              {:ok, %{optional(String.t()) => float()}} | :error

  @doc "The score a `learned` memory must clear before it interrupts a turn."

  @callback floor() :: float()

  @doc "Whether this backend is configured on this deployment."

  @callback available?() :: boolean()

  @doc """

  Ranks `candidates` against `query`, highest first.

  Returns `{backend, ranked, floor}`. Every candidate appears, scored; the

  caller decides what the floor means for each bucket, because a `user` memory

  the reader asked for is attached whether or not it shares vocabulary with

  this turn, and a `learned` one is not.

  """

  @spec rank(String.t(), String.t(), [Memory.t()]) :: {backend(), [ranked()], float()}

  def rank(user_id, query, candidates)

      when is_binary(user_id) and is_binary(query) and is_list(candidates) do

    rank(user_id, query, candidates, backend())

  end

  @doc "Ranks with a named backend. Falls back to the stand-in when it cannot answer."

  @spec rank(String.t(), String.t(), [Memory.t()], module()) :: {backend(), [ranked()], float()}

  def rank(_user_id, _query, [], module), do: {name(module), [], module.floor()}

  def rank(user_id, query, candidates, module) do

    case module.score(user_id, query, candidates) do

      {:ok, scores} ->

        {name(module), ordered(candidates, scores), module.floor()}

      :error when module != Lexical ->

        Logger.info("memory_retrieval_fell_back backend=#{name(module)}")

        rank(user_id, query, candidates, Lexical)

      :error ->

        {name(module), ordered(candidates, %{}), module.floor()}

    end

  end

  @doc "The backend this deployment ranks with."

  @spec backend() :: module()

  def backend do

    if Semantic.available?(), do: Semantic, else: Lexical

  end

  @doc "The short name of a backend module, for a note or a log line."

  @spec name(module()) :: backend()

  def name(Semantic), do: :semantic

  def name(_lexical), do: :lexical

  # Highest score first, then newest first, so an account whose memories all

  # score zero still gets a stable, meaningful order rather than table order.

  defp ordered(candidates, scores) do

    candidates

    |> Enum.map(&{&1, Map.get(scores, &1.id, 0.0)})

    |> Enum.sort_by(fn {memory, score} -> {-score, -stamp(memory)} end)

  end

  defp stamp(%Memory{inserted_at: %DateTime{} = at}), do: DateTime.to_unix(at, :microsecond)

  defp stamp(_memory), do: 0

end

defmodule OpenAgents.Memories.Retrieval.Lexical do

  @moduledoc """

  PostgreSQL full-text scoring over memory bodies. **This is the stand-in, not

  the target.**

  The workspace retrieval rule is that user-facing retrieval routes on meaning,

  not on words: embeddings and cosine similarity, which

  `OpenAgents.Memories.Retrieval.Semantic` implements. This module exists so a

  deployment with no embedding credential still recalls something, and it is

  marked here the way the capability rail marks its own lexical scorer

  (`OpenAgents.Tools.Selector`) rather than being presented as the answer.

  What it cannot do is the reason the marking matters. "Remember I use pnpm,

  not npm" shares no word with "install the deps", so word overlap scores that

  pair at zero. The `user` bucket is attached regardless of score for exactly

  that reason (`OpenAgents.Memories.recall/3`), which is a bound on the damage

  rather than a fix. Only the semantic backend actually connects the two.

  Scoring is `ts_rank_cd` over the generated `search_vector` column, read

  through the partial GIN index on live rows, under the `english` text-search

  configuration so that stop words drop out and words stem.

  The query is the turn's words joined with `or`, not with the `and` that

  `websearch_to_tsquery` defaults to. A turn is a sentence, not a search box:

  requiring every word of "the migration failed" to appear in a memory would

  match nothing an account ever wrote. Sharing one content word is the bar,

  which is the same bar the issue states for the `learned` bucket, and stop

  words are gone before it is applied so "the" cannot clear it.

  """

  @behaviour OpenAgents.Memories.Retrieval

  import Ecto.Query

  alias OpenAgents.Memories.Memory

  alias OpenAgents.Repo

  # Only what fits a text-search query. A whole conversation turn pasted into

  # `websearch_to_tsquery` costs more to parse than the ranking is worth.

  @maximum_query_characters 512

  @maximum_query_words 64

  # A word the parser would read as an operator rather than as a word. They are

  # stripped so a turn saying "or" cannot produce `or or or` and fail to parse.

  @operators ~w(or and not)

  @impl true

  def available?, do: true

  # Any match at all. `ts_rank_cd` is unnormalized and its magnitude means

  # nothing across queries, so the only honest floor is "the words appear".

  @impl true

  def floor, do: 0.0

  @impl true

  def score(user_id, query, candidates) do

    text = prepare(query)

    ids = Enum.map(candidates, & &1.id)

    if text == "" or ids == [] do

      {:ok, %{}}

    else

      {:ok, ranked(user_id, text, ids)}

    end

  rescue

    _error -> :error

  end

  # `user_id` is the scope predicate, and it is written here rather than

  # inherited from the candidate ids (MEMORY-010). The id list narrows the

  # read; it does not bound it, and a caller that assembled that list wrongly

  # would otherwise reach another account's rows through this query.

  defp ranked(user_id, text, ids) do

    from(memory in Memory,

      where: memory.user_id == ^user_id,

      where: is_nil(memory.superseded_by_id),

      where: memory.id in ^ids,

      where: fragment("? @@ websearch_to_tsquery('english', ?)", memory.search_vector, ^text),

      select: {

        memory.id,

        fragment(

          "ts_rank_cd(?, websearch_to_tsquery('english', ?), 32)",

          memory.search_vector,

          ^text

        )

      }

    )

    |> Repo.all()

    |> Map.new(fn {id, rank} -> {id, rank / 1} end)

  end

  # `websearch_to_tsquery` reads `-` as negation, quotes as phrases, and `or`

  # as disjunction. A turn carrying punctuation would ask for something the

  # reader did not, so everything but letters and digits goes first; then the

  # words are rejoined with `or`, which is the operator this wants and not the

  # one the parser assumes.

  defp prepare(query) do

    query

    |> String.slice(0, @maximum_query_characters)

    |> String.replace(~r/[^\p{L}\p{N}\s]/u, " ")

    |> String.split(~r/\s+/u, trim: true)

    |> Enum.reject(&(&1 in @operators))

    |> Enum.take(@maximum_query_words)

    |> Enum.join(" or ")

  end

end

defmodule OpenAgents.Memories.Retrieval.Semantic do

  @moduledoc """

  Cosine similarity over memory embeddings. **This is the target backend.**

  Retrieval routes on meaning here rather than on shared words, which is what

  lets "remember I use pnpm, not npm" reach a later "install the deps" — a pair

  the lexical stand-in scores at zero because they have no word in common.

  An account holds few memories, and `OpenAgents.Memories` caps how many are

  live, so the vectors are compared in this process rather than through an

  index. That is the same trade the tool catalog makes in

  `OpenAgents.Tools.Embeddings`: a bounded set does not earn a per-turn

  pgvector query. The embedding itself is written on the row at create time by

  `OpenAgents.Memories.create/2`, under the same provider boundary the rest of

  this repository embeds through (`OpenAgents.Memory.EmbeddingProvider`).

  Every failure degrades rather than raising. No credential, a provider that

  errors, a query the provider will not embed, a row whose vector was written

  under a different model — each one leaves the caller falling back to the

  stand-in or scoring nothing, never failing the turn.

  """

  @behaviour OpenAgents.Memories.Retrieval