Seal non-searched private content columns at rest under ContentVault (#193)

f3b54f298fea · AtlantisPleb · · parent c9e6fc8e671a

Seal non-searched private content columns at rest under ContentVault (#193)

Deploy story

What this commit did to the running system — joined from the forge receipt chain, the part a commit page elsewhere cannot show.

pushed
by user · WAL seq 412 · 2026-08-25T18:19:04.915470Z

Changed files

  • modified INVARIANTS.md
  • modified config/config.exs
  • modified config/dev.exs
  • modified config/runtime.exs
  • modified config/test.exs
  • modified docs/2026-08-25-encryption-at-rest.md
  • added docs/2026-08-25-ox-alpha-stress-test-log.md
  • modified docs/runtime-configuration.md
  • modified docs/security/secrets-and-log-handling.md
  • modified infra/staging/main.tf
  • modified infra/staging/templates/fleet-startup.sh.tftpl
  • modified lib/openagents/application.ex
  • added lib/openagents/content_vault.ex
  • modified lib/openagents/forge/at_rest.ex
  • modified lib/openagents/preferences/observation.ex
  • modified lib/openagents/projects/project_note.ex
  • modified lib/openagents/runtime_config.ex
  • modified lib/openagents/timeline.ex
  • modified lib/openagents/voice/retention.ex
  • modified lib/openagents/voice/session.ex
  • modified lib/openagents/voice/transcript_item.ex
  • modified lib/openagents_web/controllers/project_json.ex
  • modified lib/openagents_web/live/project_show_live.ex
  • modified ops/ci/release-smoke.sh
  • modified ops/deploy/fleet-startup.template.sh
  • modified ops/relup-proof/common.sh
  • modified ops/staging/gate-5-profile.sh
  • modified ops/staging/validate-isolation.sh
  • modified priv/migration_lineages/prior-2026-08-19.json
  • added priv/repo/migrations/20260825170000_add_private_content_ciphertext_columns.exs
  • added priv/repo/migrations/20260825170100_seal_private_content.exs
  • modified test/openagents/forge/at_rest_test.exs
  • modified test/openagents/forge/key_rotation_test.exs
  • modified test/openagents/project_notes_test.exs
  • modified test/openagents/projects_test.exs
  • modified test/openagents/runtime_config_test.exs
  • added test/openagents/seal_private_content_migration_test.exs
  • modified test/openagents/voice_sessions_test.exs
  • modified test/openagents/voice_test.exs

Diff

39 files changed, +1652 -62

INVARIANTS.md modified +78 -18

@@ -3069,8 +3069,24 @@ than its sound, but the operator route named in ADMIN-001 unseals and streams

3069 3069
it; the seal here defends against a stolen database, not against the person who
3070 3070
holds the key.
3071 3071
3072
Amended 2026-08-25 (issue #193). The asymmetry this contract carried is closed.
3073
`voice_transcript_items.content` used to rest as plaintext beside the sealed
3074
audio, so a stolen database got the words either way and the seal on the audio
3075
moved nothing. The transcript is now sealed too, under `OpenAgents.ContentVault`
3076
and its own key, along with `voice_sessions.compaction_summary`. Both are read
3077
through the schema — `OpenAgents.Voice.TranscriptItem.text/1` and
3078
`OpenAgents.Voice.Session.compaction_summary/1` — so no reader reaches the
3079
words by touching a field. What is unchanged is the boundary: these seals are
3080
under a key the operator holds, so they defend against a stolen database and
3081
not against the operator, exactly as the paragraph above says of the audio. The
3082
same words still rest in `messages.content`, which carries a generated
3083
`search_vector` and cannot be sealed without ending lexical recall; that is
3084
recorded in `EXIT-006` and in `docs/2026-08-25-encryption-at-rest.md` rather
3085
than implied here.
3086
3072 3087
Evidence: `OpenAgents.Voice.Recordings`, `OpenAgents.Voice.Recording`,
3073
`OpenAgents.Voice.RecordingChunk`, `OpenAgents.Voice.RecordingVault`, the
3088
`OpenAgents.Voice.RecordingChunk`, `OpenAgents.Voice.RecordingVault`,
3089
`OpenAgents.ContentVault`, `OpenAgents.Voice.TranscriptItem`, the
3074 3090
`create_voice_recordings` migration, `OpenAgentsWeb.VoiceRecordingController`,
3075 3091
`assets/js/voice_recording.mjs`, `OpenAgents.Voice.RecordingsTest`,
3076 3092
`OpenAgentsWeb.VoiceRecordingControllerTest`, `assets/test/voice_recording_test.mjs`,

@@ -3499,11 +3515,12 @@ Evidence: `OpenAgents.GitHubOAuth.RuntimeConfig`,

3499 3515
3500 3516
Status: Current
3501 3517
3502
The application holds three hand-rolled encryption vaults —
3518
The application holds four hand-rolled encryption vaults —
3503 3519
`OpenAgents.Accounts.TokenVault` for GitHub access tokens,
3504 3520
`OpenAgents.Machines.TokenVault` for computer tokens awaiting pairing claim,
3505
and `OpenAgents.Voice.RecordingVault` for call audio — and each seals under
3506
its own configured key. Rotating one vault's key never makes another vault's
3521
`OpenAgents.Voice.RecordingVault` for call audio, and `OpenAgents.ContentVault`
3522
for private text nobody searches — and each seals under its own configured
3523
key. Rotating one vault's key never makes another vault's
3507 3524
records unreadable or unverifiable, because no vault reads another vault's
3508 3525
key to seal. A vault whose own key is absent fails with a typed configuration
3509 3526
error at its boundary rather than silently borrowing key material that

@@ -3527,21 +3544,46 @@ requires: the GitHub vault rotates losslessly through its keyed envelope and

3527 3544
keyring rewrap; the pairing vault's own rotation loses at most one ten-minute
3528 3545
window of unclaimed pairings, which retry; the recording vault has no keyring,
3529 3546
so rotating its key strands prior recordings — a bounded, recorded loss, not a
3530
silent one.
3547
silent one. The content vault has no keyring either, so rotating
3548
`CONTENT_ENCRYPTION_KEY` strands every column it seals, together. That is the
3549
price of one vault over several columns rather than one per column, and it is
3550
paid deliberately: the separation that carries weight is credential from
3551
content, and splitting the content columns from each other would buy a finer
3552
blast radius at the cost of one production secret per column while every one of
3553
them stays readable to the same operator through the same application.
3554
3555
The content vault is required rather than optional. `RuntimeConfig.validate/1`
3556
refuses a staging or production boot without its key, because the alternative
3557
is a node that accepts a voice transcript, a compaction summary, a preference
3558
observation, or a project note and then cannot seal it. There is deliberately
3559
no bridge to another vault's key of the kind `config/runtime.exs` still offers
3560
the pairing vault: that bridge is what #192 found and #253 repeated, and a
3561
vault added after both should not reintroduce it.
3531 3562
3532 3563
Amended 2026-08-25 (issue #193). `OpenAgents.Forge.AtRest.sealed_columns/0`
3533
names the column each of these three vaults seals, and
3534
`test/openagents/forge/at_rest_test.exs` reads each column back with raw SQL
3535
after a real write, so "sealed" is checked against PostgreSQL rather than
3536
against the vault's own unit tests. A fourth vault would have to appear there
3537
before `EXIT-006` could count it.
3564
names the column each vault seals, and `test/openagents/forge/at_rest_test.exs`
3565
reads each column back with raw SQL after a real write, so "sealed" is checked
3566
against PostgreSQL rather than against the vault's own unit tests.
3567
3568
Amended 2026-08-25 (issue #193, second pass). The fourth vault is
3569
`OpenAgents.ContentVault`, and it seals content rather than credentials:
3570
`voice_transcript_items.content`, `voice_sessions.compaction_summary`,
3571
`preference_observations.summary`, and `project_notes.body`. Its seals carry
3572
the column and the row identity as additional authenticated data, so ciphertext
3573
lifted from one row or one column does not open as another's sentence. The
3574
plaintext columns those replaced survive one more release, empty, because
3575
dropping a live column mid-roll breaks the nodes still writing into it; the
3576
contract migration removes them the way `machine_pairings.user_id` was removed
3577
a release after its last reader.
3538 3578
3539 3579
Evidence: `OpenAgents.Machines.TokenVault`, `OpenAgents.Accounts.TokenVault`,
3540
`OpenAgents.Voice.RecordingVault`, `OpenAgents.Forge.AtRest`,
3580
`OpenAgents.Voice.RecordingVault`, `OpenAgents.ContentVault`,
3581
`OpenAgents.Forge.AtRest`,
3541 3582
`OpenAgents.RuntimeConfig.validate/1`,
3542 3583
`config/runtime.exs`, `test/openagents/machines/token_vault_test.exs`,
3543 3584
`test/openagents/accounts/token_vault_test.exs`,
3544
`test/openagents/forge/at_rest_test.exs`, and
3585
`test/openagents/forge/at_rest_test.exs`,
3586
`test/openagents/forge/key_rotation_test.exs`, and
3545 3587
`test/openagents/runtime_config_test.exs`.
3546 3588
3547 3589
### RELEASE-003 — Every published hostname can establish LiveView

@@ -4985,12 +5027,30 @@ That is the direction every other gather in this projection fails in.

4985 5027
4986 5028
Amended 2026-08-25 (issue #193). The decision about which columns stop being
4987 5029
server-readable is recorded in `docs/2026-08-25-encryption-at-rest.md` with its
4988
threat model, the cost to an account under an account-held key, and five
4989
rejected options. No content column is encrypted, and the reason is that an
4990
operator-held key protects a stolen dump and nothing else — the claim `EXIT-006`
4991
exists to keep off this page — while an account-held key ends search,
4992
rendering, and the `TRANSPARENCY-001` projections, and makes key loss permanent.
4993
The published key set did not change, so `STATUS-001` has nothing to move.
5030
threat model, the cost to an account under an account-held key, and the options
5031
rejected. The published key set did not change, so `STATUS-001` has nothing to
5032
move.
5033
5034
Amended 2026-08-25 (issue #193, second pass). Four content columns are now
5035
sealed under `OpenAgents.ContentVault` — the voice transcript, the in-call
5036
compaction summary, the preference observation, and the project note — and the
5037
reason is narrower than "encryption at rest": each is read whole and searched
5038
by nothing, so a seal costs no feature. That is the whole test applied. The
5039
columns left in plaintext are left because a query reads them in a way a seal
5040
would end: `messages.content` carries a `search_vector` PostgreSQL generates
5041
from it, `issues.body`, `comments.body`, and `forum_posts.body_text` are
5042
matched with `ILIKE`, and the two `account_chat_runs` columns hold the same
5043
words that rest verbatim in `account_chat_events.payload` beside them, where
5044
the replay path reads them structurally. `plaintext_private_columns/0` now
5045
carries that reason per column, so the ledger records why a gap is open rather
5046
than only that it is.
5047
5048
`encrypted_at_rest` is still `false`, and sealing four columns did not move it,
5049
which is the point. The boolean is `true` only when no private column rests as
5050
plaintext, and `operator_reads_source` is its negation, so this change cannot
5051
be read as protection from the operator: these seals are under a key the
5052
operator holds and defend against a stolen dump. `EXIT-006` exists to keep the
5053
larger claim off the page, and it still does.
4994 5054
4995 5055
Amended 2026-08-24 (issue #178). The decision that a private export can be
4996 5056
encrypted to a key the operator does not hold is recorded in
config/config.exs modified +1

@@ -396,6 +396,7 @@ config :openagents,

396 396
  github_token_decryption_keys: %{},
397 397
  machine_token_encryption_key: nil,
398 398
  voice_recording_encryption_key: nil,
399
  content_encryption_key: nil,
399 400
  inference_proxy_url: nil,
400 401
  inference_grant_max_total_tokens: 2_000_000,
401 402
  inference_grant_max_calls: 64,
config/dev.exs modified +13

@@ -49,6 +49,19 @@ config :openagents,

49 49
       System.get_env("MACHINE_TOKEN_ENCRYPTION_KEY") ||
50 50
         Base.encode64("openagents-dev-machine-vault-key")
51 51
52
# The voice recording vault and the content vault, each with its own key for
53
# the same reason. Sealing the audio under one key and the words under another
54
# is the whole point of VAULT-001: neither opens the other.
55
config :openagents,
56
       :voice_recording_encryption_key,
57
       System.get_env("VOICE_RECORDING_ENCRYPTION_KEY") ||
58
         Base.encode64("openagents-dev-recording-vaultkey")
59
60
config :openagents,
61
       :content_encryption_key,
62
       System.get_env("CONTENT_ENCRYPTION_KEY") ||
63
         Base.encode64("openagents-dev-content-vault-key3")
64
52 65
# For development, we disable any cache and enable
53 66
# debugging and code reloading.
54 67
#
config/runtime.exs modified +4

@@ -490,6 +490,10 @@ if config_env() == :prod and runtime_role == :web do

490 490
    voice: voice,
491 491
    voice_recording: voice_recording,
492 492
    voice_recording_encryption_key: optional_text.("VOICE_RECORDING_ENCRYPTION_KEY"),
493
    # The content vault's own key. VAULT-001 forbids the bridge that #192
494
    # found and #253 repeated, so there is deliberately no fallback here: an
495
    # unset key is an unset key, and RuntimeConfig refuses the boot.
496
    content_encryption_key: optional_text.("CONTENT_ENCRYPTION_KEY"),
493 497
    voice_recovery_worker_enabled: voice_enabled,
494 498
    voice_retention_worker_enabled: voice_retention_enabled,
495 499
    voice_retention_enabled: voice_retention_enabled,
config/test.exs modified +5

@@ -130,6 +130,11 @@ config :openagents, :scv_codex,

130 130
131 131
config :openagents, :voice_recording_encryption_key, Base.encode64(:crypto.strong_rand_bytes(32))
132 132
133
# The content vault's own key, generated per run and shared with nothing.
134
# VAULT-001: a recording key must not open a transcript, and neither opens a
135
# GitHub token.
136
config :openagents, :content_encryption_key, Base.encode64(:crypto.strong_rand_bytes(32))
137
133 138
config :openagents, :voice_recording,
134 139
  enabled: true,
135 140
  timeslice_ms: 5_000,
docs/2026-08-25-encryption-at-rest.md modified +154 -4

@@ -4,6 +4,8 @@

4 4
**Issue:** #193
5 5
**Parent:** #94
6 6
**Companion:** `docs/2026-08-24-private-export-encryption.md`
7
**Revised:** 2026-08-25, second pass. Section 8 records what changed and why
8
the first pass's answer was narrower than the question.
7 9
8 10
#178 asked whether a private export can be encrypted to a key the operator does
9 11
not hold. It can, and is. That decision deliberately left the other half open,

@@ -130,7 +132,10 @@ another. `cloak_ecto`'s default is a single global keyring, which is the shape

130 132
already here would trade a proven property for a familiar name.
131 133
132 134
**Seal `voice_transcript_items.content` alone, since the audio beside it is
133
sealed.** Rejected, and it was the closest call. The asymmetry is real, but the
135
sealed.** Rejected in the first pass, and it was the closest call. **Overturned
136
in section 8** — the reasoning below proves the seal does not close the *threat*
137
while `messages.content` is searchable, which is true, and then wrongly
138
concludes that it is not worth doing. The asymmetry is real, but the
134 139
same words rest in `messages.content`, which is searched. Sealing one and not
135 140
the other would move a number without moving the threat model, which is the
136 141
failure mode this whole document is written against. It is recorded in section

@@ -188,10 +193,155 @@ it earned it.

188 193
189 194
## 7. What is still open
190 195
191
- **Content columns are plaintext, and the operator reads them.** That is the
192
  decision, not a gap in it. #193 stays open because the acceptance criteria
193
  say it does: encryption did not land, and `/status` keeps publishing `false`.
196
Sections 1 through 6 are the first pass and are kept as written. Section 8
197
revises them: four content columns are sealed now, and the list below is
198
narrower than it was. Read section 8 for the current state.
199
200
- **Some content columns are plaintext, and the operator reads them.** Section
201
  8.2 names each one and the query that keeps it readable. `/status` keeps
202
  publishing `encrypted_at_rest: false` while any of them remains, which is the
203
  outcome `EXIT-006` exists to produce.
194 204
- **No operator read is audited.** `ADMIN-001`. An access log the operator
195 205
  writes into the operator's own database is evidence to the operator and to
196 206
  nobody else; #151 and #168 carry the external anchor.
197 207
- **The transcript/audio asymmetry.** Section 4. Recorded, not closed.
208
209
## 8. Second pass: seal everything a seal costs nothing
210
211
The first pass above answered "should content be encrypted?" with a threat
212
model, and the threat model said an operator-held key protects a stolen dump
213
and nothing else. That is still true, and nothing below claims otherwise. But
214
it answered a question nobody had to ask as one question, when it is two:
215
216
1. **Should content stop being server-readable?** That needs an account-held
217
   key, it ends search and rendering, and key loss becomes permanent. Section 4
218
   still stands, and nobody has asked for it.
219
2. **Should content that nothing reads still rest readable in a stolen dump?**
220
   No. A seal that costs no feature is worth having even when it only defends
221
   against theft, and "it does not defend against the operator" is an argument
222
   for not *claiming* more, not for leaving plaintext on disk.
223
224
The first pass collapsed the two and answered only the first. The owner's
225
decision is the second: **encrypt as much as we can.** What follows is what
226
that turned out to mean.
227
228
### 8.1 The test that decides each column
229
230
A column is sealed when nothing reads it except whole. A column stays plaintext
231
when a query reads it in a way a seal would end — and the query is named, so
232
the reason is checkable rather than asserted.
233
234
That is the entire test. It is not about how sensitive the words are; it is
235
about whether encryption costs a working feature. `INVARIANTS.md` preamble is
236
the reason it is written this way: a gap left open needs a reason a reader can
237
falsify.
238
239
### 8.2 The inventory, from `information_schema`
240
241
Every `text`, `varchar`, and `bytea` column whose name carries content-shaped
242
vocabulary — `body`, `content`, `text`, `transcript`, `message`, `prompt`,
243
`summary`, `description`, `note`, `payload`, `data` — was listed from the
244
catalog rather than from memory. Of 972 text-shaped columns, 38 matched, and
245
most of those are digests, enum-ish kinds, or public metadata. The private
246
content columns, and what happened to each:
247
248
**Sealed under `OpenAgents.ContentVault`:**
249
250
| Column | Readers | Why a seal costs nothing |
251
| --- | --- | --- |
252
| `voice_transcript_items.content` | one: `OpenAgents.Timeline`, whole then truncated | no index, no predicate, no export path |
253
| `voice_sessions.compaction_summary` | none — written, held in process state, purged | never read back from PostgreSQL at all |
254
| `preference_observations.summary` | none — hashed into `evidence_digest` at write | the digest commits to the words; the column is not read |
255
| `project_notes.body` | REST JSON and the project page, whole | no predicate, no index; rendering decrypts |
256
257
**Left plaintext, with the query that keeps it there:**
258
259
| Column | What reads it |
260
| --- | --- |
261
| `messages.content` | a `search_vector` PostgreSQL *generates* from this column, indexed `USING GIN`, driving `OpenAgents.Memory.LexicalRecall`. Sealing it ends lexical recall over your own history. This is the one the owner named, and it is the one that cannot move. |
262
| `issues.body` | `OpenAgents.Issues.search/2` and `OpenAgents.Issues.TaskReferences`, both `ILIKE` |
263
| `comments.body` | `OpenAgents.Issues.TaskReferences`, `ILIKE` |
264
| `forum_posts.body_text` | `OpenAgents.Forum.search/2`, `ILIKE` |
265
| `account_chat_runs.user_content` | nothing searches it — but the same words rest verbatim in `account_chat_events.payload`, which the replay path reads structurally. Sealing the text column alone moves the plaintext eight inches sideways. |
266
| `account_chat_runs.assistant_content` | the same, plus the `completion` map beside it |
267
268
**Not private content**, so out of scope rather than left: digests
269
(`content_digest`, `payload_digest`), enum-ish kinds (`content_kind`,
270
`transcript_kind`, `data_type`), token counters, public repository and project
271
descriptions, published changelog and incident summaries, and
272
`verified_artifact_listings.owner_description`, which is marketplace copy the
273
listing exists to publish.
274
275
`account_chat_runs` is the honest disappointment here. It is real private
276
conversation, nothing searches it, and it is still plaintext — because the
277
duplicate beside it is jsonb the streaming replay path reads by key, and
278
sealing that is its own change with its own decisions about what stays
279
structured. `plaintext_private_columns/0` carries that reason, so it is a
280
tracked gap rather than a silence.
281
282
### 8.3 One vault, four columns, its own key
283
284
`OpenAgents.ContentVault` is the fourth vault. AES-256-GCM, versioned framing,
285
and additional authenticated data naming the column and the row, so ciphertext
286
lifted out of one row does not open as another's sentence.
287
288
It is one vault over four columns rather than four vaults. `VAULT-001`'s
289
property is that no vault reads another vault's key, and that holds: this vault
290
reads `:content_encryption_key` and nothing else, with no bridge of the kind
291
`config/runtime.exs` still offers the pairing vault — the bridge #192 found and
292
#253 repeated. What one-vault-per-column would buy is a finer rotation blast
293
radius, at the price of one production secret per column, for columns that are
294
all readable to the same operator through the same application. The cost is
295
recorded instead: rotating this key strands all four columns together.
296
297
The key is **required**, not optional. Without it a transcript, a summary, an
298
observation, and a note each refuse to be written rather than being written
299
readable, so `RuntimeConfig.validate/1` refuses a staging or production boot
300
without it. `cloak_ecto` is still rejected, for the reason section 5 gives.
301
302
### 8.4 Expand and contract, because the fleet rolls
303
304
Each column got a sibling `*_ciphertext` column, a backfill that seals every
305
existing row and nulls the plaintext, and a `_present` check constraint so "this
306
row has text" stays true while both halves are live. The plaintext columns are
307
still declared and still read as a fallback, because `RELEASE-006`'s rolling
308
replacement leaves nodes on the previous release writing into them. They are
309
dropped by a contract migration a release later, the way
310
`machine_pairings.user_id` was.
311
312
Two limits, recorded rather than implied:
313
314
- **Dead tuples.** `UPDATE ... SET content = NULL` writes a new row version and
315
  leaves the old one on disk until autovacuum reclaims it. The plaintext
316
  survives in dead tuples for a bounded window after the backfill.
317
- **The roll window.** A node still running the previous release writes
318
  plaintext for as long as the replacement takes. Those rows are readable until
319
  the contract migration, which is why the fallback reader exists.
320
321
### 8.5 What the status page says now
322
323
`encrypted_at_rest` is still `false`, and sealing four columns did not move it.
324
That is the design working. The boolean is `true` only when *no* private column
325
rests as plaintext, and `operator_reads_source` is its negation, so this change
326
cannot be mistaken for protection from the operator. `EXIT-006` exists to keep
327
that claim off the page, and it still does.
328
329
### 8.6 Proof
330
331
`test/openagents/forge/at_rest_test.exs` gained a `describe` block that, for
332
each sealed column, writes through the application path a person reaches and
333
then asks PostgreSQL two questions with raw SQL: is the ciphertext column free
334
of the words, and is the plaintext column it replaced empty. The second is the
335
one a round-trip test cannot ask — a schema that seals on write and opens on
336
read passes a round trip whether or not anything reached disk.
337
338
`test/openagents/forge/key_rotation_test.exs` adds the two questions #253 is
339
about: another vault's rotation must not reach sealed content, and this vault's
340
own rotation must strand it rather than falling back to a keyring it does not
341
have.
342
343
The population still comes from `information_schema`. `content_ciphertext`,
344
`summary_ciphertext`, `compaction_summary_ciphertext`, and `body_ciphertext`
345
all match the secret-shaped pattern on `cipher`, so each had to be classified
346
before the proof would go green — the same mechanism that catches a plaintext
347
token column now catches an unclassified ciphertext one.
docs/2026-08-25-ox-alpha-stress-test-log.md added +94

@@ -0,0 +1,94 @@

1
# Ox Alpha Stress Test & High-Throughput Token Benchmark Log
2
3
Date: 2026-08-25
4
Status: In Progress
5
Author: OpenAgents Coder & Christopher David
6
7
## 1. Objectives & Context
8
9
As discussed in Episode 275 ("Parallelizing Ox Alpha Stress Testing"), the objective is to determine the maximum sustainable tokens per second (TPS) and parallel capacity of `stealth/ox-alpha` across available providers (OpenRouter, OpenCode Zen, Nous Portal, Venice) without degrading into rate limit errors (`429`), quota exhaustion, or merge conflicts.
10
11
Initial findings are collected and logged here before running live backlog burns on video.
12
13
---
14
15
## 2. Benchmark & Concurrency Ladder Run Receipts
16
17
### Baseline Probe (Concurrency = 2)
18
- **Model / Lane:** `ox-alpha` (via OpenAgents inference proxy -> OpenRouter `stealth/ox-alpha`)
19
- **Task:** 300-word architectural analysis of swarm scaling & token economics
20
- **Result:**
21
  - 2 of 2 workers completed cleanly.
22
  - Per-worker generation latency: ~18-22s.
23
  - Estimated output generation: ~800 tokens/worker (including hidden reasoning).
24
  - Effective aggregate generation rate: ~75–90 tokens/sec across 2 lanes.
25
  - HTTP status: 200 OK (no 429s).
26
27
### Concurrency Ladder Step 1 (Concurrency = 4)
28
- **Model / Lane:** `ox-alpha`
29
- **Task:** 250-word scaling analysis on generation ceilings, prompt caching, and throughput bottlenecks
30
- **Result:**
31
  - 4 of 4 workers completed concurrently without dropouts.
32
  - System load: 4.47 (16 idle CPU threads available, memory clean).
33
  - Effective aggregate generation rate: ~160–190 tokens/sec across 4 lanes.
34
  - Error rate: 0.0% (0 / 4 failed).
35
36
---
37
38
## 3. Key Lessons Learned & Bottleneck Hierarchy
39
40
1. **Output Generation vs. TPM/Context Prefill:**
41
   - Single-agent output decode rate sits comfortably around 20–30 TPS.
42
   - The primary bottleneck in agentic loops is **input token volume and TPM ceilings**, not output decode speed. A 50k-token repository context re-sent across 4 agents consumes 200k input tokens per step, rapidly hitting Free-tier TPM ceilings (e.g., Nous Free tier observed at 500k TPM = max 2–3 active heavy workers).
43
44
2. **Prompt Caching is Mandatory for Fan-Out:**
45
   - 80–90% of prefix data (system prompts, tool definitions, file headers) is static across sibling workers.
46
   - Seeding workers with shared context prefixes ensures cache hits, minimizing time-to-first-token (TTFT) and saving API quota.
47
48
3. **Rate Limits & Provider Topologies:**
49
   - **OpenRouter:** Free model pool has a documented 20 RPM limit, plus daily request quotas (50/day before $10 lifetime credit; 1,000/day after).
50
   - **Nous Portal:** Observed 50 RPM / 500K TPM on Free plan; paid tiers scale up to 1,600 RPM / 16M TPM.
51
   - **OpenCode Zen (`x-preview-f-free`):** Free Ox Alpha endpoint can be subject to upstream availability (`503` under load).
52
53
4. **Next Step for Cloud Computers / Box Infrastructure:**
54
   - To scale past 8 concurrent agents safely on complex repositories, tasks must be isolated to separate git worktrees or cloud sandbox containers (Box / Cloud Computer leases) to avoid workspace lock contention and serial merge bottlenecks.
55
56
---
57
58
## 4. Next Ladder Steps Planned
59
60
- [x] Concurrency 2: Verified (~80 TPS)
61
- [x] Concurrency 4: Verified (~180 TPS)
62
- [ ] Concurrency 8: Test short burst synthesis across 8 workers
63
- [ ] Concurrency 15: Maximum planned fan-out test with Box/worktree isolation
64
65
## 5. Box / Cloud Computer Fleet Integration Plan
66
67
### Architecture & Current Substrate
68
The backend in `openagents.com` already holds the complete Box runtime and delegation primitives:
69
1. **Core Client & Fleet (`OpenAgents.Box`, `OpenAgents.Box.Fleet`, `OpenAgents.Box.Fanout`):**
70
   - Implements bounded multi-box allocation with labels, queues, and cost ceilings (Issue #109).
71
   - Manages active box caps per conversation (default 2, budgeted grant up to 10/15).
72
2. **Durable Runs (`OpenAgents.BoxRuns`):**
73
   - Asynchronously runs OpenCode / Coder commands on remote sandboxes (Issue #107) with resumption and bounded output tailing.
74
3. **Forge Issue Assignment (`OpenAgents.Forge.Assignments`):**
75
   - Delivers short-lived, branch-scoped credentials to a box container (Issue #108) so a remote OpenCode harness can clone, implement, run checks, and push a branch.
76
4. **Unified Delegation Layer (`OpenAgents.Delegations`):**
77
   - Provides a single seam across local harnesses and remote Box targets (`target_type: "box"`).
78
79
### Next Steps for Fan-Out Execution
80
1. **Bridge Coder Session with Box Fleet Fan-out:**
81
   - File a dedicated issue in Project 6 to wire `openagents coder` delegation directly to the remote Box fan-out controller (`POST /api/v1/conversations/:id/boxes/fanout` and `POST /api/v1/delegations/dispatch`).
82
2. **Execute Multi-Box Ox Alpha Run:**
83
   - Request 4-8 cloud boxes in parallel, dispatching OpenCode with `stealth/ox-alpha` against distinct backlog issues.
84
   - Collect and compare end-to-end cloud latency, TPS, and receipt delivery against local runs.
85
86
### Concurrency Ladder Step 2 (Concurrency = 8)
87
- **Model / Lane:** `ox-alpha` (8 parallel child coding agents)
88
- **Task:** Prompt caching optimization and TTFT reduction analysis under multi-tenant load
89
- **Result:**
90
  - 8 of 8 children completed concurrently with zero dropouts.
91
  - Per-child generation: ~750–1,200 tokens (output + reasoning).
92
  - Effective aggregate generation rate: **~350–410 tokens/sec** across 8 parallel streams.
93
  - Machine state: Load average 4.75, memory healthy, zero HTTP 429s or rate limit errors.
94
  - Error rate: 0.0% (0 / 8 failed).
docs/runtime-configuration.md modified +1

@@ -160,6 +160,7 @@ arguments, repository URLs, receipts, or checked-in environment files.

160 160
| Providers | `OPENAGENTS_INFERENCE_PROXY_URL` | HTTPS URL without credentials when computers are enabled; empty disables |
161 161
| Computers | `OPENAGENTS_MACHINE_TOKEN_TTL_SECONDS` | `300` through `2592000`; Gate 5 uses the 30-day maximum |
162 162
| Recording | `VOICE_RECORDING_ENCRYPTION_KEY` | Base64-encoded 32-byte key when recording is enabled; empty disables recording storage |
163
| Content | `CONTENT_ENCRYPTION_KEY` | Base64-encoded 32-byte key, required in staging and production. The content vault seals voice transcripts, in-call compaction summaries, preference observations, and project notes. It is the vault's own key: nothing bridges to it and nothing bridges from it (VAULT-001), so an unset value refuses the boot rather than borrowing another vault's key |
163 164
164 165
`PHX_SERVER` is optional for evaluation commands. If set, it must be exactly
165 166
`true` or `false`.
docs/security/secrets-and-log-handling.md modified +1

@@ -38,6 +38,7 @@ use distinct names and values and remains locked.

38 38
| `MACHINE_TOKEN_ENCRYPTION_KEY` | `openagents-staging-machine-vault-active` | web, fleet | Scheduled vault rotation or suspected exposure; rotation strands at most one ten-minute window of unclaimed pairings (VAULT-001). Until this secret exists, `config/runtime.exs` bridges the machine pairing vault to `GITHUB_TOKEN_ENCRYPTION_KEY` |
39 39
| `OPENAI_API_KEY` | `openagents-staging-openai-api-key` | web, fleet | Provider rotation or suspected prompt/log exposure |
40 40
| `VOICE_RECORDING_ENCRYPTION_KEY` | `openagents-staging-voice-recording-key` | web and fleet when recording is admitted | Scheduled recording-key procedure or suspected exposure |
41
| `CONTENT_ENCRYPTION_KEY` | `openagents-staging-content-vault-active` | web, fleet, migrator | Scheduled vault rotation or suspected exposure; the content vault has no keyring, so rotation strands every sealed voice transcript, compaction summary, preference observation, and project note (VAULT-001) |
41 42
| `OPENAGENTS_FORGE_OPERATOR_TOKEN` | `openagents-staging-forge-operator-token` | web, fleet, builder | Scheduled rotation, builder replacement, or suspected URL/argv/log exposure |
42 43
| `RELEASE_COOKIE` | `openagents-staging-release-cookie` | web, fleet, deployer | Fleet-wide coordinated rotation or suspected exposure |
43 44
| Connected Codex account slot 1 | `openagents-staging-scv-codex-operator-1` | web and fleet may add and read versions; an isolated Codex SCV runtime may read one exact version | Operator disconnect, OpenAI reauthentication, or suspected exposure |
infra/staging/main.tf modified +2

@@ -37,6 +37,7 @@ locals {

37 37
    "openagents-staging-github-vault-previous",
38 38
    "openagents-staging-openai-api-key",
39 39
    "openagents-staging-voice-recording-key",
40
    "openagents-staging-content-vault-active",
40 41
    "openagents-staging-forge-operator-token",
41 42
    "openagents-staging-release-cookie",
42 43
    "openagents-staging-scv-codex-operator-1",

@@ -55,6 +56,7 @@ locals {

55 56
    "openagents-staging-github-vault-previous",
56 57
    "openagents-staging-openai-api-key",
57 58
    "openagents-staging-voice-recording-key",
59
    "openagents-staging-content-vault-active",
58 60
    "openagents-staging-forge-operator-token",
59 61
    "openagents-staging-release-cookie"
60 62
  ])
infra/staging/templates/fleet-startup.sh.tftpl modified +2 -1

@@ -126,7 +126,7 @@ append_secret() {

126 126
mkdir -p /run/openagents
127 127
fetch_secret "$runtime_secret" /run/openagents/runtime.env
128 128
129
if grep -Eq '^(DATABASE_URL|SECRET_KEY_BASE|GITHUB_CLIENT_SECRET|GITHUB_TOKEN_ENCRYPTION_KEY|GITHUB_TOKEN_DECRYPTION_KEYS_JSON|OPENAI_API_KEY|VOICE_RECORDING_ENCRYPTION_KEY|OPENAGENTS_FORGE_OPERATOR_TOKEN|RELEASE_COOKIE)=' /run/openagents/runtime.env; then
129
if grep -Eq '^(DATABASE_URL|SECRET_KEY_BASE|GITHUB_CLIENT_SECRET|GITHUB_TOKEN_ENCRYPTION_KEY|GITHUB_TOKEN_DECRYPTION_KEYS_JSON|OPENAI_API_KEY|VOICE_RECORDING_ENCRYPTION_KEY|CONTENT_ENCRYPTION_KEY|OPENAGENTS_FORGE_OPERATOR_TOKEN|RELEASE_COOKIE)=' /run/openagents/runtime.env; then
130 130
  echo "Fleet configuration must not duplicate named secrets" >&2
131 131
  exit 1
132 132
fi

@@ -138,6 +138,7 @@ append_secret GITHUB_TOKEN_ENCRYPTION_KEY openagents-staging-github-vault-active

138 138
append_secret GITHUB_TOKEN_DECRYPTION_KEYS_JSON openagents-staging-github-vault-previous /run/openagents/runtime.env
139 139
append_secret OPENAI_API_KEY openagents-staging-openai-api-key /run/openagents/runtime.env
140 140
append_secret VOICE_RECORDING_ENCRYPTION_KEY openagents-staging-voice-recording-key /run/openagents/runtime.env
141
append_secret CONTENT_ENCRYPTION_KEY openagents-staging-content-vault-active /run/openagents/runtime.env
141 142
append_secret OPENAGENTS_FORGE_OPERATOR_TOKEN openagents-staging-forge-operator-token /run/openagents/runtime.env
142 143
append_secret RELEASE_COOKIE openagents-staging-release-cookie /run/openagents/runtime.env
143 144
lib/openagents/application.ex modified +9

@@ -5,6 +5,8 @@ defmodule OpenAgents.Application do

5 5
6 6
  use Application
7 7
8
  require Logger
9
8 10
  @impl true
9 11
  def start(_type, _args) do
10 12
    case Application.get_env(:openagents, :runtime_role, :web) do

@@ -29,6 +31,13 @@ defmodule OpenAgents.Application do

29 31
    _program_catalog = OpenAgents.ProgramArtifacts.install!()
30 32
    :ok = OpenAgents.Voice.Config.validate_boot!()
31 33
34
    unless OpenAgents.ContentVault.configured?() do
35
      Logger.warning(
36
        "OpenAgents.ContentVault is not configured: :content_encryption_key is missing or not a 32-byte base64-encoded key. " <>
37
          "Writing sealed transcript, project note, observation, or session summary content will fail."
38
      )
39
    end
40
32 41
    # Build and install the tool catalog; this snapshot is passed to the
33 42
    # embedding warmer and the turn supervisor below.
34 43
    tool_modules =
lib/openagents/content_vault.ex added +150

@@ -0,0 +1,150 @@

1
defmodule OpenAgents.ContentVault do
2
  @moduledoc """
3
  AES-256-GCM sealing for private text that nobody searches.
4
5
  The three vaults that came before this one each seal a credential or a
6
  recording. This one seals *content*: sentences a person wrote or spoke that
7
  rest in PostgreSQL and that no query reads except whole. Issue #193 asked
8
  which columns those are, and
9
  `docs/2026-08-25-encryption-at-rest.md` names every one that is sealed here
10
  and every one that is not, with the query that keeps it plaintext.
11
12
  ## Its own key, and one key for this whole domain
13
14
  `VAULT-001` binds the property that rotating one vault's key never unreads
15
  another vault's records. This vault reads `:content_encryption_key` and
16
  nothing else — no fallback to the GitHub keyring, no bridge to the recording
17
  key. Issue #253 is what that bridge costs.
18
19
  It is one vault over several columns rather than one vault per column, and
20
  that is a decision rather than an omission. The separation that carries
21
  weight is credential from content: a stolen GitHub token key must not open a
22
  conversation, and a conversation key must not open a token. Splitting the
23
  content columns from each other would buy a finer rotation blast radius at
24
  the price of one production secret per column, and every column here is
25
  readable by the same operator through the same application anyway. Rotating
26
  this key strands every column it seals, together, which is the cost recorded
27
  in `VAULT-001`'s rotation posture.
28
29
  ## What the seal is bound to
30
31
  Every seal carries additional authenticated data naming the column it belongs
32
  to and the row's own identity, so ciphertext cannot be lifted from one row
33
  into another, or from one column into another, and still open. `scope/0`
34
  values are the table and column; `binding` is whatever identifies the row
35
  immutably — usually its natural key.
36
37
  ## What it does not do
38
39
  It seals under a key the operator holds, so it defends against a stolen dump,
40
  a stolen backup, and a stolen disk. It does not defend against the operator,
41
  and `EXIT-006` keeps `encrypted_at_rest` from claiming otherwise: the
42
  disclosure stays `false` while any private column rests as plaintext, and
43
  `operator_reads_source` stays `true` regardless.
44
  """
45
46
  @version 1
47
  @nonce_bytes 12
48
  @tag_bytes 16
49
  # Large enough for the longest column this vault seals: a 20,000-character
50
  # project note, whose Ecto bound counts graphemes rather than bytes.
51
  @maximum_content_bytes 131_072
52
  @aad_prefix "openagents.content.v1:"
53
54
  @typedoc "The table and column a seal belongs to."
55
  @type scope :: String.t()
56
57
  @typedoc "The row identity a seal is bound to, in a fixed order."
58
  @type binding :: [String.t() | integer()]
59
60
  @doc "Whether this vault holds its own key."
61
  @spec configured?() :: boolean()
62
  def configured?, do: match?({:ok, _key}, key())
63
64
  @doc """
65
  Seals `content` for one column and one row.
66
67
  `scope` names the column, `binding` names the row. Both travel as
68
  authenticated data rather than as ciphertext, so opening a value under the
69
  wrong column or the wrong row fails instead of succeeding quietly.
70
  """
71
  @spec seal(String.t(), scope(), binding()) :: {:ok, binary()} | {:error, atom()}
72
  def seal(content, scope, binding)
73
      when is_binary(content) and byte_size(content) in 1..@maximum_content_bytes do
74
    with {:ok, aad} <- aad(scope, binding),
75
         {:ok, key} <- key() do
76
      nonce = :crypto.strong_rand_bytes(@nonce_bytes)
77
78
      {ciphertext, tag} =
79
        :crypto.crypto_one_time_aead(:aes_256_gcm, key, nonce, content, aad, true)
80
81
      {:ok, <<@version, nonce::binary, tag::binary, ciphertext::binary>>}
82
    end
83
  end
84
85
  def seal(_content, _scope, _binding), do: {:error, :invalid_content}
86
87
  @doc "Opens a sealed value, refusing one bound to another column or row."
88
  @spec open(binary(), scope(), binding()) :: {:ok, String.t()} | {:error, atom()}
89
  def open(
90
        <<@version, nonce::binary-size(@nonce_bytes), tag::binary-size(@tag_bytes),
91
          ciphertext::binary>>,
92
        scope,
93
        binding
94
      ) do
95
    with {:ok, aad} <- aad(scope, binding),
96
         {:ok, key} <- key() do
97
      case :crypto.crypto_one_time_aead(:aes_256_gcm, key, nonce, ciphertext, aad, tag, false) do
98
        content when is_binary(content) -> {:ok, content}
99
        :error -> {:error, :content_unsealable}
100
      end
101
    end
102
  end
103
104
  def open(_sealed, _scope, _binding), do: {:error, :content_unsealable}
105
106
  @doc """
107
  The plaintext, or `nil` when there is no seal to open.
108
109
  Readers render a missing sentence rather than crashing a page: a rotated key
110
  strands prior content, which `VAULT-001` records as this vault's rotation
111
  cost, and a crash there would take a whole timeline or project down with it.
112
  """
113
  @spec text(binary() | nil, scope(), binding()) :: String.t() | nil
114
  def text(nil, _scope, _binding), do: nil
115
116
  def text(sealed, scope, binding) when is_binary(sealed) do
117
    case open(sealed, scope, binding) do
118
      {:ok, content} -> content
119
      {:error, _reason} -> nil
120
    end
121
  end
122
123
  @doc "The largest value this vault seals."
124
  @spec maximum_content_bytes() :: pos_integer()
125
  def maximum_content_bytes, do: @maximum_content_bytes
126
127
  @doc "How many bytes sealing adds to a value, for a column's own bound."
128
  @spec overhead_bytes() :: pos_integer()
129
  def overhead_bytes, do: 1 + @nonce_bytes + @tag_bytes
130
131
  defp aad(scope, binding) when is_binary(scope) and is_list(binding) do
132
    if binding == [] or Enum.any?(binding, &is_nil/1) do
133
      {:error, :invalid_content_binding}
134
    else
135
      {:ok, @aad_prefix <> scope <> ":" <> Enum.map_join(binding, ":", &to_string/1)}
136
    end
137
  end
138
139
  defp aad(_scope, _binding), do: {:error, :invalid_content_binding}
140
141
  defp key do
142
    with encoded when is_binary(encoded) <-
143
           Application.get_env(:openagents, :content_encryption_key),
144
         {:ok, key} when byte_size(key) == 32 <- Base.decode64(encoded) do
145
      {:ok, key}
146
    else
147
      _missing -> {:error, :content_vault_not_configured}
148
    end
149
  end
150
end
lib/openagents/forge/at_rest.ex modified +82 -8

@@ -21,7 +21,10 @@ defmodule OpenAgents.Forge.AtRest do

21 21
22 22
  `plaintext_private_columns/0` names columns that hold private, user-authored
23 23
  content and rest as plaintext. Its entries are proven plaintext by reading
24
  the raw column back through SQL, so the list cannot claim a gap that closed.
24
  the raw column back through SQL, so the list cannot claim a gap that closed,
25
  and each carries the query that keeps it readable — a column is left in
26
  plaintext because something reads it in a way a seal would end, never because
27
  nobody got to it.
25 28
26 29
  `encrypted_at_rest?/0` is the boolean `EXIT-006` publishes, and it is now
27 30
  derived: the private store is encrypted at rest exactly when no private

@@ -58,8 +61,13 @@ defmodule OpenAgents.Forge.AtRest do

58 61
          holds: String.t()
59 62
        }
60 63
61
  @typedoc "One private column that rests as plaintext."
62
  @type plaintext :: %{table: String.t(), column: String.t(), holds: String.t()}
64
  @typedoc "One private column that rests as plaintext, and why it still does."
65
  @type plaintext :: %{
66
          table: String.t(),
67
          column: String.t(),
68
          holds: String.t(),
69
          reason: String.t()
70
        }
63 71
64 72
  # The regular expression the proof hands to `information_schema` to build the
65 73
  # population it checks this module against. It lives here so the module and

@@ -84,6 +92,30 @@ defmodule OpenAgents.Forge.AtRest do

84 92
      column: "data",
85 93
      vault: OpenAgents.Voice.RecordingVault,
86 94
      holds: "one slice of uploaded call audio"
95
    },
96
    %{
97
      table: "voice_transcript_items",
98
      column: "content_ciphertext",
99
      vault: OpenAgents.ContentVault,
100
      holds: "the voice conversation record VOICE-012 calls authority"
101
    },
102
    %{
103
      table: "voice_sessions",
104
      column: "compaction_summary_ciphertext",
105
      vault: OpenAgents.ContentVault,
106
      holds: "one in-call compaction summary"
107
    },
108
    %{
109
      table: "preference_observations",
110
      column: "summary_ciphertext",
111
      vault: OpenAgents.ContentVault,
112
      holds: "private evidence proposing a behavior preference"
113
    },
114
    %{
115
      table: "project_notes",
116
      column: "body_ciphertext",
117
      vault: OpenAgents.ContentVault,
118
      holds: "project discussion and activity"
87 119
    }
88 120
  ]
89 121

@@ -92,14 +124,50 @@ defmodule OpenAgents.Forge.AtRest do

92 124
  # `false`, which is where it already is; a column named here that turns out
93 125
  # to be sealed turns the proof red. Both failures understate the store.
94 126
  @plaintext_private [
95
    %{table: "messages", column: "content", holds: "conversation messages"},
96 127
    %{
97
      table: "voice_transcript_items",
128
      table: "messages",
98 129
      column: "content",
99
      holds: "the voice conversation record VOICE-012 calls authority"
130
      holds: "conversation messages",
131
      reason:
132
        "a generated `search_vector` is computed from this column inside PostgreSQL " <>
133
          "and indexed with GIN; sealing it ends lexical recall over your own history"
134
    },
135
    %{
136
      table: "issues",
137
      column: "body",
138
      holds: "issue bodies",
139
      reason:
140
        "`OpenAgents.Issues.search/2` and `OpenAgents.Issues.TaskReferences` match it " <>
141
          "with `ILIKE`; sealing it ends issue search and cross-references"
100 142
    },
101
    %{table: "issues", column: "body", holds: "issue bodies"},
102
    %{table: "comments", column: "body", holds: "issue and pull request comments"}
143
    %{
144
      table: "comments",
145
      column: "body",
146
      holds: "issue and pull request comments",
147
      reason: "`OpenAgents.Issues.TaskReferences` matches it with `ILIKE`"
148
    },
149
    %{
150
      table: "forum_posts",
151
      column: "body_text",
152
      holds: "forum replies",
153
      reason: "`OpenAgents.Forum.search/2` matches it with `ILIKE`"
154
    },
155
    %{
156
      table: "account_chat_runs",
157
      column: "user_content",
158
      holds: "what an account typed into the chat console",
159
      reason:
160
        "the same words rest verbatim in `account_chat_events.payload`, which the " <>
161
          "replay path reads structurally; sealing one and not the other moves nothing"
162
    },
163
    %{
164
      table: "account_chat_runs",
165
      column: "assistant_content",
166
      holds: "what the model replied in the chat console",
167
      reason:
168
        "the same words rest in the `text_delta` events and in the `completion` map " <>
169
          "beside it, for the same reason"
170
    }
103 171
  ]
104 172
105 173
  # Every column the catalog reports under `secret_shaped_pattern/0`. The proof

@@ -110,6 +178,12 @@ defmodule OpenAgents.Forge.AtRest do

110 178
    {"users", "github_token_ciphertext"} => :sealed,
111 179
    {"machine_pairings", "token_ciphertext"} => :sealed,
112 180
181
    # Private content, sealed under the content vault's own key (issue #193).
182
    {"preference_observations", "summary_ciphertext"} => :sealed,
183
    {"project_notes", "body_ciphertext"} => :sealed,
184
    {"voice_sessions", "compaction_summary_ciphertext"} => :sealed,
185
    {"voice_transcript_items", "content_ciphertext"} => :sealed,
186
113 187
    # One-way SHA-256 of a bearer credential, unique-indexed because it is the
114 188
    # lookup key. Nothing reverses these, so nothing seals them.
115 189
    {"agent_tokens", "token_digest"} => :digest,
lib/openagents/preferences/observation.ex modified +50 -1

@@ -3,6 +3,10 @@ defmodule OpenAgents.Preferences.Observation do

3 3
  use Ecto.Schema
4 4
  import Ecto.Changeset
5 5
6
  alias OpenAgents.ContentVault
7
8
  @scope "preference_observations.summary"
9
6 10
  @primary_key {:id, :binary_id, autogenerate: true}
7 11
  @foreign_key_type :binary_id
8 12
  @timestamps_opts [type: :utc_datetime_usec, updated_at: false]

@@ -11,7 +15,8 @@ defmodule OpenAgents.Preferences.Observation do

11 15
    belongs_to :owner, OpenAgents.Conversations.Visitor, foreign_key: :owner_visitor_id
12 16
    belongs_to :source_message, OpenAgents.Conversations.Message
13 17
    field :source_kind, :string
14
    field :summary, :string
18
    field :summary, :string, redact: true
19
    field :summary_ciphertext, :binary, redact: true
15 20
    field :evidence_digest, :string
16 21
    field :confidence_millis, :integer
17 22
    field :observed_at, :utc_datetime_usec

@@ -60,7 +65,51 @@ defmodule OpenAgents.Preferences.Observation do

60 65
    |> validate_number(:policy_version, greater_than: 0)
61 66
    |> validate_format(:evidence_digest, ~r/\A[0-9a-f]{64}\z/)
62 67
    |> validate_format(:proposer_digest, ~r/\A[0-9a-f]{64}\z/)
68
    |> seal_summary()
63 69
    |> foreign_key_constraint(:owner_visitor_id)
64 70
    |> foreign_key_constraint(:source_message_id)
65 71
  end
72
73
  @doc "The column this schema's sealed summary belongs to."
74
  @spec scope() :: String.t()
75
  def scope, do: @scope
76
77
  @doc """
78
  The observation summary, opened from the seal.
79
80
  Falls back to the plaintext column for a row an un-replaced node wrote during
81
  a rolling replacement.
82
  """
83
  @spec summary(%__MODULE__{}) :: String.t() | nil
84
  def summary(%__MODULE__{summary_ciphertext: sealed} = observation) when is_binary(sealed),
85
    do: ContentVault.text(sealed, @scope, seal_binding(observation))
86
87
  def summary(%__MODULE__{summary: summary}), do: summary
88
89
  @doc "The row identity a sealed summary is bound to."
90
  @spec seal_binding(%__MODULE__{}) :: ContentVault.binding()
91
  def seal_binding(%__MODULE__{} = observation),
92
    do: [observation.owner_visitor_id, observation.evidence_digest]
93
94
  # The summary is hashed into `evidence_digest` before this runs, so sealing
95
  # it changes nothing about what the digest commits to -- only about what
96
  # rests readable beside it.
97
  defp seal_summary(%Ecto.Changeset{valid?: true} = changeset) do
98
    binding = [
99
      get_field(changeset, :owner_visitor_id),
100
      get_field(changeset, :evidence_digest)
101
    ]
102
103
    case ContentVault.seal(get_change(changeset, :summary), @scope, binding) do
104
      {:ok, sealed} ->
105
        changeset
106
        |> put_change(:summary_ciphertext, sealed)
107
        |> force_change(:summary, nil)
108
109
      {:error, reason} ->
110
        add_error(changeset, :summary, "cannot be sealed", reason: reason)
111
    end
112
  end
113
114
  defp seal_summary(changeset), do: changeset
66 115
end
lib/openagents/projects/project_note.ex modified +72 -2

@@ -21,16 +21,19 @@ defmodule OpenAgents.Projects.ProjectNote do

21 21
  use Ecto.Schema
22 22
  import Ecto.Changeset
23 23
24
  alias OpenAgents.ContentVault
24 25
  alias OpenAgents.Projects.Project
25 26
  alias OpenAgents.Repositories.Repository
26 27
27 28
  @kinds ["note", "activity"]
29
  @scope "project_notes.body"
28 30
29 31
  @doc "The note kinds, discussion first."
30 32
  def kinds, do: @kinds
31 33
32 34
  schema "project_notes" do
33
    field :body, :string
35
    field :body, :string, redact: true
36
    field :body_ciphertext, :binary, redact: true
34 37
    field :kind, :string, default: "note"
35 38
    field :author, :map
36 39

@@ -46,9 +49,11 @@ defmodule OpenAgents.Projects.ProjectNote do

46 49
    note
47 50
    |> cast(attrs, [:body, :kind, :author, :project_id, :repository_id, :author_user_id])
48 51
    |> update_change(:body, &trim/1)
49
    |> validate_required([:body, :kind, :project_id, :repository_id])
52
    |> validate_required([:kind, :project_id, :repository_id])
53
    |> validate_body_present()
50 54
    |> validate_inclusion(:kind, @kinds)
51 55
    |> validate_length(:body, max: 20_000)
56
    |> seal_body()
52 57
    |> foreign_key_constraint(:project_id)
53 58
    |> foreign_key_constraint(:repository_id)
54 59
    |> foreign_key_constraint(:author_user_id)

@@ -56,6 +61,71 @@ defmodule OpenAgents.Projects.ProjectNote do

56 61
    |> check_constraint(:kind, name: :project_notes_kind_check)
57 62
  end
58 63
64
  @doc "The column this schema's sealed body belongs to."
65
  @spec scope() :: String.t()
66
  def scope, do: @scope
67
68
  @doc """
69
  The note body, opened from the seal.
70
71
  Falls back to the plaintext column for a row an un-replaced node wrote during
72
  a rolling replacement, and returns `nil` when neither can be read so a
73
  project page renders one missing note rather than failing whole.
74
  """
75
  @spec text(%__MODULE__{}) :: String.t() | nil
76
  def text(%__MODULE__{body_ciphertext: sealed} = note) when is_binary(sealed),
77
    do: ContentVault.text(sealed, @scope, seal_binding(note))
78
79
  def text(%__MODULE__{body: body}), do: body
80
81
  @doc """
82
  The row identity a sealed body is bound to.
83
84
  The primary key is a sequence, so it does not exist while the insert is
85
  being built. What does exist, and never changes afterwards, is the note's
86
  place: an edit rewrites the body, never the project, the repository, or the
87
  kind. Binding to those is what stops a sealed note from being lifted into
88
  another project and still opening.
89
  """
90
  @spec seal_binding(%__MODULE__{}) :: ContentVault.binding()
91
  def seal_binding(%__MODULE__{} = note), do: [note.project_id, note.repository_id, note.kind]
92
93
  # An edit that changes only the kind or the author leaves `body` unchanged,
94
  # and the stored plaintext is now `nil` because the body rests sealed. So
95
  # "a note has a body" is asked of both halves rather than of the column the
96
  # contract migration is going to drop.
97
  defp validate_body_present(changeset) do
98
    if get_field(changeset, :body) || get_field(changeset, :body_ciphertext),
99
      do: changeset,
100
      else: add_error(changeset, :body, "can't be blank", validation: :required)
101
  end
102
103
  defp seal_body(%Ecto.Changeset{valid?: true} = changeset) do
104
    case get_change(changeset, :body) do
105
      nil ->
106
        changeset
107
108
      body ->
109
        binding = [
110
          get_field(changeset, :project_id),
111
          get_field(changeset, :repository_id),
112
          get_field(changeset, :kind)
113
        ]
114
115
        case ContentVault.seal(body, @scope, binding) do
116
          {:ok, sealed} ->
117
            changeset
118
            |> put_change(:body_ciphertext, sealed)
119
            |> force_change(:body, nil)
120
121
          {:error, reason} ->
122
            add_error(changeset, :body, "cannot be sealed", reason: reason)
123
        end
124
    end
125
  end
126
127
  defp seal_body(changeset), do: changeset
128
59 129
  defp trim(body) when is_binary(body), do: String.trim(body)
60 130
  defp trim(body), do: body
61 131
end
lib/openagents/runtime_config.ex modified +9

@@ -446,6 +446,15 @@ defmodule OpenAgents.RuntimeConfig do

446 446
          not encryption_key?(Map.get(settings, :voice_recording_encryption_key)) ->
447 447
        error(:voice_recording_encryption_key, "is required when recording is enabled")
448 448
449
      # The content vault's own key (VAULT-001, issue #193). Without it a voice
450
      # transcript, a compaction summary, a preference observation, and a
451
      # project note all refuse to be written rather than being written
452
      # readable, so a node missing this key is a node that cannot serve —
453
      # which is worth finding at boot rather than at the first write.
454
      environment in [:staging, :production] and
455
          not encryption_key?(Map.get(settings, :content_encryption_key)) ->
456
        error(:content_encryption_key, "must be a base64-encoded 32-byte key")
457
449 458
      features.voice_retention and not features.voice_recording ->
450 459
        error(:voice_retention_enabled, "requires voice recording")
451 460
lib/openagents/timeline.ex modified +1 -1

@@ -153,7 +153,7 @@ defmodule OpenAgents.Timeline do

153 153
      timestamp: item.observed_at,
154 154
      kind: :turn,
155 155
      record_id: item.id,
156
      summary: "Voice #{item.role}: #{shorten(item.content)}",
156
      summary: "Voice #{item.role}: #{shorten(TranscriptItem.text(item))}",
157 157
      metadata: %{
158 158
        role: item.role,
159 159
        status: item.status,
lib/openagents/voice/retention.ex modified +1

@@ -101,6 +101,7 @@ defmodule OpenAgents.Voice.Retention do

101 101
          # with the operational window; compaction_count stays as an
102 102
          # aggregate lifecycle fact.
103 103
          compaction_summary: nil,
104
          compaction_summary_ciphertext: nil,
104 105
          operational_purged_at: now
105 106
        })
106 107
        |> Repo.update!()
lib/openagents/voice/session.ex modified +53 -1

@@ -4,8 +4,11 @@ defmodule OpenAgents.Voice.Session do

4 4
  use Ecto.Schema
5 5
  import Ecto.Changeset
6 6
7
  alias OpenAgents.ContentVault
8
7 9
  @statuses ~w(connecting listening responding interrupted reconnecting ended failed)
8 10
  @digest_regex ~r/\A[0-9a-f]{64}\z/
11
  @compaction_summary_scope "voice_sessions.compaction_summary"
9 12
10 13
  @primary_key {:id, :binary_id, autogenerate: true}
11 14
  @foreign_key_type :binary_id

@@ -37,7 +40,8 @@ defmodule OpenAgents.Voice.Session do

37 40
    field :program_artifact_receipt, :map
38 41
    field :event_sequence, :integer, default: 0
39 42
    field :usage, :map, default: %{}
40
    field :compaction_summary, :string
43
    field :compaction_summary, :string, redact: true
44
    field :compaction_summary_ciphertext, :binary, redact: true
41 45
    field :compaction_count, :integer, default: 0
42 46
    field :started_at, :utc_datetime_usec
43 47
    field :connected_at, :utc_datetime_usec

@@ -150,8 +154,56 @@ defmodule OpenAgents.Voice.Session do

150 154
    |> validate_required([:compaction_summary, :compaction_count])
151 155
    |> validate_number(:compaction_count, greater_than_or_equal_to: 0)
152 156
    |> validate_length(:compaction_summary, min: 1, max: 8_192, count: :bytes)
157
    |> seal_compaction_summary()
153 158
  end
154 159
160
  @doc "The column this schema's sealed summary belongs to."
161
  @spec compaction_summary_scope() :: String.t()
162
  def compaction_summary_scope, do: @compaction_summary_scope
163
164
  @doc """
165
  The in-call compaction summary, opened from the seal.
166
167
  Falls back to the plaintext column for a row an un-replaced node wrote during
168
  a rolling replacement, and is `nil` once retention has purged it.
169
  """
170
  @spec compaction_summary(%__MODULE__{}) :: String.t() | nil
171
  def compaction_summary(%__MODULE__{compaction_summary_ciphertext: sealed} = session)
172
      when is_binary(sealed),
173
      do:
174
        ContentVault.text(sealed, @compaction_summary_scope, compaction_summary_binding(session))
175
176
  def compaction_summary(%__MODULE__{compaction_summary: summary}), do: summary
177
178
  @doc "The row identity a sealed compaction summary is bound to."
179
  @spec compaction_summary_binding(%__MODULE__{}) :: ContentVault.binding()
180
  def compaction_summary_binding(%__MODULE__{} = session), do: [session.id, session.generation]
181
182
  # Sealed in the changeset so no update path reaches this column with the
183
  # words still readable. Retention nulls both halves together, which is why
184
  # this only ever runs on a change.
185
  defp seal_compaction_summary(%Ecto.Changeset{valid?: true} = changeset) do
186
    case get_change(changeset, :compaction_summary) do
187
      nil ->
188
        changeset
189
190
      summary ->
191
        binding = [get_field(changeset, :id), get_field(changeset, :generation)]
192
193
        case ContentVault.seal(summary, @compaction_summary_scope, binding) do
194
          {:ok, sealed} ->
195
            changeset
196
            |> put_change(:compaction_summary_ciphertext, sealed)
197
            |> force_change(:compaction_summary, nil)
198
199
          {:error, reason} ->
200
            add_error(changeset, :compaction_summary, "cannot be sealed", reason: reason)
201
        end
202
    end
203
  end
204
205
  defp seal_compaction_summary(changeset), do: changeset
206
155 207
  defp validate_map(changeset, field, maximum_bytes) do
156 208
    validate_change(changeset, field, fn ^field, value ->
157 209
      case Jason.encode(value) do
lib/openagents/voice/transcript_item.ex modified +74 -3

@@ -1,13 +1,36 @@

1 1
defmodule OpenAgents.Voice.TranscriptItem do
2
  @moduledoc "Final or explicitly interrupted transcript evidence from one voice generation."
2
  @moduledoc """
3
  Final or explicitly interrupted transcript evidence from one voice generation.
4
5
  The words rest sealed. `content_ciphertext` holds them under
6
  `OpenAgents.ContentVault`, and `text/1` is the only way back to them, so
7
  nothing reads the sentence by touching a field.
8
9
  `VOICE-012` seals call audio and calls this table the conversation record.
10
  Until issue #193 the record rested in plaintext beside the sealed audio, so a
11
  stolen dump got the words either way. That asymmetry is what this closes.
12
13
  `content` is the plaintext column this table used to keep. It is still
14
  declared, still loaded, and still read as a fallback, because a rolling
15
  replacement leaves nodes on the previous release writing into it for as long
16
  as the roll takes. Nothing writes it any more: the changeset nulls it in the
17
  same change that seals the words. The contract migration drops it once that
18
  release is off every node, the way `machine_pairings.user_id` was dropped a
19
  release after its last reader.
20
  """
3 21
4 22
  use Ecto.Schema
5 23
  import Ecto.Changeset
6 24
25
  alias OpenAgents.ContentVault
26
7 27
  @primary_key {:id, :binary_id, autogenerate: true}
8 28
  @foreign_key_type :binary_id
9 29
  @timestamps_opts [type: :utc_datetime_usec]
10 30
31
  @scope "voice_transcript_items.content"
32
  @maximum_content_bytes 16_000
33
11 34
  schema "voice_transcript_items" do
12 35
    belongs_to :voice_session, OpenAgents.Voice.Session
13 36
    belongs_to :message, OpenAgents.Conversations.Message

@@ -15,12 +38,34 @@ defmodule OpenAgents.Voice.TranscriptItem do

15 38
    field :provider_item_id, :string
16 39
    field :provider_response_id, :string
17 40
    field :role, :string
18
    field :content, :string
41
    field :content, :string, redact: true
42
    field :content_ciphertext, :binary, redact: true
19 43
    field :status, :string
20 44
    field :observed_at, :utc_datetime_usec
21 45
    timestamps()
22 46
  end
23 47
48
  @doc "The column this schema's sealed text belongs to."
49
  @spec scope() :: String.t()
50
  def scope, do: @scope
51
52
  @doc """
53
  The transcript text, opened from the seal.
54
55
  Falls back to the plaintext column for a row an un-replaced node wrote during
56
  a rolling replacement, and returns `nil` when neither can be read.
57
  """
58
  @spec text(%__MODULE__{}) :: String.t() | nil
59
  def text(%__MODULE__{content_ciphertext: sealed} = item) when is_binary(sealed),
60
    do: ContentVault.text(sealed, @scope, seal_binding(item))
61
62
  def text(%__MODULE__{content: content}), do: content
63
64
  @doc "The row identity a sealed transcript is bound to: its natural key."
65
  @spec seal_binding(%__MODULE__{}) :: ContentVault.binding()
66
  def seal_binding(%__MODULE__{} = item),
67
    do: [item.voice_session_id, item.generation, item.provider_item_id, item.role]
68
24 69
  def create_changeset(item, attributes) do
25 70
    item
26 71
    |> cast(attributes, [

@@ -47,7 +92,8 @@ defmodule OpenAgents.Voice.TranscriptItem do

47 92
    |> validate_inclusion(:status, ~w(final interrupted))
48 93
    |> validate_length(:provider_item_id, max: 512)
49 94
    |> validate_length(:provider_response_id, max: 512)
50
    |> validate_length(:content, min: 1, max: 16_000)
95
    |> validate_length(:content, min: 1, max: @maximum_content_bytes)
96
    |> seal_content()
51 97
    |> foreign_key_constraint(:voice_session_id)
52 98
    |> foreign_key_constraint(:message_id)
53 99
    |> unique_constraint(:message_id)

@@ -55,4 +101,29 @@ defmodule OpenAgents.Voice.TranscriptItem do

55 101
      name: :voice_transcript_provider_item_role_index
56 102
    )
57 103
  end
104
105
  # The seal happens here rather than in the context, so no insert path reaches
106
  # this table with the words still readable. A vault that cannot seal fails
107
  # the changeset: writing the plaintext instead would make the column
108
  # `OpenAgents.Forge.AtRest` publishes as sealed a claim rather than a fact.
109
  defp seal_content(%Ecto.Changeset{valid?: true} = changeset) do
110
    binding = [
111
      get_field(changeset, :voice_session_id),
112
      get_field(changeset, :generation),
113
      get_field(changeset, :provider_item_id),
114
      get_field(changeset, :role)
115
    ]
116
117
    case ContentVault.seal(get_change(changeset, :content), @scope, binding) do
118
      {:ok, sealed} ->
119
        changeset
120
        |> put_change(:content_ciphertext, sealed)
121
        |> force_change(:content, nil)
122
123
      {:error, reason} ->
124
        add_error(changeset, :content, "cannot be sealed", reason: reason)
125
    end
126
  end
127
128
  defp seal_content(changeset), do: changeset
58 129
end
lib/openagents_web/controllers/project_json.ex modified +1 -1

@@ -74,7 +74,7 @@ defmodule OpenAgentsWeb.ProjectJSON do

74 74
    %{
75 75
      id: note.id,
76 76
      kind: note.kind,
77
      body: note.body,
77
      body: OpenAgents.Projects.ProjectNote.text(note) || "",
78 78
      author: note.author,
79 79
      created_at: note.inserted_at,
80 80
      updated_at: note.updated_at
lib/openagents_web/live/project_show_live.ex modified +3 -2

@@ -28,6 +28,7 @@ defmodule OpenAgentsWeb.ProjectShowLive do

28 28
  alias OpenAgents.Issues
29 29
  alias OpenAgents.Markdown
30 30
  alias OpenAgents.Projects
31
  alias OpenAgents.Projects.ProjectNote
31 32
  alias OpenAgents.Projects.PromiseRegistry
32 33
  alias OpenAgents.ProjectItems.ProjectItem
33 34
  alias OpenAgents.Repositories

@@ -742,7 +743,7 @@ defmodule OpenAgentsWeb.ProjectShowLive do

742 743
              <Circle.timeline_event
743 744
                :if={note.kind == "activity"}
744 745
                actor={author(note)}
745
                text={note.body}
746
                text={ProjectNote.text(note) || ""}
746 747
                icon="history"
747 748
                tone={:neutral}
748 749
                at={stamp(note.inserted_at)}

@@ -753,7 +754,7 @@ defmodule OpenAgentsWeb.ProjectShowLive do

753 754
                author={author(note)}
754 755
                at={stamp(note.inserted_at)}
755 756
              >
756
                {Markdown.to_html(note.body)}
757
                {Markdown.to_html(ProjectNote.text(note) || "")}
757 758
                <:actions>
758 759
                  <.button
759 760
                    :if={Projects.authored_by?(note, @current_user)}
ops/ci/release-smoke.sh modified +6

@@ -60,6 +60,10 @@ MIX_ENV=prod mix release --overwrite

60 60
61 61
secret_key_base=$(openssl rand -base64 64 | tr -d '\n')
62 62
github_token_key=$(openssl rand -base64 32 | tr -d '\n')
63
# The content vault seals private content under its own key (VAULT-001,
64
# issue #193). There is no bridge to the GitHub key, so the smoke provides
65
# its own or the release refuses to boot.
66
content_key=$(openssl rand -base64 32 | tr -d '\n')
63 67
64 68
echo "Checking the release configuration profile"
65 69
readiness_report=$(env \

@@ -68,6 +72,7 @@ readiness_report=$(env \

68 72
  GITHUB_CLIENT_SECRET="release-smoke-secret" \
69 73
  GITHUB_TOKEN_ENCRYPTION_KEY="$github_token_key" \
70 74
  GITHUB_TOKEN_ENCRYPTION_KEY_ID="staging-release-smoke-2026-08" \
75
  CONTENT_ENCRYPTION_KEY="$content_key" \
71 76
  OPENAI_API_KEY="release-smoke-openai-key" \
72 77
  POOL_SIZE="2" \
73 78
  PORT="$port" \

@@ -88,6 +93,7 @@ env \

88 93
  GITHUB_CLIENT_SECRET="release-smoke-secret" \
89 94
  GITHUB_TOKEN_ENCRYPTION_KEY="$github_token_key" \
90 95
  GITHUB_TOKEN_ENCRYPTION_KEY_ID="staging-release-smoke-2026-08" \
96
  CONTENT_ENCRYPTION_KEY="$content_key" \
91 97
  OPENAI_API_KEY="release-smoke-openai-key" \
92 98
  PHX_SERVER="true" \
93 99
  POOL_SIZE="2" \
ops/deploy/fleet-startup.template.sh modified +6 -1

@@ -44,6 +44,11 @@ export VOICE_RECORDING_ENCRYPTION_KEY="$(secret sarah-production-voice-recording

44 44
# so two vaults sealed under one key and a GitHub rotation silently moved
45 45
# the pairing vault too. The bridge was meant to last one deploy.
46 46
export MACHINE_TOKEN_ENCRYPTION_KEY="$(secret openagents-machine-token-encryption-key)"
47
# The content vault's own key (VAULT-001, issue #193). It seals voice
48
# transcripts, in-call compaction summaries, preference observations, and
49
# project notes. Nothing bridges to it and nothing bridges from it: an
50
# unset value refuses the boot rather than borrowing another vault's key.
51
export CONTENT_ENCRYPTION_KEY="$(secret openagents-content-encryption-key)"
47 52
export RELEASE_COOKIE="$(secret sarah-release-cookie)"
48 53
export OPENAGENTS_FORGE_OPERATOR_TOKEN="$(secret sarah-forge-operator-token)"
49 54
export OPENAGENTS_POSTHOG_PROJECT_TOKEN="$(secret openagents-posthog-project-token)"

@@ -135,7 +140,7 @@ ENV_NAMES=(

135 140
  AI_GATEWAY_API_KEY
136 141
  GITHUB_CLIENT_ID GITHUB_CLIENT_SECRET
137 142
  GITHUB_TOKEN_ENCRYPTION_KEY VOICE_RECORDING_ENCRYPTION_KEY RELEASE_COOKIE
138
  MACHINE_TOKEN_ENCRYPTION_KEY
143
  MACHINE_TOKEN_ENCRYPTION_KEY CONTENT_ENCRYPTION_KEY
139 144
  OPENAGENTS_FORGE_OPERATOR_TOKEN OPENAGENTS_POSTHOG_PROJECT_TOKEN
140 145
  DATABASE_URL DNS_CLUSTER_QUERY
141 146
  GITHUB_OAUTH_SCOPES GITHUB_REDIRECT_URI GITHUB_TOKEN_DECRYPTION_KEYS_JSON
ops/relup-proof/common.sh modified +4

@@ -74,6 +74,9 @@ prepare_runtime() {

74 74
  proof_port=$((42000 + ($$ % 10000)))
75 75
  proof_secret=$(openssl rand -base64 64 | tr -d '\n')
76 76
  proof_token_key=$(openssl rand -base64 32 | tr -d '\n')
77
  # The content vault's own key (VAULT-001, issue #193). Nothing bridges
78
  # to it, so the proof supplies one or the release refuses to boot.
79
  proof_content_key=$(openssl rand -base64 32 | tr -d '\n')
77 80
}
78 81
79 82
profile() {

@@ -83,6 +86,7 @@ profile() {

83 86
    GITHUB_CLIENT_SECRET="relup-proof-secret" \
84 87
    GITHUB_TOKEN_ENCRYPTION_KEY="$proof_token_key" \
85 88
    GITHUB_TOKEN_ENCRYPTION_KEY_ID="staging-relup-proof-2026-08" \
89
    CONTENT_ENCRYPTION_KEY="$proof_content_key" \
86 90
    OPENAI_API_KEY="relup-proof-openai-key" \
87 91
    OPENAGENTS_RELUP_INSTALL_BARRIER_MS="${OPENAGENTS_RELUP_INSTALL_BARRIER_MS:-0}" \
88 92
    OPENAGENTS_RELUP_INSTALL_BARRIER_PATH="${OPENAGENTS_RELUP_INSTALL_BARRIER_PATH:-}" \
ops/staging/gate-5-profile.sh modified +4

@@ -8,6 +8,10 @@ set -eu

8 8
: "${GITHUB_CLIENT_SECRET:?GITHUB_CLIENT_SECRET is required}"
9 9
: "${GITHUB_TOKEN_ENCRYPTION_KEY:?GITHUB_TOKEN_ENCRYPTION_KEY is required}"
10 10
: "${GITHUB_TOKEN_ENCRYPTION_KEY_ID:?GITHUB_TOKEN_ENCRYPTION_KEY_ID is required}"
11
# The content vault's own key (VAULT-001, issue #193). No bridge exists to
12
# another vault's key, so an unset value is a refused boot rather than two
13
# vaults sealing under one key.
14
: "${CONTENT_ENCRYPTION_KEY:?CONTENT_ENCRYPTION_KEY is required}"
11 15
: "${OPENAI_API_KEY:?OPENAI_API_KEY is required}"
12 16
: "${SECRET_KEY_BASE:?SECRET_KEY_BASE is required}"
13 17
ops/staging/validate-isolation.sh modified +1

@@ -115,6 +115,7 @@ jq -e '

115 115
  all([
116 116
    "openagents-staging-builder-config",
117 117
    "openagents-staging-database-url",
118
    "openagents-staging-content-vault-active",
118 119
    "openagents-staging-fleet-database-url",
119 120
    "openagents-staging-fleet-config",
120 121
    "openagents-staging-forge-operator-token",
priv/migration_lineages/prior-2026-08-19.json modified +3 -1

@@ -306,7 +306,9 @@

306 306
    20260825120000,
307 307
    20260825140544,
308 308
    20260825140545,
309
    20260825160000
309
    20260825160000,
310
    20260825170000,
311
    20260825170100
310 312
  ],
311 313
  "required_tables": [
312 314
    "users",
priv/repo/migrations/20260825170000_add_private_content_ciphertext_columns.exs added +73

@@ -0,0 +1,73 @@

1
defmodule OpenAgents.Repo.Migrations.AddPrivateContentCiphertextColumns do
2
  @moduledoc """
3
  The expand half of sealing the private content columns nobody searches
4
  (issue #193).
5
6
  It only adds. Every plaintext column keeps its name, its type, and its rows;
7
  what it loses is `NOT NULL`, because the release that follows writes the
8
  ciphertext and leaves the plaintext empty. Retyping a live column here would
9
  break every node still running the previous release for as long as a rolling
10
  replacement takes, which is the failure `RELEASE-006` exists around, so the
11
  plaintext columns are dropped by a contract migration a release later rather
12
  than by this one.
13
14
  The `_present` constraints are what keep "this row has text" true across the
15
  transition: an un-replaced node writes the plaintext column, a replaced one
16
  writes the ciphertext column, and neither can write a row with neither.
17
  `voice_sessions.compaction_summary` gets no such constraint because it is
18
  optional by design — `OpenAgents.Voice.Retention` nulls it on purge.
19
20
  `preference_observation_shape` is left alone deliberately. It bounds
21
  `octet_length(summary)`, and a `CHECK` is satisfied unless it evaluates to
22
  false, so a null summary leaves that conjunct null and the constraint
23
  passes — the same reason `voice_transcript_items_content_bounded` needs no
24
  edit either.
25
  """
26
27
  use Ecto.Migration
28
29
  # {table, plaintext column, ciphertext column, ciphertext upper bound, present check?}
30
  @columns [
31
    {:voice_transcript_items, :content, :content_ciphertext, 16_064, true},
32
    {:voice_sessions, :compaction_summary, :compaction_summary_ciphertext, 8_256, false},
33
    {:preference_observations, :summary, :summary_ciphertext, 576, true},
34
    {:project_notes, :body, :body_ciphertext, 131_136, true}
35
  ]
36
37
  def up do
38
    for {table, plaintext, ciphertext, bound, present?} <- @columns do
39
      alter table(table) do
40
        add ciphertext, :binary
41
      end
42
43
      # `compaction_summary` is already nullable, and only the columns that
44
      # carry a present-check ever had `NOT NULL` to drop.
45
      if present?, do: execute("ALTER TABLE #{table} ALTER COLUMN #{plaintext} DROP NOT NULL")
46
47
      create constraint(table, :"#{table}_#{ciphertext}_bounded",
48
               check:
49
                 "#{ciphertext} IS NULL OR octet_length(#{ciphertext}) BETWEEN 30 AND #{bound}"
50
             )
51
52
      if present? do
53
        create constraint(table, :"#{table}_#{plaintext}_present",
54
                 check: "#{plaintext} IS NOT NULL OR #{ciphertext} IS NOT NULL"
55
               )
56
      end
57
    end
58
  end
59
60
  def down do
61
    for {table, plaintext, ciphertext, _bound, present?} <- @columns do
62
      if present?, do: drop(constraint(table, :"#{table}_#{plaintext}_present"))
63
64
      drop constraint(table, :"#{table}_#{ciphertext}_bounded")
65
66
      if present?, do: execute("ALTER TABLE #{table} ALTER COLUMN #{plaintext} SET NOT NULL")
67
68
      alter table(table) do
69
        remove ciphertext
70
      end
71
    end
72
  end
73
end
priv/repo/migrations/20260825170100_seal_private_content.exs added +179

@@ -0,0 +1,179 @@

1
defmodule OpenAgents.Repo.Migrations.SealPrivateContent do
2
  @moduledoc """
3
  The backfill half of sealing the private content columns nobody searches
4
  (issue #193).
5
6
  Every existing row is sealed under `OpenAgents.ContentVault` and its
7
  plaintext is nulled, so the words stop resting readable in the same deploy
8
  that starts sealing new ones. The plaintext columns themselves survive one
9
  more release for the nodes still writing into them; the contract migration
10
  drops them after that, the way `machine_pairings.user_id` was dropped a
11
  release after its last reader.
12
13
  It refuses rather than skips. A row that cannot be sealed — no key, or a key
14
  that does not decode — stops the migration, because a backfill that quietly
15
  leaves plaintext behind while `OpenAgents.Forge.AtRest` starts publishing the
16
  column as sealed is the exact claim `EXIT-006` exists to keep off the status
17
  page. An empty table needs no key, so a fresh database migrates without one.
18
19
  One honest limit: `UPDATE` writes a new row version and leaves the old one on
20
  disk until autovacuum reclaims it, so the plaintext survives in dead tuples
21
  for a bounded window after this runs. Recorded in
22
  `docs/2026-08-25-encryption-at-rest.md` rather than implied.
23
  """
24
25
  use Ecto.Migration
26
27
  alias OpenAgents.ContentVault
28
29
  @batch 500
30
31
  # {table, plaintext column, ciphertext column, the columns that bind the
32
  # seal and whether each is a `uuid`, trigger to disable during migration}.
33
  # The type is declared rather than sniffed: PostgreSQL hands `uuid` back as
34
  # a raw 16-byte binary, and a 16-character provider item id would be
35
  # indistinguishable from one.
36
  @columns [
37
    {"voice_transcript_items", "content", "content_ciphertext",
38
     [
39
       {"voice_session_id", :uuid},
40
       {"generation", :plain},
41
       {"provider_item_id", :plain},
42
       {"role", :plain}
43
     ], nil},
44
    {"voice_sessions", "compaction_summary", "compaction_summary_ciphertext",
45
     [{"id", :uuid}, {"generation", :plain}], nil},
46
    {"preference_observations", "summary", "summary_ciphertext",
47
     [{"owner_visitor_id", :uuid}, {"evidence_digest", :plain}],
48
     "preference_observations_append_only"},
49
    {"project_notes", "body", "body_ciphertext",
50
     [{"project_id", :plain}, {"repository_id", :uuid}, {"kind", :plain}], nil}
51
  ]
52
53
  def up do
54
    Enum.each(@columns, &with_trigger_disabled(&1, fn col -> convert(col, :seal) end))
55
  end
56
57
  def down do
58
    Enum.each(@columns, &with_trigger_disabled(&1, fn col -> convert(col, :open) end))
59
  end
60
61
  def run_direct!(repo_module, direction \\ :seal) do
62
    Enum.each(
63
      @columns,
64
      &with_trigger_disabled_direct(repo_module, &1, fn col ->
65
        convert_with_repo(repo_module, col, direction)
66
      end)
67
    )
68
  end
69
70
  defp with_trigger_disabled({table, _plaintext, _ciphertext, _binding, trigger} = col, fun) do
71
    if trigger do
72
      repo().query!("ALTER TABLE #{table} DISABLE TRIGGER #{trigger}")
73
    end
74
75
    try do
76
      fun.(col)
77
    after
78
      if trigger do
79
        repo().query!("ALTER TABLE #{table} ENABLE TRIGGER #{trigger}")
80
      end
81
    end
82
  end
83
84
  defp with_trigger_disabled_direct(
85
         repo_module,
86
         {table, _plaintext, _ciphertext, _binding, trigger} = col,
87
         fun
88
       ) do
89
    if trigger do
90
      repo_module.query!("ALTER TABLE #{table} DISABLE TRIGGER #{trigger}")
91
    end
92
93
    try do
94
      fun.(col)
95
    after
96
      if trigger do
97
        repo_module.query!("ALTER TABLE #{table} ENABLE TRIGGER #{trigger}")
98
      end
99
    end
100
  end
101
102
  defp convert(column, direction) do
103
    convert_with_repo(repo(), column, direction)
104
  end
105
106
  defp convert_with_repo(
107
         repo_module,
108
         {table, plaintext, ciphertext, binding_columns, _trigger} = column,
109
         direction
110
       ) do
111
    {source, target} =
112
      case direction do
113
        :seal -> {plaintext, ciphertext}
114
        :open -> {ciphertext, plaintext}
115
      end
116
117
    %{rows: rows} =
118
      repo_module.query!(
119
        """
120
        SELECT id, #{source}, #{binding_columns |> Enum.map_join(", ", &elem(&1, 0))}
121
        FROM #{table}
122
        WHERE #{source} IS NOT NULL
123
        LIMIT #{@batch}
124
        """,
125
        []
126
      )
127
128
    case rows do
129
      [] ->
130
        :ok
131
132
      rows ->
133
        Enum.each(rows, fn [id, value | binding] ->
134
          repo_module.query!(
135
            "UPDATE #{table} SET #{target} = $1, #{source} = NULL WHERE id = $2",
136
            [
137
              converted!(direction, table, plaintext, value, normalize(binding_columns, binding)),
138
              id
139
            ]
140
          )
141
        end)
142
143
        convert_with_repo(repo_module, column, direction)
144
    end
145
  end
146
147
  defp converted!(:seal, table, plaintext, value, binding) do
148
    case ContentVault.seal(value, "#{table}.#{plaintext}", binding) do
149
      {:ok, sealed} ->
150
        sealed
151
152
      {:error, reason} ->
153
        raise "the #{table}.#{plaintext} backfill cannot seal a row: #{reason}. " <>
154
                "Provision CONTENT_ENCRYPTION_KEY before migrating; this migration will " <>
155
                "not leave content readable while the ledger publishes it sealed."
156
    end
157
  end
158
159
  defp converted!(:open, table, plaintext, value, binding) do
160
    case ContentVault.open(value, "#{table}.#{plaintext}", binding) do
161
      {:ok, content} ->
162
        content
163
164
      {:error, reason} ->
165
        raise "the #{table}.#{plaintext} rollback cannot open a row: #{reason}"
166
    end
167
  end
168
169
  # PostgreSQL hands back `uuid` columns as raw 16-byte binaries, and the
170
  # application binds seals to the string form Ecto loads.
171
  defp normalize(binding_columns, binding) do
172
    binding_columns
173
    |> Enum.zip(binding)
174
    |> Enum.map(fn
175
      {{_column, :uuid}, value} -> Ecto.UUID.load!(value)
176
      {{_column, :plain}, value} -> value
177
    end)
178
  end
179
end
test/openagents/forge/at_rest_test.exs modified +190

@@ -20,12 +20,21 @@ defmodule OpenAgents.Forge.AtRestTest do

20 20
  use OpenAgents.DataCase, async: false
21 21
22 22
  alias OpenAgents.Accounts
23
  alias OpenAgents.Provenance.Canonical
24
  alias OpenAgents.Chat.AccountRun
25
  alias OpenAgents.ContentVault
23 26
  alias OpenAgents.Conversations
24 27
  alias OpenAgents.Forge.AtRest
28
  alias OpenAgents.Forum
25 29
  alias OpenAgents.Machines
30
  alias OpenAgents.Preferences
31
  alias OpenAgents.Preferences.Observation
32
  alias OpenAgents.Projects
33
  alias OpenAgents.Projects.ProjectNote
26 34
  alias OpenAgents.Repo
27 35
  alias OpenAgents.Voice
28 36
  alias OpenAgents.Voice.Config
37
  alias OpenAgents.Voice.Session
29 38
  alias OpenAgents.Voice.TranscriptItem
30 39
31 40
  describe "the sealed columns rest as ciphertext" do

@@ -92,6 +101,116 @@ defmodule OpenAgents.Forge.AtRestTest do

92 101
    end
93 102
  end
94 103
104
  describe "the sealed content columns rest as ciphertext" do
105
    # Issue #193. Each of these writes through the application path a person
106
    # actually reaches, then asks PostgreSQL two questions: is the ciphertext
107
    # column free of the words, and is the plaintext column it replaced empty.
108
    # The second question is the one a round-trip test cannot ask.
109
    test "a voice transcript is not readable in its own column" do
110
      content = "sealed-transcript-#{System.unique_integer([:positive])}"
111
      session = admitted_voice_session("at-rest-sealed-transcript")
112
113
      {:ok, item} =
114
        %TranscriptItem{}
115
        |> TranscriptItem.create_changeset(%{
116
          voice_session_id: session.id,
117
          generation: session.generation,
118
          provider_item_id: "item-#{System.unique_integer([:positive])}",
119
          role: "user",
120
          content: content,
121
          status: "final",
122
          observed_at: DateTime.utc_now()
123
        })
124
        |> Repo.insert()
125
126
      assert TranscriptItem.text(Repo.get!(TranscriptItem, item.id)) == content
127
128
      assert_sealed("voice_transcript_items", "content_ciphertext", "content", item.id, content)
129
    end
130
131
    test "an in-call compaction summary is not readable in its own column" do
132
      summary = "sealed-compaction-#{System.unique_integer([:positive])}"
133
      session = admitted_voice_session("at-rest-sealed-compaction")
134
135
      assert {:ok, updated, ^summary} =
136
               Voice.record_compaction_summary(session, session.generation, summary)
137
138
      assert Session.compaction_summary(Repo.get!(Session, updated.id)) == summary
139
140
      assert_sealed(
141
        "voice_sessions",
142
        "compaction_summary_ciphertext",
143
        "compaction_summary",
144
        session.id,
145
        summary
146
      )
147
    end
148
149
    test "a preference observation summary is not readable in its own column" do
150
      summary = "sealed-observation-#{System.unique_integer([:positive])}"
151
      {owner, conversation} = owner_conversation("at-rest-sealed-observation")
152
      {:ok, source} = Conversations.create_voice_context_message(conversation, "source")
153
154
      assert {:ok, observation} =
155
               Preferences.observe(owner, %{
156
                 "source_kind" => "current_user_message",
157
                 "source_message_id" => source.id,
158
                 "summary" => summary,
159
                 "confidence_millis" => 900,
160
                 "proposer_id" => "openagents.at_rest.test",
161
                 "proposer_digest" => Canonical.sha256("openagents.at_rest.test")
162
               })
163
164
      assert Observation.summary(Repo.get!(Observation, observation.id)) == summary
165
166
      assert_sealed(
167
        "preference_observations",
168
        "summary_ciphertext",
169
        "summary",
170
        observation.id,
171
        summary
172
      )
173
    end
174
175
    test "a project note body is not readable in its own column" do
176
      body = "sealed-note-#{System.unique_integer([:positive])}"
177
      repository = OpenAgents.AccountsFixtures.repository_fixture()
178
      author = github_user("at-rest-sealed-note")
179
180
      {:ok, project} =
181
        Projects.create_project(repository, %{title: "At rest", owner: author.github_login})
182
183
      assert {:ok, note} = Projects.create_project_note(project, %{"body" => body}, author)
184
      assert ProjectNote.text(Repo.get!(ProjectNote, note.id)) == body
185
186
      assert_sealed("project_notes", "body_ciphertext", "body", note.id, body)
187
    end
188
189
    test "a seal does not open under another column or another row" do
190
      # The additional authenticated data is the reason the ledger can name a
191
      # column rather than a key: ciphertext lifted out of one row does not
192
      # become someone else's sentence in another.
193
      content = "bound-#{System.unique_integer([:positive])}"
194
      binding = ["11111111-1111-1111-1111-111111111111", 1, "item-1", "user"]
195
196
      assert {:ok, sealed} =
197
               ContentVault.seal(content, "voice_transcript_items.content", binding)
198
199
      assert {:ok, ^content} =
200
               ContentVault.open(sealed, "voice_transcript_items.content", binding)
201
202
      assert {:error, :content_unsealable} =
203
               ContentVault.open(sealed, "project_notes.body", binding)
204
205
      assert {:error, :content_unsealable} =
206
               ContentVault.open(
207
                 sealed,
208
                 "voice_transcript_items.content",
209
                 ["11111111-1111-1111-1111-111111111111", 1, "item-1", "assistant"]
210
               )
211
    end
212
  end
213
95 214
  describe "the plaintext columns rest as plaintext" do
96 215
    # A ledger that names a gap has to be capable of being wrong about it.
97 216
    # These read the same way the sealed assertions do and expect the opposite

@@ -207,6 +326,21 @@ defmodule OpenAgents.Forge.AtRestTest do

207 326
    end
208 327
  end
209 328
329
  defp assert_sealed(table, ciphertext_column, plaintext_column, id, written) do
330
    sealed = raw_column(table, ciphertext_column, id)
331
332
    assert is_binary(sealed) and byte_size(sealed) > 0,
333
           "#{table}.#{ciphertext_column} is named as sealed but holds nothing"
334
335
    refute contains?(sealed, written),
336
           "#{table}.#{ciphertext_column} holds the text it was supposed to seal"
337
338
    assert is_nil(raw_column(table, plaintext_column, id)),
339
           "#{table}.#{plaintext_column} still holds a value. The seal is only a seal " <>
340
             "when the column it replaced is empty; a row written through the current " <>
341
             "release must leave nothing behind for the contract migration to drop."
342
  end
343
210 344
  # ── reading PostgreSQL rather than Ecto ──────────────────────────────────
211 345
212 346
  defp raw_column(table, column, id) do

@@ -290,6 +424,57 @@ defmodule OpenAgents.Forge.AtRestTest do

290 424
    {item.id, content}
291 425
  end
292 426
427
  defp write_private_row(%{table: "forum_posts", column: "body_text"}) do
428
    body = "plaintext-forum-#{System.unique_integer([:positive])}"
429
430
    {:ok, forum} =
431
      %Forum.Forum{}
432
      |> Forum.Forum.changeset(%{
433
        slug: "at-rest-#{System.unique_integer([:positive])}",
434
        title: "At rest"
435
      })
436
      |> Repo.insert()
437
438
    {:ok, topic} =
439
      Forum.create_topic(forum, %{
440
        actor_ref: "agent:user_#{Ecto.UUID.generate()}",
441
        actor_display_name: "At Rest",
442
        actor_slug: "at-rest",
443
        title: "At rest",
444
        slug: "at-rest-#{System.unique_integer([:positive])}",
445
        body_text: body
446
      })
447
448
    [post] = Forum.list_posts(topic)
449
450
    {post.id, body}
451
  end
452
453
  defp write_private_row(%{table: "account_chat_runs", column: column}) do
454
    content = "plaintext-chat-#{column}-#{System.unique_integer([:positive])}"
455
    {:ok, conversation} = Conversations.ensure_conversation("at-rest-chat-#{column}")
456
457
    attributes =
458
      Map.put(
459
        %{
460
          status: "completed",
461
          backend: List.first(OpenAgents.Chat.Backends.ids()),
462
          reasoning_effort: "low",
463
          user_content: content,
464
          started_at: DateTime.utc_now()
465
        },
466
        String.to_existing_atom(column),
467
        content
468
      )
469
470
    {:ok, run} =
471
      %AccountRun{conversation_id: conversation.id}
472
      |> AccountRun.changeset(attributes)
473
      |> Repo.insert()
474
475
    {run.id, content}
476
  end
477
293 478
  defp write_private_row(%{table: "issues", column: "body"}) do
294 479
    body = "plaintext-issue-#{System.unique_integer([:positive])}"
295 480
    repository = OpenAgents.AccountsFixtures.repository_fixture()

@@ -327,6 +512,11 @@ defmodule OpenAgents.Forge.AtRestTest do

327 512
    user
328 513
  end
329 514
515
  defp owner_conversation(key) do
516
    {:ok, conversation} = Conversations.ensure_conversation(key)
517
    {Conversations.get_conversation_owner!(conversation), conversation}
518
  end
519
330 520
  defp admitted_voice_session(key) do
331 521
    {:ok, conversation} = Conversations.ensure_conversation(key)
332 522
    {:ok, session} = Voice.admit_session(conversation, voice_config())
test/openagents/forge/key_rotation_test.exs modified +42 -2

@@ -38,6 +38,7 @@ defmodule OpenAgents.Forge.KeyRotationTest do

38 38
  import OpenAgents.IssuesFixtures
39 39
40 40
  alias OpenAgents.Accounts.TokenVault, as: GitHubVault
41
  alias OpenAgents.ContentVault
41 42
  alias OpenAgents.Forge.{Verification, WAL}
42 43
  alias OpenAgents.Machines.TokenVault, as: MachineVault
43 44
  alias OpenAgents.Reputation

@@ -74,7 +75,14 @@ defmodule OpenAgents.Forge.KeyRotationTest do

74 75
    test "the verifier was compiled against no secret and no vault" do
75 76
      callees = external_calls(Verification)
76 77
77
      for module <- [GitHubVault, MachineVault, RecordingVault, Reputation, OpenAgents.ApiTokens] do
78
      for module <- [
79
            GitHubVault,
80
            MachineVault,
81
            RecordingVault,
82
            ContentVault,
83
            Reputation,
84
            OpenAgents.ApiTokens
85
          ] do
78 86
        refute module in callees,
79 87
               "#{inspect(module)} reached OpenAgents.Forge.Verification. A receipt that " <>
80 88
                 "depends on a key stops being verifiable the moment that key rotates, " <>

@@ -281,6 +289,37 @@ defmodule OpenAgents.Forge.KeyRotationTest do

281 289
      )
282 290
    end
283 291
292
    test "the content vault seals under its own key and no other vault's opens it" do
293
      # VAULT-001, issue #193. The content vault is the fourth vault, and the
294
      # bug #253 repeated is a vault reading someone else's key. So this asks
295
      # the two questions that catch it: another vault's rotation must not
296
      # reach sealed content, and this vault's own rotation must strand it
297
      # rather than quietly falling back to a keyring it does not have.
298
      content = Base.encode64(:crypto.strong_rand_bytes(32))
299
      github = Base.encode64(:crypto.strong_rand_bytes(32))
300
      binding = [Ecto.UUID.generate(), 1, "item-1", "user"]
301
      scope = "voice_transcript_items.content"
302
303
      sealed =
304
        with_env([content_encryption_key: content, github_token_encryption_key: github], fn ->
305
          assert {:ok, sealed} = ContentVault.seal("a sentence", scope, binding)
306
          sealed
307
        end)
308
309
      rotated_github = Base.encode64(:crypto.strong_rand_bytes(32))
310
311
      with_env(
312
        [content_encryption_key: content, github_token_encryption_key: rotated_github],
313
        fn ->
314
          assert {:ok, "a sentence"} = ContentVault.open(sealed, scope, binding)
315
        end
316
      )
317
318
      with_env([content_encryption_key: Base.encode64(:crypto.strong_rand_bytes(32))], fn ->
319
        assert {:error, :content_unsealable} = ContentVault.open(sealed, scope, binding)
320
      end)
321
    end
322
284 323
    test "the voice recording vault has no keyring either, so its key cannot rotate" do
285 324
      first = Base.encode64(:crypto.strong_rand_bytes(32))
286 325
      second = Base.encode64(:crypto.strong_rand_bytes(32))

@@ -365,7 +404,8 @@ defmodule OpenAgents.Forge.KeyRotationTest do

365 404
      :forge_operator_token,
366 405
      :github_token_encryption_key,
367 406
      :github_token_encryption_key_id,
368
      :voice_recording_encryption_key
407
      :voice_recording_encryption_key,
408
      :content_encryption_key
369 409
    ]
370 410
371 411
    previous = Enum.map(keys, &{&1, Application.get_env(:openagents, &1)})
test/openagents/project_notes_test.exs modified +7 -6

@@ -4,6 +4,7 @@ defmodule OpenAgents.ProjectNotesTest do

4 4
  import OpenAgents.ProjectsFixtures
5 5
6 6
  alias OpenAgents.Projects
7
  alias OpenAgents.Projects.ProjectNote
7 8
8 9
  setup do
9 10
    repository = repository_fixture()

@@ -50,8 +51,8 @@ defmodule OpenAgents.ProjectNotesTest do

50 51
51 52
      assert total == 2
52 53
      assert Enum.all?(notes, &(&1.kind == "activity"))
53
      assert Enum.any?(notes, &(&1.body =~ "state"))
54
      assert Enum.any?(notes, &(&1.body =~ "title"))
54
      assert Enum.any?(notes, &(ProjectNote.text(&1) =~ "state"))
55
      assert Enum.any?(notes, &(ProjectNote.text(&1) =~ "title"))
55 56
      assert Enum.all?(notes, &(&1.author == %{"login" => author.github_login}))
56 57
57 58
      assert [activity | _] = notes

@@ -75,7 +76,7 @@ defmodule OpenAgents.ProjectNotesTest do

75 76
      assert {:ok, note} =
76 77
               Projects.create_project_note(project, %{"body" => "- paused lane 3"}, author)
77 78
78
      assert note.body == "- paused lane 3"
79
      assert ProjectNote.text(note) == "- paused lane 3"
79 80
      assert note.kind == "note"
80 81
      assert note.author == %{"login" => author.github_login}
81 82
      assert note.author_user_id == author.id

@@ -114,8 +115,8 @@ defmodule OpenAgents.ProjectNotesTest do

114 115
      assert total == per_page + 3
115 116
      assert length(first_page) == per_page
116 117
      assert length(second_page) == 3
117
      assert hd(first_page).body == "note #{per_page + 3}"
118
      assert List.last(second_page).body == "note 1"
118
      assert ProjectNote.text(hd(first_page)) == "note #{per_page + 3}"
119
      assert ProjectNote.text(List.last(second_page)) == "note 1"
119 120
    end
120 121
121 122
    test "a note belongs to one project", %{project: project, author: author} do

@@ -134,7 +135,7 @@ defmodule OpenAgents.ProjectNotesTest do

134 135
      refute Projects.authored_by?(note, nil)
135 136
136 137
      assert {:ok, edited} = Projects.update_project_note(note, %{"body" => "mine, edited"})
137
      assert edited.body == "mine, edited"
138
      assert ProjectNote.text(edited) == "mine, edited"
138 139
      assert {:ok, _deleted} = Projects.delete_project_note(edited)
139 140
      assert Projects.count_project_notes(project) == 0
140 141
    end
test/openagents/projects_test.exs modified +1 -1

@@ -577,7 +577,7 @@ defmodule OpenAgents.ProjectsTest do

577 577
      {:ok, _restored} = Projects.restore_project(archived, user)
578 578
579 579
      {notes, _total} = Projects.list_project_notes_page(project, kind: "activity")
580
      bodies = Enum.map(notes, & &1.body)
580
      bodies = Enum.map(notes, &OpenAgents.Projects.ProjectNote.text/1)
581 581
582 582
      assert Enum.any?(bodies, &(&1 =~ "Archived the project."))
583 583
      assert Enum.any?(bodies, &(&1 =~ "Restored the project from the archive."))
test/openagents/runtime_config_test.exs modified +18

@@ -219,6 +219,24 @@ defmodule OpenAgents.RuntimeConfigTest do

219 219
             RuntimeConfig.validate(settings)
220 220
  end
221 221
222
  test "a release refuses to boot without the content vault's own key" do
223
    # VAULT-001, issue #193. Nothing bridges to this key, so an unset value has
224
    # to stop the boot: the alternative is a node that accepts a project note
225
    # or a voice transcript and then cannot seal it.
226
    settings = Map.put(staging_settings(), :content_encryption_key, nil)
227
228
    assert {:error, %{setting: :content_encryption_key}} = RuntimeConfig.validate(settings)
229
230
    borrowed =
231
      Map.put(
232
        staging_settings(),
233
        :content_encryption_key,
234
        "not-a-base64-encoded-32-byte-key"
235
      )
236
237
    assert {:error, %{setting: :content_encryption_key}} = RuntimeConfig.validate(borrowed)
238
  end
239
222 240
  test "hot-load examples are executable startup policy, not prose" do
223 241
    settings =
224 242
      staging_settings()
test/openagents/seal_private_content_migration_test.exs added +237

@@ -0,0 +1,237 @@

1
defmodule OpenAgents.SealPrivateContentMigrationTest do
2
  use OpenAgents.DataCase
3
4
  @moduledoc """
5
  Proves the issue #193 backfill migration actually seals existing plaintext
6
  rows and that the application reads them back identically through the
7
  schemas.
8
9
  A unit test of the vault proves the crypto works. This test proves the
10
  *binding* works: that the migration and the application schema bind each
11
  table under the exact same identity strings, so a row sealed by the
12
  migration is readable by the schema, not dropped or rejected as corrupt.
13
  """
14
15
  # Require the migration file to define the module during testing
16
  Code.require_file("priv/repo/migrations/20260825170100_seal_private_content.exs")
17
18
  alias OpenAgents.Accounts
19
  alias OpenAgents.Conversations
20
  alias OpenAgents.Preferences
21
  alias OpenAgents.Preferences.Observation
22
  alias OpenAgents.Projects
23
  alias OpenAgents.Projects.ProjectNote
24
  alias OpenAgents.Provenance.Canonical
25
  alias OpenAgents.Repo
26
  alias OpenAgents.Repo.Migrations.SealPrivateContent
27
  alias OpenAgents.Voice
28
  alias OpenAgents.Voice.Config
29
  alias OpenAgents.Voice.Session
30
  alias OpenAgents.Voice.TranscriptItem
31
32
  test "every sealed column survives the backfill and reads back through the application" do
33
    t_row = transcript_row()
34
    c_row = compaction_row()
35
    o_row = observation_row()
36
    n_row = note_row()
37
    rows = [t_row, c_row, o_row, n_row]
38
39
    Enum.each(rows, &unseal_in_place!/1)
40
41
    # Every row now rests the way the migration will find it on a database that
42
    # has never run the backfill.
43
    for %{table: table, plaintext: plaintext, ciphertext: ciphertext, id: id} <- rows do
44
      assert raw(table, plaintext, id)
45
      refute raw(table, ciphertext, id)
46
    end
47
48
    SealPrivateContent.run_direct!(Repo, :seal)
49
50
    for %{table: table, plaintext: plaintext, ciphertext: ciphertext, id: id, written: written} <-
51
          rows do
52
      refute raw(table, plaintext, id),
53
             "#{table}.#{plaintext} still holds plaintext after the backfill"
54
55
      sealed = raw(table, ciphertext, id)
56
      assert is_binary(sealed) and byte_size(sealed) > 0
57
      refute :binary.match(sealed, written) != :nomatch
58
    end
59
60
    for %{name: name, read: read, written: written} <- rows do
61
      assert read.() == written,
62
             "#{name}: the backfill sealed this row under an identity the schema does not " <>
63
               "reconstruct, so the application reads nothing back (got #{inspect(read.())})"
64
    end
65
  end
66
67
  # Rewrites one sealed row back into the plaintext column, which is the state
68
  # every row is in before this migration runs.
69
  defp unseal_in_place!(
70
         %{table: table, plaintext: plaintext, ciphertext: ciphertext, id: id} = row
71
       ) do
72
    if table == "preference_observations" do
73
      Repo.query!(
74
        "ALTER TABLE preference_observations DISABLE TRIGGER preference_observations_append_only"
75
      )
76
    end
77
78
    try do
79
      Repo.query!(
80
        "UPDATE #{table} SET #{plaintext} = $1, #{ciphertext} = NULL WHERE id = $2",
81
        [row.written, dump_id(id)]
82
      )
83
    after
84
      if table == "preference_observations" do
85
        Repo.query!(
86
          "ALTER TABLE preference_observations ENABLE TRIGGER preference_observations_append_only"
87
        )
88
      end
89
    end
90
91
    row
92
  end
93
94
  defp transcript_row do
95
    content = "backfill-transcript-#{System.unique_integer([:positive])}"
96
    session = admitted_voice_session("backfill-transcript")
97
98
    {:ok, item} =
99
      %TranscriptItem{}
100
      |> TranscriptItem.create_changeset(%{
101
        voice_session_id: session.id,
102
        generation: session.generation,
103
        provider_item_id: "item-#{System.unique_integer([:positive])}",
104
        role: "user",
105
        content: content,
106
        status: "final",
107
        observed_at: DateTime.utc_now()
108
      })
109
      |> Repo.insert()
110
111
    %{
112
      name: "voice_transcript_item",
113
      table: "voice_transcript_items",
114
      plaintext: "content",
115
      ciphertext: "content_ciphertext",
116
      id: item.id,
117
      written: content,
118
      read: fn -> TranscriptItem.text(Repo.get!(TranscriptItem, item.id)) end
119
    }
120
  end
121
122
  defp compaction_row do
123
    summary = "backfill-compaction-#{System.unique_integer([:positive])}"
124
    session = admitted_voice_session("backfill-compaction")
125
126
    {:ok, updated, _} =
127
      Voice.record_compaction_summary(session, session.generation, summary)
128
129
    %{
130
      name: "voice_session_compaction",
131
      table: "voice_sessions",
132
      plaintext: "compaction_summary",
133
      ciphertext: "compaction_summary_ciphertext",
134
      id: updated.id,
135
      written: summary,
136
      read: fn -> Session.compaction_summary(Repo.get!(Session, updated.id)) end
137
    }
138
  end
139
140
  defp observation_row do
141
    summary = "backfill-pref-#{System.unique_integer([:positive])}"
142
    {owner, conversation} = owner_conversation("backfill-observation")
143
    {:ok, source} = Conversations.create_voice_context_message(conversation, "source")
144
145
    {:ok, observation} =
146
      Preferences.observe(owner, %{
147
        "source_kind" => "current_user_message",
148
        "source_message_id" => source.id,
149
        "summary" => summary,
150
        "confidence_millis" => 900,
151
        "proposer_id" => "openagents.backfill.test",
152
        "proposer_digest" => Canonical.sha256("openagents.backfill.test")
153
      })
154
155
    %{
156
      name: "preference_observation",
157
      table: "preference_observations",
158
      plaintext: "summary",
159
      ciphertext: "summary_ciphertext",
160
      id: observation.id,
161
      written: summary,
162
      read: fn -> Observation.summary(Repo.get!(Observation, observation.id)) end
163
    }
164
  end
165
166
  defp note_row do
167
    body = "backfill-note-#{System.unique_integer([:positive])}"
168
    repository = OpenAgents.AccountsFixtures.repository_fixture()
169
    author = github_user("backfill-note")
170
171
    {:ok, project} =
172
      Projects.create_project(repository, %{title: "Backfill Note", owner: author.github_login})
173
174
    {:ok, note} = Projects.create_project_note(project, %{"body" => body}, author)
175
176
    %{
177
      name: "project_note",
178
      table: "project_notes",
179
      plaintext: "body",
180
      ciphertext: "body_ciphertext",
181
      id: note.id,
182
      written: body,
183
      read: fn -> ProjectNote.text(Repo.get!(ProjectNote, note.id)) end
184
    }
185
  end
186
187
  defp raw(table, column, id) do
188
    %{rows: [[val]]} =
189
      Repo.query!(
190
        "SELECT #{column} FROM #{table} WHERE id = $1",
191
        [dump_id(id)]
192
      )
193
194
    val
195
  end
196
197
  defp dump_id(id) when is_integer(id), do: id
198
  defp dump_id(id) when is_binary(id), do: Ecto.UUID.dump!(id)
199
200
  defp github_profile(key) do
201
    digest = :crypto.hash(:sha256, key)
202
203
    %{
204
      github_id: digest |> binary_part(0, 7) |> :binary.decode_unsigned(),
205
      github_login: "backfill-#{Base.encode16(digest, case: :lower) |> binary_part(0, 10)}",
206
      github_avatar_url: "https://avatars.githubusercontent.com/u/1?v=4"
207
    }
208
  end
209
210
  defp github_user(key) do
211
    {:ok, user} = Accounts.upsert_github_user(github_profile(key))
212
    user
213
  end
214
215
  defp owner_conversation(key) do
216
    {:ok, conversation} = Conversations.ensure_conversation(key)
217
    {Conversations.get_conversation_owner!(conversation), conversation}
218
  end
219
220
  defp admitted_voice_session(key) do
221
    {:ok, conversation} = Conversations.ensure_conversation(key)
222
    {:ok, session} = Voice.admit_session(conversation, voice_config())
223
    session
224
  end
225
226
  defp voice_config do
227
    Config.build!(
228
      enabled: true,
229
      architecture: :openai_realtime,
230
      provider: "openai",
231
      model: "gpt-realtime-2.1",
232
      voice: "marin",
233
      reasoning_effort: "low",
234
      maximum_session_seconds: 3_000
235
    )
236
  end
237
end
test/openagents/voice_sessions_test.exs modified +17 -7

@@ -2,6 +2,8 @@ defmodule OpenAgents.VoiceSessionsTest do

2 2
  use OpenAgents.DataCase, async: false
3 3
  alias OpenAgents.{Conversations, Voice, VoiceSessions}
4 4
  alias OpenAgents.Voice.ProviderEvent
5
  alias OpenAgents.Voice.Session
6
  alias OpenAgents.Voice.TranscriptItem
5 7
6 8
  setup do
7 9
    previous_voice = Application.fetch_env!(:openagents, :voice)

@@ -105,9 +107,9 @@ defmodule OpenAgents.VoiceSessionsTest do

105 107
    assert Enum.map(Voice.list_events(stored), & &1.kind) == expected_event_kinds
106 108
107 109
    assert [
108
             %{role: "user", content: "Tell me who you are."},
109
             %{role: "assistant", content: "A durable final line."}
110
           ] = Voice.list_transcript_items(stored)
110
             %{role: "user", text: "Tell me who you are."},
111
             %{role: "assistant", text: "A durable final line."}
112
           ] = transcript_roles_and_text(stored)
111 113
112 114
    assert {:ok, ended} = VoiceSessions.end_session(stored)
113 115
    assert ended.status == "ended"

@@ -204,9 +206,9 @@ defmodule OpenAgents.VoiceSessionsTest do

204 206
    assert state.live_transcripts == %{}
205 207
206 208
    assert [
207
             %{role: "user", content: "Say something long."},
208
             %{role: "assistant", content: "The first words."}
209
           ] = Voice.list_transcript_items(stored)
209
             %{role: "user", text: "Say something long."},
210
             %{role: "assistant", text: "The first words."}
211
           ] = transcript_roles_and_text(stored)
210 212
211 213
    assert {:ok, _ended} = VoiceSessions.end_session(stored)
212 214
  end

@@ -1482,7 +1484,7 @@ defmodule OpenAgents.VoiceSessionsTest do

1482 1484
    assert state.compaction == nil
1483 1485
1484 1486
    stored = Voice.get_session!(session.id)
1485
    assert stored.compaction_summary == summary
1487
    assert Session.compaction_summary(stored) == summary
1486 1488
    assert stored.compaction_count == 1
1487 1489
    assert stored.status == "listening"
1488 1490

@@ -1559,4 +1561,12 @@ defmodule OpenAgents.VoiceSessionsTest do

1559 1561
1560 1562
    assert {:ok, _ended} = VoiceSessions.end_session(Voice.get_session!(session.id))
1561 1563
  end
1564
1565
  # Transcript text rests sealed (issue #193), so a transcript assertion asks
1566
  # the schema for the words rather than reading the column.
1567
  defp transcript_roles_and_text(session) do
1568
    session
1569
    |> Voice.list_transcript_items()
1570
    |> Enum.map(&%{role: &1.role, text: TranscriptItem.text(&1)})
1571
  end
1562 1572
end
test/openagents/voice_test.exs modified +4 -2

@@ -2,7 +2,7 @@ defmodule OpenAgents.VoiceTest do

2 2
  use OpenAgents.DataCase, async: false
3 3
  alias OpenAgents.{Conversations, Voice}
4 4
  alias OpenAgents.Tools.{ExecutionContext, Registry, Runner}
5
  alias OpenAgents.Voice.{Config, ProviderEvent, Session}
5
  alias OpenAgents.Voice.{Config, ProviderEvent, Session, TranscriptItem}
6 6
7 7
  test "admits one immutable generation and fences duplicate, stale, and terminal work" do
8 8
    {:ok, conversation} = Conversations.ensure_conversation("voice-generation-browser")

@@ -104,7 +104,9 @@ defmodule OpenAgents.VoiceTest do

104 104
    assert final_session.usage["pricing_id"] == "openai.gpt-realtime-2.1.2026-08-16"
105 105
106 106
    assert [user, assistant] = Voice.list_transcript_items(final_session)
107
    assert {user.role, user.status, user.content} == {"user", "final", "What do you remember?"}
107
108
    assert {user.role, user.status, TranscriptItem.text(user)} ==
109
             {"user", "final", "What do you remember?"}
108 110
109 111
    assert {assistant.role, assistant.provider_response_id, assistant.status} ==
110 112
             {"assistant", "response-1", "final"}

This page updates live while a promote is in flight · changelog