Sync engrams without ever making a turn wait

bb1d07dbdc3b · AtlantisPleb · · parent b4cefc3a43ab

Sync engrams without ever making a turn wait

The ledger is local and authoritative; sync is how a copy of it
reaches somewhere else. One rule shapes the whole thing: a turn never
waits for it and never fails because of it, because memory that can
break a conversation is worse than memory that is briefly out of date.

EngramTransport is the seam — publish and fetch, nothing about relays,
sockets, or keys — with an in-memory implementation behind it.
EngramSyncQueue sits between the ledger and a transport: publish
enqueues and returns, drain is what talks to the transport, and a host
drains between turns rather than inside one.

Three properties, each with a test. Local-first: an engram is on disk
and readable before anything is delivered. Nothing is lost: an
unreachable transport leaves the engram queued and it goes on the next
drain, so an hour of downtime costs an hour of latency and no engrams.
Degraded is a state, not an error: the queue reports what it holds and
what failed last, so a session can say it is behind instead of leaving
a reader to infer it from silence.

Two distinctions worth the code. A refusal is terminal while
unreachable and failed are not — retrying something the transport has
judged invalid only repeats the judgement. And a transport that throws
is treated as failing rather than propagating, because a throw from
there would reach the caller's turn, which is the one thing this may
not do.

CoderMemory queues every engram it records and exposes flush and
syncStatus, so the seam is exercised in the live path rather than
sitting unused behind tests.

Real Nostr transport is deliberately not here: it drops in behind
EngramTransport without any caller changing, and should build on the
workspace's shared nostr-effect rather than reimplementing NIP
primitives.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GoYpb8FEmdxVErsv7ABCYi
Co-Authored-By
Claude Fable 5 <noreply@anthropic.com>

Deploy story

What this commit did to the running system — joined from the forge receipt chain, the part a commit page elsewhere cannot show.

Not deployed through the forge lane

No push, promotion, build, or deploy receipt references this commit (receipts are scanned over a bounded recent window). Changes shipped by full node replacement carry their proof in the release gate receipt instead.

Changed files

  • modified docs/assure-repo/false-green-candidates.v1.json
  • modified docs/assure-repo/surface-inventory.v1.json
  • modified packages/agent-experience-memory/src/index.ts
  • added packages/agent-experience-memory/src/sync.test.ts
  • added packages/agent-experience-memory/src/sync.ts
  • modified packages/openagents-cli/scripts/vendor-memory.mjs
  • modified packages/openagents-cli/src/coder-memory.ts
  • modified packages/openagents-cli/src/memory/index.ts
  • added packages/openagents-cli/src/memory/sync.ts
  • modified packages/openagents-cli/test/coder-memory.test.ts

Diff

10 files changed, +737 -3

docs/assure-repo/false-green-candidates.v1.json modified +1 -1

@@ -4,7 +4,7 @@

4 4
  "note": "Heuristic false-green LEADS, not findings. A finding requires a demonstrated reproduction (surviving mutation via mutation-runner). Do not treat a candidate as a confirmed false green. Coverage-theater leads may include tests that delegate their assertion to a custom helper the classifier does not recognise; verify before acting.",
5 5
  "sourceDigest": "sha256:dd810dd48c5bdbc9becd7fcc01dd41a4ca2abf0b2d6f6a545907247f6e3e8361",
6 6
  "summary": {
7
    "filesScanned": 2468,
7
    "filesScanned": 2469,
8 8
    "candidateCount": 16,
9 9
    "byMode": {
10 10
      "false_green_coverage_theater": 15,
docs/assure-repo/surface-inventory.v1.json modified +2 -2

@@ -1,7 +1,7 @@

1 1
{
2 2
  "schemaVersion": "1",
3 3
  "repository": "OpenAgentsInc/openagents",
4
  "sourceDigest": "sha256:359b6533e42f95b7605e98f21e6ed7a4f837f917a73ad1723aab8cee17ec0752",
4
  "sourceDigest": "sha256:d6a28da451ce9ef6f5cf228f613eb7a4a8e932087110a750f3bb6d198737304c",
5 5
  "surfaces": [
6 6
    {
7 7
      "id": "app:@openagentsinc/acceptance-runner",

@@ -1533,7 +1533,7 @@

1533 1533
      "oracles": [
1534 1534
        {
1535 1535
          "type": "test",
1536
          "ref": "packages/agent-experience-memory (10 tracked test files)"
1536
          "ref": "packages/agent-experience-memory (11 tracked test files)"
1537 1537
        }
1538 1538
      ],
1539 1539
      "obligation": {
packages/agent-experience-memory/src/index.ts modified +1

@@ -40,3 +40,4 @@ export * from "./engram.js";

40 40
export * from "./consolidation.js";
41 41
export * from "./subagent-memory.js";
42 42
export * from "./projection.js";
43
export * from "./sync.js";
packages/agent-experience-memory/src/sync.test.ts added +167

@@ -0,0 +1,167 @@

1
import { describe, expect, test } from "vite-plus/test";
2
3
import {
4
  buildEngramBody,
5
  buildEngramEvent,
6
  engramContentDigest,
7
  type EngramEvent,
8
} from "./engram.js";
9
import {
10
  EngramSyncQueue,
11
  MemoryTransport,
12
  type EngramTransport,
13
  type PublishResult,
14
} from "./sync.js";
15
16
const PUBKEY = "a".repeat(64);
17
const sign = (eventId: string): string => `sig-${eventId.slice(0, 16)}`;
18
19
const engram = (slug: string, value: string, createdAt: number): EngramEvent => {
20
  const body = buildEngramBody(slug, value, {
21
    admission: "admitted",
22
    entityId: "entity-1",
23
    contentDigest: engramContentDigest(value),
24
    sourceEventRefs: [],
25
    relations: [],
26
    derivedFromSlugs: [],
27
  });
28
  return buildEngramEvent(PUBKEY, createdAt, slug, JSON.stringify(body), sign);
29
};
30
31
describe("the memory transport", () => {
32
  test("stores what it accepts and finds it by filter", async () => {
33
    const transport = new MemoryTransport();
34
    const first = engram("note/one", "first", 1_000);
35
    const second = engram("note/two", "second", 2_000);
36
    await transport.publish(first);
37
    await transport.publish(second);
38
39
    expect(await transport.fetch({ slugs: ["note/one"] })).toEqual([first]);
40
    expect(await transport.fetch({ since: 1_500 })).toEqual([second]);
41
    expect(await transport.fetch({ authors: [PUBKEY] })).toHaveLength(2);
42
    expect(await transport.fetch({ limit: 1 })).toEqual([first]);
43
  });
44
45
  test("refuses an engram whose id does not verify", async () => {
46
    const transport = new MemoryTransport();
47
    const authentic = engram("note/one", "authentic", 1_000);
48
    const tampered: EngramEvent = {
49
      ...authentic,
50
      content: authentic.content.replace("authentic", "forged"),
51
    };
52
53
    const result = await transport.publish(tampered);
54
    expect(result).toEqual({ ok: false, reason: "refused", detail: "event id does not verify" });
55
    expect(transport.stored()).toHaveLength(0);
56
  });
57
});
58
59
describe("the sync queue", () => {
60
  test("publishing does not wait on the transport", () => {
61
    const queue = new EngramSyncQueue(new MemoryTransport());
62
    // No await: publish returns nothing to wait on, and the engram is queued.
63
    queue.publish(engram("note/one", "first", 1_000));
64
    expect(queue.status().pending).toBe(1);
65
    expect(queue.status().delivered).toBe(0);
66
  });
67
68
  test("a drain delivers what is queued", async () => {
69
    const transport = new MemoryTransport();
70
    const queue = new EngramSyncQueue(transport);
71
    queue.publish(engram("note/one", "first", 1_000));
72
    queue.publish(engram("note/two", "second", 2_000));
73
74
    expect(await queue.drain()).toBe(2);
75
    expect(queue.status()).toMatchObject({ pending: 0, delivered: 2, refused: 0 });
76
    expect(transport.stored()).toHaveLength(2);
77
  });
78
79
  test("an unreachable transport loses nothing and says it is behind", async () => {
80
    const transport = new MemoryTransport();
81
    transport.reachable = false;
82
    const queue = new EngramSyncQueue(transport);
83
    queue.publish(engram("note/one", "first", 1_000));
84
85
    expect(await queue.drain()).toBe(0);
86
    expect(queue.behind()).toBe(true);
87
    expect(queue.status()).toMatchObject({ pending: 1, delivered: 0 });
88
    expect(queue.status().lastFailure?.reason).toBe("unreachable");
89
90
    // The engram is still there when the transport comes back.
91
    transport.reachable = true;
92
    expect(await queue.drain()).toBe(1);
93
    expect(queue.behind()).toBe(false);
94
    expect(transport.stored()).toHaveLength(1);
95
  });
96
97
  test("a refusal is terminal and does not retry forever", async () => {
98
    const transport = new MemoryTransport();
99
    const queue = new EngramSyncQueue(transport);
100
    const authentic = engram("note/one", "authentic", 1_000);
101
    queue.publish({ ...authentic, content: authentic.content.replace("authentic", "forged") });
102
103
    expect(await queue.drain()).toBe(0);
104
    expect(queue.status()).toMatchObject({ pending: 0, refused: 1 });
105
    // Nothing left to retry: the transport judged it, and repeating the call
106
    // only repeats the judgement.
107
    expect(await queue.drain()).toBe(0);
108
    expect(queue.behind()).toBe(false);
109
  });
110
111
  test("a transport that throws is treated as failing, not as a crash", async () => {
112
    const throwing: EngramTransport = {
113
      publish(): Promise<PublishResult> {
114
        throw new Error("socket exploded");
115
      },
116
      fetch(): Promise<ReadonlyArray<EngramEvent>> {
117
        throw new Error("socket exploded");
118
      },
119
    };
120
    const queue = new EngramSyncQueue(throwing);
121
    queue.publish(engram("note/one", "first", 1_000));
122
123
    await expect(queue.drain()).resolves.toBe(0);
124
    expect(queue.status().lastFailure).toMatchObject({
125
      reason: "failed",
126
      detail: "socket exploded",
127
    });
128
    expect(queue.status().pending).toBe(1);
129
    await expect(queue.fetch({})).resolves.toEqual([]);
130
  });
131
132
  test("an engram is never delivered twice", async () => {
133
    const transport = new MemoryTransport();
134
    const queue = new EngramSyncQueue(transport);
135
    const event = engram("note/one", "first", 1_000);
136
137
    queue.publish(event);
138
    expect(await queue.drain()).toBe(1);
139
140
    // Republishing the whole ledger costs one pass, not one delivery per pass.
141
    queue.publish(event);
142
    expect(queue.status().pending).toBe(0);
143
    expect(await queue.drain()).toBe(0);
144
    expect(queue.status().delivered).toBe(1);
145
  });
146
147
  test("the same engram queued twice before a drain delivers once", async () => {
148
    const transport = new MemoryTransport();
149
    const queue = new EngramSyncQueue(transport);
150
    const event = engram("note/one", "first", 1_000);
151
    queue.publish(event);
152
    queue.publish(event);
153
154
    expect(queue.status().pending).toBe(1);
155
    expect(await queue.drain()).toBe(1);
156
  });
157
158
  test("fetching reads back what was delivered", async () => {
159
    const transport = new MemoryTransport();
160
    const queue = new EngramSyncQueue(transport);
161
    const event = engram("note/one", "first", 1_000);
162
    queue.publish(event);
163
    await queue.drain();
164
165
    expect(await queue.fetch({ slugs: ["note/one"] })).toEqual([event]);
166
  });
167
});
packages/agent-experience-memory/src/sync.ts added +242

@@ -0,0 +1,242 @@

1
import { Schema as S } from "effect";
2
3
import { computeEngramEventId, type EngramEvent } from "./engram.js";
4
5
/**
6
 * The engram sync seam (issue #222).
7
 *
8
 * The ledger is local and authoritative. Sync is how a copy of it reaches
9
 * somewhere else, and the whole design follows from one rule: **a turn never
10
 * waits for it, and never fails because of it.** A relay that is slow, down,
11
 * or gone must be indistinguishable from one that is fine, as far as the
12
 * caller is concerned — because memory that can break a conversation is worse
13
 * than memory that is briefly out of date.
14
 *
15
 * That gives the three properties the tests hold this to:
16
 *
17
 * 1. **Local-first.** `publish` records the engram as pending and returns
18
 *    immediately. The engram is already in the local ledger by then; sync is
19
 *    catching up, not gatekeeping.
20
 * 2. **Nothing is lost.** A failed publish stays queued and is retried. The
21
 *    queue only forgets an engram once a transport has acknowledged it, so a
22
 *    transport that is down for an hour costs an hour of latency and no
23
 *    engrams.
24
 * 3. **Degraded is a state, not an error.** The queue reports what it is
25
 *    holding and what failed last, so a caller can *say* it is behind rather
26
 *    than discovering it by silence.
27
 *
28
 * What is deliberately not here: relays, sockets, keys, encryption. A
29
 * transport is anything that satisfies `EngramTransport`, and the real Nostr
30
 * one lands behind this interface without any caller changing. The workspace
31
 * has a shared Nostr implementation (`nostr-effect`) that a real transport
32
 * should build on rather than reimplement.
33
 */
34
35
export const SYNC_SCHEMA_ID = "openagents.engram_sync.v1" as const;
36
37
/** Why a publish did not land. Distinct because they need distinct responses. */
38
export const SyncFailureReason = S.Literals([
39
  /** The transport could not be reached at all. Retry later, unchanged. */
40
  "unreachable",
41
  /** The transport reached and refused this engram. Retrying will not help. */
42
  "refused",
43
  /** The transport accepted the call and failed inside it. Retry later. */
44
  "failed",
45
]);
46
export type SyncFailureReason = typeof SyncFailureReason.Type;
47
48
export type PublishResult =
49
  | { readonly ok: true; readonly eventId: string }
50
  | { readonly ok: false; readonly reason: SyncFailureReason; readonly detail?: string };
51
52
/** A filter for what to fetch back. Absent fields mean "no constraint". */
53
export interface EngramFilter {
54
  readonly authors?: ReadonlyArray<string>;
55
  /** `d` tag values — engram slugs. */
56
  readonly slugs?: ReadonlyArray<string>;
57
  /** Unix seconds; inclusive. */
58
  readonly since?: number;
59
  readonly until?: number;
60
  readonly limit?: number;
61
}
62
63
/**
64
 * Whatever carries engrams somewhere else.
65
 *
66
 * Implementations must not throw: a transport that cannot answer returns a
67
 * failure, because a throw from here would reach the caller's turn and the
68
 * whole point is that it cannot.
69
 */
70
export interface EngramTransport {
71
  publish(event: EngramEvent): Promise<PublishResult>;
72
  fetch(filter: EngramFilter): Promise<ReadonlyArray<EngramEvent>>;
73
}
74
75
/** What the queue is holding, for a caller that wants to say so. */
76
export interface SyncStatus {
77
  readonly pending: number;
78
  readonly delivered: number;
79
  /** Engrams the transport refused outright. They are not retried. */
80
  readonly refused: number;
81
  readonly lastFailure?: { readonly reason: SyncFailureReason; readonly detail?: string };
82
}
83
84
const matches = (event: EngramEvent, filter: EngramFilter): boolean => {
85
  if (filter.authors !== undefined && !filter.authors.includes(event.pubkey)) return false;
86
  if (filter.since !== undefined && event.created_at < filter.since) return false;
87
  if (filter.until !== undefined && event.created_at > filter.until) return false;
88
  if (filter.slugs !== undefined) {
89
    const slug = event.tags.find((tag) => tag[0] === "d")?.[1];
90
    if (slug === undefined || !filter.slugs.includes(slug)) return false;
91
  }
92
  return true;
93
};
94
95
/**
96
 * A transport that keeps engrams in memory.
97
 *
98
 * The reference implementation and what the tests run against. It is also a
99
 * useful real transport for a single process that wants sync's shape without
100
 * a relay.
101
 */
102
export class MemoryTransport implements EngramTransport {
103
  private readonly events = new Map<string, EngramEvent>();
104
  /** Set to fail every publish, for degraded-mode tests. */
105
  reachable = true;
106
107
  publish(event: EngramEvent): Promise<PublishResult> {
108
    if (!this.reachable) {
109
      return Promise.resolve({ ok: false, reason: "unreachable", detail: "transport is down" });
110
    }
111
    if (event.id !== computeEngramEventId(event)) {
112
      return Promise.resolve({ ok: false, reason: "refused", detail: "event id does not verify" });
113
    }
114
    this.events.set(event.id, event);
115
    return Promise.resolve({ ok: true, eventId: event.id });
116
  }
117
118
  fetch(filter: EngramFilter): Promise<ReadonlyArray<EngramEvent>> {
119
    if (!this.reachable) return Promise.resolve([]);
120
    const found = [...this.events.values()]
121
      .filter((event) => matches(event, filter))
122
      .sort((left, right) =>
123
        left.created_at !== right.created_at
124
          ? left.created_at - right.created_at
125
          : left.id.localeCompare(right.id),
126
      );
127
    return Promise.resolve(filter.limit === undefined ? found : found.slice(0, filter.limit));
128
  }
129
130
  /** Everything the transport holds, for assertions. */
131
  stored(): ReadonlyArray<EngramEvent> {
132
    return [...this.events.values()];
133
  }
134
}
135
136
/**
137
 * The queue between the ledger and a transport.
138
 *
139
 * `publish` is synchronous from the caller's side: it enqueues and returns.
140
 * `drain` is what actually talks to the transport, and a host calls it
141
 * whenever it likes — after a turn, on a timer, at exit. Nothing about the
142
 * caller's turn depends on when that happens.
143
 */
144
export class EngramSyncQueue {
145
  private readonly queue: Array<EngramEvent> = [];
146
  private readonly deliveredIds = new Set<string>();
147
  private readonly refusedIds = new Set<string>();
148
  private lastFailure: SyncStatus["lastFailure"];
149
  private draining = false;
150
151
  constructor(private readonly transport: EngramTransport) {}
152
153
  /**
154
   * Enqueue an engram for delivery. Returns nothing to wait on.
155
   *
156
   * An engram already delivered or already refused is not enqueued twice, so
157
   * a caller that republishes its whole ledger costs one pass, not one
158
   * delivery per pass.
159
   */
160
  publish(event: EngramEvent): void {
161
    if (this.deliveredIds.has(event.id) || this.refusedIds.has(event.id)) return;
162
    if (this.queue.some((queued) => queued.id === event.id)) return;
163
    this.queue.push(event);
164
  }
165
166
  /**
167
   * Try to deliver everything queued.
168
   *
169
   * Returns how many landed. An unreachable or failing transport leaves the
170
   * engram queued for the next drain; a refusal is terminal, because retrying
171
   * something the transport has judged invalid only repeats the judgement.
172
   * Never throws — a transport that throws anyway is treated as failing.
173
   */
174
  async drain(): Promise<number> {
175
    if (this.draining) return 0;
176
    this.draining = true;
177
    try {
178
      let delivered = 0;
179
      // Copy: a publish during a drain lands in the next one rather than
180
      // mutating the array being walked.
181
      const attempting = [...this.queue];
182
      for (const event of attempting) {
183
        let result: PublishResult;
184
        try {
185
          result = await this.transport.publish(event);
186
        } catch (cause) {
187
          result = {
188
            ok: false,
189
            reason: "failed",
190
            detail: cause instanceof Error ? cause.message : String(cause),
191
          };
192
        }
193
        if (result.ok) {
194
          this.deliveredIds.add(event.id);
195
          this.removeFromQueue(event.id);
196
          delivered += 1;
197
          continue;
198
        }
199
        this.lastFailure = {
200
          reason: result.reason,
201
          ...(result.detail === undefined ? {} : { detail: result.detail }),
202
        };
203
        if (result.reason === "refused") {
204
          this.refusedIds.add(event.id);
205
          this.removeFromQueue(event.id);
206
        }
207
        // "unreachable" and "failed" stay queued for the next drain.
208
      }
209
      return delivered;
210
    } finally {
211
      this.draining = false;
212
    }
213
  }
214
215
  private removeFromQueue(eventId: string): void {
216
    const at = this.queue.findIndex((queued) => queued.id === eventId);
217
    if (at >= 0) this.queue.splice(at, 1);
218
  }
219
220
  /** Fetch from the transport. An unreachable transport yields nothing. */
221
  async fetch(filter: EngramFilter): Promise<ReadonlyArray<EngramEvent>> {
222
    try {
223
      return await this.transport.fetch(filter);
224
    } catch {
225
      return [];
226
    }
227
  }
228
229
  status(): SyncStatus {
230
    return {
231
      pending: this.queue.length,
232
      delivered: this.deliveredIds.size,
233
      refused: this.refusedIds.size,
234
      ...(this.lastFailure === undefined ? {} : { lastFailure: this.lastFailure }),
235
    };
236
  }
237
238
  /** Whether the queue is holding anything the transport has not taken. */
239
  behind(): boolean {
240
    return this.queue.length > 0;
241
  }
242
}
packages/openagents-cli/scripts/vendor-memory.mjs modified +2

@@ -46,6 +46,7 @@ export const VENDORED = [

46 46
      ["./internal/sha256.js", "./sha256.js"],
47 47
    ],
48 48
  ],
49
  ["agent-experience-memory/src/sync.ts", "sync.ts", []],
49 50
  [
50 51
    "agent-experience-memory/src/consolidation.ts",
51 52
    "consolidation.ts",

@@ -87,6 +88,7 @@ export * from "./engram.js";

87 88
export * from "./consolidation.js";
88 89
export * from "./subagent-memory.js";
89 90
export * from "./projection.js";
91
export * from "./sync.js";
90 92
`;
91 93
92 94
export const writeAll = () => {
packages/openagents-cli/src/coder-memory.ts modified +35

@@ -21,6 +21,8 @@ import {

21 21
  ledgerEntriesAsHeuristics,
22 22
  promoteHeuristicToPattern,
23 23
  signSupersedingEngram,
24
  EngramSyncQueue,
25
  MemoryTransport,
24 26
  project,
25 27
  projectedValue,
26 28
  projectionMatches,

@@ -29,8 +31,10 @@ import {

29 31
  HarvestedLedgerEntry,
30 32
  type EngramEvent,
31 33
  type EngramBody,
34
  type EngramTransport,
32 35
  type ParentHeuristic,
33 36
  type Projection,
37
  type SyncStatus,
34 38
} from "./memory/index.js";
35 39
import { Schema as S } from "effect";
36 40

@@ -69,6 +73,12 @@ export interface CoderMemoryOptions {

69 73
  readonly projectScope?: string;
70 74
  /** Epoch-milliseconds clock, injectable for tests. */
71 75
  readonly now?: () => number;
76
  /**
77
   * Where engrams are mirrored. Defaults to an in-process transport, which
78
   * keeps the seam exercised without a relay; a real Nostr transport drops in
79
   * here without any caller changing (#222).
80
   */
81
  readonly transport?: EngramTransport;
72 82
  /**
73 83
   * New harvests needed before the harvest path schedules a dream.
74 84
   * `Infinity` turns the automatic pass off, leaving `dream` explicit.

@@ -117,6 +127,7 @@ export class CoderMemory implements CoderDelegationMemory {

117 127
  private readonly dreamThreshold: number;
118 128
  private key: Buffer | undefined;
119 129
  private cachedProjection: Projection | undefined;
130
  private readonly sync: EngramSyncQueue;
120 131
121 132
  constructor(options: CoderMemoryOptions = {}) {
122 133
    this.directory = options.directory ?? join(homedir(), ".openagents", "memory");

@@ -125,6 +136,26 @@ export class CoderMemory implements CoderDelegationMemory {

125 136
    this.projectScope = sanitizeScope(options.projectScope ?? `project:${process.cwd()}`);
126 137
    this.now = options.now ?? Date.now;
127 138
    this.dreamThreshold = options.dreamThreshold ?? DREAM_THRESHOLD;
139
    this.sync = new EngramSyncQueue(options.transport ?? new MemoryTransport());
140
  }
141
142
  /**
143
   * What sync is holding. A session can say it is behind rather than leaving
144
   * a reader to infer it from silence.
145
   */
146
  syncStatus(): SyncStatus {
147
    return this.sync.status();
148
  }
149
150
  /**
151
   * Hand everything queued to the transport.
152
   *
153
   * Called between turns, never inside one: the local ledger is already
154
   * authoritative by the time this runs, so a transport that is down costs
155
   * latency and nothing else.
156
   */
157
  async flush(): Promise<number> {
158
    return this.sync.drain();
128 159
  }
129 160
130 161
  /** The local signing key, created on first use. */

@@ -180,6 +211,8 @@ export class CoderMemory implements CoderDelegationMemory {

180 211
    );
181 212
    appendFileSync(this.ledgerPath, `${JSON.stringify(event)}\n`, { mode: 0o600 });
182 213
    this.cachedProjection = undefined;
214
    // Local-first: the engram is on disk and readable now. Sync catches up.
215
    this.sync.publish(event);
183 216
    return event;
184 217
  }
185 218

@@ -203,6 +236,8 @@ export class CoderMemory implements CoderDelegationMemory {

203 236
      sign,
204 237
    );
205 238
    appendFileSync(this.ledgerPath, `${JSON.stringify(event)}\n`, { mode: 0o600 });
239
    this.cachedProjection = undefined;
240
    this.sync.publish(event);
206 241
    return event;
207 242
  }
208 243
packages/openagents-cli/src/memory/index.ts modified +1

@@ -3,3 +3,4 @@ export * from "./engram.js";

3 3
export * from "./consolidation.js";
4 4
export * from "./subagent-memory.js";
5 5
export * from "./projection.js";
6
export * from "./sync.js";
packages/openagents-cli/src/memory/sync.ts added +245

@@ -0,0 +1,245 @@

1
// Vendored from packages/agent-experience-memory/src/sync.ts by scripts/vendor-memory.mjs — do not edit here.
2
// The drift guard (test/vendored-memory-drift.test.ts) fails when this copy
3
// no longer matches the canonical source.
4
import { Schema as S } from "effect";
5
6
import { computeEngramEventId, type EngramEvent } from "./engram.js";
7
8
/**
9
 * The engram sync seam (issue #222).
10
 *
11
 * The ledger is local and authoritative. Sync is how a copy of it reaches
12
 * somewhere else, and the whole design follows from one rule: **a turn never
13
 * waits for it, and never fails because of it.** A relay that is slow, down,
14
 * or gone must be indistinguishable from one that is fine, as far as the
15
 * caller is concerned — because memory that can break a conversation is worse
16
 * than memory that is briefly out of date.
17
 *
18
 * That gives the three properties the tests hold this to:
19
 *
20
 * 1. **Local-first.** `publish` records the engram as pending and returns
21
 *    immediately. The engram is already in the local ledger by then; sync is
22
 *    catching up, not gatekeeping.
23
 * 2. **Nothing is lost.** A failed publish stays queued and is retried. The
24
 *    queue only forgets an engram once a transport has acknowledged it, so a
25
 *    transport that is down for an hour costs an hour of latency and no
26
 *    engrams.
27
 * 3. **Degraded is a state, not an error.** The queue reports what it is
28
 *    holding and what failed last, so a caller can *say* it is behind rather
29
 *    than discovering it by silence.
30
 *
31
 * What is deliberately not here: relays, sockets, keys, encryption. A
32
 * transport is anything that satisfies `EngramTransport`, and the real Nostr
33
 * one lands behind this interface without any caller changing. The workspace
34
 * has a shared Nostr implementation (`nostr-effect`) that a real transport
35
 * should build on rather than reimplement.
36
 */
37
38
export const SYNC_SCHEMA_ID = "openagents.engram_sync.v1" as const;
39
40
/** Why a publish did not land. Distinct because they need distinct responses. */
41
export const SyncFailureReason = S.Literals([
42
  /** The transport could not be reached at all. Retry later, unchanged. */
43
  "unreachable",
44
  /** The transport reached and refused this engram. Retrying will not help. */
45
  "refused",
46
  /** The transport accepted the call and failed inside it. Retry later. */
47
  "failed",
48
]);
49
export type SyncFailureReason = typeof SyncFailureReason.Type;
50
51
export type PublishResult =
52
  | { readonly ok: true; readonly eventId: string }
53
  | { readonly ok: false; readonly reason: SyncFailureReason; readonly detail?: string };
54
55
/** A filter for what to fetch back. Absent fields mean "no constraint". */
56
export interface EngramFilter {
57
  readonly authors?: ReadonlyArray<string>;
58
  /** `d` tag values — engram slugs. */
59
  readonly slugs?: ReadonlyArray<string>;
60
  /** Unix seconds; inclusive. */
61
  readonly since?: number;
62
  readonly until?: number;
63
  readonly limit?: number;
64
}
65
66
/**
67
 * Whatever carries engrams somewhere else.
68
 *
69
 * Implementations must not throw: a transport that cannot answer returns a
70
 * failure, because a throw from here would reach the caller's turn and the
71
 * whole point is that it cannot.
72
 */
73
export interface EngramTransport {
74
  publish(event: EngramEvent): Promise<PublishResult>;
75
  fetch(filter: EngramFilter): Promise<ReadonlyArray<EngramEvent>>;
76
}
77
78
/** What the queue is holding, for a caller that wants to say so. */
79
export interface SyncStatus {
80
  readonly pending: number;
81
  readonly delivered: number;
82
  /** Engrams the transport refused outright. They are not retried. */
83
  readonly refused: number;
84
  readonly lastFailure?: { readonly reason: SyncFailureReason; readonly detail?: string };
85
}
86
87
const matches = (event: EngramEvent, filter: EngramFilter): boolean => {
88
  if (filter.authors !== undefined && !filter.authors.includes(event.pubkey)) return false;
89
  if (filter.since !== undefined && event.created_at < filter.since) return false;
90
  if (filter.until !== undefined && event.created_at > filter.until) return false;
91
  if (filter.slugs !== undefined) {
92
    const slug = event.tags.find((tag) => tag[0] === "d")?.[1];
93
    if (slug === undefined || !filter.slugs.includes(slug)) return false;
94
  }
95
  return true;
96
};
97
98
/**
99
 * A transport that keeps engrams in memory.
100
 *
101
 * The reference implementation and what the tests run against. It is also a
102
 * useful real transport for a single process that wants sync's shape without
103
 * a relay.
104
 */
105
export class MemoryTransport implements EngramTransport {
106
  private readonly events = new Map<string, EngramEvent>();
107
  /** Set to fail every publish, for degraded-mode tests. */
108
  reachable = true;
109
110
  publish(event: EngramEvent): Promise<PublishResult> {
111
    if (!this.reachable) {
112
      return Promise.resolve({ ok: false, reason: "unreachable", detail: "transport is down" });
113
    }
114
    if (event.id !== computeEngramEventId(event)) {
115
      return Promise.resolve({ ok: false, reason: "refused", detail: "event id does not verify" });
116
    }
117
    this.events.set(event.id, event);
118
    return Promise.resolve({ ok: true, eventId: event.id });
119
  }
120
121
  fetch(filter: EngramFilter): Promise<ReadonlyArray<EngramEvent>> {
122
    if (!this.reachable) return Promise.resolve([]);
123
    const found = [...this.events.values()]
124
      .filter((event) => matches(event, filter))
125
      .sort((left, right) =>
126
        left.created_at !== right.created_at
127
          ? left.created_at - right.created_at
128
          : left.id.localeCompare(right.id),
129
      );
130
    return Promise.resolve(filter.limit === undefined ? found : found.slice(0, filter.limit));
131
  }
132
133
  /** Everything the transport holds, for assertions. */
134
  stored(): ReadonlyArray<EngramEvent> {
135
    return [...this.events.values()];
136
  }
137
}
138
139
/**
140
 * The queue between the ledger and a transport.
141
 *
142
 * `publish` is synchronous from the caller's side: it enqueues and returns.
143
 * `drain` is what actually talks to the transport, and a host calls it
144
 * whenever it likes — after a turn, on a timer, at exit. Nothing about the
145
 * caller's turn depends on when that happens.
146
 */
147
export class EngramSyncQueue {
148
  private readonly queue: Array<EngramEvent> = [];
149
  private readonly deliveredIds = new Set<string>();
150
  private readonly refusedIds = new Set<string>();
151
  private lastFailure: SyncStatus["lastFailure"];
152
  private draining = false;
153
154
  constructor(private readonly transport: EngramTransport) {}
155
156
  /**
157
   * Enqueue an engram for delivery. Returns nothing to wait on.
158
   *
159
   * An engram already delivered or already refused is not enqueued twice, so
160
   * a caller that republishes its whole ledger costs one pass, not one
161
   * delivery per pass.
162
   */
163
  publish(event: EngramEvent): void {
164
    if (this.deliveredIds.has(event.id) || this.refusedIds.has(event.id)) return;
165
    if (this.queue.some((queued) => queued.id === event.id)) return;
166
    this.queue.push(event);
167
  }
168
169
  /**
170
   * Try to deliver everything queued.
171
   *
172
   * Returns how many landed. An unreachable or failing transport leaves the
173
   * engram queued for the next drain; a refusal is terminal, because retrying
174
   * something the transport has judged invalid only repeats the judgement.
175
   * Never throws — a transport that throws anyway is treated as failing.
176
   */
177
  async drain(): Promise<number> {
178
    if (this.draining) return 0;
179
    this.draining = true;
180
    try {
181
      let delivered = 0;
182
      // Copy: a publish during a drain lands in the next one rather than
183
      // mutating the array being walked.
184
      const attempting = [...this.queue];
185
      for (const event of attempting) {
186
        let result: PublishResult;
187
        try {
188
          result = await this.transport.publish(event);
189
        } catch (cause) {
190
          result = {
191
            ok: false,
192
            reason: "failed",
193
            detail: cause instanceof Error ? cause.message : String(cause),
194
          };
195
        }
196
        if (result.ok) {
197
          this.deliveredIds.add(event.id);
198
          this.removeFromQueue(event.id);
199
          delivered += 1;
200
          continue;
201
        }
202
        this.lastFailure = {
203
          reason: result.reason,
204
          ...(result.detail === undefined ? {} : { detail: result.detail }),
205
        };
206
        if (result.reason === "refused") {
207
          this.refusedIds.add(event.id);
208
          this.removeFromQueue(event.id);
209
        }
210
        // "unreachable" and "failed" stay queued for the next drain.
211
      }
212
      return delivered;
213
    } finally {
214
      this.draining = false;
215
    }
216
  }
217
218
  private removeFromQueue(eventId: string): void {
219
    const at = this.queue.findIndex((queued) => queued.id === eventId);
220
    if (at >= 0) this.queue.splice(at, 1);
221
  }
222
223
  /** Fetch from the transport. An unreachable transport yields nothing. */
224
  async fetch(filter: EngramFilter): Promise<ReadonlyArray<EngramEvent>> {
225
    try {
226
      return await this.transport.fetch(filter);
227
    } catch {
228
      return [];
229
    }
230
  }
231
232
  status(): SyncStatus {
233
    return {
234
      pending: this.queue.length,
235
      delivered: this.deliveredIds.size,
236
      refused: this.refusedIds.size,
237
      ...(this.lastFailure === undefined ? {} : { lastFailure: this.lastFailure }),
238
    };
239
  }
240
241
  /** Whether the queue is holding anything the transport has not taken. */
242
  behind(): boolean {
243
    return this.queue.length > 0;
244
  }
245
}
packages/openagents-cli/test/coder-memory.test.ts modified +41

@@ -4,6 +4,7 @@ import { join } from "node:path";

4 4
import { afterEach, describe, expect, it } from "vitest";
5 5
6 6
import { CoderMemory } from "../src/coder-memory.js";
7
import { MemoryTransport } from "../src/memory/index.js";
7 8
import { DelegateFleet, type DelegateEvent, type DelegateHarness } from "../src/coder-delegate.js";
8 9
import { CoderTaskRegistry } from "../src/coder-tasks.js";
9 10

@@ -232,3 +233,43 @@ describe("the dreaming pass", () => {

232 233
    expect(distilled.length).toBeGreaterThan(0);
233 234
  });
234 235
});
236
237
describe("memory sync", () => {
238
  it("queues every recorded engram and delivers it on flush", async () => {
239
    const dir = freshDir();
240
    const memory = memoryAt(dir);
241
    memory.record("note/one", "a finding worth keeping", "note-1");
242
243
    // Local-first: the engram is readable before anything is delivered.
244
    expect(memory.recall("note/one")).toBe("a finding worth keeping");
245
    expect(memory.syncStatus().pending).toBe(1);
246
    expect(memory.syncStatus().delivered).toBe(0);
247
248
    await memory.flush();
249
    expect(memory.syncStatus()).toMatchObject({ pending: 0, delivered: 1 });
250
  });
251
252
  it("keeps working when the transport is down, and loses nothing", async () => {
253
    const dir = freshDir();
254
    const transport = new MemoryTransport();
255
    transport.reachable = false;
256
    const memory = new CoderMemory({
257
      directory: dir,
258
      projectScope: "project:test",
259
      now: () => 1_756_000_000_000,
260
      dreamThreshold: Number.POSITIVE_INFINITY,
261
      transport,
262
    });
263
264
    memory.record("note/one", "recorded while the relay was gone", "note-1");
265
    expect(memory.recall("note/one")).toBe("recorded while the relay was gone");
266
    await memory.flush();
267
    expect(memory.syncStatus().pending).toBe(1);
268
    expect(memory.syncStatus().lastFailure?.reason).toBe("unreachable");
269
270
    transport.reachable = true;
271
    await memory.flush();
272
    expect(memory.syncStatus()).toMatchObject({ pending: 0, delivered: 1 });
273
    expect(transport.stored()).toHaveLength(1);
274
  });
275
});

This page updates live while a promote is in flight · changelog