Export consent-gated thread traces in the WEKA format #218

Closed AtlantisPleb opened this 2d ago 3 comments

Evidence

Shipped in the release at 8c53d09, promoted 2026-08-26.

1 pushes receipt

Outcome

An exporter from thread_events to the WEKA trace format used by AgentX: content tokenized into 64-token blocks and replaced with session-scoped chained hashes, preserving timing, context growth, prefix-reuse patterns, and (once the nested-thread ledger exists) sub-agent structure, while revealing no content — the same privacy construction the AgentX dataset uses.

Contract

  • Strictly consent-gated: only threads whose visibility tier permits it (THREAD-002), under the opt-in trace-licensing posture from the registry strategy. A corpus built from this exporter must be reproducible from a recorded thread-id set and code revision.
  • Output replays under the workbench's AIPerf harness (OpenAgentsInc/openagents bench lane) so lane comparisons run against traffic shaped like OpenAgents users, not only the SemiAnalysis corpus.
  • Publication of any corpus is a separate, explicit decision, not a side effect of exporting.

Acceptance

An exported OpenAgents corpus from consenting threads replays under the axis-1 harness with prefix-reuse characteristics measurably matching the source sessions.

Source

docs/2026-08-24-benchmark-workbench-agentx.md section 5; anonymization method per the AgentX methodology (chained-hash blocks).

  1. AtlantisPleb opened this issue 2d ago
  2. A AtlantisPleb Author 2d ago

    Exporter core landed on main in d808cf1: OpenAgents.Threads.WekaExport with the chained-hash 64-token-block anonymization, wider-than-dark consent gate, and byte-identical reproducibility for a fixed thread and salt (43 thread tests green). Remaining for acceptance: the AIPerf replay proof and any HTTP/corpus surface, which stays a separate decision.

  3. A AtlantisPleb Author 2d ago

    The replay half landed on monorepo main in 2a54d8f5aa: bench/replay_weka.py reconstructs each session's request shape from a weka-trace-v1 document — per turn, how many context blocks it carried and how many repeat blocks already seen earlier in that session. That repetition is the prefix reuse this format exists to measure, and the chained block hashes make it computable, which is the point of the construction: the tool reads hashes and never content, because there is no plaintext to read.

    It reports per session and corpus-wide (turns, total blocks, repeated blocks, reuse ratio, context growth), has a --json mode, skips a malformed document rather than crashing, and ships a synthetic fixture with a self-check pinning the reported ratio to a hand-computed expectation.

    So the acceptance is now evaluable, which it was not before. What remains to actually close this issue is running it against a real exported corpus from consenting threads and comparing the measured reuse against the source sessions — that needs real consenting threads and is a run rather than a build. Leaving open for that.

  4. AtlantisPleb closed this as completed in e57f5ea 1d ago
  5. A AtlantisPleb Author 1d ago

    Shipped in d808cf1 (exporter core) and e57f5ea (the format, the corpus, and the measurement).

    Consent gate — the contract's first clause. OpenAgents.Threads.WekaExport.export/2 refuses a dark thread with :consent_required, by struct and by id. The gate is per thread, not per subtree, because a consenting parent may spawn a narrower child (THREAD-003): a dark child produces no sub-agent entry, no request, no block count, and no id. corpus/2 takes a recorded thread-id set rather than running a query — a query is how a consent gate gets widened by accident — and records each refusal by id and reason instead of the trace. Proven in test/openagents/threads/weka_export_test.exs ("a dark thread refuses, by struct and by id", "a dark child of a consenting parent contributes nothing", "a named thread that has not consented is refused, not exported") and again at the operator surface in test/mix/tasks/openagents_weka_export_test.exs.

    The format. d808cf1 emitted a weka-trace-v1 document with an events array — plausible-looking, but not the format AIPerf reads, and per-event chunking destroyed the one property a corpus exists to carry. e57f5ea emits the real WEKA v1 session: block_size: 64, hash_id_scope: "local", and an ordered requests list of {t, type, model, in, out, hash_ids, api_time, think_time}, matching proxy_to_weka.py in the InferenceX reference. Only whole blocks are hashed, so in is exactly what a replay would send. Consenting children become subagent entries with fresh context in the shared id space — the nested-thread ledger THREAD-003 added, so the conditional clause is no longer conditional. Proven by "a consenting thread exports a replayable weka trace" and "a consenting child becomes a sub-agent entry".

    Anonymization. 64-token blocks replaced by session-salted chained SHA-256, remapped to session-local integers. The chain is load-bearing: a repeated block at a different context position is not cache-equivalent, and "the same block text at a different context position gets a different id" proves the two are told apart. "no transcript text survives the export" proves nothing readable leaves.

    Acceptance — prefix reuse measurably matching the source. prefix_reuse/1 reports reused leading blocks over all blocks, per agent. "the anonymized trace reports the source session's prefix reuse" computes the same statistic independently over the raw transcript — plain 64-token text blocks, no hashing, no id remap — and holds the two equal, plus the structural identity that every block of every call but the last is carried forward. An exporter that chunked per event, salted per request, or reordered would fail it.

    Reproducibility. "a corpus is reproducible from its recorded thread-id set and revision" rebuilds a corpus from its own requested_thread_ids and code_revision and compares bytes.

    Surface. mix openagents.weka.export --threads SET_FILE --out CORPUS_FILE. No route publishes a corpus; publication stays a separate, explicit decision, as the contract says.

    THREAD-002 records the new consumer; section 5 of docs/2026-08-24-benchmark-workbench-agentx.md records what shipped.

    Not in scope and not claimed: the replay itself runs in the OpenAgentsInc/openagents bench lane (#33), which now has a corpus to read. The trace family in the export ledger stays :blocked — that is the ATIF upload surface's missing read-back (#217), a different family from thread transcripts, and this change adds no route.

Sign in with GitHub to comment on this issue.