Post-2.7.9 QA-traffic export pipeline for RATCHET validation corpus

[emooreatx]

Context

RATCHET (research / validation platform; not a production evaluator) needs a fixed, full-fidelity corpus of post-2.7.9 production traffic to calibrate detector math against real federation behavior, construct Sybil-archetype red-team fixtures against a known-honest baseline, and validate ρ_critical = 0.43 + k_eff predictions on production data rather than only physical-analog (GPU strain array) substrates.

CIRISLens hot-path scoring continues to be the production evaluator. RATCHET stays at corpus-snapshot scope; no live read access requested, no production-API changes asked for.

Trace standard: CIRISAgent/FSD/TRACE_WIRE_FORMAT.md @ v2.7.9-stable (PR #714).
Persist substrate: already shipped (trace_events + trace_llm_calls live; persist v0.3.0 pinned via 45a63e2).
Lens-side overlay: FSD/trace_format_specification.md (per dab6df6).

What this repo needs to do

Ship: post-2.7.9 QA-traffic snapshot pipeline

Periodic JSONL export filtered to trace_schema_version = "2.7.9":

• cirislens.trace_events — full row, signed bytes preserved
• cirislens.trace_llm_calls — joined on canonical parent linkage (agent_id_hash, trace_id, thought_id, parent_event_type, parent_attempt_index)
• cirislens.accord_public_keys — for agent_id_hash → signature_key_id resolution

Frozen accord_traces not needed (pre-cutover history isn't 2.7.9-shaped).

Scoping

Dimension	Value	Why
Trace levels	detailed + full_traces	generic is content-free; Layer 1 reasoning-topology features need structured data payloads
Scrubbing	Post-scrub, same envelope shape as release/data_scrubbed_v1/	RATCHET does not need pre-scrub content for the H1–H7 detectors
Wire version filter	Strictly "2.7.9"	Pre-2.7.9 has nullable parent_event_type / parent_attempt_index and envelope-only agent_id_hash, which degrades H3 (shared LLM infra) + H7 (timing coincidence) detectors
Traffic subset	QA-evaluation traces (theology / politics / technology / history per release/README.md)	High prompt-variance reveals constraint-space topology; honest agents produce varied reasoning patterns, Sybils stylize/lockstep
Format	JSONL per table, signed bytes preserved	Matches existing release/data_scrubbed_v1/ precedent; lets RATCHET re-verify provenance

Authorization

This is a federation-trust question, not a technical question. The public release/data_scrubbed_v1/ corpus already has a defined scrubbing + signing posture; if QA traffic is already in scope for that pipeline, the export is just a periodic snapshot of that pipeline's output with the version filter applied. If QA traffic is out of scope for the public release pipeline, please flag — that's a separate decision before this issue progresses.

Cadence

• First snapshot: ~2-4 weeks after 2.7.9 cutover begins (allows corpus depth to accumulate)
• Re-export quarterly, or at lens-team discretion

Non-goals

• No changes to persistence schema (V001 is correct as-is; AV-9 closure holds)
• No changes to wire format (locked at v2.7.9-stable)
• No live API access for RATCHET
• No commitment from RATCHET to publish the corpus onward — separate federation-trust review if that ever comes up

Open questions for the lens team

1. Where does the export land? (S3 bucket / lens-published distribution / separate release branch?)
2. What authorization gate? (Public per existing release/ posture, or authenticated consumer list?)
3. Is QA traffic in scope for the existing release pipeline today? If yes, this is mostly a filter + cadence decision. If no, the prerequisite is bringing QA traffic into the public-release scrubbing path.

References

• Trace standard (locked): CIRISAgent/FSD/TRACE_WIRE_FORMAT.md @ feat/trace-wire-format-712 HEAD; tag target v2.7.9-stable
• Persist V001 dedup tuple (canonical): CIRISPersist migrations/postgres/lens/V001__trace_events.sql
• Lens-side trace overlay FSD: FSD/trace_format_specification.md
• Public corpus precedent: CIRISAgent/release/data_scrubbed_v1/
• Federation threat model interface contract: CIRISVerify/docs/FEDERATION_THREAT_MODEL.md §2.4 (N_eff) + §6 (F-AV catalog) + §10 (RATCHET interface)
• Vendored threat-model snapshots in RATCHET: RATCHET/FEDERATION_THREAT_MODELS/ (snapshot 2026-05-02)