Lens-core fold onto persist: drop legacy fallback path + identify persist API gaps

[emooreatx]

Summary

Lens-core still has direct write/delete/update sites against persist-owned tables that should fold onto Engine API once persist exposes the right primitives. This issue tracks the fold as a unit + names the persist asks needed to complete it.

The PoB §3.1 trajectory says "lens-core folds into agent" — that fold is much cleaner if every persist-substrate touch goes through Engine instead of direct SQL, because Engine survives the move and direct SQL doesn't.

Direct-touch inventory (lens-core only)

Site                                          Disposition         Persist API exists?
─────────────────────────────────────────────────────────────────────────────────────
Reads from accord_traces / accord_public_keys Stay on db_pool;    n/a
                                              owner relationship —
                                              no security boundary
                                              between lens-core and
                                              persist (same trust)

INSERT accord_traces                          DROP — legacy        n/a
   (accord_api.py:2369)                       fallback ingestion
                                              path; CIRISLENS_USE_
                                              PERSIST_ENGINE has
                                              been on in production
                                              since the v0.2.x
                                              cutover, this is dead
                                              code

INSERT accord_public_keys                     KEEP for now;        Engine.put_public_key
   (accord_api.py:2629                        document v0.4.0      exists (already mirrored
   register_public_key handler)               retirement timing    via federation_mirror).
                                                                   Direct INSERT stays as
                                                                   defense-in-depth until
                                                                   v0.4.0 retires the
                                                                   dual-read fallback in
                                                                   persist's verify path.

DELETE accord_traces                          NEEDS persist API    NO — no Engine
   (accord_api.py:2880,2891                                        primitive for DSAR-
   dsar_delete_traces handler)                                     style row deletion
                                                                   today. See §"Persist
                                                                   asks" below.

UPDATE accord_traces                          MOVE to lens-derived NO — and shouldn't
   (accord_api.py:3570,3636                   schema (cirislens_   exist on persist's
   set_trace_public_sample,                    derived.trace_       side. partner_access
   partner_access)                            visibility, joined   and public_sample are
                                              to accord_traces)    lens-policy decisions
                                                                   on persist's data; they
                                                                   shouldn't co-locate
                                                                   with substrate columns

INSERT accord_traces_mock                     dev-only path; not   n/a
   (accord_api.py:1323)                       in production

Recommended scope for the actual fold (separate PRs)

1. Drop the legacy fallback ingestion path. The if not _persist_engine_active() branch in receive_accord_events is dead code in production (Phase 2a flag has been on since v0.2.x cutover). Connectivity-event handling and mock-LLM filtering — which are NOT dead — need to be teased out and kept as their own routes/checks rather than living inside the legacy fallback. ~200-300 lines net delete after the connectivity/mock split.

2. Move partner_access + public_sample to a lens-derived schema. New table cirislens_derived.trace_visibility(trace_id, partner_access, public_sample, set_at, set_by) joined to accord_traces at query time. Removes lens's UPDATE on persist's substrate columns. ~150 lines + a migration.

3. Wait for persist's DSAR primitive (see asks below) before migrating DELETE.

4. Wait for persist v0.4.0 (drop dual-read fallback) before stopping accord_public_keys writes.

Asks for persist (file separately if/when they pick this up)

Ask 1: Engine.delete_traces_for_agent_id_hash(agent_id_hash) -> DeleteSummary

GDPR right-to-erasure (DSAR) primitive. Lens currently does direct DELETE on accord_traces; with trace_events post-cutover that path needs an Engine equivalent.

pub struct DeleteSummary {
    pub trace_events_deleted: u64,
    pub trace_llm_calls_deleted: u64,
    pub federation_keys_deleted: u64,  // if revoking the agent's key too
    pub deleted_at: DateTime<Utc>,
    // Audit: lens consumes this for its own DSAR ledger;
    //        persist returns the row counts, lens records
    //        the request envelope.
}

Mission-aligned because GDPR Article 17 is a federation-grade ethical requirement, not a lens-side convenience. Persist owns substrate row deletion; lens orchestrates the audit + signature verification of the DSAR request.

Ask 2: v0.4.0 retirement plan

What's the timeline for persist dropping the accord_public_keys dual-read fallback in Backend::lookup_public_key? Lens-core's direct INSERT into accord_public_keys can retire the same release. Coordinate.

Ask 3 (deferred until needed): bulk-iter primitive for analytical reads

Engine.iterate_trace_events(filter, callback) for streaming analytical work that doesn't fit the SQL-via-cirislens_reader pattern. Not blocking; the export script's asyncpg-cursor pattern works fine for now. Mention only if persist team wants to standardize.

What's NOT in this issue's scope

• Reads stay on db_pool. No security boundary between lens-core and persist; routing in-process analytical reads through cirislens_reader is plumbing without benefit. The peer/owner distinction matters: cirislens_reader is for OUT-OF-PROCESS consumers (export script, future federation peers), not lens-core itself.
• Lens-derived tables stay lens-owned. coherence_ratchet_alerts, pdma_events, wbd_deferrals, creator_ledger, sunset_ledger — these are lens's analytical output, not persist's substrate. They don't fold.

References

• PoB §3.1 — lens-core folds into agent
• v0.3.4 commit (5fcd550) — pin bump for deployment_profile wire-format compatibility
• v0.2.x cutover — Phase 2a flag flip that made the legacy ingestion path dead code
• CIRISPersist v0.4.0 roadmap — accord_public_keys dual-read retirement

[emooreatx]

v0.3.5 ships ASKs 1 + 3 → https://pypi.org/project/ciris-persist/0.3.5/

• `Engine.delete_traces_for_agent(agent_id_hash, include_federation_key=False)` — DSAR primitive. Always deletes trace_events + trace_llm_calls; `include_federation_key=True` cascades federation_keys + attestations + revocations. Idempotent. Persist owns substrate; lens owns the DSAR audit + signature verification.
• `Engine.fetch_trace_events_page(after_event_id=0, limit=1000, agent_id_hash=None)` — page-cursor read primitive. Caller orchestrates the cursor. For cross-process consumers (export script, future federation peers); lens-core analytical reads stay on `cirislens_reader` + direct SQL.

CHANGELOG: https://github.com/CIRISAI/CIRISPersist/blob/v0.3.5/CHANGELOG.md#035--2026-05-03

ASK 2 — v0.4.0 timing

Persist commits to dropping the `accord_public_keys` dual-read fallback in v0.4.0. Lens-core can drop its direct INSERT into `accord_public_keys` the same release. Coordinate timing — when lens flips, persist flips, both ship in the same window.

Trigger criteria for v0.4.0 (suggested):

1. Federation_keys carries equivalent or greater traffic than accord_public_keys for verify lookups. Today persist's `Backend::lookup_public_key` reads federation_keys first and falls back to accord_public_keys only on miss. Once the fallback's hit-rate drops to ~0 in production, the legacy table is shadowable. Suggested observation window: 2 weeks of stable lens 0.3.x + agent 2.7.9 traffic.

2. No production verify failures attributable to federation_keys lookups in that window. The dual-read has been carrying its weight as defense-in-depth; v0.4.0 retires it, so we want clean signals first.

3. Lens commits to dropping the direct INSERT into accord_public_keys — same release window, no flag-day across the two repos. Lens's `register_public_key` handler at `accord_api.py:2629` becomes a no-op or routes everything through `engine.put_public_key` (already wired via `federation_mirror.py` per the inventory).

Suggested cadence: aim for v0.4.0 ~2 weeks after agent 2.7.9 enters stable production. That gives both teams a real-traffic signal before retiring the fallback. Persist-side prep is small (drop the fallback branch in `PostgresBackend::lookup_public_key` + the equivalent in sqlite + memory; ~30 lines net delete + a CHANGELOG entry).

If lens's timeline slips or wants to land sooner, ping here — we can do v0.4.0 whenever both sides are ready, no fixed calendar lock.

ASK 3 status

Shipped as `Engine.fetch_trace_events_page`. Page-cursor shape per the lens#8 rationale — no callback shape across PyO3, caller drives the cursor.

The export script's existing asyncpg-cursor pattern still works fine; `fetch_trace_events_page` is for consumers that want typed Rust-shape rows or can't take cirislens_reader (e.g., out-of-process partner sites).

[emooreatx]

CIRISPersist v0.3.5 closes the asks; partial fold landed at 99359e8

ASK 1: Engine.delete_traces_for_agent — ✓ closed (lens commit 99359e8)

DSAR handler at accord_api.py:dsar_delete_traces now folds the
persist-owned half of the deletion:

• Persist-owned data (trace_events + trace_llm_calls):
  engine.delete_traces_for_agent(agent_id_hash, include_federation_key=False)
• Lens-owned legacy (accord_traces, accord_traces_mock): direct DELETE on db_pool
• Lens-derived (coherence_ratchet_alerts): direct DELETE on db_pool

Best-effort: persist-side delete failure logs a warning, doesn't roll
back the DSAR (lens-owned succeeded; partial deletion documented in
response).

Scope shift on the new persist call: per-agent_id_hash (broader
than lens's pre-fold per-(agent_id_hash, signature_key_id) scope).
Correct for GDPR Article 17 — data subject is the person, not the
key. Pre-cutover accord_traces retains the per-key scope; post-cutover
trace_events follows the agent-scoped contract.

include_federation_key=False — leaves the agent's signing key alive
so they can register fresh consent + start a new corpus.

ASK 2: v0.4.0 timing — confirmed in v0.3.5 release notes

Persist team posted: drop accord_public_keys dual-read in v0.4.0;
coordinate with lens's direct-INSERT drop. Lens-side action item
queued for v0.4.0 cut — not in scope for this commit.

ASK 3: Engine.fetch_trace_events_page — ✓ shipped, not yet consumed

Page-cursor read primitive available in v0.3.5. Lens has no immediate
consumer (existing analytical paths read via SQL on db_pool, which
works for owner-context). Will land when an analytical migration
needs cursor-style streaming.

What's still open under CIRISLens#8

• Drop the legacy fallback ingestion path in receive_accord_events.
  CIRISLENS_USE_PERSIST_ENGINE has been on in production since the
  v0.2.x cutover; the off-path's INSERT into accord_traces is dead
  code. ~200-300 lines after teasing connectivity-event handling and
  mock-LLM filtering out (those are NOT dead and need their own routes).
• Move partner_access + public_sample to lens-derived schema.
  New table cirislens_derived.trace_visibility joined to accord_traces
  at query time. Removes lens's UPDATE on persist's substrate columns.
• Drop accord_public_keys direct INSERT at v0.4.0 (paired with
  persist's dual-read retirement).

[emooreatx]

Correction on ASK 1 — DSAR fold reverted; persist needs a per-key API

The 99359e8 fold was wrong. Reverted at fbbd844.

The lens DSAR contract is per-(agent_id_hash, signature_key_id), and the signature_key_id filter is the authorization scope of the DSAR — not just an identity filter. A DSAR signed by key A is only authorized to delete traces signed by key A. If a key is lost, those traces stay undeletable by that key. That's the contract; it preserves the property that no one with any one valid key can delete traces that belong to a different agent instance using the same logical name with a different key.

CIRISPersist v0.3.5's Engine.delete_traces_for_agent(agent_id_hash, include_federation_key=False) is per-agent, not per-key — broader than the contract requires.

Filed CIRISPersist#15 asking for an Option<signature_key_id> parameter. Once that lands, the DSAR fold proceeds cleanly: trace_events + trace_llm_calls deletion goes through Engine with the per-key filter intact; lens-owned legacy (accord_traces, accord_traces_mock) and lens-derived (coherence_ratchet_alerts) deletes continue using direct DELETE.

Updated ASK status

• ASK 1 — Engine.delete_traces_for_agent: ✗ shape needs revision. CIRISPersist#15 tracks the per-key parameter add. Re-fold once the new API lands.
• ASK 2 — v0.4.0 timing: ✓ on persist's roadmap.
• ASK 3 — Engine.fetch_trace_events_page: ✓ shipped in v0.3.5; not yet consumed.

Pin stays at 0.3.5 — Engine.fetch_trace_events_page (ASK 3) and the deployment_profile work are still useful at this version even though ASK 1 needs another iteration.

[emooreatx]

v0.4.0 shipped → https://pypi.org/project/ciris-persist/0.4.0/

The schema-stabilization release. Three architectural commitments closed in one cut:

ASK 2 — accord_public_keys dual-read fallback retired ✅

`Backend::lookup_public_key` on postgres + memory + sqlite reads only from `federation_keys`. The v0.2.1 fallback path is gone. `sample_public_keys` diagnostic also queries federation_keys so the verify-unknown-key breadcrumb sample matches the actual lookup query.

The legacy `accord_public_keys` table stays in the schema for historical reads via `cirislens_reader` (V005 read-only role) but the verify path no longer touches it. v0.5.0 may drop the table itself once historical-reads consumers migrate.

Lens action: drop the direct INSERT into `accord_public_keys` (`accord_api.py:2629` register_public_key handler). The fold attempt that was reverted at `fbbd844` lands cleanly on v0.4.0.

Full verify surface for agent cutover ✅

Five new `Engine` verify methods so the agent can cut over to persist for runtime verify when brought in via lenscore:

```python

Full CompleteTrace verify with internal directory lookup

result = engine.verify_trace(complete_trace_json)

{"verified": True, "schema_version": "2.7.0"|"2.7.9"}

Hybrid verify with internal lookup_public_key (convenience wrapper)

result = engine.verify_hybrid_via_directory(
canonical_bytes, signature_key_id,
ed25519_sig_b64, ml_dsa_65_sig_b64,
policy="strict|soft_freshness|ed25519_fallback",
soft_freshness_window_seconds=..., row_age_seconds=...,
)

Federation directory row verify (verify-without-store)

engine.verify_signed_key_record(json, policy, ...)
engine.verify_signed_attestation(json, policy, ...)
engine.verify_signed_revocation(json, policy, ...)
```

The agent's runtime verify needs (CompleteTrace, peer-message envelopes, federation directory rows, ACK envelopes, arbitrary canonical bytes) all map onto Engine methods. No federation peer needs to call `ciris_crypto::HybridVerifier` directly — verify-via-persist is the single-source-of-truth (CIRISPersist#7 architectural closure repeated for the verify path).

Edge outbound queue (CIRISPersist#16) ✅

Companion deliverable — durable substrate for `CIRISEdge::send_durable()`. Lens uses ephemeral channels (`AccordEventsBatch` = Ephemeral; persist's AV-9 dedup recovers). Lens-side impact: none for now; relevant when lens-core folds into agent and federation gossip uses `Delivery::Durable` message types.

Bridge action

```
ciris-persist==0.3.6 → ciris-persist==0.4.0
```

Lens DSAR handler still uses the per-key contract from v0.3.6:

```python
engine.delete_traces_for_agent(agent_id_hash, signature_key_id)
```

Plus the lens-core fold tracked in this issue can now drop the direct INSERT into `accord_public_keys` cleanly.

Tests

177 lib + 22 integration green; clippy clean. Two legacy tests targeting accord_public_keys round-trip rewritten to use federation_keys (the renamed test `expired_keys_filtered` reflects the v0.4.0 model — federation revocations live in `federation_revocations`, a separate concern from key validity windows).

Threat model

AV-40 (outbound queue disk exhaustion) + AV-41 (spoofed in_reply_to ACK matching) added in `docs/THREAT_MODEL.md` §3.11.