Inconsistent base_url scrubbing + correlated prompt_hash null pattern across LLM_CALL rows

[emooreatx]

Context

First post-2.7.9 export to RATCHET (per #4) shows trace_llm_calls.base_url scrubbed on ~56% of rows and intact on ~44%, with a perfectly-correlated null pattern on prompt_hash. Suggests two distinct ingestion paths with different scrubbing/enrichment posture.

Export: release_v2/data/lens_export_v2_fixed_2026-05-03T04-00-04Z/
Rows examined: 3,615

Finding

base_url value	rows	prompt_hash populated	top handlers
[URL] (scrubbed)	2,049 (56.7%)	100%	BaseDSDMA, ActionSelectionPDMAEvaluator, entropy_conscience, coherence_conscience, CSDMAEvaluator
https://api.deepinfra.com/v1/openai (intact)	1,566 (43.3%)	0%	BaseDSDMA, ActionSelectionPDMAEvaluator, entropy_conscience, coherence_conscience, CSDMAEvaluator

Same handlers in both subsets — not a per-handler scrubbing rule. Two real anomalies:

1. Provider URL leaks on 1,566 rows. The deployment's LLM provider identity (DeepInfra in this corpus) is exposed in the public-release export. Privacy posture probably doesn't intend this.
2. prompt_hash is null exactly when base_url is intact. The scrubber that strips base_url → [URL] also computes prompt_hash. The path that leaves base_url intact does not compute prompt_hash. Two different ingestion / enrichment paths.

Additionally: in the scrubbed-URL subset, prompt_hash value is sometimes literally "[IDENTIFIER]" (a placeholder, not a hash). Sample:

fa9a996f5730178e51ea54e6ee17b74957a82d64b35b23c21e0171b90f1e493b   ← real sha256
7679d565a9af6980acf72b177b849d9d669200da1eea44945913ad79e2ad43ba   ← real sha256
fad44638f46d46459a7c7035fb5936326013429c7383ccda1ae0ab683b348525   ← real sha256
94afdeb839bd61c1c74aeab136f650c7ca923ddadf614c35c50ec94e8a30f383   ← real sha256
[IDENTIFIER]                                                       ← placeholder

prompt_hash is a content fingerprint, not PII — replacing it with [IDENTIFIER] defeats its purpose (downstream comparison / dedup) without any privacy gain.

Why this matters for RATCHET

H3 (shared LLM infrastructure detection) is one of the seven Sybil hypotheses the federation evaluator uses. The detector clusters agents by (model, base_url, parent-topology) — agents that fall in the same dense cluster at high frequency are evidence of shared LLM infrastructure under different pseudonyms. With base_url scrubbed inconsistently:

• The 56.7% scrubbed subset is useless for clustering on base_url (all rows hash to [URL]).
• The 43.3% intact subset works for clustering — but only includes ~44% of the corpus.
• The split is per-row, not per-agent or per-handler, so we can't even reliably partition the corpus.

prompt_hash similarly: in the scrubbed subset where prompt text is stripped, prompt_hash is the only signal of prompt content fingerprint. Null on 43.3% halves the cross-call comparability.

What this repo needs to do

1. Identify the two ingestion paths. Likely candidates:

   • Old vs new persist path (pre-cutover legacy ingestion vs post-cutover engine.receive_and_persist)
   • Two different scrubber versions running in parallel during a v1↔v2 shadow window (per R3_5_DIVERGENCE_AUDIT.md)
   • Different middlewares applying scrubbing on different routes

2. Decide consistent posture:

   • base_url: scrub everywhere (matches public-release privacy intent), or keep everywhere (matches research-utility intent for H3 detector)? Pick one.
   • prompt_hash: always populate (it's a fingerprint, not PII). Never replace with [IDENTIFIER].

3. Re-export the QA corpus once paths are unified.

Non-blocking for RATCHET

H3 detector calibration on the current corpus will be at degraded power (~44% of rows usable for base_url clustering). Workaround documented in RATCHET's evaluation pipeline; filing this so the next export window has unified scrubbing.

References

• Adjacent: Post-2.7.9 QA-traffic export pipeline for RATCHET validation corpus #4 (RATCHET export pipeline)
• Scrubber promotion gate: RATCHET/LENS/R3_5_DIVERGENCE_AUDIT.md
• Privacy boundary spec: CIRISLens/docs/THREAT_MODEL.md §1 ("Privacy boundary by trace tier")

[emooreatx]

Investigation: split is trace_level-driven, not two ingestion paths

Tighter look at the production data:

SELECT
  CASE WHEN te.payload->>'base_url' = '[URL]' THEN 'scrubbed'
       ELSE 'intact' END AS base_url_state,
  te.trace_level, te.schema_version, COUNT(*)
FROM cirislens.trace_events te
WHERE te.event_type = 'LLM_CALL' AND te.ts > NOW() - INTERVAL '2 days'
GROUP BY 1,2,3 ORDER BY COUNT(*) DESC;

 base_url_state | trace_level | schema_version | count
----------------+-------------+----------------+-------
 intact         | generic     | 2.7.9          |  1769
 intact         | generic     | 2.7.0          |  3000
 scrubbed       | detailed    | 2.7.0          |  1483
 scrubbed       | detailed    | 2.7.9          |   970
 intact         | generic     | 2.7.9          |   396

Detailed → scrubbed. Generic → intact. Same code path, level-conditional behavior. The "two ingestion paths" framing was wrong; it's one path with the lens scrubber callback bypassed at generic per persist's v0.2.x design (generic traces are content-free → scrubber bypassed → metadata fields like base_url survive intact).

prompt_hash null pattern: natural consequence of generic tier — no prompt text → no sha256 to compute. Not a bug.

What's actually broken — two real items

1. Privacy posture inversion at generic tier

The lens FSD §1 says generic is the public-dashboard-safe tier. But:

• base_url (LLM provider identity — DeepInfra in this corpus) is unscrubbed at generic.
• service_name, model, handler_name, prompt_bytes, cost_usd all survive at generic.

Generic should have the most sanitized metadata, not the least. The current behaviour ships operational identity (which provider, which model, cost-per-call) in the public-dashboard tier.

This is a lens design decision, not strictly a bug. Two paths:

• Tighten generic: extend the scrubber callback to run a metadata-stripping pass at generic too (base_url → [URL], prompt_bytes → null, cost_usd → bucketed).
• Document it: explicitly accept that base_url is operational metadata, not PII, and adjust the FSD's privacy-tier description to match what's actually shipped.

H3 (shared LLM infrastructure) clustering on base_url works on the unscrubbed subset, so RATCHET prefers option 1 (tighten) only if it doesn't break their detector — they need base_url clusterable, just not in the public-dashboard view.

2. [IDENTIFIER] over-scrub on agent_id_hash

Fresh finding from the same investigation:

SELECT agent_id_hash, COUNT(*) FROM cirislens.trace_events
WHERE event_type='LLM_CALL' AND ts > NOW() - INTERVAL '2 days'
  AND agent_id_hash NOT SIMILAR TO '[a-f0-9]{16}|unknown'
GROUP BY 1;

  agent_id_hash | count
----------------+-------
  [IDENTIFIER]  |  534

534 rows have agent_id_hash='[IDENTIFIER]' — a placeholder string, not a hex hash. [IDENTIFIER] isn't in the standard pii_scrubber.py substitution table ([PERSON_*], [ORG_*], [EMAIL], [PHONE], [IP_ADDRESS], [URL], [SSN], [CREDIT_CARD]). So either:

• cirislens-core's Rust scrubber has a stricter regex that classifies hex hashes as identifiers and replaces them.
• The agent itself emits [IDENTIFIER] (different bug, would show up at the agent side).

agent_id_hash is a stable opaque identifier required by AV-9 dedup; replacing it with a placeholder defeats the structural binding without any privacy gain. Investigation TBD on whether this is in cirislens-core/src/validation/ or upstream.

Upstream relationship

• The base_url leakage IS the design (tier-conditional scrubber bypass) — needs a lens FSD clarification + maybe a generic-tier metadata-strip extension.
• The [IDENTIFIER] over-scrub is a real bug — investigation continuing on lens side.
• Generic-tier identity stripping (agent_id_hash="unknown") is a separate agent-side bug, filed at CIRISAgent#716.

Keeping this issue open as the umbrella for the lens-side action items (privacy posture decision + scrubber over-scrub investigation).