Skip to content

fix(semantic): Re-evaluate stored vectors and entries when privacy policy changes#85

Open
mhiro2 wants to merge 5 commits into
mainfrom
fix/semantic-index-policy-reindex
Open

fix(semantic): Re-evaluate stored vectors and entries when privacy policy changes#85
mhiro2 wants to merge 5 commits into
mainfrom
fix/semantic-index-policy-reindex

Conversation

@mhiro2

@mhiro2 mhiro2 commented Jul 23, 2026

Copy link
Copy Markdown
Owner

Summary

  • The semantic index froze each entry's capture-time sensitivity forever: adding a regex_denylist / app_denylist rule (or enabling OTP detection) after an entry was embedded neither deleted its stored vector nor stopped a manual Rebuild from re-embedding the same forbidden text verbatim
  • Fingerprint the embedding privacy policy (AppSettings::semantic_policy_hash) and store it with the index metadata, so any policy edit purges the stored vectors immediately — even while the index is disabled or no embedding model is available — and gates semantic queries until the rebuild lands
  • Re-assess every entry against the current policy right before embedding, over every persisted text projection (raw body, normalized text, rich-text markup, stored representations): re-assessed Secret/Blocked entries are refused and tombstoned in a new semantic_exclusions table, Private ones are redacted, and only Public text reaches the model verbatim
  • Make the settings write the atomicity anchor: a policy-changing save erases the index in its own transaction, the indexer's guarded upsert re-checks the policy hash inside the write transaction, and the ranking SQL re-checks it inside the query snapshot — so no interleaving persists or serves old-policy vectors past the settings commit
  • Migration 105 rebuilds the meta table with the policy_hash column and deletes pre-existing vectors outright, since they were embedded under a policy whose fingerprint predates tracking; policy-driven erases follow the hard-delete WAL-scrub contract

Changes

  • 3f063cf : fix(semantic): re-evaluate stored vectors and entries when privacy policy changes
    • Track the embedding privacy policy as a fingerprint over regex_denylist, app_denylist, OTP detection, and the entry-size ceiling; include it in SemanticIndexMeta::is_compatible_with so a policy edit invalidates the index like a model change, purged before the enabled / availability / battery guards (the no-embedder loop now watches settings too)
    • Re-assess pending entries at embed time via the new SensitivityClassifier assessment API, treating the stored verdict as a floor: refused entries get a semantic_exclusions tombstone (also dropping any leftover vector) that keeps them out of the backlog until their content or the policy changes
    • Add regression tests covering the three review scenarios (a new rule erases an already-embedded Public row's vector, Rebuild never hands the raw text to the embedder, and the purge still runs with no embedding model available); update ARCHITECTURE.md and docs/privacy.md to describe the actual lifecycle
  • cc87c58 : fix(semantic): close policy races and widen re-assessment to all stored payloads
    • Drop an in-flight batch when the settings watch reports a change before the model call or before persisting, so vectors shaped by the previous policy never become durable
    • Widen the re-assessment to every persisted text projection — rich-text markup and stored HTML/RTF/plain representation rows included — with the capture classifier's aggregate-size shape, and refuse bodies whose content_json cannot be parsed back (fail closed)
    • Pin the semantic ranking SQL to the validated policy fingerprint inside its own snapshot, anti-join the exclusion tombstones in search, and follow the hard-delete WAL-scrub contract (wal_checkpoint(TRUNCATE)) after policy purges, exclusions that dropped vectors, and applied migrations
  • a794315 : fix(semantic): anchor policy/index atomicity on the settings write
    • Erase the semantic index (vectors, tombstones, metadata) in the same transaction as any settings save whose policy fingerprint changed, on both the plain and revision-checked save paths, so every snapshot either predates the save or sees no vectors — closing the races the watch checks and query pin could not close alone
    • Persist indexing batches through a guarded upsert that re-checks the expected policy hash against semantic_index_meta inside the write transaction, aborting when a policy-changing save erased it mid-flight
    • Decode text/uri-list (FilePaths) representations back to the newline-joined shape the capture-time classifier scanned, so anchored or quote-sensitive rules re-assess identically
  • 0f310f6 : ci: bust the stale Linux apt package cache
    • Bump cache-apt-pkgs-action's version key in ci.yaml and release.yaml so the Linux package set is resolved fresh against the updated ubuntu-latest image, fixing the glib-sys / glib-2.0 build failure caused by a stale cached package set omitting transitive dev dependencies
  • 6faa98f : fix(semantic): make the rebuild and exclusion writes policy-guarded too
    • Replace the worker's separate clear + set-metadata calls with semantic_rebuild_index_meta, a single write transaction validated against the committed settings row, so a pass shaped under previous settings can never resurrect the old policy fingerprint after a policy-changing save erased it
    • Apply the same in-transaction policy guard to exclusion tombstone writes, and open the guarded write transactions IMMEDIATE to avoid SQLITE_BUSY_SNAPSHOT on deferred read-to-write upgrades

…licy changes

The semantic index trusted each entry's capture-time sensitivity forever:
adding a regex_denylist / app_denylist rule (or toggling OTP detection)
after an entry was embedded neither purged its stored vector nor stopped a
rebuild from re-embedding the same forbidden text verbatim, because vector
staleness was keyed on content_hash alone and the indexer only redacted
rows already stored as Private.

Track the embedding privacy policy explicitly and enforce it at both ends:

- AppSettings::semantic_policy_hash() fingerprints the settings that decide
  what may reach the embedding model (regex_denylist, app_denylist, OTP
  detection, entry-size ceiling; order-insensitive over the rule lists).
- semantic_index_meta gains a policy_hash column; SemanticIndexMeta carries
  it and includes it in is_compatible_with, so a policy edit invalidates
  the index like a model change: the worker purges stored vectors before
  the enabled / availability / battery guards (even with the index off or
  no embedder on the host — the no-embedder loop now watches settings), and
  semantic queries return nothing until the rebuild lands.
- Before embedding, each pending entry is re-assessed against the current
  policy via SensitivityClassifier::assess_semantic_text, over both the raw
  body and the normalized projection (normalization folds case, which
  case-sensitive rules and detectors would miss). The stored verdict is a
  floor: re-assessed Secret/Blocked entries are refused, Private ones
  redacted, and only Public text embeds verbatim.
- Refused entries get a semantic_exclusions tombstone (new table, migration
  105) that also drops any leftover vector, keeps them out of the pending
  backlog until their content or the policy changes, and is cleared with
  the vectors on every rebuild. Migration 105 also rebuilds the meta table
  and deletes existing vectors outright: they were embedded under a policy
  whose fingerprint predates tracking, so they cannot be verified.
@mhiro2 mhiro2 self-assigned this Jul 23, 2026
mhiro2 added 4 commits July 24, 2026 22:06
…ed payloads

Follow-ups to the policy-hash re-indexing change, from review:

- The pre-batch settings check left a window where a batch shaped by the
  previous policy could still embed and persist. embed_and_store now
  re-checks the settings watch right before the model call and again
  before persisting, dropping the whole batch on a mid-flight change
  (the policy-hash purge on the next wake remains the authoritative
  backstop for the un-closable model-call window).
- The re-assessment only saw the raw plain text and normalized
  projection, missing rich-text markup and stored HTML/RTF/plain
  representation rows that the capture-time classifier scans — and an
  unparseable content_json silently skipped the raw-side scan.
  semantic_pending now recovers every persisted text projection,
  assess_semantic_texts applies the capture classifier's aggregate-size
  and per-payload scanning shape to all of them, and a body that cannot
  be parsed back is refused (fail closed).
- The query path validated the policy hash before embedding, then
  searched without it (TOCTOU). semantic_search now re-checks
  semantic_index_meta.policy_hash inside the ranking SQL itself, and
  anti-joins semantic_exclusions, so a racing policy edit yields no
  rows within the same snapshot.
- semantic_clear, semantic_exclude_batch, and the migration runner now
  follow the hard-delete WAL-scrub contract with a best-effort
  wal_checkpoint(TRUNCATE), so vectors erased for policy reasons do not
  linger in historical WAL frames.
Review follow-up: the watch-based mid-batch check and the query-side
policy pin both raced the settings save itself, because the index
metadata was only invalidated later by the background worker — an
indexing batch could commit old-policy vectors after the save, and a
search could validate against not-yet-updated metadata.

Make the settings write the atomicity anchor:

- A save whose semantic policy fingerprint differs from the stored
  settings erases the vectors, exclusion tombstones, and metadata row
  in the same transaction (both the plain and the revision-checked
  save paths), followed by the hard-delete WAL checkpoint. Any
  snapshot now either predates the save (old policy fully in force)
  or sees no vectors at all, which also makes the query-side
  policy-hash pin authoritative.
- The indexer persists batches through a guarded upsert that
  re-checks the expected policy hash against semantic_index_meta
  inside the write transaction, aborting with a conflict when a
  policy-changing save erased it mid-flight — so a racing batch
  either commits before the save (whose erase then covers it) or
  drops with nothing persisted. The watch-based checks remain as
  cheap early exits that keep stale inputs away from the model.
- FilePaths representations are decoded from their JSON storage form
  back to the newline-joined shape the capture-time classifier
  scanned, so anchored or quote-sensitive rules re-assess identically.
The ubuntu-latest Rust job started failing to build glib-sys: pkg-config
could not find glib-2.0, a transitive dev dependency of libgtk-3-dev.
cache-apt-pkgs-action restores previously downloaded package files
instead of re-running apt's resolver, so a cache built against an older
runner image can silently omit dev packages the image used to provide.
Bump the action's version key (its cache-busting knob) in both ci.yaml
and release.yaml so the package set is resolved fresh against the
current image, and document the failure mode next to the knob.
Review follow-up closing the remaining resurrection races: the worker's
rebuild ran semantic_clear and semantic_set_meta as separate
transactions, so a pass still shaped under the previous settings could
re-create the metadata row — with the old policy fingerprint — after a
policy-changing save had erased it, quietly re-arming the guarded
upsert and the query-side pin for the old policy. Exclusion tombstones
had the same hole in the other direction: a stale refusal could
re-insert tombstones the save had erased, hiding now-permitted entries
from the backlog until the next rebuild.

- semantic_rebuild_index_meta replaces the clear+set pair: one write
  transaction that validates the new metadata's policy fingerprint
  against the committed settings row and erases + re-records the index
  atomically, failing closed (Conflict) on a mismatch or an unreadable
  settings row. The reconcile treats the conflict as a stale pass and
  lets the settings-change wakeup retry with fresh settings.
- Exclusion writes go through the same in-transaction policy guard as
  the vector upsert.
- The guarded write transactions open IMMEDIATE so the guard's read and
  the writes share one writer snapshot, instead of a DEFERRED
  read-to-write upgrade that can fail with SQLITE_BUSY_SNAPSHOT under a
  concurrent commit and stall the backfill.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant