fix(semantic): Re-evaluate stored vectors and entries when privacy policy changes#85
Open
mhiro2 wants to merge 5 commits into
Open
fix(semantic): Re-evaluate stored vectors and entries when privacy policy changes#85mhiro2 wants to merge 5 commits into
mhiro2 wants to merge 5 commits into
Conversation
…licy changes The semantic index trusted each entry's capture-time sensitivity forever: adding a regex_denylist / app_denylist rule (or toggling OTP detection) after an entry was embedded neither purged its stored vector nor stopped a rebuild from re-embedding the same forbidden text verbatim, because vector staleness was keyed on content_hash alone and the indexer only redacted rows already stored as Private. Track the embedding privacy policy explicitly and enforce it at both ends: - AppSettings::semantic_policy_hash() fingerprints the settings that decide what may reach the embedding model (regex_denylist, app_denylist, OTP detection, entry-size ceiling; order-insensitive over the rule lists). - semantic_index_meta gains a policy_hash column; SemanticIndexMeta carries it and includes it in is_compatible_with, so a policy edit invalidates the index like a model change: the worker purges stored vectors before the enabled / availability / battery guards (even with the index off or no embedder on the host — the no-embedder loop now watches settings), and semantic queries return nothing until the rebuild lands. - Before embedding, each pending entry is re-assessed against the current policy via SensitivityClassifier::assess_semantic_text, over both the raw body and the normalized projection (normalization folds case, which case-sensitive rules and detectors would miss). The stored verdict is a floor: re-assessed Secret/Blocked entries are refused, Private ones redacted, and only Public text embeds verbatim. - Refused entries get a semantic_exclusions tombstone (new table, migration 105) that also drops any leftover vector, keeps them out of the pending backlog until their content or the policy changes, and is cleared with the vectors on every rebuild. Migration 105 also rebuilds the meta table and deletes existing vectors outright: they were embedded under a policy whose fingerprint predates tracking, so they cannot be verified.
…ed payloads Follow-ups to the policy-hash re-indexing change, from review: - The pre-batch settings check left a window where a batch shaped by the previous policy could still embed and persist. embed_and_store now re-checks the settings watch right before the model call and again before persisting, dropping the whole batch on a mid-flight change (the policy-hash purge on the next wake remains the authoritative backstop for the un-closable model-call window). - The re-assessment only saw the raw plain text and normalized projection, missing rich-text markup and stored HTML/RTF/plain representation rows that the capture-time classifier scans — and an unparseable content_json silently skipped the raw-side scan. semantic_pending now recovers every persisted text projection, assess_semantic_texts applies the capture classifier's aggregate-size and per-payload scanning shape to all of them, and a body that cannot be parsed back is refused (fail closed). - The query path validated the policy hash before embedding, then searched without it (TOCTOU). semantic_search now re-checks semantic_index_meta.policy_hash inside the ranking SQL itself, and anti-joins semantic_exclusions, so a racing policy edit yields no rows within the same snapshot. - semantic_clear, semantic_exclude_batch, and the migration runner now follow the hard-delete WAL-scrub contract with a best-effort wal_checkpoint(TRUNCATE), so vectors erased for policy reasons do not linger in historical WAL frames.
Review follow-up: the watch-based mid-batch check and the query-side policy pin both raced the settings save itself, because the index metadata was only invalidated later by the background worker — an indexing batch could commit old-policy vectors after the save, and a search could validate against not-yet-updated metadata. Make the settings write the atomicity anchor: - A save whose semantic policy fingerprint differs from the stored settings erases the vectors, exclusion tombstones, and metadata row in the same transaction (both the plain and the revision-checked save paths), followed by the hard-delete WAL checkpoint. Any snapshot now either predates the save (old policy fully in force) or sees no vectors at all, which also makes the query-side policy-hash pin authoritative. - The indexer persists batches through a guarded upsert that re-checks the expected policy hash against semantic_index_meta inside the write transaction, aborting with a conflict when a policy-changing save erased it mid-flight — so a racing batch either commits before the save (whose erase then covers it) or drops with nothing persisted. The watch-based checks remain as cheap early exits that keep stale inputs away from the model. - FilePaths representations are decoded from their JSON storage form back to the newline-joined shape the capture-time classifier scanned, so anchored or quote-sensitive rules re-assess identically.
The ubuntu-latest Rust job started failing to build glib-sys: pkg-config could not find glib-2.0, a transitive dev dependency of libgtk-3-dev. cache-apt-pkgs-action restores previously downloaded package files instead of re-running apt's resolver, so a cache built against an older runner image can silently omit dev packages the image used to provide. Bump the action's version key (its cache-busting knob) in both ci.yaml and release.yaml so the package set is resolved fresh against the current image, and document the failure mode next to the knob.
Review follow-up closing the remaining resurrection races: the worker's rebuild ran semantic_clear and semantic_set_meta as separate transactions, so a pass still shaped under the previous settings could re-create the metadata row — with the old policy fingerprint — after a policy-changing save had erased it, quietly re-arming the guarded upsert and the query-side pin for the old policy. Exclusion tombstones had the same hole in the other direction: a stale refusal could re-insert tombstones the save had erased, hiding now-permitted entries from the backlog until the next rebuild. - semantic_rebuild_index_meta replaces the clear+set pair: one write transaction that validates the new metadata's policy fingerprint against the committed settings row and erases + re-records the index atomically, failing closed (Conflict) on a mismatch or an unreadable settings row. The reconcile treats the conflict as a stale pass and lets the settings-change wakeup retry with fresh settings. - Exclusion writes go through the same in-transaction policy guard as the vector upsert. - The guarded write transactions open IMMEDIATE so the guard's read and the writes share one writer snapshot, instead of a DEFERRED read-to-write upgrade that can fail with SQLITE_BUSY_SNAPSHOT under a concurrent commit and stall the backfill.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
regex_denylist/app_denylistrule (or enabling OTP detection) after an entry was embedded neither deleted its stored vector nor stopped a manual Rebuild from re-embedding the same forbidden text verbatimAppSettings::semantic_policy_hash) and store it with the index metadata, so any policy edit purges the stored vectors immediately — even while the index is disabled or no embedding model is available — and gates semantic queries until the rebuild landssemantic_exclusionstable, Private ones are redacted, and only Public text reaches the model verbatimpolicy_hashcolumn and deletes pre-existing vectors outright, since they were embedded under a policy whose fingerprint predates tracking; policy-driven erases follow the hard-delete WAL-scrub contractChanges
regex_denylist,app_denylist, OTP detection, and the entry-size ceiling; include it inSemanticIndexMeta::is_compatible_withso a policy edit invalidates the index like a model change, purged before the enabled / availability / battery guards (the no-embedder loop now watches settings too)SensitivityClassifierassessment API, treating the stored verdict as a floor: refused entries get asemantic_exclusionstombstone (also dropping any leftover vector) that keeps them out of the backlog until their content or the policy changescontent_jsoncannot be parsed back (fail closed)wal_checkpoint(TRUNCATE)) after policy purges, exclusions that dropped vectors, and applied migrationssemantic_index_metainside the write transaction, aborting when a policy-changing save erased it mid-flighttext/uri-list(FilePaths) representations back to the newline-joined shape the capture-time classifier scanned, so anchored or quote-sensitive rules re-assess identicallysemantic_rebuild_index_meta, a single write transaction validated against the committed settings row, so a pass shaped under previous settings can never resurrect the old policy fingerprint after a policy-changing save erased itSQLITE_BUSY_SNAPSHOTon deferred read-to-write upgrades