Phase 2: detection engine v2 — archive/text/audio/video similarity + near-dupe correctness - #16
Open
zackkitzmiller wants to merge 2 commits into
Open
Conversation
…ect near-dupe matching - zip ↔ extracted-folder matching: every entry streamed through blake3 and matched against on-disk files at the same relative paths; the archive surfaces as the deletable side of the duplicate set - fuzzy text similarity: 64-bit simhash over word 3-shingles catches lightly-edited copies of the same document - audio: chromaprint (fpcalc) acoustic fingerprints match the same recording across formats/bitrates; graceful no-op when not installed - video: keyframe phash via ffmpeg joins videos into the image near-dupe pool (video↔video and video↔image); no-op without ffmpeg - near-dupe correctness: mismatched hash lengths are 'not comparable' instead of a fabricated prefix distance (was a debug_assert only) - near-dupe scale: pigeonhole banding (threshold+1 bands) replaces the O(n²) pairwise sweep with bucket candidates — identical results - archive introspection now preserves entry paths when extracting (same-named files in different folders no longer clobber each other) - TUI dupe mode explains the new match kinds (MATCH TYPE / WHY panels)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Detection was shallow: near-duplicates covered images only via an O(n²) sweep,
hamming_distancesilently computed wrong distances for mismatched hash lengths in release builds, the required zip↔extracted-folder matching didn't exist, and text/audio/video similarity was absent. Archive introspection also flattened entries by filename, soa/index.htmlandb/index.htmlclobbered each other.Stacked on #15.
Changes
fingerprint/archive.rs): every zip entry is streamed through blake3 (no extraction) and matched against scanned files at the same relative paths under a common root. A full match emitsDuplicateType::ArchiveMatchwith the archive as the deletable side — the extracted tree is never the default deletion.fingerprint/text.rs): 64-bit simhash over word 3-shingles, threshold 6 — catches lightly-edited copies (contract v1 vs v2). Byte-identical pairs are left to exact dedupe; tiny/huge files skipped.fingerprint/audio.rs): chromaprint viafpcalcwhen on PATH — same recording across mp3/flac/bitrates at ≥90% bit-similarity. One log line + no-op otherwise.fingerprint/video_frame.rs): keyframe phash viaffmpegwhen on PATH feeds videos into the existing perceptual pool — video↔video and video↔image matches for free. No-op otherwise.hamming_distancereturnsNoneon length mismatch (wasdebug_assert+ wrong result in release).--dupes-onlyruns all detectors.Test plan
cargo test— 366 passed (new: zip↔folder positive/negative/missing-entry fixtures, simhash edited-copy + unrelated-doc + identical-skip, banding-vs-bruteforce, mismatched-length regression, fpcalc similarity scoring, garbage-video none)cargo clippy --all-targets— cleanfpcalc/ffmpegpresence with a single informational log