Skip to content

Phase 2: detection engine v2 — archive/text/audio/video similarity + near-dupe correctness - #16

Open
zackkitzmiller wants to merge 2 commits into
feat/phase1-journaled-rollbackfrom
feat/phase2-detection-v2
Open

Phase 2: detection engine v2 — archive/text/audio/video similarity + near-dupe correctness#16
zackkitzmiller wants to merge 2 commits into
feat/phase1-journaled-rollbackfrom
feat/phase2-detection-v2

Conversation

@zackkitzmiller

Copy link
Copy Markdown
Member

Why

Detection was shallow: near-duplicates covered images only via an O(n²) sweep, hamming_distance silently computed wrong distances for mismatched hash lengths in release builds, the required zip↔extracted-folder matching didn't exist, and text/audio/video similarity was absent. Archive introspection also flattened entries by filename, so a/index.html and b/index.html clobbered each other.

Stacked on #15.

Changes

  • Archive ≡ folder (fingerprint/archive.rs): every zip entry is streamed through blake3 (no extraction) and matched against scanned files at the same relative paths under a common root. A full match emits DuplicateType::ArchiveMatch with the archive as the deletable side — the extracted tree is never the default deletion.
  • Text similarity (fingerprint/text.rs): 64-bit simhash over word 3-shingles, threshold 6 — catches lightly-edited copies (contract v1 vs v2). Byte-identical pairs are left to exact dedupe; tiny/huge files skipped.
  • Audio (fingerprint/audio.rs): chromaprint via fpcalc when on PATH — same recording across mp3/flac/bitrates at ≥90% bit-similarity. One log line + no-op otherwise.
  • Video (fingerprint/video_frame.rs): keyframe phash via ffmpeg when on PATH feeds videos into the existing perceptual pool — video↔video and video↔image matches for free. No-op otherwise.
  • Correctness: hamming_distance returns None on length mismatch (was debug_assert + wrong result in release).
  • Scale: pigeonhole banding (threshold+1 bands) generates near-dupe candidates from shared band buckets — provably identical results to the pairwise sweep, near-linear time.
  • extract_zip preserves entry relative paths under the temp dir.
  • TUI: dupe mode renders MATCH TYPE / WHY panels for the three new kinds; --dupes-only runs all detectors.

Test plan

  • cargo test — 366 passed (new: zip↔folder positive/negative/missing-entry fixtures, simhash edited-copy + unrelated-doc + identical-skip, banding-vs-bruteforce, mismatched-length regression, fpcalc similarity scoring, garbage-video none)
  • cargo clippy --all-targets — clean
  • Degradation: audio/video detectors are runtime-gated on fpcalc/ffmpeg presence with a single informational log

…ect near-dupe matching

- zip ↔ extracted-folder matching: every entry streamed through blake3
  and matched against on-disk files at the same relative paths; the
  archive surfaces as the deletable side of the duplicate set
- fuzzy text similarity: 64-bit simhash over word 3-shingles catches
  lightly-edited copies of the same document
- audio: chromaprint (fpcalc) acoustic fingerprints match the same
  recording across formats/bitrates; graceful no-op when not installed
- video: keyframe phash via ffmpeg joins videos into the image
  near-dupe pool (video↔video and video↔image); no-op without ffmpeg
- near-dupe correctness: mismatched hash lengths are 'not comparable'
  instead of a fabricated prefix distance (was a debug_assert only)
- near-dupe scale: pigeonhole banding (threshold+1 bands) replaces the
  O(n²) pairwise sweep with bucket candidates — identical results
- archive introspection now preserves entry paths when extracting
  (same-named files in different folders no longer clobber each other)
- TUI dupe mode explains the new match kinds (MATCH TYPE / WHY panels)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant