Skip to content

feat: replace SHA-256 with xxHash3-64 for file hashing - #372

Merged
cebarks merged 3 commits into
mainfrom
worktree-xxhash-convoy
Jul 31, 2026
Merged

feat: replace SHA-256 with xxHash3-64 for file hashing#372
cebarks merged 3 commits into
mainfrom
worktree-xxhash-convoy

Conversation

@cebarks

@cebarks cebarks commented Jul 30, 2026

Copy link
Copy Markdown
Owner

Summary

  • Replaces sha2 crate with xxhash-rust (xxh3 feature) for all file hashing — ~15x faster across all file sizes
  • Swaps compute_file_hash, compute_hash, and HashingWriter in src/spt/mods.rs from SHA-256 to xxHash3-64
  • Adds DB migration (022) to clear stale SHA-256 hashes so they get lazily recomputed
  • Hash output changes from 64-char hex to 16-char hex; all consumers treat hashes as opaque strings

Convoy catalog generation, integrity checks, mod install/update, and backup/restore all benefit. Benchmark results (release mode, 20 iterations avg):

Size SHA-256 xxHash3-64 Speedup
1 KB 528 ns 59 ns 8.9x
64 KB 32 µs 1.65 µs 19.5x
1 MB 424 µs 24.9 µs 17.0x
10 MB 4.27 ms 264 µs 16.1x
50 MB 21.5 ms 1.40 ms 15.3x

Test plan

  • All 825+ existing tests pass
  • Verified mod install produces 16-char xxHash3-64 hashes in DB
  • Verified integrity check detects tampered files using new hash algorithm
  • Migration clears old hashes; existing code handles NULL hashes gracefully

Implemented with the help of Claude Code

cebarks and others added 3 commits July 30, 2026 05:11
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@smashedbanan

Copy link
Copy Markdown
Contributor

sick benchmark results

@cebarks
cebarks merged commit 672224c into main Jul 31, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants