Skip to content

Skip per-entry range checks for leaves fully inside an extract range#1289

Closed
cberner wants to merge 1 commit into
masterfrom
claude/redb-extract-if-perf-5psncr
Closed

Skip per-entry range checks for leaves fully inside an extract range#1289
cberner wants to merge 1 commit into
masterfrom
claude/redb-extract-if-perf-5psncr

Conversation

@cberner

@cberner cberner commented Jul 4, 2026

Copy link
Copy Markdown
Owner

Summary

First of a series of PRs splitting the extract_if/retain optimizations into individually reviewable pieces, ordered by expected impact on fast storage (NVMe), where the remaining ~2x gap vs lmdb is CPU-bound rather than fsync-bound.

Each entry visited by extract_from_if settled its cursor end individually: re-activating the live cursor, re-reading the entry, and comparing its key against the other end's bound — even though every key in a leaf that lies entirely inside the range passes the same checks. This was the largest single CPU cost in the scan.

Now, when an entry settles, the leaf's furthest key is compared against the opposite bound once, and the number of entries ahead of the gap that share the verdict is remembered; their settles skip the cursor work entirely. Only advancing the gap by one entry may consume from this run; every other reposition (leaf flushes, parking the end, errors) clears it. The final leaf at the range boundary keeps the per-entry comparison.

Benchmark (lmdb_benchmark extract_if phase, 5M entries, file in memory to isolate CPU)

scan commit total
master ~225ms ~92ms ~320ms
this PR ~180ms ~92ms ~271ms
lmdb ~110ms median ~65ms ~175ms

Planned follow-up PRs (one at a time, in this order)

  1. this PR — per-leaf range check fast path
  2. Zero-copy splice sources (build replacement leaves directly from source pages)
  3. Batched page freeing at commit (one lock acquisition per batch instead of per page)
  4. Contiguous batch allocation for replacement leaves + coalesced commit writes (mainly benefits slower block storage; ~6x fewer write syscalls everywhere)

Testing

  • just test green (deny, fmt, clippy, full suite)
  • fuzz_redb 52k iterations, no findings
  • Existing extract/retain bound-semantics and double-ended iteration tests cover the fast path's correctness surface

🤖 Generated with Claude Code

https://claude.ai/code/session_019fyYXygUxss6KcMo5jmBnw

@codecov

codecov Bot commented Jul 4, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 90.19608% with 5 lines in your changes missing coverage. Please review.
✅ Project coverage is 90.15%. Comparing base (f4d86a0) to head (94b3e2b).

Files with missing lines Patch % Lines
src/tree_store/btree_cursor.rs 90.19% 5 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #1289      +/-   ##
==========================================
- Coverage   90.16%   90.15%   -0.02%     
==========================================
  Files          36       36              
  Lines       16161    16212      +51     
==========================================
+ Hits        14572    14616      +44     
- Misses       1589     1596       +7     
Files with missing lines Coverage Δ
src/tree_store/btree_cursor.rs 97.34% <90.19%> (-0.32%) ⬇️

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@cberner cberner force-pushed the claude/redb-extract-if-perf-5psncr branch from 9b84014 to a811708 Compare July 4, 2026 16:31
Each entry visited by extract_from_if settled its cursor end
individually: re-activating the live cursor, re-reading the entry, and
comparing its key against the other end's bound, even though every key
in a leaf that lies entirely inside the range passes the same checks.

When an entry settles, compare the leaf's furthest key against the
opposite bound once and remember how many entries ahead of the gap
share the verdict; their settles then skip the cursor work entirely.
Only advancing the gap by one entry may consume from this run, and
every other reposition (leaf flushes, parking the end, errors) clears
it, so the fast path never outlives the position it was computed for.
The final leaf at the range boundary keeps the per-entry comparison.

On the lmdb_benchmark extract_if phase (5M entries) with the file in
memory, the scan drops from about 225ms to 180ms; commit time is
unchanged.

Assisted-by: Claude Code
https://claude.ai/code/session_019fyYXygUxss6KcMo5jmBnw
@cberner cberner force-pushed the claude/redb-extract-if-perf-5psncr branch from cc33268 to 94b3e2b Compare July 4, 2026 17:51
@cberner cberner changed the title Optimize retain/extract_if bulk removals to match lmdb Skip per-entry range checks for leaves fully inside an extract range Jul 4, 2026
@cberner cberner closed this Jul 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant