Skip to content

Fix lock-order inversion in MultimapTable::remove subtree collapse#1292

Merged
cberner merged 1 commit into
masterfrom
claude/abba-deadlock-multimap-kd93z3
Jul 5, 2026
Merged

Fix lock-order inversion in MultimapTable::remove subtree collapse#1292
cberner merged 1 commit into
masterfrom
claude/abba-deadlock-multimap-kd93z3

Conversation

@cberner

@cberner cberner commented Jul 5, 2026

Copy link
Copy Markdown
Owner

Problem

MultimapTable::remove()'s subtree→inline conversion locked the two page-tracking mutexes in the wrong order. When the collapsed subtree root is a committed page, it locked allocated_pages and then, while still holding it, locked freed_pages:

let mut allocated_pages = self.allocated_pages.lock().unwrap();       // A
if !self.page_allocator.free_if_uncommitted(new_root, &mut allocated_pages) {
    (*self.freed_pages).lock().unwrap().push(new_root);               // B, while holding A
}

Every other site takes these locks in the opposite order, freed_pagesallocated_pages: Btree::insert (freed_pages held while LeafBuilder::build locks allocated_pages), Btree::get_mut, MultimapValue::drop, and table removal in table_tree.rs.

All tables in one WriteTransaction share the same two Arc<Mutex>es, and using two tables from two threads is allowed (WriteTransaction: Sync). A concurrent insert on a sibling table holds freed_pages and then waits on allocated_pages inside LeafBuilder::build, so it can deadlock against the conversion holding allocated_pages and waiting on freed_pages — an ABBA cycle that permanently hangs the writer.

The inverted branch is reachable, not just theoretical: insert two small values plus one large value (the value-set is promoted to a subtree with a branch root), commit, then remove the large value. The branch collapses and leaves the untouched committed small-value leaf as the new root; it is below half a page, so it is rewritten back to inline storage and the committed leaf is freed via freed_pages — exactly the branch with the reversed lock order.

This is present in the released v4.1.0, not a master-only regression, so it's included in the changelog.

Fix

Lock freed_pages before allocated_pages here, matching every other site, and use the existing conditional_free() helper for the free-or-defer logic.

Testing

  • Added multimap_remove_collapses_committed_subtree_to_inline, which reproduces the exact collapse-to-committed-leaf scenario. I verified (via a temporary probe) that no existing test reached the committed-root branch and that this test does. The test is single-threaded, so it locks in correctness of the code path but cannot itself trigger the timing-dependent deadlock — the guarantee comes from the lock order now being globally consistent.
  • cargo fmt --check, cargo clippy --all-targets --all-features (no warnings), and cargo test --all-features all pass.

🤖 Generated with Claude Code

https://claude.ai/code/session_01YBCeHTcEZagQHCFMCcnqKn


Generated by Claude Code

MultimapTable::remove()'s subtree->inline conversion locked allocated_pages
and then, when freeing a committed root page, locked freed_pages -- the
reverse of the freed_pages -> allocated_pages order used everywhere else
(Btree::insert and get_mut, MultimapValue::drop, and table removal). All
tables in one WriteTransaction share the same two mutexes, and using two
tables from two threads is allowed since WriteTransaction is Sync. A
concurrent insert on another table holds freed_pages and then waits on
allocated_pages inside LeafBuilder::build, so it could deadlock against the
conversion holding allocated_pages and waiting on freed_pages -- an ABBA
cycle that permanently hangs the writer.

The inverted path is reachable, not just theoretical: two small values plus
one large value build a subtree with a branch root; after committing,
removing the large value collapses the branch and leaves the untouched
committed small-value leaf as the new root. That leaf is below half a page,
so it is rewritten back to inline storage and the committed leaf is freed
via freed_pages -- exactly the branch with the reversed lock order.

Lock freed_pages before allocated_pages here, matching every other site, and
use conditional_free() for the free-or-defer logic. Add a regression test
that drives the collapse through the committed-root branch, which no
existing test covered.

Assisted-by: Claude Code
Claude-Session: https://claude.ai/code/session_01YBCeHTcEZagQHCFMCcnqKn
@codecov

codecov Bot commented Jul 5, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 90.18%. Comparing base (8e04e95) to head (42088d4).
⚠️ Report is 1 commits behind head on master.

Additional details and impacted files
@@           Coverage Diff           @@
##           master    #1292   +/-   ##
=======================================
  Coverage   90.18%   90.18%           
=======================================
  Files          36       36           
  Lines       16227    16226    -1     
=======================================
  Hits        14634    14634           
+ Misses       1593     1592    -1     
Files with missing lines Coverage Δ
src/multimap_table.rs 93.64% <100.00%> (+0.26%) ⬆️

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@cberner cberner merged commit df0c318 into master Jul 5, 2026
8 checks passed
@cberner cberner deleted the claude/abba-deadlock-multimap-kd93z3 branch July 5, 2026 15:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant