Fix lock-order inversion in MultimapTable::remove subtree collapse#1292
Merged
Conversation
MultimapTable::remove()'s subtree->inline conversion locked allocated_pages and then, when freeing a committed root page, locked freed_pages -- the reverse of the freed_pages -> allocated_pages order used everywhere else (Btree::insert and get_mut, MultimapValue::drop, and table removal). All tables in one WriteTransaction share the same two mutexes, and using two tables from two threads is allowed since WriteTransaction is Sync. A concurrent insert on another table holds freed_pages and then waits on allocated_pages inside LeafBuilder::build, so it could deadlock against the conversion holding allocated_pages and waiting on freed_pages -- an ABBA cycle that permanently hangs the writer. The inverted path is reachable, not just theoretical: two small values plus one large value build a subtree with a branch root; after committing, removing the large value collapses the branch and leaves the untouched committed small-value leaf as the new root. That leaf is below half a page, so it is rewritten back to inline storage and the committed leaf is freed via freed_pages -- exactly the branch with the reversed lock order. Lock freed_pages before allocated_pages here, matching every other site, and use conditional_free() for the free-or-defer logic. Add a regression test that drives the collapse through the committed-root branch, which no existing test covered. Assisted-by: Claude Code Claude-Session: https://claude.ai/code/session_01YBCeHTcEZagQHCFMCcnqKn
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #1292 +/- ##
=======================================
Coverage 90.18% 90.18%
=======================================
Files 36 36
Lines 16227 16226 -1
=======================================
Hits 14634 14634
+ Misses 1593 1592 -1
🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
MultimapTable::remove()'s subtree→inline conversion locked the two page-tracking mutexes in the wrong order. When the collapsed subtree root is a committed page, it lockedallocated_pagesand then, while still holding it, lockedfreed_pages:Every other site takes these locks in the opposite order,
freed_pages→allocated_pages:Btree::insert(freed_pagesheld whileLeafBuilder::buildlocksallocated_pages),Btree::get_mut,MultimapValue::drop, and table removal intable_tree.rs.All tables in one
WriteTransactionshare the same twoArc<Mutex>es, and using two tables from two threads is allowed (WriteTransaction: Sync). A concurrent insert on a sibling table holdsfreed_pagesand then waits onallocated_pagesinsideLeafBuilder::build, so it can deadlock against the conversion holdingallocated_pagesand waiting onfreed_pages— an ABBA cycle that permanently hangs the writer.The inverted branch is reachable, not just theoretical: insert two small values plus one large value (the value-set is promoted to a subtree with a branch root), commit, then remove the large value. The branch collapses and leaves the untouched committed small-value leaf as the new root; it is below half a page, so it is rewritten back to inline storage and the committed leaf is freed via
freed_pages— exactly the branch with the reversed lock order.This is present in the released v4.1.0, not a master-only regression, so it's included in the changelog.
Fix
Lock
freed_pagesbeforeallocated_pageshere, matching every other site, and use the existingconditional_free()helper for the free-or-defer logic.Testing
multimap_remove_collapses_committed_subtree_to_inline, which reproduces the exact collapse-to-committed-leaf scenario. I verified (via a temporary probe) that no existing test reached the committed-root branch and that this test does. The test is single-threaded, so it locks in correctness of the code path but cannot itself trigger the timing-dependent deadlock — the guarantee comes from the lock order now being globally consistent.cargo fmt --check,cargo clippy --all-targets --all-features(no warnings), andcargo test --all-featuresall pass.🤖 Generated with Claude Code
https://claude.ai/code/session_01YBCeHTcEZagQHCFMCcnqKn
Generated by Claude Code