confirm: probe in-flight outs once, not per parent - #266
Closed
rearden-grok[bot] wants to merge 1 commit into
Closed
confirm: probe in-flight outs once, not per parent#266rearden-grok[bot] wants to merge 1 commit into
rearden-grok[bot] wants to merge 1 commit into
Conversation
Post-milestone unique spent parents were walking every in-flight layer via get_out in fill_missing and pin adopt. Union the CreatePin keys instead. Name that wall fill=/idx= on ibd: perf. Co-authored-by: Cursor <cursoragent@cursor.com>
Owner
|
No thanks. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Post-milestone IBD (~843k, ~8.3 blk/s) saturates the load OS thread, not scripts. A 5s window was
load_thr busy/wait ≈ 4475/430ms; pin_hit% ≈ 19% (expected: fat blocks, 18 pipeline maps cover few heights). Load split was stamp pack ~2.1s + pin ~2.1s;stamp_sub batch=~1.9s with only ~0.5s in the named nested timers.Root cause:
fill_missing_parent_rangesand pin adopt calledInFlightView::get_out(id)per unique spent parent (~160k) × every in-flight layer (~15) → millions of HashMap probes.get_outassumed L and pack counts stay small; unique parents now dwarf pipeline CreatePins.This PR:
out_ids/for_each_out, newest-first, each id once) and skips that way in fill_missing + pin adopt.batch=gap onibd: perf:fill=/idx=(spent/body range batches).Not in this PR (follow-ups): overlap
txout.bodywith stamp CPU; carryspent_rangeon published live_union; invertget_create_fkthe same way; sequential fk index vsbatch_pin_by_id; lookuphead=probe (~3.5s onlookup_thr).Does not flatten io_uring machines. Pins stay plan/batch only.
Test plan
cargo test -p rbitcoin-query out_ids_unions fill_missing_skipscargo test -p rbitcoin-consensus pin_for_wirecargo test -p rbitcoin-net format_info_and_debug_optional_subblockscargo fmt --allfmt,deny,clippy,ast-grep,test,windows,macos,multinode,coverage)stamp_sub fill=should drop vs priorbatch=gap;idx=shows remaining spent.idx costMade with Cursor