fix: prevent sync deadlock when no peer can serve the requested header range - #2
Draft
nonsense wants to merge 3 commits into
Draft
fix: prevent sync deadlock when no peer can serve the requested header range#2nonsense wants to merge 3 commits into
nonsense wants to merge 3 commits into
Conversation
An empty response to a headers or bodies request is the protocol-correct answer of a peer that does not have the requested data, but the downloaders reported it as a bad message (ReputationChangeKind::BadMessage, -16384): four honest answers cross the ban threshold and disconnect the peer for 12 hours. When no peer can serve the requested range - e.g. a fork choice target ahead of every currently connected peer - the node bans its entire honest peer set within seconds, guaranteeing the download it is waiting on can never complete. This wedged OP Stack nodes in production (Ink Mainnet 2026-07-16, unichain 2026-08-05), where a >32 block gap after a sequencer stall sent all replicas into backfill with no peer able to serve the range. Empty responses are no longer reported; the failed request is resubmitted as before, and the fetcher already deprioritizes peers whose last response was unsatisfactory when picking a peer for the next request. Malformed responses (wrong start block, non-contiguous headers, invalid seals, too many bodies) keep the full penalty. The engine's single-block and block-range fetch paths already treat empty responses as non-reportable. Also promotes the "Penalizing peer" log from trace to debug so reputation destruction is visible closer to default verbosity. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The Headers stage propagated the downloader's Pending forever: a fork choice target that no connected peer can serve - e.g. after a sequencer stall leaves every replica equally behind - parked the pipeline at stage 1 indefinitely, with no retry, no error and no log line. Combined with fork choice updates only being tracked during backfill, this wedged OP Stack nodes in production until an operator restart (Ink Mainnet 2026-07-16, unichain 2026-08-05). Two bounds, both progress-aware so a slow but healthy sync is never interrupted: - `HeaderStage` now arms a stall deadline (default 30s, configurable via `stages.headers.stall_timeout`) while the downloader is pending and resets it whenever headers arrive. On expiry the stage returns a new recoverable `StageError::Stalled` instead of pending forever. The ETL and downloader state are kept so a retry with an unchanged sync gap resumes the download; the ETL collectors are instead cleared whenever the sync gap changes, which also fixes stale collector entries leaking into a later run with a different target. A timeout wrapped around the `execute_ready` future itself would not work here: the headers stage legitimately downloads the entire gap - potentially for hours during initial sync - before it resolves. - The pipeline now gives a stage a readiness budget (default 5m, `PipelineBuilder::with_execute_ready_timeout`) armed on the first `execute_ready` failure and cleared once the stage becomes ready. When a stage keeps failing readiness beyond the budget, the pipeline emits `PipelineEvent::TimedOut` and gives up on the stage for the current run with `ControlFlow::NoProgress`, so control returns to the caller (the engine re-evaluates the backfill target) instead of retrying a stale target forever. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A finished backfill only returned the network to `SyncState::Idle` when the debug-only `--debug.startup-sync-state-idle` flag was set (the unconditional transition was removed in paradigmxyz#16742 and re-added behind the flag in paradigmxyz#19429). A backfill that ends without reaching its target - now reachable via the stage stall handling - therefore left the node marked syncing indefinitely, suppressing transaction gossip, until live sync happened to advance. This contributed to wedged OP Stack nodes in production (Ink Mainnet 2026-07-16, unichain 2026-08-05). Make the idle transition on `BackfillSyncFinished` unconditional again. If the node is still behind, `on_backfill_sync_finished` re-checks the distance against the latest tracked fork choice state and re-triggers backfill (emitting `BackfillSyncStarted`, which marks the network as syncing again) or issues the remaining downloads, so the idle window between consecutive runs is brief. This also means the initial-sync latch flips after the first backfill run instead of at the first live-sync block, matching pre-paradigmxyz#16742 behavior; a brief syncing/idle flap between runs is preferable to a node that stays "syncing" forever while wedged. The flag keeps its startup semantics. Also warn when the connected-peer count reaches zero while the node is syncing - a sync that has no peers cannot make progress, and this state was previously invisible at default log levels. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When a fork choice target is >32 blocks ahead and no connected peer can serve the header
range — e.g. a fleet of equally-behind replicas resuming after a sequencer stall — the node
bans its entire honest peer set (4 empty responses × BadMessage = 12h ban), the Headers
stage pends forever with no error or log line, and the network stays marked syncing until
an operator restart. Three production incidents on 1s-blocktime OP Stack chains hit this
exact shape; on 12s chains the same path is reachable after ~6.4 min of drift.
EmptyResponseas a bad message — it is the protocol-correct answer of apeer that lacks the data, and the fetcher already deprioritizes such peers for the next
request (sibling fix to fix(net): recover partial header responses paradigmxyz/reth#26482)
StageError::Stalledwhen the download makesno progress for
stages.headers.stall_timeout(default 30s), keeping ETL/downloaderstate so a same-gap retry resumes; end the pipeline run with
ControlFlow::NoProgressonce readiness failures exceed a budget (default 5m), so the engine re-targets from the
latest fork choice state instead of retrying a stale target forever
SyncState::Idleunconditionally onBackfillSyncFinished(reverts the fix: small networking fixes paradigmxyz/reth#16742 gate — sticky "syncing" after an unproductive backfill suppresses tx
gossip indefinitely; the debug flag keeps its startup semantics)