fix: reachable replay anchor - #307
Merged
skylenet merged 2 commits intoAug 12, 2026
Merged
Conversation
A prestate keeps whatever forkchoice markers it was built with. Prestates are produced by advancing a snapshot with forkchoiceUpdated calls setting head = safe = finalized on every block (builder/engine.go, builder/prerun_bundle.go), so the block they end on — the one every fixture replays from — arrives already finalized. geth will not move its head back to a block at or below the one it considers finalized: it answers "Skipping beacon update to finalized ancestor" with VALID and leaves the head alone. That stays invisible until a test moves the head far enough that the anchor's state is no longer retained. test_blockhash builds 256 dummy blocks to fill the BLOCKHASH window, putting the head 258 blocks out; the next test's first newPayload then names a parent whose state the client cannot serve, is answered ACCEPTED, and every later payload is orphaned. One such test stranded 4,213 of 4,884 tests in a jochemnet compute run, which still exited 0. Send one forkchoiceUpdated per instance, before any test, pointing safe and finalized at a block strictly below the head, and prefix each test's replay with a forkchoiceUpdated back to its anchor. On the reproduction the anchor stops being finalized, the forkchoice rewinds 258 blocks, and eth_getBalance at the anchor goes from "historical state ... is not available" to a balance. The zero hash cannot do this: clients read it as "no update", which is what lets the stale marker survive. It is also only permitted "unless transition block is finalized" (ForkchoiceStateV1), which mainnet-fork datadirs do not satisfy. The anchor is resolved once and kept: a configured hash, else the client's own finalized block when it already sits below the head, else the head's parent. For a datadir built by advancing a snapshot, the snapshot block is the natural configured value. This runs unconditionally rather than inside bootstrap_fcu, an optional readiness probe most configs leave unset.
SkipUntilBlockNumber drops leading lines until the first engine_newPayload past
the datadir head, so a partly-applied pre-run bundle is not replayed from the
start. Its doc says "skip pre-run RPC lines", but it ran on every step file,
including each test's own setup and test steps.
That was invisible while every step file began with an engine_newPayload, which
satisfies the condition immediately. It stops being invisible as soon as a step
starts with anything else: a setup opening with a forkchoiceUpdated — the call
that brings the head back to the block the fixture replays from — had that line
removed before it was sent, and the client never saw it:
Resuming pre-run replay resumed_at_block=24410464 resumed_at_line=2
skip_until_block=24410463 skipped_lines=1
A test's steps are never partly applied; only the pre-run replay resumes.
With this, the two-test reproduction goes from 516 executor failures to 0, and
the full 4,884-test compute suite passes with none.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Due to changes to the execution api https://github.com/ethereum/execution-apis/pull/786/changes and to Geth ethereum/go-ethereum#34767 the reorg logic of Geth changed which causes problems on jochemnet compute tests, where an attempted reorg derails the entire storage and makes it impossible to run any tests on it.
Prestates are built by advancing a snapshot with forkchoice calls that set
head = safe = finalizedon every block, so the block every fixture replays from arrives already finalized. Geth won't move its head back to a block at or below what it considers finalized. That stays invisible until a test drifts the head far enough that the anchor's state is no longer retained (test_blockhashbuilds 258 blocks), after which the next test's firstnewPayloadnames a parent whose state the client can't serve, getsACCEPTED, and every later payload is orphaned: 4,213 of 4,884 tests in one jochemnet compute run, which still exited 0.This PR sends one forkchoice per instance pointing
safe/finalizedat a block strictly below the head, and prefixes each test's replay with a forkchoice back to its anchor, so the client is asked for the anchor rather than assumed to be on it. It also scopesSkipUntilBlockNumberto the pre-run step, which is what its doc always claimed: it was running on every step file and silently dropping that new leading forkchoice before it was sent. Verified under exact CI conditions: the two-test reproduction goes from 516 executor failures to 0, and the full 4,884-test compute suite passes with none.