rpc, execution, db: FcuBackgroundCommit groundwork, coherent cache fixes

[yperbasis]

Summary

Groundwork for enabling FcuBackgroundCommit — the default flip and the standalone cache-retention increase from 0MB to 128MB are in the stacked #22269; defaults are unchanged here. With the flag on, the FCU response returns to the consensus client before the MDBX flush+commit lands: the commit runs in a background goroutine, and the ExecModule semaphore serializes successive FCUs, so FCU N+1 always reads FCU N's committed state. This PR makes RPC reads, the version-keyed state cache, and post-FCU consumers coherent under that mode. Several of the fixes matter already today, because state-change notifications are dispatched pre-commit.

RPC: head-sensitive reads resolve on a well-defined view

They see the pre-commit head everywhere or nowhere — never a mix.

Overlay view — the head and every dependent block-table read are served from the published BlockOverlay: bor getSnapshot/getAuthor/getSigners/getSnapshotProposer{,Sequence}/latest-block, eth_getBlockTransactionCountBy{Number,Hash}, graphql latest-block, debug_setHead, debug_getRawHeader, eth_getTransactionByHash, txpool_content{,From}, and erigon_getBlockByTimestamp. Behavior change on Polygon: for bor getSnapshot/getSigners/getSnapshotProposer{,Sequence}, nil/latest now resolves to the overlay-aware forkchoice/executed head instead of the header-stage tip (ReadCurrentHeader), so a catching-up node answers for its executed position rather than the downloaded-header tip — consistent with eth_blockNumber; getAuthor additionally fixes explicit-tag resolution (negative tags were cast to uint64 and returned "unknown block"). BaseAPI.headerByHash is overlay-aware, covering every by-hash consumer that stays on block tables. Each request pins one overlay read view up front and reuses it for all dependent reads, so an overlay unpublished mid-request cannot drop the request onto the older MDBX snapshot.

Committed view — the dependent reads use SD-temporal data, which the overlay does not expose, so tags resolve with nil filters and the bounds agree with the data: eth_getLogs/overlay_* range resolution (including eth_getLogs block-hash filters), trace_filter, eth_getProof, eth_simulateV1, debug_traceBlockBy*, debug_storageRangeAt, debug_accountRange, debug_accountAt (by-hash included — an overlay-resolved head would have no committed history), eth_getWitness. The filters-param contract is documented on rpchelper.GetBlockNumber. eth_getProof additionally keeps header lookup, commitment reconstruction, and state reads on the caller's single RO snapshot; shared branch-cache reads are bound-gated (servableUnderBound, #22467), so a concurrent commit cannot mix snapshots. Nil guards cover parity_listStorageKeys, trace_filter, and the eth_getProof header lookup.

Compatibility: on payload-building nodes, "pending" resolves to the latest executed block in every committed-view method that accepts the tag (eth_getLogs/overlay_*, trace_filter, debug_traceBlockByNumber, eth_simulateV1, eth_getProof, eth_getWitness). This keeps tag resolution aligned with the committed data those methods read. debug_accountRange keeps its explicit pending rejection.

Known embedded-daemon limitation: generic latest-state calls (eth_call, eth_getBalance, eth_getStorageAt, eth_getCode) resolve the overlay head while their temporal state reads stay on the committed snapshot — head N with state N-1 for the commit duration. The SD-aware temporal view needed to close this is tracked in #21314.

Coherent state cache: roots keyed by the version readers observe

db/kv/kvcache, execution/execmodule/notification_dispatcher.go. The pre-commit dispatch announces the post-commit PlainStateVersion, so version-keyed roots match exactly the readers that should see them: post-commit transactions hit the new canonical root, pre-commit transactions keep the previous one. Batch-fed storage and code entries use the key shapes readers look up (address+location; code by address). A view that outlives KeepViews version advances falls back to its own tx snapshot instead of erroring too old ViewID; reads through non-latest views are not cached (they bypass eviction accounting and would grow without bound). Retained roots share the configured memory budgets: state and code entries in older roots are cleared on version advance, and views pinned to them fall back to their own snapshots. Canonical roots start from their own batch, with no carry-over: the state-change producers don't announce every mutation, so a carried entry could stay stale — at a deliberate hit-rate cost: every version starts cold except the batch's changed keys, so hot-but-unchanged keys miss once per key per block, and steady-state hit rates are much lower than with carry-over. Completing the producers and restoring carry-over (which recovers the warmth) is tracked in #22276. The standalone daemon uses this coherent cache at every configured budget, in remote and --datadir modes alike. Positive budgets retain entries; 0MB retains none and disables new-block waiting, so reads fall back to the caller's tx snapshot while remaining coherent. The default remains 0MB until #22269 raises it to 128MB.

Engine: busy-tolerant batch execution

A background FCU commit briefly holds the exec semaphore, so ValidateChain/UpdateForkChoice can routinely return Busy. execDownloadedBatch waits it out via retryBusy: a ctx-aware 50 ms poll with a periodic debug log so a stuck commit surfaces.

Overlay safe-close invariant

db/kv/membatchwithdb. Shared-tx read views are safe only on the pure-Go memStore backing, whose Rollback/Close are no-ops on the data — that is what lets the bg-commit goroutine close the published overlay while RPC readers still iterate views. The memTx field's type restricts overlays to that backing.

Post-FCU readers wait for the commit

State-change events are dispatched pre-commit, so the notification stream alone does not guarantee MDBX contains the head. import_cmd and the execmodule tester call ExecModule.WaitIdle before opening a fresh tx after UpdateForkChoice.

Tests

TestStateChangeVersionMatchesCommitted pins announce-vs-committed version parity across {fg, bg} × {1-by-1, batched} (the batched case crosses the initial-cycle threshold, covering mid-FCU version bumps). kvcache tests pin fresh roots, retained-root aggregate budgets, reader-shaped feed keys, the evicted-view fallback, and non-latest no-cache. TestZeroBudgetRemoteCachePinsCommittedState pins snapshot-consistent reads with entry retention disabled. Overlay tests pin view selection in TestGetBlockTransactionCountByHash_SeesOverlayHead, TestDebugAccountAt_OverlayHeadHash_CommittedView, and TestGetLogsBlockHashUsesCommittedView, plus view lifetime under concurrent unpublish (the three *_PinsOverlayView tests). TestGetProofPinsReadSnapshot pins all proof reads to one RO snapshot; TestGetProofMissingHeader pins a clean error for a missing header. TestNotificationDispatchBackgroundCommit covers notification dispatch under background commit; multi-block bg-commit coverage lives in TestReorgBackAndForwardIntoCanonicalChain (bg mode) and TestInsertBlocksWithBatchedFCU_BadBlockRecovery_Background.

Safety (with the flag enabled)

FCU sequencing. The bg goroutine releases the exec semaphore only after the commit completes and PublishOverlay(nil). AssembleBlock/ValidateChain share the semaphore, so engine_newPayload/engine_getPayload serialize behind a pending commit — ~commit-duration added latency; a SYNCING response in a pipelined burst is spec-valid and self-heals.

Embedded rpcdaemon. Overlay-aware paths read FCU N's block-table writes pre-commit. In-flight readers keep their pinned view across unpublish; new readers cascade to MDBX, which by then contains N. No stale window for block-table paths; latest-state calls carry the #21314 limitation above.

Standalone rpcdaemon (remote or with datadir). With a positive entry-retention budget, cache entries are keyed by PlainStateVersion, bumped inside the commit batch and announced at the post-commit value — a daemon tx opened during the commit window resolves the N-1 root, so pre-commit batch data is never served against the older head. At 0MB, no entries are retained and each view falls back to its own tx snapshot, preserving the same consistency without serving batch data. Incomplete producer batches can still leave a retained entry stale for one version (#22276).

Interplay with #21414 (FCU semaphore decouple)

#21414 releases the exec semaphore as soon as updateForkChoice returns and moves the commit to a FIFO background worker, chaining FCU N+1's in-memory state onto FCU N's in-flight commit generation. Whichever PR lands second must re-validate the premises stated above in semaphore terms:

• "FCU N+1 reads FCU N's committed state" becomes "reads the chained in-flight generation" — TestStateChangeVersionMatchesCommitted and the cache's pre-commit-window reasoning are the regression net.
• WaitIdle currently implies the commit has landed; import_cmd and the execmodule tester rely on that.
• Overlay fallbacks rely on PublishOverlay(nil) happening only after the commit lands; the worker must preserve that ordering.
• The Busy window retryBusy waits out mostly disappears (the helper stays correct, just rarely loops).

Flag gating composes: #21414 inherits --fcu.background.commit (default false); the flip stays with #22269.

[Copilot]

Pull request overview

Enables FcuBackgroundCommit by default and updates JSON-RPC handlers to read head-sensitive data through the SharedDomains overlay so RPC responses can reflect the new FCU head before the MDBX commit/fsync lands.

Changes:

• Flip ethconfig.Defaults.FcuBackgroundCommit to true and update the rationale comment to reflect current overlay + notification/coherent-cache wiring.
• Wrap a set of head-sensitive rawdb.Read* / rpchelper.Get{Latest,Safe,Finalized}BlockNumber call sites with filters.WithOverlay / WithTemporalOverlay.
• Minor call-site refactors (introducing overlayTx locals) to reuse overlay-backed transactions.

Reviewed changes

Copilot reviewed 17 out of 17 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
rpc/jsonrpc/txpool_api.go	Read current header via overlay for txpool content endpoints.
rpc/jsonrpc/trace_filtering.go	Use overlay-backed head hash/number when ToBlock is omitted.
rpc/jsonrpc/parity_api.go	Read current block number via overlay for parity storage-key listing.
rpc/jsonrpc/overlay_api.go	Use overlay-backed latest block number when capping ranges.
rpc/jsonrpc/graphql_api.go	Resolve “latest” block number via overlay for GraphQL API.
rpc/jsonrpc/eth_txs.go	Read current header via overlay for pending-tx responses.
rpc/jsonrpc/eth_system.go	Use temporal overlay for gas oracle backend and overlay for head header reads (base fee/blob base fee).
rpc/jsonrpc/eth_simulation.go	Compare requested block to overlay-backed latest head to avoid “future block” false positives.
rpc/jsonrpc/eth_receipts.go	Use overlay-backed latest block number in max-uint32 capping path.
rpc/jsonrpc/eth_call.go	Use overlay-backed latest block number when building proof/witness preconditions.
rpc/jsonrpc/eth_block.go	Use overlay-backed latest block number for transaction-count-by-number guard.
rpc/jsonrpc/erigon_receipts.go	Use overlay-backed latest block number for log range defaults.
rpc/jsonrpc/erigon_block.go	Read current header via overlay for timestamp-based block lookup.
rpc/jsonrpc/debug_execution_witness.go	Use overlay-backed latest block number when building expected post-state.
rpc/jsonrpc/debug_api.go	Use overlay-backed latest block number for debug_setHead baseline.
rpc/jsonrpc/bor_api_impl.go	Use overlay-backed latest header/number in several Bor RPCs.
node/ethconfig/config.go	Default FcuBackgroundCommit to true and expand explanatory comment.

Comments suppressed due to low confidence (4)

rpc/jsonrpc/parity_api.go:80

• bn is read through the overlay, but subsequent reads (_txNumReader.Min and tx.RangeAsOf) still use the original temporal tx. During a background-commit window this can leave parity_listStorageKeys querying state/txnums from the committed DB while targeting the overlay head number, producing stale or inconsistent results. Consider wrapping the temporal tx once (e.g., via filters.WithTemporalOverlay) and using that wrapped tx consistently for the latest-state reader, txnum lookup, and the RangeAsOf scan.

	bn := rawdb.ReadCurrentBlockNumber(api.filters.WithOverlay(tx))
	minTxNum, err := api._txNumReader.Min(ctx, tx, *bn)
	if err != nil {
		return nil, err
	}

rpc/jsonrpc/trace_filtering.go:351

• toBlock is now derived from an overlay-backed head hash/number, but the rest of the method still passes the original dbtx into filterV3, which computes txnum bounds and scans indexes against the committed DB view. This means the requested range can reference an overlay head while the underlying trace scan is still anchored to the pre-commit state (and the block-range limit check uses the overlay height). Consider using a temporal overlay tx (filters.WithTemporalOverlay(dbtx)) and passing that through to filterV3 (and any txnum/index reads) so the head number and the scanned data come from the same view.

	if req.ToBlock == nil {
		overlayTx := api.filters.WithOverlay(dbtx)
		headNumber, err := api._blockReader.HeaderNumber(ctx, overlayTx, rawdb.ReadHeadHeaderHash(overlayTx))
		if err != nil {
			return err
		}
		toBlock = *headNumber

rpc/jsonrpc/bor_api_impl.go:111

• latestBlockNum is resolved using an overlay-backed tx, but the subsequent HeaderByNumber call still uses the original tx. During background commit this can cause the latest header lookup to miss the overlay head (returning errUnknownBlock even though the overlay has the header). Consider reusing the same overlay-wrapped tx for the HeaderByNumber/HeaderByHash reads in the “latest” path.

	//nolint:nestif
	if blockNrOrHash == nil {
		latestBlockNum, err2 := rpchelper.GetLatestBlockNumber(api.filters.WithOverlay(tx))
		if err2 != nil {
			return accounts.NilAddress, err2
		}
		header, err = api._blockReader.HeaderByNumber(ctx, tx, latestBlockNum)
	} else {

rpc/jsonrpc/eth_block.go:383

• This method now uses an overlay-backed tx to compute latestBlockNumber, but it still reads the block body/tx count through the original tx later in the function. In a background-commit window, blockNum can be the overlay head while _blockReader.Body on the committed tx returns nil, so the RPC may still return null for latest. Consider using the same overlay-wrapped view for the subsequent body read when serving head-sensitive queries.

	latestBlockNumber, err := rpchelper.GetLatestBlockNumber(api.filters.WithOverlay(tx))
	if err != nil {
		return nil, err
	}
	if blockNum > latestBlockNumber {

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[yperbasis]

Manually dispatched the three CI workflows that were skipped because the PR is in draft state:

• Test Hive — https://github.com/erigontech/erigon/actions/runs/26146859675 (auto-skipped per test-hive.yml:17-19, guards on !pull_request.draft && !labels.skip-uncaching)
• Hive EEST tests — https://github.com/erigontech/erigon/actions/runs/26146860896 (only auto-runs via workflow_call from the EEST scheduler, so it never auto-triggers on PRs)
• Kurtosis GLOAS Tests — https://github.com/erigontech/erigon/actions/runs/26146862426 (auto-skipped per test-kurtosis-gloas.yml:36, guards on !pull_request.draft)

Why I want them: this PR flips FcuBackgroundCommit to true and re-routes head-sensitive RPC reads through the SD overlay. The earlier auto hive run on commit d7bbe14 (run 26126332101) hit a real bug in the import path (nil-deref in BlockReader.CurrentBlock because the post-FCU WriteHeadBlockHash raced the bg-commit goroutine), fixed in commit 9e61f87. Want the dispatched run to confirm that fix, and to surface any other paths with the same shape across the broader hive + gloas surface.

[yperbasis]

Re-dispatched the three skipped workflows against HEAD 79ce7d94ef (includes the import_cmd WaitIdle fix from 9e61f87, the bor/eth_block overlay-consistency extensions from d0a8e31, and the head-vs-data revert sweep from 79ce7d9):

• Test Hive — https://github.com/erigontech/erigon/actions/runs/26148490758
• Hive EEST tests — https://github.com/erigontech/erigon/actions/runs/26148492430
• Kurtosis GLOAS Tests — https://github.com/erigontech/erigon/actions/runs/26148493906

[Copilot]

Pull request overview

Copilot reviewed 20 out of 20 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 36 out of 36 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 37 out of 37 changed files in this pull request and generated 5 comments.

[yperbasis]

Splitting this PR into independently-mergeable pieces, all based on current main (none stacked), kept as drafts:

• db, execution, rpcdaemon: coherent version-keyed state cache #22532 — db, execution, rpcdaemon: coherent version-keyed state cache — post-commit version announce, kvcache fresh-roots rework, rpcdaemon wiring at every budget, the execmoduletester WaitIdle (required by any bg-commit test: the tester's setup InsertChain otherwise returns pre-commit and GenerateChain builds on stale genesis state), and the un-skipped TestNotificationDispatchBackgroundCommit.
• rpc: resolve head-sensitive reads on a well-defined view (overlay vs committed) #22533 — rpc: resolve head-sensitive reads on a well-defined view — the overlay-view vs committed-view split across rpc/jsonrpc, including the eth_getProof single-snapshot fix and the bor "latest" semantics change.
• db/kv/membatchwithdb: restrict overlays to the pure-Go memStore backing #22534 — db/kv/membatchwithdb: restrict overlays to the pure-Go memStore backing — the safe-close invariant, made structural.
• execution, cmd, docs: make FcuBackgroundCommit consumers wait out the async commit #22535 — execution, cmd, docs: make FcuBackgroundCommit consumers wait out the async commit — retryBusy, import_cmd WaitIdle, flag documentation.

The union of the four reproduces this PR's diff exactly; only rpc/rpchelper/helper.go and the generated llms files are touched by two PRs, with non-overlapping hunks. #22269 (the default flip) conceptually follows #22532 + #22535.

[yperbasis]

Closing in favor of the split (map in the comment above): #22532 (coherent version-keyed state cache), #22533 (RPC overlay/committed view split), #22534 (membatchwithdb safe-close invariant), #22535 (bg-commit consumers + flag docs) — with #22269 (default flip) and #22278 (SimpleCache rename) re-stacked onto #22532. Together they reproduce this branch's diff exactly.

The "Interplay with #21414" checklist from this description now lives as a comment on #21414, mapped to the split PRs. The branch stays in place for reference.