v3.5.2/v3.5.3 mainnet archive: small reorgs near tip leave persistent corruption in the derived receipt/log domains; eth_getLogs contradicts the node's own eth_getBlockReceipts, survives restarts, gets frozen into snapshot files

[MoonBoi9001]

System information

Erigon version: erigon version 3.5.3-ee865f2c (occurrences 2-4; occurrence 1 was on v3.5.2)

OS & Version: Linux (Ubuntu 24.04, unprivileged LXC on Proxmox, datadir on ZFS)

Commit hash: ee865f2

Erigon Command (with flags/config):

erigon --datadir=/eth-data/erigon --chain=mainnet --prune.mode=archive --batchSize=1G \
  --externalcl --authrpc.addr=127.0.0.1 --authrpc.port=8551 \
  --http --http.api=eth,erigon,web3,net,trace,debug,txpool

Consensus Layer: Lighthouse v8.1.3 (external, Engine API on 127.0.0.1:8551)

Consensus Layer Command (with flags/config): standard beacon_node setup; full unit file available on request

Chain/Network: Ethereum mainnet, archive

Expected behaviour

eth_getLogs and eth_getBlockReceipts are 2 views of the same data, so for any block the log set they return should be identical: same logs, same logIndex, same topics.

Actual behaviour

After some small reorgs near the tip, the derived domains that serve eth_getLogs end up persistently corrupt for 1 block: eth_getLogs returns a phantom log with the wrong logIndex (and in one occurrence wrong topics) stitched onto the correct transaction and data, while the block's true log entries are missing from both filtered and unfiltered results. eth_getBlockReceipts for the same block on the same node stays fully correct (verified byte-identical to independent providers each time), so the node contradicts itself. The corruption survives restarts, and within hours it is merged into the frozen snapshot files, after which no restart or reset revisits it. The only repair that works is the documented derived-domain rebuild (snapshots rm-state-snapshots --domain=receipt,logtopics,logaddrs,tracesfrom,tracesto --step=<range>, then integration stage_custom_trace --reset and a run, per cmd/integration/Readme.md), which heals it completely every time.

The asymmetry between the 2 endpoints is structural (details in the root cause section below): Generator.GetReceipts (eth_getBlockReceipts) re-executes the whole block and derives every logIndex natively, never consulting the derived domains, while Generator.GetReceipt (eth_getLogs, eth_getTransactionReceipt) takes each transaction's log index base from ReceiptDomain history and re-executes only that one transaction. Every domain the repair rebuilds sits exclusively on that second path. Note also that unfiltered eth_getLogs never consults the logtopics/logaddrs indexes (with no filter it walks every txNum in the range), so the "missing from unfiltered results too" symptom points at the ReceiptDomain log-index metadata as the primary corrupt artifact, with the inverted indexes as a possible second (they are append-only and are skipped by some unwind paths, see below). This node runs without --persist.receipts (archive default), so the RCacheDomain path is inert and not a factor here.

This has hit the same node 4 times in 14 days:

date (UTC)	version	corrupt block	observed damage
2026-07-23	v3.5.2	25,592,780-787	after a 3-block reorg at 03:33 (blocks 25,592,769-771), Arbitrum bridge MessageDelivered index 2,514,362 returned twice with different payloads, 2,514,363 missing, and block 25,592,780 served a stale duplicate of 2,514,361. Notably no Unwind Execution line was logged for this reorg (the blocks arrived via the engine-api backward block downloader), while 2 other 1-block reorgs the same day did log unwinds and did no damage.
2026-07-25	v3.5.3	(same signature)	recurred within a day of upgrading to v3.5.3.
2026-08-05	v3.5.3	25,688,847	written during a window (12:45-12:53) where execution lagged to ~1 block per 20-30s under BuildFilesInBackground and the RPC answered eth_getLogs with "block range extends beyond current head block" for ~8 minutes. Afterwards: the block's 2 bridge logs (true logIndex 269, 270) missing from eth_getLogs; a phantom at logIndex 698 carrying topic1 = message index 2,527,576, which truly lives in the previous block (25,688,846, logIndex 965). Same transactionHash and data as the true log; logIndex and topics wrong.
2026-08-06	v3.5.3	25,692,570	overnight, ~12h after repairing the previous one: bridge MessageDelivered index 2,528,110 served at logIndex 337 where receipts (and independent providers) say 26.

Note the fingerprint difference: occurrence 4 served the correct event with only logIndex wrong, while occurrence 3 also had wrong topics; a persistent logIndex corruption and the transient, cache-held mechanism of #22106 layering during a lagging window would account for that. In every occurrence the wrong value traces back to the previous block.

How we notice at all: a co-located Arbitrum nitro node reads its L1 inbox from this Erigon and verifies every message against its on-chain hash commitment, so it permanently loops error reading inbox err="found message N data with mismatched hash" and freezes. Any consumer without cryptographic verification would ingest the phantom silently.

Root cause analysis (from reading the source at commit ee865f2, the v3.5.3 tag)

We traced the read, write, and unwind paths at the affected commit. The mechanisms below are confirmed by direct code reading (and are still present on main today); the final attribution of our specific bad values to this chain is inference, not an executed reproduction, so we present it as a proposed root cause.

Defect 1: some reorg unwinds never roll back the receipt domains. TemporalMemBatch.GetDiffset (db/state/temporal_mem_batch.go:504-521) builds the unwind changeset as a [kv.DomainLen] (= 6) array but supplies only 4 elements: Accounts, Storage, Code, Commitment. ReceiptDomain (index 4) and RCacheDomain (index 5) come back nil. DomainRoTx.unwind (db/state/domain.go:1216) restores values purely from those diffs, so for the receipt domains it restores nothing and only prunes their history above the unwind point (domain.go:1300), leaving the orphaned fork's latest values in the values table. The receipt diffs themselves are recorded and serialized correctly (the changeset sink covers all 6 domains, and changeset.ReadDiffSet/deserializeKeys, db/state/changeset/state_changeset.go:306-315, read all 6 back; the "there are 4 domains of state changes" comment at state_changeset.go:36 is stale). Git history shows the omission is an oversight rather than a decision: the literal was written in June 2024 (#10714) when kv.DomainLen was 4, so it enumerated every domain that existed; ReceiptDomain arrived as the 5th domain in September 2024 (#12012), and the serialization loops picked it up automatically while the hand-enumerated literal never did. A plain single-FCU reorg reads its changeset from disk and unwinds correctly, which matches the harmless reorgs we observed the same day. The defect is confined to the in-RAM branch of GetDiffset and fires when 2 conditions hold together: the orphaned blocks were already committed to disk, and the unwind runs inside the same SharedDomains instance that executed them. The code paths satisfying both are multi-block FCU cycles that commit mid-cycle and then unwind within the same cycle: the backward-downloader catch-up batch (engine_block_downloader/block_downloader.go:245-299, executing up to LoopBlockLimit blocks then forcing an FCU, with CommitCycle, forkchoice.go:535-570, committing mid-loop while the instance lives on), a bad-block unwind during the forward run (exec3.go:749, exec3_parallel.go:707), and MergeExtendingFork copying a fork validator's accumulated changesets into the FCU's instance (forkchoice.go:509).

How the stale value becomes a persistent wrong answer. The values table itself self-heals: re-execution rewrites the same 3 keys and the newer write replaces the orphan's. The lasting damage is in history. AppendReceipt (db/rawdb/rawtemporaldb/accessors_receipt.go:59-82) passes prevVal=nil, so SharedDomains.domainPut (db/state/execctx/domain_shared.go:856-882) fetches the previous value itself, and after such an unwind every layer that should supply the true pre-reorg value misses: the RAM overlay was pruned, and the unwoundLatest safety net (temporal_mem_batch.go:225-244) consults sd.unwindChangeset[domain], which is empty for the receipt domains for exactly the same reason the value restore was empty (1 dropped diff breaks both the write-back and the read-back). The fetch falls through to the DB and returns the orphaned chain's un-reverted value, and the first post-unwind write records it into history as the value-before-this-txNum (db/state/domain.go:342). ReceiptAsOf resolves a query via GetAsOf/HistorySeek, which returns the first history record at or after the requested txNum; the unwind pruned everything above the unwind point, so this poisoned boundary record is exactly the one that answers queries for the last pre-reorg block's txNums. And because logIndexAfterTx is cumulative and the receipt generator derives each transaction's first log index from ReceiptAsOf at the preceding txNum, a single poisoned boundary record misnumbers a whole block: 1 block of wrong log indexes, committed to disk, which is the observed shape, with the wrong values tracing back to the orphaned chain segment (our occurrences 1 and 4; compare also the fingerprint in #22504, where the leaked value was exactly the previous block's final log count).

Defect 2: some reorg paths skip the on-disk unwind entirely and log nothing. Unwind Execution has 1 call site (execution/stagedsync/stage_execute.go:405) and only fires when the unwind point is below execution progress. Branches that switch the canonical chain without it: the else-branch at stage_execute.go:447-458 does a RAM-only prune with a nil changeset, so TemporalMemBatch.Flush (temporal_mem_batch.go:765-780) skips the disk unwind completely, including the logtopics/logaddrs/tracesfrom/tracesto truncation; execution/execmodule/forkchoice.go:394 skips the unwind while still rewriting canonical hashes and the block-to-txNum mapping (forkchoice.go:422-456); execution/stagedsync/sync.go:142 swallows ErrBehindCommitment and reports success; sync.go:539 skips when stage progress already equals the target. On these paths the append-only inverted-index postings from the orphaned fork survive permanently (the writers have no delete path), and once the txNum mapping is rewritten they attach to transactions they never belonged to. Independently, the buffered ETL writers are never rolled back by an unwind and are flushed after the disk unwind runs (temporal_mem_batch.go:522-601, 765-790), so orphaned-block writes can reach MDBX even when an unwind did run. This matches our occurrence 1: the damaging reorg logged no Unwind Execution while 2 clean same-day reorgs logged it.

Why it reaches the frozen files. Domain and history collation copy the MDBX tables verbatim with no canonicality check, and the freeze-side guard (reorgSafeBlock, default depth 96 blocks) only delays collation; the corruption above lives at committed txNums that look canonical, so the next BuildFilesInBackground seals it into the immutable files, after which the MDBX-level unwind cannot reach it. Precedent that this class of corruption reaches published files exists: #22142 documented persisted ReceiptDomain corruption (receipts not hashing to the block's own receiptsRoot) and was resolved by regenerating files.

The transient layer on top. The mechanism analysed in #22106 (a block admitted through the RPC overlay while its commit is in flight, ReceiptAsOf reading the committed tx, GetAsOf falling back to GetLatest: receipts_generator.go:278 vs :283) mints wrong log indexes into the RPC caches. Its fix (#22511 + #22893, backported to release/3.5 as #22951 on 2026-08-04) is in no released build: not in v3.5.3 (this node's build), and v3.5.4 was tagged before the backport landed. That mechanism plausibly explains the wrong-topics component of our occurrence 3, minted during the 8-minute lagging window and held by the caches. What it cannot explain is our restart-surviving, frozen-into-files evidence, which contradicts the general statement in #22951's notes that "the wrong values were never persisted" and no rewind is ever needed. Both variants appear real; this report is about the persistent one.

Test coverage. We could not find a test on any branch that asserts ReceiptDomain, RCacheDomain, logtopics or logaddrs contents after a reorg or unwind; existing reorg tests check accounts, storage and commitment only. Two tests would settle this. The one-line root cause proof: assert that GetDiffset returns non-empty slices at indexes 4 and 5 for a block that wrote receipts (today it cannot, on any branch). The end-to-end one: execute and commit several blocks inside one SharedDomains instance, unwind them through UnwindExecutionStage on that same instance, re-execute a different chain, then assert ReceiptAsOf at the boundary txNum; per the analysis it should return the orphaned chain's logIndexAfterTx. We have opened #23061 with a fix for defect 1 (loop over all domains in GetDiffset) together with the first of these tests.

Relation to existing issues: #22106 (open) is the same symptom family; its analysed transient variant is complementary to this report, and its fix is unreleased for 3.5 as noted above. #22142 (closed) is the persistence precedent. #22110 fixed a different writer of the same wrong data (firstLogIndex = 0 written into ReceiptDomain on mid-block resume under parallel exec) on main/3.6 only, never on 3.5. #21515 and #22423 (closed, v3.4.x) established the post-reorg family, #22423 with corruption in persistent domains surviving cold restarts. #22504 (closed) is the transient fingerprint. #22622 (post-unwind overlay off-by-one at the boundary txNum, main only, never backported to 3.5) may also be relevant at the unwind boundary.

Steps to reproduce the behaviour

No deterministic trigger observed live, but both observed onset conditions match the defect 1 trigger paths above. At our observed rate (2 clusters of 2 in 14 days on 1 mainnet archive node) it recurs within days, so an instrumented build could likely catch it live; happy to run one. The 2 regression tests sketched in the root cause section should reproduce the defect deterministically without any live reorg.

Detecting it on any node is deterministic and self-contained (no external providers needed):

B=0x188099a   # 25,692,570, occurrence 4; any suspect block
curl -s -X POST -H 'content-type: application/json' localhost:8545 \
  -d '{"jsonrpc":"2.0","id":1,"method":"eth_getLogs","params":[{"fromBlock":"'$B'","toBlock":"'$B'"}]}' \
  | jq '[.result[] | {logIndex, address, topics}] | sort_by(.logIndex)' > a.json
curl -s -X POST -H 'content-type: application/json' localhost:8545 \
  -d '{"jsonrpc":"2.0","id":1,"method":"eth_getBlockReceipts","params":["'$B'"]}' \
  | jq '[.result[].logs[] | {logIndex, address, topics}] | sort_by(.logIndex)' > b.json
diff a.json b.json   # any output = eth_getLogs disagrees with the node's own receipts

On a corrupted node this diff shows the phantom/missing entries; after the derived-domain rebuild it is empty.

A single-transaction cross-check should also be conclusive (predicted from the code, not yet run here): eth_getTransactionReceipt for the phantom's transaction hash goes through the same per-transaction path as eth_getLogs, so on a corrupted node it should show the same wrong log positions while eth_getBlockReceipts shows the correct ones for the very same transaction.

Backtrace

No panic; the node keeps running and serving. Evidence trail for occurrence 3 (2026-08-05):

erigon journal during the corruption window (execution lagging, ~1 blk per 20-30s):
[WARN] [08-05|12:45:08.526] [rpc] served  conn=... method=eth_getLogs err="block range extends beyond current head block"
  (repeats for ~8 minutes; no "Unwind Execution" line in the window)
[INFO] [08-05|12:45:12.205] BuildFilesInBackground  step=9477 lastInDB=9477 targetStep=9478 reorgSafeBlock=25688713

self-consistency diff at block 25,688,847 afterwards (getLogs vs the node's own receipts):
  getLogs: 834 logs, but logIndex 269 and 270 missing; phantom present at logIndex 698
    phantom: address 0x8315177a... topics[1]=0x...2691d8 (msg 2,527,576)  logIndex 0x2ba
    true (receipts): same transactionHash, same data, topics[1] differs, logIndex 0x10d
  receipts: 834 logs, byte-identical to independent public providers

downstream client (Arbitrum nitro, reading this node as its L1), looping once per second:
WARN error reading inbox  err="found message 2527576 data with mismatched hash"

[taratorio]

thank you. solved by #23061 ?

[MoonBoi9001]

Partially. #23061 fixes the main defect (defect 1 in the report): the in-RAM unwind changeset was missing the 2 receipt domains, which is the traced cause of the persistent corruption. It has since been backported to release/3.5 (#23064) and release/3.6 (#23063).

---

Still open: the defect 2 paths that skip the on-disk unwind entirely (listed in the report, each needs a call on intended behavior), and a tagged release containing #22951 and #23064 (both are merged to release/3.5 but in no released build yet). Note the fix is preventive only: nodes that already have corrupt values need the derived-domain rebuild to heal.

[VBulikov]

yperbasis added Imp1 as Importance