BlockStateCache vs StateCache

[sudeepdino008]

Erigon has two caches over the same data: BlockStateCache and StateCache. Both hold Accounts, Storage and Code. They sit on top of each other in the read chain, and they work against each other.

The two caches

	BlockStateCache	StateCache
Where	execution/state/rw_v3.go:987	execution/cache/state_cache.go:59
Created	fresh for every block (exec3.go:670, exec3_parallel.go:2451)	once per process (exec_module.go:317)
Holds	pre-block values, plus this block's writes	committed values only
Freshness	none — entries carry no txNum	txNum-stamped, frontier-gated, unwind-invalidated
Size limit	none — plain maps	byte budget, registered in the shared memory envelope

The read chain is: BlockStateCache → sd.mem → parent.mem → StateCache → BranchCache → files.

What goes wrong

BlockStateCache sits above StateCache, so it absorbs the reads StateCache exists to serve — then throws them away at the end of the block.

1. StateCache sees a decimated read stream. Its Account/Storage tiers are freelru.ShardedLRU, which promotes an entry on every Get. A slot read 5000 times in a block reaches it once. Eviction then ranks keys by "how many blocks touched this key first", not by how hot they are. The hottest keys look the coldest.
2. Hit-rate numbers are not comparable. UpdateStateCacheHit/Miss (domain_shared.go:1271) only fires after both BlockStateCache and sd.mem miss. The reported rate is conditional on two upstream misses.
3. Only one of them is budgeted. StateCache has byte budgets and releases its reservation in ExecModule.Close. BlockStateCache is unbounded maps, sized by the block's working set, outside that accounting.
4. Two freshness models for one problem. StateCache stamps every entry with a txNum and invalidates by epoch and floor. BlockStateCache stamps nothing. Its correctness rests on a short life, so the two cannot be merged as they stand.

The committed tier does not isolate anything

BlockStateCache's docstring says the committed tier gives "a stable view for GetCommittedState that isn't affected by intra-block DomainPut calls" (rw_v3.go:972). In the parallel executor there are no such calls for these three domains:

• sd.mem changes only in blockStateCache.Flush during completeBlock, and block N+1 is not scheduled until block N's completeBlock returns (exec3_parallel.go:1218-1233).
• StateCache is written only at flush and only invalidated on unwind — "the executor does not touch it during forward execution" (exec3_parallel.go:1264).
• The other in-block writers, the commitment calculator and ApplyTxIndexes, write CommitmentDomain, ReceiptDomain and the inverted indices. BlockStateCache covers none of those.

So the whole chain below is already frozen for the length of a block. The committed tier is a plain read cache, not an isolation mechanism. The hazard the docstring names is real only when blockCache == nil — the serial executor, which does not use this cache (rw_v3.go:401).

The write buffer half of BlockStateCache is a different matter. It has to be per-block, and it is what freezes the chain in the first place.

What the committed tier still earns

It is not dead weight today:

• It saves a chain walk per read: an sd.mem probe under latestStateLock.RLock, then a StateCache probe.
• Mid-batch there is a band of keys written by earlier blocks in the same batch. Those live in sd.mem only and are absent from StateCache, which is populated at flush. For that band the committed tier is not redundant at all.

Suggested direction

Close the gap between sd.mem and StateCache first. If StateCache can serve values that are still sitting in sd.mem, the second reason above disappears, the first shrinks to a cheap probe, and the committed tier can go — which removes the redundancy and gives StateCache back its real read stream. This overlaps with #23139.

Worth measuring before changing anything: how often the committed tier hits, and how much of that would have hit StateCache instead.

Adjacent cleanup

execution/state/block_cache_multiblock_flush_test.go opens with "The parallel executor reuses one BlockStateCache across all blocks in a batch". That was true before #20805, which made the cache per-block and removed the stale-committed dedup in Flush that caused a wrong trie root at block 24839762. The test still earns its place — it pins Flush behaviour whatever the allocation model — but the docstring reads as a description of current behaviour, and it is not.

[sudeepdino008]

Following up: the redundancy is wider than the description says. BlockStateCache does not just duplicate StateCache — it duplicates VersionedIO and the version map, and it does so on both of its tiers.

Every write set already reaches three stores:

store	fed by
be.blockIO (VersionedIO)	RecordWrites — exec3_parallel.go:2701, :2727, :2865, :2994, :3333
be.versionMap	FlushVersionedWrites — :2814, :2995, :3334
BlockStateCache	ApplyStateWrites — :3140

So BSC.current holds nothing the version map does not already hold, and BSC.writeLog holds nothing blockIO does not already hold.

The wiring routes around the version map on purpose

newFinalizeState gives the finalize IBS a fresh empty version map — ibs.SetVersionMap(state.NewVersionMap(nil)) (:3290) — so its reads resolve only through the reader chain, which means through BlockStateCache. Twenty lines later in the same function, writes.Normalize(be.versionMap, finalVersion.TxIndex, ...) (:3342) does consult the real version map. The version map is already trusted for normalization but not for the IBS's reads.

Worker reads never needed BSC in the first place: read_paths.go:474-488 tries versionMap.Read* first and falls back to the state reader only when no prior tx wrote the key, so the state reader is the pre-block base by design.

What removal would take

1. Pass be.versionMap at finalVersion.TxIndex to the finalize IBS instead of an empty map. Same for the consumers of lazyStateReader (:2766 — fee crediting and normalization).
2. Replace Flush's writeLog replay with a replay over blockIO's per-tx write sets in tx order.

Two hazards

• The comment at :3273 warns that "reading around the block cache lets the finalize stomp an in-block update". The version-map path has to close that too, and the two stores are not shaped alike: BSC.current is keyed by address with last-write-wins, while the version map is keyed by (addr, path, key, txIdx) with incarnations and a separate self-destruct / revival path (read_paths.go:385-403). Reading at txIndex = last should agree, but self-destruct and revival are where it would not.
• writeLog reproduces the serial executor's IBS dedup — the DeleteAccount short-circuit at rw_v3.go:1189 that emits one DomainDel per address per block. A blockIO replay has to reproduce it, or the domain history stops matching serial, which is what Flush's docstring says it is matching.

Effect on sequencing

• This no longer depends on the sd.mem / latestStateLock work. The version map replaces BSC for reads, so worker read cost is not the gating factor.
• It does make execution/state: make BlockStateCache committed storage reads lock-free #23134 moot — that PR makes the committed tier's storage reads lock-free, and this removes the tier. Worth deciding the direction before it merges.

[sudeepdino008]

BlockStateCache --

• it's cleared at end of every block exec.
• it's got a "current" state i.e. accumulated writes collected from finalized Tx
• it's got a committed state i.e. values at the start of block execution, which get read as execution goes on.
• the state reader for tx workers is created from this: BlockStateCache + sd.mem + sd.parent.mem + StateCache + temporaldb
  • note that versionedio is already layered on top of this by txworkers.
• BlockStateCache - flushes to sd.mem after every block exec, and a new one used for next block.
• from sd.mem - things can be flushed to db based on batched execution or chaintip execution.

StateCache is the inter-block cache; we can probably get rid of BlockStateCache by:

• remove committed state from BlockSTateCache; continue accumulating current data in BSC though (for the 3 apply steps - Fee crediting, normalizationand engine.Finalize)
• create a reader for workers on: in.mem + STateCache + temporaldb... (first layer is versionedio) -- we can remove BlockSTateCache in state reader, because the other layers already cover those data
• get the 3 apply steps to use the duplicated info in versionedio instead -- this will probably help us get rid of BlockStateCache completely.

---

note that it's not required for tx workers to have the tx ApplyState available after each finalized tx.