execution/cache: a failed RwTx commit leaves applied-but-never-committed state in StateCache

[yperbasis]

Problem

The cache-apply runs before the MDBX commit: SharedDomains.Commit flushes and applies the pending updates to the shared StateCache (advancing appliedEnd), and the caller then commits the RwTx (runForkchoiceFlushCommit in the FCU flow; SetHead and the integration/custom-trace commit paths have the same shape). If that commit fails and the process survives:

1. The DB rolls back to the pre-block state.
2. The cache keeps the post-block values, with appliedEnd advanced — applied-but-never-committed state, presented as the authority.
3. Erigon re-executes the same block. A fresh SharedDomains has an empty mem overlay, so reads hit the poisoned cache entries and execution runs on top of the block's own post-state — wrong results until a commitment mismatch forces an unwind (which does clean the poison: epoch bump + floor + lowered appliedEnd).
4. Embedded RPC at latest serves the never-committed values from cache hits in the meantime.

Fills are only over-rejected (new views sit below the inflated appliedEnd), so the fill-admission gate from #22444 is not bypassed — the gap is that the authority itself is untruthful after a failed commit.

This is the third shape in the #22356 family: not a stale fill (forward, fixed by #22444) and not a dead-fork refill (#22463), but applied-but-never-committed state that is neither unwound nor evicted.

Qualifiers

• Pre-existing: the flush-time cache writes predate execution, db: bind StateCache fills to transaction views and reject stale fills #22444 (that PR changed the fill side, not the apply timing), so this is main's exposure too.
• Severity is gated by recovery behavior: an MDBX commit failure at FCU is often fatal to the process, and a process death clears the in-memory cache. The hazard exists only where the node survives the error and retries.

Fix directions

1. Failed commit = unwind of the never-committed range (preferred near-term): on the commit-error path, call Applier.Unwind(pre-block txNum) — poison evicted via epoch/floor, appliedEnd lowered, admission consistent. Small and local to the commit call sites; needs a red test of the shape: apply → fail the commit → assert hits return nothing and appliedEnd is lowered.
2. Apply only after a successful commit (structural): moves the authority update behind durability. Interacts with the background-commit design in execution: FCU background-commit worker + generation-chain reader consistency #21414 (the apply would move to the bg worker and lag the next FCU, which chains reads through the mem overlay), so it needs joint design with that stack.

Refs: #22444 (fill admission and the apply/appliedEnd vocabulary), #22463 (unwind direction), #21414 (background commit widens the window where the ordering of apply vs commit matters).

[VBulikov]

yperbasis added Imp2 as Importance

[yperbasis]

Right — history/II rows ride ETL collectors into the RwTx (historyBufferedWriter.historyVals, the II writer's index/indexKeys) and become visible to other readers only at the MDBX commit; the RPC overlay path also has in-mem history reads disabled. Two consequences for this issue:

1. On a failed commit they are discarded with the tx — consistent with the rolled-back DB. So the StateCache is the only component that keeps the never-committed state: everything else self-heals by rollback, the cache alone needs the explicit Unwind. That sharpens the problem statement.
2. During the applied-but-not-committed window, embedded RPC visibility is mixed: latest (overlay + cache) runs ahead of history/II. In normal operation that is the overlay's by-design behavior, bounded by commit latency; after a failed commit it persists wrongly until an unwind or re-exec — one more reason to prefer fix direction 1 (treat the failed commit as an unwind), which restores cache↔history consistency immediately.

[yperbasis]

Closing as invalid — I had the ordering backwards. sd.Commit commits the RwTx first and walks the pending updates into the cache only after tx.Commit returns (domain_shared.go, the if err := tx.Commit(); err != nil { return err } immediately before the pending loop). Every production apply path routes through sd.Commit (FCU, SetHead, executor, custom trace, integration), and plain Flush deliberately never touches the cache. So a failed commit leaves the cache un-applied and consistent with the rolled-back DB — there is no applied-but-never-committed state.

Two things pointed at the truth: @AskAlexSharov's green guard test on #23005 (f0e1981835 — a reader ahead of appliedEnd must still fill, because commit-then-apply makes "ahead" the normal window), and, in hindsight, an earlier review suggestion by codex to describe applies as happening "after successful database commit," which I wrongly rejected. The PR #22444 description is corrected accordingly.

What survives from this issue: nothing needing a fix. The ETL/II observation above stands as further confirmation that everything rides the tx — including, it turns out, the cache applies themselves.