execution/cache: stale-fill admission is one-sided — pre-unwind read views refill dead-fork values, and unwound keys bypass the flush cache-apply

[yperbasis]

Summary

#22444 linearizes cache-fill admission and closes #22356 for the forward direction: a read view older than the last committed apply cannot fill. The unwind direction remains open, because the freshness gate cannot express it and the unwind flush never reaches the cache-apply. Around a reorg, a dead-fork value can enter the shared StateCache as a live entry and be served as canonical after the unwind completes, until the key is rewritten by the new chain, dropped by a later unwind floor, or evicted.

Found during the #22159 review (the in-flight-unwind read-fill analysis); scoped out of both #22159 and #22444.

Mechanics

Three cooperating gaps. Only delete-shape unwound keys (created on the dead fork, Value == nil in the unwind changeset) are exposed — restore-shape keys are served from the mem overlay during the window and their stale cache entries drop lazily via epoch/floor afterwards.

1. In-flight fill of a dying row. sd.Unwind bumps the cache epoch and StateCache unwind lowers appliedEnd to the unwind point. A read of a delete-shape unwound key then misses in mem (which publishes only a per-key maxStep bound), and the maxStep-bounded MDBX read returns the not-yet-deleted dying row (the bound is inclusive; TestReadFill_DoesNotClobberLiveEntry on #22159 pins exactly this read). The fill is admitted — the reader's frontier is pre-unwind, far above the lowered appliedEnd, so frontier >= appliedEnd passes — and the entry is stamped with the current post-bump epoch: a live cache entry holding dead-fork bytes. The linearized admission cannot help: the exclusive Unwind completed before the fill's shared section began, so there is no interleaving to forbid.

During the in-flight window this is even path-consistent (the cache-disabled read returns the same dying row). The problem is persistence past the flush:

2. The unwind flush never corrects the cache. SharedDomains.Commit's cache-apply iterates pending, collected by WithFlushCallback from the forward overlay (sd.domains / sd.storage). The unwind replay runs via tx.Unwind(unwindChangesetRaw) inside flushLocked, before and outside the callback pass, so unwound keys get neither a physical Delete nor a restore Apply. After the flush MDBX serves the restored state while the cache still serves the dead-fork value.

3. Post-flush fills from pre-unwind read views. Any still-open pre-unwind read-only tx (an embedded-RPC view on the published overlay — the reader #22444's integration test wires up) still sees the dying row via MVCC and passes the freshness gate, because its frontier exceeds the lowered appliedEnd. The window closes only when the new canonical chain's Apply calls raise the applied end past that view's frontier — roughly unwind → first post-reorg flush cache-apply.

Why the #22444 gate cannot catch this

The admission predicate answers "is the read view at least as new as the last apply?" — after a rewind, newer is precisely what is wrong. Unwind lowering appliedEnd makes every pre-unwind read view pass by construction (#22357's design note — "a pre-delete read view is by construction behind the delete-commit" — inverts for unwinds). Eviction-independence does not help either: no marker for an unwound key is ever written in the first place (gap 2).

Fix directions

1. Cheap, closes gap 1: skip the read-fill in getLatestMetered when the mem overlay bounded the key (maxStep != math.MaxUint64) — the same per-key in-flight-unwind signal the ASSERT_STATE_CACHE gate uses. A bounded read's result is transient by definition, so caching it is at best useless warmup.
2. Principled, closes gaps 1+3: make admission two-sided — also reject frontier > appliedEnd — now expressible in one place under admissionMu. Requires seeding Clear with the current domain progress rather than 0, else a cleared cache rejects every fill until the first flush-apply.
3. Full closure, also gap 2: route the unwind replay's per-key deletes/restores through Apply, which needs tx.Unwind / mem-batch flush-callback plumbing so unwound keys reach the cache-apply.

Direction 2 composes with #22444's design cleanly; direction 1 is worth doing regardless as a one-line hardening.

Severity

Requires a reorg plus a concurrent reader of a fork-created key (embedded RPC on the published overlay, or the FCU flow's own reads inside the window). The result is dead-fork account/storage/code state served as canonical from the cache after the unwind — the same class as #22356, concentrated in exactly the windows where the affected keys are hottest.

Refs: #22444, #22356, #22357, #22159, #22120.

[VBulikov]

yperbasis added Imp2 as Importance

[yperbasis]

Step-0 blind spot in the per-key maxStep signal — affects fix direction one here and the ASSERT_STATE_CACHE gate from #22467.

The mem overlay publishes the bound as a miss-with-step: for a delete-shape entry unwoundLatest returns (nil, keyStep, false) (temporal_mem_batch.go), and getLatestMetered accepts it only when step > 0, because a plain miss is also (nil, 0, false). A delete-shape whose encoded step is 0 is therefore indistinguishable from "no signal" and publishes no bound. For such keys during the in-flight window:

• the db, execution: harden StateCache read-fills against stale overwrites and unwinds #22467 assert skip doesn't engage — the divergence assert compares against an unbounded read that can still return the dying step-0 row, the same false-panic mechanism that PR fixed for steps ≥ 1. Concrete repro: a step-0 variant of TestAssertStateCache_NoFalsePanicDuringInFlightUnwind (single row written in step 0, delete-diff encoding ^0, cache holding the correct negative below the floor) panics on the pre- and post-db, execution: harden StateCache read-fills against stale overwrites and unwinds #22467 code alike;
• direction one's fill-skip (maxStep != math.MaxUint64) would likewise not engage, so the dying-row fill it targets stays admitted for step-0 keys;
• the state-cache read gate also stays disengaged, though this costs only path-consistency, not correctness: any entry reflecting a reverted step-0 write is stamped at or above the unwind floor, so the epoch floor drops it regardless.

Exposure is bounded on mainnet (step 0 is frozen in files) but not on young chains: with the default 1.56M step size an entire fresh devnet lives inside step 0, so every reorg there exercises the blind spot — which is also where the reorg-heavy test setups (kurtosis, hive) run.

Since the fill-skip, the assert gate, and the read gate all key off the same signal, the fix belongs in the signal rather than its consumers: let the overlay distinguish "no signal" from "bound at step 0" (an explicit bounded flag on the miss, or a kv.Step(math.MaxUint64) no-signal sentinel), after which direction one can be implemented without inheriting the gap.

[yperbasis]

The step-0 blind spot from the previous comment is closed on #22467 (c69bbc3): a plain mem-batch miss now returns kv.NoStepBound instead of 0, so a delete-shape bound at step 0 publishes normally and the assert gate, the read gates, and fix direction one all see it. TestAssertStateCache_NoFalsePanicDuringInFlightUnwindStepZero pins the signal. Direction one's fill-skip can now be implemented without inheriting the gap.