[r3.6] execution, db: bind StateCache fills to transaction views and reject stale fills - #23031
Merged
Conversation
…stale fills (#22444) Closes #22356 — the forward stale-fill direction: a fill from a read view older than the latest apply is rejected. The reverse direction around unwind is tracked in #22463 and stays open (details in the scope section). `StateCache` is one process-global in-memory cache of the **latest committed values** of accounts, storage and code. It exists to skip the file-accessor/MDBX stack on repeated `GetLatest` reads. It is not a snapshot: per key it holds one value — the newest committed value known to the process, whether applied by execution or filled from a read. It is on by default; `USE_STATE_CACHE=false` constructs no cache at all (readers go straight to the backing tx), and `STATE_CACHE_FILLS=false` disables every reader write (apply-only mode) — the kill switch and the A/B lever for measuring what fills contribute. | Actor | Reads | Writes | When | |---|---|---|---| | Canonical execution (`SharedDomains.Commit`) | via `GetLatest` | **apply**: put committed updates, physically delete deletions | after the RwTx commit succeeds — `sd.Commit` walks the pending updates into the cache only once `tx.Commit` returns | | Embedded RPC (`execmodule.CacheView` → `SharedDomains.AsGetter`) | via `GetLatest` | **fill**: on a miss, offer the value read from its own tx | during any request at `latest` | | Read-ahead warmup (`blocks_read_ahead`) | — | **fill**: same, to pre-warm keys for exec | while exec runs | A **fill** is a cache write performed on behalf of a *database* reader (fill-on-miss); an **apply** is the authoritative post-commit write from `SharedDomains.Commit`. A **read view** is what a read-only temporal tx sees (its MDBX read tx + pinned files view); its **frontier** is the exclusive txNum end of what it can see — a view with frontier N sees txNums < N. This sharing — including the RPC fills — is `main`'s code today (wiring: `node/eth/backend.go` hands `execmodule.Cache` to the embedded rpcdaemon as its kvcache; `CacheView.Get` → `SharedDomains.AsGetter` → `getLatestMetered`, which consults and fills the shared cache). The PR does not add the sharing; it adds the missing ordering. 1. **Reader** begins a read-only tx; its frontier is N. 2. **Exec** at txNum M >= N deletes key K and commits; the post-commit apply empties K's slot. 3. **Reader** reads K: cache miss → reads its own older tx → gets the pre-delete value → its fill re-inserts it (the slot is empty, so `PutIfAbsent` lands). 4. Anyone's next `GetLatest(K)` serves the deleted value from the cache. Each step is consistent inside its own tx. The bug is the unordered hand-off between two txs through the shared cache — no per-tx consistency property can prevent it. ```mermaid sequenceDiagram participant Reader as Reader with frontier N participant Cache as shared StateCache participant Exec as Exec commit Exec->>Cache: apply at txNum M >= N. delete K, appliedEnd = M+1 Reader->>Cache: GetLatest(K). miss, the slot is empty Reader->>Reader: falls back to its own older tx. pre-delete value Reader->>Cache: fill(K, pre-delete value) Note over Cache: main. PutIfAbsent lands, K is resurrected Note over Cache: this PR. rejected, frontier N < appliedEnd ``` `StateCache` has no data methods (like `db`). Access goes through two handles (like `tx`): - **`ReadView`** — reads plus admission-gated fills, bound to one tx's read view; it must not outlive the tx. `SharedDomains.AsGetter*(tx)` builds one per getter; the plain `GetLatest` wrappers use a frontier-less view and bind a frontier only on the miss path. The view resolves its frontier internally, so a fill can no longer be paired with another tx's frontier. - **`Applier`** — `Apply` / `Unwind` / `Clear`. Used by the authoritative `SharedDomains` commit/unwind paths (every SD holds a handle; the operations serialize under the admission lock). The cache tracks, per domain, the exclusive end of what has been applied (`appliedEnd`), guarded by one `RWMutex`: - `Applier.Apply` (write lock) advances `appliedEnd` and mutates the cache in the same critical section. Applying txNum X records `appliedEnd = X+1`. - `ReadView.Fill` (read lock) rechecks its view's frontier and inserts before releasing: admitted iff `frontier >= appliedEnd`. Fills are put-if-absent, so they never replace a live authoritative entry. Ordering proof, two cases: if the fill takes the lock first, the later apply overwrites or deletes it; if the apply takes the lock first, the stale fill is rejected. Equality is safe because a view whose frontier equals the applied end has seen the apply. Checking freshness only at view creation would not be enough — an apply can land between the check and the fill — which is why admission happens at fill time, under the same lock applies take. `DomainVisibleEnd` reports the exact exclusive frontier of a tx's domain read view: a file covering `[0,N)` reports N; a hot-DB view containing txNum N reports N+1. Views without an exact frontier answer `ok=false` and never fill — remote and history-disabled backends, and dependency-clamped values views (their reads mix ages, and the clamped-away state can appear later without any apply); such reads still work, they just skip cache population. Two memos, one per path: read-only temporal txs memoize the frontier per domain in a tx-local cache, reset only by `ForceReopenUnderlyingFilesTx` (which can only extend it); the writable path memoizes in `SharedDomains`, reset at flush and on `ViewID` change. A compile-time assert guards each memo's bitmask against domain-count growth. Positive entries are stamped with their step-derived txNum bound. A negative entry is stamped with the last txNum its read view included, `max(frontier-1, 0)`: since `Unwind(N)` treats N as the first rolled-back txNum, the negative survives `Unwind(N)` and is invalidated by `Unwind(N-1)`. Admission safety needs one more invariant: a view's frontier never decreases in a process that fills the cache (the DB component is frozen at tx begin; a files reopen only extends it). This is enforced, not just documented: wiring a fill-enabled `StateCache` forbids visibility lowering on that aggregator, and `recalcVisibleFiles` panics on the one transition that breaks admission — a cached state domain's visible end decreasing — whichever entry point causes it. Raising visibility (unaligning a lagging entity) stays allowed, and apply-only caches (`STATE_CACHE_FILLS=false`) skip the forbid: with no fills there is nothing for a lowered frontier to poison. `Apply` physically deletes account, storage and code entries; on the authoritative side absence is represented by absence, with no deletion markers (a marker can be evicted, re-opening the fill window). Reader-side negative entries do exist — they are ordinary admission-gated fills, stamped as below. An account update invalidates the derived address→codeHash mapping; an account deletion also removes the address's code binding; an account-only deletion does not advance the code-domain frontier (that would suppress valid code fills for unrelated contracts). On authoritative code apply, bytes are cloned before hashing so cached code and its codeHash cannot diverge. `SeedAddrCodeHash` accepts only view-sourced account records (a cache-sourced record can lag the latest apply and would carry pre-apply state past the gate); a read error does not seed the zero-hash sentinel. - **Reads are not snapshot-isolated**: a `ReadView` hit can be newer than the view's tx. That is the same direction the Overlay already serves (embedded RPC at `latest` reads exec's newest state on purpose), and a single-version LRU cannot give stable per-view reads without becoming a second kvcache — `node/shards` already provides that model for the remote daemon. The cache's contract, in the forward direction, is monotonicity: content never regresses behind the applied frontier; unwinds invalidate by epoch and floor. - **The reverse direction around unwind** — a view opened before an unwind refilling a value from the discarded fork — is tracked separately in #22463. - Cache hits are unchanged: no frontier query, no admission lock. - The memoized frontier read is `~2.4 ns/op`, 0 allocs (Apple M2 Max); the memo adds `+64 B/op` to a read-only temporal tx (`BenchmarkBeginTemporalRo/WithBlockSnaps`; absolute numbers drift with unrelated changes on `main`). - Getter paths (exec workers, RPC) build their frontier-carrying `ReadView` once per getter; frontier-less views elsewhere (`GetCode`/`GetCodeSize` fast paths) are by-value and allocation-free, and a frontier is bound — one allocation — only on cold fill/seed paths. A cold negative fill through the plain `GetLatest` wrappers is `~0.3 µs/op`, 48 B/op, 2 allocs: one allocation is the fill itself, one is binding the frontier on the miss path — both amortize against the backing read they follow (the no-cache baseline of the same read is `~0.1 µs`). Production-path reproductions through embedded RPC (`execmodule.Cache.View` over a published `SharedDomains` overlay and a real temporal DB): account, storage and code resurrection from an old RPC read view, and code-of-a-deleted-account refill (pinning both the SD-level paired deletion and the cache-level accounts-frontier check on code fills). Plus: admission at the exact frontier boundary and a 20,000-round concurrent apply/delete vs fill interleaving; physical deletion and derived code-mapping invalidation; account-only deletion not suppressing unrelated code fills; the address→codeHash mapping seeding only from view-sourced records; exact file/DB frontier calculation and memo re-derivation after files-view reopen and after flush; read-ahead freshness, no-exact-frontier backends never filling; negatives surviving `Unwind(N)` but not `Unwind(N-1)`; the admission frontier surviving `Clear`; the `STATE_CACHE_FILLS=false` apply-only switch; `USE_STATE_CACHE=false` constructing no cache; node close releasing every cache-budget reservation (a full EngineApiTester lifecycle); and the visibility-lowering assert. All cross-package tests exercise the public `ReadView`/`Applier` API. The three resurrection tests use no API introduced by this PR: ported unchanged to `main`, all three fail with the resurrected values served — they pin the bug, not the refactor. Verified with `go test` (and `-race`) on `execution/cache`, `execution/exec`, `db/state/execctx`, `db/kv/temporal`, full-tree build, benchmarks with `-benchmem`, and repeated clean `make lint` runs. Suggested order: 1. `execution/cache/cache.go` (package doc) and `execution/cache/view.go` — the contract and the API: `Frontier`, `ReadView`, `Applier`. Small files; everything else implements them. 2. `execution/cache/state_cache.go` — admission internals: `appliedEnd`, `fillIfFresh`, `apply`. 3. `db/state/execctx/domain_shared.go` — wiring: getters hold a `ReadView`, commit/unwind hold the `Applier`; `getLatestMetered` shows the full read chain (mem → cache → backing tx → fill). 4. `execution/exec/blocks_read_ahead.go` — the read-ahead fill path. 5. `db/state/execctx/statecache_rpc_integration_test.go` — the resurrection reproduced end-to-end through the embedded-RPC path, and the fix pinned. --------- Co-authored-by: Alexey Sharov <askalexsharov@gmail.com>
yperbasis
requested review from
AskAlexSharov,
mh0lt and
sudeepdino008
as code owners
August 5, 2026 13:58
yperbasis
enabled auto-merge
August 5, 2026 13:59
AskAlexSharov
approved these changes
Aug 6, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cherry-pick of #22444 to release/3.6.
r3.6-specific adaptations
cmd/integrationon release/3.6 has no state-cache wiring (execBlocksBatchtakes no cache), so the PR's three integration-wiring lines were dropped in conflict resolution — the integration tool runs cache-less through that path, which needs no admission gate.