Investigate how much RPC↔exec state sharing to keep: STATE_CACHE_FILLS default, RPC cache reads, and the SharedDomains overlay

[yperbasis]

How much RPC↔execution state sharing should erigon keep on by default? This covers the STATE_CACHE_FILLS default and, prompted by #22444 (comment), the wider concern behind it: whether the embedded RPC daemon should use the shared StateCache and the global SharedDomains overlay at all.

The sharing has three separable layers, with different value claims and different risks:

Layer	What it is	Value claim	Risk/cost
1. RPC/reader fills into the shared cache	readers write the cache on miss (admission-gated since #22444)	warms read-mostly keys (hot storage, popular code) that applies can never cache	the entire stale-fill bug family (#22356, #22463, #23028); admission machinery on the hot read path (~48 B / 2 allocs per cold fill, exact-frontier lookups)
2. RPC reads from the shared cache	RPC-originated getters consult the cache and serve hits (latest-wins, possibly newer than the request's tx)	skips the file-accessor/MDBX stack on repeated reads	latest-wins semantics for RPC; ties RPC read behavior to exec's cache contents
3. The SharedDomains overlay in RPC	execmodule.Cache.View hands RPC a getter over exec's current/published SD	embedded-RPC latest sees the newest (even uncommitted, mid-FCU) state	RPC reads coupled to exec internals (published-SD lifetime, background-commit windows — the #21613/#23003 family); the "premature design" objection

Layer 1 — should STATE_CACHE_FILLS default to false (apply-only cache)?

Apply-only mode (introduced in #22444) disables every reader write: admission-gated fills, addr→codeHash seeds, code-size fills. Applies from SharedDomains.Commit remain the only writer.

Why consider it:

• Every bug in this family is a fill bug. execution/cache: stale read-view fills can resurrect deleted state #22356 (resurrection), execution/cache: stale-fill admission is one-sided — pre-unwind read views refill dead-fork values, and unwound keys bypass the flush cache-apply #22463 (unwind readmission), StateCache: fills can go stale when a snapshot download extends file visibility with never-applied state #23028 (publication staleness) all poison the cache through reader fills; applies are authoritative by construction. Apply-only eliminates the class rather than gating it.
• Fills may pay less than assumed. Parallel-exec workers hold one read tx per run, so their fills self-starve after the first mid-run commit. With frozen-block catchup wired into the apply stream (execution/execmodule: publish frozen-block commits to StateCache #23033), execution keeps the cache warm for every written key. The residual value is warming for read-mostly keys — only fills can cache those.

This was the explicit open crux of the #22444 review (the admission gate vs. dropping reader fills entirely); the gate was built, the default deferred pending measurement.

Layers 2+3 — should the RPC daemon use the shared cache and overlay at all?

The comment linked above argues the sharing is unnecessary and premature, and asks for a feature flag disabling global shared domains and caching in the RPC daemon. Mechanically that flag is small: CacheView already has fallback paths for the nil-SD case, so an RPC-isolation mode = bypass the overlay getter and the cache, read straight through the plain temporal tx.

The two halves need separate verdicts:

• Cache reads (layer 2) are a pure perf claim — measurable (see below). If RPC-originated cache hits are rare or cheap to lose, disabling cache use for RPC costs little and decouples RPC from exec's cache entirely.
• The overlay (layer 3) is semantics, not perf: without it, embedded-RPC latest lags execution until commit. That gap is exactly what the flush+commit-online simplification (the same PR-22444 discussion thread) shrinks — with synchronous per-FCU commits, latest≈committed most of the time, and disabling the overlay becomes semantically cheap. Verdict on layer 3 should therefore be coordinated with that work, not taken standalone.

Measurements the decisions need

1. Interleaved A/B on execution throughput — eest benchmark shards and/or a tip-following node, STATE_CACHE_FILLS on vs off (an earlier single-pass attempt was discarded: run-order cold-cache artifact, env propagation unverified).
2. RPC latency A/B at latest on an RPC-serving node: (a) fills on vs off; (b) cache reads on vs off; (c) overlay on vs off (correctness caveat above).
3. Fill-outcome counters over a real sync (PrintStatsAndReset: admitted / rejected / no-frontier, execution/execmodule: publish frozen-block commits to StateCache #23033) plus hit ratios: if admitted fills are a small fraction of attempts, or fill-originated entries rarely produce hits, layer 1's value collapses.
4. Read-ahead interaction: STATE_CACHE_FILLS=false also silences BlockReadAheader warming; the execution: read-ahead warmup must not clobber fresher StateCache entries #22146 lineage suggests warming mattered there — quantify.

Possible outcomes (not mutually exclusive)

• Flip STATE_CACHE_FILLS default to false — the machinery stays as an opt-in lever; execution/cache: stale-fill admission is one-sided — pre-unwind read views refill dead-fork values, and unwound keys bypass the flush cache-apply #22463 drops in urgency (guards an opt-in path); release lines can adopt the flipped default as a mitigation posture (3.4/3.5 lack the admission gate).
• Add an RPC-isolation flag (no overlay, no cache for RPC-originated reads), default per measurement; coordinate the overlay half with flush+commit-online.
• Keep defaults as-is with the measurements as evidence that the sharing pays.

[VBulikov]

yperbasis added Imp2 as Importance

[VBulikov]

yperbasis changed Importance from Imp2 to Imp1