Skip to content

feat: warmup_test_payload phase between setup and test - #221

Open
skylenet wants to merge 6 commits into
masterfrom
warmup-impl
Open

feat: warmup_test_payload phase between setup and test#221
skylenet wants to merge 6 commits into
masterfrom
warmup-impl

Conversation

@skylenet

@skylenet skylenet commented Apr 28, 2026

Copy link
Copy Markdown
Member

Summary

Adds an optional warmup phase that runs between setup and test. For each engine_newPayload* call in the test step, the runner sends a modified copy with blockHash recomputed so the EL still does real payload decoding, transaction validation, and (depending on the chosen method) state-root execution before rejecting on the mismatch. The value is in that work — cache fills and codepath warming — not in the eventual rejection.

Two mutation strategies are supported, selected via method:

  • invalid-stateroot (default): replaces stateRoot with a deterministic placeholder derived from a fork-specific salt and the iteration index, then recomputes blockHash. EL typically rejects on state-root mismatch after executing the block.
  • invalid-gasused: keeps stateRoot intact and subtracts (1+i) from gasUsed for iteration i (so iteration 0 = original-1, iteration 1 = original-2, …), then recomputes blockHash. EL typically rejects once it notices the gas mismatch.

count controls how many times each engine_newPayload* line is sent. Each iteration produces a distinct payload (per-iteration salted stateRoot, or per-iteration gasUsed decrement) so the client treats the calls as distinct. Non-newPayload lines (e.g. engine_forkchoiceUpdated*) are sent once regardless.

Configured via:

runner:
  client:
    config:
      warmup_test_payload:
        enabled: true
        fork: osaka
        method: invalid-stateroot   # or invalid-gasused; defaults to invalid-stateroot
        count: 3                    # optional; defaults to 1

Currently only fork: osaka is supported. Standard global/instance override pattern applies — instance-level config (when non-nil) fully replaces the global default.

  • New pkg/warmup package: rewrites engine_newPayload* JSON-RPC lines via a method-dispatched mutator, recomputes blockHash using go-ethereum's engine.ExecutableDataToBlockNoHash. Non-newPayload lines pass through unchanged. invalid-gasused returns an error if count would underflow the original gasUsed.
  • Config: WarmupTestPayloadConfig (with Method, Count, Fork, Enabled), GetWarmupTestPayload(instance) getter, validator that requires fork == "osaka", accepts either method, and rejects negative counts.
  • Executor: new StepTypeWarmup, runs after setup / before test. Results land at <test>/warmup.{response,result-details.json,result-aggregated.json} and surface in result.json and per-suite stats. Warmup gas/time is recorded but not folded into the test totals so test metrics stay comparable across warmup-on/off runs.
  • Runner: warmup config plumbed into all three ExecuteOptions call sites (lifecycle + container + checkpoint) and the resolved value lands in the run's config.json under instance.warmup_test_payload.
  • UI: rendered in the run-detail Configuration panel (shows method, fork, count) and as a row in the compare ConfigDiff (with "differs" highlighting).
  • Docs: docs/configuration.md updated with a Warmup Test Payload section covering both methods + entries in the global and instance-override tables.

New direct dep: github.com/ethereum/go-ethereum v1.17.2.

Test plan

  • go test ./... passes (incl. new pkg/warmup tests for both methods, count-produces-distinct-variants, underflow guard, and pkg/config tests for getter + validator)
  • golangci-lint run --new-from-rev="origin/master" clean
  • tsc -b and eslint clean for the UI changes
  • End-to-end with method: invalid-stateroot, fork: osaka: confirm warmup payloads are sent and the client reaches state-root execution before rejecting (vs. rejecting at hash check)
  • End-to-end with method: invalid-gasused, fork: osaka: confirm warmup payloads are sent and the client reaches gas-validation before rejecting; verify count=N produces N distinct payloads per newPayload
  • Verify <test>/warmup.* files are written and result.json carries steps.warmup
  • Verify the run-detail page shows the Warmup Test Payload card with the correct method and count, and the compare page row reflects differences across runs

Adds an optional warmup phase that takes each engine_newPayload* call from
the test step, replaces stateRoot with a fork-specific placeholder, and
recomputes blockHash so the EL accepts the header and starts execution.
The expected outcome is a state-root rejection, but the work the client
performs first warms its caches before the real test runs.

Configured via runner.client.config.warmup_test_payload.{enabled, fork}
with the standard global/instance override pattern. Currently only
fork=osaka is supported. Block-hash recomputation is delegated to
go-ethereum's engine.ExecutableDataToBlockNoHash.

Warmup results write under <test>/warmup.* and surface in result.json
and per-suite stats without folding warmup gas/time into test totals.
The resolved value lands in the run's config.json and the UI's run
detail and compare views.
…mup-impl

# Conflicts:
#	pkg/config/config_test.go
#	ui/src/components/compare/ConfigDiff.tsx
Adds runner.client.config.warmup_test_payload.count to control how many
times each engine_newPayload* line is sent during the warmup phase.
Defaults to 1; must be >= 1 when enabled.

Each iteration uses a different stateRoot deterministically derived as
keccak256(salt || uint64BE(iteration)), so the client treats the calls
as distinct payloads (different blockHashes) instead of de-duplicating
them. The hardcoded OsakaWarmupStateRoot constant becomes the salt
input (renamed OsakaWarmupSalt) rather than the literal stateRoot.

Generator.Transform now returns a slice: non-newPayload lines (FCUs,
etc.) come back as a single element regardless of count; newPayload
lines expand to count variants with distinct stateRoots and
recomputed blockHashes.

Resolved instance config.json, run-detail UI card, compare diff row,
and docs all show the count value.
Adds runner.client.config.warmup_test_payload.method to select the
warmup strategy. Today only "invalid-stateroot" (the existing
stateRoot-rewrite + blockHash-recompute behavior) is supported; the
field is in place so future strategies can be added without breaking
existing configs.

EffectiveMethod() returns "invalid-stateroot" when the field is
unset/empty so existing configs keep working unchanged. The validator
rejects any non-empty method other than "invalid-stateroot".

The runWarmupStep dispatch now switches on EffectiveMethod(); the
invalid-stateroot branch wraps the existing generator code.

UI run-detail card and compare diff render the method value; docs
document the field, its default, and explicitly note that
warmup_test_payload supports both global and per-instance config.
skylenet added 2 commits May 11, 2026 16:24
# Conflicts:
#	docs/configuration.md
#	pkg/config/config.go
#	pkg/runner/lifecycle.go
#	pkg/runner/runner.go
#	ui/src/api/types.ts
#	ui/src/components/run-detail/RunConfiguration.tsx
Adds a second warmup mutation strategy that keeps stateRoot intact and
instead subtracts (1+i) from gasUsed for iteration i, then recomputes
blockHash so the client still does a real payload-decode and tx pass
before rejecting on the gas mismatch. Iteration 0 = original-1,
iteration 1 = original-2, and so on, which preserves the
"subtract 1 from the original" semantics for the common count=1 case
while keeping every iteration distinct for count>1.

The Generator now takes a Method parameter and dispatches the per-
iteration mutation; underflow on gasUsed surfaces as an error so we
don't silently send malformed payloads. The config validator accepts
either method; the executor's switch passes both through to the same
generator pipeline.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant