feat: warmup_test_payload phase between setup and test - #221
Open
skylenet wants to merge 6 commits into
Open
Conversation
Adds an optional warmup phase that takes each engine_newPayload* call from
the test step, replaces stateRoot with a fork-specific placeholder, and
recomputes blockHash so the EL accepts the header and starts execution.
The expected outcome is a state-root rejection, but the work the client
performs first warms its caches before the real test runs.
Configured via runner.client.config.warmup_test_payload.{enabled, fork}
with the standard global/instance override pattern. Currently only
fork=osaka is supported. Block-hash recomputation is delegated to
go-ethereum's engine.ExecutableDataToBlockNoHash.
Warmup results write under <test>/warmup.* and surface in result.json
and per-suite stats without folding warmup gas/time into test totals.
The resolved value lands in the run's config.json and the UI's run
detail and compare views.
…mup-impl # Conflicts: # pkg/config/config_test.go # ui/src/components/compare/ConfigDiff.tsx
Adds runner.client.config.warmup_test_payload.count to control how many times each engine_newPayload* line is sent during the warmup phase. Defaults to 1; must be >= 1 when enabled. Each iteration uses a different stateRoot deterministically derived as keccak256(salt || uint64BE(iteration)), so the client treats the calls as distinct payloads (different blockHashes) instead of de-duplicating them. The hardcoded OsakaWarmupStateRoot constant becomes the salt input (renamed OsakaWarmupSalt) rather than the literal stateRoot. Generator.Transform now returns a slice: non-newPayload lines (FCUs, etc.) come back as a single element regardless of count; newPayload lines expand to count variants with distinct stateRoots and recomputed blockHashes. Resolved instance config.json, run-detail UI card, compare diff row, and docs all show the count value.
Adds runner.client.config.warmup_test_payload.method to select the warmup strategy. Today only "invalid-stateroot" (the existing stateRoot-rewrite + blockHash-recompute behavior) is supported; the field is in place so future strategies can be added without breaking existing configs. EffectiveMethod() returns "invalid-stateroot" when the field is unset/empty so existing configs keep working unchanged. The validator rejects any non-empty method other than "invalid-stateroot". The runWarmupStep dispatch now switches on EffectiveMethod(); the invalid-stateroot branch wraps the existing generator code. UI run-detail card and compare diff render the method value; docs document the field, its default, and explicitly note that warmup_test_payload supports both global and per-instance config.
# Conflicts: # docs/configuration.md # pkg/config/config.go # pkg/runner/lifecycle.go # pkg/runner/runner.go # ui/src/api/types.ts # ui/src/components/run-detail/RunConfiguration.tsx
Adds a second warmup mutation strategy that keeps stateRoot intact and instead subtracts (1+i) from gasUsed for iteration i, then recomputes blockHash so the client still does a real payload-decode and tx pass before rejecting on the gas mismatch. Iteration 0 = original-1, iteration 1 = original-2, and so on, which preserves the "subtract 1 from the original" semantics for the common count=1 case while keeping every iteration distinct for count>1. The Generator now takes a Method parameter and dispatches the per- iteration mutation; underflow on gasUsed surfaces as an error so we don't silently send malformed payloads. The config validator accepts either method; the executor's switch passes both through to the same generator pipeline.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds an optional warmup phase that runs between setup and test. For each
engine_newPayload*call in the test step, the runner sends a modified copy withblockHashrecomputed so the EL still does real payload decoding, transaction validation, and (depending on the chosen method) state-root execution before rejecting on the mismatch. The value is in that work — cache fills and codepath warming — not in the eventual rejection.Two mutation strategies are supported, selected via
method:invalid-stateroot(default): replacesstateRootwith a deterministic placeholder derived from a fork-specific salt and the iteration index, then recomputesblockHash. EL typically rejects on state-root mismatch after executing the block.invalid-gasused: keepsstateRootintact and subtracts(1+i)fromgasUsedfor iterationi(so iteration 0 = original-1, iteration 1 = original-2, …), then recomputesblockHash. EL typically rejects once it notices the gas mismatch.countcontrols how many times eachengine_newPayload*line is sent. Each iteration produces a distinct payload (per-iteration salted stateRoot, or per-iteration gasUsed decrement) so the client treats the calls as distinct. Non-newPayload lines (e.g.engine_forkchoiceUpdated*) are sent once regardless.Configured via:
Currently only
fork: osakais supported. Standard global/instance override pattern applies — instance-level config (when non-nil) fully replaces the global default.pkg/warmuppackage: rewrites engine_newPayload* JSON-RPC lines via a method-dispatched mutator, recomputesblockHashusing go-ethereum'sengine.ExecutableDataToBlockNoHash. Non-newPayload lines pass through unchanged.invalid-gasusedreturns an error ifcountwould underflow the originalgasUsed.WarmupTestPayloadConfig(withMethod,Count,Fork,Enabled),GetWarmupTestPayload(instance)getter, validator that requiresfork == "osaka", accepts either method, and rejects negative counts.StepTypeWarmup, runs after setup / before test. Results land at<test>/warmup.{response,result-details.json,result-aggregated.json}and surface inresult.jsonand per-suite stats. Warmup gas/time is recorded but not folded into the test totals so test metrics stay comparable across warmup-on/off runs.ExecuteOptionscall sites (lifecycle + container + checkpoint) and the resolved value lands in the run'sconfig.jsonunderinstance.warmup_test_payload.method,fork,count) and as a row in the compareConfigDiff(with "differs" highlighting).docs/configuration.mdupdated with a Warmup Test Payload section covering both methods + entries in the global and instance-override tables.New direct dep:
github.com/ethereum/go-ethereum v1.17.2.Test plan
go test ./...passes (incl. newpkg/warmuptests for both methods, count-produces-distinct-variants, underflow guard, andpkg/configtests for getter + validator)golangci-lint run --new-from-rev="origin/master"cleantsc -bandeslintclean for the UI changesmethod: invalid-stateroot, fork: osaka: confirm warmup payloads are sent and the client reaches state-root execution before rejecting (vs. rejecting at hash check)method: invalid-gasused, fork: osaka: confirm warmup payloads are sent and the client reaches gas-validation before rejecting; verifycount=Nproduces N distinct payloads per newPayload<test>/warmup.*files are written andresult.jsoncarriessteps.warmupmethodandcount, and the compare page row reflects differences across runs