feat: Benchmarkoor - Nethermind CI (podman + CRIU memory checkpoints, per-test summary, dotTrace profiling) - #159
Merged
Merged
Conversation
…eckpoints) Integrates ethpandaops/benchmarkoor for Nethermind only, running on the stateful-generator self-hosted runner. - container_runtime: podman with rollback_strategy: container-checkpoint-restore by default for stateful suites: pre-run steps (gas-bump/funding) are baked into a CRIU memory checkpoint + ZFS datadir snapshot, then every test gets an instant rollback/restore instead of a full container recreate. - Bootstraps a file-backed zpool on the runner and seeds a dataset from /mnt/sda/<network>/nethermind when the snapshot is not already on ZFS (one-time rsync, reused across runs). - Tests/opcodes pulled from gas-benchmarks release archives (default amsterdam-repricings-v5.2.0, jochemnet), chainspec from scripts/genesisfiles/nethermind at the triggering commit. - Rollback strategy selectable (checkpoint-restore / container-recreate / rpc-debug-setHead / none) for A/B comparison; extra_run_config input deep-merges arbitrary benchmarkoor YAML last. - Results uploaded as workflow artifact + job summary via the ethpandaops/benchmarkoor composite action, which also installs podman/CRIU/crun and starts the rootful podman socket.
The jochemnet nethermind snapshot is ~900G; the fixed 500G pool filled at 57% of the seed copy. Pool size now defaults to 'auto' (snapshot + 15% + 32G) with a free-space preflight, existing pools are grown in place (sparse truncate + zpool online -e), and seed completion is tracked via the gb:seeded ZFS user property so interrupted copies resume instead of being mistaken for complete. Also add --Init.BaseDbPath=/data to the default Nethermind flags: the runner snapshots hold the nethermind_db content at their root, which the stock --datadir=/data would miss.
…nnet) Nethermind's BaseDbPath convention already contains the per-network dir (mainnet config uses nethermind_db/mainnet; our own scripts/nethermind/docker-compose.yaml uses /nethermind/data/mainnet). With --Init.BaseDbPath=/data the client created an empty DB next to the snapshot's mainnet/ dir and booted at genesis (chain head 0, endless SYNCING). Point it at /data/mainnet instead.
…nly-scenarios main's copy schedules the Amsterdam EIP set (missing 7954/7976/7981/8037) at 0x697D1108, which falls inside the snapshot's synthetic block range — the node then computes BAL hashes for stored blocks that have none and rejects them (InvalidBlockLevelAccessListHash). The corrected file activates 7708,7778,7843,7928,7954,7976,7981,8024,8037 at 0x697ddeff (head block 24402727 timestamp + 1), matching the chainspec ethpandaops use for this exact snapshot (verified params/genesis/accounts-identical to skylenet's devnet-6 gist).
Full Nethermind output is always written to runs/<run_id>/container.log in the results artifact; client_logs_to_stdout only duplicates it into the workflow log, which drowns the job output during the startup block replay. Re-enable per run via extra_run_config when live-debugging.
Needed to validate benchmarkoor branches (e.g. the restore_in_place fork) on the runner before they land upstream.
cancel-in-progress: true killed a 3h validation run when the next dispatch arrived on the same ref. Benchmarks should queue.
GitHub substitutes the default for explicitly-empty dispatch inputs, so passing benchmarkoor_image='' still pulled ghcr master and silently skipped the fork build. A non-empty git ref now implies image=''.
Stateful suites now get checkpoint-restore with restore_in_place out of the box (rollback auto already maps compute to 'none', so compute runs carry no checkpoint burden). Upstream benchmarkoor binaries ignore the option until ethpandaops/benchmarkoor#282 merges and simply use the export/import path; the fork ref enables the ~4s in-place restores now.
- Append a per-test throughput table (median/min/max + collapsible all-tests table) to the GitHub job summary, generated from the run's result.json by .github/scripts/benchmarkoor-pertest-summary.jq. - diagnostics_mode input (dottrace/dotmemory/dotnet-trace) runs the client through the diag-image DIAG_WITH entrypoint; snapshots are collected via the fork's new instance-level extra_mounts option (bind mount at /nethermind/diag), uploaded as an artifact, and dotTrace snapshots are converted to XML on windows-latest with Reporter.exe (mirrors nethermind's run-expb-reproducible-benchmarks). - trace_blocks input (implies dottrace) injects NETHERMIND_PROFILE_BLOCKS for the BlockProfiler plugin (NethermindEth/nethermind#12444): per-block dotTrace windows so only testing blocks are traced, excluding gas-bump and setup blocks; 'auto' maps to the jochemnet testing blocks 24407730,24407731. - Profiling runs resolve rollback auto -> rpc-debug-setHead (one live process; dotTrace is incompatible with CRIU restores) and default benchmarkoor to the fork's gas-benchmarks branch for extra_mounts.
The diag entrypoint's --save-to base produces a chunked snapshot: an extensionless 'dottrace' index plus dottrace.XXXX chunk files. Match the index (plus *.dtp/*.dtc) like repricing-nethermind does instead of requiring a .dtp extension.
masterdiag profiles with dotTrace 2026.1.4; Reporter 2026.1.0.1 rejects
its snapshots ('unknown or unsupported snapshot format'). Also make the
XML step's exit explicit: fail only when snapshots existed but nothing
converted, instead of leaking the last Reporter exit code.
masterdiag's diag-entrypoint profiles with --profiling-type=timeline; Reporter.exe cannot read Timeline snapshots (MFDTEM storage, 'Invalid storage type'), verified by local bisection: default/sampling capture produces MFDTPF storage that converts fine. In dottrace mode the container entrypoint is now overridden with an expb-style wrapper: default (sampling) profiling, .dtp snapshot naming, and --use-api only when NETHERMIND_PROFILE_BLOCKS is set (MeasureProfiler per-block windows need API mode; API mode suppresses whole-run collection, so trace_blocks needs an image with the BlockProfiler plugin).
…onfig
benchmarkoor os.Expand()s the raw config text at load time, so the
bash wrapper's runtime variables ($tool, $c, $@) were blanked before
the container ever started ('dottrace tool not found in image').
Replace the shell wrapper with a flat argv entrypoint — the client
flags (benchmarkoor's command) are appended by the runtime,
ENTRYPOINT+CMD style — and decide --use-api at compose time.
The dottrace path (/opt/diag-tools/dottrace) is where nethermind
diag images install it (verified in masterdiag and bal-devnet-7-diag).
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
…ed by the branch's full version)
…e runs rpc-debug-setHead keeps client caches warm across tests, so only the first test in a profiling run measures cold state (observed ~4x speedup on subsequent tests). Single-test filters give pure-cold profiles.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Integrates ethpandaops/benchmarkoor into gas-benchmarks CI, exclusively for Nethermind, as the Benchmarkoor - Nethermind workflow running on the
stateful-generatorself-hosted runner — with CRIU memory-checkpoint rollbacks instead of the restart-per-test flow used for the stateful category today.Highlights
container-checkpoint-restore, withrestore_in_place): pre-run steps (gas-bump/funding) are baked into a checkpoint once; every test then gets an instant ZFS datadir rollback + ~4 s in-place process restore. Full 550-test jochemnet stateful suite: 1 h 22 m, 550/550 passed (vs ~3 h+ with per-test restores through archive import, and far longer with container recreation)./mnt/sda/<network>/nethermind) is seeded once into an auto-sized file-backed zpool (grown in place when undersized, resumable seeding tracked via agb:seededdataset property). The source snapshot is never written to..github/scripts/benchmarkoor-pertest-summary.jqover the run'sresult.json).diagnostics_mode: dottrace+ a diag image, e.g.nethermindeth/nethermind:bal-devnet-7-diag): sampling snapshot collected via a bind-mounted diag dir, uploaded as an artifact, and auto-converted to XML by a WindowsReporter.exejob (same approach and package as nethermind'srun-expb-reproducible-benchmarks). Note: the diag images' ownDIAG_WITHentrypoint profiles with--profiling-type=timeline, whichReporter.execannot convert — the workflow overrides the entrypoint with a flat-argv sampling invocation instead.trace_blocks(NethermindEth/nethermind#12444 contract): injectsNETHERMIND_PROFILE_BLOCKS+--use-apifor per-block dotTrace windows scoped to testing blocks only (excluding gas-bump and setup);automaps to the jochemnet testing blocks. Requires an image with the BlockProfiler plugin to produce output.generated-tests-<type>-<network>.tar.gz, defaultamsterdam-repricings-v5.2.0), overridable to any URL including GitHub Actions artifacts;filternarrows the suite per run.Layout
.github/workflows/benchmarkoor-nethermind.yml— the dispatchable workflow (network/test-type/image/filter/rollback/diagnostics/... inputs)benchmarkoor/*.yaml— benchmarkoor config fragments fetched at the pinned commit and merged in order (global, test source, datadir, per-strategy rollback)benchmarkoor/README.md— full docs: defaults, ZFS bootstrap, datadir layout (--Init.BaseDbPath=/data/mainnet), profiling, caveatsscripts/genesisfiles/nethermind/generator-amsterdam-jochemnet.json— corrected jochemnet chainspec (main's copy scheduled an incomplete Amsterdam EIP set at a timestamp inside the snapshot's synthetic block range, causingInvalidBlockLevelAccessListHashon startup replay; this copy matches the ethpandaops devnet-7 chainspec, params/genesis/accounts-identical)Dependencies
Two benchmarkoor capabilities live in a fork until upstreamed (
kamilchodola/benchmarkoor@gas-benchmarks):restore_in_place— upstream PR open: ethpandaops/benchmarkoor#282extra_mounts(diag-output egress) — candidate follow-up PRUpstream images ignore both options gracefully (stateful runs fall back to archive-import restores; diagnostics runs auto-default to the fork build). Once merged upstream, no changes needed here.
Validation
.dtpsnapshot + XML report artifactsClient/fixture pairing note: the v5.2.0 fixtures carry bal-devnet-7 block access lists —
nethermind:masterrejects them (InvalidBlockLevelAccessList), so master runs need fixtures refilled against master.