feat: testing build source — exact blocks from a private tx intake via testing_buildBlockV1

[qu0b]

Testing build source: controlled blocks from a private tx intake via testing_buildBlockV1

--build-source testing turns buildoor into a block load and edge-case tool for devnets and shadowforks. You decide exactly which transactions a block holds and in what order, buildoor gets the block proposed through its normal ePBS bid/reveal or Builder API path, and it verifies on the EL that the included block holds exactly the plan. Any deviation is loud.

What changed

Tx intake (pkg/tx_intake, served at POST /rpc on the API port). A JSON-RPC 2.0 endpoint transaction sources point at instead of the EL. eth_sendRawTransaction lands in buildoor's private queue, eth_getTransactionCount(addr, "pending") answers from the queue on top of the EL's latest nonce, everything else is forwarded verbatim (batches too). geth gossips every RPC transaction and cannot be told not to, so a private queue is the only way to guarantee that nobody else includes the transactions. spamoor works unchanged against it. buildoor's own lifecycle transactions are teed into the queue as well, otherwise its intake-built blocks would never carry its deposits.

Queue. Per-sender nonce chains keyed by (sender, nonce) and hash. Same hash is idempotent, same (sender, nonce) is replaced by the newer transaction. Nothing is removed at build time: the parent state nonce is the truth. At pack time nonces below it are evicted and gaps stop a sender's chain, which covers reorgs, lost slots, duplicates and restarts with no special cases. Bounded by capacity (reject) and age (evict).

Packer. Deterministic pre-filter with the caps geth will enforce: gas limit stepped toward the target, next base fee, blob base fee bound, blob cap from eth_config, byte cap, per-sender budget. Policies: fifo, fee, round_robin, as_given. as_given and the per-slot build.txs list are exact: any deviation is an error, never a trim.

Build (pkg/payload_builder/testing_build.go). Bare forkchoiceUpdated to the parent (geth builds on its head only), pack, testing_buildBlockV1, then verify the returned payload holds exactly the plan before it is bid or served. geth returns the getPayload envelope shape, so the whole existing tail (extraData stamp, bid gas-limit enforcement, beacon conversion, bids, reveals, transforms, slot results) is reused unchanged. An EL refusal is attributed to the sender or index, the attributed transactions get a strike, and the build retries within testing.max_attempts. The build runs under a slot deadline (default: ePBS bid start minus 300 ms); a miss means no payload unless testing.on_failure: pool is chosen. Only the canonical candidate builds in this mode.

Verifier (pkg/tx_plan_verifier). On every included testing-built payload it fetches the block from the EL and compares transaction order and count with the plan. Gloas missed and orphaned verdicts are mirrored. The verdict lands on SlotResult.tx_plan.status, in buildoor_testing_plan_checks_total, and as an error log line prefixed TX PLAN CHECK FAILED.

Control surface. Flags --build-source and --testing-*, all live settings (POST /api/config/testing, build.source / testing.* keys). Per-slot overrides through the action plan build category: source, fill: {gas_pct, max_txs, max_blobs, policy}, txs (explicit ordered hashes). GET/DELETE /api/buildoor/tx-queue. Metrics in pkg/metrics.

The mapping this guarantees

For every testing-built block there is one tx_plan on the slot result: the ordered list of transaction hashes buildoor asked geth to build. It is checked three times: the packer refuses to deviate from an explicit list, the built payload is compared with the plan before it leaves buildoor, and the canonical block is compared with the plan on the EL after inclusion. Expected order, expected count, or a recorded failure.

Verified on a kurtosis devnet

.github/e2e/kurtosis-testing-build.yaml + .github/scripts/e2e-testing-build.sh: minimal preset, gloas at epoch 1, one geth (--http.api ...,testing) + lighthouse participant, one dedicated buildoor from this branch. The script switches the source to testing, runs spamoor eoatx against the intake, and requires that every won block after the switch verified as an exact match with at least 6 blocks carrying 20+ transactions, then cross-checks one block's transaction list on the EL itself.

Result of the run on this branch (image built from the worktree, ethereum-package main, geth master, lighthouse unstable):

Check	Result
plan checks after the switch	14 checked, 14 match, 0 mismatch, 0 block_not_found, 0 missed
eoatx blocks (spamoor at 300 tx/slot)	150 transactions per block, built in 14 to 16 ms, included through ePBS
gasburner blocks (4M gas per tx)	46 transactions, gasUsed / gasLimit 0.98 to 0.99, built in about 270 ms, packer reports the overflow as skipped.gas_cap
pre-Gloas	builder API blocks at slots 4 to 7 verified as matches (empty plans)
EL cross-check	eth_getBlockByHash transaction list equals tx_plan.expected_hashes

Explicit per-slot plan, same enclave: six transfers from two senders were queued while two slots ran on the pool source, slot 117 was planned with the interleaved order s3[0], s2[0], s2[1], s3[1], s3[2], s2[2], and the included block holds exactly that list (tx_plan.status: match, expected_hashes equal to the plan, eth_getBlockByHash equal to the plan). A second plan naming a hash that was no longer queued failed the build before any bid: packing: tx 0 (0x82eb…) is not queued, slot result build.status: failed, buildoor_testing_build_failures_total{reason="packing"} 1. Across the whole session: 50 plan checks, 50 matches.

The assertoor playbook in ethpandaops/assertoor (playbooks/dev/buildoor-testing-build.yaml, needs the executionGroups endpoint field from the same PR) passed all nine tasks against this enclave: switch, spamoor through the builder group, block proposals with 20+ transactions, zero mismatches, no forks, switch back.

The run also caught a real gap: buildoor's lifecycle deposit went to geth's public pool and never landed in its own intake-built blocks, so the builder stayed unregistered after the fork. The lifecycle RPC client now tees every transaction it sends into the intake.

Unit tests cover the queue, the packer (fifo, nonce rule, fee and balance filters, exact as_given), error attribution, the proxy (intercept, forward, batches), the block comparison and the gas-limit step.

Review fixes (redpandabot)

All three findings were real and are fixed with regression tests:

• Snapshot race that failed whole slots (🟡). Build snapshotted the queue for the sender states and Pack snapshotted again, so a sender whose first transaction arrived in between had no parent state and the entire slot build errored — under exactly the sustained intake load this exists for. Pack now takes the caller's snapshot, so one consistent view covers states and packing. Test asserts a sender arriving after the snapshot leaves the build unaffected and stays queued for the next block.
• as_given was trimmed by the retry path (🟡). The EL-refusal path attributed and dropped transactions for every policy, so an explicit build.txs list was silently reduced, its transactions accrued strikes, and the verifier then reported match against the reduced plan. An exact plan now fails the build instead; fill policies keep the attributed retry. Two tests pin both halves of that contract.
• Unsynchronized Strikes read (🟢). Snapshot entries are shared, so the API's read raced the queue's write. Strikes is now atomic behind an accessor, with a concurrent striker/reader test that runs under -race.

The whole suite runs under -race; docs updated for the as_given refusal behaviour and the snapshot contract.

Second round, both fixed:

• The base-fee ceiling shrank exact plans (🟡). It forced the fill to 50% for every policy, so an explicit as_given list that fits the block but not half of it failed with "does not fit: gas_cap" — the operator's plan broken by an unrelated sustainability knob. The reduction is now a fill-policy knob only, extracted into a named helper and covered per policy.
• not_included was never produced (🟢). A testing slot whose bid simply lost sat at pending for its whole retention window, the one silent outcome in a feature whose point is being loud. The slot-results tick now records it once the slot is past the inclusion tracker's reorg window, and only for slots that already carry a pending plan, so it never races a real verdict or creates a record.

Third round, both fixed:

• A blob-capped transaction did not stop its sender's chain (🟡). The packer advanced past a transaction it had not added, so that sender's higher nonces landed in the plan at a gap. geth refuses such a list, the error attributes to the sender, and every one of its transactions is dropped and striked — an attempt wasted on each blob-heavy fill. Any skip now ends the chain, which is what the packer's own nonce rule always said, with a regression test.
• The intake forwarded anything to the EL, unauthenticated (🟡). It sat on the API port and passed every non-intercepted method through, and in this mode the EL must serve the testing namespace, so anyone reaching the WebUI port could drive testing_buildBlockV1 and build blocks on the devnet EL. It now forwards only the sender-facing namespaces (eth_, net_, web3_, txpool_, rpc_) and refuses the rest, and it sits behind the same token as the mutating API endpoints when an auth provider is configured. Verified on the live devnet: testing_buildBlockV1 and debug_setHead are refused, eth_chainId still answers, and the e2e passes unchanged.

Fourth round, both fixed:

• The default deadline sat before the slot started (🟡). It always derived from the ePBS bid window (bid start minus 300 ms, about 700 ms before slot start), including in the Builder-API-only mode the README itself shows, so attributes arriving near the slot start permanently skipped the build. It now follows whoever will ask for the payload: the bid window when ePBS is enabled, otherwise the slot start, since getHeader can come that early. Table test over all three cases.
• A refused candidate left a dangling in-progress marker (🟡). The build was announced before the testing spec was resolved, so a candidate the testing source does not build (only the canonical one is supported) showed as building forever with no outcome. The spec resolves first, so nothing is announced that will not run.

Notes for reviewers

• Only geth implements testing_buildBlockV1 today; startup fails fast when the EL does not serve the testing namespace. besu and nethermind are blocked upstream (state-root mismatch, missing debug_setHead).
• Every full block raises the base fee 12.5 percent and only buildoor's blocks carry the queue. --testing-base-fee-ceiling-gwei drops the fill to the 1559 target above a ceiling; a transaction source needs a high max fee for long max-fill runs.
• Two one-line upstream geth improvements would make error attribution exact and allow speculative candidates: include the tx index and hash in BuildBlockV1 errors, and accept any parent whose state is available.
• The slot results tracker gains a tx_plan field; no schema migration, older records simply have none.

https://claude.ai/code/session_01P3LSEorv3qsJrWb12ZKHNo

[redpandabot]

Summary

feat: a private tx intake (JSON-RPC on /rpc) plus a geth testing_buildBlockV1 build path and an on-chain tx-plan verifier, so a buildoor slot's payload can be built from an exact operator-controlled transaction list and the included block is later checked against it. The code is unusually careful (snapshot/state coupling, nonce-chain gap rules, exact-as_given contracts, loud verifier) and I found no blockers; the two items below are a genuine but minor shutdown race and a config footgun.

Issues

• 🟡 pkg/tx_plan_verifier/verifier.go:140 — WaitGroup Add races Wait on shutdown (inclusion tracker still live) — see the thread on that line
• 🟢 pkg/action_plan/frozen.go:660 — Global or per-slot as_given policy without a tx list can never build — The only source of the explicit tx list is the per-slot build.txs (resolveSource at frozen.go:660), yet testing.policy: as_given is accepted by settings/plan validation (config/settings.go ValidTestingPolicy). Any slot that resolves as_given without build.txs fails every build with "policy as_given needs a non-empty tx list" (packer.go packExplicit); a global --testing-policy as_given therefore bricks every un-overridden slot. Worth a validation guard (as_given requires a tx list) or a loud startup/plan warning.

---

_{Reviewed @ 1aad0a08
"All non-trivial abstractions, to some degree, are leaky." — Joel Spolsky}