fix(payload_builder): supersede synthesized-attribute builds with node attributes

[pk910]

Problem

On glamsterdam-devnet-8, buildoor-teku-nethermind-1 produced an invalid payload in slot 172896 (first slot of epoch 5403, parent slots 172893-172895 missed). Every node rejected the envelope:

Envelope verification error: Withdrawals mismatch between payload and expected
payload=0xe153a69a… expected=0xe47b0e93…

Root cause (from the node's otel logs and the envelope SSZ Teku dumped):

• The missing-block fallback fires at the slot's build start time (-2900 ms) and copies the previous slot's attributes.
• Teku emits the next slot's attributes only ~30 ms before that point. The epoch transition delayed the emission by ~0.8 s, so the fallback won the race and the build ran from slot 172895's attributes, whose withdrawals carry the pre-transition amounts (same indices and validators, amounts lower by exactly the epoch rewards).
• The bid was still accepted on chain because the randao mix of an epoch's first slot equals the previous one; only the envelope failed.
• The real attributes arrived 0.8 s later (1.7 s before the first bid) and were ignored: a build for the same parent tuple had already started.

Fix

• Fallback events carry a Synthesized marker; every started candidate build keeps its cancel func, the attributes it ran from and the payload it emitted.
• A node-received event for the same parent tuple supersedes a build that ran from synthesized attributes unless the build inputs are identical (PayloadAttributesEvent.BuildInputsEqual): the in-flight engine build is cancelled, an already emitted payload is withdrawn from the payload cache (recorded as a failed build, reason superseded by beacon node attributes) and the tuple is released so the existing late-build path rebuilds from the node's attributes.
• Publishing a payload and superseding it serialize on the build lock, so a stale payload can never be emitted after its abort.
• Slot results: a failed outcome for the same candidate replaces the primary ready outcome, so a withdrawn payload is not left rendered as ready.

E2E: two-node enclave (unrelated pre-existing failure)

The PR check's p2p phase has been failing on every branch since 2026-09-06, including a dependabot UI-deps PR. The ethpandaops/lodestar:glamsterdam-devnet-8 image moved to v1.47.0 on 2026-09-04, which no longer inserts a beacon-API-published bid into the publishing node's own bid pool (ChainSafe/lodestar#9998), and gossipsub never loops a self-published message back. In the single-node enclave buildoor's bids went to zero peers and node 1's proposers logged builder_no_bid right after Published execution payload bid, every slot.

The participant now runs with count: 2: buildoor stays wired to node 1, gossip feeds node 2's bid pool, and the p2p phase wins on a node 2 proposal. Verified locally: all three phases pass, node 2 logs Selected builder block for the bid node 1 published.

Note for devnet operators: on real devnets a lodestar node paired with a buildoor only takes that buildoor's bids through the Builder API from v1.47.0 on; its own validators never see the p2p bids.

Review follow-ups

• Ready event after supersede: a build records when its ready dispatch finished. A supersede that finds the payload cached but not yet dispatched leaves the failure report to the build goroutine, which fires it right after the ready event, so the superseded failure always follows the same build's ready (exactly once per build).
• Old build's failure clobbering the rebuild: every build carries a monotonic BuildSeq on its started/ready/failed events. The slot results tracker keeps it per outcome (in memory only) and drops events of an older build once a newer build of the same candidate has reported; a newer build's start restarts the candidate instead of being refused as a regression.
• E2E timing: with two participants buildoor comes up ~90 s after the enclave starts (already slot 1 on a runner); the lodestar VCs register with the builder once at epoch 0 and retry only at the next epoch (Gloas), so the pre-Gloas phase needs buildoor up before slot 0. genesis_delay is now 120 s.

Testing

• New unit tests: abort in flight, keep identical inputs, ignore node builds, withdraw an emitted payload, handler-driven rebuild, cache removal, BuildInputsEqual table test.
• go test -race ./... passes; go vet clean.
• Kurtosis e2e passes locally with the two-node enclave.
• golangci-lint could not run locally (installed binary panics on the Go 1.27 toolchain requirement).

Follow-up (not in this PR)

The fallback still fires ~30 ms after Teku's usual emission point, so every delayed emission now costs an aborted engine build plus a rebuild. Delaying the fallback slightly past the CL's emission point would avoid that.

[redpandabot]

Summary

Fix for the glamsterdam-devnet-8 invalid-payload incident: builds started from synthesized (missing-block-fallback) attributes are now tracked per parent tuple with their cancel func and emitted payload, and a node-received event for the same tuple cancels the in-flight engine build, withdraws an already-emitted payload from the cache (recorded as 'superseded by beacon node attributes') and releases the tuple for a rebuild via the late-build path. The cache withdrawal is authoritative and sound for both bid paths (p2p scheduler and builderapi read the cache at bid time, never a stored reference), the locking is consistent (scheduledBuildMu→payloadCache.mu in one direction only, no leaked unlocks), and BuildInputsEqual correctly captures the epoch-transition withdrawals/randao drift that caused the incident. Two residual display/tracking races remain around the ready-event emission not being serialized with the supersede.

Issues

• 🟡 pkg/payload_builder/service.go:911 — Withdrawn payload's ready event can still fire after the supersede — see the thread on that line
• 🟡 pkg/slot_results/tracker.go:598 — Same-candidate failure can clobber a successful rebuild's ready outcome — The new rule where a failed outcome replaces a ready one for the same candidate also applies in the reverse direction: the aborted old build's late reportBuildSuperseded (fired from its goroutine only after the engine call finally observes the cancellation) can arrive after the rebuild has already emitted its ready outcome for the same candidate, replacing the valid rebuilt payload's ready state with a stale 'superseded' failure. The tracker's single per-candidate outcome cannot represent the sequential old-aborted + new-successful lifecycle, so which state renders depends on inter-dispatcher arrival order.

---

_{Reviewed @ 2b2209a0
"Programs must be written for people to read, and only incidentally for machines to execute." — Abelson & Sussman}

[redpandabot]

Summary

The PR makes a node-received payload_attributes event supersede a build that started from locally synthesized attributes (the missing-block fallback won the race against a late node event, producing a payload that would be rejected for withdrawals/randao staleness): the in-flight engine build is cancelled, an already-emitted payload is withdrawn from the payload cache and recorded as failed with reason "superseded by beacon node attributes", and the tuple is released so the late-build path rebuilds from the node's real attributes. I traced the locking (publish and supersede serialize on scheduledBuildMu), the emission ordering (the superseded failure always follows that build's ready event, exactly once, via the readyFired flag), and the slot-results BuildSeq ordering (a superseded build's late events never clobber its replacement's); these are all handled correctly and well unit-tested. The E2E two-node change is a coherent workaround for the lodestar self-publish regression.

Issues

• 🟢 pkg/payload_builder/service.go:893 — Superseded build misreported as "context canceled" when a jq payload transform is configured — If a supersede fires after the engine build returned successfully but before applyPayloadTransform runs, the cancelled ctx makes the jq transform fail and the build fires a PayloadBuildFailedEvent with Error="context canceled" (or the transform error) and returns — never checking buildAborted, so reportBuildSuperseded is never called (the supersede's own reportHere was false because the payload was never published). The slot ends up showing a misleading failure reason instead of "superseded by beacon node attributes". Only reachable when a jq payload transform is set for the slot (a testing knob); the default no-transform path takes the correct publish-lock abort check.

---

_{Reviewed @ f56f94fc
"Programs must be written for people to read, and only incidentally for machines to execute." — Abelson & Sussman}