Skip to content

ePBS (EIP-7732 / Gloas) — SIP-94 - #2901

Open
iurii-ssv wants to merge 161 commits into
stagefrom
epbs-gloas
Open

ePBS (EIP-7732 / Gloas) — SIP-94#2901
iurii-ssv wants to merge 161 commits into
stagefrom
epbs-gloas

Conversation

@iurii-ssv

@iurii-ssv iurii-ssv commented Jun 23, 2026

Copy link
Copy Markdown
Contributor

Node-side ePBS (EIP-7732 / Gloas) support, per SIP ssvlabs/SIPs#94.

Built on the Boole code baseline — consolidated roles, ProposerConsensusData, and the node-side role switches — which has already merged to stage (via the integration/boole-convergence merge). This is a compile-time dependency on that code being present; it is not a dependency on the Boole fork activating. ePBS gates every execution path on the beacon node's GLOAS_FORK_EPOCH alone and runs correctly with Boole dormant (Forks.Boole = math.MaxUint64 — the default on every network, mainnet/sepolia/hoodi/holesky and now local-testnet), which is exactly the state the Glamsterdam transition runs in.

The ePBS wire constants (runner/beacon roles, partial-sig types, beacon domains 0x0B/0C/0D, and the #2962 request-auth pair DomainBuilderRequestAuth 0x0B000001 / RequestAuthPartialSig(9)) live in ssv-spec via PR #632 — the established spectypes pattern, and required because ekm/ssvsigner reaches signing domains only through spectypes. Their golden test lives there; both go.mods are pinned to that branch until it merges.

What's here

  • protocol/v2/types/gloas package — the Gloas beacon-chain containers (the §4 block family with the EIP-8282 five-list ExecutionRequests, the §6 envelope, the PTC containers, ProposerPreferences) are go-eth2-client's spec/gloas types, aliased here; their SSZ is pk910/dynamic-ssz with the progressive-container/list merkleization of EIP-7688/EIP-7916 (Gloas SSZ types give a wrong hash_tree_root. All Gloas proposals fail. #3008). The SSV-owned types (GloasBeaconVote — fixed 120-byte, cross-fork decode fails cleanly — EnvelopeConsensusData, the blinded envelope, PTCDuty, builder config and request auth) have dynamic-ssz encoders. Roots are pinned by a devnet-8 golden fixture: a finalized Gloas block whose HashTreeRoot must equal the chain's header root and whose proposer signature must verify, plus its execution-payload envelope.
  • Fork gating & retiming (SIP §1)GLOAS_FORK_EPOCH read from the BN spec into the fork map; IsGloas/IsGloasAtSlot, safe on pre-Gloas networks. IntervalDuration becomes slot-keyed (1/3 of the slot → 1/4 at the fork) and drives duty deadlines, QBFT round-1 head starts, aggregation waits, and attestation fetch budgets. QuickTimeout is deliberately not retimed — the Gloas proposer is round-1-must-succeed pending real round-trip data. The §6 envelope proposer shares the proposer's round-relative timers (roundtimer.RoundRelativeRole, also what message validation's round-spread exemption keys off): its instance starts only after the §4 block is published, so a slot-anchored timer would already have expired.
  • §2 Gloas attestations — the committee runner agrees on GloasBeaconVote (carrying the BN's payload-status index) on Gloas slots: fork-aware decode across runner, post-consensus validation, committee observer, and duty tracer; NewGloasVoteChecker; index-preserving aggregation; and a hand-rolled attestation-data GET on Gloas slots (go-eth2-client's post-Electra Index==0 check rejects the healthy FULL status). Aggregation on Gloas slots uses go-eth2-client's dedicated gloas.AggregateAndProof/SignedAggregateAndProof end to end (fetch, consensus-data decode, signed submit): the container serializes like Electra's but merkleizes differently, so the Electra path would sign the wrong root (All AGGREGATOR duties fail after the Gloas fork. The library refuses the version header "gloas". #3009). ForkAtEpoch resolves Gloas from GLOAS_FORK_EPOCH on (no more Fulu cap, in both the node and ssvsigner), so attestations ride gloas.Attestation and the aggregator-committee consensus data stamps DataVersionGloas — the SIP §2 byte-parity stamp Anchor uses (ePBS/Gloas: stamp DataVersionGloas into AggregatorCommitteeConsensusData.Version (drop the Fulu-cap workaround) #2998). This relies on ssv-spec's Gloas arms building the Gloas containers (on the dynssz branch, ssv-spec#643, vectors regenerated); with the Electra-reusing arms the fork's submit path would reject the attestations.
  • §3 PTC attestation — full slice — goclient endpoints; PTCAttesterRunner (no consensus, partial-signature only, honest convergence over the frozen 75%-cutoff observation, abstains when no block is seen); scheduler handler firing at the cutoff; duty store + message validation; ekm signing.
  • §4 Gloas block proposal — produce/publish via the merged beacon-APIs endpoints (v4 produce, include_payload=false); the bid-only block is the QBFT value directly (no blinding); every operator submits the decided block (BN dedupes by root — to be re-confirmed on a Gloas BN); slashing-protected local signing (check→record→sign under a dedicated lock); build-source telemetry; new ProposerDelayEPBS knob (hard-capped at 1s, no dangerous override).
  • §5 Proposer preferences — one duty per upcoming proposal slot across the lookahead, plus the MIN_SEED_LOOKAHEAD pre-fork emission window; a multi-slot dispatcher runner (honest convergence over fee recipient / target gas limit / dependent_root, singleflighted dependent_root fetch); message validation with a future-slot allowance and a bounded distinct-root dedup (admitting dependent_root re-emissions); ValidatorRegistration is deprecated at the fork (rejected wire-side, drained scheduler-side).
  • §6 Execution-payload envelope (self-build only) — the proposer records its §4-decided block root and triggers a second QBFT instance (EnvelopeProposerRunner, role 9) over the blinded envelope; only the operator whose cached envelope content-matches the decided value publishes the full body; the value check ties the envelope to the §4-decided root. Only the beacon node that built the decided block can produce its envelope, so on a cluster whose operators run separate beacon nodes every non-builder joins the §6 instance as a voter (Controller.JoinInstance): it validates and votes on the builder's proposal and signs the decided root, but never proposes a value of its own — the round relies on the builder leading round 1, which holds whenever §4 decided in round 1. The envelope QBFT window is now [25%, 50%] of the slot — consensus-specs#5414 lowered PAYLOAD_DUE_BPS 75%→50% (§3 PTC's PAYLOAD_ATTESTATION_DUE stays 75%).
  • ePBS/Gloas: external-builder authentication & per-builder bid preferences — design & implementation plan #2962 — direct-builder request auth + overlay (opt-in) — the threshold BuilderRequestAuth{data, proposal_slot} signing (builder-specs DOMAIN_BUILDER_REQUEST_AUTH, genesis-style) rides the §5 dispatcher: per configured builder — Builders cluster config in keymanager-APIs#88 vocabulary, validated at startup, required identical across all n operators — one single-root RequestAuthPartialSig per distinct auth root per proposal slot, collected in a dedicated container with no succeeded-gate, reconstructed into a per-validator auth cache. Message validation admits the new type under role 8 with its own distinct-root budget (§5 two-tier REJECT/IGNORE). Sub-quorum degrades silently to the enshrined flow — never blocks the proposal; reconstructions and auth-unavailable are counted, and the §4 build-source telemetry is a typed enum. The upstream specs this was written against have merged and the branch is reconciled to them (builder-specs#165 renames; keymanager-APIs#88 supersedes validate change round justification signer uniqueness like we do in regular agg. messages #87; beacon-APIs#630 supersedes Verify signed message with domain #625), and phases 2–3 are implemented here: the produceBlockV4 POST attach with the Eth-Builder-Url block-forwarding echo (§4), and the BN-mediated ahead-of-time submitBuilderPreferences — both e2e-gated on a Round Timer Fix #630 beacon node. Design/tracking: #2962.
  • Riding along (not Gloas-specific) — duty-fetch retry when no validators are eligible yet (attester/proposer/sync-committee handlers); scheme-less beacon-address normalization; zero-registry-address guard in SSVConfigByName; GetStateRoot nil-state guard; runner-state JSON dedup.
  • Exporter cleanup (unrelated tagalong) — removed a dead vendor/ sszgen directive; documented that exporter/model_encoding.go is hand-maintained.

Validation — hermetic Gloas devnet (local_testnet_gloas)

Validated on a hermetic Gloas devnet (geth/lighthouse glamsterdam-devnet-6, lighthouse v8.2.0-36b70da, Gloas fork @ epoch 2), read from Loki node logs + the beacon API. Suites: ssvlabs/aetheria#130 (proposer, §4/§5/§6) + #128 (ptc, §3).

§ Duty Status
§2 Attestation ✅ works
§2 Aggregation 🔁 Gloas arms landed (#3009 / #3021); Aetheria re-run pending
§3 PTC (payload attestation) ✅ works post-fork
§4 Proposer block ✅ works — publication passes; proposed block canonical on-chain
§5 ProposerPreferences ✅ works — emit → submit, 0 errors; POST …/validator/proposer_preferences → 200
§6 Execution-payload envelope ✅ publishes + lands on Lodestar v1.43.0 (#2921 — full/unblinded body); residual all-publish dedup on the shared-BN devnet → #2923

The node's ePBS implementation is spec-correct — §2–§5 are confirmed working on-chain, and §6 publishes and lands on Lodestar v1.43.0 (the live devnet-6 CL). §6 produce is missing in lighthouse v8.2.0 (a generic beacon-APIs#580 404) but works on Lodestar. #2921 now publishes the full/unblinded envelope (SignedExecutionPayloadEnvelopeContents), which Lodestar accepts — the earlier 400 "Offset out of bounds" (blinded body) is gone and the envelope lands on-chain. #2922 makes the all-operators §4 submit handle Lodestar's 500 BLOCK_ERROR_ALREADY_KNOWN as success (the "BN dedupes by root" assumption doesn't hold on Lodestar; the block still lands). Residual: on the devnet §6 showed the same all-publish dedup as §4 — every operator published the identical envelope, so the redundant ones hit 500 EXECUTION_PAYLOAD_ENVELOPE_ERROR_ALREADY_KNOWN (the §6 analog of #2922, tracked in #2923). That only holds because the devnet's operators share one beacon node, which produces the same envelope for all of them; with a beacon node per operator only the builder holds the payload and publishes, and the other operators take part as voters (see §6 above) — a topology not yet exercised end to end. E2E-verify context in #2920 (V1/V2/V6 — resolved there; the issue is closed, see the e2e strategy below). (§3 also shows a transient pre-fork 500 IncorrectStateVariant on lighthouse's PTC-duties lookahead, self-clearing once the head crosses the fork.)

E2E strategy. The hermetic Aetheria local_testnet_gloas net above is the ePBS proving ground, and public-network e2e rides the Sepolia/Hoodi Gloas forks on the existing SSV deployments there — GLOAS_FORK_EPOCH is read from the BN at runtime, so no new networkconfig is needed. A dedicated SSV cluster on the public glamsterdam devnet was considered and dropped: standing up a fresh SSV environment (contract deploy, registry bootstrap, operator onboarding) buys little over those two paths. #2920 is closed accordingly, its remaining verify items split across the successors: the manual Sepolia fork log-check (#2953), the node-design verification items V5/V8/V9/V11 (#2954), the Aetheria harness on Hoodi post-fork (ssvlabs/aetheria#139), and automated fork-transition coverage (ssvlabs/aetheria#141). The GlamsterdamDevnet networkconfig stub has been dropped from the branch accordingly (the zero-registry-address guard it motivated in SSVConfigByName stays).

Known limitations / deferred

  • Remote signing (Web3Signer): no request types for the four new domains (incl. ePBS/Gloas: external-builder authentication & per-builder bid preferences — design & implementation plan #2962's request auth) and no Gloas block path — remote-signing operators fail those duties (bounded: the cluster reconstructs while ≤ f operators are remote). Remote fork_info now carries the Gloas fork so every other remote duty signs under the correct domain (whether that alone satisfies a live Web3Signer is an e2e-confirm — #2954 (V11)). Deferred §4 fix direction: pass the block header (HTR-equal to the block, nameable across the module boundary) instead of the Gloas block; whether Web3Signer accepts a Gloas-version block request is unverified.
  • §5 publish-finality: preferences publish on quorum with no hold-until-final guard, so a preference published just before a reorg shifts its dependent_root is not retracted. Re-emission itself now works — message validation admits up to N=4 distinct signing roots per (slot, signer) and the scheduler re-emits only on a real dependent_root change (the now-agreed SIP-94 §5 rule, still to be matched by Anchor). Only the pre-publish finality hold stays deferred. Low severity (reorg-gated, §5 is observational).
  • §6 round-1 dependence: a voter never proposes a value of its own, so the envelope round decides only if the builder leads round 1 (§4 decided in round 1) or a prepared value carries over a round change; otherwise the self-built payload stays unrevealed for that slot. Non-builders cannot fetch the payload from their own beacon node, so a fallback would need the builder to hand it over.
  • Upstream-gated: the Gloas endpoints are hand-rolled HTTP — the repinned go-eth2-client fork now ships typed calls for the PTC and envelope endpoints (moving onto them is a follow-up) but none for proposer/builder preferences or the proposer-duties dependent_root, and its ePBS proposal call is the pre-Round Timer Fix #630 GET with no BuilderConfig body or Eth-Builder-Url echo; DataVersionGloas aliases the fork's spec.DataVersionGloas, ForkAtEpoch resolves it, and the aggregator consensus data stamps it (ePBS/Gloas: stamp DataVersionGloas into AggregatorCommitteeConsensusData.Version (drop the Fulu-cap workaround) #2998 — done on the branch). ssv-spec's Gloas arms (GetAggregateAndProofs/GetAggregateAndProofHashRoot, the reference constructor, the attestation helpers) switch from the Electra container to the fork's Gloas one on the dynssz branch (ssv-spec#643), which both go.mods pin; what remains for ePBS/Gloas: stamp DataVersionGloas into AggregatorCommitteeConsensusData.Version (drop the Fulu-cap workaround) #2998 is the Anchor byte-parity check; eth2-key-manager purpose-named root signer.

Merge & follow-up checklist

Merge blockers — to clear before this PR lands on stage:

  • Land the outstanding ePBS/Gloas: external-builder authentication & per-builder bid preferences — design & implementation plan #2962 review fixes on the branch: the stash-replay gate on re-emission (hasDutyRunninghasDutyAssigned), domain-fetch hard-fail, and minor polish
  • ssv-spec's Gloas container arms for the §2 aggregate and attestation landed on the dynssz branch (ssv-spec#643) → both go.mods pin its head. Required pairing: with the previous pin's Electra-reusing arms, the node's ForkAtEpoch Gloas resolution would make attestation submits fail on Gloas slots.
  • ssv-spec#632 merges → re-point both go.mods at the tagged ssv-spec (today they pin the branch tip, which already carries the EnvelopeBuilderEnvelopeProposer rename and the ePBS/Gloas: external-builder authentication & per-builder bid preferences — design & implementation plan #2962 request-auth constants).
  • Boole lands on stage → move the ePBS commits onto stagedone: Boole landed via the integration/boole-convergence merge (so the merge-base was already on stage and a plain rebase sufficed), and the branch is rebased onto the latest stage tip (136 commits, linear; 134/136 patches byte-identical per range-diff, two mechanical conflict resolutions in committee.go/aggregator.go) — and re-rebased onto the current stage tip after the review round (one import-block conflict in aggregator_committee.go)
  • Re-run the Aetheria local_testnet_gloas suites (proposer + ptc) after the rebase and after each boole-fork refresh

Tracked follow-ups (not merge blockers):

  • E2E verify — the ePBS devnet e2e — glamsterdam-devnet (SIP-94, #2901) #2920 successors (that issue is closed; no dedicated SSV cluster on the public glamsterdam devnet, see the e2e strategy under Validation): manual Sepolia fork log-check #2953; node-design verification items V5/V8/V9/V11 #2954; Aetheria harness on Hoodi post-fork ssvlabs/aetheria#139; automated fork-transition coverage ssvlabs/aetheria#141. The hermetic local net is done & green — ssvlabs/aetheria#125.
  • Builders-configured local_testnet_gloas run — gauge the restart re-broadcast cost (a restart mid-lookahead re-broadcasts every auth root, up to entry-cap+1 same-peer REJECTs per pending slot; noted at broadcastAuthRoots) and the request-auth reconstruction telemetry
  • Cross-client / upstream: re-check the §5/§6 wire constants against Anchor once it implements them (PTC already verified against sigp/anchor) — now also RequestAuthPartialSig(9)/DomainBuilderRequestAuth and the role-8 dual-type validation rule; run a computational SSZ/HTR cross-check (incl. the EIP-8282 five-list ExecutionRequests) once canonical Gloas spec vectors exist
  • §6 on separate beacon nodes — exercise the voter path end to end on a topology with a beacon node per operator (the hermetic devnet shares one BN, so every operator produced the same envelope there): confirm a non-builder's envelope produce fails as expected and that the round decides with the builder leading round 1. In the same pass, move the PTC and envelope requests onto the go-eth2-client fork's typed calls.
  • Map the spec's Gloas runner/committee vectors onto the node (#3017) — the repinned spec's Gloas runner tests and the Gloas-era variants of shared vectors are skipped in the mapping suites under gloasSpecRunnerSkipReason (the Gloas valcheck vectors already run against the node's real checkers); natural to do together with the ssv-spec tag repoint above
  • ePBS/Gloas: external-builder authentication & per-builder bid preferences — design & implementation plan #2962 phases 2–3 — implemented on this branch: the produceBlockV4 GET→POST attach (the BuilderConfig body assembled from config + the per-slot auth cache, Eth-Builder-Url block-forwarding echo gated by the §6-style owner-match, per-node GET fallback) and the BN-mediated ahead-of-time submitBuilderPreferences (reuses the reconstructed auth, all-operators submit). E2E gated on a beacon node shipping beacon-APIs#630 (lodestar#9832 is an open draft); the GET fallback keeps current beacon nodes working.
  • Flip §4 produce to POST-first (beacon-APIs#630 POST)done: produce always POSTs a BuilderConfig body (the direct-builder overlay when configured, else a neutral local-build config — builders: [], builder_boost_factor 100); the per-node GET fallback stays for pre-Round Timer Fix #630 nodes (only on a 404/405). Round Timer Fix #630 merged and Teku went POST-only, so this became a live bug (reported in produceBlockV4 must use the POST method and send a BuilderConfig body #3002), not just a future risk.
  • MEV knobs: docs/MEV_CONSIDERATIONS.md ePBS rewrite (config.example.yaml already points readers there for ProposerDelayEPBS); re-evaluate proposals: mev/commit-boost driven MEV flow #2855 (ProposalSoftDeadline) against the tighter 25% proposal deadline once ePBS lands — decision + rationale recorded on proposals: mev/commit-boost driven MEV flow #2855.
  • §5 publish-finality — re-emission (N=4 distinct-root dedup + dependent_root-gated re-emit) is implemented; only the pre-publish finality hold remains deferred (low severity). Anchor still to match the §5 validation rule.
  • Deferred code refinements: assert the Eth-Consensus-Version response header on Gloas produce; unify the hand-rolled JSON/SSZ HTTP helpers (shared httpDo core + a single httpStatusError); reuse the shared Gloas test-block fixture in the goclient proposer tests.

@iurii-ssv
iurii-ssv force-pushed the epbs-gloas branch 2 times, most recently from bc977ed to 838fe4b Compare June 24, 2026 13:01
iurii-ssv added a commit that referenced this pull request Jun 28, 2026
Promote EPBS_IMPLEMENTATION_PLAN.md from local-only (.git/info/exclude) into
the branch so the in-flight ePBS planning context is shared, not local. The
file carries an explicit action item: before #2901 is marked ready for review,
move all remaining/useful action items into the PR description and delete this
file — it must not outlive the PR.
@iurii-ssv
iurii-ssv marked this pull request as ready for review June 28, 2026 12:10
@iurii-ssv
iurii-ssv requested review from a team as code owners June 28, 2026 12:10
@iurii-ssv iurii-ssv changed the title ePBS (EIP-7732 / Gloas): node-side foundation — wire types + fork gating ePBS: (EIP-7732 / Gloas / SIP-94) Jun 28, 2026
@greptile-apps

greptile-apps Bot commented Jun 28, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR adds the node-side foundation for ePBS/Gloas. The main changes are:

  • New Gloas wire types and SSZ/JSON encoders.
  • Beacon API clients for PTC, proposer preferences, Gloas blocks, and envelopes.
  • Fork gating and Glamsterdam devnet configuration.
  • PTC, ProposerPreferences, and EnvelopeBuilder runner wiring.
  • Local signer support for the new Gloas domains.

Confidence Score: 4/5

The Gloas message validation and block production paths need fixes before merging.

  • First-fork ProposerPreferences messages can be rejected during the intended lookahead window.
  • Valid payload-included Gloas block responses can fail to decode and skip proposal work.
  • The new runner and type surfaces otherwise look consistent with the inspected changes.

message/validation/signed_ssv_message.go; beacon/goclient/gloas_proposer.go

Important Files Changed

Filename Overview
message/validation/signed_ssv_message.go Adds fork-aware role validation for new Gloas roles, but ProposerPreferences does not account for the intended prior-window broadcast.
beacon/goclient/gloas_proposer.go Adds Gloas block produce and publish clients, but produce response handling only supports the bare block variant.
protocol/v2/types/gloas Adds node-side Gloas wire structures with generated SSZ support and round-trip tests.
protocol/v2/ssv/runner Adds PTC, proposer-preferences, envelope, and Gloas proposer runner paths.
operator/duties Adds duty handlers and scheduler wiring for PTC and ProposerPreferences.

Reviews (1): Last reviewed commit: "gloas: log §2 vote index + §5 proposer-p..." | Re-trigger Greptile

Comment thread message/validation/signed_ssv_message.go Outdated
Comment thread beacon/goclient/gloas_proposer.go
Comment thread message/validation/common_checks.go
Comment thread protocol/v2/ssv/runner/ptc_attester.go Outdated
Comment thread beacon/goclient/proposer_preferences.go
Comment thread networkconfig/glamsterdam-devnet.go Outdated
@iurii-ssv
iurii-ssv requested a review from momosh-ssv June 29, 2026 16:15
iurii-ssv added a commit that referenced this pull request Jun 30, 2026
Comment/doc-only response to a #2901 review (most findings were non-issues or
over-stated — assessed in the plan); the actionable bits:

- beacon_block.go: note blob KZG commitments also leave the body (the payload
  and blobs ship in the §6 envelope) — they were missing from the drops list.
- ptc.go: clarify the hand-rolled client's missing custom-TLS matches the main
  eth2clienthttp path (system-CA https + basic-auth), so it's no regression.
- plan §2: devnet-verify that a Gloas BN accepts the Fulu-tagged attestation
  submission (BeaconForkAtEpoch caps at Fulu; TODO(gloas) to extend if rejected).
- plan §2b: record the remote-signer limitation — Web3Signer has no PTC /
  proposer-preferences / envelope sign types; bounded by f, local-sign
  workaround; operator-facing.
@iurii-ssv

iurii-ssv commented Jun 30, 2026

Copy link
Copy Markdown
Contributor Author

The ePBS validation log that lived in this comment has been moved into the PR description (the Validation section) — so it sits alongside the implementation summary and is easy to find. This comment is kept only as a pointer.

iurii-ssv added a commit that referenced this pull request Jun 30, 2026
Promote EPBS_IMPLEMENTATION_PLAN.md from local-only (.git/info/exclude) into
the branch so the in-flight ePBS planning context is shared, not local. The
file carries an explicit action item: before #2901 is marked ready for review,
move all remaining/useful action items into the PR description and delete this
file — it must not outlive the PR.
iurii-ssv added a commit that referenced this pull request Jun 30, 2026
Comment/doc-only response to a #2901 review (most findings were non-issues or
over-stated — assessed in the plan); the actionable bits:

- beacon_block.go: note blob KZG commitments also leave the body (the payload
  and blobs ship in the §6 envelope) — they were missing from the drops list.
- ptc.go: clarify the hand-rolled client's missing custom-TLS matches the main
  eth2clienthttp path (system-CA https + basic-auth), so it's no regression.
- plan §2: devnet-verify that a Gloas BN accepts the Fulu-tagged attestation
  submission (BeaconForkAtEpoch caps at Fulu; TODO(gloas) to extend if rejected).
- plan §2b: record the remote-signer limitation — Web3Signer has no PTC /
  proposer-preferences / envelope sign types; bounded by f, local-sign
  workaround; operator-facing.
iurii-ssv added a commit that referenced this pull request Jul 1, 2026
The proposer, attester and sync-committee duty-fetch handlers marked an
epoch/period intent fulfilled even when no validators were eligible at fetch
time, so the duties were never fetched once validators did become eligible
(e.g. after a beacon-metadata sync that arrives without an accompanying
indices-change event). On the Gloas devnet this surfaced as the proposer
missing every block it was assigned.

fetchAndProcessDuties now returns (fetched bool, err error); the caller marks
the intent fulfilled only when a beacon fetch actually ran. "No eligible
validators" returns fetched=false, leaving the intent pending so a later tick
retries — the same model the PTC and proposer-preferences handlers already use.

Also drop the now-redundant per-fetch bracket log lines, and add a temporary
proposer diagnostic (#2901) that dumps the Validators()/
SelfValidators() view on no-eligible to confirm the root cause on devnet.
Scheduler tests updated to assert the intent stays pending and that a late
indices-change remains the sole re-fetch trigger.
iurii-ssv added a commit that referenced this pull request Jul 1, 2026
Promote EPBS_IMPLEMENTATION_PLAN.md from local-only (.git/info/exclude) into
the branch so the in-flight ePBS planning context is shared, not local. The
file carries an explicit action item: before #2901 is marked ready for review,
move all remaining/useful action items into the PR description and delete this
file — it must not outlive the PR.
iurii-ssv added a commit that referenced this pull request Jul 1, 2026
Comment/doc-only response to a #2901 review (most findings were non-issues or
over-stated — assessed in the plan); the actionable bits:

- beacon_block.go: note blob KZG commitments also leave the body (the payload
  and blobs ship in the §6 envelope) — they were missing from the drops list.
- ptc.go: clarify the hand-rolled client's missing custom-TLS matches the main
  eth2clienthttp path (system-CA https + basic-auth), so it's no regression.
- plan §2: devnet-verify that a Gloas BN accepts the Fulu-tagged attestation
  submission (BeaconForkAtEpoch caps at Fulu; TODO(gloas) to extend if rejected).
- plan §2b: record the remote-signer limitation — Web3Signer has no PTC /
  proposer-preferences / envelope sign types; bounded by f, local-sign
  workaround; operator-facing.
iurii-ssv added a commit that referenced this pull request Jul 1, 2026
The proposer, attester and sync-committee duty-fetch handlers marked an
epoch/period intent fulfilled even when no validators were eligible at fetch
time, so the duties were never fetched once validators did become eligible
(e.g. after a beacon-metadata sync that arrives without an accompanying
indices-change event). On the Gloas devnet this surfaced as the proposer
missing every block it was assigned.

fetchAndProcessDuties now returns (fetched bool, err error); the caller marks
the intent fulfilled only when a beacon fetch actually ran. "No eligible
validators" returns fetched=false, leaving the intent pending so a later tick
retries — the same model the PTC and proposer-preferences handlers already use.

Also drop the now-redundant per-fetch bracket log lines, and add a temporary
proposer diagnostic (#2901) that dumps the Validators()/
SelfValidators() view on no-eligible to confirm the root cause on devnet.
Scheduler tests updated to assert the intent stays pending and that a late
indices-change remains the sole re-fetch trigger.
momosh-ssv
momosh-ssv previously approved these changes Jul 1, 2026

@momosh-ssv momosh-ssv left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

amazing, gj!

iurii-ssv added a commit that referenced this pull request Jul 1, 2026
Two TEMP logging-only diagnostics to pinpoint where an assigned Gloas proposer
duty is lost between fetch and the runner. The existing zero-eligible diagnostic
only covers the "never fetched" case; these cover the loaded-epoch case:

- logSlotDispatchDiagnostic (processExecution): on any slot carrying a stored
  proposer duty, reports stored_any / in_committee / executable so a run can
  separate an InCommittee-flag drop from a one-slot-window miss from a downstream
  dispatch loss (cross-checked against the existing 🔧 executing validator duty /
  could not find validator logs).
- logFetchDispatchDiagnostic (fetchAndProcessDuties): flags in-committee duties
  stored for already-passed slots (fetched-too-late), and surfaces the InCommittee
  split on the fetch-success path.

Read-only; no control-flow change. Remove after devnet confirmation.
@ovidiu-ssv-labs ovidiu-ssv-labs added this to the QA - Glamsterdam Fork milestone Jul 2, 2026
@iurii-ssv iurii-ssv changed the title ePBS: (EIP-7732 / Gloas / SIP-94) ePBS (EIP-7732 / Gloas) — SIP-94 Jul 2, 2026
RunnerRoleToString/FromString were missing PTC_ATTESTER, PROPOSER_PREFERENCES
and ENVELOPE_PROPOSER — the same registry-drift class as the validRoleUnion gap
(#2999), caught here by the existing round-trip sweep and the observability
lockstep test the moment the pinned spec gave the roles real String() values.
The strings follow the established trimmed-_RUNNER convention, keeping the
mappers in lockstep with ssvtypes.RunnerRoleToString's spec passthrough.
The trace store has no schema for the SIP #94 par.3/par.5/par.6 duties, and with
message validation now admitting their roles on the wire (#2999) the collector
would error per message in toBNRole. Skip the three roles explicitly at the
collect entry instead; tracing them is deliberate future exporter work. The
toBNRole table now pins the intentional non-mapping.
The four Gloas domains that Web3Signer can't yet sign (PTC payload attestation,
proposer preferences, execution-payload envelope, builder request-auth) each
return an explicit "not supported" error with a TODO(gloas). Reference the new
tracking issue (#3000) from each so the code points at the actionable tracker.
Five items from a review of the direct-builder overlay:

- gloasHTTPDo doc: note Eth-Consensus-Version is forced to the Gloas version
  and not overridable by extraHeaders (no caller overrides it).
- Refresh the two role-sweep test bounds: the spec's max RunnerRole is now 9
  (RoleEnvelopeProposer), not 6; the 15 headroom stays.
- Add a publish-echo test asserting the Eth-Builder-Url header actually reaches
  the submitGloasBeaconBlock POST (owner-match forwarding).
- BuildProduceConfig: fold the two defensive decode-skips into the
  authUnavailable count so a (validated-at-startup, unreachable) skip can't drop
  a builder silently.
- Rename the now-generic hand-rolled HTTP helpers ptcHTTPClient -> gloasHTTPClient
  and ptcDo -> jsonDo (they serve all Gloas endpoints, not just PTC);
  genuinely-PTC identifiers untouched.
Decode auth_data / builder_pubkeys and resolve the effective knobs a single time,
at load, into a ResolvedBuilderConfig — the runtime form the §4 produce path and
§5 signing round read — instead of re-parsing operator config every proposal.

- Add ResolvedBuilderConfig / ResolvedBuilderEntry and ResolveBuilderConfig, which
  validates + decodes + resolves in one pass. ValidateBuilderConfig is now a thin
  wrapper over it, removing the duplicate decode validation used to do.
- Each runner resolves once per validator: NewProposerRunner (§4) and
  NewProposerPreferencesRunner (§5, threaded to the per-slot sub-runners).
- BuildProduceConfig and the §5 loop read pre-decoded fields; both drop their
  per-use decode and its unreachable error branch — including the item-4 defensive
  skips folded in the prior commit.
- The §5 signing identity and the §4 auth-cache-lookup identity are now the same
  stored Identity, so they match by construction rather than by re-decoding
  identically at each read.

Behavior-preserving for valid config (the only reachable state — startup
validates); an invalid config now fails fast at runner construction instead of
silently dropping entries. No cli/controller or serialization impact.
Post-review tidy of the direct-builder overlay:

- Document the sharing contract on ResolvedBuilderEntry: its slices are never
  copied (frozen auths alias AuthData, produce bodies alias BuilderPubKeys), so
  treat them as immutable.
- Align ResolvedBuilderConfig.Configured with (*BuilderConfig).Configured on a
  pointer receiver.
- Point the produce telemetry method-label note at issue #2962 (the POST-first
  flip tracks resolving it) and drop the clause that duplicated it.
- Tighten the resolve-once comments to present-tense statements and dedupe the
  gloasHTTPDo consensus-version note.
- Preallocate submittedURLs in the request-auth test (pinned golangci-lint
  prealloc finding).
The producePayloadAttestationData slot moved from a path segment to a query
parameter in ethereum/beacon-APIs#626 (to match produceAttestationData);
Lodestar and Prysm followed. Our hand-rolled GET still used the old path
form, so a spec-compliant beacon node 404s it and the PTC duty (SIP-94 §3)
never submits. That 404 — not a missing CL endpoint — is what #2963 saw:
the duties call keeps its epoch path param and succeeds, only the data call
404s. Reported in #3001.

Point the constant at the query form, matching the sibling attestation_data
call already in this package.

Also handle the endpoint's 204 No Content, the spec's "no block seen for
this slot" abstain signal: httpDo now returns the status code so
requestPayloadAttestationData yields (nil, nil) on a 204 instead of failing
to decode the empty body, and the runner treats nil data as the §3 abstain
(markDutyNotRequired) rather than marking the duty failed.
The header key was a bare string literal at five goclient call sites (only
its value, consensusVersionGloas, was already a constant). Add
consensusVersionHeader alongside it and use it everywhere, clearing the
goconst lint hit. Tests keep the literal, to pin the wire header independently.
produceBlockV4 became POST-only with a required BuilderConfig body in
ethereum/beacon-APIs#630 (merged 2026-08-24); Teku already dropped the GET.
The node sent a bare GET whenever no direct builder was configured — always,
today — so Gloas block production fails against any spec-current beacon node
(only Lighthouse/Lodestar/Prysm still keep the GET, hiding it for now).

Always POST the BuilderConfig body: the direct-builder overlay when
configured, else a neutral local-build config (empty builders,
builder_boost_factor 100 — SSV's neutral, so p2p bids compete at par with the
local build). The per-node GET fallback stays for pre-#630 nodes, only on a
404/405. Docs and comments describing the old "unconfigured -> GET" behavior
are updated to match.

Reported in #3002; completes the POST-first flip deferred in this PR.
Review follow-ups on the PTC (#3001) and produceBlockV4 (#3002) fixes:

- Add a runner unit test for the PTC abstain path: executeDuty calls
  markDutyNotRequired and freezes no observation when the beacon node reports
  no block — nil data (a 204) or, defensively, a 200 with an all-zero root.
- Fix a stale doc comment on gloasBuilderConfig: a nil config no longer means
  "the enshrined GET"; the goclient now POSTs a neutral local-build config.
- Document that a 204 stops the client fallback (an authoritative "no block",
  so the operator abstains on its own node's view rather than polling the
  rest) — a deliberate multi-BN choice, not an accident.
…signed-blinded code

beacon-APIs#624 removed the BlindedExecutionPayloadEnvelope container from the
spec. SSV keeps a type of that name purely as the §6 QBFT consensus value (the
full SignedExecutionPayloadEnvelope is what's published), so document that and
drop the now-dead code for the removed blinded publication path:

- Note on BlindedExecutionPayloadEnvelope: an SSV-internal consensus value
  only, never on the wire (#624 removed the identically named spec container).
- Remove SignedBlindedExecutionPayloadEnvelope, SignedExecutionPayloadEnvelope
  .Blinded(), their generated SSZ, the round-trip test, and the sszgen obj.
- Correct the stale "blinded publication path" comments to say #624 removed the
  blinded body; the only deferred alternative is the full-envelope Contents.

Confirms #3003 (BlindedExecutionPayloadEnvelope stays an SSV-internal
container); SIP-94 §6 already documents this ("No public
SignedBlindedExecutionPayloadEnvelope is used"), so no SIP change is needed.
…publish

beacon-APIs#624 makes Eth-Blob-Data-Included a required header on POST
/eth/v1/beacon/execution_payload_envelopes: it selects the request body —
false for the full SignedExecutionPayloadEnvelope (stateful flow, the beacon
node attaches the blobs it cached at production), true for the blobs-carrying
Contents — and the SSZ octet-stream body is ambiguous without it. The node
published the full envelope with no such header, so a spec-compliant beacon
node rejects it (the same class as the produceBlockV4 gap in #3002); it only
worked because Lodestar v1.43.0 implements just the full form.

Set Eth-Blob-Data-Included: false on the envelope publish, and correct two
comments that still named the old #580 Eth-Execution-Payload-Blinded header
(the pre-#624 §6 name). The pre-Gloas produceBlockV3 use of that header is
unaffected.
The old ssv-spec pin pointed at an orphaned commit; move both go.mods (root
and ssvsigner) to the current epbs-gloas-types head.

That head splits the generic spectypes.NewMsgID(domain, []byte, role) into the
fixed-size spectypes.NewValidatorMsgID / spectypes.NewCommitteeMsgID. Migrate
all production call sites to the typed constructors — output is byte-identical,
so message ids stay wire-compatible: validator-keyed runners use
NewValidatorMsgID, committee-keyed ones NewCommitteeMsgID, and the generic
[]byte identifier resolver picks by executor length.

Tests build synthetic and intentionally-malformed ids from arbitrary-length
executor bytes, which the fixed-size constructors cannot express, so add
protocol/v2/types/ssvtestingutils.NewMsgID — a test-only reproduction of the
removed constructor — and repoint the test call sites at it.

Also:
- type the []byte validator-pubkey params of the signAndBroadcast* and
  dutyDataToSSVMsg helpers as spectypes.ValidatorPK, dropping [:] at callers;
- build the identifier directly per switch case in the testing runner helper;
- go mod tidy: prune stale ssv-spec go.sum entries and mark golang.org/x/exp
  (imported directly by the ssv spectest helpers) as a direct dependency.
local-testnet was the only network config scheduling Boole at genesis
(Boole: 0). Pin it to math.MaxUint64 like the production networks so the
default local substrate matches production through the Glamsterdam (Gloas/
ePBS) transition, where Boole stays dormant. The post-Boole path is still
reachable via an explicit epoch in a custom config or SSV_TEST_BOOLE_FORK=post.
…oost policy

Review follow-ups on the recent commit range:

- beacon/goclient: hoist the §6 envelope publish body selector into a
  blobDataIncludedHeader constant next to the envelope endpoints (tests keep
  the literal, pinning the wire header independently).
- docs/EXTERNAL_BUILDERS.md: state what the unconfigured cluster's neutral
  produce body means — builder_boost_factor 100 weighs gossiped p2p bids at
  par with the local build, the same default the pre-#630 GET applied — and
  name the local-preferred opt-out (a knobs-only BuilderBoostFactor: 0).
- ssvtestingutils.NewMsgID: document the one deliberate divergence from the
  removed spec constructor (an oversized executor id is truncated to the
  48-byte slot instead of overflowing into the role bytes).
The repinned ssv-spec head moved its generated vectors out of the module into
the sibling ssvlabs/spec-tests repo: its UnmarshalStateComparison now resolves
<module go.mod root>/../spec-tests/<module>, and its vector generator writes
there, panicking without a go.mod above its cwd. Neither path can exist under
the Go module cache, so TestQBFTMapping/TestSSVMapping died generating vectors
and the alan variants failed all 110 msg-processing state comparisons (the
alan pin still ships vectors in-module, but the new resolver looks beside the
module root instead).

Teach the harness both layouts:

- GenerateSpecTestJSON runs the split-layout generator from a scaffold dir
  carrying a go.mod inside the artifact cache, so it writes to
  <artifact>/spec-tests/<module>; the generated state_comparison vectors are
  kept for the per-test overrides, and the cached-tests fast path applies
  only while those vectors are still available.
- New storage helpers (StateComparisonDir / ReadStateComparisonFile /
  UnmarshalStateComparison) resolve the state-comparison vectors in-module
  for pre-split pins (alan) and from the artifact scaffold for split pins,
  regenerating on a cold cache. All spectest call sites are repointed at
  them, replacing the spec's own resolver, which hardcodes the
  sibling-checkout paths.
…ped runner vectors

The repinned ssv-spec added spec-side Gloas runners (PTC attester, proposer
preferences, envelope proposer) with spec tests, Gloas-era variants of the
shared runner/committee vectors, and Gloas valcheck suites. Wire what the node
can genuinely run today and skip the rest explicitly:

- The Gloas valcheck vectors run against the node's real checkers:
  NewGloasVoteChecker is selected for Gloas-era committee vectors by the
  spec's testing fork epoch (mirroring the committee runner's slot-keyed
  selection), the proposer checker gets a Gloas-scheduled beacon config, and
  NewEnvelopeChecker runs over the spec's seeded §4-root store.
- Two node fixes those vectors flushed out: the §6 envelope value check now
  pins the wire-carried duty's beacon role (slot/index/pubkey were already
  pinned), and the validator-registration runner gains the runner-side Gloas
  deprecation guard matching ssv-spec's (the scheduler drain and the wire
  rejection already existed). The guard is covered by a node unit test, since
  the spec's own vector for it cannot run yet (below).
- The Gloas checkers' error codes are aligned with the spec's new dedicated
  codes (DecodeGloasBeaconVote, the Envelope* family).
- The unmapped Gloas vectors are skipped under one documented reason: the new
  runner roles (the node's implementations deliberately differ in internals)
  and the Gloas-era variants of shared vectors (their post-state roots embed
  the network config, which the ssvtesting constructors cannot schedule Gloas
  into without diverging those roots; the testing beacon-node wrapper also
  has no Gloas produce surface). Also skipped: the new different-slots
  aggregator-committee vector, whose expected post-state records the runner
  before the failing duty validation, where the node validates first.

Together with the split-layout support this turns the PR's spec-test CI green;
mapping the skipped vectors onto the node is tracked as a repin follow-up.
The skip reason and TODOs referenced the PR as the tracker; issue #3017 now
tracks the mapping work durably, so reference it instead.
Bump the Ethereum deps that don't require the go-eth2-client Gloas/dynamic-ssz
migration (tracked in #3014):

  go-ethereum  1.16.4 -> 1.17.5
  fastssz      0.1.4  -> 1.0.0
  herumi/bls   1.29.1 -> 1.37.0
  prysm/v4     4.0.8  -> 4.2.1   (root)
  wealdtech    go-eth2-types/util 2.8.1->2.8.2 / 1.8.1->1.8.2 (root),
               keystorev4 1.1.3->1.4.1 (ssvsigner)

go-eth2-client, ssv-spec and eth2-key-manager stay pinned: bumping go-eth2-client
to the Gloas fork forces a coordinated dynamic-ssz migration across ssv-spec and
eth2-key-manager (#3014). Auxiliary modfiles are left untouched (go.spec.alan.mod
is a frozen spec-vector snapshot; tool.mod is codegen tooling).

Also realign the two go-ethereum source permalinks in network/discovery/shared_conn.go
to v1.17.5 (the cited lines and the size 100 are unchanged there).
…-ssz

protocol/v2/types/gloas now aliases go-eth2-client's spec/gloas for the Gloas
(ePBS) beacon-chain containers — the canonical, progressive-SSZ-correct types —
instead of hand-rolling them (~4k fewer lines). Only SSV-owned types remain:
the consensus values (GloasBeaconVote, EnvelopeConsensusData, the blinded
envelope) and the node types ssv-spec lacks (builder config/entry, PTC duty,
builder request auth), with encoders now generated by pk910/dynamic-ssz —
fastssz cannot merkleize the progressive containers/lists.

go-eth2-client's move to dynamic-ssz drops fastssz's HashRoot (GetTree), so
ssz.HashRoot becomes the narrow spectypes.HashRoot across the runner package,
consensus_data.go, and ssvsigner; go-bitfield switches to OffchainLabs where
go-eth2-client types are composed. BlockSlot()/Blinded() become free
functions, and ssvsigner's block-signing path type-switches on
*gloas.BeaconBlock directly, dropping the structural slashableBeaconBlock
interface.

Drop the SSZ round-trip/JSON tests of the now-aliased types (go-eth2-client
and ssv-spec's dynssz parity suite own that coverage). Repin ssv-spec,
eth2-key-manager, and the go-eth2-client fork in both modules.
…-ups)

Address the review findings on the ePBS branch.

- roundtimer: the §6 envelope proposer gets the proposer's round-relative
  timeouts. It fell into the slot-anchored branch with a zero head start,
  so round 1 expired the moment the instance started whenever the §4 block
  published later than QuickTimeout into the slot, and with the envelope
  round cap of 2 peers then rejected the round. RoundRelativeRole is the
  one predicate the timer and message validation's round-spread exemption
  now share, so the two cannot drift apart again.

- §6: an operator whose beacon node cannot produce the envelope (every
  non-builder on a cluster whose operators run separate beacon nodes) joins
  the QBFT instance as a voter instead of failing the duty before the
  instance exists, which left the round short of a quorum.
  Controller.JoinInstance starts the instance without a value; a leader
  with an empty start value skips its round-1 proposal rather than
  broadcast an empty one; a later round it leads re-proposes only an
  already-prepared value, since the justification's value check rejects an
  empty one. The round still relies on the builder leading round 1, which
  holds whenever §4 decided in round 1.

- duties/proposer: a reorg refetches the current epoch only when the
  current dependent root changed, as its comment already said. The
  unconditional refetch raced the idle-time indices-change case and made
  TestScheduler_Proposer_Reorg_Previous_Indices_Changed flaky.

- §5: a partial for a proposal slot with no local sub-runner is a plain
  error; the dispatcher stash replays it on duty start, so a queue retry
  only churned.

- proposer: reset the Gloas produce markers per duty so a stale owner-match
  cannot echo a previous slot's Eth-Builder-Url, and treat a produced block
  that fails to hash as an error. The envelope runner requires its root
  store at construction.

- DataVersionGloas aliases the fork's spec.DataVersionGloas (same value) in
  both modules; correct the request-auth cache comment and the goclient
  comments that claimed go-eth2-client has no Gloas types; the beacon
  address normalizer matches the two schemes explicitly; drop the unrelated
  tla/ ignore entries.
Picks up the #643 review round: valid max-attestation fixtures in the maxmsgsize
suite, module-graph-resolved go-eth2-client includes in the generators,
ExecutionPayloadBid.PrevRandao aligned with the fork, and doc/cleanup touch-ups.
ssv-spec test/generator/doc changes only — the node builds unchanged and
TestSSVMapping stays green against the regenerated vectors.
otel/sdk v1.41 changed resource.Default() to advertise OTel schema URL
1.39.0, which conflicts with the hand-built resource pinned to
semconv/v1.37.0 in buildResources. resource.Merge returns
ErrSchemaURLConflict and buildResources treats it as fatal, so the node
aborted at startup during observability init before any duty logic ran.

Bump the semconv import v1.37.0 -> v1.39.0 across all in-tree users
(v1.39.0 already ships in otel@v1.41.0, so no go.mod change). Only
resources.go triggered the crash; the other files are metric-name-only and
their attribute keys are value-identical between 1.37 and 1.39, so emitted
metrics are unchanged. Also note the version-alignment requirement on the
import and fix an "OTeL" typo.

Fixes #3020.
… roots

Aggregator duties failed on Gloas slots with "unknown data version: 8" (#3009): the fork already
decodes the gloas Eth-Consensus-Version into VersionedAttestation.Gloas and ships a distinct
gloas.AggregateAndProof (its Attestation merkleizes progressively, so its root differs from the
byte-identical Electra container's), but the node's version switches had no Gloas arm. Add it to the
goclient aggregate conversions, the consensus-data decode, and a signed-aggregate constructor now
shared by both aggregator runners; the remote signer rejects Gloas aggregates explicitly (#3000).

Drop ForkAtEpoch's Fulu cap in the node and ssvsigner: a scheduled Gloas fork resolves from its epoch
on and absent forks are skipped, so attestations ride gloas.Attestation and the aggregator-committee
consensus data stamps DataVersionGloas (SIP #94 §2, #2998). This needs ssv-spec's Gloas arms to build
the Gloas containers, so both go.mods repin to the ssv-spec dynssz branch head that carries them
(ssvlabs/ssv-spec#643). The remote signer's fork_info substitution is no longer needed.

Pin the Gloas roots to the chain (#3008): a devnet-8 finalized block whose HashTreeRoot must equal the
chain's header root and whose proposer signature must verify, plus its execution-payload envelope,
which must blind to the full envelope's root. EIP-7688/7916 change only merkleization, so round-trip
tests alone cannot catch a wrong root.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants