Attestation data service - #24
Open
eserilev wants to merge 368 commits into
Open
Conversation
Ari4ka
approved these changes
Feb 16, 2026
Co-Authored-By: dapplion <35266934+dapplion@users.noreply.github.com>
Since sigp#6847, invalid `BlocksByRange`/`BlobsByRange` requests, which do not comply with the spec, are [handled in the Handler](https://github.com/sigp/lighthouse/blob/3d16d1080f5b93193404967dcb5525fa68840ea0/beacon_node/lighthouse_network/src/rpc/handler.rs#L880-L911). Any peer that sends an invalid request is penalized and disconnected. However, other kinds of invalid rpc request, which result in decoding errors, are just dropped. No penalty is applied and the connection with the peer remains. I have added handling for the `ListenUpgradeError` event to notify the application of an `RPCError:InvalidData` error and disconnect to the peer that sent the invalid rpc request. I also added tests for handling invalid rpc requests. Co-Authored-By: ackintosh <sora.akatsuki@gmail.com>
Co-Authored-By: Jimmy Chen <jchen.tc@gmail.com>
N/A Handle snappy crate errors as InvalidData instead of IoError. Co-Authored-By: Pawan Dhananjay <pawandhananjay@gmail.com>
Fix a bug in v8.1.0 whereby the VC times out continuously with:
> Feb 18 02:03:48.030 WARN Head service failed retrying starting next slot error: "Head monitoring stream error, node: 0, error: SseClient(Transport(reqwest::Error { kind: Decode, source: reqwest::Error { kind: Body, source: TimedOut } }))"
- Remove the existing timeout for the events API by using `Duration::MAX`. This is necessary as the client is configured with a default timeout. This is the only way to override/remove it.
- DO NOT add a `read_timeout` (yet), as this would need to be configured on a per-client basis. We do not want to create a new Client for every call as the early commits on this branch were doing, as this would bypass the TLS cert config, and is also wasteful.
Co-Authored-By: hopinheimer <knmanas6@gmail.com>
Co-Authored-By: Michael Sproul <michael@sigmaprime.io>
Co-Authored-By: Michael Sproul <michaelsproul@users.noreply.github.com>
Co-Authored-By: Jimmy Chen <jchen.tc@gmail.com>
Fix the failure of the beacon-chain tests for phase0/altair, which now only runs nightly. Just skip the payload invalidation tests, they don't make any sense prior to Bellatrix anyway. Co-Authored-By: Michael Sproul <michael@sigmaprime.io>
N/A Another find by @gitToki. Sort the preferred_ids in descending order as originally intended from the comment in the function. Co-Authored-By: Pawan Dhananjay <pawandhananjay@gmail.com>
Co-Authored-By: Jimmy Chen <jchen.tc@gmail.com>
sigp#8547 Bump the following crates to remove `hashlink 0.8`: - `rusqlite` - `r2d2-sqlite` - `yaml-rust2` Co-Authored-By: Mac L <mjladson@pm.me>
Co-Authored-By: Jimmy Chen <jchen.tc@gmail.com>
Co-Authored-By: Jimmy Chen <jchen.tc@gmail.com>
Which issue # does this PR address?
None
All of these are performing a check, and adding a batch, or creating a new lookup, or a new query, etc..
Hence all of these limits would be off by one.
Example:
```rust
// BACKFILL_BATCH_BUFFER_SIZE = 5
if self.batches.iter().filter(...).count() >= BACKFILL_BATCH_BUFFER_SIZE {
return None; // ← REJECT
}
// ... later adds batch via Entry::Vacant(entry).insert(...)
```
Without the `>` being changed to a `>=` , we would allow 6. The same idea applies to all changes proposed.
Co-Authored-By: Antoine James <antoine@ethereum.org>
Co-Authored-By: Jimmy Chen <jimmy@sigmaprime.io>
Co-Authored-By: Jimmy Chen <jchen.tc@gmail.com>
Co-Authored-By: Eitan Seri-Levi <eserilev@ucsc.edu> Co-Authored-By: Eitan Seri- Levi <eserilev@gmail.com> Co-Authored-By: Michael Sproul <michaelsproul@users.noreply.github.com>
Co-Authored-By: Jimmy Chen <jchen.tc@gmail.com>
I found myself having to update this code for Gloas, and figured we may as well delete it seeing as it doesn't work. See: - sigp#4198 Delete all `fork_revert` logic and the accompanying test. Co-Authored-By: Michael Sproul <michael@sigmaprime.io>
Co-Authored-By: Jimmy Chen <jchen.tc@gmail.com>
Co-Authored-By: dapplion <35266934+dapplion@users.noreply.github.com>
Co-Authored-By: Jimmy Chen <jchen.tc@gmail.com>
Co-Authored-By: Michael Sproul <michael@sigmaprime.io> Co-Authored-By: Michael Sproul <michaelsproul@users.noreply.github.com>
Co-Authored-By: Jimmy Chen <jchen.tc@gmail.com>
We received a bug report of a node restarting custody backfill unnecessarily after upgrading to Lighthouse v8.1.1. What happened is:
- User started LH v8.0.1 many months ago, CGC updated 0 -> N but the CGC was not eagerly persisted.
- LH experienced an unclean shutdown (not sure of what type).
- Upon restarting (still running v8.0.1), the custody context read from disk contains CGC=0: `DEBUG Loaded persisted custody context custody_context: CustodyContext { validator_custody_count: 0, ...`).
- CGC updates again to N, retriggering custody backfill: `DEBUG Validator count at head updated old_count: 0, new_count: N`.
- Custody backfill does a bunch of downloading for no gain: `DEBUG Imported historical data columns epoch: Epoch(428433), total_imported: 0`
- While custody backfill is running user updated to v8.1.1, and we see logs for the CGC=N being peristed upon clean shutdown, and then correctly read on startup with v8.1.1.
- Custody backfill keeps running and downloading due to the CGC change still being considered in progress.
- Call `persist_custody_context` inside the `register_validators` handler so that it is written to disk eagerly whenever it changes. The performance impact of this should be minimal as the amount of data is very small and this call can only happen at most ~128 times (once for each change) in the entire life of a beacon node.
- Call `persist_custody_context` inside `BeaconChainBuilder::build` so that changes caused by CLI flags are persisted (otherwise starting a node with `--semi-supernode` and no validators, then shutting it down uncleanly would cause use to forget the CGC).
These changes greatly reduce the timespan during which an unclean shutdown can create inconsistency. In the worst case, we only lose backfill progress that runs concurrently with the `register_validators` handler (should be extremely minimal, nigh impossible).
Co-Authored-By: Michael Sproul <michael@sigmaprime.io>
sigp#8959 WIP still working on adding more re-org tests and refactoring existing. Co-Authored-By: hopinheimer <knmanas6@gmail.com> Co-Authored-By: Michael Sproul <michael@sigmaprime.io>
…gp#9361) This PR fixes two issues: 1. This condition is inverted: https://github.com/sigp/lighthouse/blob/dfb259171a65cacd6db57b8874af8f543cabcb7a/beacon_node/network/src/network_beacon_processor/gossip_methods.rs#L1507-L1508 We are supposed to filter out incomplete columns when we DON'T have local blobs yet! 2. When the EL returns no blobs, we never store a partial in the assembler, and this code fails to publish our need to the network, as no partials are returned: https://github.com/sigp/lighthouse/blob/dfb259171a65cacd6db57b8874af8f543cabcb7a/beacon_node/network/src/network_beacon_processor/mod.rs#L1038-L1050 The simple fix for 1 would be to invert the condition, but we can improve the flow here: Instead of not publishing anything, we can publish what we got, but not request anything. This ties into the fix for 2: After get blobs completes, we not only publish anything in the partial assembler, but also for every missing custody column in there, publish an empty column and a request for all cells. In particular: - When sending a partial message to `network`, allow specifying a request bitmap instead of hardcoding an all-ones bitmap. - For clarity and to prepare for Gloas integration, add a `PubsubPartialMessage` enum with a `DataColumnFulu` variant. - On republishing after merging a gossip column: always publish, but only request cells if local blobs are known or get blobs is disabled. This also prepares us to request only *some* cells, e.g. in cases where we are aware of the blobs that the EL is going to send us, e.g. via `engine_hasBlobs`. - Move guards in `fetch_engine_blobs_and_publish` to ensure everything works fine if there are no blobs or if get_blobs is disabled. Co-Authored-By: Daniel Knopik <daniel@dknopik.de>
…erences (sigp#9309) The proposer preferences service was attempting to publish preferences at the start of each epoch. This caused it to race with the validator duties service, it wouldn't calculate validator duties in time for the proposer preference service. This PR first updates the validator duties service to calculate proposer duties for the current epoch and the next epoch. After Fulu we have the ability to look ahead one epoch for proposer duties, but we never updated the vc to leverage this feature. This PR also updates the proposer preferences service to fire at every slot. We have an `(Epoch, DependentRoot)` map that prevents us from publishing the same preferences twice. The changes here should prevent the race condition between the two services and make the proposer preferences service more robust in general. Co-Authored-By: Eitan Seri- Levi <eserilev@gmail.com> Co-Authored-By: Eitan Seri-Levi <eserilev@ucsc.edu> Co-Authored-By: Michael Sproul <michaelsproul@users.noreply.github.com>
This is needed to connect to buildoor (Kurtosis package, acts as a trustless builder) Co-Authored-By: Eitan Seri- Levi <eserilev@gmail.com> Co-Authored-By: Eitan Seri-Levi <eserilev@ucsc.edu> Co-Authored-By: Michael Sproul <michaelsproul@users.noreply.github.com>
…gp#9456) Co-Authored-By: dapplion <35266934+dapplion@users.noreply.github.com>
- Remove deprecated geth flag - Remove duplicate entries for crate `syn` Co-Authored-By: Eitan Seri-Levi <eserilev@ucsc.edu>
I noticed that `beacon_node/http_api/src/beacon/execution_payload_envelope.rs` was recently removed but not added to the forbidden-files.txt. Add the removed file to the forbidden list to ensure it isn't accidentally re-added by a merge or rebase. Co-Authored-By: Mac L <mjladson@pm.me>
Co-Authored-By: dapplion <35266934+dapplion@users.noreply.github.com> Co-Authored-By: Pawan Dhananjay <pawandhananjay@gmail.com>
Co-Authored-By: dapplion <35266934+dapplion@users.noreply.github.com> Co-Authored-By: Pawan Dhananjay <pawandhananjay@gmail.com>
Co-Authored-By: dapplion <35266934+dapplion@users.noreply.github.com> Co-Authored-By: Pawan Dhananjay <pawandhananjay@gmail.com>
Remove some `is_gloas` checks that are unnecessary in the `gloas_reorg_tests.rs`. I found myself wanting to make this change while tweaking these tests in another PR. Figured it makes sense as a simple standalone PR. Co-Authored-By: Michael Sproul <michael@sigmaprime.io> Co-Authored-By: hopinheimer <48147533+hopinheimer@users.noreply.github.com>
`dequeue_attestations` released votes by splitting the queue at the first entry with `slot >= current_slot`, which assumes the queue is sorted by slot. It isn't: `on_attestation` pushes attestations in arrival order and never sorts. When a future-slot vote sits ahead of a vote that is already due, the split happens at the future-slot vote and the due vote stays stuck behind it and is never applied to fork choice, even after its slot is in the past. The PR current uses a naive solution to solve the bug and also adds regression tests to exercise the bug. There are other competing solutions which can be used which also optimize this path at the same time. sigp#8378 sigp#8378 (comment) Co-Authored-By: hopinheimer <knmanas6@gmail.com> Co-Authored-By: hopinheimer <48147533+hopinheimer@users.noreply.github.com> Co-Authored-By: Michael Sproul <michael@sigmaprime.io>
Co-Authored-By: Mark Mackey <mark@sigmaprime.io>
- Bump `warp` to 0.4. This unifies `warp` and `axum` onto the same `http`, `hyper`, `h2`, `rustls`, etc versions. - Create `axum_utils` which contain common functions and types - Begins migration of all HTTP API servers from warp to axum Co-Authored-By: Mac L <mjladson@pm.me>
- Generalize the custody-by-root request to accept a `Vec<Hash256>` of block roots so a whole batch is fetched in one request; `DataColumnsByRootRequestParams { block_roots, indices }` completes at `block_roots.len() * indices.len()`.
- `custody_lookup_request` takes `block_roots: &[Hash256]` + `block_epoch`; `cached_data_column_indexes` takes `block_epoch`.
- `block_sidecar_coupling.rs`: couple a batch's blocks with the columns returned by the single custody-by-root request; drop the per-column request map and retry/faulty-peer machinery.
- Remove the now-unused columns-by-range range-sync path and the serialize-downloads gate.
Co-Authored-By: dapplion <35266934+dapplion@users.noreply.github.com>
Range/backfill sync already de-prioritizes peers that failed a batch download (batch.failed_peers() -> peers_to_deprioritize). Block and payload lookups did not: SingleLookupRequestState only tracked a failure counter, not which peer failed, so a retry could re-select the same flaky peer. Track failed_peers per request-state, record the peer on download failure, and feed it into the block_lookup_request / payload_lookup_request peer-selection sort key, mirroring range sync. Custody lookups already have their own per-peer-attempt mechanism and are left unchanged. Co-Authored-By: dapplion <35266934+dapplion@users.noreply.github.com>
…igp#9548) Closes sigp#9545 `BeaconNodeHttpClient::eth_path` clones the configured server URL and pushes path segments onto it. When the base URL ends with a slash (such as a QuickNode-style endpoint that embeds the API key in the path, `https://<host>/<api-key>/`), the URL already carries an empty trailing segment, so pushing `eth` yields a double slash: ``` https://<host>/<api-key>//eth/v2/debug/beacon/states/finalized ``` The provider responds to the `//` form with a `307` redirect to a normalized path that drops the key segment, so the redirected request arrives unauthenticated and checkpoint sync fails with `Error loading checkpoint state from remote: StatusCode(401)`. This PR calls `pop_if_empty()` before pushing segments, the standard `url`-crate idiom for trailing-slash-tolerant joining, so a trailing slash no longer changes the resulting path. The fix is applied in `eth_path` (the shared prefix builder that every `/eth/vX` endpoint derives from) and in `post_lighthouse_liveness`, the only other builder that extends the base URL directly. Co-Authored-By: rahulbarman <itsrahulbarman1@gmail.com>
Alpha spec 11 changes Co-Authored-By: Eitan Seri-Levi <eserilev@ucsc.edu> Co-Authored-By: Michael Sproul <michael@sigmaprime.io>
Tweaking/optimising `get_expected_withdrawals` on the pre-Gloas path, after the removal of some optimisations in: - sigp#9102 - Remove `prepare_beacon_proposer` calls from `register_validator`. These calls were: heavily duplicated, happening _too early_ (at the start of the slot before the block arrived), expensive on epoch boundaries (forcing an early epoch transition), _and_ unnecessary. They are completely safe to remove because the scheduled proposer preparation routine exists. - Instrument `get_expected_withdrawals`/`prepare_beacon_proposer` to find more unexpected callers. Co-Authored-By: Michael Sproul <michael@sigmaprime.io> Co-Authored-By: Tan Chee Keong <tanck@sigmaprime.io> Co-Authored-By: Michael Sproul <michaelsproul@users.noreply.github.com>
sigp#8817 we were not emitting payload attribute SSE events for gloas. Since the head snapshot stores the canonical latest execution payload envelope we have all the info we need to emit the event. Co-Authored-By: Eitan Seri-Levi <eserilev@ucsc.edu> Co-Authored-By: Eitan Seri-Levi <eserilev@gmail.com>
Co-Authored-By: Age Manning <Age@AgeManning.com> Co-Authored-By: Daniel Knopik <daniel@dknopik.de> Co-Authored-By: João Oliveira <hello@jxs.pt>
Co-Authored-By: banteg <4562643+banteg@users.noreply.github.com>
Ensure we are using the correct state when validating proposer preferences over gossip. Previously we were only using the head state. At epoch boundaries the head state could be at `current_epoch - 1`. Peers submitting proposer preferences for `current_epoch + 1` would be penalized because our head states lookahead did not have proposer duties for `current_epoch + 1` Co-Authored-By: Eitan Seri-Levi <eserilev@ucsc.edu>
Move attestation data fetching out of the attestation service into a dedicated AttestationDataService. Add first_success_with_index and first_success_from_index to BeaconNodeFallback so callers can prefer a specific beacon node and learn which node served the response. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
eserilev
force-pushed
the
attestation-data-service
branch
from
July 6, 2026 11:52
833e074 to
a3e4868
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Issue Addressed
Which issue # does this PR address?
Proposed Changes
Please list or describe the changes introduced by this PR.
Additional Info
Please provide any additional information. For example, future considerations
or information useful for reviewers.