Implement PTC reprocess queue by conache · Pull Request #9458 · sigp/lighthouse

conache · 2026-06-11T17:47:04Z

Issue Addressed

This PR closes issue #9421.

Proposed Changes

Payload attestation messages that reference an unknown block are now queued for reprocessing and retried once the block is imported, instead of being dropped.
Retained the existing sync lookup trigger; the gossipsub validation result is withheld until the retry resolves so late-validated votes still propagate.
Follows the same reprocess-queue pattern used by unaggregated/aggregated attestations (UnknownBlockUnaggregate).
Includes e2e tests mirroring attestation_to_unknown_block_processed (gossip and RPC block import variants).

…c-payload-attestations-reprocess-queue # Conflicts: # beacon_node/beacon_processor/src/scheduler/work_reprocessing_queue.rs # beacon_node/network/src/network_beacon_processor/tests.rs

eserilev

Looks pretty good, just some small changes

Also would be good to add reprocess metrics, see BEACON_PROCESSOR_AGGREGATED_ATTESTATION_REQUEUED_TOTAL and BEACON_PROCESSOR_UNAGGREGATED_ATTESTATION_REQUEUED_TOTAL as examples

We can introduce a BEACON_PROCESSOR_PAYLOAD_ATTESTATION_REQUEUED_TOTAL when a payload attestation gets queued for reprocessing

Also we have a test case requeue_unknown_block_gossip_attestation_without_import. I think we could add a similar test cases for payload attestations.

eserilev · 2026-06-23T13:02:25Z

+            // PTC size ~512 per slot, buffer 2-3 slots for reorgs and processing delays (512 * 3 = 1536)
+            // TODO(EIP-7732): verify if this is preferable queue length or otherwise


yeah this is fine, you can remove this TODO

eserilev · 2026-07-07T10:01:32Z

                self.send_sync_message(SyncMessage::UnknownBlockHashFromAttestation(
                    peer_id,
                    *beacon_block_root,
-                ))
+                ));
+
+                if allow_reprocess {
+                    // Queue the payload attestation for re-processing
+                    let processor = self.clone();
+                    let msg = ReprocessQueueMessage::UnknownBlockPayloadAttestation(


we call UnknownBlockHashFromAttestation both inside and outside the if allow_reprocess. we should only call it inside the if allow_reprocess

Should be fixed in 577bc98

eserilev · 2026-07-07T10:12:45Z

        error: PayloadAttestationError,
-        message_slot: Slot,
+        payload_attestation_message: Box<PayloadAttestationMessage>,
+        allow_reprocess: bool,


PR #9440 introduced a ReprocessAllowance enum. We should use that instead of the allow_reprocess flag. We'd probably want to add a new variant to the enum BlockOnly

Thanks for the context on this! I added the new variant and switched to using the ReprocessAllowance enum in 918ea83

…reporcess bool flag

conache · 2026-07-08T08:45:18Z

Looks pretty good, just some small changes

Also would be good to add reprocess metrics, see BEACON_PROCESSOR_AGGREGATED_ATTESTATION_REQUEUED_TOTAL and BEACON_PROCESSOR_UNAGGREGATED_ATTESTATION_REQUEUED_TOTAL as examples

We can introduce a BEACON_PROCESSOR_PAYLOAD_ATTESTATION_REQUEUED_TOTAL when a payload attestation gets queued for reprocessing

Also we have a test case requeue_unknown_block_gossip_attestation_without_import. I think we could add a similar test cases for payload attestations.

Thank you for the review and the guidance! 🙏 I addressed the comments and added the suggested test and metric.
The PR should be ready for another look.

eserilev

LGTM!

mergify · 2026-07-08T13:22:01Z

Queued — the merge queue status continues in this comment ↓.

mergify · 2026-07-08T13:22:37Z

Merge Queue Status

✅ Entered queue — 2026-07-08 13:22 UTC · Rule: default · triggered by @eserilev with the merge queue checkbox
❌ Checks failed · on draft merge queue: checking unstable (81d5769) and #9458 together #9599
🚫 Left the queue — 2026-07-08 13:51 UTC · at bc49d70551037e1f7b28e81952812d8d1277304c

This pull request spent 29 minutes in the queue, including 27 minutes 12 seconds running CI.

Waiting for

check-success=local-testnet-success

All conditions

check-success=local-testnet-success
check-success=test-suite-success
github-review-approved [🛡 GitHub branch protection]
- Implement PTC reprocess queue #9458

Reason

The merge conditions cannot be satisfied due to failing checks

local-testnet-success

Hint

You may have to fix your CI before adding the pull request to the queue again.
If you update this pull request, to fix the CI, it will automatically be requeued once the queue conditions match again.
If you think this was a flaky issue instead, you can requeue the pull request, without updating it, by posting a @mergifyio queue comment.

Requeued — the merge queue status continues in this comment ↓.

eserilev · 2026-07-08T14:18:52Z

@mergify requeue

mergify · 2026-07-08T14:19:03Z

Merge Queue Status

✅ Entered queue — 2026-07-08 14:19 UTC · Rule: default · triggered by @eserilev with the @mergifyio queue command
❌ Checks failed · on draft merge queue: checking unstable (81d5769) and #9458 together #9601
🚫 Left the queue — 2026-07-08 14:49 UTC · at bc49d70551037e1f7b28e81952812d8d1277304c

This pull request spent 30 minutes 11 seconds in the queue, including 28 minutes 29 seconds running CI.

Waiting for

check-success=local-testnet-success

All conditions

check-success=local-testnet-success
check-success=test-suite-success
github-review-approved [🛡 GitHub branch protection]
- Implement PTC reprocess queue #9458

Reason

The merge conditions cannot be satisfied due to failing checks

local-testnet-success

Hint

You may have to fix your CI before adding the pull request to the queue again.
If you update this pull request, to fix the CI, it will automatically be requeued once the queue conditions match again.
If you think this was a flaky issue instead, you can requeue the pull request, without updating it, by posting a @mergifyio queue comment.

Requeued — the merge queue status continues in this comment ↓.

eserilev · 2026-07-09T06:42:47Z

@mergify requeue

mergify · 2026-07-09T06:43:02Z

Merge Queue Status

✅ Entered queue — 2026-07-09 06:43 UTC · Rule: default · triggered by @eserilev with the @mergifyio queue command
❌ Checks failed · on draft merge queue: checking unstable (81d5769) and #9458 together #9603
🚫 Left the queue — 2026-07-09 07:12 UTC · at bc49d70551037e1f7b28e81952812d8d1277304c

This pull request spent 29 minutes 13 seconds in the queue, including 27 minutes 46 seconds running CI.

Waiting for

check-success=local-testnet-success
check-success=test-suite-success

All conditions

check-success=local-testnet-success
check-success=test-suite-success
github-review-approved [🛡 GitHub branch protection]
- Implement PTC reprocess queue #9458

Reason

The merge conditions cannot be satisfied due to failing checks

local-testnet-success

Hint

You may have to fix your CI before adding the pull request to the queue again.
If you update this pull request, to fix the CI, it will automatically be requeued once the queue conditions match again.
If you think this was a flaky issue instead, you can requeue the pull request, without updating it, by posting a @mergifyio queue comment.

Requeued — the merge queue status continues in this comment ↓.

mergify · 2026-07-09T21:19:59Z

Merge Queue Status

✅ Entered queue — 2026-07-09 21:19 UTC · Rule: default · triggered by @pawanjay176 with the merge queue checkbox
✅ Checks passed · on draft merge queue: checking unstable (81d5769) and [#9607 + #9458 + #9608] together #9610
✅ Merged — 2026-07-09 21:48 UTC · at bc49d70551037e1f7b28e81952812d8d1277304c · squash

This pull request spent 28 minutes 47 seconds in the queue, including 27 minutes 32 seconds running CI.

Required conditions to merge

check-success=local-testnet-success
check-success=test-suite-success
github-review-approved [🛡 GitHub branch protection]

conache added 2 commits June 11, 2026 19:15

Setup payload attestation reprocess queue

aa371fb

Trigger payload attestations reprocessing for unknwon blocks

085dcb8

conache changed the title ~~Ptc payload attestations reprocess queue~~ Implement PTC reprocess queue Jun 11, 2026

conache added 3 commits June 12, 2026 11:00

Add e2e network tests

73ef69b

Merge branch 'unstable' into ptc-payload-attestations-reprocess-queue

84eea65

Remove misleading comment

b6b6893

conache marked this pull request as ready for review June 12, 2026 08:41

conache requested a review from jxs as a code owner June 12, 2026 08:41

conache mentioned this pull request Jun 12, 2026

Implement PTC reprocess queue #9421

Closed

michaelsproul added the gloas label Jun 13, 2026

conache and others added 2 commits June 15, 2026 10:34

Merge branch 'unstable' into ptc-payload-attestations-reprocess-queue

11d3e61

Merge branch 'unstable' of https://github.com/sigp/lighthouse into pt…

ac94703

…c-payload-attestations-reprocess-queue # Conflicts: # beacon_node/beacon_processor/src/scheduler/work_reprocessing_queue.rs # beacon_node/network/src/network_beacon_processor/tests.rs

eserilev requested changes Jul 7, 2026

View reviewed changes

eserilev added the waiting-on-author The reviewer has suggested changes and awaits thier implementation. label Jul 7, 2026

conache added 5 commits July 8, 2026 10:11

Remove unknown block ptc queue todo

44599ef

Flag unknown block only if reprocess is allowed

577bc98

Use ReprocessAllowance with a new BlockOnly variant instead of allow_…

918ea83

…reporcess bool flag

Cover payload attestation timeout case without block import

6e44d89

Add payload attestation requeued counter metric

bc49d70

conache requested a review from eserilev July 8, 2026 08:45

eserilev approved these changes Jul 8, 2026

View reviewed changes

eserilev removed the waiting-on-author The reviewer has suggested changes and awaits thier implementation. label Jul 8, 2026

eserilev added the ready-for-merge This PR is ready to merge. label Jul 8, 2026

mergify Bot added the queued label Jul 8, 2026

mergify Bot mentioned this pull request Jul 8, 2026

merge queue: checking unstable (81d5769) and #9458 together #9599

Closed

8 tasks

mergify Bot added dequeued and removed queued labels Jul 8, 2026

mergify Bot removed the dequeued label Jul 8, 2026

mergify Bot mentioned this pull request Jul 8, 2026

merge queue: checking unstable (81d5769) and #9458 together #9600

Closed

8 tasks

mergify Bot added dequeued and removed queued labels Jul 8, 2026

mergify Bot added queued and removed dequeued labels Jul 8, 2026

mergify Bot mentioned this pull request Jul 8, 2026

merge queue: checking unstable (81d5769) and #9458 together #9601

Closed

8 tasks

mergify Bot added dequeued and removed queued labels Jul 8, 2026

mergify Bot added queued and removed dequeued labels Jul 9, 2026

mergify Bot added a commit that referenced this pull request Jul 9, 2026

Merge of #9458

b1b50ef

mergify Bot mentioned this pull request Jul 9, 2026

merge queue: checking unstable (81d5769) and #9458 together #9603

Closed

8 tasks

mergify Bot added dequeued and removed queued labels Jul 9, 2026

mergify Bot added the queued label Jul 9, 2026

mergify Bot added a commit that referenced this pull request Jul 9, 2026

Merge of #9458

e1ec480

This was referenced Jul 9, 2026

Fix rust 1.97 lints #9607

Merged

Pin genesis generator to devnet-6 #9608

Merged

merge queue: checking unstable (81d5769) and [#9607 + #9458 + #9608] together #9610

Closed

mergify Bot removed the dequeued label Jul 9, 2026

mergify Bot merged commit 3d9b46e into sigp:unstable Jul 9, 2026
38 checks passed

mergify Bot removed the queued label Jul 9, 2026

eserilev mentioned this pull request Jul 10, 2026

gloas: Lighthouse Payload Attestation Unknown-Block Permanent Drop #9585

Closed

conache deleted the ptc-payload-attestations-reprocess-queue branch July 23, 2026 15:07

		// PTC size ~512 per slot, buffer 2-3 slots for reorgs and processing delays (512 * 3 = 1536)
		// TODO(EIP-7732): verify if this is preferable queue length or otherwise

Uh oh!

Conversation

conache commented Jun 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Issue Addressed

Proposed Changes

Uh oh!

eserilev left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

eserilev Jun 23, 2026

Choose a reason for hiding this comment

Uh oh!

eserilev Jul 7, 2026

Choose a reason for hiding this comment

Uh oh!

conache Jul 8, 2026

Choose a reason for hiding this comment

Uh oh!

eserilev Jul 7, 2026

Choose a reason for hiding this comment

Uh oh!

conache Jul 8, 2026

Choose a reason for hiding this comment

Uh oh!

conache commented Jul 8, 2026

Uh oh!

eserilev left a comment

Choose a reason for hiding this comment

Uh oh!

mergify Bot commented Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mergify Bot commented Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merge Queue Status

Reason

Hint

Uh oh!

eserilev commented Jul 8, 2026

Uh oh!

mergify Bot commented Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merge Queue Status

Reason

Hint

Uh oh!

eserilev commented Jul 9, 2026

Uh oh!

mergify Bot commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merge Queue Status

Reason

Hint

Uh oh!

mergify Bot commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merge Queue Status

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

conache commented Jun 11, 2026 •

edited

Loading

eserilev left a comment •

edited

Loading

mergify Bot commented Jul 8, 2026 •

edited

Loading

mergify Bot commented Jul 8, 2026 •

edited

Loading

mergify Bot commented Jul 8, 2026 •

edited

Loading

mergify Bot commented Jul 9, 2026 •

edited

Loading

mergify Bot commented Jul 9, 2026 •

edited

Loading