fix: don't spend retry budget on undeliverable dispatch#95
Open
tjluyao wants to merge 1 commit into
Open
Conversation
When a worker is selected but the task can't be delivered to it (its node has no live dispatch subscriber, or publish raises), the dispatcher requeued with count_retry=True, exhausting the task's max_attempts in seconds and failing with a null-error max_attempts_exceeded that masks the real cause. Route no_dispatch_subscriber and publish_failed through a new _grace_then_fail_undeliverable() keyed on a dedicated TaskRecord.no_dispatch_since clock (so the eligibility path's no_eligible_since reset can't clobber it): retry without consuming the execution budget, then fail after no_worker_grace_sec with a descriptive error naming the orphaned worker+node. Signed-off-by: Yao Lu <ylu@yao.lu>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
When the dispatcher selects a worker but the task cannot be delivered to it — the worker's node has no live subscriber on
node:<id>:dispatch(receivers <= 0, logged as "No subscribers on tasks channel"), orpublish_taskraises — it requeued with_requeue_task(..., front=True), which defaultscount_retry=True.That spends the task's execution
max_attemptson an infrastructure condition: the task burns all 3 attempts in ~3s and fails witherror: max_attempts_exceededand a nulllast_error, which reads like an executor failure and hides the real cause (an orphaned/stale worker registration — heartbeating but not consuming its dispatch channel).Observed in production: an
echotask died in 2.9s withattempts: 3,dispatched_ts: null,assigned_worker: null,last_error: nullwhile the whole fleet was registered-but-dispatch-dead.Fix
New
_grace_then_fail_undeliverable()indispatcher/base.py, keyed on a newTaskRecord.no_dispatch_sincefield (a separate clock, so the eligibility path resettingno_eligible_sinceeach iteration can't clobber the timer). Bothno_dispatch_subscriberandpublish_failedroute through it:count_retry=False),no_worker_grace_sec, fail with a descriptive error naming the orphaned worker + node,no_dispatch_sinceis reset on a successfulmark_dispatched.This does not restore dispatch (that's an operational/field-side concern) — it makes the failure legible and stops infra faults from consuming retries.
Tests
tests/server/dispatcher/test_dispatch_undeliverable.py(+2): within-grace requeues withcount_retry=False; post-grace fails with the descriptive message + reason/worker/node payload.