Skip to content

fix(channels): recover exhausted final-delivery jobs - #309

Draft
AsperforMias wants to merge 3 commits into
langgenius:mainfrom
AsperforMias:fix/channel-final-delivery-recovery
Draft

fix(channels): recover exhausted final-delivery jobs#309
AsperforMias wants to merge 3 commits into
langgenius:mainfrom
AsperforMias:fix/channel-final-delivery-recovery

Conversation

@AsperforMias

Copy link
Copy Markdown
Collaborator

Closes #304.

Summary

  • Mark final-delivery jobs that reach the retry limit as recoverable rather than allowing them to become terminally lost.
  • Redrive those jobs from the Worker scheduled handler.
  • Preserve the recovery marker when a recovery enqueue fails, and clear it only after Queue acceptance.

Why

A final-delivery job that exhausted Queue retries was ACKed and sent to DLQ with no recovery path, allowing a terminal user response to be permanently lost.

Verification

  • Commands: just test-file apps/api/tests/channel-final-delivery-scheduling.test.ts (15 pass); just tc-package @mosoo/api.
  • Manual steps: drove a real final-delivery job through its maximum provider failure count, injected a failed recovery enqueue followed by a successful one, then exercised the actual Worker scheduled handler and queue consumer. The job remained recoverable after the first enqueue failure and was eventually delivered and ACKed.
  • Not run: just check was not run; no real Cloudflare production account or remote Queue/D1 was used.

Impact

  • User/API/contract changes: None.
  • Generated files / GraphQL / DB / lockfile: None.
  • Env or config changes: None.
  • Risk and rollback: Recovery can produce a later delivery after an ambiguous Queue response; the existing job ledger remains the idempotency boundary. Revert this PR to stop new recovery scheduling.

Review

@Yevanchen
Yevanchen force-pushed the fix/channel-final-delivery-recovery branch 7 times, most recently from 44b53c8 to 033cc6b Compare July 21, 2026 16:05
@Yevanchen
Yevanchen force-pushed the fix/channel-final-delivery-recovery branch 2 times, most recently from 3ff34c0 to 11f0d3e Compare July 25, 2026 16:12
@Yevanchen
Yevanchen force-pushed the fix/channel-final-delivery-recovery branch 6 times, most recently from 51ca5b9 to 5f57862 Compare July 31, 2026 16:08

@Yevanchen Yevanchen left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the retry-exhaustion recovery path and did not find a code-level blocker in this pass. It cannot merge while Channels remains deliberately unavailable, and it is stacked on #308.

Keep it draft; if the product surface is revived, land/review #308 first, then rebase this branch so only the exhausted-delivery recovery remains and rerun current repository plus live-boundary verification.

@Yevanchen
Yevanchen force-pushed the fix/channel-final-delivery-recovery branch 3 times, most recently from a25462b to 78e1c2a Compare August 20, 2026 16:13
@Yevanchen
Yevanchen force-pushed the fix/channel-final-delivery-recovery branch from 78e1c2a to f6417f8 Compare August 21, 2026 16:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(api): recover terminal channel final-delivery failures

2 participants