[Deferred] Dead letter queue for deferred tasks

[rsamoilov]

Description

Currently, when a deferred task exhausts all its retry attempts, it is silently discarded. There is no record of what failed, why it failed, or what arguments it was called with. This makes it difficult to diagnose recurring issues and impossible to recover failed work without re-triggering it from scratch.

A dead letter queue (DLQ) would capture tasks that have exhausted all retries, preserving the task class, arguments, exception details, and retry history. This gives operators visibility into persistent failures and the ability to investigate and replay failed tasks.

Implementation

The DLQ should store:

• Task class name — which task failed.
• Task arguments — the arguments passed to perform.
• Last exception — the exception class, message, and backtrace from the final attempt.
• Retry count — how many times the task was attempted.
• Timestamps — when the task was originally enqueued (if available) and when it was moved to the DLQ.

Storage

Since Rage::Deferred already uses a write-ahead log (WAL) with no external dependencies, the DLQ should follow the same philosophy. A disk-based approach (e.g. a separate file) would be a natural fit. Consider whether the DLQ should share the WAL's format, or if a substitute-all-on-write approach would fit better.

API surface

At a minimum, operators need a way to:

1. List dead-lettered tasks.
2. Inspect a specific task's details.
3. Retry one or more dead-lettered tasks (re-enqueue them).
4. Delete tasks from the DLQ.

Whether this is exposed as a Ruby API, a Rake task, or both is a design decision worth discussing.

Tips

• Review the Deferred docs — especially the retry mechanism and WAL sections — to understand the current behavior when tasks fail.
• Look at how the existing WAL persistence works in the codebase to inform the DLQ storage design.
• Check the architecture doc to see how Rage's core components interact and to understand the design principles.
• Read the contributing guide for coding conventions and design principles used across the codebase.
• Before starting the implementation, please share your proposed design approach. Discussing the approach early will drastically increase the chances of acceptance and help avoid rework.
• Feel free to ask any questions or request help in the comments below!

[alex-rogachev]

Spec: Dead Letter Queue for Deferred Tasks

Problem

When a deferred task exhausts its retries (or aborts via retry_interval), Rage::Deferred::Queue removes it from the WAL and discards it:

# lib/rage/deferred/queue.rb (simplified)
if retry_in
  enqueue(context, delay: retry_in, task_id:)
else
  @backend.remove(task_id) # permanent failure — currently silent
end

There is no durable record of what failed, with which arguments, or why. Operators cannot diagnose recurring failures or replay work without re-triggering it from scratch.

Goals

1. Persist tasks that have permanently failed (retries exhausted or explicitly aborted).
2. Preserve enough detail to investigate and replay: task class, arguments, last exception, attempt count, timestamps.
3. Expose list / inspect / retry / delete for operators.
4. Follow Deferred’s storage philosophy: disk-based, no external dependencies.

Recommended approach

Separate disk store + Ruby API.

Hook the permanent-failure branch in Queue#schedule, write a DLQ record, then remove the task from the WAL as today.

  perform task
       │
       ├─ success ──────────────► WAL remove
       │
       └─ failure
            │
            ├─ will retry ──────► WAL re-enqueue (unchanged)
            │
            └─ give up ─────────► DLQ add ──► WAL remove

Ordering and crash edge case. Always write the DLQ record before removing the task from the WAL. If the process crashes (or remove fails) after the DLQ write:

1. The failure is already durable in the DLQ — nothing is silently lost.
2. On restart, the WAL still lists the task as pending, so Deferred may run it again (existing at-least-once behavior).
3. Another permanent failure would call DLQ add again; upsert by task_id so the same work becomes one DLQ entry, not duplicates.
4. Until WAL remove eventually succeeds, operators may briefly see the task as both pending and dead-lettered.

The reverse order (WAL remove first, then DLQ) is unsafe: a crash leaves the task neither queued nor dead-lettered — the silent loss this feature exists to prevent. Prefer possible re-execution over losing the failure record.

Why not extend the WAL?

Alternatives considered:

Option	Pros	Cons	Decision
A. New op in the existing WAL (dlq:…)	Reuses CRC / append / flock	Mixes hot queue traffic with cold failure records; rotation must specially preserve DLQ entries; list/delete become awkward (tombstones or rewrite of a large log)	Reject
B. Separate append-only WAL for DLQ	Familiar pattern	Overkill for low volume; delete/retry still need compaction	Reject
C. Separate file, rewrite-on-write	Fits small, long-lived sets; list/delete/retry are simple; isolated from WAL rotation	Whole-file rewrite under lock	Accept

Sharing the WAL's format or file would tie records meant for long retention and explicit deletion to a log built for high-churn add/rem and periodic rotation that drops processed history.

Substitute-all-on-write (load → mutate → rewrite under exclusive lock) is appropriate because DLQ cardinality is expected to stay small (operators clear or replay failures). If volume later becomes an issue, the storage backend can change without changing the public API.

Storage

Location

Same directory as Deferred disk storage (config.deferred.backend :path, default storage/), with a distinct prefix, e.g. deferred-dlq-.

• Reuses existing path configuration; no new config required for the common case.
• Survives process restarts independently of WAL rotation.
• Easy for operators to find next to WAL files.

Format

One JSON document (or a versioned envelope) rewritten atomically when possible (write temp file + rename):

{
  "version": 1,
  "entries": [
    {
      "id": "1735689600-12345-1",
      "task_class": "SendWelcomeEmail",
      "args": "BAhbBkkiFXVzZXJAZXhhbXBsZS5jb20GOgZFVA==",
      "kwargs": "BAh7AA==",
      "attempts": 21,
      "exception": {
        "class": "RuntimeError",
        "message": "SMTP timeout",
        "backtrace": ["..."]
      },
      "enqueued_at": 1735690000,
      "dead_lettered_at": 1735690000
    }
  ]
}

Why JSON (not Marshal for the whole file)?

• DLQ is operator-facing: readable, greppable, recoverable after class renames.
• Exception payloads are stored as strings/hashes, not live exception objects, so inspect stays stable across deploys.

Arguments are the exception to that. perform takes arbitrary Ruby objects, so args and kwargs are stored as Base64-encoded Marshal.dump blobs inside the JSON envelope, matching the fidelity the WAL already gets from Marshal.dump(task).dump. Anything that can be enqueued can therefore be dead-lettered, and no task is dropped because its arguments would not serialize. Everything around them stays plain JSON, so an entry remains greppable by class, message and timestamp even when its arguments are opaque.

Multi-worker

How the WAL does it today. Each worker owns its own storage file for its entire lifetime. On boot, Backends::Disk globs the storage directory and tries a non-blocking exclusive flock on every file it finds. A file whose lock is already held belongs to a live worker and is skipped; a file whose lock is free is an orphan left by a crashed or previous worker and gets claimed. If the worker claims nothing it creates a fresh file; if it claims several, the first becomes its working storage and the rest are merged into it and deleted. Because ownership is established once at boot and held, individual add/remove calls are plain unlocked appends to a file no one else can touch.

Why the DLQ should not copy that. Per-worker files work for the WAL because nobody reads a worker's file except that worker. The DLQ is the opposite: an operator running list expects to see every failure across all workers. Per-worker DLQ files would force a find-and-merge on every read, and delete/retry would first have to locate which worker's file owns the entry.

Proposed approach. A single shared DLQ file under the configured path, with an exclusive lock taken around each load → mutate → rewrite cycle and released immediately after. DLQ writes only happen on permanent failure, so per-write locking is cheap. A writer that finds the file locked waits and then rewrites against the updated contents rather than skipping the write — a busy file must never cause a dropped dead letter.

Nil backend

When config.deferred.backend = nil, permanent failures continue to be discarded (current behavior). DLQ list/retry APIs return empty / no-op. Feature isolation: apps without persistence do not get a half-working DLQ.

Integration point

Single call site in Rage::Deferred::Queue#schedule:

if result == true
  @backend.remove(task_id)
else
  attempts = Rage::Deferred::Context.inc_attempts(context)
  retry_in = task.__next_retry_in(attempts, result)
  if retry_in
    enqueue(context, delay: retry_in, task_id:)
  else
    dead_lettered = begin
      @dead_letters.add(context, exception: result, task_id:) # new
      true
    rescue => e
      Rage.logger.error("Could not dead-letter task #{task_id}: #{e.class} (#{e.message})")
      false
    end

    @backend.remove(task_id) if dead_lettered
  end
end

A failed DLQ write must never remove the task from the WAL. If add raises — unwritable path, full disk, a corrupt DLQ file — the task stays pending and will be picked up again after the next boot, and dead-lettered again once it fails for good. That is the same trade-off as the crash case above: an extra execution is acceptable, a failure nobody recorded is not. The upsert by task_id keeps the eventual retry from producing a second entry.

Public API

Ruby API

# List summaries
Rage::Deferred.dead_letters.all
# => [#<Rage::Deferred::DeadLetter ...>, ...]

Rage::Deferred.dead_letters.find(id)
# => entry or nil

Rage::Deferred.dead_letters.retry(id)   # re-enqueue via normal queue path; remove from DLQ on success
Rage::Deferred.dead_letters.delete(id) # drop without replaying

Rage::Deferred.dead_letters.retry_all  # optional convenience
Rage::Deferred.dead_letters.clear      # optional convenience

retry should:

1. Load the entry.
2. Resolve the task class (raise a clear error if missing/renamed).
3. Enqueue through the normal Deferred path (Task.enqueue / queue) with original args/kwargs — reset attempts so the task gets a fresh retry budget.
4. Delete the DLQ entry after successful enqueue.

Internal construction / storage types marked @private per CONTRIBUTING.

Rake tasks

Thin wrappers over the Ruby API, living in lib/rage/tasks/deferred.rake

deferred:dlq:list
deferred:dlq:show[id]
deferred:dlq:retry[id]
deferred:dlq:delete[id]

Each task boots the environment, calls the matching Ruby method and prints the result. No logic of its own, so there is still one implementation to document and test. The reason to have them at all is that operators frequently have shell access without a REPL, and a single command drops into a runbook or a deploy hook far more easily than a console session does.

Configuration

No new knobs: the backend already decides. A disk backend gets a file-backed DLQ, a nil backend gets none.

Questions / Concerns

Questions

1. Default retention — Do we need a default retention period (or max age) for DLQ records, or do entries live until an operator retries or deletes them?
2. Storage via backends — Should DLQ persistence follow the same backends approach as the WAL (Disk / Nil, and possibly other types later), so different storage backends can be used for the DLQ?
3. Aborted retries — Should aborted retries go to the DLQ the same as exhausted max retries?
4. Out-of-band retry — retry invoked from rake or rage c runs in a process with no Iodine loop, so the re-enqueued task sits in a storage file until the next boot. Do we document the delay, have workers periodically adopt orphaned storage files, or expose retry from inside the running server?

Concerns

1. Unbounded growth — Operators may not clear the DLQ. With rewrite-on-write, a large active set makes every add/retry/delete rewrite more data and hold the lock longer. Mitigations to decide: soft cap + warning, refuse new DLQ writes while keeping the task in the WAL, drop-oldest retention, or revisit storage if large/uncleared DLQs are expected in practice.

[rsamoilov]

@serhii-sadovskyi @alex-rogachev

Separate file, rewrite-on-write
Same directory as Deferred disk storage, with a distinct prefix
rewritten atomically when possible (write temp file + rename)
an exclusive lock taken around each load → mutate → rewrite cycle
Rake tasks

Love these!

"version": 1,

The current WAL storage keeps version in the file name - let's follow the same pattern and drop version from the file itself.

args and kwargs are stored as Base64-encoded

Any specific reason you're using Base64 here?

Rage::Deferred.dead_letters

I'm not sure about the dead_letters name. Have you considered any alternatives? What do you think about dead_set, failed_tasks, or dead_tasks?

Rage::Deferred.dead_letters.all

Exposing all might not be the best idea if we expect the storage to grow. What about implementing Enumerable instead? This way we could expose each which would be iterating over small batches internally.

Rage::Deferred.dead_letters.retry(id)
Rage::Deferred.dead_letters.delete(id)

I like the approach where the entry is just a data wrapper, but a more convenient approach could be to allow the entries be retried/deleted in-place. Compare this:

Rage::Deferred.dead_letters.each { |entry| Rage::Deferred.dead_letters.retry(entry.id) }

And this:

Rage::Deferred.dead_letters.each(&:retry)

Such implementation can be a bit more complex, especially considering you'd need to ensure calling entry.delete inside a loop doesn't try to modify the collection in-place, so let me know what you think.

Default retention — Do we need a default retention period (or max age) for DLQ records, or do entries live until an operator retries or deletes them?

I think the default should be no retention period - entries live forever.

Storage via backends — Should DLQ persistence follow the same backends approach as the WAL (Disk / Nil, and possibly other types later), so different storage backends can be used for the DLQ?

Yes, I belive this is already covered in the design - "When config.deferred.backend = nil, permanent failures continue to be discarded".

Aborted retries — Should aborted retries go to the DLQ the same as exhausted max retries?

Currently, there's no API to abort a retry.

Out-of-band retry
the re-enqueued task sits in a storage file until the next boot

This is likely one of the biggest challenges of this issue. Currently, the trade-off is that if you're enqueueing a task from the console, it will only be picked up after a restart. With the DLQ in place, I don't think it will be acceptable anymore.

When you run MyTask.enqueue from the console, it will create a new WAL and add the task in there. There's currently no mechanism to pick up these "orphaned" WALs because if we only think about the app, there's always a process attached to a WAL - if a worker process dies, the master process sees it, spawns another worker, and it picks up the orphaned WAL.

However, when a WAL is created from another process (e.g. the console), this expectation breaks. So what we'll need is a mechanism to periodically check for such orphaned WALs, pick them up, merge into the current WAL, and enqueue respective tasks. It's going to be quite challenging to separate the poller/enqueue/storage responsibilities correctly, so make sure you understand well how it's supposed to work. And let me know if I you have any questions.

P.S. another small part here would be to ensure Deferred::Queue#schedule is a no-op unless Iodine.running?.

Unbounded growth — Operators may not clear the DLQ. With rewrite-on-write, a large active set makes every add/retry/delete rewrite more data and hold the lock longer

This is a valid point. What do you think about a hybrid approach - using append to add an entry and rewrite-on-write on delete? This would likely contradict the idea of using JSON to store dead tasks, but it would give us instant writes inside the server process. Retries/deletes would still need to load the whole data set in memory, but it's acceptable since these would mostly be running from separate processes, like IRB or Rake. Just thinking aloud here - let me know what you think. "Nah, this is dumb" also works 😄

[alex-rogachev]

The current WAL storage keeps version in the file name - let's follow the same pattern and drop version from the file itself.

gotcha

Any specific reason you're using Base64 here?

So the binary Marshal payload of args and kwargs can live inside a JSON string (Marshal.dump returns a byte string that can contain nulls and non-UTF-8 sequences).
But anyway we might drop the idea of using JSON at all.

I'm not sure about the dead_letters name. Have you considered any alternatives? What do you think about dead_set, failed_tasks, or dead_tasks?

dead_tasks sounds great, we'll use it.

Exposing all might not be the best idea if we expect the storage to grow. What about implementing Enumerable instead? This way we could expose each which would be iterating over small batches internally.

Sounds good, we'll consider this approach

Currently, there's no API to abort a retry.

I see that there's no specific method to abort a retry, but abortion is still possible by explicitly returning nil from the retry_interval method.
https://github.com/rage-rb/rage/blob/main/lib/rage/deferred/task.rb#L143

This is likely one of the biggest challenges of this issue. Currently, the trade-off is that if you're enqueueing a task from the console, it will only be picked up after a restart. With the DLQ in place, I don't think it will be acceptable anymore.

Could you please elaborate on why "With the DLQ in place, I don't think it will be acceptable anymore."? I feel like implementing such mechanism that tracks and merges orphaned WALs should be implemented as a separate task. Currently we can add some kind of message (STDOUT) for a user that re-enqueued dead tasks will be processed after a restart. Just like it works now for newly enqueued tasks from the console (or other separate process). But anyway if you think that this approach is too kludgy, let us know so we'll work on it within the current issue.

What do you think about a hybrid approach - using append to add an entry and rewrite-on-write on delete? This would likely contradict the idea of using JSON to store dead tasks, but it would give us instant writes inside the server process.

Sounds reasonable, but again let us think a bit on this!

[rsamoilov]

I see that there's no specific method to abort a retry, but abortion is still possible by explicitly returning nil from the retry_interval method.

Then it would also go to the DLQ.

I feel like implementing such mechanism that tracks and merges orphaned WALs should be implemented as a separate task. Currently we can add some kind of message (STDOUT) for a user that re-enqueued dead tasks will be processed after a restart.

Fair enough - let's ditch it for now 👍