[Deferred] Dead letter queue for deferred tasks

[alex-rogachev]

Work In Progress

[alex-rogachev]

Deferred Backends Approach

Class dependencies

classDiagram
    class Queue {
        -backend
        +initialize(backend)
        +enqueue(...)
        +schedule(...)
    }

    class Nil {
        +TasksBackend tasks
        +DeadTasksBackend dead_tasks
        +initialize(**)
    }

    class Disk {
        +TasksBackend tasks
        +DeadTasksBackend dead_tasks
        +initialize(tasks_options:, dead_tasks_options:)
    }

    class NilTasksBackend["Nil::TasksBackend"] {
        +add(...)
        +remove(...)
        +pending_tasks()
    }

    class NilDeadTasksBackend["Nil::DeadTasksBackend"] {
        +add(...)
        +remove(...)
        +retry(...)
    }

    class DiskTasksBackend["Disk::TasksBackend"] {
        - path
        - prefix
        - fsync_frequency
        +add(context, publish_at:, task_id:)
        +remove(task_id)
        +pending_tasks()
    }

    class DiskDeadTasksBackend["Disk::DeadTasksBackend"] {
        - path
        - prefix
        +add(context, exception:, task_id:)
        +remove(task_id)
        +retry(...)
    }

    Queue --> Nil : backend
    Queue --> Disk : backend

    Nil *-- NilTasksBackend : tasks
    Nil *-- NilDeadTasksBackend : dead_tasks

    Disk *-- DiskTasksBackend : tasks
    Disk *-- DiskDeadTasksBackend : dead_tasks

Loading

Runtime call shape:

backend.tasks.add(...)
backend.tasks.remove(...)
backend.tasks.pending_tasks
backend.dead_tasks.add(...)
backend.dead_tasks.remove(...)

File interfaces

lib/rage/deferred/backends/disk.rb

class Rage::Deferred::Backends::Disk
  attr_reader :tasks, :dead_tasks

  def initialize(tasks_options: {}, dead_tasks_options: {})
    @tasks = TasksBackend.new(**tasks_options)
    @dead_tasks = DeadTasksBackend.new(**dead_tasks_options)
  end

  class TasksBackend
    def initialize(path:, prefix:, fsync_frequency:)
      # WAL setup (current Disk body)
    end

    def add(context, publish_at: nil, task_id: nil) end
    def remove(task_id) end
    def pending_tasks end
  end

  class DeadTasksBackend
    def initialize(path:, prefix:)
      ...
    end

    def add(context, exception:, task_id:) end
    def remove(task_id) end
    def retry(...) end
  end
end

lib/rage/deferred/backends/nil.rb

class Rage::Deferred::Backends::Nil
  attr_reader :tasks, :dead_tasks

  def initialize(**)
    @tasks = TasksBackend.new
    @dead_tasks = DeadTasksBackend.new
  end

  class TasksBackend
    def add(_, **) end
    def remove(_) end
    def pending_tasks = []
  end

  class DeadTasksBackend
    def add(_, **) end
    def remove(_) end
    def retry(_, **) end
  end
end

Configuration

lib/rage/configuration.rb (Deferred) exposes three knobs:

1. backend= — selects the facade type only (:disk or nil). No longer carries path/prefix/fsync (those move to the collection setters). Setting backend= still sets @configured = true.

2. tasks= — options for the pending-task store (:path, :prefix, :fsync_frequency). Parsed like today’s parse_disk_backend_options, with defaults storage/, deferred-, 0.5s.

3. dead_tasks= — options for the dead-tasks store. Defaults: same path as tasks defaults, prefix dead_tasks-. Explicit dead_tasks= overrides win; unset keys fall back to tasks defaults where that is sensible (path), never reuse the tasks prefix.

Rage.configure do
  config.deferred.backend = :disk
  config.deferred.tasks = { path: "storage", prefix: "deferred-", fsync_frequency: 0.5 }
  config.deferred.dead_tasks = { path: "storage", prefix: "dead_tasks-" }
end

backend getter builds one instance:

@backend_class.new(tasks_options: @tasks_options, dead_tasks_options: @dead_tasks_options)

For backend = nil, both option hashes are ignored (Nil facade’s nested collections are no-ops).

[alex-rogachev]

@rsamoilov @serhii-sadovskyi please take a look at the proposed approach above for the deferred backends implementation. I'd like to hear your thoughts!

[rsamoilov]

Hey @alex-rogachev ,

This looks great. Love the diagram!

Several thoughts:

backend= — selects the facade type only (:disk or nil). No longer carries path/prefix/fsync

This would be a breaking change, which we would ideally want to avoid.

dead_tasks= — options for the dead-tasks store

This exposes the fact that dead tasks are stored in another file, which is an internal implementation detail that shouldn't be exposed. Consider a hypothetical Redis backend - would it need a separate dead_tasks configuration?

The DLQ backend should be reusing the same path and prefix options that are already set in config.backend because, from the user's perspective, it's just one storage. In the future, we might add some DLQ-specific knobs, like dead_tasks_retention_period, but they would be part of config.backend because the framework should not expose that internally it's a two-file implementation.

Disk::DeadTasksBackend#retry

If this method is supposed to reenqueue the task and remove it from the DLQ, then it should be part of another class, potentially Queue. The storage layer should not know the logic behind retrying a task - it only knows how to read from and write into the storage.

[alex-rogachev]

Hey @alex-rogachev ,

This looks great. Love the diagram!

Several thoughts:

backend= — selects the facade type only (:disk or nil). No longer carries path/prefix/fsync

This would be a breaking change, which we would ideally want to avoid.

dead_tasks= — options for the dead-tasks store

This exposes the fact that dead tasks are stored in another file, which is an internal implementation detail that shouldn't be exposed. Consider a hypothetical Redis backend - would it need a separate dead_tasks configuration?

The DLQ backend should be reusing the same path and prefix options that are already set in config.backend because, from the user's perspective, it's just one storage. In the future, we might add some DLQ-specific knobs, like dead_tasks_retention_period, but they would be part of config.backend because the framework should not expose that internally it's a two-file implementation.

Disk::DeadTasksBackend#retry

If this method is supposed to reenqueue the task and remove it from the DLQ, then it should be part of another class, potentially Queue. The storage layer should not know the logic behind retrying a task - it only knows how to read from and write into the storage.

All these points make sense. We'll consider them if we select this approach!