Skip to content

Support increasing priority of an already queued task #679

Description

@wjmorland

Is your feature request related to a problem? Please describe.

When a job with a given dedupe_key is already queued, there isn't a clean way to update its priority.

A concrete example from our use case: we enqueue a job at a low priority, and later some trigger decides the same work is now urgent and tries to enqueue the same dedupe_key at a higher priority to move it toward the top of the queue. Today that raises DuplicateJobError (the partial unique index on dedupe_key rejects the second insert), and the already-queued job keeps its original low priority. As far as we can tell there isn't a supported way to promote it.

This is also pretty closely related to #678 (configurable duplicate handling on enqueue). Probably the same interface, just another option for how it handles duplicates.

Describe the solution you'd like

It would be great to have a configurable option in enqueue that updates the priority of the already-queued job instead of raising, taking the max of the existing priority and the new priority from the enqueue attempt:

  • If no queued/picked job with that dedupe_key exists → insert normally.
  • If one exists → UPDATE its priority to GREATEST(existing_priority, new_priority) and leave it in the queue, so priority is only ever raised, never lowered.

Under the hood this could map to INSERT ... ON CONFLICT (dedupe_key) WHERE (...) DO UPDATE SET priority = GREATEST(<queue_table>.priority, EXCLUDED.priority).

# promote an already-queued job; keep the higher of the two priorities
await queries.enqueue(
    "process_report",
    payload,
    priority=100,
    dedupe_key="report:42",
    on_conflict="update_priority",
)

If we implement #678 as well, the full on_conflict surface would then be raise | skip | update_priority.

Describe alternatives you've considered

  • Dequeuing/cancelling the existing job and re-enqueuing at the higher priority — this has a race condition (the job may already be picked), loses ordering/history, and needs extra round-trips.
  • Manually UPDATE-ing the queue table's priority column out of band — this works, but it sidesteps the public API and is fragile against schema changes. I'd prefer not to have to do this.
  • Not using dedupe keys and inserting a new task with higher priority — possible, but then we lose ability to avoid extra work, causes a lot of churn in idempotent tasks where it gets executed twice.
  • More of an alternative implementation, but always taking the new priority rather than the max — this could accidentally lower priority; GREATEST seems like it would be safer and solves the use case of bumping up a tasks priority.

Additional context

Same stuff from other issue probably also applies

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions