Distributed Job Queue

A production-grade, SQS-style distributed job queue built with Java 21 and Spring Boot 3. Producers submit jobs, the queue stores them durably in PostgreSQL (with Redis as a fast-path cache), and a horizontally scalable fleet of workers processes them with visibility timeouts, exponential-backoff retries, and a dead-letter queue. The whole stack is observable through Prometheus and Grafana and runs locally with a single docker compose up.

---

Table of Contents

• Architecture
• Why it's built this way (design decisions)
• Tradeoffs
• Delivery semantics
• Quick start
• Local development
• API documentation
• Observability
• Load testing & benchmarks
• Testing
• Configuration reference
• Future improvements

---

Architecture

                         ┌───────────────────┐
   POST /jobs            │  Producer Service │  validates + forwards
   GET  /jobs/{id}  ───▶ │   (Spring Boot)   │
                         └─────────┬─────────┘
                                   │ HTTP
                                   ▼
        ┌───────────────────────────────────────────────┐
        │              Queue Service                     │
        │  POST /jobs            GET  /jobs/next          │
        │  POST /jobs/{id}/ack   POST /jobs/{id}/fail     │
        │  GET  /jobs/{id}                                │
        │                                                 │
        │   ┌──────────────┐        ┌──────────────────┐  │
        │   │ Visibility   │        │  Backoff / retry │  │
        │   │ reaper (cron)│        │  + dead-letter   │  │
        │   └──────────────┘        └──────────────────┘  │
        └──────┬───────────────────────────────┬─────────┘
               │ source of truth               │ fast cache
               ▼                               ▼
        ┌──────────────┐                ┌──────────────┐
        │  PostgreSQL  │                │    Redis     │
        │  (jobs table)│                │ (queue depth)│
        └──────────────┘                └──────────────┘
               ▲
               │ claim (SELECT … FOR UPDATE SKIP LOCKED)
               │ ack / fail
   ┌───────────┴───────────────────────────────┐
   │           Worker Service  (× N replicas)   │
   │   fetch ─▶ process (JobProcessor) ─▶ ack/fail │
   └───────────────────────────────────────────┘

        Prometheus scrapes /actuator/prometheus on every service
        Grafana visualises throughput, depth, latency, retries, DLQ

Components

Module	Responsibility
common	Shared DTOs and the JobStatus enum — the contract between services.
producer-service	Public ingress. Validates submissions and forwards them to the queue.
queue-service	Durable queue core: storage, visibility timeouts, retries, DLQ, metrics.
worker-service	Polls, processes (pluggable JobProcessors), acks/fails. Scales out.
infra	Dockerfile, Docker Compose, Prometheus config, Grafana dashboards.
benchmark	k6 load tests and throughput tooling.

---

Why it's built this way (design decisions)

PostgreSQL is the single source of truth. Every state transition (enqueue, claim, ack, fail, requeue) is one atomic SQL statement. Durability and correctness never depend on Redis or application-level locking.

SELECT … FOR UPDATE SKIP LOCKED for claiming. This is the canonical PostgreSQL pattern for a concurrent work queue: many workers can pull disjoint jobs at the same time without blocking each other and without ever handing the same job to two workers. A partial index on (visible_at) WHERE status = 'PENDING' keeps the claim query fast as the table grows.

Visibility timeouts instead of distributed locks. When a worker claims a job it becomes invisible (visible_at = now + timeout). If the worker crashes, a scheduled reaper makes the job visible again — no lock leases, no ZooKeeper, no split-brain. This is exactly how Amazon SQS behaves.

Redis is an accelerator, not a dependency. It caches queue depth for O(1) reads and is updated on a timer from the PostgreSQL truth. Every Redis call is best-effort: a Redis outage degrades performance, never correctness.

Workers are stateless and pluggable. New job types are added by implementing the JobProcessor interface and registering it as a Spring bean — the worker loop never changes (open/closed principle). Scale by running more replicas; PostgreSQL coordinates them.

Clean separation of concerns / SOLID. Repository (atomic SQL) → service (business rules: backoff, DLQ) → web (HTTP). The producer holds no durable state so ingress concerns stay independent of the queue.

---

Tradeoffs

Decision	Benefit	Cost
Postgres-backed queue (vs. Kafka/RabbitMQ/SQS)	One datastore, transactional, trivially inspectable, no extra ops	Throughput bounded by Postgres; not millions of msg/s
SKIP LOCKED polling	Simple, correct, no broker	Workers poll; idle polling has a small floor cost (mitigated by idle backoff)
At-least-once delivery	No lost jobs on worker crash	Processors must be idempotent; duplicates are possible
Visibility reaper on a timer	No leases/heartbeats to manage	Recovery latency is bounded by the reaper interval, not instant
Redis cache for depth	Cheap reads, decoupled from hot path	Depth gauge is eventually consistent (refresh interval)
HTTP between services	Language-agnostic, easy to scale/inspect	Slightly more latency than in-process calls

---

Delivery semantics

• At-least-once. A job is delivered until it is successfully acked. If a worker dies mid-processing, the visibility timeout returns the job to the queue. Design processors to be idempotent.
• Retries use exponential backoff: delay = min(base × multiplier^(n-1), maxDelay). With defaults: 5s, 10s, 20s, 40s, … capped at 5m.
• Dead-letter queue. After maxRetries failed attempts (or an explicit non-retryable failure), the job moves to DEAD_LETTER and stops being delivered. A visibility-timeout expiry counts as a consumed attempt, so even "poison" jobs that crash workers eventually dead-letter.

Job lifecycle:

PENDING ──claim──▶ PROCESSING ──ack──▶ COMPLETED
   ▲                   │
   │ retry (backoff)   │ fail / timeout (retries left)
   └───────────────────┘
                       │ fail / timeout (retries exhausted)
                       ▼
                  DEAD_LETTER

---

Quick start

Prerequisites: Docker + Docker Compose. (Java 21 / Maven only needed for local non-Docker development.)

cd infra
docker compose up --build                    # single worker
docker compose up --build --scale worker-service=3   # three workers

Services:

Service	URL
Producer API	http://localhost:8080
Queue API	http://localhost:8081
Worker actuator	(internal, scaled)
Prometheus	http://localhost:9090
Grafana	http://localhost:3000 (admin/admin)

Submit a job and watch it complete:

# Create a job
curl -s -X POST http://localhost:8080/jobs \
  -H 'Content-Type: application/json' \
  -d '{"taskType":"email","payload":{"to":"alice@example.com"}}'
# => {"jobId":"3f2c...":}

# Inspect it (status flips PENDING -> PROCESSING -> COMPLETED)
curl -s http://localhost:8080/jobs/<jobId>

---

Local development

Build and test everything (requires Java 21 + Docker for Testcontainers):

mvn clean install            # build all modules + run tests
mvn -pl queue-service test   # one module

Run a single service against local Postgres/Redis:

# start just the datastores
cd infra && docker compose up -d postgres redis

# run the queue service from your IDE or:
mvn -pl queue-service spring-boot:run
mvn -pl worker-service spring-boot:run
mvn -pl producer-service spring-boot:run

Adding a new job type

@Component
public class ResizeImageProcessor implements JobProcessor {
    public String taskType() { return "resize-image"; }
    public void process(JobMessage job) {
        // throw NonRetryableJobException for bad input (-> dead-letter)
        // throw any other exception to retry with backoff
    }
}

That's it — the registry wires it in automatically; the worker loop is untouched.

---

API documentation

Producer Service (:8080) — public

Method & path	Body	Response
POST /jobs	{"taskType":"email","payload":{...}}	201 {"jobId":"<uuid>"}
GET /jobs/{id}	—	200 JobView / 404

Queue Service (:8081) — internal

Method & path	Body	Response
POST /jobs	CreateJobRequest	201 {"jobId"}
GET /jobs/next?worker=	—	200 JobMessage / 204 (empty)
POST /jobs/{id}/ack	—	200 JobView
POST /jobs/{id}/fail	{"error":"...","retryable":true} (optional)	200 JobView
GET /jobs/{id}	—	200 JobView / 404

JobView:

{
  "id": "uuid", "taskType": "email", "payload": { },
  "status": "PENDING|PROCESSING|COMPLETED|FAILED|DEAD_LETTER",
  "retries": 0, "maxRetries": 5, "lastError": null,
  "createdAt": "…", "updatedAt": "…", "visibleAt": "…"
}

Errors use a uniform envelope: { "timestamp", "status", "error", "message", "path" }.

---

Observability

Every service exposes Prometheus metrics at /actuator/prometheus.

Metric	Source	Meaning
jobs_processed_total	queue	Jobs acknowledged complete
jobs_failed_total	queue	Failed processing attempts
jobs_retried_total	queue	Jobs requeued for retry
queue_depth	queue	Pending jobs (gauge)
dead_letter_count	queue	Jobs in the DLQ (gauge)
job_processing_time	worker	Processor execution time (timer, p50/p95/p99)
worker_latency	worker	Full claim→settle time (timer)
worker_active	worker	In-flight jobs per replica (gauge)

Grafana ships a provisioned Distributed Job Queue dashboard (throughput, queue depth, failure rate, retry rate, P95 latency, active workers, DLQ count). Prometheus discovers worker replicas over DNS, so --scale worker-service=N is picked up automatically.

---

Load testing & benchmarks

See benchmark/README.md. In short:

cd infra && docker compose up --build --scale worker-service=3 -d
cd ../benchmark
RATE=800 DURATION=60s ./run.sh          # drive the producer
./measure-throughput.sh 15              # sample worker throughput

run.sh reports requests/sec, jobs/sec, P95 latency, and failure rate, with thresholds that fail the run if P95 > 250 ms or errors exceed 1%.

Benchmark numbers are hardware-dependent. Record your own results here, e.g.:

Scenario	Workers	Accept RPS	Worker throughput	P95 (accept)	Errors
sleep job, 5 ms	3	your number	your number	your number	0%

---

Testing

mvn test

• Unit — BackoffPolicyTest (exponential schedule + cap), JobDispatcherTest (ack/retry/dead-letter routing) with Mockito.
• Repository / integration — JobRepositoryIntegrationTest against real PostgreSQL via Testcontainers, including a concurrency test proving SKIP LOCKED never delivers a job twice.
• Service — QueueServiceIntegrationTest: retry-with-backoff, dead-letter promotion, visibility-timeout recovery, idempotent ack.
• API — JobControllerApiTest (MockMvc) and ProducerControllerTest (@WebMvcTest): happy paths, validation (400), not-found (404), 204-on-empty.

Integration tests need a running Docker daemon (Testcontainers).

---

Configuration reference

All values have sane defaults and are overridable via environment variables.

Variable	Service	Default	Description
QUEUE_VISIBILITY_TIMEOUT	queue	30s	Invisibility window after claim
QUEUE_MAX_RETRIES	queue	5	Retries before dead-letter
QUEUE_RETRY_BASE_DELAY	queue	5s	First backoff delay
QUEUE_RETRY_MULTIPLIER	queue	2.0	Backoff growth factor
QUEUE_RETRY_MAX_DELAY	queue	5m	Backoff cap
QUEUE_REAPER_INTERVAL_MS	queue	5000	Visibility reaper cadence
WORKER_CONCURRENCY	worker	4	Polling threads per replica
WORKER_IDLE_BACKOFF	worker	500ms	Sleep when no work is found
QUEUE_SERVICE_URL	producer/worker	http://localhost:8081	Queue endpoint

---

Future improvements

• Priority queues / fairness via an ordering column or per-tenant weighting.
• Scheduled / delayed jobs as a first-class field (the visible_at column already makes this a small step).
• LISTEN/NOTIFY to wake idle workers instantly instead of polling.
• Batch claim (LIMIT N) to amortise round-trips at very high throughput.
• DLQ replay endpoint and an admin UI for inspection/requeue.
• Idempotency keys on submission to make producers exactly-once.
• Partitioning / sharding the jobs table for horizontal write scale.
• Authentication & multi-tenancy on the producer ingress.