🚀 Distributed Job Queue System

A scalable multi-queue job distribution system built with Node.js, Express, PostgreSQL, Redis, and Docker. Designed for parallel worker processing, real-time queue management, and fault-tolerant task execution.

📌 Features

✅ Multi-Queue Architecture (High / Medium / Low Priority)

✅ Redis-Based Queue Management

✅ Distributed Worker Processing

✅ Heartbeat-Based Worker Health Monitoring

✅ Automatic Job Reassignment on Worker Failure

✅ Retry Mechanism for Failed Jobs

✅ PostgreSQL Persistent Storage

✅ Fully Dockerized Microservice Setup

✅ Horizontal Worker Scaling

🏗️ System Architecture Client → API Server → Redis (Queue Layer) ↓ Worker Containers ↓ PostgreSQL (Persistent Store)

API server pushes jobs into Redis queues

Workers consume jobs from Redis

PostgreSQL stores job metadata & status

Health-check system monitors worker availability

🛠️ Tech Stack

Backend: Node.js, Express.js

Queue System: Redis

Database: PostgreSQL

Containerization: Docker, Docker Compose

Architecture: Distributed Multi-Queue Worker System

🐳 Running with Docker docker-compose up --build

Services:

API Server

Redis

PostgreSQL

Worker Instances

📊 Job Lifecycle Pending → Queued → Processing → Completed ↘ Failed → Retry 🔥 Future Improvements

Dead Letter Queue (DLQ)

Rate Limiting

Auto Worker Scaling

Monitoring Dashboard (Prometheus + Grafana)

Kubernetes Deployment