Distributed Job Orchestration Service in Go

A production-grade, distributed job orchestration engine built in Go, emphasizing explicit concurrency, resiliency patterns, and cloud-native readiness.

🚀 Overview

This service accepts asynchronous jobs via REST APIs, persists their metadata, and processes them using a bounded worker pool. It implements standard distributed systems patterns to ensure reliability when interacting with downstream services.

🏗️ Architecture & Design

Core Components

• API Layer: Echo-based REST server handles job submission and status tracking.
• Worker Pool: A bounded pool of goroutines using explicit concurrency patterns (channels, WaitGroups).
• Persistence Layer: Abstracted Store interface (implemented with an in-memory MockStore for demo, designed for Postgres).
• Resiliency Layer: Integrated Circuit Breaker, Exponential Backoff Retries, and Rate Limiting.
• Observability Layer: Structured JSON logging (zerolog) and Prometheus metrics.

Architecture Design Diagram

graph TD
    API[REST API /jobs] -->|Enqueue| DB[(Postgres)]
    API -->|Signal| Dispatcher[Job Dispatcher]
    Dispatcher -->|Work| WorkerPool[Bounded Worker Pool]
    WorkerPool -->|Downstream Call| SvcA[Service A]
    WorkerPool -->|Downstream Call| SvcB[Service B]
    WorkerPool -->|Aggregation| Aggregator[Result Aggregator]
    Aggregator -->|Update Status| DB

Loading

🛠️ Prerequisites

• Go: 1.22 or higher
• Docker: For containerized deployment
• Terraform / Helm: For infrastructure and K8s deployment (optional)

🚦 Getting Started

1. Install Dependencies

go mod download

2. Run the Service

go run cmd/server/main.go

3. Run Automated Tests

go test -v -race ./...

🧪 Testing with curl

Submit a Job

curl -X POST http://localhost:8080/jobs \
     -H "Content-Type: application/json" \
     -H "X-Correlation-ID: job-123" \
     -d '{"type": "ekyc-verification", "payload": {"user_id": 456}}'

Check Job Status

# Replace {id} with the ID from the submit response
curl http://localhost:8080/jobs/{id}

Health & Metrics

# Liveness/Readiness
curl http://localhost:8080/health

# Prometheus Metrics
curl http://localhost:8080/metrics

🛡️ Resiliency & Observability Features

• Backpressure: Returns 429 Too Many Requests when the internal job queue is saturated.
• Circuit Breaker: Trips after a 60% failure rate over 3 requests, protecting downstream systems.
• Graceful Shutdown: Allots 20 seconds for in-flight jobs to complete during termination.
• Structured Logs: Every log entry contains correlation_id, job_id, and worker_id for easy debugging.
• Metrics:
  • active_workers_count: Real-time worker utilization.
  • job_processing_duration_seconds: Histogram of processing latency.
  • job_queue_depth: Monitor for consumer lag.

🚢 Deployment

• Docker: docker build -t scheduler .
• Kubernetes: Manifests located in deployments/k8s-manifests.yaml.
• Terraform: AWS and GCP modules in terraform/.

---

Developed as a production-grade template for distributed orchestration in Go.