Distributed Code Execution Engine

A horizontally scalable code execution engine built in Go, capable of running untrusted user code in isolated Docker sandboxes with strict resource enforcement. Modeled after the Judge0 architecture, rebuilt from scratch in Go.

Architecture

Users
  |
  v
API Gateway (Go)
├── JWT Authentication
├── Token-bucket Rate Limiting (per IP)
├── Circuit Breaker
└── Reverse Proxy / Load Balancer
        |
        v
  PostgreSQL (submission store)
  Redis (job queue — LPUSH/BRPOP)
        |
        v
  CEE Worker Pool (horizontally scaled)
        |
        v
  Docker Sandbox
  ├── --network none
  ├── --memory 128m
  ├── --cpus 0.5
  ├── --pids-limit 64
  └── coreutils timeout (wall-clock enforcement)
        |
        v
  Verdict → PostgreSQL
  (ok | tle | mle | runtime_error | internal_error)

Endpoints

Method	Path	Description
POST	/auth	Issue JWT token
POST	/submissions	Submit code for execution
GET	/submissions/:id	Poll submission verdict
GET	/metrics	Prometheus metrics

Submit

curl -X POST http://localhost:8080/submissions \
  -H "Authorization: Bearer <token>" \
  -H "Content-Type: application/json" \
  -d '{"language": "python", "code": "print(1+1)"}'

{
  "id": "ac260428-fa52-4ead-af36-858d064d28eb",
  "language": "python",
  "status": "queued",
  "created_at": "2026-06-15T13:12:40Z"
}

Poll

curl http://localhost:8080/submissions/ac260428-fa52-4ead-af36-858d064d28eb \
  -H "Authorization: Bearer <token>"

{
  "id": "ac260428-fa52-4ead-af36-858d064d28eb",
  "language": "python",
  "status": "ok",
  "output": "2",
  "finished_at": "2026-06-15T13:12:41Z"
}

Execution Pipeline

1. POST /submissions writes a record to PostgreSQL with status: queued
2. Submission ID + code is pushed onto a Redis list (LPUSH submissions:queue)
3. Worker blocks on BRPOP submissions:queue — zero CPU burn while idle
4. On dequeue: marks submission running, spins up a Docker container, runs code
5. Exit code mapped to verdict: 0 → ok, 124 → tle, 137 → mle, else runtime_error
6. Verdict + output written back to PostgreSQL
7. Client polls GET /submissions/:id until status != queued | running

Sandbox Security

• --network none — no outbound network access from user code
• --memory 128m — OOM kill on excess memory (exit 137 → mle)
• --cpus 0.5 — cgroup CPU throttle
• --pids-limit 64 — fork bomb mitigation
• Read-only volume mount (-v tmpDir:/sandbox:ro) — no host filesystem writes
• Named containers + explicit docker kill on timeout — no leaked containers on Go-side hang
• Dual timeout: inner coreutils timeout 5s (precise wall-clock) + outer context.WithTimeout (container hang recovery)

Project Structure

.
├── cmd/
│   ├── api/main.go           # API Gateway entrypoint
│   └── worker/main.go        # CEE Worker entrypoint
├── internal/
│   ├── api/
│   │   ├── handler/          # submit, status, auth
│   │   ├── middleware/        # JWT, rate limiter, logger
│   │   └── router.go
│   ├── worker/
│   │   └── worker.go         # dequeue → execute → write verdict
│   ├── executor/
│   │   ├── executor.go       # Executor interface
│   │   ├── python.go
│   │   └── registry.go       # language → executor dispatch
│   ├── sandbox/
│   │   └── docker.go         # docker run wrapper + exit code mapping
│   ├── queue/
│   │   └── redis.go          # LPUSH / BRPOP
│   ├── store/
│   │   ├── postgres.go       # submission CRUD
│   │   └── schema.sql
│   └── model/
│       └── submission.go
├── deploy/
│   ├── docker-compose.yml
│   └── k8s/
│       ├── api-deployment.yaml
│       ├── worker-deployment.yaml
│       └── worker-hpa.yaml   # KEDA redis-list scaler
└── pkg/
    └── config/config.go

Running Locally

Prerequisites: Go 1.22+, Docker, PostgreSQL, Redis

# Start dependencies
docker run -d --name pg -e POSTGRES_PASSWORD=postgres -p 5432:5432 postgres:16-alpine
docker run -d --name redis -p 6379:6379 redis:7-alpine

# Pull sandbox image
docker pull python:3.11-slim

# Apply schema
psql $DATABASE_URL -f internal/store/schema.sql

# Set env
export DATABASE_URL="postgres://postgres:postgres@localhost:5432/execution_service?sslmode=disable"
export REDIS_URL="redis://localhost:6379"

# Run API and worker in separate terminals
go run ./cmd/api
go run ./cmd/worker

Observability

Prometheus metrics exposed at /metrics:

Metric	Description
submissions_total	Total submissions by verdict
queue_depth	Current Redis queue length
execution_duration_seconds	Sandbox execution latency histogram
worker_utilization	Active workers / total workers

Grafana dashboard included in deploy/grafana/.

Kubernetes Deployment

Workers scale horizontally via KEDA on Redis queue depth:

# worker-hpa.yaml (excerpt)
triggers:
  - type: redis
    metadata:
      listName: submissions:queue
      listLength: "5"       # scale up when queue depth > 5 per replica

kubectl apply -f deploy/k8s/

Supported Languages

Language	Image
Python 3.11	python:3.11-slim
JavaScript (Node 20)	node:20-slim (coming soon)
Go 1.22	golang:1.22-alpine (coming soon)

Adding a language: implement executor.Executor, register in registry.go. One file, one line.

Tech Stack

Go · PostgreSQL · Redis · Docker · Kubernetes · Prometheus · KEDA