LLM serving · FastAPI · vLLM · Redis Streams · Docker · Prometheus · Grafana
I built a production-style API that serves a quantized 7B language model with caching, fair rate limits, GPU queueing, and full observability. The goal was not a demo notebook—it was a system that stays stable when many clients hit it at once.
Most LLM tutorials stop at model.generate(). In production you need fair access, predictable latency, cost control, and visibility when things break. This project applies backend engineering—queues, backpressure, metrics, caching—to GPU inference.
| What I optimized for | How |
|---|---|
| Cost & speed on repeats | Redis cache (~5 ms vs ~8 s on GPU) |
| Fair usage | Per–API-key sliding-window rate limits |
| Overload behavior | Redis Streams queue + HTTP 503 when full |
| GPU efficiency | Dedicated worker with micro-batching (vLLM) |
| Operations | Prometheus metrics + Grafana dashboard |
| Metric | Value |
|---|---|
| Cache-hit latency | ~5 ms |
| Inference p50 / p95 / p99 | 8.3 s / 8.4 s / 8.4 s |
| Cache hit rate (test workload) | ~60% |
| GPU memory (7B AWQ) | ~14 GB / 16 GB VRAM |
| Model | Mistral-7B-Instruct (AWQ 4-bit) |
Dedicated datacenter GPUs (A10/A100) would lower inference latency; T4 numbers reflect a cost-conscious deployment target.
Charts from Locust load tests on NVIDIA T4.
Latency: cache vs GPU · inference percentiles
Request flow (where time goes)
GPU memory with AWQ quantization
- REST API (FastAPI):
/infer,/health,/metrics - Two deployment modes: Redis Streams + GPU worker (default), or single-process inline mode for local/Colab
- Micro-batching on the worker: groups compatible prompts to improve throughput
- Backpressure: rejects new work with HTTP 503 when the queue is saturated
- Caching & rate limiting in Redis
- Observability: Prometheus + pre-provisioned Grafana dashboard
- Load testing: Locust scenarios included
flowchart LR
Client --> API[FastAPI API]
API --> Cache{Redis cache}
Cache -->|hit| Response
Cache -->|miss| Queue[Redis Stream]
Queue --> Worker[GPU worker + vLLM]
Worker --> Response
API --> Metrics[Prometheus / Grafana]
FastAPI · vLLM · Mistral-7B AWQ · Redis (Streams, cache, rate limits) · Docker Compose · Prometheus · Grafana · Locust
Prerequisites: NVIDIA GPU (16 GB+ VRAM recommended), Docker & Docker Compose, NVIDIA Container Toolkit
git clone <your-repo-url>
cd LIFT
cp .env.example .env # edit if needed; .env is not committed
docker compose up --build| Service | URL |
|---|---|
| API | http://localhost:8000 |
| Prometheus | http://localhost:9090 |
| Grafana | http://localhost:3000 (admin / admin) |
Example request
curl -X POST "http://localhost:8000/infer" \
-H "Content-Type: application/json" \
-H "X-API-Key: demo-key" \
-d '{"prompt": "Explain microservices in one paragraph.", "max_tokens": 256}'Configuration is documented in .env.example. Copy it to .env for local overrides—secrets and machine-specific values stay out of git.
├── app/ # API, worker, cache, queue, metrics
├── docs/assets/ # Benchmark charts (published)
├── grafana/ # Dashboard provisioning
├── docker-compose.yml
├── locustfile.py # Load tests
├── prometheus.yml
├── .env.example # Safe template (committed)
└── README.md
- Designing async APIs that delegate blocking GPU work to workers
- Distributed queueing with Redis Streams and consumer groups
- SRE-minded behavior: metrics, health checks, graceful degradation
- Performance engineering: caching, batching, quantization for hardware limits
- Containerized multi-service deployment
Architecture and benchmarks build on Jeet-51/llm-inference-service (MIT). This fork adds Redis Streams, a GPU worker, micro-batching, backpressure, and expanded operations docs.
MIT — see LICENSE.