A self-hosted Datadog alternative with real-time ML anomaly detection. Ingests logs, metrics, and traces from microservices, detects anomalies using Isolation Forest and LSTM Autoencoders, and visualizes everything on a live dashboard.
┌─────────────────────────────────────────────────────────────────────────┐
│ Demo Microservices │
│ ┌──────────────┐ ┌───────────────┐ ┌────────────────┐ │
│ │ auth-service │ │ order-service │ │payment-service │ │
│ │ (port 8001) │ │ (port 8002) │ │ (port 8003) │ │
│ └──────┬───────┘ └───────┬───────┘ └───────┬────────┘ │
│ │ OpenTelemetry SDK │ │ │
└──────────┼────────────────────┼────────────────────┼─────────────────────┘
│ │ │
▼ ▼ ▼
┌─────────────────────────────────────────────────────────────────────────┐
│ Kafka Cluster │
│ ┌────────────┐ ┌───────────────┐ ┌──────────────┐ │
│ │ logs topic │ │ metrics topic │ │ traces topic │ │
│ └─────┬──────┘ └───────┬───────┘ └──────┬───────┘ │
└─────────┼──────────────────┼───────────────────┼────────────────────────┘
│ │ │
▼ ▼ ▼
┌─────────────────────────────────────────────────────────────────────────┐
│ Spark Streaming Jobs │
│ • Windowed p50/p95/p99 latency (10s windows) │
│ • Error rate per service │
│ • Throughput aggregation │
└──────────────────────────┬──────────────────────────────────────────────┘
│
┌────────────┴─────────────┐
▼ ▼
┌─────────────────────┐ ┌───────────────────────┐
│ TimescaleDB │ │ Elasticsearch │
│ (time-series │ │ (log storage + │
│ metrics store) │ │ full-text search) │
└──────────┬──────────┘ └───────────┬────────────┘
│ │
└────────────┬───────────────┘
▼
┌─────────────────────────────────────────────────────────────────────────┐
│ FastAPI Backend (port 8000) │
│ REST: /metrics /logs /traces /alerts /anomalies /services │
│ WebSocket: /ws/dashboard (live streaming) │
│ │
│ ┌──────────────────────────────────────────────────────────────────┐ │
│ │ ML Pipeline │ │
│ │ • Isolation Forest (point anomalies) │ │
│ │ • LSTM Autoencoder (temporal sequence anomalies) │ │
│ │ • Retrains every N windows via APScheduler │ │
│ │ • MLflow experiment tracking │ │
│ └──────────────────────────────────────────────────────────────────┘ │
│ │
│ ┌──────────────────────────────────────────────────────────────────┐ │
│ │ Alert Engine │ │
│ │ • Configurable threshold rules │ │
│ │ • State tracked in Redis │ │
│ │ • Webhook delivery with retry logic │ │
│ └──────────────────────────────────────────────────────────────────┘ │
└───────────────────────────┬─────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────────┐
│ React Frontend (port 3000) │
│ • Live time-series charts (Recharts) │
│ • Log search with filters │
│ • Anomaly overlays on metric charts │
│ • Active alert feed │
│ • Service dependency graph (React Flow) │
└─────────────────────────────────────────────────────────────────────────┘
| Layer | Technology |
|---|---|
| Message Bus | Apache Kafka |
| Stream Processing | Apache Spark Structured Streaming |
| Time-Series DB | TimescaleDB (PostgreSQL extension) |
| Log Storage | Elasticsearch |
| Alert State | Redis |
| ML Models | scikit-learn (Isolation Forest), PyTorch (LSTM Autoencoder) |
| Experiment Tracking | MLflow |
| Backend API | FastAPI (Python 3.11) |
| Frontend | React 18 + Recharts + React Flow |
| Instrumentation | OpenTelemetry SDK |
| Infra | Docker Compose |
# 1. Clone and configure
cp .env.example .env
# 2. Start everything
docker-compose up -d
# 3. Wait ~60s for services to be healthy, then open the dashboard
open http://localhost:3000
# 4. Run the load test to trigger anomalies
cd load-testing && python load_test.py --spike payment-serviceargus/
├── ingestion/ # Kafka setup and topic configuration
├── processing/ # Spark Streaming jobs
│ └── spark/ # Windowed aggregation jobs
├── ml/ # ML pipeline
│ ├── models/ # Isolation Forest + LSTM Autoencoder
│ ├── training/ # Training scripts
│ └── evaluation/ # Metrics and evaluation
├── api/ # FastAPI backend
│ ├── routers/ # Route handlers
│ ├── models/ # Pydantic schemas
│ ├── services/ # Business logic
│ └── ml_serving/ # Model inference layer
├── frontend/ # React dashboard
│ └── src/
│ ├── components/ # UI components
│ ├── pages/ # Page-level views
│ ├── hooks/ # Custom React hooks
│ └── api/ # API client
├── demo-services/ # Instrumented microservices
│ ├── auth-service/
│ ├── order-service/
│ └── payment-service/
├── infra/ # Docker configs, init scripts
├── load-testing/ # Traffic spike scripts
├── docker-compose.yml
├── .env.example
└── README.md
- Overview — all services at a glance, active alerts, anomaly count
- Service Detail — p50/p95/p99 latency charts with anomaly overlays, error rate, throughput
- Logs — full-text search with service/level/time filters
- Traces — distributed trace waterfall view
- Alerts — active and resolved alerts, rule configuration
- ML Models — anomaly score time series, model accuracy, MLflow run history
See Instructions.md for a step-by-step demo walkthrough including how to trigger and observe anomalies.