SentinelAI Incident Commander

Production-grade, self-hosted AI incident investigation platform built with Spring Boot 3, Spring AI, and 100% open-source infrastructure.

When a production incident occurs, SentinelAI autonomously collects logs and metrics, searches historical incidents and runbooks, correlates evidence across agents, and generates root cause hypotheses with remediation recommendations and postmortem reports.

---

Architecture

┌─────────────┐     ┌──────────────┐     ┌─────────────────────────────────────────┐
│   Frontend  │────▶│  API Gateway │────▶│  incident-service │ agent-service       │
│  React/MUI  │     │ Spring Cloud │     │  rag-service      │ ingestion-service   │
└─────────────┘     └──────────────┘     │  notification-service                   │
       │                  │              └─────────────────────────────────────────┘
       │                  │                              │
       ▼                  ▼                              ▼
┌─────────────┐   ┌──────────────┐   ┌────────┐  ┌──────────┐  ┌─────────┐
│  Keycloak   │   │    Redis     │   │Redpanda│  │PostgreSQL│  │ Ollama  │
│    (Auth)   │   │Rate Limiting │   │ Events │  │ pgvector │  │  (LLM)  │
└─────────────┘   └──────────────┘   └────────┘  └──────────┘  └─────────┘
                                              │
                    ┌─────────────────────────┼─────────────────────────┐
                    ▼                         ▼                         ▼
              ┌──────────┐            ┌────────────┐            ┌──────────┐
              │Prometheus│            │   Grafana  │            │   Loki   │
              └──────────┘            └────────────┘            └──────────┘

Microservices

Service	Port	Responsibility
api-gateway	8080	Auth, routing, rate limiting, correlation IDs
incident-service	8081	Incident CRUD, timeline, reports, Flyway migrations
agent-service	8082	Multi-agent AI orchestration (Log, Metrics, Historical, Runbook, RCA)
rag-service	8083	Vector search, embeddings, knowledge retrieval
ingestion-service	8084	Log/metrics collection from Loki/Prometheus
notification-service	8085	Event-driven notifications via Redis
frontend	3000	React dashboard UI

AI Agents

Agent	Input	Output
Log Agent	Loki logs / sample logs	{ summary, severity, confidence }
Metrics Agent	Prometheus metrics	{ anomaly, metric, reason }
Historical Incident Agent	Vector search	Similar past incidents
Runbook Agent	Hybrid RAG search	Remediation guidance
RCA Agent	All agent outputs	{ rootCause, confidence, evidence, recommendedActions }

---

Quick Start

Prerequisites

• Docker & Docker Compose v2
• 8 GB+ RAM (16 GB recommended for Ollama)
• 20 GB disk space

One-Command Startup

./run.sh

Or manually:

docker compose up -d

Access Points

Service	URL	Credentials
Frontend	http://localhost:3000	See below
API Gateway	http://localhost:8080	JWT required
Keycloak	http://localhost:8180	admin / admin
Grafana	http://localhost:3001	admin / admin
Prometheus	http://localhost:9090	—
Swagger (Incident)	http://localhost:8081/swagger-ui.html	—

Login Credentials

User	Password	Role
admin	admin	ADMIN
sre	sre	SRE
engineer	engineer	ENGINEER
viewer	viewer	VIEWER

Stop Platform

./stop.sh

---

Incident Workflow

1. User creates incident via frontend or API
2. incident.created event published to Redpanda
3. User triggers Run AI Analysis
4. incident.analysis.started event consumed by agent-service
5. Log Agent analyzes logs from ingestion-service
6. Metrics Agent analyzes Prometheus metrics
7. Historical Incident Agent retrieves similar incidents via RAG
8. Runbook Agent retrieves remediation runbooks
9. RCA Agent aggregates all evidence and generates root cause
10. Report saved; postmortem generated; user notified

---

Configuration

All configuration via .env:

POSTGRES_DB=incidentdb
POSTGRES_USER=incident
POSTGRES_PASSWORD=incident

OLLAMA_MODEL=qwen2.5:7b          # Change to qwen3:8b when available
OLLAMA_EMBEDDING_MODEL=nomic-embed-text

KEYCLOAK_ADMIN=admin
KEYCLOAK_ADMIN_PASSWORD=admin

REDIS_HOST=redis
KAFKA_BOOTSTRAP_SERVERS=redpanda:9092

No secrets in source code.

---

API Reference

All APIs proxied through gateway at http://localhost:8080/api/.

Incidents

POST   /api/incidents              Create incident
GET    /api/incidents              List incidents
GET    /api/incidents/{id}         Get incident
GET    /api/incidents/search?q=    Search incidents
POST   /api/incidents/{id}/analyze Trigger AI analysis
GET    /api/incidents/{id}/report  Get analysis report
GET    /api/incidents/{id}/timeline Get timeline

Knowledge (RAG)

POST   /api/knowledge/search       Semantic search
POST   /api/knowledge/search/hybrid Hybrid search
GET    /api/knowledge/search?q=     Keyword search

Agents

GET    /api/agents/log/{id}          Log agent output
GET    /api/agents/metrics/{id}      Metrics agent output
GET    /api/agents/historical?q=     Historical search
GET    /api/agents/runbooks?q=       Runbook search
GET    /api/agents/stream/{id}       SSE streaming analysis

See requests.http for complete test cases.

---

Development

Build Locally

mvn clean package -DskipTests

Run Tests

mvn test

Uses Testcontainers for PostgreSQL integration tests.

Project Structure

sentinel-ai/
├── common/                 # Shared DTOs, events
├── api-gateway/            # Spring Cloud Gateway
├── incident-service/       # Incident domain + Flyway
├── agent-service/          # AI multi-agent orchestration
├── rag-service/            # pgvector RAG
├── ingestion-service/      # Loki/Prometheus ingestion
├── notification-service/   # Event notifications
├── frontend/               # React + TypeScript + MUI
├── infra/                  # Prometheus, Grafana, Loki, Keycloak
├── docker-compose.yml
├── run.sh / stop.sh
└── requests.http

---

Observability

• Micrometer metrics exposed at /actuator/prometheus on each service
• Structured JSON logging with correlationId, incidentId, requestId
• Grafana dashboards pre-provisioned (SentinelAI Platform Overview)
• Loki for centralized log aggregation
• Health probes at /actuator/health/liveness and /actuator/health/readiness

---

Security

• Keycloak JWT authentication with OAuth2 Resource Server
• RBAC roles: ADMIN, SRE, ENGINEER, VIEWER
• Rate limiting via Redis in API Gateway
• CORS configured for frontend origins
• Input validation on all REST endpoints
• Audit logging table in PostgreSQL

---

Resilience

• Resilience4j circuit breakers on Ollama, Loki, Prometheus, inter-service calls
• Retries on LLM calls
• Graceful fallbacks to sample logs/metrics when observability stack unavailable
• Graceful shutdown configured on all services

---

Seed Data

Automatically loaded on startup via Flyway:

• 5 historical incidents
• 5 runbooks
• 2 postmortems
• 5 knowledge documents
• 4 users

---

Technology Stack

Layer	Technology
Language	Java 21
Framework	Spring Boot 3.4, Spring AI 1.0
Build	Maven
Database	PostgreSQL 16 + pgvector
Messaging	Redpanda (Kafka-compatible)
Cache	Redis 7
LLM	Ollama (qwen2.5:7b)
Auth	Keycloak 26
Gateway	Spring Cloud Gateway
Frontend	React 18, TypeScript, Vite, MUI
Observability	Prometheus, Grafana, Loki
Containers	Docker Compose

---

Documentation

• Deployment Guide
• Architecture Details
• Postman Collection

---

License

MIT