⎈ KAMA — Kubernetes Agentic Management Assistant

Enterprise-grade, AI-driven Kubernetes operations through natural language.
Plan → govern → execute → audit — every operation, every time.

What is KAMA?

KAMA replaces the manual kubectl / Helm / YAML workflow with a multi-agent AI platform that understands plain-English intent, retrieves relevant runbooks automatically, dry-runs every operation, enforces human approval for high-risk changes, and writes an immutable audit record — all in one conversation.

"Scale the payment-api in production to 5 replicas"
         │
         ▼
  Classify intent  →  Recall cluster memory  →  Retrieve runbooks (RAG)
         │
         ▼
  Plan multi-step operation  →  Risk-score (score: 5 → ops-lead approval)
         │
         ▼
  HITL approval card  →  KubeOps dry-runs  →  Executes  →  Audit record written

No kubectl syntax to memorise. No manual runbook lookup. No unreviewed production changes.

Key Features

Feature	Detail
Natural-language operations	Plain-English chat drives all Kubernetes operations
4-stage agentic pipeline	Classify → Plan → Execute → Reflect (4 LLM calls per request, semantic memory feedback)
9-factor risk scoring	Deterministic weighted matrix; production always requires approval
Human-in-the-Loop (HITL)	Inline approval cards with runbook context, TTL, and execution feedback
Immutable audit trail	SQLite trigger-protected; covers SOC2 / ISO27001 change management
RAG knowledge base	ChromaDB — 4 collections: Runbooks, K8s Docs, Incident History, Helm Charts
Semantic memory	KubeBrain learns cluster-specific facts across sessions
Continuous monitoring	KubeMonitor polls every 30 s; LLM-generated RCA on every anomaly
Multi-cloud FinOps	KubeCost — AWS / Azure / GCP billing, chargeback, RI/Spot, budget forecast
Capacity planning	KubeCapacity — growth forecasting, rightsizing, node utilisation analysis
Dual LLM provider	Switch between local Ollama (air-gap safe) and Anthropic Claude with one `.env` line
35 MCP tools	K8s workloads, services, config, secrets, RBAC, Helm, storage, cloud billing, capacity

Architecture

┌──────────────────────────────────────────────────────────┐
│  Streamlit UI  (port 8501)                               │
│  Chat · Dashboard · Cost · Capacity · Approvals · Audit  │
│  Knowledge · Config · Logs · Agent Monitor               │
└─────────────────────────┬────────────────────────────────┘
                          │ HTTP / REST + JWT
┌─────────────────────────▼────────────────────────────────┐
│  FastAPI  (port 8000)                                    │
│  14 router modules · JWT Auth · Rate Limiting · CORS     │
└─────────────────────────┬────────────────────────────────┘
                          │ async Python
┌─────────────────────────▼────────────────────────────────┐
│  Agent Fabric  (LangGraph StateGraph)                    │
│                                                          │
│  KubeBrain ──A2A──► KubeOps                              │
│      └──────A2A──► KubeMonitor                           │
│  KubeCost  (standalone FinOps)                           │
│  KubeCapacity  (standalone capacity planner)             │
└──────────────┬───────────────────────────────────────────┘
               │ 35 MCP tool calls
┌──────────────▼───────────────────────────────────────────┐
│  Tool Layer  ·  kubernetes-client SDK                    │
│  K8s cluster  (kubectl proxy :8001)                      │
└──────────────────────────────────────────────────────────┘

Knowledge:   ChromaDB  (4 RAG collections — always local)
Persistence: SQLite / aiosqlite  (9 tables)
LLM:         Ollama  ←→  Anthropic Claude  (switchable via .env)

Five Agents

Agent	Role	Key capability
KubeBrain	Orchestrator	4-LLM Plan-and-Execute pipeline; semantic memory; RAG; A2A dispatch
KubeOps	Safe executor	ReAct loop; mandatory dry-run; 9-factor risk score; HITL gate; audit
KubeMonitor	Observer	30 s polling; 6 anomaly types; LLM RCA; auto-remediation for safe fixes
KubeCost	FinOps analyst	AWS / Azure / GCP billing; chargeback; RI/Spot; budget forecast
KubeCapacity	Capacity planner	Usage collection; linear forecast; rightsizing; node utilisation

Prerequisites

Windows Host (required)

Python 3.11+ (3.13.3 recommended)
pip install -r requirements.txt
LLM — choose one:
- Remote (recommended): ANTHROPIC_API_KEY set as a Windows environment variable, LLM_PROVIDER=remote in .env
- Local (air-gap): Ollama running with gemma4:latest pulled
nomic-embed-text pulled in Ollama (always needed for RAG, regardless of LLM provider)

WSL2 Ubuntu (Kubernetes tier)

Docker Engine
Minikube or any kubectl-accessible cluster

Cloud Kubernetes (optional — skip WSL2)

EKS / AKS / GKE — configure via aws eks update-kubeconfig or equivalent

Quick Start

1 — Clone and install

git clone <repo-url>
cd kama
pip install -r requirements.txt

2 — Configure environment

Copy-Item .env.example .env

Edit .env — minimum required settings:

KAMA_JWT_SECRET=<64-char random string>   # required — change before first run
LLM_PROVIDER=remote                       # or "local" for Ollama
CLAUDE_MODEL=claude-sonnet-4-6
# ANTHROPIC_API_KEY — set as Windows environment variable, not in this file

Set the API key securely as a Windows user environment variable:

Win+R → sysdm.cpl → Advanced → Environment Variables
New user variable: ANTHROPIC_API_KEY = sk-ant-...
Restart your terminal

3 — Start Kubernetes (WSL2 / Minikube)

wsl -e sudo service docker start
wsl -e minikube start --driver=docker
.\start_proxy.ps1          # keep this window open — bridges WSL to Windows :8001
kubectl get nodes          # verify

4 — Start the platform

# Terminal 1 — API server
uvicorn api.main:app --host 0.0.0.0 --port 8000 --reload

# Terminal 2 — Streamlit UI
streamlit run ui/app.py --server.port 8501

5 — Open browser

http://localhost:8501

Default credentials:

Username	Password	Role
`admin`	`Admin@123`	platform-admin
`opsuser`	`Ops@123`	ops-lead
`viewer`	`Viewer@123`	operator

Change default passwords after first login via System Config (platform-admin).

One-click startup (optional)

.\start_kama.ps1                  # starts everything
.\start_kama.ps1 -UseVenv         # with virtual environment
.\start_kama.ps1 -SkipMinikube    # if Minikube is already running
.\stop_kama.ps1                   # graceful shutdown

LLM Provider

KAMA supports two backends, switchable with a single .env change — no code modifications required.

Setting	`LLM_PROVIDER=local`	`LLM_PROVIDER=remote`
Engine	Ollama (self-hosted)	Anthropic Claude API
Model	`gemma4:latest`	`claude-sonnet-4-6`
Response time	2–5 min (CPU) · 10–30 s (GPU)	5–15 seconds
Data residency	100% on-premises — air-gap safe	Sent to Anthropic API
Cost	Free (hardware only)	Pay-per-token
Internet required	No	Yes

# Switch to Claude (fast — recommended for development and production)
LLM_PROVIDER=remote
# ANTHROPIC_API_KEY must be set as OS environment variable

# Switch to Ollama (on-premises / air-gap)
LLM_PROVIDER=local
OLLAMA_MODEL=gemma4:latest

Embeddings (nomic-embed-text) always run locally via Ollama regardless of the chat provider.

Project Structure

kama/
├── agents/                    # LangGraph agents
│   ├── kubebrain/             #   Orchestrator — Plan-and-Execute, 4 LLM calls
│   ├── kubeops/               #   Executor — ReAct, dry-run, risk, HITL, audit
│   ├── kubemonitor/           #   Observer — 30s polling, anomaly detection, RCA
│   ├── kubecost/              #   FinOps — multi-cloud billing, chargeback
│   ├── kubecapacity/          #   Capacity — forecast, rightsizing, nodes
│   └── a2a/                   #   Agent-to-Agent protocol and dispatcher
│
├── api/
│   ├── main.py                # FastAPI app + lifespan startup
│   ├── routers/               # 14 route modules (chat, cluster, cost, logs, …)
│   └── middleware/            # JWT auth, rate limiting, tracing, request logging
│
├── config/
│   ├── settings.py            # Pydantic BaseSettings — all config from .env / OS env
│   ├── llm_factory.py         # build_llm() — the only place that imports ChatOllama/ChatAnthropic
│   └── logging_config.py      # Console + rotating file + SQLite log sinks
│
├── db/
│   ├── schema.py              # DDL for all 9 tables + run_migrations()
│   ├── base.py                # DatabaseManager singleton + get_db()
│   └── repositories/          # Async data access — episodes, audit, approvals, logs, …
│
├── governance/
│   ├── hitl.py                # Human-in-the-Loop gate
│   ├── risk_scorer.py         # 9-factor weighted risk matrix
│   ├── sanitizer.py           # Input sanitisation — prompt injection protection
│   └── rbac.py                # Role permission helpers
│
├── rag/
│   ├── retrieval.py           # 6-stage RAG pipeline — query expansion → semantic → rerank
│   ├── ingestion.py           # Document chunking, embedding, ChromaDB indexing
│   ├── embeddings.py          # nomic-embed-text via Ollama
│   └── setup.py               # ChromaDB client initialisation
│
├── tools/
│   ├── registry.py            # MCP tool registry — register_all_tools()
│   ├── k8s/                   # 24 K8s tools — workloads, services, config, Helm, RBAC, …
│   └── cloud/                 # 11 cloud tools — AWS/Azure/GCP billing, capacity
│
├── ui/
│   ├── app.py                 # Streamlit entry point + login page
│   ├── pages/                 # 11 pages — chat, dashboard, approvals, audit, cost, …
│   └── components/            # Shared components — auth guard, sidebar, approval card
│
├── specgen/                   # Spec validator — python -m specgen.validator
├── kama_master_spec.yaml      # Single source of truth for all routes, tools, tables
├── KAMA_Overview.md           # Full architecture + design patterns + value proposition
├── CLAUDE.md                  # Developer conventions and known gotchas
├── .env.example               # Environment variable template
├── requirements.txt           # Python dependencies
├── start_kama.ps1             # One-click startup script
└── stop_kama.ps1              # Graceful shutdown script

UI Pages

Page	URL	Who can access	Purpose
Home / Login	`/`	All	Authentication and quick navigation
Chat Console	`/chat`	operator+	Natural-language Kubernetes operations
Dashboard	`/dashboard`	operator+	Live cluster health, deployments, pods, anomalies
Approvals	`/approvals`	ops-lead+	HITL approval queue — approve / reject pending operations
Audit Log	`/audit`	operator+	Immutable operation history with before/after state diff
Knowledge	`/knowledge`	platform-admin	RAG document upload, indexing, test search
Config	`/config`	platform-admin	Runtime configuration — HITL thresholds, LLM params
Help	`/help`	All	Full user manual — setup, operations, troubleshooting
Cloud Cost	`/cost`	operator+	Multi-cloud spend, chargeback, RI/Spot, budget
Capacity	`/capacity`	operator+	Resource usage, forecast, rightsizing, node analysis
Logs	`/logs`	operator+	Centralised log viewer — backend API, agents, frontend
Agent Monitor	`/agents`	operator+	Real-time agent status, anomaly feed, recent activity

API Reference

Interactive API documentation available at http://localhost:8000/docs (Swagger UI) once the server is running.

Key endpoint groups:

Prefix	Purpose
`/api/v1/auth`	Login, logout, JWT refresh, user profile
`/api/v1/chat`	Chat message submission and session management
`/api/v1/cluster`	Namespaces, deployments, pods, warning events
`/api/v1/approvals`	HITL approval queue — list, approve, reject
`/api/v1/audit`	Audit log query with filters and CSV export
`/api/v1/monitor`	KubeMonitor status and anomaly feed
`/api/v1/cost`	KubeCost analysis endpoint
`/api/v1/capacity`	KubeCapacity analysis endpoint
`/api/v1/knowledge`	RAG document management — upload, index, search
`/api/v1/config`	Runtime platform configuration
`/api/v1/logs`	Operation log query and purge
`/api/v1/health`	System health — LLM, K8s, vector store, agent status
`/api/v1/metrics`	Prometheus-format metrics endpoint

RBAC Roles

viewer  <  operator  <  ops-lead  <  platform-admin

Role	Typical user	Permissions
`viewer`	Auditors, management	Read-only — chat history, dashboard, audit log
`operator`	L1 support, developers	Chat with KubeBrain, view all dashboards
`ops-lead`	Senior SRE, team lead	All operator + approve/reject HITL for non-production (risk 5–7)
`platform-admin`	Platform team, CTO	Everything — approve production (risk 5–10), manage config and knowledge

Risk Scoring

Every mutating operation is scored before execution. The same operation always gets the same score.

Risk Factor	Weight
Target namespace is production / prod / prd	+4
Operation is a DELETE	+3
Secret creation or modification	+3
Replica delta > 50% of current count	+2
Image tag is `:latest` or untagged	+2
Helm upgrade with custom values	+2
Resource not in known cluster snapshot	+1
Operation during business hours (Mon–Fri 09:00–18:00 UTC)	+1
Operator tenure < 30 days	+1

Score	Outcome
0 – 4	Auto-execute + audit
5 – 7	ops-lead approval required
8 – 10	platform-admin approval required
Any score in production	Always requires approval

Database Schema

SQLite database at ./data/kama.db — created automatically on first startup.

Table	Purpose
`agent_episodes`	Full episodic memory — intent, plan, tool calls, outcome
`audit_log`	Immutable trigger-protected append-only audit trail
`hitl_approvals`	HITL approval queue with TTL and execution result columns
`knowledge_documents`	RAG document registry and indexing status
`cluster_snapshots`	Cluster state cache (TTL-based, used in risk scoring)
`platform_config`	Runtime configuration key-value store
`users`	Operator accounts with bcrypt passwords and RBAC roles
`token_blocklist`	JWT revocation list
`operation_logs`	Centralised log sink for backend, agents, and frontend events

Schema migrations run automatically on every API startup (db/schema.py → run_migrations()).

RAG Knowledge Base

Upload documents to ChromaDB to ground agent responses in your organisation's knowledge:

Collection	What to upload
`RUNBOOKS`	Deployment runbooks, scaling guides, incident response playbooks
`K8S_DOCS`	Kubernetes API reference, internal architecture docs
`INCIDENT_HISTORY`	Past postmortems, RCA reports, known failure modes
`HELM_CHARTS`	Chart README files, values documentation, upgrade notes

Supported formats: .txt, .md, .pdf

Upload via the Knowledge page in the UI or directly via POST /api/v1/knowledge/upload.

Embeddings always run locally via nomic-embed-text on Ollama — document content never leaves the network.

Development

Validate spec compliance

python -m specgen.validator

The kama_master_spec.yaml is the single source of truth. All routes, tools, and tables must be declared in the spec before being implemented.

Run end-to-end agent tests

python test_agents_e2e.py

Add a new MCP tool

Define the tool in kama_master_spec.yaml
Implement in tools/k8s/ or tools/cloud/ extending BaseTool
Register in tools/registry.py → register_all_tools()
Run python -m specgen.validator to confirm compliance

Add a new API route

Add the route definition to kama_master_spec.yaml
Create or update the router in api/routers/
Register the router in api/main.py
Run python -m specgen.validator

Repository pattern (important)

All database access must use:

async with get_db().get_connection() as db:
    ...

Never use async for db in get_db() — get_db() returns a DatabaseManager object (not an async iterable) and will raise a TypeError caught as a 500.

Configuration Reference

Variable	Default	Description
`KAMA_JWT_SECRET`	(required)	64-char random string for JWT signing
`LLM_PROVIDER`	`local`	`local` (Ollama) or `remote` (Claude API)
`ANTHROPIC_API_KEY`	(OS env var)	Claude API key — set as system env, not in `.env`
`CLAUDE_MODEL`	`claude-sonnet-4-6`	Claude model ID
`OLLAMA_HOST`	`http://localhost:11434`	Ollama base URL
`OLLAMA_MODEL`	`gemma4:latest`	Local chat model
`OLLAMA_EMBEDDING_MODEL`	`nomic-embed-text:latest`	Embedding model for RAG
`KAMA_DB_PATH`	`./data/kama.db`	SQLite database path
`CHROMA_PERSIST_DIR`	`./data/chromadb`	ChromaDB persistence directory
`KUBECONFIG`	`~/.kube/config`	Path to kubeconfig file
`KAMA_ENV`	`development`	`development` or `production` (affects log format)
`KAMA_LOG_LEVEL`	`INFO`	`DEBUG` / `INFO` / `WARNING` / `ERROR`

Full configuration reference: config/settings.py

Troubleshooting

Symptom	Quick fix
`kubectl get nodes` — connection refused	Restart `.\start_proxy.ps1` — the kubectl bridge window was closed
Dashboard hangs for > 10 seconds	K8s health check timeout is 5 s — restart API if hanging; check proxy bridge
Logs page returns 500	API restart required after code update to `db/repositories/logs.py`
Sidebar nav link does nothing	Restart Streamlit — page not in startup registry; HTML anchor fallback should navigate via URL
LLM shows Offline (remote)	Check `ANTHROPIC_API_KEY` is set as OS env var; restart terminal after setting it
LLM shows Offline (local)	Run `Invoke-RestMethod http://localhost:11434/api/tags`; start Ollama if not running
500 on any API endpoint	Check API logs; confirm `async with get_db().get_connection() as db:` pattern in repositories
Approval expired	Re-issue the chat request; extend `governance.approval_expiry_minutes` in System Config

Full troubleshooting guide: Help page in the UI (/help) or KAMA_Overview.md.

Documentation

Document	Contents
`KAMA_Overview.md`	Full architecture, design patterns, traditional vs agentic comparison, value proposition, Mermaid diagram
`CLAUDE.md`	Developer conventions, file locations, key patterns, known gotchas
`kama_master_spec.yaml`	Spec-driven development source of truth — all routes, tools, tables
`http://localhost:8000/docs`	Live Swagger UI (when API server is running)
UI Help page `/help`	End-user manual — setup, daily ops, troubleshooting, examples

Technology Stack

Component	Technology
Language	Python 3.13.3
Agent framework	LangGraph 0.4.7
API layer	FastAPI 0.115.6 + Uvicorn
UI	Streamlit 1.41
LLM — local	Ollama · gemma4:latest · langchain-ollama 0.3.2
LLM — remote	Anthropic Claude · claude-sonnet-4-6 · langchain-anthropic 0.3.15
Embeddings	Ollama · nomic-embed-text (always local)
Vector store	ChromaDB 0.5.23
Relational store	SQLite via aiosqlite — 9 tables
Data validation	Pydantic v2
Kubernetes SDK	kubernetes-client 31.0.0
Auth	JWT HS256 · 8-hour access token · 7-day refresh token
HTTP client	httpx 0.28.1 (async)

Built with LangGraph · FastAPI · Streamlit · ChromaDB · Anthropic Claude

Name		Name	Last commit message	Last commit date
.claude		.claude
.streamlit		.streamlit
agents		agents
api		api
config		config
db		db
governance		governance
rag		rag
specgen		specgen
tools		tools
ui		ui
.env.example		.env.example
.gitignore		.gitignore
CLAUDE.md		CLAUDE.md
KAMA_Architecture_Spec.docx		KAMA_Architecture_Spec.docx
KAMA_OPS_GUIDE.md		KAMA_OPS_GUIDE.md
KAMA_Overview.md		KAMA_Overview.md
KAMA_USER_GUIDE.md		KAMA_USER_GUIDE.md
README.md		README.md
kama_master_spec.yaml		kama_master_spec.yaml
kama_spec_text.txt		kama_spec_text.txt
requirements.txt		requirements.txt
restartapi.bat		restartapi.bat
run_ui.py		run_ui.py
start_kama.ps1		start_kama.ps1
start_proxy.ps1		start_proxy.ps1
stop_kama.ps1		stop_kama.ps1
test_agents_e2e.py		test_agents_e2e.py

Latest commit

History

Folders and files

Repository files navigation