E14: Investigation Brain (deterministic, rule-based, abstaining root-cause)

[gnanirahulnutakki]

Goal. A deterministic, rule-based, transparent root-cause reasoner over the E2 four-lens operational graph that turns a symptom into a ranked, evidence-cited set of root-cause hypotheses and a suggested remediation plan — the capability that makes Sith more than a fleet viewer. It proposes, never executes: advisory in the local wedge (a suggested command / PR diff the user runs), a governed typed-intent proposal through the PEP (E4/E5) in the hub. One brain, two modes.

Spec. docs/specs/E2-readfed-brain-integrations.md §3 (PR #43). Depends on E2 (four-lens graph, #20 · F2.6/F2.7), E12 (connector coverage, #30), E4/E5 (governed-plan hand-off in hub mode, #22/#23). Substrate keys: OpenTelemetry Kubernetes semantic conventions (§2.3).

Phase / milestone. Center of gravity P1 (the deterministic brain needs E2's four-lens graph). A local advisory subset ships in Phase L (see child, over locally-reachable lenses); the governed-plan behaviour lands with the write path (P2/P3).

---

Why this is now a first-class epic (research)

docs/research/MARKET-RESEARCH-2026-07.md §2D/§6 (theme T2): the entire 2026 AI-SRE / auto-triage wave — k8sgpt, HolmesGPT, Robusta, Botkube, Komodor/Klaudia, Cleric — converges on one shape: LLM-agentic, investigate/advise, read-only or action-gated. None ships deterministic rule-based root-cause; none ships governed typed action. Two structural openings fall out, and E14 occupies both:

1. Deterministic root-cause is unoccupied in OSS. Every serious tool routes root-cause through an LLM. Sith's rule-based brain is offline, reproducible, explainable, needs no model or data egress — credible to security-conscious / air-gapped / China estates that cannot send cluster telemetry to an LLM. Position it as deterministic, not "our AI is better."
2. The action gap the AI-SRE tools admit to is Sith's moat. They all stop at "here's the fix"; Sith's answer to "and now safely do it" is typed-intent + PR-first + multi-approver + abstention + signed dispatch + per-spoke re-validation — and Sith can be the governed MCP server these very agents call (→ E7 E7: Governed MCP server #25, E4 E4: Action federation #22).

The determinism / positioning contract (hard requirements)

1. Rule-based and transparent, not a black box — every hypothesis names the exact signals that raised it and the evidence that validated it. No LLM in the critical path; an LLM is an optional client, never the engine.
2. Proposes, never executes — output is a plan, gated downstream. No autonomy (the Replit database-deletion lesson).
3. Reads the graph; adds no data source — all evidence flows through the same connector verbs and the same freshness/coverage rules.
4. Honest about coverage — a missing/stale required lens ⇒ abstain on the dependent hypothesis with the missing lens named, never a false-confident verdict. Sith would rather say "I don't know" than guess — the trust differentiator over autonomy-first tools.

The pipeline

symptom → scope → evidence → hypotheses → validation → plan (spec §3.2). The brain consumes the four read verbs (discover/read/query/diff) and emits a plan; it never calls execute (owned by the PEP, E4/E5).

Features

• F14.1 — Rule engine + rule schema (Rule/Signal/PlanTemplate, spec §3.3): weighted signals per lens, NeedsLenses/NeedsVerbs coverage gates, CauseOf chaining.
• F14.2 — The six canonical rules R1–R6 (spec §3.4): R1 bad deploy · R2 OOMKilled · R3 CrashLoopBackOff · R4 config drift · R5 cert expiry · R6 node pressure.
• F14.3 — Composition & arbitration (spec §3.5): cause-of chaining, weight+temporal+specificity ranking, fleet-correlation dominates (same image/config across clusters outranks per-cluster), abstention-beats-a-guess.
• F14.4 — Coverage gate / abstention wired to F2.5 per-lens coverage (spec §2.6).
• F14.5 — Local advisory renderer (Phase L) — plan → suggested kubectl/PR diff the user runs. (→ child issue, ships Phase L.)
• F14.6 — Governed-plan renderer (hub) — plan → typed-intent proposal routed to the PEP (E4/E5); advisory and governed share the same rules, differ only in plan rendering + gating.
• F14.7 — Fixtures/replay harness so rule-weight changes are regression-tested against a corpus of real incidents (spec §7.2 open decision 4).
• Adjacent rules as coverage lands: ImagePullBackOff, FailingDependency, Pipeline/SyncFailure, HPA-thrash, PVC-full — the schema admits them without change.

Connector → rule coverage (spec §4.3)

With only the Wave-1 core (Kubernetes + Argo + GitHub + Prometheus + a log store), R1/R2/R4/R5/R6 reach a confident verdict and R3 reaches detect (needs a log connector for confidence). This is why W1 is the daily core — it is exactly the coverage the six rules need (→ E12 #30 wave matrix).

Exit criteria (spec §6.2)

• The pipeline runs end-to-end over the graph for at least R1–R6.
• Each of R1–R6 produces, on a seeded fixture, the correct top hypothesis with cited signals and a suggested plan naming only closed-vocabulary intents (or an advisory command in local mode).
• Cause-of chaining works (CrashLoop caused by a bad deploy reports R1 as root cause of R3, attaches R1's plan).
• Coverage gate / abstention: with a required lens made stale/absent, the dependent hypothesis is reported unconfirmed with the missing lens named.
• Fleet correlation dominates: when the same cause spans ≥ 2 clusters, the fleet-wide hypothesis outranks per-cluster.
• The brain never calls execute — verified by the seam (no execute code path in brain).
• Local vs hub: the identical rule set yields an advisory command in local mode and a typed-intent proposal routed to the PEP in hub mode.

Open decisions carried from spec §7.2

1. Brain placement — recommendation adopted: E14 owns the reasoner; E2 F2.6/F2.7 own the graph substrate (E2: Read federation #20).
2. Local telemetry reachability — opt-in endpoint + best-effort API-proxy discovery to in-cluster Prom/Loki; abstain cleanly when absent.
3. Timeline ring size — small fixed window (last N events / 24–72h), not a series; tune against real incidents.
4. Rule weights — indicative in the spec; tune via the F14.7 replay harness.
5. Node/cluster remediation (R6) — outside the v1 closed vocabulary; advisory now, a cloud-scoped typed verb is a later E4 decision.
6. EntityRef canonicalization — OTel semconv attribute names as canonical keys (T5).

---

Source: docs/specs/E2-readfed-brain-integrations.md (PR #43) · docs/research/MARKET-RESEARCH-2026-07.md (T2). Child: local advisory subset (Phase L). Related: #20 (E2 graph), #30 (E12 coverage), #22 (E4), #23 (E5), #25 (E7).

[gnanirahulnutakki]

E14 progress: the Phase-L local advisory subset F14.5 #48 is released to main via #63 and #64.

The shipped engine is deterministic and I/O-free; R1-R6 produce ranked, evidence-cited verdicts with entity-local coverage gates, stale-evidence abstention, cause composition, same-digest fleet correlation, and inert advisory output. Unit fixtures surface R1-R4; real two-cluster kind proves fleet-wide R3 and clean TELEMETRY abstention.

Full four-lens connectors and the governed hub renderer remain later E14/E2/E4 work; the local brain does not dispatch or execute.