Phases 1-7: classifier, optimizer, infra/observability, stream processing, drift detection, and full MLOps retraining loop

CLAUDE.md

@@ -135,7 +135,7 @@ sentinel/
     evaluate/          ← accuracy + latency benchmarks (K8s Job)
     retrain/           ← fine-tuning scripts, called by Airflow
     drift/             ← PySpark PSI/JSD jobs (spark-submit, own pyproject.toml)
-  dags/                ← Airflow DAG definitions (orchestrates pipelines/)
+  orchestration/       ← Airflow DAG definitions (orchestrates pipelines/)
   datasets/            ← shared data loading utilities
   infra/               ← Terraform + Helm charts
 ```

infra/explanation.md

@@ -1,8 +1,12 @@
 # Infra — Overview
 
 The `infra/` directory contains everything needed to run Sentinel's supporting
-infrastructure. All services run inside a local k3d Kubernetes cluster managed
-entirely by Terraform — no `docker-compose.yml`, no manual `kubectl apply`.
+infrastructure. All services — including the classifier and stream processor
+themselves, as of Phase 5 — run inside a local k3d Kubernetes cluster managed
+entirely by Terraform. No `kubectl apply` files exist outside of what
+Terraform generates; `docker-compose.yml` at the repo root is a secondary,
+lightweight alternative for running just the classifier + Prometheus + Grafana
+without a full cluster (see its own comments for the tradeoffs).
 
 ---
 
@@ -13,11 +17,18 @@ infra/
   terraform/
     local/                    — Terraform workspace for the k3d dev cluster
       providers.tf            — kubernetes + helm provider config
-      variables.tf            — all tuneable knobs (passwords, sizes)
-      main.tf                 — every K8s resource (namespaces → all services)
+      variables.tf            — all tuneable knobs (passwords, sizes, keys)
+      main.tf                 — data/monitoring/app layer resources
+      airflow.tf              — Airflow (Phase 7 orchestration)
+      mlflow.tf               — MLflow tracking server (Phase 7 experiment tracking)
+      label-ui.tf             — manual-labelling UI (Phase 7 retraining loop)
       outputs.tf              — port-forward commands printed after apply
+  mlflow/
+    Dockerfile                — extends ghcr.io/mlflow/mlflow with psycopg2 + boto3
   prometheus/
-    prometheus.yml            — global config, scrape jobs, rule file references
+    prometheus.yml            — k8s-path scrape config (rule_files: rules/)
+    prometheus.compose.yml    — docker-compose-path scrape config (different
+                                 classifier target — see its own comment)
     rules/
       classifier.yml          — alert + recording rules for the classifier
   grafana/
@@ -30,21 +41,48 @@ infra/
 
 ## Component explanations
 
-Each sub-directory has its own detailed explanation file:
+Each sub-directory (and several outside `infra/`) has its own detailed
+explanation file:
 
-- [`terraform/local/explanation.md`](terraform/local/explanation.md) — all Terraform
-  resources: namespaces, PostgreSQL, MongoDB, MinIO, Prometheus, Grafana, Kafka,
-  Jaeger, OTel Collector. Covers every provider block, variable, resource type,
-  and the patterns used throughout (StatefulSet vs Deployment, wait_for_rollout,
-  lifecycle hooks, etc.).
+- [`terraform/local/explanation.md`](terraform/local/explanation.md) — every
+  Terraform resource: namespaces, PostgreSQL, MongoDB, MinIO, Prometheus,
+  Grafana, Kafka, Jaeger, OTel Collector, spark-operator, Airflow, MLflow,
+  and the label-ui service. Covers every provider block, variable, resource
+  type, and the patterns used throughout (StatefulSet vs Deployment,
+  `wait_for_rollout`, ConfigMap-as-file mounting, etc.) — plus a growing
+  list of live-debugged gotchas.
+
+- [`mlflow/explanation.md`](mlflow/explanation.md) — why MLflow needs a
+  custom image on top of the official one, and how to bump its version.
 
 - [`prometheus/explanation.md`](prometheus/explanation.md) — global Prometheus
-  config, the k3d host scraping trick, scrape job relabeling, recording rules,
-  alert thresholds, and how to add new services to monitoring.
+  config, scrape job relabeling, recording rules, alert thresholds, and how to
+  add new services to monitoring.
+
+- [`grafana/explanation.md`](grafana/explanation.md) — Grafana provisioning
+  system, datasource proxy model, editable vs managed resources, and how to
+  build and persist dashboards.
+
+- [`../pipelines/drift/explanation.md`](../pipelines/drift/explanation.md) —
+  PySpark-based drift detection (PSI/JSD), the SparkApplication CRD, and how
+  it's scheduled to run.
+
+- [`../pipelines/evaluation/explanation.md`](../pipelines/evaluation/explanation.md)
+  — the model quality gate: benchmark metrics, the ground-truth dataset, and
+  what "passing" actually means before a model can be promoted.
 
-- [`grafana/explanation.md`](grafana/explanation.md) — Grafana provisioning system,
-  datasource proxy model, editable vs managed resources, and how to build and
-  persist dashboards.
+- [`../pipelines/retraining/explanation.md`](../pipelines/retraining/explanation.md)
+  — fine-tuning on manually-labelled data, full MLflow logging, and how it
+  hands off to the optimizer/evaluation pipelines unchanged.
+
+- [`../services/label-ui/explanation.md`](../services/label-ui/explanation.md)
+  — the manual-labelling web UI that feeds the retraining pipeline and
+  triggers it via Airflow's REST API.
+
+- [`../orchestration/explanation.md`](../orchestration/explanation.md) — how
+  Airflow DAGs get into the cluster, `retrain_dag.py`'s three-task promotion
+  flow, `drift_dag.py`'s hourly drift-check-and-auto-retrain loop, and the
+  CLI commands used to inspect DAGs day to day.
 
 ---
 
@@ -53,25 +91,45 @@ Each sub-directory has its own detailed explanation file:
 ```
 dev-start.sh
   → k3d cluster create/start sentinel
+  → docker build + k3d image import (classifier, stream-processor, drift,
+                                      mlflow, label-ui, retraining)
   → terraform apply (deploys everything below)
 
 K8s cluster (sentinel-data namespace)
-  PostgreSQL  :5432   — classification results, model registry
-  MongoDB     :27017  — flagged content for retraining
-  MinIO       :9000   — ONNX model artifacts
+  PostgreSQL  :5432   — classification results, model registry, drift stats,
+                        Airflow's own metadata + MLflow's backend store
+                        (each a separate database on the same instance)
+  MongoDB     :27017  — flagged content for retraining, manual labels
+  MinIO       :9000   — ONNX model artifacts + MLflow's artifact store
+                        (models/, datasets/, mlflow/ buckets)
   Kafka       :9092   — traces.raw topic (3 partitions)
 
+K8s cluster (sentinel-app namespace)
+  classifier         — FastAPI + ONNX inference, /v1/moderations primary endpoint
+  stream-processor   — Kafka consumer → classify → PG + Mongo
+  label-ui           — manual labelling UI for flagged_content, triggers retrain_dag
+
 K8s cluster (sentinel-monitoring namespace)
-  Prometheus  :9090   — scrapes classifier at host.k3d.internal:8000
+  Prometheus  :9090   — scrapes classifier via in-cluster Service DNS
   Grafana     :3000   — queries Prometheus via in-cluster DNS
   Jaeger      :16686  — receives OTLP traces from OTel Collector
   OTel Collector :4317/:4318  — receives spans, fans out to Kafka + Jaeger
-
-Host machine (started by dev-start.sh, not in K8s)
-  classifier  :8000   — FastAPI + ONNX inference
-  stream-processor    — Kafka consumer → classify → PG + Mongo
+  MLflow      :5000   — experiment tracking for the retraining pipeline
+
+K8s cluster (sentinel-pipeline namespace)
+  spark-operator      — manages the drift job's driver/executor pods
+  Airflow             — scheduler + webserver (LocalExecutor), orchestrates
+                        pipelines/ jobs. drift_dag.py runs hourly: submits
+                        the drift Spark job, and if drift_flagged is true,
+                        triggers retrain_dag.py, which fine-tunes (via a
+                        KubernetesPodOperator pod), gates on quality, and
+                        promotes + rolls out a new model. Both DAGs are also
+                        reachable manually (services/label-ui's button, or
+                        `airflow dags trigger`).
 ```
 
+Everything now runs in-cluster — there is no "host machine" component left.
 Port-forwards from the cluster to localhost are opened automatically by
-`dev-start.sh`. Each can also be opened individually — see each service's
-section in `docs/local-dev.md`.
+`dev-start.sh` for local access (curl, browser UIs, psql, etc.); each can also
+be opened individually via the `terraform output` commands in `outputs.tf`.
+See each service's section in `docs/local-dev.md` for URLs and credentials.

infra/mlflow/Dockerfile

@@ -0,0 +1,8 @@
+# The official image doesn't bundle a Postgres driver or S3 client — the base
+# install only supports SQLite/MySQL backends and local-disk artifact storage.
+# psycopg2-binary (backend-store-uri postgresql://...) and boto3
+# (default-artifact-root s3://... against MinIO) are both required for the
+# `mlflow server` command in mlflow.tf to start at all.
+FROM ghcr.io/mlflow/mlflow:v3.13.0
+
+RUN pip install --no-cache-dir psycopg2-binary boto3

infra/mlflow/explanation.md

@@ -0,0 +1,73 @@
+# MLflow image — Explanation
+
+This directory holds exactly one file: a `Dockerfile`. There's no application
+code here — MLflow's tracking server is a pip-installable CLI (`mlflow
+server ...`), not something this repo builds. The Dockerfile exists to
+extend the official image with the two extra clients it needs to talk to
+this project's backing stores.
+
+---
+
+## Why a custom image at all
+
+```dockerfile
+FROM ghcr.io/mlflow/mlflow:v3.13.0
+RUN pip install --no-cache-dir psycopg2-binary boto3
+```
+
+The official `ghcr.io/mlflow/mlflow` image only supports SQLite/MySQL
+backend stores and local-disk artifact storage out of the box — it doesn't
+bundle a PostgreSQL driver or an S3 client. This project's MLflow deployment
+(`infra/terraform/local/mlflow.tf`) needs both:
+
+- `--backend-store-uri postgresql://...` (experiments/runs/params/metrics)
+  → needs `psycopg2-binary`
+- `--default-artifact-root s3://mlflow/` (against MinIO) → needs `boto3`
+
+Without them, `mlflow server` fails immediately at startup trying to
+construct the SQLAlchemy engine or the S3 client. This is a
+well-documented, common gap — most MLflow-on-Postgres-and-S3 deployment
+guides extend the base image the same way.
+
+**Version pinned to an exact tag** (`v3.13.0`, not `latest`), matching this
+repo's general "never `:latest`" rule. Confirmed to exist in the `ghcr.io/
+mlflow/mlflow` registry before pinning to it, rather than guessed.
+
+---
+
+## Built and imported like every other local image
+
+```bash
+docker build -t sentinel-mlflow:local infra/mlflow/
+k3d image import sentinel-mlflow:local -c sentinel
+```
+
+Same pattern as classifier/stream-processor/drift/label-ui/retraining —
+`dev-start.sh` does both steps automatically. `imagePullPolicy: Never` in
+`mlflow.tf`'s Deployment spec tells Kubernetes to use only the local image
+store, never attempt a registry pull.
+
+---
+
+## Tips and tricks
+
+**Checking what's actually installed in the image:**
+```bash
+docker run --rm sentinel-mlflow:local pip list | grep -iE "mlflow|psycopg2|boto3"
+```
+
+**`mlflow server --help` is the fastest way to check a flag's exact
+semantics** rather than guessing from memory — this is how the
+`--allowed-hosts` and `--workers` behavior (see
+[`../terraform/local/explanation.md`](../terraform/local/explanation.md)'s
+MLflow section for what those two flags fixed) was actually confirmed,
+not assumed:
+```bash
+docker run --rm sentinel-mlflow:local mlflow server --help
+```
+
+**Bumping the MLflow version later**: change the `FROM` tag, confirm the
+new tag exists in the registry first (`ghcr.io/mlflow/mlflow`'s package
+page, or just let `docker build` fail fast if it doesn't), rebuild, and
+re-run `dev-start.sh` — no other file needs to change unless the new
+version's CLI flags differ (check `--help` again).