mck21 · mck21 · Jun 18, 2026 · Jun 18, 2026
diff --git a/AGENTS.md b/AGENTS.md
@@ -21,7 +21,7 @@ Entry point for AI agents working on this repository. Read this file first, then
 | Dev cluster | Minikube (local) |
 | Staging / Production | EKS on AWS |
 
-**Current phase:** Phase 5 implemented (staging-first CD). **Next:** Validate CD on EKS staging, then Phase 6 — Security Hardening. See [STATUS.md](STATUS.md).
+**Current phase:** Phases 1–9 built as code/manifests/docs (staging-first; production off). **Next:** live EKS validation + screenshots (`terraform apply`, deploy, capture evidence). See [STATUS.md](STATUS.md).
 
 ---
 
@@ -33,7 +33,7 @@ devsecops-platform/
 ├── app/frontend/         # Empty — reserved for future UI
 ├── infrastructure/       # Terraform modules + staging/production envs
 ├── k8s/                  # Kustomize manifests (base + overlays + blue-green + argocd)
-├── .github/workflows/    # ci.yaml, cd.yaml, security.yaml, terraform.yaml
+├── .github/workflows/    # ci.yaml, cd.yaml (main)
 ├── scripts/              # blue/green CD automation (Phase 5)
 ├── docs/                 # Placeholder — full docs in Phase 9
 ├── images/phase-N/       # Screenshot evidence per phase
@@ -114,16 +114,26 @@ Summary:
 
 ---
 
-## Phase Boundaries — What NOT to Do Yet
+## Phase Boundaries — what's built vs deferred
 
-| Deferred to | Do not implement yet |
-|-------------|---------------------|
-| Phase 6 | Kyverno policies, NetworkPolicy, External Secrets, manifest `securityContext` |
-| Phase 7 | Grafana dashboards, Loki, custom Prometheus ServiceMonitors |
-| Phase 9 | Full `docs/`, ADRs, architecture diagrams |
-| End of PLAN | Production CD auto-sync — see [docs/cd-production-promotion.md](docs/cd-production-promotion.md) |
+Phases 1–9 are now **implemented as code/manifests/docs**. What remains is
+**manual/live** work, intentionally deferred:
 
-**Phase 5 (staging-first):** CD auto-deploys to **staging EKS only**. Dev is manual. Run EKS bootstrap once: [k8s/argocd/install-notes.md](k8s/argocd/install-notes.md). Set GitHub vars `EKS_CLUSTER_NAME`, `STAGING_HEALTH_URL`.
+| Status | Item |
+|--------|------|
+| Built | Phase 6 security (`k8s/security`, `k8s/kyverno`, `k8s/external-secrets`, securityContext, Cosign in `ci.yaml`) |
+| Built | Phase 7 monitoring (`monitoring/`, SLO rules, dashboards, `docs/sre.md`) |
+| Built | Phase 8 DR/resilience (`k8s/velero`, `tests/k6`, DR + resilience docs) |
+| Built | Phase 9 docs (`docs/` guides, `docs/adr/`, `docs/diagrams/*.mmd`) |
+| Deferred (manual) | Live EKS apply of the above, screenshots, diagram PNG exports, `terraform apply` |
+| Undecided | Production **runtime** (EKS apply, CD promote, ECR push) — **off**; reference [docs/cd-production-promotion.md](docs/cd-production-promotion.md) |
+
+> When changing app manifests, keep Kyverno policies and securityContext intact
+> (Phase 6 enforces them on staging/prod).
+
+**Showcase mode:** Staging is the only environment with a working pipeline. Production is **off** — it stays in Git as a **mirror** of staging overlays/Terraform, with no build, push, or deploy. Turning production runtime on is an open decision, not a committed phase. Runbook: [docs/showcase-staging-only.md](docs/showcase-staging-only.md). When staging overlay changes, align `k8s/overlays/production/` in the same PR.
+
+**Phase 5 CD:** auto-deploys to **staging EKS only**. Set GitHub vars `EKS_CLUSTER_NAME`, `STAGING_HEALTH_URL` (optional until Ingress ready).
 
 ---
 
@@ -146,6 +156,7 @@ Summary:
 | CI / CD workflows | `.github/workflows/ci.yaml`, `.github/workflows/cd.yaml` |
 | CD scripts | `scripts/gitops-bump.sh`, `scripts/blue-green-switch.sh`, `scripts/rollback.sh` |
 | SonarCloud config | `sonar-project.properties` |
+| Showcase runbook (staging-only) | [docs/showcase-staging-only.md](docs/showcase-staging-only.md) |
 | GitHub OIDC (Terraform) | `infrastructure/modules/github-oidc/` |
 
 ---

diff --git a/README.md b/README.md
@@ -33,9 +33,14 @@ The application is a **Feature Flag Service** — manage feature toggles per env
 | 3 | Feature Flag Service + K8s manifests | **Done** |
 | 4 | CI Pipeline (GitHub Actions, SonarCloud, ECR) | **Done** |
 | 5 | CD Pipeline & GitOps (ArgoCD, staging-first) | **Implemented** — validate on EKS |
-| 6–9 | Security, Monitoring, DR, Docs | Planned |
+| 6 | Security hardening (Kyverno, Cosign, NetworkPolicy, External Secrets) | **Implemented** |
+| 7 | Monitoring & SRE (Prometheus, Grafana, Loki, SLOs) | **Implemented** |
+| 8 | DR & resilience (Velero, k6) | **Implemented** |
+| 9 | Docs, ADRs, cost | **Implemented** |
 
-Full tracker: [STATUS.md](STATUS.md) · Full roadmap: [PLAN.md](PLAN.md)
+All manifests/config/docs are committed. Live validation (EKS deploy + screenshots)
+is the remaining step — staging is the only counted runtime; production is a Git
+mirror that stays **off**. Full tracker: [STATUS.md](STATUS.md) · Roadmap: [PLAN.md](PLAN.md)
 
 ---
 
@@ -105,7 +110,27 @@ See [app/backend/README.md](app/backend/README.md) for `.env` and Prisma setup.
 
 ## Architecture
 
-> _Diagram coming in Phase 9_
+```mermaid
+flowchart LR
+    dev[Developer] -->|git push| GH[GitHub]
+    GH -->|CI: lint, test, Sonar, Trivy, build, sign| ECR[(ECR)]
+    GH -->|CD: GitOps bump| GH
+    subgraph AWS[AWS - staging EKS]
+      ArgoCD -->|sync| ns[staging namespace]
+      ns --> bg[Blue/Green Deployments]
+      bg --> istio[Istio VirtualService]
+      bg --> pg[(PostgreSQL)]
+      bg --> redis[(Redis cache)]
+      ECR -.image.-> bg
+      kyverno[Kyverno admission] -. verify .-> bg
+      prom[Prometheus] --> graf[Grafana]
+      bg -.metrics.-> prom
+    end
+    ArgoCD -. watches .-> GH
+```
+
+Detailed diagrams (AWS infra, CI/CD flow, Kubernetes layout) and rationale:
+[docs/architecture.md](docs/architecture.md). Decision records: [docs/adr/](docs/adr/).
 
 ---
 
@@ -140,11 +165,20 @@ docs/              Full documentation (Phase 9)
 
 ---
 
-## Deployment Guide
-
-> _Full Minikube + EKS guide coming in Phase 9 (`docs/deployment-guide.md`)_
-
-Minikube dev deploy: [k8s/README.md](k8s/README.md)
+## Documentation
+
+| Topic | Doc |
+|-------|-----|
+| Architecture & diagrams | [docs/architecture.md](docs/architecture.md), [docs/diagrams/](docs/diagrams/) |
+| Deployment (Minikube + EKS) | [docs/deployment-guide.md](docs/deployment-guide.md), [k8s/README.md](k8s/README.md) |
+| Security hardening | [docs/security.md](docs/security.md) |
+| Monitoring & observability | [docs/monitoring.md](docs/monitoring.md), [monitoring/README.md](monitoring/README.md) |
+| SRE — SLI/SLO/error budget | [docs/sre.md](docs/sre.md) |
+| Disaster recovery | [docs/disaster-recovery.md](docs/disaster-recovery.md) |
+| Resilience testing | [docs/resilience-testing.md](docs/resilience-testing.md) |
+| SonarCloud quality gate | [docs/sonarqube.md](docs/sonarqube.md) |
+| Cost estimates | [docs/cost.md](docs/cost.md) |
+| Architecture Decision Records | [docs/adr/](docs/adr/) |
 
 ---
 

diff --git a/STATUS.md b/STATUS.md
@@ -2,7 +2,7 @@
 
 Phase tracker for the DevSecOps platform. Updated after Phase 5 implementation.
 
-**Active work:** Phase 5 validation on EKS staging (screenshots)  
+**Active work:** Phases 6–9 built as code/manifests/docs; pending live EKS validation + screenshots  
 **Agent entry point:** [AGENTS.md](AGENTS.md)  
 **Deploy runbook:** [TROUBLESHOOTING.md](TROUBLESHOOTING.md)  
 **Full roadmap:** [PLAN.md](PLAN.md)
@@ -19,34 +19,53 @@ Phase tracker for the DevSecOps platform. Updated after Phase 5 implementation.
 | 3 | Feature Flag Service + K8s manifests | **Done** | `app/backend/`, `k8s/`, `images/phase-3/` |
 | 4 | CI Pipeline (GitHub Actions, SonarCloud, ECR) | **Done** | `.github/workflows/`, `sonar-project.properties`, `images/phase-4/` |
 | 5 | CD & GitOps (ArgoCD sync, blue/green automation) | **Implemented** (staging-first) | `.github/workflows/cd.yaml`, `scripts/`, `images/phase-5/` |
-| 6 | Security hardening (Kyverno, Cosign, NetworkPolicy) | Not started | — |
-| 7 | Monitoring & SRE (Prometheus, Grafana, Loki) | Not started | `monitoring/` placeholder |
-| 8 | DR & resilience (Velero, k6) | Not started | — |
-| 9 | Documentation, ADRs, diagrams, cost docs | Not started | `docs/` placeholder |
+| 6 | Security hardening (Kyverno, Cosign, NetworkPolicy, External Secrets) | **Implemented** | `k8s/security/`, `k8s/kyverno/`, `k8s/external-secrets/`, securityContext in deployments, Cosign in `ci.yaml` |
+| 7 | Monitoring & SRE (Prometheus, Grafana, Loki) | **Implemented** | `monitoring/`, `docs/sre.md` |
+| 8 | DR & resilience (Velero, k6) | **Implemented** | `k8s/velero/`, `tests/k6/`, `scripts/resilience-test.sh`, `docs/disaster-recovery.md`, `docs/resilience-testing.md` |
+| 9 | Documentation, ADRs, diagrams, cost docs | **Implemented** | `docs/` (architecture, deployment-guide, security, monitoring, sonarqube, cost), `docs/adr/`, `docs/diagrams/` |
+
+> **Phases 6–9 are built as code/manifests/docs.** Live validation (EKS deploy,
+> screenshots, `terraform apply`) is the remaining manual step and is intentionally
+> deferred. All new manifests pass `kubectl kustomize` + `yamllint`.
+
+---
+
+## Showcase mode — Git vs runtime
+
+**Scope:** Staging is the only environment with a working pipeline right now. Production is **off** — kept in Git as a mirror of staging only. Whether/when to lift production runtime is an open decision, not a committed phase.
+
+| Layer | Staging | Production |
+|-------|---------|------------|
+| **Git** | Active source | **Mirror** of staging — kept aligned |
+| **Runtime** | EKS + CD on every `main` push | **Off** — not lifted; next steps TBD |
+| **Pipeline (CI/CD)** | Counted as the working pipeline | Excluded — no build, push, or deploy |
+
+**Before screenshots:** follow [docs/showcase-staging-only.md](docs/showcase-staging-only.md) (terraform apply **staging only**, bootstrap EKS staging, push to `main`).
 
 ---
 
 ## Phase 5 — Implementation Details (staging-first)
 
-Automatic CD targets **staging EKS only**. Dev (Minikube) remains manual. Production CD deferred — see [docs/cd-production-promotion.md](docs/cd-production-promotion.md).
+Automatic CD targets **staging EKS only** — staging is the only counted pipeline for now. Dev (Minikube) remains manual. Production is **off**: no CI build/push, no CD deploy. Production **manifests** stay in Git as a mirror of staging; turning production runtime on later is an undecided next step — reference: [docs/cd-production-promotion.md](docs/cd-production-promotion.md).
 
 ### Delivered
 
 - [`.github/workflows/cd.yaml`](.github/workflows/cd.yaml) — post-CI GitOps bump, rollout health check, traffic switch, rollback
 - [`scripts/`](scripts/) — `gitops-bump.sh`, `blue-green-switch.sh`, `blue-green-health.sh`, `rollback.sh`
 - [`k8s/overlays/staging/`](k8s/overlays/staging/) — blue/green overlay, sync waves, `cd-active-color` ConfigMap
+- [`k8s/overlays/production/`](k8s/overlays/production/) — Git mirror of staging (blue/green, production ECR, HA patches)
 - [`k8s/argocd/install-notes.md`](k8s/argocd/install-notes.md) — EKS staging bootstrap runbook
 - Terraform EKS access entry for staging `cicd` role ([`infrastructure/modules/eks/`](infrastructure/modules/eks/))
-- CI: `[skip ci]` guard; production ECR gated by `ENABLE_PROD_ECR` (default off)
+- CI: `[skip ci]` guard; **no production ECR push** until final promotion phase
 
 ### GitHub variables required
 
 | Variable | Example |
 |----------|---------|
 | `EKS_CLUSTER_NAME` | `mck21-devsecops-staging-eks` |
-| `STAGING_HEALTH_URL` | `https://<alb-hostname>/health` |
+| `STAGING_HEALTH_URL` | `https://<alb-hostname>/health` (optional — leave empty until Ingress ready) |
 
-Optional: `ENABLE_PROD_ECR=true`, secret `CD_BOT_TOKEN` if branch protection blocks `GITHUB_TOKEN`.
+Optional: secret `CD_BOT_TOKEN` if branch protection blocks `GITHUB_TOKEN`.
 
 ### Pending validation
 
@@ -61,17 +80,17 @@ Optional: `ENABLE_PROD_ECR=true`, secret `CD_BOT_TOKEN` if branch protection blo
 
 | Item | Reason | Target |
 |------|--------|--------|
-| Production CD | Staging-first cost/risk strategy | [docs/cd-production-promotion.md](docs/cd-production-promotion.md) |
+| Production runtime (EKS, CD) | Off for now — Git mirror only; staging is the working pipeline | TBD — decide next steps; reference [docs/cd-production-promotion.md](docs/cd-production-promotion.md) |
 | Dev automated CD | Minikube not reachable from GitHub-hosted runners | Manual dev deploy |
-| Architecture diagram, full deployment guide | Portfolio polish | Phase 9 |
-| Grafana dashboards | Monitoring stack not installed | Phase 7 |
+| Diagram PNG exports | Mermaid sources committed (`docs/diagrams/*.mmd`); render deferred with screenshots | Manual |
+| Live EKS validation of Phases 6–9 | Manifests/Helm values committed; not yet applied to a cluster | Manual |
 
 ---
 
 ## Environment Strategy
 
-| Environment | Kubernetes | CD |
-|-------------|------------|-----|
-| `dev` | Minikube (local) | Manual |
-| `staging` | EKS (AWS) | **Automatic** (merge to `main`) |
-| `production` | EKS (AWS) | Deferred |
+| Environment | Kubernetes (runtime) | Git manifests | CD |
+|-------------|----------------------|---------------|-----|
+| `dev` | Minikube (local) | `overlays/dev/` | Manual |
+| `staging` | EKS (AWS) | `overlays/staging/` | **Automatic** |
+| `production` | Off — not lifted | `overlays/production/` mirror | None (excluded; next steps TBD) |
diff --git a/docs/adr/001-argocd-over-fluxcd.md b/docs/adr/001-argocd-over-fluxcd.md
@@ -0,0 +1,23 @@
+# ADR-001: ArgoCD over Flux CD
+
+## Status
+Accepted
+
+## Context
+GitOps needs a controller that reconciles cluster state from Git. The two mature
+CNCF options are ArgoCD and Flux CD. We want clear visibility of sync/health for a
+portfolio project, multi-environment app management, and a gentle learning curve.
+
+## Decision
+Use **ArgoCD**. It ships a first-class UI showing per-Application sync and health
+status, the `Application` CRD maps cleanly to our per-overlay environments, and it
+integrates well with the blue/green traffic-switch flow and sync waves we use.
+
+## Consequences
+- **Easier:** observing drift/sync state visually (great for screenshots and
+  demos), modelling dev/staging/production as separate Applications, manual
+  sync/rollback from the UI or CLI.
+- **Harder / cost:** ArgoCD is a heavier install than Flux and is less "pure
+  GitOps" for image automation (we drive image bumps from CI rather than an
+  in-cluster image automation controller). Acceptable: the CD pipeline already
+  owns tag bumping and blue/green switching.
diff --git a/docs/adr/002-eks-over-ecs.md b/docs/adr/002-eks-over-ecs.md
@@ -0,0 +1,23 @@
+# ADR-002: EKS over ECS
+
+## Status
+Accepted
+
+## Context
+The platform needs a container orchestrator on AWS. ECS/Fargate is simpler and
+cheaper to operate; EKS is managed Kubernetes. The project's goal is to
+demonstrate portable, industry-standard Kubernetes + ecosystem skills (Istio,
+ArgoCD, Kyverno, Prometheus), and to run the *same* manifests locally on Minikube.
+
+## Decision
+Use **EKS**. Kubernetes is the portable target: identical manifests run on
+Minikube (dev) and EKS (staging/production) via Kustomize overlays, and the whole
+CNCF tooling stack (Istio, ArgoCD, Kyverno, External Secrets, kube-prometheus)
+assumes Kubernetes.
+
+## Consequences
+- **Easier:** one manifest set across local and cloud; access to the CNCF
+  ecosystem; transferable skills.
+- **Harder / cost:** EKS control plane (~$73/mo) and operational complexity vs
+  ECS. Mitigated with Spot nodes, single-AZ NAT on staging, off-hours shutdown,
+  and running **staging-only** during showcase.
diff --git a/docs/adr/003-kustomize-for-app-helm-for-infra.md b/docs/adr/003-kustomize-for-app-helm-for-infra.md
@@ -0,0 +1,25 @@
+# ADR-003: Kustomize for app manifests, Helm for platform components
+
+## Status
+Accepted
+
+## Context
+We need to template per-environment app config and also install large third-party
+platform components (Prometheus, Loki, Kyverno, ArgoCD, Velero, External Secrets).
+Helm and Kustomize solve overlapping but different problems.
+
+## Decision
+Use **Kustomize for our own application manifests** and **Helm for upstream
+platform components**.
+
+- App: a single `base/` with `overlays/{dev,staging,production}` patches. No
+  templating language, transparent diffs, native `kubectl -k` / ArgoCD support.
+- Platform: install via upstream Helm charts with environment-specific
+  `values-*.yaml` (kube-prometheus-stack, loki, promtail, kyverno, velero,
+  external-secrets).
+
+## Consequences
+- **Easier:** readable, reviewable app overlays; we don't reinvent complex charts
+  that vendors maintain; values files keep env differences explicit.
+- **Harder:** two mental models in one repo. Acceptable and conventional — the
+  boundary (our code = Kustomize, their code = Helm) is easy to reason about.
diff --git a/docs/adr/004-github-actions-over-jenkins.md b/docs/adr/004-github-actions-over-jenkins.md
@@ -0,0 +1,20 @@
+# ADR-004: GitHub Actions over Jenkins
+
+## Status
+Accepted
+
+## Context
+CI/CD needs an automation engine. Jenkins is powerful but self-hosted (a server to
+patch, secure, and scale). The repo already lives on GitHub.
+
+## Decision
+Use **GitHub Actions**. It is fully managed, co-located with the code, has a large
+marketplace, supports **OIDC federation to AWS** (no long-lived keys), and pins
+actions by commit SHA for supply-chain safety.
+
+## Consequences
+- **Easier:** no CI server to operate; OIDC removes static AWS credentials;
+  native PR checks and Sonar/Cosign integrations; `workflow_run` chaining for
+  CI→CD.
+- **Harder:** vendor lock-in to GitHub; less flexible than Jenkins for exotic
+  agents. Acceptable for this project's scope and the security benefit of OIDC.
diff --git a/docs/adr/005-istio-tradeoffs.md b/docs/adr/005-istio-tradeoffs.md
@@ -0,0 +1,24 @@
+# ADR-005: Keep Istio despite its operational cost
+
+## Status
+Accepted
+
+## Context
+Istio adds real complexity: sidecar injection, TCP services needing `appProtocol`
++ DestinationRules, LimitRange tuning for the proxy, and higher resource use
+(documented in [../../TROUBLESHOOTING.md](../../TROUBLESHOOTING.md)). A simpler
+ingress-only setup would avoid these. But the platform relies on mesh features.
+
+## Decision
+**Keep Istio.** It provides the blue/green traffic switching via `VirtualService`
+weights (core to the Phase 5 CD flow), consistent mesh metrics
+(`istio_requests_total`) that power the SLOs independently of app code, and
+distributed tracing via Jaeger.
+
+## Consequences
+- **Easier:** zero-downtime blue/green by shifting weights; uniform telemetry for
+  SLOs/dashboards; mTLS-capable mesh.
+- **Harder:** operational gotchas (TCP routing for postgres/redis, sidecar
+  resource overhead, NetworkPolicy must allow Istiod/DNS). These are documented
+  and templated, so the cost is paid once. For a tiny single-service app this is
+  arguably over-engineered — accepted deliberately to demonstrate mesh skills.
diff --git a/docs/adr/006-feature-flag-service-domain.md b/docs/adr/006-feature-flag-service-domain.md
@@ -0,0 +1,23 @@
+# ADR-006: Feature Flag Service as the demo domain
+
+## Status
+Accepted
+
+## Context
+A DevSecOps platform needs a demo workload. It should make each infrastructure
+decision feel *earned* rather than arbitrary (why a cache? why an HPA? why
+overlays?), while staying small enough to not distract from the platform.
+
+## Decision
+Build a **Feature Flag Service** (manage toggles per environment; Redis-cached
+reads; PostgreSQL audit log). Real-world equivalents: LaunchDarkly, Unleash,
+Flipt.
+
+## Consequences
+- **Earned decisions:** flag reads happen on every client request → justifies the
+  **Redis cache** and the **HPA**; per-environment toggles → justify **Kustomize
+  overlays**; the audit log → justifies **PostgreSQL** persistence; rare writes /
+  frequent reads → a clean cache-invalidation story.
+- **Bounded scope:** five small NestJS modules (`flags`, `audit`, `health`,
+  `cache`, `prisma`) keep app code from overshadowing the platform.
+- **Harder:** none significant; the domain is intentionally simple.