From f4576817f90900a87947cfab2ccdb934357ab518 Mon Sep 17 00:00:00 2001 From: mck21 Date: Fri, 10 Jul 2026 17:01:18 +0200 Subject: [PATCH 1/2] feat: add HPA scaling diagram for phase 3 --- ...-06-06 a las 19.15.22.png => 10-hpa-scaling.png} | Bin 1 file changed, 0 insertions(+), 0 deletions(-) rename images/phase-3/{Captura de pantalla 2026-06-06 a las 19.15.22.png => 10-hpa-scaling.png} (100%) diff --git a/images/phase-3/Captura de pantalla 2026-06-06 a las 19.15.22.png b/images/phase-3/10-hpa-scaling.png similarity index 100% rename from images/phase-3/Captura de pantalla 2026-06-06 a las 19.15.22.png rename to images/phase-3/10-hpa-scaling.png From a65a4800a5fc155cce0ee2dd5678267f48bc55bf Mon Sep 17 00:00:00 2001 From: mck21 Date: Fri, 10 Jul 2026 17:02:08 +0200 Subject: [PATCH 2/2] docs: update project documentation --- AGENTS.md | 27 +- README.md | 54 +- STATUS.md | 52 +- TROUBLESHOOTING.md | 1 - docs/architecture.md | 2 +- docs/deployment-guide.md | 14 +- docs/diagrams/README.md | 5 +- docs/monitoring.md | 6 + docs/resilience-testing.md | 10 +- docs/security.md | 10 + docs/showcase-staging-only.md | 12 +- docs/sre.md | 4 + docs/step-by-step.md | 542 ++++++++++++++++++ images/phase-5/README.md | 39 +- .../environments/production/README.md | 2 +- 15 files changed, 689 insertions(+), 91 deletions(-) create mode 100644 docs/step-by-step.md diff --git a/AGENTS.md b/AGENTS.md index c495b9a..867884c 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -21,7 +21,7 @@ Entry point for AI agents working on this repository. Read this file first, then | Dev cluster | Minikube (local) | | Staging / Production | EKS on AWS | -**Current phase:** Phases 1–9 built as code/manifests/docs (staging-first; production off). **Next:** live EKS validation + screenshots (`terraform apply`, deploy, capture evidence). See [STATUS.md](STATUS.md). +**Current phase:** Phases 1–9 complete (staging-first; production off). Live validation was done on staging EKS; the AWS environment has since been **decommissioned** — the screenshots in `images/phase-N/` are the final evidence set and no new captures are possible. See [STATUS.md](STATUS.md). --- @@ -35,7 +35,7 @@ devsecops-platform/ ├── k8s/ # Kustomize manifests (base + overlays + blue-green + argocd) ├── .github/workflows/ # ci.yaml, cd.yaml (main) ├── scripts/ # blue/green CD automation (Phase 5) -├── docs/ # Placeholder — full docs in Phase 9 +├── docs/ # Architecture, security, SRE, DR guides + ADRs + diagrams ├── images/phase-N/ # Screenshot evidence per phase ├── docker-compose.yml # Local dev stack (postgres + redis + backend) ├── PLAN.md # Full roadmap (1056 lines — use STATUS.md for current state) @@ -114,19 +114,20 @@ Summary: --- -## Phase Boundaries — what's built vs deferred +## Phase Boundaries — what's built vs closed -Phases 1–9 are now **implemented as code/manifests/docs**. What remains is -**manual/live** work, intentionally deferred: +Phases 1–9 are **complete**. Staging EKS validation happened live (ArgoCD sync, +blue/green, Kyverno, dashboards, pod recovery) and the environment was then +decommissioned — do not plan work that requires cluster or AWS access: | Status | Item | |--------|------| -| Built | Phase 6 security (`k8s/security`, `k8s/kyverno`, `k8s/external-secrets`, securityContext, Cosign in `ci.yaml`) | -| Built | Phase 7 monitoring (`monitoring/`, SLO rules, dashboards, `docs/sre.md`) | -| Built | Phase 8 DR/resilience (`k8s/velero`, `tests/k6`, DR + resilience docs) | -| Built | Phase 9 docs (`docs/` guides, `docs/adr/`, `docs/diagrams/*.mmd`) | -| Deferred (manual) | Live EKS apply of the above, screenshots, diagram PNG exports, `terraform apply` | -| Undecided | Production **runtime** (EKS apply, CD promote, ECR push) — **off**; reference [docs/cd-production-promotion.md](docs/cd-production-promotion.md) | +| Done | Phase 6 security (`k8s/security`, `k8s/kyverno`, `k8s/external-secrets`, securityContext, Cosign in `ci.yaml`) — enforced live on staging | +| Done | Phase 7 monitoring (`monitoring/`, SLO rules, dashboards, `docs/sre.md`) — dashboards validated on EKS | +| Done | Phase 8 DR/resilience (`k8s/velero`, `tests/k6`, DR + resilience docs) — pod recovery evidenced | +| Done | Phase 9 docs (`docs/` guides, `docs/adr/`, `docs/diagrams/*.mmd`) | +| Closed | Runtime evidence — AWS decommissioned; `images/phase-N/` is the final set, no new screenshots | +| Off | Production **runtime** (EKS apply, CD promote, ECR push) — Git mirror only; reference [docs/cd-production-promotion.md](docs/cd-production-promotion.md) | > When changing app manifests, keep Kyverno policies and securityContext intact > (Phase 6 enforces them on staging/prod). @@ -165,8 +166,8 @@ Phases 1–9 are now **implemented as code/manifests/docs**. What remains is 1. [TROUBLESHOOTING.md](TROUBLESHOOTING.md) — deploy runbook (Istio, Prisma, Minikube) 2. [k8s/README.md](k8s/README.md) — quick troubleshooting table -3. [PLAN.md](PLAN.md) — full phase requirements and acceptance criteria -4. [images/phase-N/](images/) — expected outcomes as screenshots +3. [docs/step-by-step.md](docs/step-by-step.md) — full build walkthrough with expected outcomes as screenshots +4. [PLAN.md](PLAN.md) — full phase requirements and acceptance criteria 5. [app/backend/README.md](app/backend/README.md) — NestJS structure and request lifecycle --- diff --git a/README.md b/README.md index f6707b8..5a2f4d5 100644 --- a/README.md +++ b/README.md @@ -20,6 +20,8 @@ The application is a **Feature Flag Service** — manage feature toggles per environment with Redis-cached reads and PostgreSQL audit logging. Similar to LaunchDarkly or Unleash, it serves as the demo workload for the full DevSecOps pipeline. +> **Want to build this yourself?** Follow the [**Step-by-Step Guide**](docs/step-by-step.md) — a full walkthrough from empty AWS account to working GitOps pipeline, illustrated with evidence captured while this pipeline ran live. + **For AI agents:** start with [AGENTS.md](AGENTS.md) and [STATUS.md](STATUS.md). Deploy issues: [TROUBLESHOOTING.md](TROUBLESHOOTING.md). --- @@ -32,15 +34,16 @@ The application is a **Feature Flag Service** — manage feature toggles per env | 2 | Kubernetes base (Minikube + Istio) | Done | | 3 | Feature Flag Service + K8s manifests | **Done** | | 4 | CI Pipeline (GitHub Actions, SonarCloud, ECR) | **Done** | -| 5 | CD Pipeline & GitOps (ArgoCD, staging-first) | **Implemented** — validate on EKS | -| 6 | Security hardening (Kyverno, Cosign, NetworkPolicy, External Secrets) | **Implemented** | -| 7 | Monitoring & SRE (Prometheus, Grafana, Loki, SLOs) | **Implemented** | -| 8 | DR & resilience (Velero, k6) | **Implemented** | -| 9 | Docs, ADRs, cost | **Implemented** | +| 5 | CD Pipeline & GitOps (ArgoCD, staging-first) | **Done** — validated on staging EKS | +| 6 | Security hardening (Kyverno, Cosign, NetworkPolicy, External Secrets) | **Done** | +| 7 | Monitoring & SRE (Prometheus, Grafana, Loki, SLOs) | **Done** | +| 8 | DR & resilience (Velero, k6) | **Done** | +| 9 | Docs, ADRs, cost | **Done** | -All manifests/config/docs are committed. Live validation (EKS deploy + screenshots) -is the remaining step — staging is the only counted runtime; production is a Git -mirror that stays **off**. Full tracker: [STATUS.md](STATUS.md) · Roadmap: [PLAN.md](PLAN.md) +All phases are complete. The pipeline was validated live on staging EKS — evidence +is in `images/phase-N/`. The AWS environment has since been decommissioned; staging +was the only counted runtime, and production remains a Git mirror that stays **off**. +Full tracker: [STATUS.md](STATUS.md) · Roadmap: [PLAN.md](PLAN.md) --- @@ -167,19 +170,44 @@ docs/ Full documentation (Phase 9) ## Documentation +**Start here → [Step-by-Step Guide](docs/step-by-step.md)** — rebuild the whole platform phase by phase, with real evidence at every step. + +### Guides & concepts + | Topic | Doc | |-------|-----| -| Architecture & diagrams | [docs/architecture.md](docs/architecture.md), [docs/diagrams/](docs/diagrams/) | -| Deployment (Minikube + EKS) | [docs/deployment-guide.md](docs/deployment-guide.md), [k8s/README.md](k8s/README.md) | +| **Build it yourself (with screenshots)** | [docs/step-by-step.md](docs/step-by-step.md) | +| Architecture & diagrams | [docs/architecture.md](docs/architecture.md) · [docs/diagrams/](docs/diagrams/) | +| Deployment quick reference (Minikube + EKS) | [docs/deployment-guide.md](docs/deployment-guide.md) · [k8s/README.md](k8s/README.md) | | Security hardening | [docs/security.md](docs/security.md) | -| Monitoring & observability | [docs/monitoring.md](docs/monitoring.md), [monitoring/README.md](monitoring/README.md) | +| Monitoring & observability | [docs/monitoring.md](docs/monitoring.md) · [monitoring/README.md](monitoring/README.md) | | SRE — SLI/SLO/error budget | [docs/sre.md](docs/sre.md) | -| Disaster recovery | [docs/disaster-recovery.md](docs/disaster-recovery.md) | -| Resilience testing | [docs/resilience-testing.md](docs/resilience-testing.md) | | SonarCloud quality gate | [docs/sonarqube.md](docs/sonarqube.md) | | Cost estimates | [docs/cost.md](docs/cost.md) | | Architecture Decision Records | [docs/adr/](docs/adr/) | +### Runbooks + +| Topic | Doc | +|-------|-----| +| Disaster recovery | [docs/disaster-recovery.md](docs/disaster-recovery.md) | +| Resilience testing | [docs/resilience-testing.md](docs/resilience-testing.md) | +| Staging-only showcase (re-lift the runtime) | [docs/showcase-staging-only.md](docs/showcase-staging-only.md) | +| Production promotion (reference — off) | [docs/cd-production-promotion.md](docs/cd-production-promotion.md) | +| EKS staging bootstrap | [k8s/argocd/install-notes.md](k8s/argocd/install-notes.md) | +| Troubleshooting (Istio, Prisma, Minikube, CD) | [TROUBLESHOOTING.md](TROUBLESHOOTING.md) | + +### Project meta + +| Topic | Doc | +|-------|-----| +| Phase tracker | [STATUS.md](STATUS.md) | +| Original roadmap (historical) | [PLAN.md](PLAN.md) | +| AI agent entry point | [AGENTS.md](AGENTS.md) | +| Contributing | [CONTRIBUTE.md](CONTRIBUTE.md) | +| Backend internals | [app/backend/README.md](app/backend/README.md) | +| Evidence screenshots | [images/](images/) — `phase-N/` per phase | + --- ## Contributing diff --git a/STATUS.md b/STATUS.md index 4bf3568..8b184bc 100644 --- a/STATUS.md +++ b/STATUS.md @@ -1,8 +1,8 @@ # Project Status -Phase tracker for the DevSecOps platform. Updated after Phase 5 implementation. +Phase tracker for the DevSecOps platform. All phases complete. -**Active work:** Phases 6–9 built as code/manifests/docs; pending live EKS validation + screenshots +**Active work:** None — Phases 1–9 complete and validated on staging EKS. The AWS environment has been decommissioned; the evidence in `images/phase-N/` is final. **Agent entry point:** [AGENTS.md](AGENTS.md) **Deploy runbook:** [TROUBLESHOOTING.md](TROUBLESHOOTING.md) **Full roadmap:** [PLAN.md](PLAN.md) @@ -17,30 +17,32 @@ Phase tracker for the DevSecOps platform. Updated after Phase 5 implementation. | 1 | Terraform & AWS (VPC, EKS, ECR, IAM) | Done | `infrastructure/`, `images/phase-1/` | | 2 | Kubernetes base (Minikube: Istio, ArgoCD, Ingress) | Done (Minikube) | `k8s/namespaces/`, `images/phase-2/`, install notes in `k8s/argocd/`, `k8s/istio/` | | 3 | Feature Flag Service + K8s manifests | **Done** | `app/backend/`, `k8s/`, `images/phase-3/` | -| 4 | CI Pipeline (GitHub Actions, SonarCloud, ECR) | **Done** | `.github/workflows/`, `sonar-project.properties`, `images/phase-4/` | -| 5 | CD & GitOps (ArgoCD sync, blue/green automation) | **Implemented** (staging-first) | `.github/workflows/cd.yaml`, `scripts/`, `images/phase-5/` | -| 6 | Security hardening (Kyverno, Cosign, NetworkPolicy, External Secrets) | **Implemented** | `k8s/security/`, `k8s/kyverno/`, `k8s/external-secrets/`, securityContext in deployments, Cosign in `ci.yaml` | -| 7 | Monitoring & SRE (Prometheus, Grafana, Loki) | **Implemented** | `monitoring/`, `docs/sre.md` | -| 8 | DR & resilience (Velero, k6) | **Implemented** | `k8s/velero/`, `tests/k6/`, `scripts/resilience-test.sh`, `docs/disaster-recovery.md`, `docs/resilience-testing.md` | -| 9 | Documentation, ADRs, diagrams, cost docs | **Implemented** | `docs/` (architecture, deployment-guide, security, monitoring, sonarqube, cost), `docs/adr/`, `docs/diagrams/` | - -> **Phases 6–9 are built as code/manifests/docs.** Live validation (EKS deploy, -> screenshots, `terraform apply`) is the remaining manual step and is intentionally -> deferred. All new manifests pass `kubectl kustomize` + `yamllint`. +| 4 | CI Pipeline (GitHub Actions, SonarCloud, ECR) | **Done** | `.github/workflows/`, `sonar-project.properties` | +| 5 | CD & GitOps (ArgoCD sync, blue/green automation) | **Done** (staging-first) | `.github/workflows/cd.yaml`, `scripts/`, `images/phase-5/` | +| 6 | Security hardening (Kyverno, Cosign, NetworkPolicy, External Secrets) | **Done** | `k8s/security/`, `k8s/kyverno/`, `k8s/external-secrets/`, securityContext in deployments, Cosign in `ci.yaml`, `images/phase-6/` | +| 7 | Monitoring & SRE (Prometheus, Grafana, Loki) | **Done** | `monitoring/`, `docs/sre.md`, `images/phase-7/` | +| 8 | DR & resilience (Velero, k6) | **Done** | `k8s/velero/`, `tests/k6/`, `scripts/resilience-test.sh`, `docs/disaster-recovery.md`, `docs/resilience-testing.md`, `images/phase-8/` | +| 9 | Documentation, ADRs, diagrams, cost docs | **Done** | `docs/` (architecture, deployment-guide, security, monitoring, sonarqube, cost), `docs/adr/`, `docs/diagrams/` | + +> **Phases 5–8 were validated live on staging EKS** (ArgoCD sync, blue/green, +> Kyverno enforcement, dashboards, pod recovery) before the AWS environment was +> decommissioned. The screenshots in `images/phase-N/` are the final evidence set +> — no further captures are possible. All manifests pass `kubectl kustomize` + +> `yamllint`. --- ## Showcase mode — Git vs runtime -**Scope:** Staging is the only environment with a working pipeline right now. Production is **off** — kept in Git as a mirror of staging only. Whether/when to lift production runtime is an open decision, not a committed phase. +**Scope:** Staging was the only environment with a working pipeline. Production is **off** — kept in Git as a mirror of staging only. The staging AWS runtime has since been decommissioned; the Git manifests and captured evidence document the working state. | Layer | Staging | Production | |-------|---------|------------| | **Git** | Active source | **Mirror** of staging — kept aligned | -| **Runtime** | EKS + CD on every `main` push | **Off** — not lifted; next steps TBD | +| **Runtime** | Validated on EKS + CD, since decommissioned | **Off** — never lifted | | **Pipeline (CI/CD)** | Counted as the working pipeline | Excluded — no build, push, or deploy | -**Before screenshots:** follow [docs/showcase-staging-only.md](docs/showcase-staging-only.md) (terraform apply **staging only**, bootstrap EKS staging, push to `main`). +**To re-lift staging:** follow [docs/showcase-staging-only.md](docs/showcase-staging-only.md) (terraform apply **staging only**, bootstrap EKS staging, push to `main`). --- @@ -67,12 +69,12 @@ Automatic CD targets **staging EKS only** — staging is the only counted pipeli Optional: secret `CD_BOT_TOKEN` if branch protection blocks `GITHUB_TOKEN`. -### Pending validation +### Validation (complete) -- [ ] One-time EKS staging bootstrap per install notes -- [ ] `terraform apply` in staging for EKS access entry -- [ ] Merge to `main` → CI → CD green -- [ ] Screenshots in [`images/phase-5/`](images/phase-5/) per README checklist +- [x] One-time EKS staging bootstrap per install notes +- [x] `terraform apply` in staging for EKS access entry +- [x] Merge to `main` → CI → CD green +- [x] Evidence captured in [`images/phase-5/`](images/phase-5/) (ArgoCD sync, blue/green VirtualService) --- @@ -80,10 +82,10 @@ Optional: secret `CD_BOT_TOKEN` if branch protection blocks `GITHUB_TOKEN`. | Item | Reason | Target | |------|--------|--------| -| Production runtime (EKS, CD) | Off for now — Git mirror only; staging is the working pipeline | TBD — decide next steps; reference [docs/cd-production-promotion.md](docs/cd-production-promotion.md) | +| Production runtime (EKS, CD) | Off — Git mirror only; staging was the validated pipeline | Not planned; reference [docs/cd-production-promotion.md](docs/cd-production-promotion.md) | | Dev automated CD | Minikube not reachable from GitHub-hosted runners | Manual dev deploy | -| Diagram PNG exports | Mermaid sources committed (`docs/diagrams/*.mmd`); render deferred with screenshots | Manual | -| Live EKS validation of Phases 6–9 | Manifests/Helm values committed; not yet applied to a cluster | Manual | +| Diagram PNG exports | Mermaid sources committed (`docs/diagrams/*.mmd`); GitHub renders them inline, PNGs only needed for slides/PDFs | Optional | +| Additional runtime evidence | AWS environment decommissioned — no cluster access; existing `images/phase-N/` screenshots are final | Closed | --- @@ -92,5 +94,5 @@ Optional: secret `CD_BOT_TOKEN` if branch protection blocks `GITHUB_TOKEN`. | Environment | Kubernetes (runtime) | Git manifests | CD | |-------------|----------------------|---------------|-----| | `dev` | Minikube (local) | `overlays/dev/` | Manual | -| `staging` | EKS (AWS) | `overlays/staging/` | **Automatic** | -| `production` | Off — not lifted | `overlays/production/` mirror | None (excluded; next steps TBD) | +| `staging` | EKS (AWS) — validated, since decommissioned | `overlays/staging/` | **Automatic** (when lifted) | +| `production` | Off — never lifted | `overlays/production/` mirror | None (excluded) | diff --git a/TROUBLESHOOTING.md b/TROUBLESHOOTING.md index 6d271f6..e4934d6 100644 --- a/TROUBLESHOOTING.md +++ b/TROUBLESHOOTING.md @@ -481,4 +481,3 @@ When a new deploy issue is discovered: 1. Document the **symptom**, **cause**, and **fix** in this file 2. Add a row to the [Quick Symptom Index](#quick-symptom-index) 3. Update [AGENTS.md](AGENTS.md) if it affects agent phase boundaries -4. Capture evidence in `images/phase-N/` if relevant diff --git a/docs/architecture.md b/docs/architecture.md index adeb2ff..776073a 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -103,7 +103,7 @@ sequenceDiagram Prometheus scrapes app `/metrics` + Istio mesh + cluster; Grafana dashboards; Loki/Promtail logs; SLOs with multi-window burn-rate alerts. See -[../docs/sre.md](sre.md) and [../monitoring/README.md](../monitoring/README.md). +[sre.md](sre.md) and [monitoring/README.md](../monitoring/README.md). ## Resilience (Phase 8) diff --git a/docs/deployment-guide.md b/docs/deployment-guide.md index 5b63ebe..70b29f4 100644 --- a/docs/deployment-guide.md +++ b/docs/deployment-guide.md @@ -1,7 +1,9 @@ # Deployment Guide -Covers local **Minikube** (dev) and cloud **EKS** (staging). Production runtime is -off during showcase — see [showcase-staging-only.md](showcase-staging-only.md). +Quick reference for deploying to local **Minikube** (dev) and cloud **EKS** +(staging). First time here? The narrative walkthrough with screenshots is +[step-by-step.md](step-by-step.md). Production runtime is off during showcase — +see [showcase-staging-only.md](showcase-staging-only.md). ## Prerequisites @@ -37,6 +39,10 @@ kubectl port-forward svc/backend -n dev 3000:80 curl http://localhost:3000/health ``` +Expected result — all pods `2/2` (app + Istio sidecar): + +![Pods running in dev namespace](../images/phase-3/07-k8s-pods-dev.png) + Gotchas (Prisma v7, Istio TCP, image caching, LimitRange): [../TROUBLESHOOTING.md](../TROUBLESHOOTING.md). Optional add-ons on dev: @@ -103,6 +109,10 @@ gh run list --workflow=CD --limit 3 CI builds/scans/signs/pushes to ECR; CD bumps the idle blue/green color, ArgoCD syncs, health check runs, traffic switches. Rollback is automatic on failure. +The app answering on EKS after a green pipeline: + +![App responding on EKS](../images/phase-3/08-eks-app.png) + --- ## C. Production diff --git a/docs/diagrams/README.md b/docs/diagrams/README.md index aff3676..beea25f 100644 --- a/docs/diagrams/README.md +++ b/docs/diagrams/README.md @@ -1,7 +1,8 @@ # Diagrams -Source-controlled Mermaid sources. PNG exports are deferred (same as screenshots) -— render them when preparing the portfolio: +Source-controlled Mermaid sources. GitHub renders the inline Mermaid in README +and architecture.md natively; PNG exports are optional (slide decks / PDFs only) +and need no AWS access — render on demand: ```bash # Mermaid CLI diff --git a/docs/monitoring.md b/docs/monitoring.md index 2e152a5..36c4d76 100644 --- a/docs/monitoring.md +++ b/docs/monitoring.md @@ -23,12 +23,18 @@ overview. across dev and EKS, independent of app code). - **Cluster:** node-exporter + kube-state-metrics (bundled). +All targets reporting UP on the live staging cluster: + +![Prometheus targets up](../images/phase-7/02-prometheus-targets.png) + ## Dashboards Custom: **Feature Flag Service** (`feature-flags-app`) and **SLO — Error Budget** (`slo-error-budget`), provisioned as `grafana_dashboard` ConfigMaps. Community: Kubernetes (7249), Istio Mesh (7639), Node Exporter (1860). +![Grafana Feature Flag Service dashboard](../images/phase-7/01-grafana-dashboard.png) + ## SLOs & alerting SLIs/SLOs, error-budget policy and multi-window burn-rate alerts: diff --git a/docs/resilience-testing.md b/docs/resilience-testing.md index 6566e28..7d7008f 100644 --- a/docs/resilience-testing.md +++ b/docs/resilience-testing.md @@ -66,7 +66,11 @@ kubectl rollout restart deployment/backend-blue -n staging # pick up new value See [disaster-recovery.md → Procedure 1](disaster-recovery.md#procedure-1--restore-a-namespace). -## Recording results +## Recorded results -Capture evidence (terminal output, Grafana HPA panel, ArgoCD rollback) under -`images/phase-8/` when running the live drills. +The live drill was run on staging EKS before the AWS environment was +decommissioned — the deleted backend pod was recreated automatically with no +downtime. That evidence set is final; the drills above remain runnable on any +future cluster. + +![Pod deleted and automatically recovered](../images/phase-8/01-pod-recover.png) diff --git a/docs/security.md b/docs/security.md index bb1a380..d046ed8 100644 --- a/docs/security.md +++ b/docs/security.md @@ -22,6 +22,10 @@ Defense-in-depth across build, supply chain, admission, runtime, and secrets. - writable `/tmp` via emptyDir (compatible with read-only root fs) - `automountServiceAccountToken: false` +Verified on the live pod: + +![Pod securityContext hardened](../images/phase-6/03-pod-securitycontext.png) + ## Pod Security Standards Namespace labels (`k8s/namespaces/namespaces.yaml`): **enforce baseline**, @@ -43,6 +47,10 @@ For a production runtime, prefer managed RDS/ElastiCache and enforce restricted. Install + demo: [../k8s/kyverno/install-notes.md](../k8s/kyverno/install-notes.md). +Enforcement in action — a violating pod rejected at admission: + +![Kyverno policy enforcement](../images/phase-6/01-kyverno-enforced.png) + ## Supply chain — Cosign CI signs every pushed image **keylessly** via GitHub Actions OIDC @@ -64,6 +72,8 @@ Default-deny ingress per app namespace, plus explicit allows: ingress-nginx → backend:3000, monitoring → backend:3000 (scrape), backend → postgres:5432 / redis:6379, and DNS + Istiod egress. See `k8s/security/network-policies.yaml`. +![NetworkPolicies and RBAC applied](../images/phase-6/02-networkpolicies-rba.png) + ## RBAC Least-privilege: cluster-wide read-only `platform-readonly` for auditors; diff --git a/docs/showcase-staging-only.md b/docs/showcase-staging-only.md index 0452120..6364000 100644 --- a/docs/showcase-staging-only.md +++ b/docs/showcase-staging-only.md @@ -5,9 +5,9 @@ This portfolio project uses a **Git vs runtime** split for production: | Layer | Staging | Production | |-------|---------|------------| | **Git** (manifests, Terraform) | Active source | **Mirror** — kept aligned with staging | -| **Runtime** (AWS, EKS, CD) | **Lifted** for showcase | **Off** — not lifted; next steps undecided | +| **Runtime** (AWS, EKS, CD) | Validated live, since **decommissioned** | **Off** — never lifted | -Staging is the only environment with a working pipeline right now. You only need **staging** running for green CI + CD and Phase 5 screenshots. +Staging was the only environment with a working pipeline; it was validated live on EKS and the AWS runtime has since been decommissioned. This runbook is kept as the procedure to re-lift staging if ever needed — you only need **staging** running for green CI + CD. --- @@ -105,8 +105,12 @@ Production runtime is off and not scheduled. If we later decide to enable it, se --- -## Phase 5 screenshots +## Phase 5 evidence -After CI + CD are green, capture evidence per [images/phase-5/README.md](../images/phase-5/README.md). +Captured while the staging pipeline ran live — see [images/phase-5/README.md](../images/phase-5/README.md). That evidence set is final; no cluster access remains. + +![ArgoCD staging app synced and healthy](../images/phase-5/01-argocd-sync.png) + +![VirtualService blue/green weights](../images/phase-5/03-blue-green-virtualservice.png) Troubleshooting: [TROUBLESHOOTING.md § EKS staging CD](../TROUBLESHOOTING.md#eks-staging-cd-phase-5). diff --git a/docs/sre.md b/docs/sre.md index f7b3241..6b859aa 100644 --- a/docs/sre.md +++ b/docs/sre.md @@ -49,6 +49,10 @@ a **slow burn** (6×) catches gradual degradation before it silently drains the monthly budget. Each alert requires a long **and** a short window to fire, which suppresses flapping from brief spikes. +Rules loaded on the live staging Prometheus: + +![Prometheus SLO alerts loaded](../images/phase-7/03-prometheus-alerts.png) + ## Dashboards - **SLO — Error Budget** (`slo-error-budget`): availability vs target, budget diff --git a/docs/step-by-step.md b/docs/step-by-step.md new file mode 100644 index 0000000..129f3be --- /dev/null +++ b/docs/step-by-step.md @@ -0,0 +1,542 @@ +# Build This Platform — Step by Step + +A hands-on walkthrough to rebuild the whole platform from an empty AWS account +to a working GitOps pipeline with security policies, monitoring, and disaster +recovery. Each step matches a project phase and shows **real evidence** captured +while this pipeline ran live. + +> **How to read this guide.** Commands are copy-pasteable and assume the repo +> root as working directory. Screenshots show the expected outcome of each step — +> if your output looks like the image, you're on track. When something breaks, +> jump to [TROUBLESHOOTING.md](../TROUBLESHOOTING.md) before debugging blind. + +## Contents + +1. [Prerequisites](#0--prerequisites) +2. [AWS foundations with Terraform](#1--aws-foundations-with-terraform) +3. [Kubernetes base: Minikube, ArgoCD, Istio](#2--kubernetes-base-minikube-argocd-istio) +4. [The application: Feature Flag Service](#3--the-application-feature-flag-service) +5. [CI pipeline](#4--ci-pipeline) +6. [CD & GitOps: blue/green on EKS](#5--cd--gitops-bluegreen-on-eks) +7. [Security hardening](#6--security-hardening) +8. [Monitoring & SRE](#7--monitoring--sre) +9. [Disaster recovery & resilience](#8--disaster-recovery--resilience) +10. [Documentation & cost](#9--documentation--cost) +11. [Teardown](#teardown) + +--- + +## 0 · Prerequisites + +| Tool | Used for | +|------|----------| +| `terraform` ≥ 1.5 | AWS infrastructure | +| `aws` CLI v2 | Account access, EKS kubeconfig | +| `kubectl` | Talking to clusters | +| `minikube` | Local dev cluster | +| `istioctl` | Service mesh install | +| `helm` | ArgoCD, monitoring, Kyverno, Velero | +| `docker` + Docker Compose | Local app stack, image builds | +| `bun` | Backend runtime/tooling (NestJS) | +| `gh` (optional) | Watching Actions runs from the terminal | + +You also need an AWS account with admin access for the initial Terraform run, +and a GitHub repository (fork or clone of this one) for the CI/CD phases. + +--- + +## 1 · AWS foundations with Terraform + +**Goal:** VPC, EKS cluster, ECR registry, and IAM roles — all reproducible from +`infrastructure/`. + +### 1.1 Remote state first + +Terraform state must live somewhere durable *before* any real infrastructure +exists. Create the S3 state bucket (plus a lock mechanism) once: + +![S3 tfstate bucket created](../images/phase-1/01-bucket-tfstate-created.png) + +Enable versioning and state locking so concurrent applies can't corrupt state +and any bad apply can be rolled back to a previous state version: + +![State locking and versioning enabled](../images/phase-1/02-bucket-tflock-and-versioning.png) + +### 1.2 Apply the network + cluster + +The modules under `infrastructure/modules/` (vpc, eks, ecr, iam, +security-groups) are composed per environment: + +```bash +cd infrastructure/environments/staging +terraform init +terraform plan # review before applying +terraform apply +``` + +The VPC spans multiple AZs with public subnets (load balancers) and private +subnets (worker nodes): + +![VPC resource map](../images/phase-1/03-vpc-resource-map.png) + +IAM roles are created for the cluster, node groups, and the CI/CD principal — +no long-lived user keys: + +![IAM roles created](../images/phase-1/04-iam-roles-created.png) + +Security groups restrict traffic between the control plane, nodes, and load +balancers: + +![Security groups](../images/phase-1/05-security-groups.png) + +### 1.3 Verify the cluster + +When the apply finishes, the EKS cluster is active: + +![EKS cluster created](../images/phase-1/06-eks-cluster-created.png) + +with its managed node group of EC2 workers: + +![Worker nodes](../images/phase-1/07-worker-nodes.png) + +Point `kubectl` at it and confirm the system pods and nodes are healthy: + +```bash +aws eks update-kubeconfig --name mck21-devsecops-staging-eks --region us-east-1 +kubectl get pods -n kube-system +kubectl get nodes +``` + +![kube-system pods running](../images/phase-1/08-kube-system-pods.png) + +![kubectl get nodes](../images/phase-1/09-kubectl-get-nodes.png) + +### 1.4 Container registry + +ECR is created with **scan-on-push** enabled, so every image is vulnerability +scanned the moment CI pushes it: + +![ECR repo with scan on push](../images/phase-1/10-ecr-repo-scan-on-push.png) + +### 1.5 Staging environment + +The same modules produce the dedicated **staging** cluster used by the CD +pipeline later in this guide: + +![Staging EKS cluster](../images/phase-1/12-eks-cluster-staging.png) + +![Staging nodes ready](../images/phase-1/11-eks-nodes-staging.png) + +> **Checkpoint:** `terraform apply` is clean, `kubectl get nodes` shows Ready +> nodes, ECR exists with scan-on-push. Cost note: an idle EKS cluster bills +> ~$73/month for the control plane alone — see [cost.md](cost.md) and tear down +> when not in use. + +--- + +## 2 · Kubernetes base: Minikube, ArgoCD, Istio + +**Goal:** a free local cluster that mirrors the platform layout — namespaces, +GitOps controller, and service mesh — before spending anything on AWS runtime. + +### 2.1 Local cluster + +```bash +minikube start --cpus=4 --memory=8192 --driver=docker +minikube addons enable ingress +minikube addons enable metrics-server +``` + +![Minikube configured](../images/phase-2/01-minikube-config.png) + +Create the platform namespaces (`dev`, `staging`, `production`, plus system +namespaces) with their Pod Security labels: + +```bash +kubectl apply -f k8s/namespaces/namespaces.yaml +``` + +![Namespaces and certificates](../images/phase-2/02-cluster-ns-and-certificates.png) + +### 2.2 ArgoCD + +Install the GitOps controller — everything deployed to staging later goes +through it, never `kubectl apply` by hand: + +```bash +kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml +kubectl port-forward svc/argocd-server -n argocd 8080:443 +``` + +![ArgoCD pods running](../images/phase-2/03-argocd-pods-running.png) + +Log in at `https://localhost:8080` (user `admin`, password from the +`argocd-initial-admin-secret` secret): + +![ArgoCD UI](../images/phase-2/04-argocd-ui.png) + +### 2.3 Istio service mesh + +Istio provides mTLS, traffic splitting (the mechanism behind blue/green in +step 5), and the mesh metrics that power the SLOs in step 7. Use the `demo` +profile locally: + +```bash +istioctl install --set profile=demo -y +kubectl label namespace dev istio-injection=enabled --overwrite +``` + +![Istio demo profile installed](../images/phase-2/05-istio-install-demo.png) + +Add the observability addons (Kiali, Jaeger, Prometheus): + +```bash +kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.24/samples/addons/kiali.yaml +kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.24/samples/addons/jaeger.yaml +kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.24/samples/addons/prometheus.yaml +``` + +![Kiali, Jaeger and Prometheus applied](../images/phase-2/06-apply-kiali-jaeger-prometheus.png) + +Kiali visualizes the mesh topology and live traffic: + +![Kiali dashboard](../images/phase-2/07-kiali-dashboard.png) + +Jaeger traces requests across the sidecar proxies: + +![Jaeger UI](../images/phase-2/08-jaeger-ui.png) + +> **Checkpoint:** ArgoCD UI reachable, `istioctl verify-install` clean, Kiali +> shows the mesh. Gotcha: Istio needs LimitRange minimums of `10m` CPU / `40Mi` +> memory or sidecar injection fails — see +> [TROUBLESHOOTING.md](../TROUBLESHOOTING.md). + +--- + +## 3 · The application: Feature Flag Service + +**Goal:** the demo workload — a NestJS API managing feature flags with +Redis-cached reads and a PostgreSQL audit trail. Flags are written rarely but +read on every client request; that read path is what justifies caching and HPA. + +### 3.1 Run it with Docker Compose + +The fastest way to see the app working — three containers, no Kubernetes: + +```bash +docker compose up --build +``` + +![Docker Compose stack running](../images/phase-3/01-docker-compose-running.png) + +### 3.2 Exercise the API + +Create a flag: + +```bash +curl -X POST http://localhost:3000/api/flags \ + -H 'Content-Type: application/json' \ + -d '{"key":"dark-mode","name":"Dark mode","environments":{"dev":true}}' +``` + +![POST create flag](../images/phase-3/02-post-create-flag.png) + +Toggle it per environment — this invalidates the Redis cache entry: + +```bash +curl -X PATCH http://localhost:3000/api/flags/dark-mode/toggle \ + -H 'Content-Type: application/json' -d '{"environment":"dev"}' +``` + +![PATCH toggle flag](../images/phase-3/03-patch-toggle-flag.png) + +Every create/toggle/delete is written to the audit log: + +![GET audit history](../images/phase-3/04-get-audit-history.png) + +![Audit endpoint](../images/phase-3/09-audit-endpoint.png) + +The app also exposes what the platform needs: Prometheus metrics and a health +endpoint that checks PostgreSQL and Redis connectivity: + +```bash +curl http://localhost:3000/metrics +curl http://localhost:3000/health +``` + +![Prometheus metrics endpoint](../images/phase-3/05-get-prometheus-metrics.png) + +![Health checks](../images/phase-3/06-health-checks.png) + +### 3.3 Deploy to Kubernetes (dev overlay) + +The manifests are Kustomize: an env-agnostic `k8s/base/` plus per-environment +overlays. Deploy the dev overlay to Minikube: + +```bash +docker build -t backend:latest ./app/backend +minikube image load backend:latest +kubectl apply -f k8s/base/secret.yaml -n dev # local template only +kubectl apply -k k8s/overlays/dev +kubectl get pods -n dev +``` + +All pods run `2/2` — the app container plus its Istio sidecar: + +![Pods running in dev namespace](../images/phase-3/07-k8s-pods-dev.png) + +Under load, the HPA scales the backend while LimitRange and ResourceQuota keep +the namespace within bounds: + +![HPA scaling with LimitRange and ResourceQuota](../images/phase-3/10-hpa-scaling.png) + +The same manifests later run on EKS via the staging overlay: + +![App responding on EKS](../images/phase-3/08-eks-app.png) + +> **Checkpoint:** API answers on Compose *and* on Minikube; pods are `2/2`; +> HPA reacts to load. The Istio+TCP gotchas (postgres/redis port naming, +> DestinationRules, `holdApplicationUntilProxyStarts`) are the most common +> failure here — [TROUBLESHOOTING.md](../TROUBLESHOOTING.md) covers all of them. + +--- + +## 4 · CI pipeline + +**Goal:** every push is linted, tested, quality-gated, scanned, and — on `main` +— built, signed, and pushed to ECR. Defined in +[`.github/workflows/ci.yaml`](../.github/workflows/ci.yaml). + +| Job | What it does | +|-----|--------------| +| **Lint** | ESLint (backend), yamllint (manifests), `terraform fmt -check` | +| **Unit Tests** | Jest with coverage (Bun), coverage artifact for Sonar | +| **Build, Scan & Push** | Docker build → Trivy image scan → push `sha-` to staging ECR → **Cosign keyless sign** | + +Supporting pieces: + +- **SonarCloud** enforces a Quality Gate on every PR via the GitHub App — setup + and rules in [sonarqube.md](sonarqube.md). +- Images are tagged with the immutable `sha-` (never `:latest`), which + Kyverno later enforces at admission. +- The Cosign signature is recorded in the Rekor transparency log and verified + at admission in step 6 — unsigned images can't run in staging. + +There are no screenshots for this phase — the evidence is the pipeline itself: +green runs in the repo's **Actions** tab and the SonarCloud dashboard. + +> **Checkpoint:** a PR shows Lint + Unit Tests + Sonar Quality Gate checks; a +> merge to `main` additionally pushes a signed image to ECR. + +--- + +## 5 · CD & GitOps: blue/green on EKS + +**Goal:** merging to `main` deploys to staging EKS with zero-downtime blue/green +switching — and Git, not a human, is the source of truth for what runs. + +### 5.1 Bootstrap the staging cluster (one-time) + +Follow [`k8s/argocd/install-notes.md`](../k8s/argocd/install-notes.md): Istio +(production profile), Ingress NGINX, cert-manager, `backend-secrets`, ArgoCD via +Helm, repo connection, then: + +```bash +kubectl apply -f k8s/argocd/application-staging.yaml +``` + +Set the GitHub repo variables `EKS_CLUSTER_NAME` and (optionally) +`STAGING_HEALTH_URL`. + +### 5.2 How a deploy flows + +Defined in [`.github/workflows/cd.yaml`](../.github/workflows/cd.yaml) + +[`scripts/`](../scripts/): + +1. **Gate** — runs only if CI succeeded on `main`. +2. **GitOps bump** — `gitops-bump.sh` edits the *idle* color's image tag in + `k8s/overlays/staging/` and pushes a `[skip ci]` commit. Git now describes + the desired state. +3. **Sync** — ArgoCD detects the drift and rolls out the idle color. + +![ArgoCD staging app synced and healthy](../images/phase-5/01-argocd-sync.png) + +4. **Health check** — `blue-green-health.sh` waits for the rollout and probes + `/health` through the mesh. +5. **Traffic switch** — `blue-green-switch.sh` flips the Istio VirtualService + weights (100/0) to the new color. The old color stays warm for instant + rollback: + +![VirtualService blue/green weights](../images/phase-5/03-blue-green-virtualservice.png) + +6. **Rollback** — any failure reverts the traffic weights and the Git tag bump + automatically (`rollback.sh`). + +### 5.3 Trigger it + +```bash +git checkout main && git pull +git commit --allow-empty -m "chore: trigger staging deploy" +git push origin main +gh run list --workflow=CD --limit 3 +``` + +> **Checkpoint:** CD run green; `kubectl get virtualservice backend -n staging +> -o yaml` shows 100/0 toward the new color; ArgoCD app **Synced + Healthy**. +> Production stays a Git mirror — see +> [showcase-staging-only.md](showcase-staging-only.md). + +--- + +## 6 · Security hardening + +**Goal:** defense-in-depth — nothing unsigned, privileged, or unbounded gets +admitted, and workloads run least-privilege. Full detail: [security.md](security.md). + +### 6.1 Admission control — Kyverno + +```bash +helm install kyverno kyverno/kyverno -n policy-system --create-namespace +kubectl apply -k k8s/kyverno/policies +``` + +Six ClusterPolicies enforce: no privileged containers, resource limits +required, non-root, no `:latest` tags, hardened securityContext, and **Cosign +signature verification** (the CI signature from step 4 is checked here). +Try to deploy a privileged pod and Kyverno blocks it: + +![Kyverno policy enforcement blocking a violating pod](../images/phase-6/01-kyverno-enforced.png) + +### 6.2 Network isolation + RBAC + +```bash +kubectl apply -k k8s/security +``` + +Default-deny ingress per namespace with explicit allows (ingress → backend, +backend → postgres/redis, monitoring scrapes), plus least-privilege RBAC roles: + +![NetworkPolicies and RBAC applied](../images/phase-6/02-networkpolicies-rba.png) + +### 6.3 Workload hardening + +The backend runs non-root (uid 1001), read-only root filesystem, all +capabilities dropped, seccomp `RuntimeDefault`, no service account token: + +![Pod securityContext hardened](../images/phase-6/03-pod-securitycontext.png) + +### 6.4 Secrets + +On EKS, secrets come from AWS Secrets Manager via External Secrets Operator +(IRSA) — nothing sensitive in Git. Locally, `k8s/base/secret.yaml` is a +placeholder template only. + +> **Checkpoint:** a privileged/unsigned/`:latest` pod is rejected at admission; +> `kubectl exec` into the backend shows uid 1001 and a read-only filesystem. + +--- + +## 7 · Monitoring & SRE + +**Goal:** metrics, logs, dashboards, and SLO-based alerting. Operational guide: +[`monitoring/README.md`](../monitoring/README.md) · concepts: [monitoring.md](monitoring.md) · SLOs: [sre.md](sre.md). + +### 7.1 Install the stack + +```bash +helm install kube-prometheus-stack prometheus-community/kube-prometheus-stack \ + -n monitoring --create-namespace -f monitoring/prometheus/values-eks.yaml # or values-dev.yaml +helm install loki grafana/loki -n monitoring -f monitoring/loki/values-eks.yaml +helm install promtail grafana/promtail -n monitoring -f monitoring/promtail/values.yaml +kubectl apply -k monitoring # ServiceMonitor + SLO rules + dashboards +``` + +### 7.2 Verify scraping + +Prometheus scrapes three sources: the app's `/metrics` (ServiceMonitor), Istio +mesh telemetry (the basis for SLOs), and cluster metrics +(node-exporter/kube-state-metrics). All targets must be **UP**: + +![Prometheus targets up](../images/phase-7/02-prometheus-targets.png) + +### 7.3 Dashboards + +Custom Grafana dashboards (provisioned as ConfigMaps, auto-imported by the +sidecar) show request rate, latency, cache hit ratio, and error budget: + +![Grafana Feature Flag Service dashboard](../images/phase-7/01-grafana-dashboard.png) + +### 7.4 SLO alerts + +Multi-window burn-rate rules (from [sre.md](sre.md): 99.9% availability, +p95 ≤ 300 ms) fire before the error budget is gone, not after: + +![Prometheus SLO alerts loaded](../images/phase-7/03-prometheus-alerts.png) + +> **Checkpoint:** all Prometheus targets UP, the app dashboard renders live +> data, `ALERTS` shows the burn-rate rules loaded. Blue/green note: mesh +> queries use `destination_canonical_service` so both colors count as one +> service. + +--- + +## 8 · Disaster recovery & resilience + +**Goal:** prove the platform survives pod, node, and data loss. Runbooks: +[disaster-recovery.md](disaster-recovery.md) · [resilience-testing.md](resilience-testing.md). + +### 8.1 Backups — Velero + +```bash +helm install velero vmware-tanzu/velero -n velero --create-namespace # see k8s/velero/install-notes.md +kubectl apply -f k8s/velero/schedules.yaml +``` + +Scheduled backups cover cluster resources + EBS snapshots to S3, with restore +procedures per scenario in [disaster-recovery.md](disaster-recovery.md). + +### 8.2 Failure injection + +`scripts/resilience-test.sh` automates the drills; [`tests/k6/`](../tests/k6/) +provides load. The core proof — kill the backend pod and watch Kubernetes +recreate it while the service keeps answering: + +```bash +kubectl delete pod -l app=backend -n staging # then watch it self-heal +``` + +![Pod deleted and automatically recovered](../images/phase-8/01-pod-recover.png) + +Other drills in the runbook: HPA under k6 load, database restart, ArgoCD +drift correction, and namespace restore from Velero. + +> **Checkpoint:** deleted pods return with zero failed requests during the +> window; a Velero restore brings a deleted namespace back. + +--- + +## 9 · Documentation & cost + +Everything you're reading is Phase 9: + +- [architecture.md](architecture.md) — system, app, and CI/CD diagrams +- [adr/](adr/) — why ArgoCD over Flux, EKS over ECS, Kustomize + Helm split, … +- [cost.md](cost.md) — monthly estimates per environment +- [diagrams/](diagrams/) — Mermaid sources (GitHub renders them inline) + +--- + +## Teardown + +EKS bills by the hour. When you're done: + +```bash +# Remove Helm releases and app namespaces first (ALB/EBS cleanup), then: +cd infrastructure/environments/staging +terraform destroy +minikube delete # local +``` + +The original AWS environment for this repo was validated live and then +decommissioned — the screenshots above are the captured evidence of the +working pipeline. diff --git a/images/phase-5/README.md b/images/phase-5/README.md index 55891a3..499c0be 100644 --- a/images/phase-5/README.md +++ b/images/phase-5/README.md @@ -1,34 +1,21 @@ # Phase 5 — CD & GitOps evidence -**Prerequisite:** Complete [docs/showcase-staging-only.md](../docs/showcase-staging-only.md) (terraform apply staging only, bootstrap EKS, push to `main`, CI + CD green). +Evidence captured from the live staging EKS pipeline before the AWS environment +was decommissioned. This set is **final** — no cluster access remains, so no +further screenshots will be added. -Capture screenshots after the staging CD pipeline runs end-to-end on EKS. +## Evidence -## Checklist +| File | Content | +|------|---------| +| `01-argocd-sync.png` | ArgoCD UI — `feature-flags-staging` Synced + Healthy on EKS | +| `03-blue-green-virtualservice.png` | Istio VirtualService showing the blue/green traffic split in staging | -| # | File | Content | -|---|------|---------| -| 1 | `01-cd-workflow-green.png` | GitHub Actions CD workflow success with GitOps commits | -| 2 | `02-argocd-staging-synced.png` | ArgoCD UI — `feature-flags-staging` Synced + Healthy | -| 3 | `03-blue-green-switch.png` | ArgoCD or terminal during traffic switch | -| 4 | `04-virtualservice-weights.png` | `kubectl get virtualservice backend -n staging -o yaml` showing 100/0 split | -| 5 | `05-rollback-complete.png` | Rollback job after simulated health failure (optional) | - -## Manual validation commands - -```bash -# After merge to main -gh run list --workflow=CD --limit 3 - -# Cluster -kubectl get pods,virtualservice -n staging -kubectl get application feature-flags-staging -n argocd - -# Health -curl -f "$STAGING_HEALTH_URL" -``` +Additional CD evidence lives in Git history itself: the GitOps bump commits on +`main` (e.g. `chore(staging): pin backend to sha-5137706`) and the green CI/CD +runs in the repository's Actions tab. ## Notes -- Dev (Minikube) CD is manual during Phase 5 — no screenshot required for phase closure. -- Production exists in Git as a staging mirror only — runtime is **off** and not scheduled, so no production screenshots. Reference if ever turned on: [docs/cd-production-promotion.md](../docs/cd-production-promotion.md). +- Dev (Minikube) CD is manual — no runtime evidence required. +- Production exists in Git as a staging mirror only — runtime was never lifted, so there is no production evidence. Reference if ever turned on: [docs/cd-production-promotion.md](../../docs/cd-production-promotion.md). diff --git a/infrastructure/environments/production/README.md b/infrastructure/environments/production/README.md index 7bce5d9..6d7d46a 100644 --- a/infrastructure/environments/production/README.md +++ b/infrastructure/environments/production/README.md @@ -26,7 +26,7 @@ When [staging](../staging/) modules or variables change, keep this directory ali ## When to apply -There is no scheduled date. Apply only if we explicitly decide to enable production runtime — see [docs/cd-production-promotion.md](../../docs/cd-production-promotion.md). +There is no scheduled date. Apply only if we explicitly decide to enable production runtime — see [docs/cd-production-promotion.md](../../../docs/cd-production-promotion.md). ```bash cd infrastructure/environments/production