Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 14 additions & 13 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ Entry point for AI agents working on this repository. Read this file first, then
| Dev cluster | Minikube (local) |
| Staging / Production | EKS on AWS |

**Current phase:** Phases 1–9 built as code/manifests/docs (staging-first; production off). **Next:** live EKS validation + screenshots (`terraform apply`, deploy, capture evidence). See [STATUS.md](STATUS.md).
**Current phase:** Phases 1–9 complete (staging-first; production off). Live validation was done on staging EKS; the AWS environment has since been **decommissioned** — the screenshots in `images/phase-N/` are the final evidence set and no new captures are possible. See [STATUS.md](STATUS.md).

---

Expand All @@ -35,7 +35,7 @@ devsecops-platform/
├── k8s/ # Kustomize manifests (base + overlays + blue-green + argocd)
├── .github/workflows/ # ci.yaml, cd.yaml (main)
├── scripts/ # blue/green CD automation (Phase 5)
├── docs/ # Placeholder — full docs in Phase 9
├── docs/ # Architecture, security, SRE, DR guides + ADRs + diagrams
├── images/phase-N/ # Screenshot evidence per phase
├── docker-compose.yml # Local dev stack (postgres + redis + backend)
├── PLAN.md # Full roadmap (1056 lines — use STATUS.md for current state)
Expand Down Expand Up @@ -114,19 +114,20 @@ Summary:

---

## Phase Boundaries — what's built vs deferred
## Phase Boundaries — what's built vs closed

Phases 1–9 are now **implemented as code/manifests/docs**. What remains is
**manual/live** work, intentionally deferred:
Phases 1–9 are **complete**. Staging EKS validation happened live (ArgoCD sync,
blue/green, Kyverno, dashboards, pod recovery) and the environment was then
decommissioned — do not plan work that requires cluster or AWS access:

| Status | Item |
|--------|------|
| Built | Phase 6 security (`k8s/security`, `k8s/kyverno`, `k8s/external-secrets`, securityContext, Cosign in `ci.yaml`) |
| Built | Phase 7 monitoring (`monitoring/`, SLO rules, dashboards, `docs/sre.md`) |
| Built | Phase 8 DR/resilience (`k8s/velero`, `tests/k6`, DR + resilience docs) |
| Built | Phase 9 docs (`docs/` guides, `docs/adr/`, `docs/diagrams/*.mmd`) |
| Deferred (manual) | Live EKS apply of the above, screenshots, diagram PNG exports, `terraform apply` |
| Undecided | Production **runtime** (EKS apply, CD promote, ECR push) — **off**; reference [docs/cd-production-promotion.md](docs/cd-production-promotion.md) |
| Done | Phase 6 security (`k8s/security`, `k8s/kyverno`, `k8s/external-secrets`, securityContext, Cosign in `ci.yaml`) — enforced live on staging |
| Done | Phase 7 monitoring (`monitoring/`, SLO rules, dashboards, `docs/sre.md`) — dashboards validated on EKS |
| Done | Phase 8 DR/resilience (`k8s/velero`, `tests/k6`, DR + resilience docs) — pod recovery evidenced |
| Done | Phase 9 docs (`docs/` guides, `docs/adr/`, `docs/diagrams/*.mmd`) |
| Closed | Runtime evidence — AWS decommissioned; `images/phase-N/` is the final set, no new screenshots |
| Off | Production **runtime** (EKS apply, CD promote, ECR push) — Git mirror only; reference [docs/cd-production-promotion.md](docs/cd-production-promotion.md) |

> When changing app manifests, keep Kyverno policies and securityContext intact
> (Phase 6 enforces them on staging/prod).
Expand Down Expand Up @@ -165,8 +166,8 @@ Phases 1–9 are now **implemented as code/manifests/docs**. What remains is

1. [TROUBLESHOOTING.md](TROUBLESHOOTING.md) — deploy runbook (Istio, Prisma, Minikube)
2. [k8s/README.md](k8s/README.md) — quick troubleshooting table
3. [PLAN.md](PLAN.md) — full phase requirements and acceptance criteria
4. [images/phase-N/](images/) — expected outcomes as screenshots
3. [docs/step-by-step.md](docs/step-by-step.md) — full build walkthrough with expected outcomes as screenshots
4. [PLAN.md](PLAN.md) — full phase requirements and acceptance criteria
5. [app/backend/README.md](app/backend/README.md) — NestJS structure and request lifecycle

---
Expand Down
54 changes: 41 additions & 13 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,8 @@

The application is a **Feature Flag Service** — manage feature toggles per environment with Redis-cached reads and PostgreSQL audit logging. Similar to LaunchDarkly or Unleash, it serves as the demo workload for the full DevSecOps pipeline.

> **Want to build this yourself?** Follow the [**Step-by-Step Guide**](docs/step-by-step.md) — a full walkthrough from empty AWS account to working GitOps pipeline, illustrated with evidence captured while this pipeline ran live.

**For AI agents:** start with [AGENTS.md](AGENTS.md) and [STATUS.md](STATUS.md). Deploy issues: [TROUBLESHOOTING.md](TROUBLESHOOTING.md).

---
Expand All @@ -32,15 +34,16 @@ The application is a **Feature Flag Service** — manage feature toggles per env
| 2 | Kubernetes base (Minikube + Istio) | Done |
| 3 | Feature Flag Service + K8s manifests | **Done** |
| 4 | CI Pipeline (GitHub Actions, SonarCloud, ECR) | **Done** |
| 5 | CD Pipeline & GitOps (ArgoCD, staging-first) | **Implemented** — validate on EKS |
| 6 | Security hardening (Kyverno, Cosign, NetworkPolicy, External Secrets) | **Implemented** |
| 7 | Monitoring & SRE (Prometheus, Grafana, Loki, SLOs) | **Implemented** |
| 8 | DR & resilience (Velero, k6) | **Implemented** |
| 9 | Docs, ADRs, cost | **Implemented** |
| 5 | CD Pipeline & GitOps (ArgoCD, staging-first) | **Done** — validated on staging EKS |
| 6 | Security hardening (Kyverno, Cosign, NetworkPolicy, External Secrets) | **Done** |
| 7 | Monitoring & SRE (Prometheus, Grafana, Loki, SLOs) | **Done** |
| 8 | DR & resilience (Velero, k6) | **Done** |
| 9 | Docs, ADRs, cost | **Done** |

All manifests/config/docs are committed. Live validation (EKS deploy + screenshots)
is the remaining step — staging is the only counted runtime; production is a Git
mirror that stays **off**. Full tracker: [STATUS.md](STATUS.md) · Roadmap: [PLAN.md](PLAN.md)
All phases are complete. The pipeline was validated live on staging EKS — evidence
is in `images/phase-N/`. The AWS environment has since been decommissioned; staging
was the only counted runtime, and production remains a Git mirror that stays **off**.
Full tracker: [STATUS.md](STATUS.md) · Roadmap: [PLAN.md](PLAN.md)

---

Expand Down Expand Up @@ -167,19 +170,44 @@ docs/ Full documentation (Phase 9)

## Documentation

**Start here → [Step-by-Step Guide](docs/step-by-step.md)** — rebuild the whole platform phase by phase, with real evidence at every step.

### Guides & concepts

| Topic | Doc |
|-------|-----|
| Architecture & diagrams | [docs/architecture.md](docs/architecture.md), [docs/diagrams/](docs/diagrams/) |
| Deployment (Minikube + EKS) | [docs/deployment-guide.md](docs/deployment-guide.md), [k8s/README.md](k8s/README.md) |
| **Build it yourself (with screenshots)** | [docs/step-by-step.md](docs/step-by-step.md) |
| Architecture & diagrams | [docs/architecture.md](docs/architecture.md) · [docs/diagrams/](docs/diagrams/) |
| Deployment quick reference (Minikube + EKS) | [docs/deployment-guide.md](docs/deployment-guide.md) · [k8s/README.md](k8s/README.md) |
| Security hardening | [docs/security.md](docs/security.md) |
| Monitoring & observability | [docs/monitoring.md](docs/monitoring.md), [monitoring/README.md](monitoring/README.md) |
| Monitoring & observability | [docs/monitoring.md](docs/monitoring.md) · [monitoring/README.md](monitoring/README.md) |
| SRE — SLI/SLO/error budget | [docs/sre.md](docs/sre.md) |
| Disaster recovery | [docs/disaster-recovery.md](docs/disaster-recovery.md) |
| Resilience testing | [docs/resilience-testing.md](docs/resilience-testing.md) |
| SonarCloud quality gate | [docs/sonarqube.md](docs/sonarqube.md) |
| Cost estimates | [docs/cost.md](docs/cost.md) |
| Architecture Decision Records | [docs/adr/](docs/adr/) |

### Runbooks

| Topic | Doc |
|-------|-----|
| Disaster recovery | [docs/disaster-recovery.md](docs/disaster-recovery.md) |
| Resilience testing | [docs/resilience-testing.md](docs/resilience-testing.md) |
| Staging-only showcase (re-lift the runtime) | [docs/showcase-staging-only.md](docs/showcase-staging-only.md) |
| Production promotion (reference — off) | [docs/cd-production-promotion.md](docs/cd-production-promotion.md) |
| EKS staging bootstrap | [k8s/argocd/install-notes.md](k8s/argocd/install-notes.md) |
| Troubleshooting (Istio, Prisma, Minikube, CD) | [TROUBLESHOOTING.md](TROUBLESHOOTING.md) |

### Project meta

| Topic | Doc |
|-------|-----|
| Phase tracker | [STATUS.md](STATUS.md) |
| Original roadmap (historical) | [PLAN.md](PLAN.md) |
| AI agent entry point | [AGENTS.md](AGENTS.md) |
| Contributing | [CONTRIBUTE.md](CONTRIBUTE.md) |
| Backend internals | [app/backend/README.md](app/backend/README.md) |
| Evidence screenshots | [images/](images/) — `phase-N/` per phase |

---

## Contributing
Expand Down
52 changes: 27 additions & 25 deletions STATUS.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
# Project Status

Phase tracker for the DevSecOps platform. Updated after Phase 5 implementation.
Phase tracker for the DevSecOps platform. All phases complete.

**Active work:** Phases 6–9 built as code/manifests/docs; pending live EKS validation + screenshots
**Active work:** None — Phases 1–9 complete and validated on staging EKS. The AWS environment has been decommissioned; the evidence in `images/phase-N/` is final.
**Agent entry point:** [AGENTS.md](AGENTS.md)
**Deploy runbook:** [TROUBLESHOOTING.md](TROUBLESHOOTING.md)
**Full roadmap:** [PLAN.md](PLAN.md)
Expand All @@ -17,30 +17,32 @@ Phase tracker for the DevSecOps platform. Updated after Phase 5 implementation.
| 1 | Terraform & AWS (VPC, EKS, ECR, IAM) | Done | `infrastructure/`, `images/phase-1/` |
| 2 | Kubernetes base (Minikube: Istio, ArgoCD, Ingress) | Done (Minikube) | `k8s/namespaces/`, `images/phase-2/`, install notes in `k8s/argocd/`, `k8s/istio/` |
| 3 | Feature Flag Service + K8s manifests | **Done** | `app/backend/`, `k8s/`, `images/phase-3/` |
| 4 | CI Pipeline (GitHub Actions, SonarCloud, ECR) | **Done** | `.github/workflows/`, `sonar-project.properties`, `images/phase-4/` |
| 5 | CD & GitOps (ArgoCD sync, blue/green automation) | **Implemented** (staging-first) | `.github/workflows/cd.yaml`, `scripts/`, `images/phase-5/` |
| 6 | Security hardening (Kyverno, Cosign, NetworkPolicy, External Secrets) | **Implemented** | `k8s/security/`, `k8s/kyverno/`, `k8s/external-secrets/`, securityContext in deployments, Cosign in `ci.yaml` |
| 7 | Monitoring & SRE (Prometheus, Grafana, Loki) | **Implemented** | `monitoring/`, `docs/sre.md` |
| 8 | DR & resilience (Velero, k6) | **Implemented** | `k8s/velero/`, `tests/k6/`, `scripts/resilience-test.sh`, `docs/disaster-recovery.md`, `docs/resilience-testing.md` |
| 9 | Documentation, ADRs, diagrams, cost docs | **Implemented** | `docs/` (architecture, deployment-guide, security, monitoring, sonarqube, cost), `docs/adr/`, `docs/diagrams/` |

> **Phases 6–9 are built as code/manifests/docs.** Live validation (EKS deploy,
> screenshots, `terraform apply`) is the remaining manual step and is intentionally
> deferred. All new manifests pass `kubectl kustomize` + `yamllint`.
| 4 | CI Pipeline (GitHub Actions, SonarCloud, ECR) | **Done** | `.github/workflows/`, `sonar-project.properties` |
| 5 | CD & GitOps (ArgoCD sync, blue/green automation) | **Done** (staging-first) | `.github/workflows/cd.yaml`, `scripts/`, `images/phase-5/` |
| 6 | Security hardening (Kyverno, Cosign, NetworkPolicy, External Secrets) | **Done** | `k8s/security/`, `k8s/kyverno/`, `k8s/external-secrets/`, securityContext in deployments, Cosign in `ci.yaml`, `images/phase-6/` |
| 7 | Monitoring & SRE (Prometheus, Grafana, Loki) | **Done** | `monitoring/`, `docs/sre.md`, `images/phase-7/` |
| 8 | DR & resilience (Velero, k6) | **Done** | `k8s/velero/`, `tests/k6/`, `scripts/resilience-test.sh`, `docs/disaster-recovery.md`, `docs/resilience-testing.md`, `images/phase-8/` |
| 9 | Documentation, ADRs, diagrams, cost docs | **Done** | `docs/` (architecture, deployment-guide, security, monitoring, sonarqube, cost), `docs/adr/`, `docs/diagrams/` |

> **Phases 5–8 were validated live on staging EKS** (ArgoCD sync, blue/green,
> Kyverno enforcement, dashboards, pod recovery) before the AWS environment was
> decommissioned. The screenshots in `images/phase-N/` are the final evidence set
> — no further captures are possible. All manifests pass `kubectl kustomize` +
> `yamllint`.

---

## Showcase mode — Git vs runtime

**Scope:** Staging is the only environment with a working pipeline right now. Production is **off** — kept in Git as a mirror of staging only. Whether/when to lift production runtime is an open decision, not a committed phase.
**Scope:** Staging was the only environment with a working pipeline. Production is **off** — kept in Git as a mirror of staging only. The staging AWS runtime has since been decommissioned; the Git manifests and captured evidence document the working state.

| Layer | Staging | Production |
|-------|---------|------------|
| **Git** | Active source | **Mirror** of staging — kept aligned |
| **Runtime** | EKS + CD on every `main` push | **Off** — not lifted; next steps TBD |
| **Runtime** | Validated on EKS + CD, since decommissioned | **Off** — never lifted |
| **Pipeline (CI/CD)** | Counted as the working pipeline | Excluded — no build, push, or deploy |

**Before screenshots:** follow [docs/showcase-staging-only.md](docs/showcase-staging-only.md) (terraform apply **staging only**, bootstrap EKS staging, push to `main`).
**To re-lift staging:** follow [docs/showcase-staging-only.md](docs/showcase-staging-only.md) (terraform apply **staging only**, bootstrap EKS staging, push to `main`).

---

Expand All @@ -67,23 +69,23 @@ Automatic CD targets **staging EKS only** — staging is the only counted pipeli

Optional: secret `CD_BOT_TOKEN` if branch protection blocks `GITHUB_TOKEN`.

### Pending validation
### Validation (complete)

- [ ] One-time EKS staging bootstrap per install notes
- [ ] `terraform apply` in staging for EKS access entry
- [ ] Merge to `main` → CI → CD green
- [ ] Screenshots in [`images/phase-5/`](images/phase-5/) per README checklist
- [x] One-time EKS staging bootstrap per install notes
- [x] `terraform apply` in staging for EKS access entry
- [x] Merge to `main` → CI → CD green
- [x] Evidence captured in [`images/phase-5/`](images/phase-5/) (ArgoCD sync, blue/green VirtualService)

---

## Blockers / Deferred

| Item | Reason | Target |
|------|--------|--------|
| Production runtime (EKS, CD) | Off for now — Git mirror only; staging is the working pipeline | TBD — decide next steps; reference [docs/cd-production-promotion.md](docs/cd-production-promotion.md) |
| Production runtime (EKS, CD) | Off — Git mirror only; staging was the validated pipeline | Not planned; reference [docs/cd-production-promotion.md](docs/cd-production-promotion.md) |
| Dev automated CD | Minikube not reachable from GitHub-hosted runners | Manual dev deploy |
| Diagram PNG exports | Mermaid sources committed (`docs/diagrams/*.mmd`); render deferred with screenshots | Manual |
| Live EKS validation of Phases 6–9 | Manifests/Helm values committed; not yet applied to a cluster | Manual |
| Diagram PNG exports | Mermaid sources committed (`docs/diagrams/*.mmd`); GitHub renders them inline, PNGs only needed for slides/PDFs | Optional |
| Additional runtime evidence | AWS environment decommissioned — no cluster access; existing `images/phase-N/` screenshots are final | Closed |

---

Expand All @@ -92,5 +94,5 @@ Optional: secret `CD_BOT_TOKEN` if branch protection blocks `GITHUB_TOKEN`.
| Environment | Kubernetes (runtime) | Git manifests | CD |
|-------------|----------------------|---------------|-----|
| `dev` | Minikube (local) | `overlays/dev/` | Manual |
| `staging` | EKS (AWS) | `overlays/staging/` | **Automatic** |
| `production` | Off — not lifted | `overlays/production/` mirror | None (excluded; next steps TBD) |
| `staging` | EKS (AWS) — validated, since decommissioned | `overlays/staging/` | **Automatic** (when lifted) |
| `production` | Off — never lifted | `overlays/production/` mirror | None (excluded) |
1 change: 0 additions & 1 deletion TROUBLESHOOTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -481,4 +481,3 @@ When a new deploy issue is discovered:
1. Document the **symptom**, **cause**, and **fix** in this file
2. Add a row to the [Quick Symptom Index](#quick-symptom-index)
3. Update [AGENTS.md](AGENTS.md) if it affects agent phase boundaries
4. Capture evidence in `images/phase-N/` if relevant
2 changes: 1 addition & 1 deletion docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -103,7 +103,7 @@ sequenceDiagram

Prometheus scrapes app `/metrics` + Istio mesh + cluster; Grafana dashboards;
Loki/Promtail logs; SLOs with multi-window burn-rate alerts. See
[../docs/sre.md](sre.md) and [../monitoring/README.md](../monitoring/README.md).
[sre.md](sre.md) and [monitoring/README.md](../monitoring/README.md).

## Resilience (Phase 8)

Expand Down
14 changes: 12 additions & 2 deletions docs/deployment-guide.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,9 @@
# Deployment Guide

Covers local **Minikube** (dev) and cloud **EKS** (staging). Production runtime is
off during showcase — see [showcase-staging-only.md](showcase-staging-only.md).
Quick reference for deploying to local **Minikube** (dev) and cloud **EKS**
(staging). First time here? The narrative walkthrough with screenshots is
[step-by-step.md](step-by-step.md). Production runtime is off during showcase —
see [showcase-staging-only.md](showcase-staging-only.md).

## Prerequisites

Expand Down Expand Up @@ -37,6 +39,10 @@ kubectl port-forward svc/backend -n dev 3000:80
curl http://localhost:3000/health
```

Expected result — all pods `2/2` (app + Istio sidecar):

![Pods running in dev namespace](../images/phase-3/07-k8s-pods-dev.png)

Gotchas (Prisma v7, Istio TCP, image caching, LimitRange): [../TROUBLESHOOTING.md](../TROUBLESHOOTING.md).

Optional add-ons on dev:
Expand Down Expand Up @@ -103,6 +109,10 @@ gh run list --workflow=CD --limit 3
CI builds/scans/signs/pushes to ECR; CD bumps the idle blue/green color, ArgoCD
syncs, health check runs, traffic switches. Rollback is automatic on failure.

The app answering on EKS after a green pipeline:

![App responding on EKS](../images/phase-3/08-eks-app.png)

---

## C. Production
Expand Down
5 changes: 3 additions & 2 deletions docs/diagrams/README.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,8 @@
# Diagrams

Source-controlled Mermaid sources. PNG exports are deferred (same as screenshots)
— render them when preparing the portfolio:
Source-controlled Mermaid sources. GitHub renders the inline Mermaid in README
and architecture.md natively; PNG exports are optional (slide decks / PDFs only)
and need no AWS access — render on demand:

```bash
# Mermaid CLI
Expand Down
6 changes: 6 additions & 0 deletions docs/monitoring.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,12 +23,18 @@ overview.
across dev and EKS, independent of app code).
- **Cluster:** node-exporter + kube-state-metrics (bundled).

All targets reporting UP on the live staging cluster:

![Prometheus targets up](../images/phase-7/02-prometheus-targets.png)

## Dashboards

Custom: **Feature Flag Service** (`feature-flags-app`) and **SLO — Error Budget**
(`slo-error-budget`), provisioned as `grafana_dashboard` ConfigMaps. Community:
Kubernetes (7249), Istio Mesh (7639), Node Exporter (1860).

![Grafana Feature Flag Service dashboard](../images/phase-7/01-grafana-dashboard.png)

## SLOs & alerting

SLIs/SLOs, error-budget policy and multi-window burn-rate alerts:
Expand Down
10 changes: 7 additions & 3 deletions docs/resilience-testing.md
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,11 @@ kubectl rollout restart deployment/backend-blue -n staging # pick up new value

See [disaster-recovery.md → Procedure 1](disaster-recovery.md#procedure-1--restore-a-namespace).

## Recording results
## Recorded results

Capture evidence (terminal output, Grafana HPA panel, ArgoCD rollback) under
`images/phase-8/` when running the live drills.
The live drill was run on staging EKS before the AWS environment was
decommissioned — the deleted backend pod was recreated automatically with no
downtime. That evidence set is final; the drills above remain runnable on any
future cluster.

![Pod deleted and automatically recovered](../images/phase-8/01-pod-recover.png)
Loading
Loading