diff --git a/docs/disaster-recovery.md b/docs/disaster-recovery.md new file mode 100644 index 0000000..336e2d9 --- /dev/null +++ b/docs/disaster-recovery.md @@ -0,0 +1,71 @@ +# Disaster Recovery Runbook (Phase 8) + +How the platform is backed up and how to recover from data, namespace, or state +loss. Backups use **Velero** (cluster + EBS snapshots) and AWS-native snapshots. + +## Backup inventory + +| What | Tool | Schedule | Retention | Location | +|------|------|----------|-----------|----------| +| All namespaces (manifests + PVs) | Velero | daily 01:00 | 7 days | S3 `mck21-devsecops-velero-backups` | +| Production namespace | Velero | every 6h | 14 days | S3 | +| PV data (postgres/redis) | Velero fs-backup | daily 01:30 | 7 days | S3 | +| Terraform state | S3 versioning | on write | versioned | S3 `devsecops-tfstate-125156866917` | + +Install + schedule definitions: [../k8s/velero/install-notes.md](../k8s/velero/install-notes.md), +[../k8s/velero/schedules.yaml](../k8s/velero/schedules.yaml). + +## RTO / RPO targets + +| Scenario | RPO (data loss) | RTO (time to restore) | +|----------|-----------------|------------------------| +| Production namespace loss | ≤ 6h | ≤ 30 min | +| Full cluster loss | ≤ 24h | ≤ 2h (terraform apply + restore) | +| PostgreSQL corruption | ≤ 6h (or 5 min with RDS PITR) | ≤ 30 min | + +## Procedure 1 — Restore a namespace + +```bash +velero backup get +velero restore create prod-restore --from-backup production-6h- +velero restore describe prod-restore --details +kubectl get pods -n production +``` + +Then let ArgoCD re-sync to reconcile any drift: +```bash +argocd app sync feature-flags-production # only if production runtime is on +``` + +## Procedure 2 — PostgreSQL point-in-time + +- **In-cluster postgres (dev/showcase):** restore the PV via the `daily-volumes` + Velero backup, then `kubectl rollout restart deployment/postgres`. +- **Managed RDS (recommended for prod runtime):** use RDS automated snapshots / + PITR: `aws rds restore-db-instance-to-point-in-time ...` then repoint + `DATABASE_URL` (via External Secrets / AWS Secrets Manager). + +## Procedure 3 — Full cluster rebuild + +1. `terraform apply` the environment (recreates VPC/EKS/ECR). +2. Bootstrap the platform (Istio, Ingress, cert-manager, ArgoCD) per + [../k8s/argocd/install-notes.md](../k8s/argocd/install-notes.md). +3. Reinstall Velero (same S3 location auto-discovers backups). +4. `velero restore create --from-backup daily-all-namespaces-`. +5. ArgoCD reconciles app state from Git. + +## Procedure 4 — Terraform state corruption + +- State is versioned in S3 with `use_lockfile = true`. Recover a prior version: + ```bash + aws s3api list-object-versions --bucket devsecops-tfstate-125156866917 \ + --prefix staging/terraform.tfstate + aws s3api get-object --bucket devsecops-tfstate-125156866917 \ + --key staging/terraform.tfstate --version-id terraform.tfstate + ``` +- If a lock is stuck: `terraform force-unlock `. + +## Verification + +DR is only real if tested — see the namespace-restore and secret-rotation drills +in [resilience-testing.md](resilience-testing.md). Run a restore drill quarterly. diff --git a/docs/resilience-testing.md b/docs/resilience-testing.md new file mode 100644 index 0000000..6566e28 --- /dev/null +++ b/docs/resilience-testing.md @@ -0,0 +1,72 @@ +# Resilience Testing Runbook (Phase 8) + +Failure-injection scenarios that prove the platform self-heals. Run on Minikube +first, then repeat the cluster-only tests on EKS. Helper: +[../scripts/resilience-test.sh](../scripts/resilience-test.sh); load: +[../tests/k6/](../tests/k6/). + +## Scenarios + +| # | Test | Method | Run on | Expected result | +|---|------|--------|--------|-----------------| +| 1 | Pod crash | `kubectl delete pod` | Minikube + EKS | ReplicaSet recreates pod; no 5xx (other replica serves) | +| 2 | Failed deployment | deploy broken image | Minikube + EKS | Probes fail → CD/ArgoCD rollback; traffic stays on healthy color | +| 3 | CPU stress | k6 `stress-hpa.js` | Minikube + EKS | HPA scales out to maxReplicas, then back in | +| 4 | Node drain | `kubectl drain` | EKS only | Pods rescheduled to other nodes | +| 5 | Secret rotation | rotate AWS secret | EKS only | External Secrets re-syncs `backend-secrets` within refresh interval | +| 6 | Namespace restore | Velero restore | EKS only | Namespace fully recovered (see disaster-recovery.md) | + +## 1. Pod crash + +```bash +scripts/resilience-test.sh pod-crash dev +# Expect: deleted pod replaced within seconds, READY count restored. +``` + +## 2. Failed deployment → rollback + +```bash +scripts/resilience-test.sh bad-deploy dev backend +# Expect: rollout stuck (ImagePullBackOff / failing readiness). +kubectl rollout undo deployment/backend -n dev +``` +On staging, the CD pipeline's health check fails and triggers +`scripts/rollback.sh` (blue/green traffic reverts to the previous color). + +## 3. CPU stress → HPA scale-out + +```bash +kubectl get hpa -n dev -w +BASE_URL=http://localhost:3000 k6 run tests/k6/stress-hpa.js +# Expect: current replicas climb toward maxReplicas (10) as CPU passes 70%, +# then scale back down after load stops (stabilization window). +``` + +## 4. Node drain (EKS) + +```bash +kubectl get nodes +kubectl drain --ignore-daemonsets --delete-emptydir-data +kubectl get pods -o wide -n staging # pods rescheduled onto other nodes +kubectl uncordon +``` + +## 5. Secret rotation (EKS) + +```bash +aws secretsmanager put-secret-value \ + --secret-id mck21-devsecops/staging/backend \ + --secret-string '{"DATABASE_URL":"...","REDIS_URL":"..."}' +# Within refreshInterval (1h), ESO updates backend-secrets: +kubectl get externalsecret backend-secrets -n staging +kubectl rollout restart deployment/backend-blue -n staging # pick up new value +``` + +## 6. Namespace restore (EKS) + +See [disaster-recovery.md → Procedure 1](disaster-recovery.md#procedure-1--restore-a-namespace). + +## Recording results + +Capture evidence (terminal output, Grafana HPA panel, ArgoCD rollback) under +`images/phase-8/` when running the live drills. diff --git a/k8s/velero/install-notes.md b/k8s/velero/install-notes.md new file mode 100644 index 0000000..0026ee9 --- /dev/null +++ b/k8s/velero/install-notes.md @@ -0,0 +1,56 @@ +# Velero — install & backup/restore (Phase 8) + +Cluster backup/restore for EKS, backed by S3 + EBS volume snapshots. Auth via +IRSA (no static keys). + +## Prerequisites (Terraform) + +- S3 bucket `mck21-devsecops-velero-backups`. +- IRSA role `mck21-devsecops-velero` with S3 read/write on that bucket and + `ec2:CreateSnapshot` / `DescribeSnapshots` / `CreateTags` for EBS snapshots. + +## Install (Helm) + +```bash +helm repo add vmware-tanzu https://vmware-tanzu.github.io/helm-charts +helm repo update + +helm install velero vmware-tanzu/velero \ + --namespace velero --create-namespace \ + --set configuration.backupStorageLocation[0].name=default \ + --set configuration.backupStorageLocation[0].provider=aws \ + --set configuration.backupStorageLocation[0].bucket=mck21-devsecops-velero-backups \ + --set configuration.backupStorageLocation[0].config.region=us-east-1 \ + --set configuration.volumeSnapshotLocation[0].name=default \ + --set configuration.volumeSnapshotLocation[0].provider=aws \ + --set configuration.volumeSnapshotLocation[0].config.region=us-east-1 \ + --set initContainers[0].name=velero-plugin-for-aws \ + --set initContainers[0].image=velero/velero-plugin-for-aws:v1.10.0 \ + --set initContainers[0].volumeMounts[0].mountPath=/target \ + --set initContainers[0].volumeMounts[0].name=plugins \ + --set serviceAccount.server.annotations."eks\.amazonaws\.com/role-arn"=arn:aws:iam::125156866917:role/mck21-devsecops-velero +``` + +## Apply schedules + +```bash +kubectl apply -f k8s/velero/schedules.yaml +velero schedule get +``` + +| Schedule | Cron | Scope | Retention | +|----------|------|-------|-----------| +| `daily-all-namespaces` | `0 1 * * *` | all namespaces | 7 days | +| `production-6h` | `0 */6 * * *` | production | 14 days | +| `daily-volumes` | `30 1 * * *` | staging, production PVs | 7 days | + +## Restore drill + +```bash +velero backup get +velero restore create --from-backup daily-all-namespaces- +velero restore describe +``` + +Full procedures (namespace restore, PostgreSQL point-in-time, tfstate recovery) +are in [../../docs/disaster-recovery.md](../../docs/disaster-recovery.md). diff --git a/k8s/velero/schedules.yaml b/k8s/velero/schedules.yaml new file mode 100644 index 0000000..ca712ca --- /dev/null +++ b/k8s/velero/schedules.yaml @@ -0,0 +1,55 @@ +# Velero backup schedules (EKS). Backups stored in S3, volumes via EBS snapshots. +# Requires Velero installed with the AWS plugin — see install-notes.md. +--- +# All namespaces, daily, 7-day retention. +apiVersion: velero.io/v1 +kind: Schedule +metadata: + name: daily-all-namespaces + namespace: velero +spec: + schedule: "0 1 * * *" + template: + includedNamespaces: + - "*" + excludedResources: + - events + - events.events.k8s.io + snapshotVolumes: true + storageLocation: default + volumeSnapshotLocations: + - default + ttl: 168h0m0s +--- +# Production namespace, every 6 hours, 14-day retention. +apiVersion: velero.io/v1 +kind: Schedule +metadata: + name: production-6h + namespace: velero +spec: + schedule: "0 */6 * * *" + template: + includedNamespaces: + - production + snapshotVolumes: true + storageLocation: default + volumeSnapshotLocations: + - default + ttl: 336h0m0s +--- +# Persistent volumes (postgres/redis data), daily fs-backup, 7-day retention. +apiVersion: velero.io/v1 +kind: Schedule +metadata: + name: daily-volumes + namespace: velero +spec: + schedule: "30 1 * * *" + template: + includedNamespaces: + - staging + - production + defaultVolumesToFsBackup: true + storageLocation: default + ttl: 168h0m0s diff --git a/scripts/resilience-test.sh b/scripts/resilience-test.sh new file mode 100755 index 0000000..ed65e02 --- /dev/null +++ b/scripts/resilience-test.sh @@ -0,0 +1,44 @@ +#!/usr/bin/env bash +# Resilience drill helper — inject a failure and observe recovery. +# Usage: scripts/resilience-test.sh [namespace] +# scenarios: pod-crash | bad-deploy | cpu-stress | help +# Defaults to namespace "dev". Read-only on the cluster except the chosen action. +set -euo pipefail + +SCENARIO="${1:-help}" +NS="${2:-dev}" + +usage() { + cat <<'EOF' +Resilience drill helper + pod-crash Delete a backend pod; expect auto-restart, no downtime + bad-deploy Set an invalid image; expect probes to fail (roll back via GitOps/CD) + cpu-stress Print the k6 command that drives HPA scale-out +EOF +} + +case "$SCENARIO" in + pod-crash) + echo "[*] Pods before:"; kubectl get pods -n "$NS" -l app=backend + POD="$(kubectl get pods -n "$NS" -l app=backend -o jsonpath='{.items[0].metadata.name}')" + echo "[*] Deleting pod $POD ..." + kubectl delete pod "$POD" -n "$NS" + echo "[*] Watch recovery (Ctrl-C to stop):" + kubectl get pods -n "$NS" -l app=backend -w + ;; + bad-deploy) + DEPLOY="${3:-backend}" + echo "[*] Setting invalid image on deployment/$DEPLOY in $NS ..." + kubectl set image "deployment/$DEPLOY" backend=backend:does-not-exist -n "$NS" + echo "[*] Watch rollout fail (probes/imagepull). Roll back with:" + echo " kubectl rollout undo deployment/$DEPLOY -n $NS" + kubectl rollout status "deployment/$DEPLOY" -n "$NS" --timeout=90s || true + ;; + cpu-stress) + echo "[*] In one shell: kubectl get hpa -n $NS -w" + echo "[*] In another: BASE_URL=http://localhost:3000 k6 run tests/k6/stress-hpa.js" + ;; + help|*) + usage + ;; +esac diff --git a/tests/k6/README.md b/tests/k6/README.md new file mode 100644 index 0000000..0aecaf2 --- /dev/null +++ b/tests/k6/README.md @@ -0,0 +1,45 @@ +# k6 Load & Resilience Tests (Phase 8) + +Load tests for the Feature Flag Service, used to validate SLOs and trigger HPA +scale-out during resilience drills. + +## Install k6 + +```bash +brew install k6 # macOS +# or: docker run --rm -i grafana/k6 run - < tests/k6/load-flags-read.js +``` + +## Tests + +| Script | Purpose | Pass criteria | +|--------|---------|---------------| +| `load-flags-read.js` | Steady 200 rps on the cache-hot read path | P95 < 300ms, errors < 0.1% (thresholds enforced) | +| `stress-hpa.js` | Ramp load to push CPU past 70% | HPA scales replicas up, then back down | + +## Run against dev (Minikube) + +```bash +kubectl port-forward svc/backend -n dev 3000:80 & +# seed a flag first if needed: +curl -X POST localhost:3000/api/flags -H 'content-type: application/json' \ + -d '{"key":"checkout-v2","description":"demo","environments":{"production":true}}' + +BASE_URL=http://localhost:3000 FLAG_KEY=checkout-v2 k6 run tests/k6/load-flags-read.js +``` + +## Run against staging (EKS) + +```bash +BASE_URL=https://flags.staging.example.com k6 run tests/k6/load-flags-read.js +``` + +## Observe HPA during stress + +```bash +kubectl get hpa -n dev -w # or -n staging +BASE_URL=http://localhost:3000 k6 run tests/k6/stress-hpa.js +``` + +Scenarios and expected outcomes are tracked in +[../../docs/resilience-testing.md](../../docs/resilience-testing.md). diff --git a/tests/k6/load-flags-read.js b/tests/k6/load-flags-read.js new file mode 100644 index 0000000..aefcfcd --- /dev/null +++ b/tests/k6/load-flags-read.js @@ -0,0 +1,35 @@ +// k6 — steady-state read load on the cache-hot path GET /api/flags/{key}. +// Validates SLOs (P95 <= 300ms, error rate < 0.1%) under normal traffic. +// +// BASE_URL=http://localhost:3000 FLAG_KEY=checkout-v2 k6 run tests/k6/load-flags-read.js +import http from 'k6/http'; +import { check, sleep } from 'k6'; + +const BASE_URL = __ENV.BASE_URL || 'http://localhost:3000'; +const FLAG_KEY = __ENV.FLAG_KEY || 'checkout-v2'; + +export const options = { + scenarios: { + steady: { + executor: 'constant-arrival-rate', + rate: 200, // requests per second + timeUnit: '1s', + duration: '5m', + preAllocatedVUs: 50, + maxVUs: 200, + }, + }, + thresholds: { + http_req_duration: ['p(95)<300'], // SLO: P95 <= 300ms + http_req_failed: ['rate<0.001'], // SLO: error rate < 0.1% + }, +}; + +export default function () { + const res = http.get(`${BASE_URL}/api/flags/${FLAG_KEY}`); + check(res, { + 'status is 200': (r) => r.status === 200, + 'has body': (r) => r.body && r.body.length > 0, + }); + sleep(0.1); +} diff --git a/tests/k6/stress-hpa.js b/tests/k6/stress-hpa.js new file mode 100644 index 0000000..850ff90 --- /dev/null +++ b/tests/k6/stress-hpa.js @@ -0,0 +1,33 @@ +// k6 — ramping load designed to drive CPU up and trigger the HPA scale-out. +// Watch `kubectl get hpa -n -w` and the Grafana "HPA replicas" panel. +// +// BASE_URL=http://localhost:3000 k6 run tests/k6/stress-hpa.js +import http from 'k6/http'; +import { check } from 'k6'; + +const BASE_URL = __ENV.BASE_URL || 'http://localhost:3000'; +const FLAG_KEY = __ENV.FLAG_KEY || 'checkout-v2'; + +export const options = { + scenarios: { + ramp: { + executor: 'ramping-vus', + startVUs: 0, + stages: [ + { duration: '2m', target: 100 }, // warm up + { duration: '3m', target: 400 }, // push CPU past the 70% HPA target + { duration: '3m', target: 400 }, // hold to let HPA stabilise replicas + { duration: '2m', target: 0 }, // ramp down -> observe scale-in + ], + }, + }, +}; + +export default function () { + // Mix cached reads with an occasional list to add DB/compute pressure. + const res = http.get(`${BASE_URL}/api/flags/${FLAG_KEY}`); + check(res, { 'status is 200': (r) => r.status === 200 }); + if (Math.random() < 0.1) { + http.get(`${BASE_URL}/api/flags?env=production`); + } +}