feat: add phase 8 disaster recovery and resilience testing

docs/disaster-recovery.md

@@ -0,0 +1,71 @@
+# Disaster Recovery Runbook (Phase 8)
+
+How the platform is backed up and how to recover from data, namespace, or state
+loss. Backups use **Velero** (cluster + EBS snapshots) and AWS-native snapshots.
+
+## Backup inventory
+
+| What | Tool | Schedule | Retention | Location |
+|------|------|----------|-----------|----------|
+| All namespaces (manifests + PVs) | Velero | daily 01:00 | 7 days | S3 `mck21-devsecops-velero-backups` |
+| Production namespace | Velero | every 6h | 14 days | S3 |
+| PV data (postgres/redis) | Velero fs-backup | daily 01:30 | 7 days | S3 |
+| Terraform state | S3 versioning | on write | versioned | S3 `devsecops-tfstate-125156866917` |
+
+Install + schedule definitions: [../k8s/velero/install-notes.md](../k8s/velero/install-notes.md),
+[../k8s/velero/schedules.yaml](../k8s/velero/schedules.yaml).
+
+## RTO / RPO targets
+
+| Scenario | RPO (data loss) | RTO (time to restore) |
+|----------|-----------------|------------------------|
+| Production namespace loss | ≤ 6h | ≤ 30 min |
+| Full cluster loss | ≤ 24h | ≤ 2h (terraform apply + restore) |
+| PostgreSQL corruption | ≤ 6h (or 5 min with RDS PITR) | ≤ 30 min |
+
+## Procedure 1 — Restore a namespace
+
+```bash
+velero backup get
+velero restore create prod-restore --from-backup production-6h-<timestamp>
+velero restore describe prod-restore --details
+kubectl get pods -n production
+```
+
+Then let ArgoCD re-sync to reconcile any drift:
+```bash
+argocd app sync feature-flags-production   # only if production runtime is on
+```
+
+## Procedure 2 — PostgreSQL point-in-time
+
+- **In-cluster postgres (dev/showcase):** restore the PV via the `daily-volumes`
+  Velero backup, then `kubectl rollout restart deployment/postgres`.
+- **Managed RDS (recommended for prod runtime):** use RDS automated snapshots /
+  PITR: `aws rds restore-db-instance-to-point-in-time ...` then repoint
+  `DATABASE_URL` (via External Secrets / AWS Secrets Manager).
+
+## Procedure 3 — Full cluster rebuild
+
+1. `terraform apply` the environment (recreates VPC/EKS/ECR).
+2. Bootstrap the platform (Istio, Ingress, cert-manager, ArgoCD) per
+   [../k8s/argocd/install-notes.md](../k8s/argocd/install-notes.md).
+3. Reinstall Velero (same S3 location auto-discovers backups).
+4. `velero restore create --from-backup daily-all-namespaces-<timestamp>`.
+5. ArgoCD reconciles app state from Git.
+
+## Procedure 4 — Terraform state corruption
+
+- State is versioned in S3 with `use_lockfile = true`. Recover a prior version:
+  ```bash
+  aws s3api list-object-versions --bucket devsecops-tfstate-125156866917 \
+    --prefix staging/terraform.tfstate
+  aws s3api get-object --bucket devsecops-tfstate-125156866917 \
+    --key staging/terraform.tfstate --version-id <id> terraform.tfstate
+  ```
+- If a lock is stuck: `terraform force-unlock <LOCK_ID>`.
+
+## Verification
+
+DR is only real if tested — see the namespace-restore and secret-rotation drills
+in [resilience-testing.md](resilience-testing.md). Run a restore drill quarterly.

docs/resilience-testing.md

@@ -0,0 +1,72 @@
+# Resilience Testing Runbook (Phase 8)
+
+Failure-injection scenarios that prove the platform self-heals. Run on Minikube
+first, then repeat the cluster-only tests on EKS. Helper:
+[../scripts/resilience-test.sh](../scripts/resilience-test.sh); load:
+[../tests/k6/](../tests/k6/).
+
+## Scenarios
+
+| # | Test | Method | Run on | Expected result |
+|---|------|--------|--------|-----------------|
+| 1 | Pod crash | `kubectl delete pod` | Minikube + EKS | ReplicaSet recreates pod; no 5xx (other replica serves) |
+| 2 | Failed deployment | deploy broken image | Minikube + EKS | Probes fail → CD/ArgoCD rollback; traffic stays on healthy color |
+| 3 | CPU stress | k6 `stress-hpa.js` | Minikube + EKS | HPA scales out to maxReplicas, then back in |
+| 4 | Node drain | `kubectl drain` | EKS only | Pods rescheduled to other nodes |
+| 5 | Secret rotation | rotate AWS secret | EKS only | External Secrets re-syncs `backend-secrets` within refresh interval |
+| 6 | Namespace restore | Velero restore | EKS only | Namespace fully recovered (see disaster-recovery.md) |
+
+## 1. Pod crash
+
+```bash
+scripts/resilience-test.sh pod-crash dev
+# Expect: deleted pod replaced within seconds, READY count restored.
+```
+
+## 2. Failed deployment → rollback
+
+```bash
+scripts/resilience-test.sh bad-deploy dev backend
+# Expect: rollout stuck (ImagePullBackOff / failing readiness).
+kubectl rollout undo deployment/backend -n dev
+```
+On staging, the CD pipeline's health check fails and triggers
+`scripts/rollback.sh` (blue/green traffic reverts to the previous color).
+
+## 3. CPU stress → HPA scale-out
+
+```bash
+kubectl get hpa -n dev -w
+BASE_URL=http://localhost:3000 k6 run tests/k6/stress-hpa.js
+# Expect: current replicas climb toward maxReplicas (10) as CPU passes 70%,
+# then scale back down after load stops (stabilization window).
+```
+
+## 4. Node drain (EKS)
+
+```bash
+kubectl get nodes
+kubectl drain <node> --ignore-daemonsets --delete-emptydir-data
+kubectl get pods -o wide -n staging   # pods rescheduled onto other nodes
+kubectl uncordon <node>
+```
+
+## 5. Secret rotation (EKS)
+
+```bash
+aws secretsmanager put-secret-value \
+  --secret-id mck21-devsecops/staging/backend \
+  --secret-string '{"DATABASE_URL":"...","REDIS_URL":"..."}'
+# Within refreshInterval (1h), ESO updates backend-secrets:
+kubectl get externalsecret backend-secrets -n staging
+kubectl rollout restart deployment/backend-blue -n staging   # pick up new value
+```
+
+## 6. Namespace restore (EKS)
+
+See [disaster-recovery.md → Procedure 1](disaster-recovery.md#procedure-1--restore-a-namespace).
+
+## Recording results
+
+Capture evidence (terminal output, Grafana HPA panel, ArgoCD rollback) under
+`images/phase-8/` when running the live drills.

k8s/velero/install-notes.md

@@ -0,0 +1,56 @@
+# Velero — install & backup/restore (Phase 8)
+
+Cluster backup/restore for EKS, backed by S3 + EBS volume snapshots. Auth via
+IRSA (no static keys).
+
+## Prerequisites (Terraform)
+
+- S3 bucket `mck21-devsecops-velero-backups`.
+- IRSA role `mck21-devsecops-velero` with S3 read/write on that bucket and
+  `ec2:CreateSnapshot` / `DescribeSnapshots` / `CreateTags` for EBS snapshots.
+
+## Install (Helm)
+
+```bash
+helm repo add vmware-tanzu https://vmware-tanzu.github.io/helm-charts
+helm repo update
+
+helm install velero vmware-tanzu/velero \
+  --namespace velero --create-namespace \
+  --set configuration.backupStorageLocation[0].name=default \
+  --set configuration.backupStorageLocation[0].provider=aws \
+  --set configuration.backupStorageLocation[0].bucket=mck21-devsecops-velero-backups \
+  --set configuration.backupStorageLocation[0].config.region=us-east-1 \
+  --set configuration.volumeSnapshotLocation[0].name=default \
+  --set configuration.volumeSnapshotLocation[0].provider=aws \
+  --set configuration.volumeSnapshotLocation[0].config.region=us-east-1 \
+  --set initContainers[0].name=velero-plugin-for-aws \
+  --set initContainers[0].image=velero/velero-plugin-for-aws:v1.10.0 \
+  --set initContainers[0].volumeMounts[0].mountPath=/target \
+  --set initContainers[0].volumeMounts[0].name=plugins \
+  --set serviceAccount.server.annotations."eks\.amazonaws\.com/role-arn"=arn:aws:iam::125156866917:role/mck21-devsecops-velero
+```
+
+## Apply schedules
+
+```bash
+kubectl apply -f k8s/velero/schedules.yaml
+velero schedule get
+```
+
+| Schedule | Cron | Scope | Retention |
+|----------|------|-------|-----------|
+| `daily-all-namespaces` | `0 1 * * *` | all namespaces | 7 days |
+| `production-6h` | `0 */6 * * *` | production | 14 days |
+| `daily-volumes` | `30 1 * * *` | staging, production PVs | 7 days |
+
+## Restore drill
+
+```bash
+velero backup get
+velero restore create --from-backup daily-all-namespaces-<timestamp>
+velero restore describe <restore-name>
+```
+
+Full procedures (namespace restore, PostgreSQL point-in-time, tfstate recovery)
+are in [../../docs/disaster-recovery.md](../../docs/disaster-recovery.md).

k8s/velero/schedules.yaml

@@ -0,0 +1,55 @@
+# Velero backup schedules (EKS). Backups stored in S3, volumes via EBS snapshots.
+# Requires Velero installed with the AWS plugin — see install-notes.md.
+---
+# All namespaces, daily, 7-day retention.
+apiVersion: velero.io/v1
+kind: Schedule
+metadata:
+  name: daily-all-namespaces
+  namespace: velero
+spec:
+  schedule: "0 1 * * *"
+  template:
+    includedNamespaces:
+      - "*"
+    excludedResources:
+      - events
+      - events.events.k8s.io
+    snapshotVolumes: true
+    storageLocation: default
+    volumeSnapshotLocations:
+      - default
+    ttl: 168h0m0s
+---
+# Production namespace, every 6 hours, 14-day retention.
+apiVersion: velero.io/v1
+kind: Schedule
+metadata:
+  name: production-6h
+  namespace: velero
+spec:
+  schedule: "0 */6 * * *"
+  template:
+    includedNamespaces:
+      - production
+    snapshotVolumes: true
+    storageLocation: default
+    volumeSnapshotLocations:
+      - default
+    ttl: 336h0m0s
+---
+# Persistent volumes (postgres/redis data), daily fs-backup, 7-day retention.
+apiVersion: velero.io/v1
+kind: Schedule
+metadata:
+  name: daily-volumes
+  namespace: velero
+spec:
+  schedule: "30 1 * * *"
+  template:
+    includedNamespaces:
+      - staging
+      - production
+    defaultVolumesToFsBackup: true
+    storageLocation: default
+    ttl: 168h0m0s

scripts/resilience-test.sh

@@ -0,0 +1,44 @@
+#!/usr/bin/env bash
+# Resilience drill helper — inject a failure and observe recovery.
+# Usage: scripts/resilience-test.sh <scenario> [namespace]
+#   scenarios: pod-crash | bad-deploy | cpu-stress | help
+# Defaults to namespace "dev". Read-only on the cluster except the chosen action.
+set -euo pipefail
+
+SCENARIO="${1:-help}"
+NS="${2:-dev}"
+
+usage() {
+  cat <<'EOF'
+Resilience drill helper
+  pod-crash   Delete a backend pod; expect auto-restart, no downtime
+  bad-deploy  Set an invalid image; expect probes to fail (roll back via GitOps/CD)
+  cpu-stress  Print the k6 command that drives HPA scale-out
+EOF
+}
+
+case "$SCENARIO" in
+  pod-crash)
+    echo "[*] Pods before:"; kubectl get pods -n "$NS" -l app=backend
+    POD="$(kubectl get pods -n "$NS" -l app=backend -o jsonpath='{.items[0].metadata.name}')"
+    echo "[*] Deleting pod $POD ..."
+    kubectl delete pod "$POD" -n "$NS"
+    echo "[*] Watch recovery (Ctrl-C to stop):"
+    kubectl get pods -n "$NS" -l app=backend -w
+    ;;
+  bad-deploy)
+    DEPLOY="${3:-backend}"
+    echo "[*] Setting invalid image on deployment/$DEPLOY in $NS ..."
+    kubectl set image "deployment/$DEPLOY" backend=backend:does-not-exist -n "$NS"
+    echo "[*] Watch rollout fail (probes/imagepull). Roll back with:"
+    echo "    kubectl rollout undo deployment/$DEPLOY -n $NS"
+    kubectl rollout status "deployment/$DEPLOY" -n "$NS" --timeout=90s || true
+    ;;
+  cpu-stress)
+    echo "[*] In one shell: kubectl get hpa -n $NS -w"
+    echo "[*] In another:   BASE_URL=http://localhost:3000 k6 run tests/k6/stress-hpa.js"
+    ;;
+  help|*)
+    usage
+    ;;
+esac

tests/k6/README.md

@@ -0,0 +1,45 @@
+# k6 Load & Resilience Tests (Phase 8)
+
+Load tests for the Feature Flag Service, used to validate SLOs and trigger HPA
+scale-out during resilience drills.
+
+## Install k6
+
+```bash
+brew install k6            # macOS
+# or: docker run --rm -i grafana/k6 run - < tests/k6/load-flags-read.js
+```
+
+## Tests
+
+| Script | Purpose | Pass criteria |
+|--------|---------|---------------|
+| `load-flags-read.js` | Steady 200 rps on the cache-hot read path | P95 < 300ms, errors < 0.1% (thresholds enforced) |
+| `stress-hpa.js` | Ramp load to push CPU past 70% | HPA scales replicas up, then back down |
+
+## Run against dev (Minikube)
+
+```bash
+kubectl port-forward svc/backend -n dev 3000:80 &
+# seed a flag first if needed:
+curl -X POST localhost:3000/api/flags -H 'content-type: application/json' \
+  -d '{"key":"checkout-v2","description":"demo","environments":{"production":true}}'
+
+BASE_URL=http://localhost:3000 FLAG_KEY=checkout-v2 k6 run tests/k6/load-flags-read.js
+```
+
+## Run against staging (EKS)
+
+```bash
+BASE_URL=https://flags.staging.example.com k6 run tests/k6/load-flags-read.js
+```
+
+## Observe HPA during stress
+
+```bash
+kubectl get hpa -n dev -w        # or -n staging
+BASE_URL=http://localhost:3000 k6 run tests/k6/stress-hpa.js
+```
+
+Scenarios and expected outcomes are tracked in
+[../../docs/resilience-testing.md](../../docs/resilience-testing.md).

tests/k6/load-flags-read.js

@@ -0,0 +1,35 @@
+// k6 — steady-state read load on the cache-hot path GET /api/flags/{key}.
+// Validates SLOs (P95 <= 300ms, error rate < 0.1%) under normal traffic.
+//
+//   BASE_URL=http://localhost:3000 FLAG_KEY=checkout-v2 k6 run tests/k6/load-flags-read.js
+import http from 'k6/http';
+import { check, sleep } from 'k6';
+
+const BASE_URL = __ENV.BASE_URL || 'http://localhost:3000';
+const FLAG_KEY = __ENV.FLAG_KEY || 'checkout-v2';
+
+export const options = {
+  scenarios: {
+    steady: {
+      executor: 'constant-arrival-rate',
+      rate: 200, // requests per second
+      timeUnit: '1s',
+      duration: '5m',
+      preAllocatedVUs: 50,
+      maxVUs: 200,
+    },
+  },
+  thresholds: {
+    http_req_duration: ['p(95)<300'], // SLO: P95 <= 300ms
+    http_req_failed: ['rate<0.001'], // SLO: error rate < 0.1%
+  },
+};
+
+export default function () {
+  const res = http.get(`${BASE_URL}/api/flags/${FLAG_KEY}`);
+  check(res, {
+    'status is 200': (r) => r.status === 200,
+    'has body': (r) => r.body && r.body.length > 0,
+  });
+  sleep(0.1);
+}