Feat/phase 7 monitoring sre

.github/workflows/ci.yaml

@@ -137,6 +137,7 @@ jobs:
           echo "sha_tag=sha-${{ github.sha }}" >> "$GITHUB_OUTPUT"
 
       - name: Push to staging ECR
+        id: push
         uses: docker/build-push-action@10e90e3645eae34f1e60eeb005ba3a3d33f178e8 # v6
         with:
           context: app/backend
@@ -147,14 +148,13 @@ jobs:
           cache-from: type=gha
           cache-to: type=gha,mode=max
 
-      - name: Push to production ECR
-        if: github.ref == 'refs/heads/main' && vars.ENABLE_PROD_ECR == 'true'
-        uses: docker/build-push-action@10e90e3645eae34f1e60eeb005ba3a3d33f178e8 # v6
-        with:
-          context: app/backend
-          push: true
-          tags: |
-            ${{ vars.AWS_ACCOUNT_ID }}.dkr.ecr.${{ vars.AWS_REGION }}.amazonaws.com/${{ vars.PROJECT_NAME }}-production-backend:${{ steps.tags.outputs.sha_tag }}
-            ${{ vars.AWS_ACCOUNT_ID }}.dkr.ecr.${{ vars.AWS_REGION }}.amazonaws.com/${{ vars.PROJECT_NAME }}-production-backend:latest
-          cache-from: type=gha
-          cache-to: type=gha,mode=max
+      # Supply-chain: keyless Cosign signature (Sigstore/Fulcio via GitHub OIDC).
+      # Kyverno verify-image-signatures admits only images signed here. (Phase 6)
+      - name: Install Cosign
+        uses: sigstore/cosign-installer@d58896d6a1865668819e1d91763c7751a165e159 # v3.9.2
+
+      - name: Sign image (keyless)
+        env:
+          COSIGN_YES: "true"
+        run: |
+          cosign sign "${{ steps.tags.outputs.staging }}@${{ steps.push.outputs.digest }}"

.yamllint

@@ -0,0 +1,20 @@
+# Relaxed yamllint config for Kubernetes manifests and GitHub workflows.
+# Kustomize/Helm-style YAML uses long lines and no document-start markers.
+extends: relaxed
+
+rules:
+  line-length: disable
+  document-start: disable
+  comments:
+    min-spaces-from-content: 1
+  comments-indentation: disable
+  truthy:
+    allowed-values: ['true', 'false', 'on']
+    check-keys: false
+  indentation:
+    spaces: 2
+    indent-sequences: consistent
+  empty-lines:
+    max: 2
+  trailing-spaces: enable
+  new-line-at-end-of-file: enable

docs/cd-production-promotion.md

@@ -1,22 +1,47 @@
-# Production CD promotion (deferred)
+# Production CD promotion (reference — not scheduled)
 
-Production deploy is **intentionally disabled** during Phase 5 build to keep the pipeline green and AWS costs low. Staging is the sole automated CD target until the rest of the PLAN is complete.
+> **Status:** Production runtime is **off** and not on the roadmap right now. Staging is the only counted pipeline. This document is a reference for *how* production could be turned on later; enabling it is an open decision, not a committed phase.
 
-## When to enable
+Production exists in **Git** as a mirror of staging (Kustomize overlays, Terraform, ArgoCD Application). **Runtime** (EKS cluster, deploy, ECR push) stays off — see [showcase-staging-only.md](showcase-staging-only.md).
 
-After Phase 5 staging CD is stable and remaining PLAN phases are done (or when you explicitly choose to turn on multi-env).
+## If/when production runtime is enabled
+
+There is no scheduled date. Revisit this only if we explicitly decide to go multi-environment. Prerequisite either way: staging CD proven stable.
 
 ## Steps
 
 ### 1. GitHub repository variables
 
 | Variable | Value |
 |----------|-------|
-| `ENABLE_PROD_ECR` | `true` — restores production ECR push on `main` in CI |
-| `ENABLE_PROD_CD` | `true` — enable when the promote job is wired (future) |
+| `ENABLE_PROD_CD` | `true` — enable promote job when wired |
 | `PRODUCTION_HEALTH_URL` | `https://flags.example.com/health` |
 
-### 2. Bootstrap production EKS platform
+### 2. Re-enable production ECR push in CI
+
+Add this step back to [`.github/workflows/ci.yaml`](../.github/workflows/ci.yaml) after `Push to staging ECR`:
+
+```yaml
+      - name: Push to production ECR
+        if: github.ref == 'refs/heads/main'
+        uses: docker/build-push-action@10e90e3645eae34f1e60eeb005ba3a3d33f178e8 # v6
+        with:
+          context: app/backend
+          push: true
+          tags: |
+            ${{ vars.AWS_ACCOUNT_ID }}.dkr.ecr.${{ vars.AWS_REGION }}.amazonaws.com/${{ vars.PROJECT_NAME }}-production-backend:${{ steps.tags.outputs.sha_tag }}
+            ${{ vars.AWS_ACCOUNT_ID }}.dkr.ecr.${{ vars.AWS_REGION }}.amazonaws.com/${{ vars.PROJECT_NAME }}-production-backend:latest
+          cache-from: type=gha
+          cache-to: type=gha,mode=max
+```
+
+Or gate behind `vars.ENABLE_PROD_ECR == 'true'` if you prefer opt-in.
+
+### 3. Apply production Terraform
+
+See [infrastructure/environments/production/README.md](../infrastructure/environments/production/README.md).
+
+### 4. Bootstrap production EKS platform
 
 Repeat the staging bootstrap from [k8s/argocd/install-notes.md](../k8s/argocd/install-notes.md) on the production cluster:
 
@@ -25,13 +50,13 @@ Repeat the staging bootstrap from [k8s/argocd/install-notes.md](../k8s/argocd/in
 - ArgoCD Helm install
 - `backend-secrets` in namespace `production`
 
-### 3. Register ArgoCD Application
+### 5. Register ArgoCD Application
 
 ```bash
 kubectl apply -f k8s/argocd/application-production.yaml
 ```
 
-### 4. Add promote job to CD workflow
+### 6. Add promote job to CD workflow
 
 Add a job (manual or automatic) after staging public health check succeeds:
 
@@ -50,7 +75,7 @@ promote-production:
   # workflow_dispatch for manual approval is recommended first
 ```
 
-### 5. Blue/green on production (optional)
+### 7. Blue/green on production (optional)
 
 Reuse `scripts/blue-green-switch.sh` and production overlay changes mirroring staging if zero-downtime promotion is required on production.
 

docs/showcase-staging-only.md

@@ -0,0 +1,112 @@
+# Showcase — staging-only runtime
+
+This portfolio project uses a **Git vs runtime** split for production:
+
+| Layer | Staging | Production |
+|-------|---------|------------|
+| **Git** (manifests, Terraform) | Active source | **Mirror** — kept aligned with staging |
+| **Runtime** (AWS, EKS, CD) | **Lifted** for showcase | **Off** — not lifted; next steps undecided |
+
+Staging is the only environment with a working pipeline right now. You only need **staging** running for green CI + CD and Phase 5 screenshots.
+
+---
+
+## Before your first push (one-time)
+
+### 1. Terraform — staging only
+
+```bash
+cd infrastructure/environments/staging
+terraform init
+terraform apply
+```
+
+Do **not** run `terraform apply` in `infrastructure/environments/production/`. Production is off; that code stays valid in Git as a mirror only — see [infrastructure/environments/production/README.md](../infrastructure/environments/production/README.md).
+
+### 2. Configure kubectl
+
+```bash
+aws eks update-kubeconfig \
+  --name mck21-devsecops-staging-eks \
+  --region us-east-1
+
+kubectl get nodes   # all Ready
+```
+
+### 3. Bootstrap EKS staging (cluster platform)
+
+Follow [k8s/argocd/install-notes.md](../k8s/argocd/install-notes.md) steps 1–8:
+
+- Namespaces, Istio, Ingress NGINX, cert-manager, `backend-secrets`
+- ArgoCD Helm + repo connection
+- `kubectl apply -f k8s/argocd/application-staging.yaml`
+
+Do **not** apply `k8s/argocd/application-production.yaml` — production runtime is off. See [cd-production-promotion.md](cd-production-promotion.md) only if production is ever turned on.
+
+### 4. GitHub repository variables
+
+| Variable | Value |
+|----------|-------|
+| `AWS_ACCOUNT_ID` | `125156866917` |
+| `AWS_REGION` | `us-east-1` |
+| `PROJECT_NAME` | `mck21-devsecops` |
+| `EKS_CLUSTER_NAME` | `mck21-devsecops-staging-eks` |
+| `STAGING_HEALTH_URL` | Leave empty until Ingress works, or `http(s)://<alb-host>/health` |
+
+Optional: secret `CD_BOT_TOKEN` if branch protection blocks `GITHUB_TOKEN` GitOps pushes.
+
+### 5. Branch protection
+
+Allow GitHub Actions to push `[skip ci]` GitOps commits to `main`, or use `CD_BOT_TOKEN`.
+
+---
+
+## Trigger pipeline
+
+```bash
+git checkout main && git pull
+git commit --allow-empty -m "chore: trigger staging cd validation"
+git push origin main
+```
+
+Expected flow:
+
+1. **CI** — lint, test, build, Trivy, push image to **staging ECR only**
+2. **CD** — bump idle color tag → ArgoCD sync → health check → traffic switch
+
+Monitor:
+
+```bash
+gh run list --workflow=CI --limit 3
+gh run list --workflow=CD --limit 3
+```
+
+---
+
+## Keeping production in Git aligned
+
+When you change [k8s/overlays/staging/](../k8s/overlays/staging/), review [k8s/overlays/production/](../k8s/overlays/production/) in the same PR:
+
+- Same blue/green structure and sync waves
+- Production-specific diffs: replicas (3), resources, ingress host (`flags.example.com`), ECR repo name
+
+No cluster deploy required for production YAML updates during showcase.
+
+---
+
+## What production would need if ever turned on
+
+Production runtime is off and not scheduled. If we later decide to enable it, see [cd-production-promotion.md](cd-production-promotion.md):
+
+- `terraform apply` production
+- Bootstrap production EKS
+- Re-enable production ECR push in CI
+- CD promote job + `application-production.yaml`
+
+---
+
+## Phase 5 screenshots
+
+After CI + CD are green, capture evidence per [images/phase-5/README.md](../images/phase-5/README.md).
+
+Troubleshooting: [TROUBLESHOOTING.md § EKS staging CD](../TROUBLESHOOTING.md#eks-staging-cd-phase-5).

docs/sre.md

@@ -0,0 +1,62 @@
+# SRE — SLIs, SLOs & Error Budget Policy
+
+Reliability targets for the Feature Flag Service API. Measured from Istio mesh
+telemetry (`reporter="destination"`) so they reflect the client experience.
+
+## Service Level Indicators (SLIs)
+
+| SLI | Definition | Source metric |
+|-----|------------|---------------|
+| Availability | % of requests returning non-5xx | `istio_requests_total` (response_code) |
+| Latency | P95 of `GET /api/flags/{key}` (Redis-cached read) | `istio_request_duration_milliseconds_bucket` |
+| Error rate | % of requests returning 5xx | `istio_requests_total` (response_code=~"5..") |
+
+## Service Level Objectives (SLOs)
+
+| SLO | Target | Window |
+|-----|--------|--------|
+| API availability | 99.9% | 30 days rolling |
+| P95 latency | ≤ 300 ms | 30 days rolling |
+| Error rate | < 0.1% | 30 days rolling |
+
+**Error budget:** 99.9% availability ⇒ **0.1%** of requests may fail per 30-day
+window (~43 min of full downtime equivalent).
+
+## Error Budget Policy
+
+- **Budget healthy (>50% remaining):** normal feature velocity; deploys flow
+  through CI/CD freely.
+- **Budget low (<50% remaining):** prioritise reliability work; non-critical
+  risky changes held.
+- **Budget exhausted:** change freeze except reliability/security fixes until the
+  budget recovers over the rolling window.
+
+## Burn-rate alerting (multi-window, multi-burn-rate)
+
+Implemented in [../monitoring/prometheus/prometheusrules.yaml](../monitoring/prometheus/prometheusrules.yaml).
+
+| Alert | Condition | Severity | Action |
+|-------|-----------|----------|--------|
+| `BackendErrorBudgetFastBurn` | 14.4× budget over 1h **and** 5m | critical | Page on-call |
+| `BackendErrorBudgetSlowBurn` | 6× budget over 6h **and** 30m | warning | Create ticket |
+| `BackendHighErrorRate` | 5xx ratio > 5% for 5m | critical | Page |
+| `BackendHighLatencyP95` | P95 > 300 ms for 10m | warning | Investigate |
+| `BackendPodCrashLooping` | >3 restarts / 15m | critical | Page |
+| `NodeMemoryHigh` | node mem > 85% for 10m | warning | Investigate |
+
+Why two burn rates: a **fast burn** (14.4×) catches acute outages within minutes;
+a **slow burn** (6×) catches gradual degradation before it silently drains the
+monthly budget. Each alert requires a long **and** a short window to fire, which
+suppresses flapping from brief spikes.
+
+## Dashboards
+
+- **SLO — Error Budget** (`slo-error-budget`): availability vs target, budget
+  remaining gauge, current burn rate, multi-window error ratio.
+- **Feature Flag Service** (`feature-flags-app`): request rate, error ratio, P95,
+  cache hit ratio, HPA replicas.
+
+## Reviewing the budget
+
+Monthly SRE review: read the SLO dashboard, classify any incidents, and decide
+feature-vs-reliability priority for the next period per the policy above.

images/phase-5/README.md

@@ -1,5 +1,7 @@
 # Phase 5 — CD & GitOps evidence
 
+**Prerequisite:** Complete [docs/showcase-staging-only.md](../docs/showcase-staging-only.md) (terraform apply staging only, bootstrap EKS, push to `main`, CI + CD green).
+
 Capture screenshots after the staging CD pipeline runs end-to-end on EKS.
 
 ## Checklist
@@ -29,4 +31,4 @@ curl -f "$STAGING_HEALTH_URL"
 ## Notes
 
 - Dev (Minikube) CD is manual during Phase 5 — no screenshot required for phase closure.
-- Production ArgoCD screenshots deferred until [docs/cd-production-promotion.md](../docs/cd-production-promotion.md) is activated.
+- Production exists in Git as a staging mirror only — runtime is **off** and not scheduled, so no production screenshots. Reference if ever turned on: [docs/cd-production-promotion.md](../docs/cd-production-promotion.md).

infrastructure/environments/production/README.md

@@ -0,0 +1,40 @@
+# Production environment (Terraform)
+
+Valid Terraform for the production AWS stack — **Git mirror** of [../staging/](../staging/) with production sizing and a separate VPC CIDR.
+
+## Status: off (current)
+
+Production runtime is **off** and not scheduled. Staging is the only environment with a working pipeline. This directory is a Git mirror only.
+
+**Do not run `terraform apply` here.**
+
+- No production EKS cluster required for green CI/CD
+- No AWS cost from production infra
+- Code remains in the repo to demonstrate multi-environment IaC design
+
+When [staging](../staging/) modules or variables change, keep this directory aligned (same modules, production-specific values).
+
+## Differences from staging
+
+| Setting | Staging | Production |
+|---------|---------|------------|
+| VPC CIDR | `10.0.0.0/16` | `10.1.0.0/16` |
+| Node type | `t3.medium` | `t3.large` |
+| Node min | 1 | 2 |
+| ECR retention | 10 images | 20 images |
+| GitHub OIDC | Created in staging apply | Uses account-level provider |
+
+## When to apply
+
+There is no scheduled date. Apply only if we explicitly decide to enable production runtime — see [docs/cd-production-promotion.md](../../docs/cd-production-promotion.md).
+
+```bash
+cd infrastructure/environments/production
+terraform init
+terraform plan
+terraform apply   # only when promoting to production runtime
+```
+
+## State
+
+Backend key: `production/terraform.tfstate` in bucket `devsecops-tfstate-125156866917`.