From 57f193ec01f1fe98bc2784435030b207c25e4fdc Mon Sep 17 00:00:00 2001 From: Simon van Lierde Date: Wed, 19 Aug 2026 14:14:03 +0000 Subject: [PATCH 01/85] build(deps): add terraform updates for infra directory --- .github/dependabot.yml | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/.github/dependabot.yml b/.github/dependabot.yml index 6d6cc41..1f09af9 100644 --- a/.github/dependabot.yml +++ b/.github/dependabot.yml @@ -19,6 +19,13 @@ updates: schedule: interval: "monthly" + # Cloudflare provider for infra/. Dependabot bumps the constraint in + # main.tf but not the hashes in .terraform.lock.hcl — see the runbook. + - package-ecosystem: "terraform" + directory: "/infra" + schedule: + interval: "monthly" + - package-ecosystem: "github-actions" directory: "/" schedule: From 05748de6663079ad78782b6b2664c8b4d6d3d77c Mon Sep 17 00:00:00 2001 From: Simon van Lierde Date: Wed, 19 Aug 2026 14:14:15 +0000 Subject: [PATCH 02/85] docs: update RUNBOOK with OpenTofu management details for Cloudflare tunnel and access --- docs/RUNBOOK.md | 45 ++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 44 insertions(+), 1 deletion(-) diff --git a/docs/RUNBOOK.md b/docs/RUNBOOK.md index 9bf3fb9..1d1e66f 100644 --- a/docs/RUNBOOK.md +++ b/docs/RUNBOOK.md @@ -59,6 +59,8 @@ write-ahead log. Two practical notes: - The tarball is mode 0600 and contains secrets (the Grafana database among them). Copy it off-host over a private channel — a backup on the disk it protects is a decoration. +- The tarball covers the docker volumes and nothing else. The OpenTofu + state for the Cloudflare edge is not in it — see below. - During the pause the collector keeps accepting telemetry and buffers it for about five minutes. A backup that takes longer than that will drop data, so on large volumes run it at a quiet hour. @@ -74,7 +76,13 @@ the host is the intended setup. (visible as export errors on their side) until they're updated. A running demo overlay counts as a sender: re-run `just demo` to recreate it with the new token. -- **Tunnel token:** rotate in Cloudflare Zero Trust, set the new +- **Tunnel token:** the tunnel is OpenTofu-managed, so read the token back + from there rather than copying it out of the dashboard. Rotate the tunnel + secret in Cloudflare Zero Trust, then `cd infra && tofu apply` (which + refreshes the token data source) and `tofu output -raw tunnel_token`. To + rotate entirely from code instead, `tofu apply -replace=cloudflare_zero_trust_tunnel_cloudflared.monitoring` + builds a new tunnel and repoints both CNAMEs at it — ingestion and Grafana + are unreachable for the minute or so that takes. Either way: new token into `CLOUDFLARE_TUNNEL_TOKEN` in `.env`, then `just up-tunnel`. - **Grafana admin password:** change `GRAFANA_ADMIN_PASSWORD` in `.env`, then `docker compose up -d grafana`. @@ -95,6 +103,34 @@ notify error per cycle (expected, harmless), and alerts remain visible in Grafana. After changing either variable, `docker compose up -d alertmanager`. +## Changing the Cloudflare edge + +The tunnel, its ingress rules, both DNS records, and the Cloudflare Access +policy that fronts Grafana are all OpenTofu in `infra/`. Change them there, +not in the Zero Trust dashboard: the next apply reverts anything clicked in +by hand. + +```sh +just infra-validate # tofu init + validate, in a container +cd infra && tofu plan # needs CLOUDFLARE_API_TOKEN exported +cd infra && tofu apply +``` + +- **Granting or revoking Grafana access:** edit `grafana_allowed_emails` in + `infra/terraform.tfvars` and apply. That list is the entire allowlist. A + removed address keeps working until their Access session expires (24h), + so for an urgent revocation also revoke the session in Zero Trust. At + least one address has to remain — the variable's validation rejects an + empty list, which would lock everyone out of Grafana. +- **Adding a hostname:** add an `ingress` entry pointing at the service's + container port, plus a matching `cloudflare_dns_record`. The catch-all + `http_status:404` entry stays last, or it swallows everything after it. +- **State lives on this host only.** `infra/terraform.tfstate` is gitignored + and `just backup` does not touch it. Copy it off-host next to the backups. + Losing it orphans the Cloudflare resources: they keep running, but the + next apply tries to create duplicates and you get them back only by hand + with `tofu import`. + ## Upgrading images Dependabot opens PRs that bump the pinned versions, and CI runs `just @@ -104,4 +140,11 @@ binaries the stack will run — when a new version changes its config syntax, CI fails loudly before the change reaches the host. That is the point. +Dependabot also watches the Cloudflare provider in `infra/`. Those PRs need +one manual step: it bumps the constraint in `main.tf` but not the recorded +hashes in `.terraform.lock.hcl`, so check the branch out and run +`cd infra && tofu init -upgrade`, then `just infra-validate` and `tofu plan` +against the real account — validation proves the syntax parses, only a plan +proves the provider still maps the config to the same resources. + After merging: on the host, `git pull && just pull && just up`. From ec260d283548caf437ec71d53aea7116fc6232f0 Mon Sep 17 00:00:00 2001 From: Simon van Lierde Date: Thu, 20 Aug 2026 14:29:46 +0000 Subject: [PATCH 03/85] fix: repair exemplar links, error-rate alert, and logs dashboard - exemplar destination trace_id -> traceID, matching what Tempo emits - HighErrorRate filters span_kind=SPAN_KIND_SERVER on both sides; internal ASGI child spans inflated the denominator ~3x - demo load hits /work twice so the demonstrated error rate (~10%) actually crosses the 5% threshold instead of sitting on it - infrastructure-logs dashboard queries service_name (the label Loki actually indexes); drop the env variable no templating query can populate - storage-s3 overlay re-declares commands with -config.expand-env=true, or the documented activation reads the \${S3_*} placeholders literally - drop tempo datasource search.hide, restates the default --- compose.demo.yml | 2 +- compose.storage-s3.yml | 6 ++++++ config/alerts/stack.yaml | 7 +++++-- config/grafana/datasources.yaml | 4 +--- dashboards/infrastructure-logs.json | 25 +++++++------------------ 5 files changed, 20 insertions(+), 24 deletions(-) diff --git a/compose.demo.yml b/compose.demo.yml index 16c100f..8c09c9d 100644 --- a/compose.demo.yml +++ b/compose.demo.yml @@ -41,7 +41,7 @@ services: - -c - | while true; do - curl -s -o /dev/null demo-api:8000/ + curl -s -o /dev/null demo-api:8000/work curl -s -o /dev/null demo-api:8000/work sleep 1 done diff --git a/compose.storage-s3.yml b/compose.storage-s3.yml index 921a8f7..653d86b 100644 --- a/compose.storage-s3.yml +++ b/compose.storage-s3.yml @@ -31,6 +31,10 @@ # access_key: ${S3_ACCESS_KEY_ID} # secret_key: ${S3_SECRET_ACCESS_KEY} # +# Neither Loki nor Tempo expands ${...} in its config by default, so the +# overlay below re-declares each `command` with -config.expand-env=true. +# (compose replaces `command` wholesale, it does not merge it.) +# # 2. Uncomment the overlay below and run: # docker compose -f compose.yml -f compose.storage-s3.yml up -d # @@ -39,6 +43,7 @@ # volumes: # - ./config/loki.s3.yaml:/etc/loki/loki.yaml:ro # - loki_data:/loki +# command: [ "-config.file=/etc/loki/loki.yaml", "-config.expand-env=true" ] # environment: # S3_ENDPOINT: ${S3_ENDPOINT:?set S3_ENDPOINT in .env} # S3_ACCESS_KEY_ID: ${S3_ACCESS_KEY_ID:?set S3_ACCESS_KEY_ID in .env} @@ -48,6 +53,7 @@ # volumes: # - ./config/tempo.s3.yaml:/etc/tempo/tempo.yaml:ro # - tempo_data:/var/tempo +# command: [ "-config.file=/etc/tempo/tempo.yaml", "-config.expand-env=true" ] # environment: # S3_ENDPOINT: ${S3_ENDPOINT:?set S3_ENDPOINT in .env} # S3_ACCESS_KEY_ID: ${S3_ACCESS_KEY_ID:?set S3_ACCESS_KEY_ID in .env} diff --git a/config/alerts/stack.yaml b/config/alerts/stack.yaml index 07d80bd..bfe3e18 100644 --- a/config/alerts/stack.yaml +++ b/config/alerts/stack.yaml @@ -54,10 +54,13 @@ groups: - name: service-red rules: + # Server spans only, matching the Service Health dashboard. Without the + # span_kind filter, internal child spans inflate the denominator and the + # measured rate lands well under the true request error rate. - alert: HighErrorRate expr: > - sum by (service) (rate(traces_spanmetrics_calls_total{status_code="STATUS_CODE_ERROR"}[5m])) - / sum by (service) (rate(traces_spanmetrics_calls_total[5m])) > 0.05 + sum by (service) (rate(traces_spanmetrics_calls_total{span_kind="SPAN_KIND_SERVER",status_code="STATUS_CODE_ERROR"}[5m])) + / sum by (service) (rate(traces_spanmetrics_calls_total{span_kind="SPAN_KIND_SERVER"}[5m])) > 0.05 for: 5m labels: severity: warning diff --git a/config/grafana/datasources.yaml b/config/grafana/datasources.yaml index 5a2dcf4..02ae034 100644 --- a/config/grafana/datasources.yaml +++ b/config/grafana/datasources.yaml @@ -10,7 +10,7 @@ datasources: jsonData: timeInterval: 30s exemplarTraceIdDestinations: - - name: trace_id + - name: traceID datasourceUid: tempo - name: Loki @@ -50,5 +50,3 @@ datasources: datasourceUid: prometheus nodeGraph: enabled: true - search: - hide: false diff --git a/dashboards/infrastructure-logs.json b/dashboards/infrastructure-logs.json index f49dbaa..5b0c0e1 100644 --- a/dashboards/infrastructure-logs.json +++ b/dashboards/infrastructure-logs.json @@ -7,27 +7,16 @@ "time": { "from": "now-1h", "to": "now" }, "graphTooltip": 1, "tags": ["relab"], + "description": "Container logs via OTLP ingest. This stack's Loki index carries only the service_name label (everything else, service_instance_id included, is structured metadata); there is no env/service docker-driver label path exposed here, so filtering is by service_name only.", "templating": { "list": [ - { - "name": "env", - "label": "Environment", - "type": "custom", - "query": "staging,prod", - "current": { "text": "staging", "value": "staging" }, - "options": [ - { "text": "staging", "value": "staging", "selected": true }, - { "text": "prod", "value": "prod", "selected": false } - ], - "hide": 0 - }, { "name": "service", "label": "Service", "type": "query", "datasource": { "type": "loki", "uid": "loki" }, - "definition": "label_values({env=\"$env\"}, service)", - "query": { "type": "labelValues", "label": "service", "stream": "{env=\"$env\"}" }, + "definition": "label_values(service_name)", + "query": { "type": "labelValues", "label": "service_name" }, "refresh": 2, "sort": 1, "multi": true, @@ -48,8 +37,8 @@ "datasource": { "type": "loki", "uid": "loki" }, "targets": [ { - "expr": "sum by (service) (rate({env=\"$env\", service=~\"$service\"}[$__auto]))", - "legendFormat": "{{service}}", + "expr": "sum by (service_name) (rate({service_name=~\"$service\"}[$__auto]))", + "legendFormat": "{{service_name}}", "refId": "A" } ], @@ -72,7 +61,7 @@ "datasource": { "type": "loki", "uid": "loki" }, "targets": [ { - "expr": "{env=\"$env\", service=~\"$service\"}", + "expr": "{service_name=~\"$service\"}", "refId": "A" } ], @@ -92,7 +81,7 @@ "datasource": { "type": "loki", "uid": "loki" }, "targets": [ { - "expr": "{env=\"$env\", service=~\"$service\"} |~ `(?i)(error|fatal|panic|exception|warn)`", + "expr": "{service_name=~\"$service\"} |~ `(?i)(error|fatal|panic|exception|warn)`", "refId": "A" } ], From 69493b86bc86633df59679fe5ab05c9cc3bd65f1 Mon Sep 17 00:00:00 2001 From: Simon van Lierde Date: Thu, 20 Aug 2026 14:30:16 +0000 Subject: [PATCH 04/85] feat: harden the stack and make collector buffering durable - file-backed exporter queues on a new otel_queue volume; buffered telemetry survives collector restarts, retry/queue settings pinned to the documented five-minute window - mem_limit on every service, sized from observed usage; collector memory_limiter switched to absolute values tied to its container limit - prometheus healthcheck with service_healthy ordering for collector and grafana (loki/tempo are distroless, no probe binary) - scrape grafana, loki, tempo, and alertmanager so TargetDown covers them - loki indexes only service.name; service.instance.id becomes structured metadata instead of minting a stream per sender restart - tempo metrics-generator capped at 50k active series - grafana 13.1.4 (CVE-2026-13438, CVE-2026-17183), loki 3.7.6, tempo 3.0.3, prometheus v3.13.2, node-exporter v1.12.1, cloudflared 2026.8.2 - secure cookies behind GRAFANA_COOKIE_SECURE (up-tunnel enforces it); opt-in per-user Grafana login via Cloudflare Access JWT - no-new-privileges on all services; lint containers run --network none; infra-validate keeps tfstate/tfvars out of the container - up-tunnel also guards GRAFANA_ROOT_URL and warns on empty HEARTBEAT_URL; logs/ps/tail see demo services; loki/tempo config checks join just check; smoke asserts every dashboard provisioned --- .env.example | 12 ++++++ compose.tunnel.yml | 6 ++- compose.yml | 78 ++++++++++++++++++++++++++++++++++---- config/loki.yaml | 10 +++++ config/otel-collector.yaml | 44 +++++++++++++++++++-- config/prometheus.yaml | 17 +++++++++ config/tempo.yaml | 3 ++ justfile | 48 ++++++++++++++++------- 8 files changed, 193 insertions(+), 25 deletions(-) diff --git a/.env.example b/.env.example index a581ecf..569469e 100644 --- a/.env.example +++ b/.env.example @@ -6,6 +6,11 @@ GRAFANA_ADMIN_PASSWORD=change-me # GRAFANA_ROOT_URL=https://grafana.example.com GRAFANA_ROOT_URL=http://localhost:3000 +# Set to true whenever GRAFANA_ROOT_URL is https (i.e. behind the tunnel). +# Marks the session cookie Secure; `just up-tunnel` refuses to run without it. +# Keep false for plain-http localhost use, or logins break. +GRAFANA_COOKIE_SECURE=false + # Bearer token every telemetry sender must present (Authorization: Bearer ). # The default only suits local use — generate a real one for production, e.g.: openssl rand -hex 32 OTLP_AUTH_TOKEN=local-dev-token @@ -13,6 +18,7 @@ OTLP_AUTH_TOKEN=local-dev-token # Where Alertmanager delivers alert notifications (any webhook: ntfy, Slack, …). # Leave empty to run without delivery; failures are logged and harmless. ALERT_WEBHOOK_URL= + # Dead man's switch ping target (e.g. https://hc-ping.com/). The Watchdog # alert posts here every 5m; alert externally when pings stop. HEARTBEAT_URL= @@ -20,3 +26,9 @@ HEARTBEAT_URL= # Only needed for `just up-tunnel` (production exposure via Cloudflare Tunnel). # From: Cloudflare Zero Trust → Networks → Tunnels → → Configure → token CLOUDFLARE_TUNNEL_TOKEN= + +# Set to true to let Cloudflare Access sign users into Grafana individually, +# instead of everyone sharing the admin login. Needs CF_ACCESS_TEAM_DOMAIN. +GRAFANA_JWT_AUTH=false +# Your Zero Trust team name — the in https://.cloudflareaccess.com. +CF_ACCESS_TEAM_DOMAIN= diff --git a/compose.tunnel.yml b/compose.tunnel.yml index 4932e01..291daf4 100644 --- a/compose.tunnel.yml +++ b/compose.tunnel.yml @@ -9,10 +9,14 @@ services: cloudflared: - image: cloudflare/cloudflared:2026.7.0 + image: cloudflare/cloudflared:2026.8.2 restart: unless-stopped depends_on: [ grafana, otel-collector ] command: tunnel --no-autoupdate run --token ${CLOUDFLARE_TUNNEL_TOKEN:?set CLOUDFLARE_TUNNEL_TOKEN in .env} + # Ceiling sized from observed usage with headroom; see compose.yml. + mem_limit: 256m + security_opt: + - no-new-privileges:true logging: driver: json-file options: diff --git a/compose.yml b/compose.yml index f56343a..c49364d 100644 --- a/compose.yml +++ b/compose.yml @@ -18,41 +18,67 @@ x-logging: &default-logging max-size: "10m" max-file: "3" +x-security: &default-security + - no-new-privileges:true + +# The mem_limit on each service is sized from observed steady-state usage with +# generous headroom for bursts. They are ceilings, not reservations: the point +# is that one runaway component can't OOM a host that also runs production. + services: otel-collector: image: otel/opentelemetry-collector-contrib:0.156.0 restart: unless-stopped - depends_on: [ loki, tempo, prometheus ] + depends_on: + loki: + condition: service_started + tempo: + condition: service_started + prometheus: + condition: service_healthy ports: - "127.0.0.1:4317:4317" # OTLP gRPC - "127.0.0.1:4318:4318" # OTLP HTTP volumes: - ./config/otel-collector.yaml:/etc/otelcol/config.yaml:ro + # File-backed exporter queue: telemetry buffered during a backend outage + # survives a collector restart. Owned by uid 10001 — see justfile. + - otel_queue:/var/lib/otelcol/queue command: [ "--config=/etc/otelcol/config.yaml" ] environment: OTLP_AUTH_TOKEN: ${OTLP_AUTH_TOKEN:?set OTLP_AUTH_TOKEN in .env} + # Keep in step with memory_limiter in config/otel-collector.yaml. + mem_limit: 512m + security_opt: *default-security logging: *default-logging loki: - image: grafana/loki:3.7.3 + image: grafana/loki:3.7.6 restart: unless-stopped volumes: - ./config/loki.yaml:/etc/loki/loki.yaml:ro - loki_data:/loki command: [ "-config.file=/etc/loki/loki.yaml" ] + # No healthcheck: the image is distroless, so there is no shell, wget or + # curl to probe /ready with. Grafana's Stack Health dashboard covers it. + mem_limit: 2g + security_opt: *default-security logging: *default-logging tempo: - image: grafana/tempo:3.0.2 + image: grafana/tempo:3.0.3 restart: unless-stopped volumes: - ./config/tempo.yaml:/etc/tempo/tempo.yaml:ro - tempo_data:/var/tempo command: [ "-config.file=/etc/tempo/tempo.yaml" ] + # Distroless too — see the note on loki. + mem_limit: 2g + security_opt: *default-security logging: *default-logging prometheus: - image: prom/prometheus:v3.13.0 + image: prom/prometheus:v3.13.2 restart: unless-stopped volumes: - ./config/prometheus.yaml:/etc/prometheus/prometheus.yaml:ro @@ -68,6 +94,14 @@ services: - --enable-feature=native-histograms,exemplar-storage # Whichever hits first wins; the disk-space alert is the backstop. - --storage.tsdb.retention.size=15GB + healthcheck: + test: [ "CMD", "wget", "-q", "--spider", "http://localhost:9090/-/ready" ] + interval: 15s + timeout: 3s + retries: 5 + start_period: 30s + mem_limit: 2g + security_opt: *default-security logging: *default-logging alertmanager: @@ -90,13 +124,17 @@ services: printf '%s' "$$ALERT_WEBHOOK_URL" > /run/am/webhook_url printf '%s' "$$HEARTBEAT_URL" > /run/am/heartbeat_url exec /bin/alertmanager --config.file=/etc/alertmanager/alertmanager.yaml --storage.path=/alertmanager + mem_limit: 256m + security_opt: *default-security logging: *default-logging node-exporter: - image: prom/node-exporter:v1.11.1 + image: prom/node-exporter:v1.12.1 restart: unless-stopped command: [ "--path.rootfs=/host" ] pid: host + mem_limit: 128m + security_opt: *default-security volumes: # No rslave: unsupported on Docker Desktop; only affects mounts added after start. # No network_mode:host either, so node_network_* describes the container @@ -105,9 +143,15 @@ services: logging: *default-logging grafana: - image: grafana/grafana:13.1.0 + image: grafana/grafana:13.1.4 restart: unless-stopped - depends_on: [ loki, tempo, prometheus ] + depends_on: + loki: + condition: service_started + tempo: + condition: service_started + prometheus: + condition: service_healthy ports: - "127.0.0.1:3000:3000" volumes: @@ -121,6 +165,23 @@ services: GF_SERVER_ROOT_URL: ${GRAFANA_ROOT_URL:-http://localhost:3000} # Land on Stack Health instead of the empty welcome page. GF_DASHBOARDS_DEFAULT_HOME_DASHBOARD_PATH: /var/lib/grafana/dashboards/stack-health.json + # Secure cookies would break plain-http localhost logins, so this is + # opt-in; `just up-tunnel` refuses to expose Grafana without it. + GF_SECURITY_COOKIE_SECURE: ${GRAFANA_COOKIE_SECURE:-false} + GF_SECURITY_COOKIE_SAMESITE: strict + # Opt-in per-user identity from Cloudflare Access, replacing the shared + # admin login. Off until GRAFANA_JWT_AUTH=true and CF_ACCESS_TEAM_DOMAIN + # are set in .env; new users land on the org's default role (Viewer). + # NOTE: the JWK set is team-wide, so if the Zero Trust team fronts more + # than one app, also pin GF_AUTH_JWT_EXPECT_CLAIMS to this app's aud tag. + GF_AUTH_JWT_ENABLED: ${GRAFANA_JWT_AUTH:-false} + GF_AUTH_JWT_HEADER_NAME: Cf-Access-Jwt-Assertion + GF_AUTH_JWT_JWK_SET_URL: https://${CF_ACCESS_TEAM_DOMAIN:-example}.cloudflareaccess.com/cdn-cgi/access/certs + GF_AUTH_JWT_EMAIL_CLAIM: email + GF_AUTH_JWT_USERNAME_CLAIM: email + GF_AUTH_JWT_AUTO_SIGN_UP: "true" + mem_limit: 1g + security_opt: *default-security logging: *default-logging volumes: @@ -129,3 +190,6 @@ volumes: prometheus_data: grafana_data: alertmanager_data: + # Deliberately not in `just backup`: queue contents are seconds of in-flight + # telemetry, worthless by restore time. + otel_queue: diff --git a/config/loki.yaml b/config/loki.yaml index 96d3512..4ed036b 100644 --- a/config/loki.yaml +++ b/config/loki.yaml @@ -39,6 +39,16 @@ limits_config: # Keep ingestion generous for a single-tenant homelab. ingestion_rate_mb: 16 ingestion_burst_size_mb: 32 + # Index only service.name. Loki's default OTLP mapping also indexes + # service.instance.id, which mints a fresh stream on every sender restart; + # it is now structured metadata, so query it with `| service_instance_id=...`. + otlp_config: + resource_attributes: + ignore_defaults: true + attributes_config: + - action: index_label + attributes: + - service.name compactor: working_directory: /loki/compactor diff --git a/config/otel-collector.yaml b/config/otel-collector.yaml index 338ab99..a797daf 100644 --- a/config/otel-collector.yaml +++ b/config/otel-collector.yaml @@ -7,6 +7,12 @@ extensions: bearertokenauth: token: ${env:OTLP_AUTH_TOKEN} + # Backs the exporter queues below so buffered telemetry survives collector + # restarts (every image bump is one). The volume must be writable by the + # image's uid 10001 — `just up` handles that (see _queue-volume). + file_storage: + directory: /var/lib/otelcol/queue + create_directory: true receivers: otlp: @@ -21,30 +27,60 @@ receivers: authenticator: bearertokenauth processors: + # Absolute, not percentage: percentages are read against the cgroup, and an + # unbounded container makes that the whole host. Sized to the container's + # mem_limit (512m in compose.yml) — change both together. memory_limiter: check_interval: 2s - limit_percentage: 80 - spike_limit_percentage: 15 + limit_mib: 400 + spike_limit_mib: 100 batch: timeout: 5s send_batch_size: 1024 +# The queue/retry settings repeated on each exporter below are pinned rather +# than left to upstream defaults, because the RUNBOOK's "buffers about five +# minutes" promise is exactly those numbers. (No YAML anchor: the collector +# rejects unknown top-level keys, so there is nowhere to hang one.) The queue +# is file-backed, so both a backend outage and a collector restart are ridden out. exporters: # Loki 3.x accepts OTLP natively — no Promtail / loki exporter needed. otlp_http/loki: endpoint: http://loki:3100/otlp + sending_queue: + enabled: true + queue_size: 1000 + storage: file_storage + retry_on_failure: + enabled: true + max_elapsed_time: 300s otlp_grpc/tempo: endpoint: tempo:4317 tls: insecure: true + sending_queue: + enabled: true + queue_size: 1000 + storage: file_storage + retry_on_failure: + enabled: true + max_elapsed_time: 300s - # Prometheus native OTLP receiver (requires otlp-write-receiver feature flag). + # Prometheus native OTLP receiver (needs --web.enable-otlp-receiver, set in + # compose.yml; it is a real flag, not an --enable-feature entry). otlp_http/prometheus: endpoint: http://prometheus:9090/api/v1/otlp + sending_queue: + enabled: true + queue_size: 1000 + storage: file_storage + retry_on_failure: + enabled: true + max_elapsed_time: 300s service: - extensions: [bearertokenauth] + extensions: [bearertokenauth, file_storage] pipelines: logs: receivers: [otlp] diff --git a/config/prometheus.yaml b/config/prometheus.yaml index 5df8860..f16ba53 100644 --- a/config/prometheus.yaml +++ b/config/prometheus.yaml @@ -25,3 +25,20 @@ scrape_configs: - job_name: node static_configs: - targets: ["node-exporter:9100"] + + # Self-metrics of the remaining stack services, so TargetDown covers them too. + - job_name: grafana + static_configs: + - targets: ["grafana:3000"] + + - job_name: loki + static_configs: + - targets: ["loki:3100"] + + - job_name: tempo + static_configs: + - targets: ["tempo:3200"] + + - job_name: alertmanager + static_configs: + - targets: ["alertmanager:9093"] diff --git a/config/tempo.yaml b/config/tempo.yaml index 245ef57..d251b60 100644 --- a/config/tempo.yaml +++ b/config/tempo.yaml @@ -39,6 +39,9 @@ overrides: defaults: metrics_generator: processors: [service-graphs, span-metrics] + # Cardinality backstop: a sender with unrouted span names (raw paths) + # would otherwise mint Prometheus series without bound. + max_active_series: 50000 compaction: block_retention: 168h # 7 days — traces are bulky, tune to disk budget diff --git a/justfile b/justfile index 7b6ce64..51e2d7d 100644 --- a/justfile +++ b/justfile @@ -8,15 +8,25 @@ backup_mounts := "-v monitoring_grafana_data:/data/grafana -v monitoring_prometh default: @just --list +# The collector's queue volume must be writable by the image's uid 10001, but +# a fresh named volume is root-owned and the image is distroless (no chown at +# startup possible). Idempotent, so every up-path just runs it. +_queue-volume: + @docker volume create monitoring_otel_queue > /dev/null + @docker run --rm --network none -v monitoring_otel_queue:/q alpine:3.24 chown 10001:10001 /q + # Core stack (no tunnel; Grafana at http://localhost:3000) -up: +up: _queue-volume docker compose up -d # Core stack + Cloudflare Tunnel (production; needs CLOUDFLARE_TUNNEL_TOKEN). # Refuses to expose the stack with the documented default credentials. -up-tunnel: +up-tunnel: _queue-volume @[ "${OTLP_AUTH_TOKEN:-}" != "local-dev-token" ] || { echo "error: OTLP_AUTH_TOKEN is still the local default; generate one (openssl rand -hex 32) before exposing ingestion" >&2; exit 1; } @[ "${GRAFANA_ADMIN_PASSWORD:-}" != "change-me" ] || { echo "error: GRAFANA_ADMIN_PASSWORD is still the documented default; change it before exposing Grafana" >&2; exit 1; } + @[ "${GRAFANA_ROOT_URL:-}" != "http://localhost:3000" ] || { echo "error: GRAFANA_ROOT_URL is still the localhost default; set it to the tunnel hostname or every absolute URL Grafana generates breaks" >&2; exit 1; } + @[ "${GRAFANA_COOKIE_SECURE:-false}" = "true" ] || { echo "error: GRAFANA_COOKIE_SECURE must be true when Grafana is served over HTTPS; set it in .env" >&2; exit 1; } + @[ -n "${HEARTBEAT_URL:-}" ] || echo "WARNING: HEARTBEAT_URL is empty; the stack goes live without a dead-man's switch" >&2 docker compose -f compose.yml -f compose.tunnel.yml up -d down: @@ -24,19 +34,25 @@ down: # Core stack + a demo telemetry source (see compose.demo.yml), then look at # Grafana: http://localhost:3000 -demo: +demo: _queue-volume docker compose -f compose.yml -f compose.demo.yml up -d --build +# Build the demo image without starting anything. Used by CI to catch a broken +# demo app before it merges. +demo-build: + docker compose -f compose.yml -f compose.demo.yml build + # Remove only the demo services; the core stack keeps running. demo-down: docker compose -f compose.yml -f compose.demo.yml rm -sf demo-api demo-load logs service="": - docker compose logs -f {{service}} + docker compose -f compose.yml -f compose.demo.yml logs -f {{service}} ps: - docker compose ps + docker compose -f compose.yml -f compose.demo.yml ps +# Targets the base stack only; demo services are recreated with `just demo`. restart service: docker compose restart {{service}} @@ -45,7 +61,7 @@ pull: # Tail a service's logs as JSON, decoded. Useful before Grafana is set up. tail service: - docker compose logs -f --no-log-prefix {{service}} | jq -R 'fromjson? // .' + docker compose -f compose.yml -f compose.demo.yml logs -f --no-log-prefix {{service}} | jq -R 'fromjson? // .' # Validate everything. All validators run in containers — no host installs. # promtool/otelcol/amtool images are read from compose.yml so they can't @@ -57,14 +73,18 @@ check: docker run --rm -v ./config/prometheus.yaml:/etc/prometheus/prometheus.yaml:ro -v ./config/alerts:/etc/prometheus/alerts:ro --entrypoint promtool $(docker compose config --images | grep prom/prometheus) check config /etc/prometheus/prometheus.yaml docker run --rm -e OTLP_AUTH_TOKEN=dummy -v ./config/otel-collector.yaml:/etc/otelcol/config.yaml:ro $(docker compose config --images | grep opentelemetry-collector) validate --config=/etc/otelcol/config.yaml docker run --rm -v ./config/alertmanager.yaml:/etc/alertmanager/alertmanager.yaml:ro --entrypoint /bin/amtool $(docker compose config --images | grep prom/alertmanager) check-config /etc/alertmanager/alertmanager.yaml - docker run --rm -v .:/code:ro pipelinecomponents/yamllint:0.35.13 yamllint -d '{extends: relaxed, ignore: [.git/, backups/, infra/.terraform/]}' . - docker run --rm -v .:/repo:ro -w /repo rhysd/actionlint:1.7.12 -color - docker run --rm -v ./infra:/infra:ro -w /infra ghcr.io/opentofu/opentofu:1.12.3 fmt -check + docker run --rm --network none -v ./config/loki.yaml:/etc/loki/loki.yaml:ro $(docker compose config --images | grep grafana/loki) -config.file=/etc/loki/loki.yaml -verify-config + docker run --rm --network none -v ./config/tempo.yaml:/etc/tempo/tempo.yaml:ro $(docker compose config --images | grep grafana/tempo) -config.file=/etc/tempo/tempo.yaml -config.verify=true + docker run --rm --network none -v .:/code:ro pipelinecomponents/yamllint:0.35.13 yamllint -d '{extends: relaxed, ignore: [.git/, backups/, infra/.terraform/]}' . + docker run --rm --network none -v .:/repo:ro -w /repo rhysd/actionlint:1.7.12 -color + docker run --rm --network none -v ./infra:/infra:ro -w /infra ghcr.io/opentofu/opentofu:1.12.3 fmt -check docker run --rm -v ./dashboards:/dashboards:ro ghcr.io/jqlang/jq:1.8.1 empty $(ls dashboards/*.json | sed 's|^dashboards|/dashboards|') # Full OpenTofu validation (downloads the provider, so not part of `check`). +# Runs against a copy of the sources only: state and tfvars never enter the +# container, which needs network access to fetch the provider. infra-validate: - docker run --rm --entrypoint sh -v ./infra:/infra -w /infra ghcr.io/opentofu/opentofu:1.12.3 -c 'tofu init -backend=false -input=false >/dev/null && tofu validate' + @d=$(mktemp -d) && cp infra/main.tf infra/.terraform.lock.hcl "$d"/ && docker run --rm --entrypoint sh -v "$d":/src:ro ghcr.io/opentofu/opentofu:1.12.3 -c 'mkdir /work && cp /src/main.tf /src/.terraform.lock.hcl /work && cd /work && tofu init -backend=false -input=false >/dev/null && tofu validate'; rc=$?; rm -rf "$d"; exit $rc # Format YAML in place (needs yamlfmt on the host; optional). fmt: @@ -89,10 +109,12 @@ restore file: docker run --rm {{backup_mounts}} -v {{absolute_path(file)}}:/backup.tar.gz:ro alpine:3.24 sh -c 'for d in /data/*; do find "$d" -mindepth 1 -delete; done && tar xzf /backup.tar.gz -C /data' @echo "Restored {{file}} — run 'just up' to start the stack." -# Boot the core stack, wait until Grafana reports healthy, and fail if any -# service is crash-looping. Used by CI. -smoke: +# Boot the core stack, wait until Grafana reports healthy, assert every +# dashboard in dashboards/ actually provisioned (Grafana skips broken ones +# silently), and fail if any service is crash-looping. Used by CI. +smoke: _queue-volume docker compose up -d n=0; until curl -sf http://localhost:3000/api/health >/dev/null; do n=$((n+3)); [ $n -ge 120 ] && { echo "Grafana not healthy after 120s" >&2; exit 1; }; sleep 3; done + @for uid in $(docker run --rm --network none -v ./dashboards:/dashboards:ro ghcr.io/jqlang/jq:1.8.1 -r .uid $(ls dashboards/*.json | sed 's|^dashboards|/dashboards|')); do n=0; until curl -sf -u "admin:${GRAFANA_ADMIN_PASSWORD}" "http://localhost:3000/api/dashboards/uid/$uid" >/dev/null; do n=$((n+3)); [ $n -ge 60 ] && { echo "error: dashboard $uid was not provisioned" >&2; exit 1; }; sleep 3; done; done @[ -z "$(docker compose ps -q --status=restarting --status=exited)" ] || { echo "error: services not running:" >&2; docker compose ps >&2; exit 1; } @echo "Stack healthy" From b0579ab87caa25fb2a90d586982bc34fbf9ff367 Mon Sep 17 00:00:00 2001 From: Simon van Lierde Date: Thu, 20 Aug 2026 14:30:29 +0000 Subject: [PATCH 05/85] ci: validate infra and demo build, pin actions to SHAs - run just infra-validate; tofu fmt alone let invalid resources merge green - build the demo image so a broken demo app fails CI - pin actions/checkout and setup-just to commit SHAs --- .github/workflows/ci.yml | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 982f69c..2975e06 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -14,16 +14,18 @@ jobs: check: runs-on: ubuntu-latest steps: - - uses: actions/checkout@v7 - - uses: extractions/setup-just@v4 + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + - uses: extractions/setup-just@53165ef7e734c5c07cb06b3c8e7b647c5aa16db3 # v4.0.0 - run: cp .env.example .env - run: just check + - run: just infra-validate + - run: just demo-build smoke: runs-on: ubuntu-latest steps: - - uses: actions/checkout@v7 - - uses: extractions/setup-just@v4 + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + - uses: extractions/setup-just@53165ef7e734c5c07cb06b3c8e7b647c5aa16db3 # v4.0.0 - run: cp .env.example .env - run: just smoke - run: just down From d00b1c4ae2aabe0d655155038d474203e3df63b5 Mon Sep 17 00:00:00 2001 From: Simon van Lierde Date: Thu, 20 Aug 2026 14:30:47 +0000 Subject: [PATCH 06/85] refactor: drop redundant dashboard, dead file, and default-restating config - delete relab-api.json; the \$service pickers on Service Health and Logs Overview already cover it and scale to future services - delete demo/__init__.py, never copied into the image or imported - allowUiUpdates: false, the :ro dashboards mount made UI saves fail anyway; drop provisioning keys that restate defaults --- config/grafana/dashboards.yaml | 4 +- dashboards/relab-api.json | 143 --------------------------------- demo/__init__.py | 1 - 3 files changed, 1 insertion(+), 147 deletions(-) delete mode 100644 dashboards/relab-api.json delete mode 100644 demo/__init__.py diff --git a/config/grafana/dashboards.yaml b/config/grafana/dashboards.yaml index dacd0fa..2b5d842 100644 --- a/config/grafana/dashboards.yaml +++ b/config/grafana/dashboards.yaml @@ -4,9 +4,7 @@ providers: - name: default folder: "" type: file - disableDeletion: false - allowUiUpdates: true + allowUiUpdates: false updateIntervalSeconds: 30 options: path: /var/lib/grafana/dashboards - foldersFromFilesStructure: true diff --git a/dashboards/relab-api.json b/dashboards/relab-api.json deleted file mode 100644 index b0abee7..0000000 --- a/dashboards/relab-api.json +++ /dev/null @@ -1,143 +0,0 @@ -{ - "title": "RELab — API", - "uid": "relab-api", - "schemaVersion": 39, - "version": 1, - "refresh": "30s", - "time": { "from": "now-1h", "to": "now" }, - "graphTooltip": 1, - "tags": ["relab"], - "panels": [ - { - "type": "row", - "title": "Logs", - "collapsed": false, - "gridPos": { "h": 1, "w": 24, "x": 0, "y": 0 }, - "id": 1 - }, - { - "type": "timeseries", - "title": "Log Rate by Level", - "gridPos": { "h": 8, "w": 12, "x": 0, "y": 1 }, - "id": 2, - "datasource": { "type": "loki", "uid": "loki" }, - "targets": [ - { - "expr": "sum by (detected_level) (rate({service_name=\"relab-api\"}[$__auto]))", - "legendFormat": "{{detected_level}}", - "refId": "A" - } - ], - "fieldConfig": { - "defaults": { - "unit": "reqps", - "custom": { "lineWidth": 2, "fillOpacity": 10 } - }, - "overrides": [ - { - "matcher": { "id": "byName", "options": "error" }, - "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "red" } }] - }, - { - "matcher": { "id": "byName", "options": "critical" }, - "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "dark-red" } }] - }, - { - "matcher": { "id": "byName", "options": "warning" }, - "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "orange" } }] - }, - { - "matcher": { "id": "byName", "options": "info" }, - "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "blue" } }] - } - ] - }, - "options": { - "tooltip": { "mode": "multi", "sort": "desc" }, - "legend": { "displayMode": "list", "placement": "bottom" } - } - }, - { - "type": "logs", - "title": "Log Stream", - "gridPos": { "h": 8, "w": 12, "x": 12, "y": 1 }, - "id": 3, - "datasource": { "type": "loki", "uid": "loki" }, - "targets": [ - { - "expr": "{service_name=\"relab-api\"}", - "refId": "A" - } - ], - "options": { - "dedupStrategy": "none", - "showLabels": false, - "wrapLogMessage": true, - "prettifyLogMessage": false, - "enableLogDetails": true, - "sortOrder": "Descending" - } - }, - { - "type": "logs", - "title": "Errors", - "gridPos": { "h": 7, "w": 24, "x": 0, "y": 9 }, - "id": 6, - "datasource": { "type": "loki", "uid": "loki" }, - "targets": [ - { - "expr": "{service_name=\"relab-api\"} | detected_level=~\"error|critical\"", - "refId": "A" - } - ], - "options": { - "dedupStrategy": "none", - "showLabels": false, - "wrapLogMessage": true, - "enableLogDetails": true, - "sortOrder": "Descending" - } - }, - { - "type": "row", - "title": "Traces", - "collapsed": false, - "gridPos": { "h": 1, "w": 24, "x": 0, "y": 16 }, - "id": 4 - }, - { - "type": "traces", - "title": "Recent Traces", - "gridPos": { "h": 12, "w": 24, "x": 0, "y": 17 }, - "id": 5, - "datasource": { "type": "tempo", "uid": "tempo" }, - "targets": [ - { - "queryType": "traceql", - "query": "{resource.service.name=\"relab-api\"}", - "refId": "A", - "tableType": "traces" - } - ], - "options": { "frameType": "TraceqlSearch" } - }, - { - "type": "traces", - "title": "Failed Requests", - "gridPos": { "h": 10, "w": 24, "x": 0, "y": 29 }, - "id": 7, - "datasource": { "type": "tempo", "uid": "tempo" }, - "targets": [ - { - "queryType": "traceql", - "query": "{resource.service.name=\"relab-api\" && status=error}", - "refId": "A", - "tableType": "traces" - } - ], - "options": { "frameType": "TraceqlSearch" } - } - ], - "templating": { "list": [] }, - "annotations": { "list": [] } -} diff --git a/demo/__init__.py b/demo/__init__.py deleted file mode 100644 index fd3aec1..0000000 --- a/demo/__init__.py +++ /dev/null @@ -1 +0,0 @@ -"""Demo telemetry source for the monitoring stack (see compose.demo.yml).""" From 3691af93985edbe3d818fd94afbabd75b214a780 Mon Sep 17 00:00:00 2001 From: Simon van Lierde Date: Thu, 20 Aug 2026 14:31:25 +0000 Subject: [PATCH 07/85] build(demo): bump dependencies, pin python patch, drop root - fastapi 0.141.1, uvicorn 0.52.4, otel 1.44.0/0.65b0, uv 0.12.1 - python:3.14.7-slim in both stages for reproducible builds - run as nobody --- demo/Dockerfile | 9 +++++---- demo/pyproject.toml | 10 +++++----- 2 files changed, 10 insertions(+), 9 deletions(-) diff --git a/demo/Dockerfile b/demo/Dockerfile index 5b429ee..103be01 100644 --- a/demo/Dockerfile +++ b/demo/Dockerfile @@ -1,13 +1,14 @@ -FROM python:3.14-slim AS builder -COPY --from=ghcr.io/astral-sh/uv:0.11.26 /uv /bin/ +FROM python:3.14.7-slim AS builder +COPY --from=ghcr.io/astral-sh/uv:0.12.1 /uv /bin/ WORKDIR /app COPY pyproject.toml . RUN uv pip install --system --no-cache -r pyproject.toml -FROM python:3.14-slim +FROM python:3.14.7-slim COPY --from=builder /usr/local/lib/python3.14/site-packages /usr/local/lib/python3.14/site-packages COPY --from=builder /usr/local/bin /usr/local/bin WORKDIR /app -COPY app.py . +COPY --chown=nobody:nogroup app.py . +USER nobody CMD ["opentelemetry-instrument", "uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8000"] diff --git a/demo/pyproject.toml b/demo/pyproject.toml index 7a86bbc..aa3ef02 100644 --- a/demo/pyproject.toml +++ b/demo/pyproject.toml @@ -3,9 +3,9 @@ name = "demo" version = "0.1.0" requires-python = ">=3.14" dependencies = [ - "fastapi==0.139.0", - "uvicorn==0.50.0", - "opentelemetry-distro==0.64b0", - "opentelemetry-exporter-otlp==1.43.0", - "opentelemetry-instrumentation-fastapi==0.64b0", + "fastapi==0.141.1", + "uvicorn==0.52.4", + "opentelemetry-distro==0.65b0", + "opentelemetry-exporter-otlp==1.44.0", + "opentelemetry-instrumentation-fastapi==0.65b0", ] From ffbe398e17cb001b3904cf531a645d883b26e6b7 Mon Sep 17 00:00:00 2001 From: Simon van Lierde Date: Thu, 20 Aug 2026 14:32:29 +0000 Subject: [PATCH 08/85] docs: catch up with the review fixes - README: prerequisites, accurate volume list and just-check coverage, dashboard JSON editing workflow, up-tunnel guard description - RUNBOOK: durable queue and just-up volume init, memory ceilings, JWT access setup with the aud-pinning caveat, manually-tracked tool images - ONBOARDING: loki's new label surface (service_name only) - CHANGELOG: Unreleased entry covering the six weeks since 0.2.0 and this batch --- CHANGELOG.md | 72 ++++++++++++++++++++++++++++++++++++++++++++++ README.md | 34 ++++++++++++++++++---- docs/ONBOARDING.md | 15 ++++++---- docs/RUNBOOK.md | 37 +++++++++++++++++++++--- 4 files changed, 143 insertions(+), 15 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index e9d0c34..4ca187e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,6 +3,78 @@ Notable changes to this stack. Format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/); versions follow [SemVer](https://semver.org/). +## [Unreleased] + +A hardening pass over the whole stack: buffering that survives a restart, +self-monitoring that covers every service, tighter container defaults, and a +handful of queries that had been quietly measuring the wrong thing. + +### Added + +- **Durable export queue**: the collector's send queues are file-backed on a + new `otel_queue` volume, so telemetry buffered during a backend outage + survives a collector restart. `just up` (and `demo`, `smoke`, `up-tunnel`) + prepares the volume's ownership; it is deliberately not backed up. +- **Full-stack scraping**: Prometheus now scrapes Grafana, Loki, Tempo, and + Alertmanager as well, so `TargetDown` covers every service. +- **Per-user Grafana logins**, optional: `GRAFANA_JWT_AUTH` plus + `CF_ACCESS_TEAM_DOMAIN` make Grafana verify the Cloudflare Access JWT + instead of everyone sharing the admin password. +- **Memory ceilings** (`mem_limit`) on every service, sized from observed + usage, so one runaway component cannot OOM the host. +- **Wider validation**: `just check` also verifies the Alertmanager, Loki, + and Tempo configs and OpenTofu formatting; `just smoke` asserts every + dashboard actually provisioned; CI additionally runs `just infra-validate` + and a new `just demo-build`. +- Dependabot now watches the Cloudflare provider in `infra/`, and the runbook + covers OpenTofu-managed tunnel and Access changes. + +### Changed + +- **Loki indexes only `service.name`.** Everything else, `service.instance.id` + included, is structured metadata now — one stream per service instead of one + per sender restart. Existing streams keep their old labels until they age + out (30 days). +- **Dashboards are provisioned, not editable**: `dashboards/*.json` is mounted + read-only and UI saves are off, making the files the source of truth. +- The collector's `memory_limiter` is sized in absolute MiB against the + container limit, and the queue and retry settings behind the runbook's + "buffers for five minutes" are pinned rather than inherited from upstream. +- Tempo's metrics generator is capped at 50k active series, so a sender with + unrouted span names cannot mint Prometheus series without bound. +- Dropped the `relab-api` dashboard: the `$service` picker on Service Health + and Logs Overview covers it. +- README rewritten for a broader CML audience. +- Image bumps: Grafana 13.1.4, Loki 3.7.6, Tempo 3.0.3, Prometheus 3.13.2, + the collector 0.156.0, Alertmanager 0.33.1, node-exporter 1.12.1, + cloudflared 2026.8.2, and the demo's Python dependencies. + +### Fixed + +- **Trace links from the latency panel**: the Prometheus datasource pointed + exemplars at a `trace_id` label, but span-metrics exemplars carry `traceID`, + so clicking a dot resolved to nothing. +- **`HighErrorRate` measured the wrong denominator**: it counted all spans, so + internal child spans diluted the ratio well below the real request error + rate. It now filters server spans, matching the Service Health dashboard. +- The demo load generator hit a never-failing endpoint half the time, so the + advertised one-in-ten error rate showed up as one in twenty. +- The Infrastructure Logs dashboard queried `env` and `service` labels this + stack does not set, and so was always empty. +- Grafana and the collector now wait for Prometheus to report ready, instead + of racing it on a cold start. +- The S3 overlay documents the `-config.expand-env=true` that Loki and Tempo + need before `${...}` in their configs expands at all. + +### Security + +- `no-new-privileges` on every service; the demo image runs as `nobody`. +- `GRAFANA_COOKIE_SECURE` marks the session cookie Secure (with strict + SameSite), and `just up-tunnel` refuses to expose the stack without it, a + non-localhost `GRAFANA_ROOT_URL`, and non-default credentials. +- GitHub Actions are pinned to commit SHAs, and `just infra-validate` runs + against a copy of the sources so state and tfvars never enter the container. + ## [0.2.0] - 2026-07-05 Makes the stack operable, not just runnable: authentication, self-monitoring, diff --git a/README.md b/README.md index 3f761d5..35f2793 100644 --- a/README.md +++ b/README.md @@ -13,6 +13,10 @@ queryable side by side. ## Try it in one command +You need Docker with the [Compose plugin](https://docs.docker.com/compose/install/) +and [`just`](https://github.com/casey/just#installation); everything else runs +in containers. + ```sh cp .env.example .env just demo @@ -30,7 +34,8 @@ purpose. Give it a minute, then open Grafana at - **Dashboards → Logs Overview** — log volume by service and level, an error feed, and a live tail of everything arriving over OTLP. - **Alerting → Alert rules** — the stack-health and error-rate rules Prometheus - is evaluating. + is evaluating. `HighErrorRate` trips on the demo service after five minutes: + one request in ten failing is twice the 5% threshold. ![Service Health (RED) dashboard](docs/img/service-health.png) @@ -83,6 +88,12 @@ just check # validate every config in the repo Grafana: (admin / whatever you set). +`up-tunnel` refuses to run until the settings that only matter once the stack +is reachable are real: a generated `OTLP_AUTH_TOKEN`, a changed +`GRAFANA_ADMIN_PASSWORD`, `GRAFANA_ROOT_URL` pointing at the tunnel hostname +rather than localhost, and `GRAFANA_COOKIE_SECURE=true` so the session cookie +is marked Secure. An empty `HEARTBEAT_URL` only warns. + In production the stack sits behind a Cloudflare Tunnel, and that edge is code too. The tunnel, its hostnames, DNS, and the Cloudflare Access rule that puts an email one-time-PIN in front of Grafana all live in `infra/` as a small @@ -91,10 +102,11 @@ OpenTofu configuration. Applying it produces the `CLOUDFLARE_TUNNEL_TOKEN` that [infra/main.tf](infra/main.tf). `just check` validates compose files, Prometheus config and alert rules, the -collector config, YAML, workflows, and dashboard JSON. Every validator runs in -a pinned container, so nothing needs to be installed on the host. CI runs the -same command on every push and pull request, plus a smoke test that boots the -stack and waits for Grafana to come up healthy. +collector, Alertmanager, Loki and Tempo configs, YAML, workflows, OpenTofu +formatting, and dashboard JSON. Every validator runs in a pinned container, so +nothing needs to be installed on the host. CI runs the same command on every +push and pull request, plus a smoke test that boots the stack, waits for +Grafana to come up healthy, and checks that every dashboard provisioned. ## Sending telemetry from a project @@ -124,7 +136,10 @@ optional; leave them unset and alerts are simply visible in Grafana. ## Storage Everything persists to local Docker volumes (`loki_data`, `tempo_data`, -`prometheus_data`, `grafana_data`). When local disk stops fitting, Loki and +`prometheus_data`, `grafana_data`, `alertmanager_data`), all of which `just +backup` captures. A sixth, `otel_queue`, holds the collector's on-disk export +queue — seconds of in-flight telemetry, worthless by the time anyone restores, +so it is deliberately left out. When local disk stops fitting, Loki and Tempo can move to any S3-compatible object store (Backblaze B2, Cloudflare R2, Hetzner, MinIO); `compose.storage-s3.yml` documents the concrete shape of that change. @@ -149,6 +164,13 @@ docs/ # runbook, onboarding templates, ADRs, screenshots infra/ # OpenTofu: Cloudflare tunnel, ingress routes, DNS ``` +`dashboards/*.json` is the source of truth for what Grafana shows: the +directory is mounted read-only and saving from the UI is disabled, so a change +made in the browser lasts until the page is reloaded. Edit the JSON and +provisioning picks it up within about 30 seconds; to keep something built +interactively, export the dashboard as JSON (or copy a single panel's JSON out +of *Inspect → Panel JSON*) and paste it back into the file. + ## Documentation | Document | What it covers | diff --git a/docs/ONBOARDING.md b/docs/ONBOARDING.md index e6e0a88..5d5e505 100644 --- a/docs/ONBOARDING.md +++ b/docs/ONBOARDING.md @@ -25,11 +25,16 @@ directly. - **`service.name`** is required: one stable name per deployable unit (`relab-api`, not `relab-api-prod-2`). Dashboards key on it. - **`env`** is `prod`, `staging`, or `dev`, set as a resource attribute. -- **Keep labels low-cardinality.** Loki and Prometheus index labels, and - every distinct value creates a new stream or series. User IDs, request - IDs, and timestamps therefore don't belong in resource attributes or log - labels. Put them in the log line or in span attributes instead — you can - still filter on them at query time, without the storage blowing up. +- **Keep labels low-cardinality.** Prometheus turns every distinct label + value into a series. User IDs, request IDs, and timestamps therefore + don't belong in resource attributes or metric labels. Put them in the log + line or in span attributes instead — you can still filter on them at + query time, without the storage blowing up. +- **In Loki, `service.name` is the only index label.** Every other + attribute, `service.instance.id` included, is stored as structured + metadata, so a query starts from the stream selector and filters after + it: `{service_name="my-service"} | env="prod"`. Streams written before + this change keep their old labels until they age out after 30 days. Traces are the most valuable signal to send. Tempo derives request-rate, error-rate, and duration ("RED") metrics from them, so a service that diff --git a/docs/RUNBOOK.md b/docs/RUNBOOK.md index 1d1e66f..b410c9f 100644 --- a/docs/RUNBOOK.md +++ b/docs/RUNBOOK.md @@ -19,7 +19,9 @@ just restart ``` Two alerts point here. `TargetDown` fires after two minutes when -Prometheus can't scrape the collector, node-exporter, or itself. +Prometheus can't scrape a target, and it scrapes every service in the +stack — collector, node-exporter, Grafana, Loki, Tempo, Alertmanager, and +itself — so the alert names whichever one went quiet. `OtelExportFailures` means the collector is up but a backend is rejecting its data, so look at that backend's logs, not the collector's. @@ -44,6 +46,11 @@ Stack Health, then either free space or shorten a retention window coming back, the durable fix is moving Loki and Tempo to object storage — see `compose.storage-s3.yml`. +Memory is bounded per service instead: every service carries a `mem_limit` +in `compose.yml`, sized from observed usage with headroom so that one +runaway component cannot take the host down with it. If a component +legitimately grows into its ceiling, raise it there. + ## Backup and restore ```sh @@ -62,8 +69,12 @@ write-ahead log. Two practical notes: - The tarball covers the docker volumes and nothing else. The OpenTofu state for the Cloudflare edge is not in it — see below. - During the pause the collector keeps accepting telemetry and buffers it - for about five minutes. A backup that takes longer than that will drop - data, so on large volumes run it at a quiet hour. + for five minutes — the `retry_on_failure.max_elapsed_time` pinned on + every exporter in `config/otel-collector.yaml`, not an upstream default + that can move under you. The queue is file-backed, so restarting the + collector inside that window keeps the buffer; a backup that runs longer + than five minutes still drops data, so on large volumes run it at a + quiet hour. How much history you can lose equals how often you run it. A daily cron on the host is the intended setup. @@ -122,6 +133,15 @@ cd infra && tofu apply so for an urgent revocation also revoke the session in Zero Trust. At least one address has to remain — the variable's validation rejects an empty list, which would lock everyone out of Grafana. +- **Per-user Grafana logins:** by default everyone who clears Access then + shares the one admin password. Setting `GRAFANA_JWT_AUTH=true` and + `CF_ACCESS_TEAM_DOMAIN=` in `.env` makes Grafana verify the Access + JWT instead, so each address signs in as itself and new ones land on the + org's default role (Viewer). The JWK set is team-wide, so if the Zero + Trust team fronts more than one Access application, also pin + `GF_AUTH_JWT_EXPECT_CLAIMS` in `compose.yml` to this app's `aud` tag — + otherwise a token minted for any other app in the team is accepted here + too. - **Adding a hostname:** add an `ingress` entry pointing at the service's container port, plus a matching `cloudflare_dns_record`. The catch-all `http_status:404` entry stays last, or it swallows everything after it. @@ -147,4 +167,13 @@ hashes in `.terraform.lock.hcl`, so check the branch out and run against the real account — validation proves the syntax parses, only a plan proves the provider still maps the config to the same resources. -After merging: on the host, `git pull && just pull && just up`. +Nothing watches the tool images pinned in the `justfile` — yamllint, +actionlint, OpenTofu, jq, and the alpine that backup, restore, and the +queue-volume setup run in. No Dependabot ecosystem covers a justfile, so +those are bumped by hand. + +After merging: on the host, `git pull && just pull && just up`. Bring the +stack up through `just up` rather than `docker compose up -d`: the recipe +first chowns the collector's queue volume to uid 10001, and on a host where +that volume is new a raw compose up leaves it root-owned and the collector +crash-looping on a queue directory it cannot write. From 8ebea55b92ebb3ac618e4a635db74208c76865e4 Mon Sep 17 00:00:00 2001 From: Simon van Lierde Date: Thu, 20 Aug 2026 17:46:04 +0200 Subject: [PATCH 09/85] docs: record the hub-and-spoke target and take over the Relab handover - add ADR 0002: hub-and-spoke observability for CML projects, superseding ADR 0001's span-metrics RED decision - move the August 2026 review handover in from the Relab repo as docs/HANDOVER.md, with an owner/review-date tripwire - point the README at both --- README.md | 4 +- docs/HANDOVER.md | 256 +++++++++++++++++++ docs/adr/0002-hub-and-spoke-observability.md | 95 +++++++ 3 files changed, 354 insertions(+), 1 deletion(-) create mode 100644 docs/HANDOVER.md create mode 100644 docs/adr/0002-hub-and-spoke-observability.md diff --git a/README.md b/README.md index 35f2793..9872684 100644 --- a/README.md +++ b/README.md @@ -75,7 +75,9 @@ everything talks over the compose network and Grafana is at `localhost:3000`. The stack deliberately runs on a single host. At CML's telemetry volume, distributed ingestion would add operational weight for no gain. The reasoning, and the alternatives we considered, are recorded in -[ADR 0001](docs/adr/0001-observability-stack.md). +[ADR 0001](docs/adr/0001-observability-stack.md). The hub-and-spoke target for +serving multiple CML projects is [ADR 0002](docs/adr/0002-hub-and-spoke-observability.md), +and the open migration work toward it lives in [docs/HANDOVER.md](docs/HANDOVER.md). ## Run it for real diff --git a/docs/HANDOVER.md b/docs/HANDOVER.md new file mode 100644 index 0000000..7714aec --- /dev/null +++ b/docs/HANDOVER.md @@ -0,0 +1,256 @@ +# Handover: the central stack's half of the August 2026 review + +Owner: _unassigned — put a name here before doing anything else._ +Review date: _set one; if nobody has picked this up by then, the tripwire below holds._ + +Relab's side of an architecture review is done and merged (its repo carries the full +review in `deploy/MONITORING-DESIGN.md`; the target architecture is recorded here as +[ADR 0002](adr/0002-hub-and-spoke-observability.md)). This document is the actionable +half for THIS repo: what Relab now emits, what the central stack has to do with it, and +what should be deleted here. It is written to be actionable without reading the Relab +repo. + +**Delete this file once the work below is done.** It describes a transition, not a system. + +______________________________________________________________________ + +## Why this exists + +A backup container crash-looped 668 times over 19 hours. Every monitor read green the +whole time, because the only backup check was newest-snapshot age — which a crash loop +makes look *better* than healthy. It was noticed by ear, from fan noise. + +Nothing here is about adding tools. It is about the fact that several failures currently +have no detector at all, and one of them is the failure that already happened. + +______________________________________________________________________ + +## What Relab now emits + +One Grafana Alloy agent per deploy host, one OTLP/HTTP endpoint, one bearer token. No new +hostnames, no new credentials, nothing to configure centrally per host. + +| Signal | Source | Notes | +| ---------------------------- | --------------------------------------- | -------------------------------------------------------------------------------------------- | +| Application traces + metrics | the API's own OTel SDK | `service.name=relab-api` | +| Container logs | Alloy `loki.source.docker` | every container's stdout, labelled by Compose service | +| Host metrics | Alloy `prometheus.exporter.unix` | CPU, memory, load, disk, network, `hwmon` | +| Per-container metrics | Alloy `prometheus.exporter.cadvisor` | CPU, memory, network, disk I/O, `container_start_time_seconds`, `container_oom_events_total` | +| GPU metrics | `nvidia_gpu_exporter`, scraped by Alloy | opt-in overlay, present only on GPU hosts | + +Every signal carries four identity labels: **`project`**, **`env`**, **`service.name`**, +**`host_name`**. Enforced by the agent config, so a host cannot omit them. + +Application logs are **no longer** exported by the API's SDK. Alloy ships that container's +stdout, so exporting them twice stored every line twice in two shapes. The stdout path is +the one kept, because it also carries what the SDK cannot report: the SDK's own export +failures. + +______________________________________________________________________ + +## The keystone alert, and the four behind it + +Nothing on the Relab side can detect its own absence. These rules are the deliverable — +without them the telemetry is a dashboard, not monitoring. + +| Rule | Shape | For | Catches | +| ---------------------------- | --------------------------------------------------------- | --- | -------------------------------------------------------------------------------------------------------------------- | +| **`ProjectTelemetrySilent`** | no logs or metrics with `project=X, env=Y` | 15m | Host down, Docker down, Alloy down, tunnel down, token rotated wrong, collector rejecting. **Write this one first.** | +| **`ContainerRestarting`** | `changes(container_start_time_seconds{name!=""}[1h]) > 3` | 10m | The 668-restart incident. A crash loop becomes louder than health instead of quieter. | +| `ContainerOOMKilled` | increase in `container_oom_events_total` | 0m | Distinct cause, distinct fix. | +| `HostDiskSpaceLow` | < 20% free, by `host_name` | 15m | Data already arrives; nothing reads it. | +| `OtelExportFailures` | collector send-failed rate > 0 | 5m | Backend rejecting data. | + +Keep the total under about ten rules. Past that the operator stops reading them. + +Only `ProjectTelemetrySilent` needs generating per project and environment. The rest are +generic once `host_name` and `project` labels are correct. + +For GPU hosts, three more: `nvidia_smi_gpu_recovery_action > 0` (the driver wants a +reset — the best single health alert), a thermal/power throttle flag, and XID faults with +an explicit code allowlist rather than every code, since most are application faults: + +```promql +time() - nvidia_smi_xid_last_timestamp_seconds{xid=~"48|62|64|74|79|95|119|120"} < 300 +``` + +XIDs are the GPU analogue of the crash-loop blind spot: a stuck kernel, an uncorrectable +memory fault or a card off the bus are invisible to utilisation graphs, and they are what +silently kills a twelve-hour training run. + +______________________________________________________________________ + +## Delete list for the monitoring repo + +- **Alertmanager**, its volume, its `url_file` entrypoint hack, its `amtool` check step and + its backup mount. Move every rule to Grafana-managed alert rules provisioned from YAML + under `/etc/grafana/provisioning/alerting/`. Three reasons, by weight: + + 1. **Grafana rules can query Loki; Prometheus rules cannot.** The alerts worth having are + log-shaped and metric-shaped at once. Splitting the rule engine means the most + valuable ones cannot be written at all. + 1. One place owns notification, instead of two answers to "who gets told". + 1. It deletes a container, a volume, a config file and a check step. + + Caveat that shapes the work: Grafana's built-in Alertmanager handles Grafana-managed + alerts only, and there is no documented endpoint for an external Prometheus to POST into + it. So the Prometheus rule files must genuinely *become* Grafana rules — this is not a + re-pointing of delivery. That is four rules; an afternoon. + + *The opposite choice is right if* you want alerting to survive Grafana being down. It + does not buy that today, because Alertmanager sits on the same host and dies with it. + The real answer to "alerting is down" is the heartbeat below. + +- **ONBOARDING templates 3 and 4** (Loki Docker driver, `loki.write` to a push hostname). + Both document a path that requires exposing Loki, which this stack deliberately does not + do because Loki has no authentication of its own. Relab followed template 3, and it cost + a day to undo. + +- **`compose.storage-s3.yml`** and **Tempo's span-metrics dependency** — neither is + load-bearing; move RED metrics onto native OTLP HTTP metrics instead. + +- **Grafana OnCall OSS** is not an option in 2026: maintenance mode March 2025, archived + read-only March 2026, paid Grafana Cloud IRM the only successor. If it comes up, that is + the answer. + +______________________________________________________________________ + +## The heartbeat, and why it is not optional + +Everything above travels through one collector over one tunnel. A dead host, a dead +collector, a broken tunnel and an expired token are indistinguishable from Grafana's point +of view: silence that looks like health. + +Relab's scheduled jobs each ping a per-job dead-man's switch (healthchecks.io) directly +from the host. That is the one signal that does not share fate with this stack, and it must +stay. On the central side, add a `Watchdog` rule (`vector(1)`, always firing) routed to a +heartbeat contact point on a short repeat interval — its *silence* is the alarm. + +healthchecks.io's free tier is exactly 20 checks. Relab uses three per environment; that is +the ceiling to plan against as projects onboard. + +______________________________________________________________________ + +## Dashboards: the actual scalability lever + +Today there is a per-project dashboard. N projects means N hand-maintained dashboards, and +that will stop this scaling long before storage does. + +Replace with three dashboards carrying a `project` template variable — *Service Health*, +*Logs*, *Host & Containers* — so a new project gets full dashboards the moment its first +telemetry lands, with nobody editing JSON. + +Import rather than author: + +- per-container resources: [15798](https://grafana.com/grafana/dashboards/15798-docker-monitoring/) + (revised 2025-07-12); [19792](https://grafana.com/grafana/dashboards/19792-cadvisor-dashboard/) + is the Compose-aware second choice. Avoid 10619 and 893 — high download counts, untouched + since 2019 and Grafana 4 respectively. Download counts measure inertia, not maintenance. +- GPU: [14574](https://grafana.com/grafana/dashboards/14574) (revised 2026-08-04) and its + multi-GPU companion 25547. Do **not** use the canonical DCGM dashboard 12239: last + revised 2021, and its panels lean on profiling fields consumer cards cannot produce. + +### The one experiment to run first + +Stock cAdvisor dashboards assume Prometheus *scraped* cAdvisor — they key on `job`, +`instance` and `name` as a scrape produces them. Relab's metrics arrive over OTLP, where +Prometheus reconstructs `job` from `service.name`, `instance` from `service.instance.id`, +and applies a translation strategy to metric names. + +Metric names are expected to survive untouched (already underscore-only, already +`_total`-suffixed, no OTLP unit set) but that could not be confirmed from documentation, +only inferred from translator behaviour. **Label values definitely change.** + +So: check whether `container_cpu_usage_seconds_total` exists in Prometheus with a `name` +label. Fifteen minutes, and it decides the wiring: + +- **Names survive** (expected): keep the single OTLP path, import 15798, repoint its two + template variables at `host_name`/`project`. One-time, ~20 minutes. One endpoint and one + credential is worth more than a dashboard's pristine defaults. +- **Names mangled**: do *not* reach for `otlp.translation_strategy: NoTranslation`, which + is experimental and documented with warnings. Route only the cAdvisor series natively + instead — that costs a second hostname and a second credential, so make it deliberately. + +______________________________________________________________________ + +## Two settings that are easy to miss + +- **`out_of_order_time_window: 30m`** on Prometheus. The official OTel guide requires it for + OTLP ingestion; without it, late batches are silently dropped. "Silently" is the operative + word. +- **Prometheus's OTLP receiver is documented as "not an efficient way of ingesting + samples"**, for "specific low-volume use cases". CML is exactly the case that sentence + carves out — a few hundred series per host at 30s. The deciding factor for revisiting is + volume: if a host exceeds a few thousand active series, move infrastructure metrics to + `remote_write` behind an authenticating proxy and leave app metrics on OTLP. + +______________________________________________________________________ + +## Onboarding a second project + +This is the part worth doing well: it is the difference between a stack that serves CML and +one that serves Relab. + +The central repo should publish: + +1. `templates/alloy/config.alloy` — one shared agent config, parameterised only by + environment variables. **Relab's `deploy/alloy/config.alloy` is this file already**, + minus two hardcoded `"relab"` strings. No project should ever edit it. +1. `templates/compose.telemetry.yml` — Relab's `compose.logging.alloy.yaml` is ~90% of it. +1. `templates/compose.telemetry.gpu.yml` — Relab's `compose.gpu.yaml`, likewise. +1. `templates/run_scheduled.sh` — the dead-man's-switch wrapper; Relab's is generic apart + from job names. +1. `templates/alerts/project.yaml.tmpl` — the rules above, with `{{project}}` and `{{env}}` + substituted. +1. `bootstrap.sh ` — renders the alert rules and reloads Grafana, creates the + healthchecks.io checks and prints their URLs, prints the `.env` block to paste on the + project host, and prints the `curl` that vendors the templates at a pinned tag. + +A new project's whole checklist then is: vendor two files, add six `.env` variables, include +the overlay, run `bootstrap.sh`. Under an hour, most of it waiting. + +One trap to design out: `ProjectTelemetrySilent` only exists for projects whose +`bootstrap.sh` was actually run. A project that ships telemetry but skipped bootstrap is +silently uncovered — the exact failure class the rule closes, one level up. Cheapest +countermeasure: one standing rule that fires on any series whose `project` label has no +matching provisioned rule group; failing that, make the onboarding doc say plainly that +bootstrap is what creates the safety net, not the telemetry. + +Take the templates from Relab, then **re-vendor Relab from them**, so Relab is proof the +path works rather than a special case that drifted. + +______________________________________________________________________ + +## Sequencing + +**Tripwire first:** the owner and review date at the top of this file are load-bearing. +Relab's local watchdog checks (service health, snapshot age, timer state) are scheduled +for deletion *only after* steps 1–2 below are live and verified — if nobody has picked +this work up by the review date, the watchdog stays, and the deletion item in Relab's +`deploy/MONITORING-DESIGN.md` §2.1 must not proceed on optimism. Deleting a weak local +signal before its central replacement exists trades a weak signal for none. + +1. `ProjectTelemetrySilent` — the keystone. Nothing else detects absence. +1. `ContainerRestarting` + `ContainerOOMKilled` — closes the incident that started this. +1. Run the metric-name experiment; import 15798 and 14574 accordingly. +1. `HostDiskSpaceLow`, `OtelExportFailures`, `Watchdog` heartbeat. +1. Grafana-managed alerting; delete Alertmanager. +1. Generic dashboards with a `project` variable. +1. Templates and `bootstrap.sh`; re-vendor Relab from them. + +Steps 1–2 convert this from telemetry into monitoring. Everything after is leverage. + +______________________________________________________________________ + +## Be skeptical of these + +This came from an architecture review that verified claims against current documentation and +flagged what it could not. Two are worth checking rather than trusting: + +- the Cloudflare Zero Trust free-plan seat count — the widely-cited figure appears only in + third-party posts, never in Cloudflare's own docs; +- the OTLP metric-name round trip, which is the experiment above. + +There is also **no official cadence recommendation** for `restic check --read-data-subset`; +the commonly repeated "1/12 monthly" is forum folklore. Pick a cadence, write down why, and +treat the number as arbitrary-but-declared. diff --git a/docs/adr/0002-hub-and-spoke-observability.md b/docs/adr/0002-hub-and-spoke-observability.md new file mode 100644 index 0000000..1f8c5fa --- /dev/null +++ b/docs/adr/0002-hub-and-spoke-observability.md @@ -0,0 +1,95 @@ +# ADR 0002: Hub-and-spoke observability for CML projects + +Date: 2026-08-20. Status: accepted (records the target architecture from the +August 2026 review; the migration work is tracked in [HANDOVER.md](../HANDOVER.md)). + +Supersedes one decision from ADR 0001: RED metrics move off Tempo's span-metrics +and onto the applications' native OTLP HTTP metrics. Everything else in ADR 0001 +stands. + +## Context + +This stack was built for RELab and must now serve multiple CML projects at very +different maturity levels, including GPU hosts for computer-vision work. One +part-time operator, Docker Compose everywhere, zero budget. The formative +incident: a backup container crash-looped 668 times over 19 hours while every +monitor read green — the failure modes that matter are the ones with no detector +at all. + +## Decision + +Three tiers, each owning distinct signals: + +- **Per-project host (spoke):** one Grafana Alloy agent per host — container + stdout, host metrics (node exporter), and container lifecycle/resources + (cAdvisor) — plus the application's own OTel SDK for traces and app metrics. + The agent config is one shared file published by this repo, parameterised only + by environment variables; no project ever edits it. systemd timers run + scheduled jobs (backups, checks) through a wrapper that pings a per-job + dead-man's switch. +- **Central host (hub):** this stack. One OTLP/HTTP endpoint, one bearer token, + no per-backend hostnames or credentials, ever. Grafana is the *single* home + for alert rules and notification (Alertmanager and Prometheus rule files go + away — Grafana-managed rules can query Loki, which the most valuable alerts + need). +- **Outside everything:** healthchecks.io as the per-job dead-man's switch, and + an external HTTP prober for public reachability. These are the only detectors + whose default state is alarm; everything else fails silent, and silence is + indistinguishable from health. + +Contracts that make it scale: + +- **Four identity labels on every signal**, enforced by the shared agent config: + `project`, `env`, `service.name`, `host_name` (Prometheus form; OTel form is + `host.name`). Cardinality rule: user ids, request ids and timestamps go in + bodies and span attributes, never labels. +- **Each signal has exactly one producer.** Alloy owns all container logs (SDK + log exporters stay off); native app metrics own RED; cAdvisor owns container + lifecycle; healthchecks.io owns "did the job run"; a host-local drift script + owns "is the deployed code the code we think"; nothing derives metrics from + logs or from traces. +- **`ProjectTelemetrySilent` per project/env is the keystone alert** — nothing + on a spoke can detect its own absence. Templated and provisioned by + `bootstrap.sh`, which is also what creates a project's healthchecks and prints + its `.env` block. Bootstrap is what creates the safety net, not the telemetry. +- **Onboarding is a copy, not a port:** vendor two template files at a pinned + tag, add six `.env` variables, include the overlay, run `bootstrap.sh`. A GPU + host is an ordinary host plus one opt-in overlay (`nvidia_gpu_exporter` + scraped by Alloy — not dcgm-exporter, whose profiling fields are + datacentre-only) and three GPU alert rules. + +## Alternatives considered + +- **Grafana Cloud free tier (no hub at all):** deletes this host, its backups + and its disk-full failure mode, and the free tier covers CML's volume. Rejected + on data protection, not economics: container logs cross the applications' + sanitization boundary (Postgres error lines can quote research and personal + data), and shipping them to a US-operated SaaS is a GDPR/university-policy + problem a department-run host does not have. Revisit only if that question is + formally cleared. +- **Per-project tokens and Loki multi-tenancy:** organisational controls for a + problem one operator does not have. The trigger for per-project tokens is the + first leaked-token incident; for multi-tenancy, the first dataset other CML + projects must not see. Both are a day of work, not a redesign. +- **Pushgateway for batch/ML jobs:** rejected on Prometheus's own guidance; + machine-level batch jobs use node_exporter's textfile collector plus a + dead-man's-switch check. +- **A second alerting engine, SLOs, paging rotations, per-project dashboards, + long retention:** all rejected — one operator, alert count capped around ten, + dashboards carry a `project` template variable instead of per-project copies. + +## Consequences + +- Tempo demotes to trace storage only; deleting its metrics-generator dependency + makes it disposable on its next breaking upgrade. +- The spokes' local watchdog checks (service health, snapshot age, timer state) + are deletable only after `ProjectTelemetrySilent` and the container-lifecycle + rules are live here — see the tripwire in [HANDOVER.md](../HANDOVER.md). +- Prometheus needs `out_of_order_time_window: 30m` for OTLP ingestion; without + it late batches drop silently. The OTLP receiver is documented as a + low-volume path — the revisit trigger is a host exceeding a few thousand + active series. +- Unverified at decision time: whether cAdvisor metric names survive the OTLP + round trip (a 15-minute experiment decides between importing dashboard 15798 + and adding a second ingestion path); Cloudflare free-plan Zero Trust seat + count. From a623e8fce1ce97ecbd5f43e5b1f956a04b5609b1 Mon Sep 17 00:00:00 2001 From: Simon van Lierde Date: Thu, 27 Aug 2026 04:35:16 +0000 Subject: [PATCH 10/85] fix: set OTLP out-of-order window, drop stale ONBOARDING reference --- config/prometheus.yaml | 6 ++++++ docs/ONBOARDING.md | 3 --- 2 files changed, 6 insertions(+), 3 deletions(-) diff --git a/config/prometheus.yaml b/config/prometheus.yaml index f16ba53..3572946 100644 --- a/config/prometheus.yaml +++ b/config/prometheus.yaml @@ -4,6 +4,12 @@ global: external_labels: origin: monitoring-host +storage: + tsdb: + # OTel's guide requires this for OTLP ingestion — without it, late + # batches are silently dropped. + out_of_order_time_window: 30m + rule_files: - /etc/prometheus/alerts/*.yaml diff --git a/docs/ONBOARDING.md b/docs/ONBOARDING.md index 5d5e505..83f25ef 100644 --- a/docs/ONBOARDING.md +++ b/docs/ONBOARDING.md @@ -98,9 +98,6 @@ logging: loki-external-labels: service={{.Name}},env=prod,host=myhost ``` -RELab's `compose.logging.loki.yml` overlay, auto-included when `LOKI_URL` -is set, is the reference implementation of this pattern. - ## Template 4 — host or file logs (Grafana Alloy) For log files that live outside containers. (Promtail is end-of-life; From e29ee45d3bd4d52cc01073a79224da1ab6220401 Mon Sep 17 00:00:00 2001 From: Simon van Lierde Date: Thu, 27 Aug 2026 04:35:23 +0000 Subject: [PATCH 11/85] feat: add ContainerRestarting and ContainerOOMKilled alerts --- config/alerts/stack.yaml | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/config/alerts/stack.yaml b/config/alerts/stack.yaml index bfe3e18..a8ca4a1 100644 --- a/config/alerts/stack.yaml +++ b/config/alerts/stack.yaml @@ -52,6 +52,26 @@ groups: summary: "Filesystem {{ $labels.mountpoint }} is over 80% full" description: "Loki and Tempo have no total-size cap, so this alert is the storage backstop. Free space or lower retention (see RUNBOOK)." + - name: container-lifecycle + rules: + - alert: ContainerRestarting + expr: changes(container_start_time_seconds{name!=""}[1h]) > 3 + for: 10m + labels: + severity: warning + annotations: + summary: "Container {{ $labels.name }} is crash-looping" + description: "{{ $labels.name }} on {{ $labels.host_name }} has restarted more than 3 times in the last hour." + + - alert: ContainerOOMKilled + expr: increase(container_oom_events_total[5m]) > 0 + for: 0m + labels: + severity: warning + annotations: + summary: "Container {{ $labels.name }} was OOM-killed" + description: "{{ $labels.name }} on {{ $labels.host_name }} hit an OOM kill in the last 5 minutes." + - name: service-red rules: # Server spans only, matching the Service Health dashboard. Without the From 01b77d864c29dcacc8f845653e346bf9ec539dd7 Mon Sep 17 00:00:00 2001 From: Simon van Lierde Date: Thu, 27 Aug 2026 04:35:29 +0000 Subject: [PATCH 12/85] feat: add ProjectTelemetrySilent keystone alert for relab --- config/alerts/projects.yaml | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) create mode 100644 config/alerts/projects.yaml diff --git a/config/alerts/projects.yaml b/config/alerts/projects.yaml new file mode 100644 index 0000000..16e0b7d --- /dev/null +++ b/config/alerts/projects.yaml @@ -0,0 +1,20 @@ +# Per-project "is anything arriving at all" alerts. One group per onboarded +# project — hand-written until step 7's bootstrap.sh generates these. +# Uses Prometheus's synthetic target_info{} (materialized from OTLP resource +# attributes on any received metric batch) rather than an app- or +# cAdvisor-specific metric name, so it doesn't depend on OTLP metric-name +# translation surviving. + +groups: + - name: project-telemetry-relab + rules: + - alert: ProjectTelemetrySilent + expr: absent(target_info{project="relab",env="production"}) + for: 15m + labels: + severity: critical + project: relab + env: production + annotations: + summary: "No telemetry from relab/production in 15m" + description: "Host down, Docker down, Alloy down, tunnel down, token rotated wrong, or the collector is rejecting relab's data. Nothing else on the Relab side can detect this on its own." From 0d8085cf53fd5aec19a51520f8a7f7381884e07e Mon Sep 17 00:00:00 2001 From: Simon van Lierde Date: Thu, 27 Aug 2026 05:00:39 +0000 Subject: [PATCH 13/85] fix: promote project, env and host.name onto every OTLP series --- config/prometheus.yaml | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/config/prometheus.yaml b/config/prometheus.yaml index 3572946..aaf142d 100644 --- a/config/prometheus.yaml +++ b/config/prometheus.yaml @@ -10,6 +10,13 @@ storage: # batches are silently dropped. out_of_order_time_window: 30m +otlp: + # Prometheus otherwise keeps these resource attributes on target_info only, + # so alerts and dashboards see no project/env/host. Promoting them puts the + # identity labels ADR 0002 requires on every series. service.name and + # service.instance.id already become job/instance without being listed. + promote_resource_attributes: [project, env, host.name] + rule_files: - /etc/prometheus/alerts/*.yaml From 1c10473bddeebcc14158a0d8de809248c73f5d2f Mon Sep 17 00:00:00 2001 From: Simon van Lierde Date: Thu, 27 Aug 2026 05:00:57 +0000 Subject: [PATCH 14/85] docs: assign handover owner, record the metric-name experiment --- docs/HANDOVER.md | 44 +++++++++++++++++++++++++------------------- 1 file changed, 25 insertions(+), 19 deletions(-) diff --git a/docs/HANDOVER.md b/docs/HANDOVER.md index 7714aec..2c8b371 100644 --- a/docs/HANDOVER.md +++ b/docs/HANDOVER.md @@ -1,7 +1,7 @@ # Handover: the central stack's half of the August 2026 review -Owner: _unassigned — put a name here before doing anything else._ -Review date: _set one; if nobody has picked this up by then, the tripwire below holds._ +Owner: Simon van Lierde +Review date: 2026-09-19 Relab's side of an architecture review is done and merged (its repo carries the full review in `deploy/MONITORING-DESIGN.md`; the target architecture is recorded here as @@ -157,19 +157,18 @@ Stock cAdvisor dashboards assume Prometheus *scraped* cAdvisor — they key on ` Prometheus reconstructs `job` from `service.name`, `instance` from `service.instance.id`, and applies a translation strategy to metric names. -Metric names are expected to survive untouched (already underscore-only, already -`_total`-suffixed, no OTLP unit set) but that could not be confirmed from documentation, -only inferred from translator behaviour. **Label values definitely change.** +**Run and answered (2026-08-27): names survive.** An OTLP batch posted straight at +Prometheus v3.13.2's receiver came back as `container_start_time_seconds` and +`container_oom_events_total`, names untouched, with the data-point `name` attribute intact +as a label. So: keep the single OTLP path, import 15798, repoint its two template variables +at `host_name`/`project`. No second hostname, no second credential, and no reaching for the +experimental `otlp.translation_strategy: NoTranslation`. -So: check whether `container_cpu_usage_seconds_total` exists in Prometheus with a `name` -label. Fifteen minutes, and it decides the wiring: - -- **Names survive** (expected): keep the single OTLP path, import 15798, repoint its two - template variables at `host_name`/`project`. One-time, ~20 minutes. One endpoint and one - credential is worth more than a dashboard's pristine defaults. -- **Names mangled**: do *not* reach for `otlp.translation_strategy: NoTranslation`, which - is experimental and documented with warnings. Route only the cAdvisor series natively - instead — that costs a second hostname and a second credential, so make it deliberately. +What the same experiment *did* turn up: resource attributes land on `target_info` only. +Until they are promoted, no series carries `project`, `env` or `host_name` — which silently +guts every by-host and by-project alert and the whole `project` template-variable plan. Now +fixed by `otlp.promote_resource_attributes` in `config/prometheus.yaml`; `target_info` keeps +its copy either way, so the keystone alert is unaffected. ______________________________________________________________________ @@ -230,9 +229,12 @@ this work up by the review date, the watchdog stays, and the deletion item in Re `deploy/MONITORING-DESIGN.md` §2.1 must not proceed on optimism. Deleting a weak local signal before its central replacement exists trades a weak signal for none. -1. `ProjectTelemetrySilent` — the keystone. Nothing else detects absence. -1. `ContainerRestarting` + `ContainerOOMKilled` — closes the incident that started this. -1. Run the metric-name experiment; import 15798 and 14574 accordingly. +1. ~~`ProjectTelemetrySilent`~~ — done; `config/alerts/projects.yaml`, keyed on + `target_info` so it does not depend on any project-side metric name. +1. ~~`ContainerRestarting` + `ContainerOOMKilled`~~ — done; `config/alerts/stack.yaml`, + group `container-lifecycle`. +1. ~~Run the metric-name experiment~~ — done, names survive. Still to do: import 15798 + and 14574. 1. `HostDiskSpaceLow`, `OtelExportFailures`, `Watchdog` heartbeat. 1. Grafana-managed alerting; delete Alertmanager. 1. Generic dashboards with a `project` variable. @@ -240,16 +242,20 @@ signal before its central replacement exists trades a weak signal for none. Steps 1–2 convert this from telemetry into monitoring. Everything after is leverage. +Steps 1–2 are live in config but **not yet verified against real Relab traffic** — the +tripwire above holds until they have fired once and been seen. Relab's local watchdog +checks stay until then. + ______________________________________________________________________ ## Be skeptical of these This came from an architecture review that verified claims against current documentation and -flagged what it could not. Two are worth checking rather than trusting: +flagged what it could not. One is still worth checking rather than trusting: - the Cloudflare Zero Trust free-plan seat count — the widely-cited figure appears only in third-party posts, never in Cloudflare's own docs; -- the OTLP metric-name round trip, which is the experiment above. +- ~~the OTLP metric-name round trip~~ — run on 2026-08-27; names survive. See above. There is also **no official cadence recommendation** for `restic check --read-data-subset`; the commonly repeated "1/12 monthly" is forum folklore. Pick a cadence, write down why, and From cec2769f4dbac46eb3c40b27c93c28b9cf028058 Mon Sep 17 00:00:00 2001 From: Simon van Lierde Date: Thu, 27 Aug 2026 05:19:09 +0000 Subject: [PATCH 15/85] fix: retarget ProjectTelemetrySilent at labels that actually exist Verified against the running stack with Relab reporting. - key the rule on project/env as ordinary series labels: Alloy attaches the identity labels to the series, not as OTLP resource attributes, so target_info carries none of them and the previous expression could never match - correct the environment to staging; no production series exist - record the live findings in the handover: cAdvisor sends only the root cgroup, so ContainerRestarting matches nothing and the crash-loop incident is still undetected; 30% of OTLP metric writes were being rejected as out-of-order until the window this branch carries was actually loaded --- config/alerts/projects.yaml | 20 ++++++---- docs/HANDOVER.md | 75 +++++++++++++++++++++++++++---------- 2 files changed, 68 insertions(+), 27 deletions(-) diff --git a/config/alerts/projects.yaml b/config/alerts/projects.yaml index 16e0b7d..c4f098f 100644 --- a/config/alerts/projects.yaml +++ b/config/alerts/projects.yaml @@ -1,20 +1,26 @@ # Per-project "is anything arriving at all" alerts. One group per onboarded # project — hand-written until step 7's bootstrap.sh generates these. -# Uses Prometheus's synthetic target_info{} (materialized from OTLP resource -# attributes on any received metric batch) rather than an app- or -# cAdvisor-specific metric name, so it doesn't depend on OTLP metric-name -# translation surviving. +# +# Keyed on a label-only selector, not on any named metric: whichever exporter +# a spoke runs, every series it sends carries the four identity labels, so this +# keeps working if Alloy's self-scrape or the cAdvisor integration is turned +# off. Deliberately NOT keyed on target_info — Alloy attaches the identity +# labels as ordinary series labels, so target_info carries none of them. +# NOTE: the selector touches every series a project sends (~3k for relab). +# Cheap at this size; revisit if a project's active series reach five figures. groups: - name: project-telemetry-relab rules: + # absent() reports silence only once the series go stale, which adds the + # 5m lookback on top of `for` — expect a page ~20m after the last sample. - alert: ProjectTelemetrySilent - expr: absent(target_info{project="relab",env="production"}) + expr: absent({project="relab",env="staging"}) for: 15m labels: severity: critical project: relab - env: production + env: staging annotations: - summary: "No telemetry from relab/production in 15m" + summary: "No telemetry from relab/staging in 15m" description: "Host down, Docker down, Alloy down, tunnel down, token rotated wrong, or the collector is rejecting relab's data. Nothing else on the Relab side can detect this on its own." diff --git a/docs/HANDOVER.md b/docs/HANDOVER.md index 2c8b371..8db2c71 100644 --- a/docs/HANDOVER.md +++ b/docs/HANDOVER.md @@ -41,6 +41,12 @@ hostnames, no new credentials, nothing to configure centrally per host. Every signal carries four identity labels: **`project`**, **`env`**, **`service.name`**, **`host_name`**. Enforced by the agent config, so a host cannot omit them. +Verified live 2026-08-27, with two corrections. Alloy attaches these as **ordinary series +labels, not OTLP resource attributes** — so `target_info` carries none of them, and any rule +keyed on `target_info` is keyed on nothing. And Relab's `env` is **`staging`**, not +`production`; no `production` series exist. `config/prometheus.yaml` promotes the four +attributes anyway, for spokes whose app SDK sends them the resource-attribute way. + Application logs are **no longer** exported by the API's SDK. Alloy ships that container's stdout, so exporting them twice stored every line twice in two shapes. The stdout path is the one kept, because it also carries what the SDK cannot report: the SDK's own export @@ -157,18 +163,15 @@ Stock cAdvisor dashboards assume Prometheus *scraped* cAdvisor — they key on ` Prometheus reconstructs `job` from `service.name`, `instance` from `service.instance.id`, and applies a translation strategy to metric names. -**Run and answered (2026-08-27): names survive.** An OTLP batch posted straight at -Prometheus v3.13.2's receiver came back as `container_start_time_seconds` and -`container_oom_events_total`, names untouched, with the data-point `name` attribute intact -as a label. So: keep the single OTLP path, import 15798, repoint its two template variables -at `host_name`/`project`. No second hostname, no second credential, and no reaching for the -experimental `otlp.translation_strategy: NoTranslation`. +**Run and answered (2026-08-27): names survive.** Confirmed twice — once against a +scratch Prometheus v3.13.2, once against the live stack, where Relab's own +`container_cpu_usage_seconds_total` and friends are present under their exact upstream +names. So: keep the single OTLP path, import 15798, and do not reach for the experimental +`otlp.translation_strategy: NoTranslation`. -What the same experiment *did* turn up: resource attributes land on `target_info` only. -Until they are promoted, no series carries `project`, `env` or `host_name` — which silently -guts every by-host and by-project alert and the whole `project` template-variable plan. Now -fixed by `otlp.promote_resource_attributes` in `config/prometheus.yaml`; `target_info` keeps -its copy either way, so the keystone alert is unaffected. +**But do not repoint 15798's variables at `host_name`/`project` yet, and do not import it +expecting per-container panels.** See the cAdvisor gap below — the per-container series the +dashboard is built on are not arriving at all. ______________________________________________________________________ @@ -229,12 +232,14 @@ this work up by the review date, the watchdog stays, and the deletion item in Re `deploy/MONITORING-DESIGN.md` §2.1 must not proceed on optimism. Deleting a weak local signal before its central replacement exists trades a weak signal for none. -1. ~~`ProjectTelemetrySilent`~~ — done; `config/alerts/projects.yaml`, keyed on - `target_info` so it does not depend on any project-side metric name. -1. ~~`ContainerRestarting` + `ContainerOOMKilled`~~ — done; `config/alerts/stack.yaml`, - group `container-lifecycle`. -1. ~~Run the metric-name experiment~~ — done, names survive. Still to do: import 15798 - and 14574. +1. ~~`ProjectTelemetrySilent`~~ — done and **verified against live traffic**; + `config/alerts/projects.yaml`. Keyed on a label-only selector, `project`/`env` on the + series themselves, because `target_info` carries no identity labels here. +1. `ContainerRestarting` + `ContainerOOMKilled` — written (`config/alerts/stack.yaml`, + group `container-lifecycle`) but **blind: no data reaches them.** See below. This is + the open item; the incident that started all of this is still undetected. +1. ~~Run the metric-name experiment~~ — done, names survive. Import of 15798/14574 is + blocked behind the same cAdvisor gap. 1. `HostDiskSpaceLow`, `OtelExportFailures`, `Watchdog` heartbeat. 1. Grafana-managed alerting; delete Alertmanager. 1. Generic dashboards with a `project` variable. @@ -242,9 +247,39 @@ signal before its central replacement exists trades a weak signal for none. Steps 1–2 convert this from telemetry into monitoring. Everything after is leverage. -Steps 1–2 are live in config but **not yet verified against real Relab traffic** — the -tripwire above holds until they have fired once and been seen. Relab's local watchdog -checks stay until then. +### What live verification found (2026-08-27) + +Checked against the running stack with Relab reporting. Step 1 holds. Step 2 does not. + +- **cAdvisor sends only the root cgroup.** `container_start_time_seconds`, + `container_oom_events_total` and every other `container_*` metric have exactly **one** + series each, `id="/"`, and there is no `name` label on any of them. So + `changes(container_start_time_seconds{name!=""}[1h]) > 3` matches zero series and + **`ContainerRestarting` can never fire**; `ContainerOOMKilled` sees only host-level OOM + and cannot say which container. The 668-restart incident would still be invisible today. + + The fix is on the spoke, not here: Relab's Alloy needs the mounts + `prometheus.exporter.cadvisor` requires to see container cgroups (Docker socket, + `/sys/fs/cgroup`, `/var/lib/docker`). Nothing in this repo can close it. **Until it is + closed, the tripwire holds and Relab's local watchdog checks stay.** + +- **30% of all OTLP metric writes were being rejected** — 19,585 HTTP 400s, 1,307 points + dropped every 30 seconds, silently, for as long as the counters go back. + `prometheus_tsdb_out_of_order_samples_total` matched the 400 count exactly: every + rejection was an out-of-order sample. Fixed by the `out_of_order_time_window` this branch + already carried; zero rejections since. + + Two operational notes worth keeping. `out_of_order_time_window` is **not applied by a + config reload** — SIGHUP logs a successful load and leaves the window at 0. It needs + `docker compose up -d --force-recreate prometheus`; plain `up -d` and `restart` are both + no-ops for a config-only change. And this failure mode is exactly what `OtelExportFailures` + exists to catch, which means it had been firing, unread, the whole time. + +- **`relab-api`'s own SDK metrics are not arriving.** The only jobs carrying + `project="relab"` are `gpu`, `integrations/self`, `integrations/unix` and + `integrations/cadvisor` — all Alloy. The emission table above claims app traces and + metrics under `service.name=relab-api`; the metrics half is not there. Traces were not + checked. ______________________________________________________________________ From abe69d4ca958dcd7631ae1a520565a217d2ad570 Mon Sep 17 00:00:00 2001 From: Simon van Lierde Date: Tue, 1 Sep 2026 12:31:13 +0000 Subject: [PATCH 16/85] refactor(tempo): drop the metrics-generator, RED comes from app OTLP - remove the generator, its remote-write path and the cardinality backstop - drop Prometheus' remote-write receiver, which existed only for it - unwire tracesToMetrics/serviceMap/nodeGraph rather than leave an empty service graph reading as "no traffic" --- compose.yml | 1 - config/grafana/datasources.yaml | 11 +++++------ config/tempo.yaml | 21 +++++---------------- 3 files changed, 10 insertions(+), 23 deletions(-) diff --git a/compose.yml b/compose.yml index c49364d..c85d52a 100644 --- a/compose.yml +++ b/compose.yml @@ -88,7 +88,6 @@ services: - --config.file=/etc/prometheus/prometheus.yaml - --storage.tsdb.path=/prometheus - --storage.tsdb.retention.time=30d - - --web.enable-remote-write-receiver # Prometheus 3.x: OTLP ingestion is its own flag, no longer a feature flag. - --web.enable-otlp-receiver - --enable-feature=native-histograms,exemplar-storage diff --git a/config/grafana/datasources.yaml b/config/grafana/datasources.yaml index 02ae034..565dde1 100644 --- a/config/grafana/datasources.yaml +++ b/config/grafana/datasources.yaml @@ -44,9 +44,8 @@ datasources: tags: - key: service.name value: service_name - tracesToMetrics: - datasourceUid: prometheus - serviceMap: - datasourceUid: prometheus - nodeGraph: - enabled: true + # No tracesToMetrics/serviceMap/nodeGraph: all three read the series Tempo's + # metrics-generator used to write, and that generator is gone (ADR 0002 — RED + # comes from the apps' own OTLP metrics). Leaving them configured would render + # a permanently empty service graph, which reads as "no traffic" rather than + # "not wired". diff --git a/config/tempo.yaml b/config/tempo.yaml index d251b60..75c1544 100644 --- a/config/tempo.yaml +++ b/config/tempo.yaml @@ -23,25 +23,14 @@ storage: local: path: /var/tempo/blocks -# Derive RED metrics (requests/errors/duration) + a service graph from traces. -# These land in Prometheus and power the Service Health dashboard. -metrics_generator: - registry: - external_labels: - source: tempo - storage: - path: /var/tempo/generator/wal - remote_write: - - url: http://prometheus:9090/api/v1/write - send_exemplars: true +# No metrics_generator: RED comes from the applications' own OTLP metrics (ADR 0002). +# Deriving it from traces as well meant two producers for one number, a Prometheus +# remote-write path that existed for nothing else, and a cardinality backstop to +# maintain. Tempo is trace storage now, which makes it disposable on its next +# breaking upgrade. overrides: defaults: - metrics_generator: - processors: [service-graphs, span-metrics] - # Cardinality backstop: a sender with unrouted span names (raw paths) - # would otherwise mint Prometheus series without bound. - max_active_series: 50000 compaction: block_retention: 168h # 7 days — traces are bulky, tune to disk budget From e2fac86f547bcb16f27980c49559a34e5dc56d43 Mon Sep 17 00:00:00 2001 From: Simon van Lierde Date: Tue, 1 Sep 2026 12:31:38 +0000 Subject: [PATCH 17/85] feat(alerting): move evaluation and delivery into Grafana - provision rules, contact points and the routing tree from config/grafana/alerting/; one engine that can query Loki as well - drop Alertmanager, its url_file entrypoint hack and its volume - add AlertDeliveryFailing, and make ALERT_WEBHOOK_URL a hard guard in up-tunnel: empty delivery fails silently while the heartbeat stays green --- .env.example | 6 +- compose.yml | 31 +- config/alertmanager.yaml | 27 -- config/alerts/projects.yaml | 26 -- config/alerts/stack.yaml | 89 ----- config/grafana/alerting/contact-points.yaml | 28 ++ .../alerting/notification-policies.yaml | 17 + config/grafana/alerting/rules.yaml | 318 ++++++++++++++++++ config/prometheus.yaml | 13 +- justfile | 8 +- 10 files changed, 379 insertions(+), 184 deletions(-) delete mode 100644 config/alertmanager.yaml delete mode 100644 config/alerts/projects.yaml delete mode 100644 config/alerts/stack.yaml create mode 100644 config/grafana/alerting/contact-points.yaml create mode 100644 config/grafana/alerting/notification-policies.yaml create mode 100644 config/grafana/alerting/rules.yaml diff --git a/.env.example b/.env.example index 569469e..f2d1f9a 100644 --- a/.env.example +++ b/.env.example @@ -15,8 +15,10 @@ GRAFANA_COOKIE_SECURE=false # The default only suits local use — generate a real one for production, e.g.: openssl rand -hex 32 OTLP_AUTH_TOKEN=local-dev-token -# Where Alertmanager delivers alert notifications (any webhook: ntfy, Slack, …). -# Leave empty to run without delivery; failures are logged and harmless. +# Where Grafana delivers alert notifications (any webhook: ntfy, Slack, …). +# Not optional. An empty value makes every alert fail delivery silently while the +# heartbeat below keeps pinging, so the dead man's switch reads healthy and nothing +# reaches anyone. `just up-tunnel` refuses to start without it. ALERT_WEBHOOK_URL= # Dead man's switch ping target (e.g. https://hc-ping.com/). The Watchdog diff --git a/compose.yml b/compose.yml index c85d52a..73ca847 100644 --- a/compose.yml +++ b/compose.yml @@ -103,30 +103,6 @@ services: security_opt: *default-security logging: *default-logging - alertmanager: - image: prom/alertmanager:v0.33.1 - restart: unless-stopped - volumes: - - ./config/alertmanager.yaml:/etc/alertmanager/alertmanager.yaml:ro - - alertmanager_data:/alertmanager - environment: - ALERT_WEBHOOK_URL: ${ALERT_WEBHOOK_URL:-} - HEARTBEAT_URL: ${HEARTBEAT_URL:-} - # Alertmanager can't expand env vars in its config; write the webhook - # URLs to the url_file paths the config points at, then start. tmpfs on - # purpose: the URLs often carry secrets and must not end up in the - # persistent volume (which `just backup` archives). - tmpfs: [ /run/am ] - entrypoint: [ "/bin/sh", "-c" ] - command: - - | - printf '%s' "$$ALERT_WEBHOOK_URL" > /run/am/webhook_url - printf '%s' "$$HEARTBEAT_URL" > /run/am/heartbeat_url - exec /bin/alertmanager --config.file=/etc/alertmanager/alertmanager.yaml --storage.path=/alertmanager - mem_limit: 256m - security_opt: *default-security - logging: *default-logging - node-exporter: image: prom/node-exporter:v1.12.1 restart: unless-stopped @@ -156,10 +132,16 @@ services: volumes: - ./config/grafana/datasources.yaml:/etc/grafana/provisioning/datasources/datasources.yaml:ro - ./config/grafana/dashboards.yaml:/etc/grafana/provisioning/dashboards/dashboards.yaml:ro + - ./config/grafana/alerting:/etc/grafana/provisioning/alerting:ro - ./dashboards:/var/lib/grafana/dashboards:ro - grafana_data:/var/lib/grafana environment: GF_SECURITY_ADMIN_PASSWORD: ${GRAFANA_ADMIN_PASSWORD:?set GRAFANA_ADMIN_PASSWORD in .env} + # Expanded by Grafana into the provisioned contact points. Alertmanager needed + # a url_file written by an entrypoint because it cannot read env vars; Grafana + # can, so that hack is gone along with its tmpfs. + ALERT_WEBHOOK_URL: ${ALERT_WEBHOOK_URL:-} + HEARTBEAT_URL: ${HEARTBEAT_URL:-} GF_USERS_ALLOW_SIGN_UP: "false" GF_SERVER_ROOT_URL: ${GRAFANA_ROOT_URL:-http://localhost:3000} # Land on Stack Health instead of the empty welcome page. @@ -188,7 +170,6 @@ volumes: tempo_data: prometheus_data: grafana_data: - alertmanager_data: # Deliberately not in `just backup`: queue contents are seconds of in-flight # telemetry, worthless by restore time. otel_queue: diff --git a/config/alertmanager.yaml b/config/alertmanager.yaml deleted file mode 100644 index 72b3289..0000000 --- a/config/alertmanager.yaml +++ /dev/null @@ -1,27 +0,0 @@ -# Alert routing. Webhook URLs are supplied at runtime via url_file (written -# from ALERT_WEBHOOK_URL / HEARTBEAT_URL by the compose entrypoint) because -# Alertmanager does not expand env vars in its config. With the env vars -# unset, delivery fails with a logged error and nothing else — the stack -# runs fine without them. - -route: - receiver: webhook - group_by: [alertname] - routes: - # Watchdog is always firing; its delivery is the heartbeat. Wire - # HEARTBEAT_URL to a dead man's switch (e.g. healthchecks.io) that - # alerts when pings STOP arriving. - - matchers: [ 'alertname = "Watchdog"' ] - receiver: heartbeat - group_wait: 0s - repeat_interval: 5m - -receivers: - - name: webhook - webhook_configs: - - url_file: /run/am/webhook_url - - - name: heartbeat - webhook_configs: - - url_file: /run/am/heartbeat_url - send_resolved: false diff --git a/config/alerts/projects.yaml b/config/alerts/projects.yaml deleted file mode 100644 index c4f098f..0000000 --- a/config/alerts/projects.yaml +++ /dev/null @@ -1,26 +0,0 @@ -# Per-project "is anything arriving at all" alerts. One group per onboarded -# project — hand-written until step 7's bootstrap.sh generates these. -# -# Keyed on a label-only selector, not on any named metric: whichever exporter -# a spoke runs, every series it sends carries the four identity labels, so this -# keeps working if Alloy's self-scrape or the cAdvisor integration is turned -# off. Deliberately NOT keyed on target_info — Alloy attaches the identity -# labels as ordinary series labels, so target_info carries none of them. -# NOTE: the selector touches every series a project sends (~3k for relab). -# Cheap at this size; revisit if a project's active series reach five figures. - -groups: - - name: project-telemetry-relab - rules: - # absent() reports silence only once the series go stale, which adds the - # 5m lookback on top of `for` — expect a page ~20m after the last sample. - - alert: ProjectTelemetrySilent - expr: absent({project="relab",env="staging"}) - for: 15m - labels: - severity: critical - project: relab - env: staging - annotations: - summary: "No telemetry from relab/staging in 15m" - description: "Host down, Docker down, Alloy down, tunnel down, token rotated wrong, or the collector is rejecting relab's data. Nothing else on the Relab side can detect this on its own." diff --git a/config/alerts/stack.yaml b/config/alerts/stack.yaml deleted file mode 100644 index a8ca4a1..0000000 --- a/config/alerts/stack.yaml +++ /dev/null @@ -1,89 +0,0 @@ -# Prometheus alert rules for the stack itself plus RED-style alerts on the -# span metrics Tempo derives from traces. Evaluated by Prometheus, delivered -# by Alertmanager (config/alertmanager.yaml → ALERT_WEBHOOK_URL/HEARTBEAT_URL), -# and visible in Grafana under Alerting → Alert rules. -# Validated by `just check` (promtool check config, which loads these rules). - -groups: - - name: meta - rules: - # Always firing on purpose: its arrival at the heartbeat receiver is - # the proof the whole alerting pipeline works. Silence = broken. - - alert: Watchdog - expr: vector(1) - labels: - severity: none - annotations: - summary: "Alerting-pipeline heartbeat (always firing)" - description: "Route this to a dead man's switch; investigate if pings stop." - - - name: stack-health - rules: - - alert: TargetDown - expr: up == 0 - for: 2m - labels: - severity: critical - annotations: - summary: "Scrape target {{ $labels.job }} is down" - description: "Prometheus cannot scrape {{ $labels.instance }} (job {{ $labels.job }}) for 2 minutes." - - - alert: OtelExportFailures - expr: sum by (exporter) (rate(otelcol_exporter_send_failed_spans[5m])) > 0 - or sum by (exporter) (rate(otelcol_exporter_send_failed_metric_points[5m])) > 0 - or sum by (exporter) (rate(otelcol_exporter_send_failed_log_records[5m])) > 0 - for: 5m - labels: - severity: warning - annotations: - summary: "OTel Collector failing to export via {{ $labels.exporter }}" - description: "The collector has been failing to deliver telemetry to a backend for 5 minutes. Check `just logs otel-collector`." - - - name: capacity - rules: - - alert: HostDiskSpaceLow - expr: > - (node_filesystem_avail_bytes{fstype!~"tmpfs|ramfs|overlay"} - / node_filesystem_size_bytes{fstype!~"tmpfs|ramfs|overlay"}) < 0.2 - for: 15m - labels: - severity: warning - annotations: - summary: "Filesystem {{ $labels.mountpoint }} is over 80% full" - description: "Loki and Tempo have no total-size cap, so this alert is the storage backstop. Free space or lower retention (see RUNBOOK)." - - - name: container-lifecycle - rules: - - alert: ContainerRestarting - expr: changes(container_start_time_seconds{name!=""}[1h]) > 3 - for: 10m - labels: - severity: warning - annotations: - summary: "Container {{ $labels.name }} is crash-looping" - description: "{{ $labels.name }} on {{ $labels.host_name }} has restarted more than 3 times in the last hour." - - - alert: ContainerOOMKilled - expr: increase(container_oom_events_total[5m]) > 0 - for: 0m - labels: - severity: warning - annotations: - summary: "Container {{ $labels.name }} was OOM-killed" - description: "{{ $labels.name }} on {{ $labels.host_name }} hit an OOM kill in the last 5 minutes." - - - name: service-red - rules: - # Server spans only, matching the Service Health dashboard. Without the - # span_kind filter, internal child spans inflate the denominator and the - # measured rate lands well under the true request error rate. - - alert: HighErrorRate - expr: > - sum by (service) (rate(traces_spanmetrics_calls_total{span_kind="SPAN_KIND_SERVER",status_code="STATUS_CODE_ERROR"}[5m])) - / sum by (service) (rate(traces_spanmetrics_calls_total{span_kind="SPAN_KIND_SERVER"}[5m])) > 0.05 - for: 5m - labels: - severity: warning - annotations: - summary: "{{ $labels.service }} error rate above 5%" - description: "More than 5% of spans from {{ $labels.service }} report errors (from Tempo span metrics)." diff --git a/config/grafana/alerting/contact-points.yaml b/config/grafana/alerting/contact-points.yaml new file mode 100644 index 0000000..feea705 --- /dev/null +++ b/config/grafana/alerting/contact-points.yaml @@ -0,0 +1,28 @@ +# Notification targets. Grafana expands $VAR in provisioning files, so the URLs come +# straight from the environment and no secret is written to disk. +# +# An unset variable leaves the URL empty and every notification fails — silently, from +# the outside. AlertDeliveryFailing and the heartbeat are what surface that; the guard +# in `just up-tunnel` is what prevents it. + +apiVersion: 1 + +contactPoints: + - orgId: 1 + name: webhook + receivers: + - uid: cp-webhook-default + type: webhook + settings: + url: $ALERT_WEBHOOK_URL + + - orgId: 1 + name: heartbeat + receivers: + - uid: cp-heartbeat-dms + type: webhook + settings: + url: $HEARTBEAT_URL + # A resolved-notification would ping the dead man's switch too, which + # would mask the very outage the switch exists to report. + disableResolveMessage: true diff --git a/config/grafana/alerting/notification-policies.yaml b/config/grafana/alerting/notification-policies.yaml new file mode 100644 index 0000000..79ba2a7 --- /dev/null +++ b/config/grafana/alerting/notification-policies.yaml @@ -0,0 +1,17 @@ +# Routing tree. Everything reaches the webhook except Watchdog, whose delivery +# IS the heartbeat: it fires permanently and its silence is the alarm, so it +# goes to the dead man's switch on a short repeat and never groups or waits. + +apiVersion: 1 + +policies: + - orgId: 1 + receiver: webhook + group_by: [alertname] + routes: + - receiver: heartbeat + object_matchers: + - [alertname, =, Watchdog] + group_wait: 0s + group_interval: 1m + repeat_interval: 5m diff --git a/config/grafana/alerting/rules.yaml b/config/grafana/alerting/rules.yaml new file mode 100644 index 0000000..3a3294c --- /dev/null +++ b/config/grafana/alerting/rules.yaml @@ -0,0 +1,318 @@ +# Grafana-managed alert rules (ADR 0002): one engine owns evaluation and notification, +# and Grafana rules can query Loki as well as Prometheus. +# +# Shape note, true for every rule here: the PromQL carries its own comparison, so it +# returns a series only when the alert should fire, and the threshold node just asks +# whether anything came back — no reduce/threshold pair to decode. +# +# Keep the total under about ten. These are the generic rules; the per-project keystone +# lives in project--.yaml beside this file, rendered by bootstrap.sh. + +apiVersion: 1 + +groups: + # The alerting pipeline's own proof of life, and the detector for it failing. + - orgId: 1 + name: meta + folder: Stack alerts + interval: 1m + rules: + # Always firing on purpose; its ARRIVAL is the proof. execErrState Error is + # deliberate: if the query breaks the heartbeat stops and the switch alarms. + - uid: watchdog-heartbeat + title: Watchdog + condition: FIRING + for: 0m + noDataState: OK + execErrState: Error + isPaused: false + data: + - refId: QUERY + datasourceUid: prometheus + relativeTimeRange: + from: 3600 + to: 0 + model: + refId: QUERY + instant: true + editorMode: code + expr: vector(1) + - refId: FIRING + datasourceUid: __expr__ + model: + refId: FIRING + type: threshold + expression: QUERY + conditions: + - evaluator: + type: gt + params: [0] + labels: + severity: none + annotations: + summary: "Alerting-pipeline heartbeat (always firing)" + description: "Routed to a dead man's switch. Investigate if the pings stop." + # Cannot page when the path it reports on is the broken one — that is what the + # heartbeat is for. Its job is to make the state visible instead of silent. + - uid: alert-delivery-failing + title: AlertDeliveryFailing + condition: FIRING + for: 10m + noDataState: OK + execErrState: Error + isPaused: false + data: + - refId: QUERY + datasourceUid: prometheus + relativeTimeRange: + from: 3600 + to: 0 + model: + refId: QUERY + instant: true + editorMode: code + expr: rate(grafana_alerting_notifications_failed_total[10m]) > 0 + - refId: FIRING + datasourceUid: __expr__ + model: + refId: FIRING + type: threshold + expression: QUERY + conditions: + - evaluator: + type: gt + params: [0] + labels: + severity: critical + annotations: + summary: "Grafana cannot deliver {{ $labels.integration }} notifications" + description: "Alerts are firing and going nowhere. Most likely an unset ALERT_WEBHOOK_URL, otherwise the receiver is rejecting. Check `just logs grafana`." + # The central stack watching itself. + - orgId: 1 + name: stack-health + folder: Stack alerts + interval: 1m + rules: + - uid: target-down + title: TargetDown + condition: FIRING + for: 2m + noDataState: OK + execErrState: Error + isPaused: false + data: + - refId: QUERY + datasourceUid: prometheus + relativeTimeRange: + from: 3600 + to: 0 + model: + refId: QUERY + instant: true + editorMode: code + expr: up == 0 + - refId: FIRING + datasourceUid: __expr__ + model: + refId: FIRING + type: threshold + expression: QUERY + conditions: + - evaluator: + type: gt + params: [0] + labels: + severity: critical + annotations: + summary: "Scrape target {{ $labels.job }} is down" + description: "Prometheus cannot scrape {{ $labels.instance }} (job {{ $labels.job }}) for 2 minutes." + - uid: otel-export-failures + title: OtelExportFailures + condition: FIRING + for: 5m + noDataState: OK + execErrState: Error + isPaused: false + data: + - refId: QUERY + datasourceUid: prometheus + relativeTimeRange: + from: 3600 + to: 0 + model: + refId: QUERY + instant: true + editorMode: code + expr: > + sum by (exporter) (rate(otelcol_exporter_send_failed_spans[5m])) > 0 + or sum by (exporter) (rate(otelcol_exporter_send_failed_metric_points[5m])) > 0 + or sum by (exporter) (rate(otelcol_exporter_send_failed_log_records[5m])) > 0 + - refId: FIRING + datasourceUid: __expr__ + model: + refId: FIRING + type: threshold + expression: QUERY + conditions: + - evaluator: + type: gt + params: [0] + labels: + severity: warning + annotations: + summary: "OTel Collector failing to export via {{ $labels.exporter }}" + description: "The collector has been failing to deliver telemetry to a backend for 5 minutes. Check `just logs otel-collector`." + # Storage backstop: Loki and Tempo have no total-size cap. + - orgId: 1 + name: capacity + folder: Stack alerts + interval: 1m + rules: + - uid: host-disk-space-low + title: HostDiskSpaceLow + condition: FIRING + for: 15m + noDataState: OK + execErrState: Error + isPaused: false + data: + - refId: QUERY + datasourceUid: prometheus + relativeTimeRange: + from: 3600 + to: 0 + model: + refId: QUERY + instant: true + editorMode: code + expr: > + (node_filesystem_avail_bytes{fstype!~"tmpfs|ramfs|overlay"} + / node_filesystem_size_bytes{fstype!~"tmpfs|ramfs|overlay"}) < 0.2 + - refId: FIRING + datasourceUid: __expr__ + model: + refId: FIRING + type: threshold + expression: QUERY + conditions: + - evaluator: + type: gt + params: [0] + labels: + severity: warning + annotations: + summary: "Filesystem {{ $labels.mountpoint }} is over 80% full" + description: "Loki and Tempo have no total-size cap, so this alert is the storage backstop. Free space or lower retention (see RUNBOOK)." + # The 668-restart incident: a crash loop must get louder, not quieter. + - orgId: 1 + name: container-lifecycle + folder: Stack alerts + interval: 1m + rules: + - uid: container-restarting + title: ContainerRestarting + condition: FIRING + for: 10m + noDataState: OK + execErrState: Error + isPaused: false + data: + - refId: QUERY + datasourceUid: prometheus + relativeTimeRange: + from: 3600 + to: 0 + model: + refId: QUERY + instant: true + editorMode: code + expr: changes(container_start_time_seconds{name!=""}[1h]) > 3 + - refId: FIRING + datasourceUid: __expr__ + model: + refId: FIRING + type: threshold + expression: QUERY + conditions: + - evaluator: + type: gt + params: [0] + labels: + severity: warning + annotations: + summary: "Container {{ $labels.name }} is crash-looping" + description: "{{ $labels.name }} on {{ $labels.host_name }} has restarted more than 3 times in the last hour." + - uid: container-oom-killed + title: ContainerOOMKilled + condition: FIRING + for: 0m + noDataState: OK + execErrState: Error + isPaused: false + data: + - refId: QUERY + datasourceUid: prometheus + relativeTimeRange: + from: 3600 + to: 0 + model: + refId: QUERY + instant: true + editorMode: code + expr: increase(container_oom_events_total[5m]) > 0 + - refId: FIRING + datasourceUid: __expr__ + model: + refId: FIRING + type: threshold + expression: QUERY + conditions: + - evaluator: + type: gt + params: [0] + labels: + severity: warning + annotations: + summary: "Container {{ $labels.name }} was OOM-killed" + description: "{{ $labels.name }} on {{ $labels.host_name }} hit an OOM kill in the last 5 minutes." + # RED from the applications' own OTLP metrics, not Tempo's span-metrics. + - orgId: 1 + name: service-red + folder: Stack alerts + interval: 1m + rules: + - uid: high-error-rate + title: HighErrorRate + condition: FIRING + for: 5m + noDataState: OK + execErrState: Error + isPaused: false + data: + - refId: QUERY + datasourceUid: prometheus + relativeTimeRange: + from: 3600 + to: 0 + model: + refId: QUERY + instant: true + editorMode: code + expr: > + sum by (job) (rate(http_server_request_duration_seconds_count{http_response_status_code=~"5.."}[5m])) + / sum by (job) (rate(http_server_request_duration_seconds_count[5m])) > 0.05 + - refId: FIRING + datasourceUid: __expr__ + model: + refId: FIRING + type: threshold + expression: QUERY + conditions: + - evaluator: + type: gt + params: [0] + labels: + severity: warning + annotations: + summary: "{{ $labels.job }} error rate above 5%" + description: "More than 5% of requests to {{ $labels.job }} returned 5xx over the last 5 minutes." diff --git a/config/prometheus.yaml b/config/prometheus.yaml index aaf142d..4e634a4 100644 --- a/config/prometheus.yaml +++ b/config/prometheus.yaml @@ -17,13 +17,8 @@ otlp: # service.instance.id already become job/instance without being listed. promote_resource_attributes: [project, env, host.name] -rule_files: - - /etc/prometheus/alerts/*.yaml - -alerting: - alertmanagers: - - static_configs: - - targets: ["alertmanager:9093"] +# No rule_files and no alerting block: Grafana owns both evaluation and +# notification (ADR 0002). Prometheus is a store and a query engine here. # Apps push metrics via OTLP (through the collector). Scrape only infra we host. scrape_configs: @@ -51,7 +46,3 @@ scrape_configs: - job_name: tempo static_configs: - targets: ["tempo:3200"] - - - job_name: alertmanager - static_configs: - - targets: ["alertmanager:9093"] diff --git a/justfile b/justfile index 51e2d7d..a1f60dc 100644 --- a/justfile +++ b/justfile @@ -2,8 +2,8 @@ set dotenv-load # Stateful services and their volumes, shared by backup/restore. The volume # names assume the compose project name "monitoring" (see guard in backup). -stateful := "grafana prometheus loki tempo alertmanager" -backup_mounts := "-v monitoring_grafana_data:/data/grafana -v monitoring_prometheus_data:/data/prometheus -v monitoring_loki_data:/data/loki -v monitoring_tempo_data:/data/tempo -v monitoring_alertmanager_data:/data/alertmanager" +stateful := "grafana prometheus loki tempo" +backup_mounts := "-v monitoring_grafana_data:/data/grafana -v monitoring_prometheus_data:/data/prometheus -v monitoring_loki_data:/data/loki -v monitoring_tempo_data:/data/tempo" default: @just --list @@ -27,6 +27,7 @@ up-tunnel: _queue-volume @[ "${GRAFANA_ROOT_URL:-}" != "http://localhost:3000" ] || { echo "error: GRAFANA_ROOT_URL is still the localhost default; set it to the tunnel hostname or every absolute URL Grafana generates breaks" >&2; exit 1; } @[ "${GRAFANA_COOKIE_SECURE:-false}" = "true" ] || { echo "error: GRAFANA_COOKIE_SECURE must be true when Grafana is served over HTTPS; set it in .env" >&2; exit 1; } @[ -n "${HEARTBEAT_URL:-}" ] || echo "WARNING: HEARTBEAT_URL is empty; the stack goes live without a dead-man's switch" >&2 + @[ -n "${ALERT_WEBHOOK_URL:-}" ] || { echo "error: ALERT_WEBHOOK_URL is empty; every alert would fire into an empty url_file and be dropped. The heartbeat keeps pinging either way, so this failure looks healthy from the outside — set it, or comment out this guard deliberately" >&2; exit 1; } docker compose -f compose.yml -f compose.tunnel.yml up -d down: @@ -64,7 +65,7 @@ tail service: docker compose -f compose.yml -f compose.demo.yml logs -f --no-log-prefix {{service}} | jq -R 'fromjson? // .' # Validate everything. All validators run in containers — no host installs. -# promtool/otelcol/amtool images are read from compose.yml so they can't +# promtool/otelcol images are read from compose.yml so they can't # drift from the versions the stack actually runs. check: docker compose config -q @@ -72,7 +73,6 @@ check: docker compose -f compose.yml -f compose.demo.yml config -q docker run --rm -v ./config/prometheus.yaml:/etc/prometheus/prometheus.yaml:ro -v ./config/alerts:/etc/prometheus/alerts:ro --entrypoint promtool $(docker compose config --images | grep prom/prometheus) check config /etc/prometheus/prometheus.yaml docker run --rm -e OTLP_AUTH_TOKEN=dummy -v ./config/otel-collector.yaml:/etc/otelcol/config.yaml:ro $(docker compose config --images | grep opentelemetry-collector) validate --config=/etc/otelcol/config.yaml - docker run --rm -v ./config/alertmanager.yaml:/etc/alertmanager/alertmanager.yaml:ro --entrypoint /bin/amtool $(docker compose config --images | grep prom/alertmanager) check-config /etc/alertmanager/alertmanager.yaml docker run --rm --network none -v ./config/loki.yaml:/etc/loki/loki.yaml:ro $(docker compose config --images | grep grafana/loki) -config.file=/etc/loki/loki.yaml -verify-config docker run --rm --network none -v ./config/tempo.yaml:/etc/tempo/tempo.yaml:ro $(docker compose config --images | grep grafana/tempo) -config.file=/etc/tempo/tempo.yaml -config.verify=true docker run --rm --network none -v .:/code:ro pipelinecomponents/yamllint:0.35.13 yamllint -d '{extends: relaxed, ignore: [.git/, backups/, infra/.terraform/]}' . From 79b2c8156476a86467f5f7447e4485e1fdbd7493 Mon Sep 17 00:00:00 2001 From: Simon van Lierde Date: Tue, 1 Sep 2026 12:31:51 +0000 Subject: [PATCH 18/85] feat(loki): index project, env and host.name as stream labels Template variables are stream selectors and label_values() cannot see structured metadata. All four are bounded, so indexing them costs nothing. --- config/loki.yaml | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/config/loki.yaml b/config/loki.yaml index 4ed036b..7d07918 100644 --- a/config/loki.yaml +++ b/config/loki.yaml @@ -44,11 +44,20 @@ limits_config: # it is now structured metadata, so query it with `| service_instance_id=...`. otlp_config: resource_attributes: + # ignore_defaults keeps Loki from indexing every resource attribute it is + # handed. These four are the identity labels ADR 0002 requires on every + # signal, and they have to be real stream LABELS, not structured metadata: + # a `project` template variable is a stream selector, and label_values() + # cannot see structured metadata. All four are bounded — one value per + # project, environment and host — so indexing them costs nothing. ignore_defaults: true attributes_config: - action: index_label attributes: - service.name + - project + - env + - host.name compactor: working_directory: /loki/compactor From 21e1189a7671f19bec4a0245bb47c8eb243309b6 Mon Sep 17 00:00:00 2001 From: Simon van Lierde Date: Tue, 1 Sep 2026 12:32:05 +0000 Subject: [PATCH 19/85] feat(dashboards): add gpu, host & containers, and logs - replace infrastructure-logs and logs-overview with one logs dashboard - rework service-health and stack-health onto the OTLP-native series --- dashboards/gpu.json | 4230 +++++++++++++++++++++++++++ dashboards/host-containers.json | 314 ++ dashboards/infrastructure-logs.json | 98 - dashboards/logs-overview.json | 130 - dashboards/logs.json | 195 ++ dashboards/service-health.json | 191 +- dashboards/stack-health.json | 168 +- 7 files changed, 5009 insertions(+), 317 deletions(-) create mode 100644 dashboards/gpu.json create mode 100644 dashboards/host-containers.json delete mode 100644 dashboards/infrastructure-logs.json delete mode 100644 dashboards/logs-overview.json create mode 100644 dashboards/logs.json diff --git a/dashboards/gpu.json b/dashboards/gpu.json new file mode 100644 index 0000000..3585d30 --- /dev/null +++ b/dashboards/gpu.json @@ -0,0 +1,4230 @@ +{ + "__elements": {}, + "annotations": { + "list": [ + { + "builtIn": 1, + "datasource": { + "type": "datasource", + "uid": "grafana" + }, + "enable": true, + "hide": true, + "iconColor": "rgba(0, 211, 255, 1)", + "name": "Annotations & Alerts", + "type": "dashboard" + } + ] + }, + "description": "Imported from grafana.com dashboard 14574 (Nvidia GPU Metrics), revised 2026-08-04, for the nvidia_gpu_exporter this stack ships. Kept close to upstream on purpose: the panels encode NVML domain knowledge \u2014 throttle-reason bitmasks, clock domains, XID handling \u2014 that is not worth reproducing by hand. Local changes: the datasource is pinned instead of picked, and the variable chain is scoped by project/env so one dashboard serves every GPU host. Panel expressions are upstream's, untouched.", + "editable": false, + "fiscalYearStartMonth": 0, + "gnetId": 14574, + "graphTooltip": 0, + "links": [ + { + "asDropdown": false, + "icon": "external link", + "includeVars": false, + "keepTime": true, + "tags": [ + "nvidia_gpu_exporter" + ], + "targetBlank": false, + "title": "Related dashboards", + "tooltip": "", + "type": "dashboards", + "url": "" + } + ], + "panels": [ + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "description": "Recovery action the driver recommends for this GPU. Healthy means none. Color reflects how disruptive the action is: Drain P2P degrades peer-to-peer but keeps computing, GPU Reset interrupts workloads, Node Reboot and Drain & Reset are the heaviest. Requires a recent driver, older drivers do not report this.", + "fieldConfig": { + "defaults": { + "color": { + "mode": "fixed", + "fixedColor": "text" + }, + "decimals": 0, + "mappings": [ + { + "type": "value", + "options": { + "0": { + "text": "Healthy", + "color": "#56A64B", + "index": 0 + }, + "1": { + "text": "GPU Reset", + "color": "#FF9830", + "index": 1 + }, + "2": { + "text": "Node Reboot", + "color": "#C4162A", + "index": 2 + }, + "3": { + "text": "Drain P2P", + "color": "#F2CC0C", + "index": 3 + }, + "4": { + "text": "Drain & Reset", + "color": "#E02F44", + "index": 4 + } + } + } + ], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "", + "noValue": "Not reported by driver" + }, + "overrides": [] + }, + "gridPos": { + "h": 3, + "w": 8, + "x": 0, + "y": 0 + }, + "id": 35, + "options": { + "colorMode": "background", + "graphMode": "none", + "justifyMode": "auto", + "orientation": "auto", + "percentChangeColorMode": "standard", + "reduceOptions": { + "calcs": [ + "last" + ], + "fields": "", + "values": false + }, + "showPercentChange": false, + "text": {}, + "textMode": "value", + "wideLayout": true + }, + "pluginVersion": "11.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "editorMode": "code", + "expr": "nvidia_smi_gpu_recovery_action{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}", + "instant": false, + "legendFormat": "__auto", + "range": true, + "refId": "A" + } + ], + "title": "Recovery Action", + "type": "stat" + }, + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "description": "Number of performance-limiting throttle reasons currently active: power cap, SW/HW thermal slowdown, HW power brake. Idle and configuration states (application clocks, sync boost) are not counted, and a flag the driver does not report counts as inactive. See the Throttle Reasons history panel for which reason and when.", + "fieldConfig": { + "defaults": { + "color": { + "mode": "fixed", + "fixedColor": "text" + }, + "decimals": 0, + "mappings": [ + { + "type": "value", + "options": { + "0": { + "text": "None", + "color": "#56A64B", + "index": 0 + }, + "1": { + "text": "1 reason active", + "color": "#FFB357", + "index": 1 + }, + "2": { + "text": "2 reasons active", + "color": "#FF9830", + "index": 2 + }, + "3": { + "text": "3 reasons active", + "color": "#E02F44", + "index": 3 + }, + "4": { + "text": "4 reasons active", + "color": "#C4162A", + "index": 4 + } + } + } + ], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "" + }, + "overrides": [] + }, + "gridPos": { + "h": 3, + "w": 8, + "x": 8, + "y": 0 + }, + "id": 40, + "options": { + "colorMode": "background", + "graphMode": "none", + "justifyMode": "auto", + "orientation": "auto", + "percentChangeColorMode": "standard", + "reduceOptions": { + "calcs": [ + "last" + ], + "fields": "", + "values": false + }, + "showPercentChange": false, + "text": {}, + "textMode": "value", + "wideLayout": true + }, + "pluginVersion": "11.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "expr": "(sum by(uuid) (nvidia_smi_clocks_event_reasons_sw_power_cap{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"} or nvidia_smi_clocks_throttle_reasons_sw_power_cap{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}) or (sum by(uuid) (nvidia_smi_gpu_info{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}) * 0)) + (sum by(uuid) (nvidia_smi_clocks_event_reasons_sw_thermal_slowdown{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"} or nvidia_smi_clocks_throttle_reasons_sw_thermal_slowdown{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}) or (sum by(uuid) (nvidia_smi_gpu_info{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}) * 0)) + (sum by(uuid) (nvidia_smi_clocks_event_reasons_hw_thermal_slowdown{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"} or nvidia_smi_clocks_throttle_reasons_hw_thermal_slowdown{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}) or (sum by(uuid) (nvidia_smi_gpu_info{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}) * 0)) + (sum by(uuid) (nvidia_smi_clocks_event_reasons_hw_power_brake_slowdown{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"} or nvidia_smi_clocks_throttle_reasons_hw_power_brake_slowdown{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}) or (sum by(uuid) (nvidia_smi_gpu_info{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}) * 0))", + "legendFormat": "", + "refId": "A" + } + ], + "title": "Throttling", + "type": "stat" + }, + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "description": "Lifetime uncorrected ECC error count. Uses the aggregate total when the driver reports one, otherwise the sum of all reported aggregate uncorrected counters (some GPUs report subcounters without a total). Any value above zero means the GPU memory has produced uncorrectable errors and the card needs attention. Shows 'No ECC reported' when the driver reports no uncorrected-error counters at all (most GeForce cards).", + "fieldConfig": { + "defaults": { + "color": { + "mode": "fixed", + "fixedColor": "text" + }, + "decimals": 0, + "mappings": [ + { + "type": "value", + "options": { + "0": { + "text": "None", + "color": "#56A64B", + "index": 0 + } + } + } + ], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "#6E7B8B", + "value": null + }, + { + "color": "#E02F44", + "value": 1 + } + ] + }, + "unit": "", + "noValue": "No ECC reported" + }, + "overrides": [] + }, + "gridPos": { + "h": 3, + "w": 8, + "x": 16, + "y": 0 + }, + "id": 41, + "options": { + "colorMode": "background", + "graphMode": "none", + "justifyMode": "auto", + "orientation": "auto", + "percentChangeColorMode": "standard", + "reduceOptions": { + "calcs": [ + "last" + ], + "fields": "", + "values": false + }, + "showPercentChange": false, + "text": {}, + "textMode": "value", + "wideLayout": true + }, + "pluginVersion": "11.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "expr": "sum by(uuid) (nvidia_smi_ecc_errors_uncorrected_aggregate_total{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}) or sum by(uuid) ({__name__=~\"nvidia_smi_ecc_errors_uncorrected_aggregate_.+\", uuid=\"$gpu\", instance=\"$node\", job=\"$job\"})", + "legendFormat": "", + "refId": "A" + } + ], + "title": "ECC Errors (uncorrected)", + "type": "stat" + }, + { + "collapsed": false, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 3 + }, + "id": 49, + "panels": [], + "title": "Overview", + "type": "row" + }, + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "description": "Current performance state, P0 (maximum performance) to P15 (minimum). Lower P means busier. Vivid blue = working hard, pale blue = idle. This is operational state, not a health signal.", + "fieldConfig": { + "defaults": { + "color": { + "mode": "fixed", + "fixedColor": "text" + }, + "decimals": 0, + "mappings": [ + { + "type": "value", + "options": { + "0": { + "text": "P0 \u00b7 Max perf", + "color": "#2B62D9", + "index": 0 + }, + "1": { + "text": "P1", + "color": "#3469DB", + "index": 1 + }, + "2": { + "text": "P2", + "color": "#3E70DD", + "index": 2 + }, + "3": { + "text": "P3", + "color": "#4777DE", + "index": 3 + }, + "4": { + "text": "P4", + "color": "#507FE0", + "index": 4 + }, + "5": { + "text": "P5", + "color": "#5986E2", + "index": 5 + }, + "6": { + "text": "P6", + "color": "#638DE4", + "index": 6 + }, + "7": { + "text": "P7", + "color": "#6C94E6", + "index": 7 + }, + "8": { + "text": "P8 \u00b7 Idle", + "color": "#759BE7", + "index": 8 + }, + "9": { + "text": "P9", + "color": "#7EA2E9", + "index": 9 + }, + "10": { + "text": "P10", + "color": "#88A9EB", + "index": 10 + }, + "11": { + "text": "P11", + "color": "#91B0ED", + "index": 11 + }, + "12": { + "text": "P12 \u00b7 Deep idle", + "color": "#9AB8EF", + "index": 12 + }, + "13": { + "text": "P13", + "color": "#A3BFF0", + "index": 13 + }, + "14": { + "text": "P14", + "color": "#ADC6F2", + "index": 14 + }, + "15": { + "text": "P15 \u00b7 Min", + "color": "#B6CDF4", + "index": 15 + } + } + } + ], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "" + }, + "overrides": [] + }, + "gridPos": { + "h": 2, + "w": 6, + "x": 0, + "y": 4 + }, + "id": 22, + "options": { + "colorMode": "background_solid", + "graphMode": "none", + "justifyMode": "auto", + "orientation": "auto", + "percentChangeColorMode": "standard", + "reduceOptions": { + "calcs": [ + "last" + ], + "fields": "", + "values": false + }, + "showPercentChange": false, + "text": { + "valueSize": 20 + }, + "textMode": "value", + "wideLayout": true + }, + "pluginVersion": "11.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "exemplar": true, + "expr": "nvidia_smi_pstate{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}", + "interval": "", + "legendFormat": "", + "refId": "A" + } + ], + "title": "P-State", + "type": "stat" + }, + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "description": "Static identity of the selected GPU as reported by the driver.", + "fieldConfig": { + "defaults": { + "color": { + "mode": "fixed", + "fixedColor": "text" + }, + "custom": { + "align": "left", + "cellOptions": { + "type": "auto" + }, + "inspect": false, + "filterable": false + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "text", + "value": null + } + ] + } + }, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "Field" + }, + "properties": [ + { + "id": "custom.width", + "value": 120 + }, + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "#8A94A6" + } + }, + { + "id": "custom.cellOptions", + "value": { + "type": "color-text" + } + } + ] + } + ] + }, + "gridPos": { + "h": 7, + "w": 6, + "x": 0, + "y": 6 + }, + "id": 42, + "options": { + "showHeader": false, + "cellHeight": "sm", + "footer": { + "show": false, + "reducer": [ + "sum" + ], + "countRows": false, + "fields": "" + } + }, + "pluginVersion": "11.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "expr": "nvidia_smi_gpu_info{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}", + "format": "table", + "instant": true, + "legendFormat": "", + "refId": "A" + } + ], + "transformations": [ + { + "id": "organize", + "options": { + "excludeByName": { + "Time": true, + "__name__": true, + "Value": true, + "instance": true, + "job": true, + "uuid": true, + "index": true, + "driver_model_current": true, + "driver_model_pending": true, + "pci_sub_device_id": true, + "serial": true + }, + "indexByName": { + "name": 0, + "driver_version": 1, + "cuda_version": 2, + "vbios_version": 3, + "compute_cap": 4, + "pci_bus_id": 5 + }, + "renameByName": { + "name": "GPU", + "driver_version": "Driver", + "vbios_version": "VBIOS", + "compute_cap": "Compute Cap", + "pci_bus_id": "PCI Bus", + "cuda_version": "CUDA" + } + } + }, + { + "id": "transpose", + "options": {} + } + ], + "title": "GPU Info", + "type": "table" + }, + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "description": "Compute mode is a configuration, not a health state.", + "fieldConfig": { + "defaults": { + "color": { + "mode": "fixed", + "fixedColor": "text" + }, + "decimals": 0, + "mappings": [ + { + "type": "value", + "options": { + "0": { + "text": "Default", + "color": "#5B7A9D", + "index": 0 + }, + "1": { + "text": "Exclusive Thread", + "color": "#8A5CD1", + "index": 1 + }, + "2": { + "text": "Prohibited", + "color": "#B455A0", + "index": 2 + }, + "3": { + "text": "Exclusive Process", + "color": "#3E9E9E", + "index": 3 + } + } + } + ], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "" + }, + "overrides": [] + }, + "gridPos": { + "h": 2, + "w": 6, + "x": 0, + "y": 13 + }, + "id": 36, + "options": { + "colorMode": "background_solid", + "graphMode": "none", + "justifyMode": "auto", + "orientation": "auto", + "percentChangeColorMode": "standard", + "reduceOptions": { + "calcs": [ + "last" + ], + "fields": "", + "values": false + }, + "showPercentChange": false, + "text": { + "valueSize": 20 + }, + "textMode": "value", + "wideLayout": true + }, + "pluginVersion": "11.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "editorMode": "code", + "expr": "nvidia_smi_compute_mode{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}", + "instant": false, + "legendFormat": "__auto", + "range": true, + "refId": "A" + } + ], + "title": "Compute Mode", + "type": "stat" + }, + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "description": "Percent of time over the past sample period during which one or more kernels was executing on the GPU.\nThe sample period may be between 1 second and 1/6 second depending on the product. MIG-enabled cards do not report a whole-GPU number, which shows as N/A; per-instance activity is in the MIG panels.", + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "mappings": [ + { + "type": "value", + "options": { + "-1": { + "text": "N/A", + "color": "#6E7B8B", + "index": 0 + } + } + } + ], + "max": 1, + "min": 0, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "percentunit" + }, + "overrides": [] + }, + "gridPos": { + "h": 5, + "w": 3, + "x": 6, + "y": 4 + }, + "id": 6, + "options": { + "minVizHeight": 75, + "minVizWidth": 75, + "orientation": "auto", + "reduceOptions": { + "calcs": [ + "last" + ], + "fields": "", + "values": false + }, + "showThresholdLabels": false, + "showThresholdMarkers": true, + "sizing": "auto", + "text": {} + }, + "pluginVersion": "11.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "exemplar": true, + "expr": "sum(nvidia_smi_utilization_gpu_ratio{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}) or sum(nvidia_smi_gpu_info{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}) * 0 - 1", + "interval": "", + "legendFormat": "{{uuid}}", + "refId": "A" + } + ], + "title": "GPU Util", + "type": "gauge" + }, + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "description": "Board power draw as a share of the power limit in force: the enforced limit when the driver reports one, otherwise the software or default limit.", + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "mappings": [], + "max": 1, + "min": 0, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "yellow", + "value": null + } + ] + }, + "unit": "percentunit" + }, + "overrides": [] + }, + "gridPos": { + "h": 5, + "w": 3, + "x": 9, + "y": 4 + }, + "id": 21, + "options": { + "minVizHeight": 75, + "minVizWidth": 75, + "orientation": "auto", + "reduceOptions": { + "calcs": [ + "last" + ], + "fields": "", + "values": false + }, + "showThresholdLabels": false, + "showThresholdMarkers": true, + "sizing": "auto", + "text": {} + }, + "pluginVersion": "11.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "exemplar": true, + "expr": "nvidia_smi_power_draw_watts{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"} / (nvidia_smi_enforced_power_limit_watts{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"} or nvidia_smi_power_limit_watts{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"} or nvidia_smi_power_default_limit_watts{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"})", + "interval": "", + "legendFormat": "", + "refId": "A" + } + ], + "title": "Power", + "type": "gauge" + }, + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "description": "The fan speed value is the percent of the product's maximum noise tolerance fan speed that the device's fan is currently intended to run at. This value may exceed 100% in certain cases. Note: The reported speed is the intended fan speed. If the fan is physically blocked and unable to spin, this output will not match the actual fan speed. Many parts do not report fan speeds because they rely on cooling via fans in the surrounding enclosure.\n", + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "mappings": [ + { + "type": "value", + "options": { + "-1": { + "text": "N/A", + "color": "#6E7B8B", + "index": 0 + } + } + } + ], + "max": 1, + "min": 0, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "orange", + "value": null + } + ] + }, + "unit": "percentunit" + }, + "overrides": [] + }, + "gridPos": { + "h": 5, + "w": 3, + "x": 12, + "y": 4 + }, + "id": 4, + "options": { + "minVizHeight": 75, + "minVizWidth": 75, + "orientation": "auto", + "reduceOptions": { + "calcs": [ + "last" + ], + "fields": "", + "values": false + }, + "showThresholdLabels": false, + "showThresholdMarkers": true, + "sizing": "auto", + "text": {} + }, + "pluginVersion": "11.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "exemplar": true, + "expr": "sum(nvidia_smi_fan_speed_ratio{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}) or sum(nvidia_smi_gpu_info{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}) * 0 - 1", + "interval": "", + "legendFormat": "", + "refId": "A" + } + ], + "title": "Fan", + "type": "gauge" + }, + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "description": "Core GPU temperature. in degrees C.", + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "mappings": [], + "max": 100, + "min": 0, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "#56A64B", + "value": null + }, + { + "color": "#F2CC0C", + "value": 70 + }, + { + "color": "#E02F44", + "value": 80 + } + ] + }, + "unit": "celsius" + }, + "overrides": [] + }, + "gridPos": { + "h": 5, + "w": 3, + "x": 15, + "y": 4 + }, + "id": 16, + "options": { + "minVizHeight": 75, + "minVizWidth": 75, + "orientation": "auto", + "reduceOptions": { + "calcs": [ + "last" + ], + "fields": "", + "values": false + }, + "showThresholdLabels": false, + "showThresholdMarkers": true, + "sizing": "auto", + "text": {} + }, + "pluginVersion": "11.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "exemplar": true, + "expr": "nvidia_smi_temperature_gpu{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}", + "interval": "", + "legendFormat": "{{uuid}}", + "refId": "A" + } + ], + "title": "Temperature", + "type": "gauge" + }, + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "description": "Current frequency of graphics (shader) clock\n/\nMaximum frequency of graphics (shader) clock.\n", + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "mappings": [], + "max": 1, + "min": 0, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "blue", + "value": null + } + ] + }, + "unit": "percentunit" + }, + "overrides": [] + }, + "gridPos": { + "h": 5, + "w": 3, + "x": 6, + "y": 9 + }, + "id": 20, + "options": { + "minVizHeight": 75, + "minVizWidth": 75, + "orientation": "auto", + "reduceOptions": { + "calcs": [ + "last" + ], + "fields": "", + "values": false + }, + "showThresholdLabels": false, + "showThresholdMarkers": true, + "sizing": "auto", + "text": {} + }, + "pluginVersion": "11.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "exemplar": true, + "expr": "nvidia_smi_clocks_current_graphics_clock_hz{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"} / nvidia_smi_clocks_max_graphics_clock_hz{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}", + "interval": "", + "legendFormat": "", + "refId": "A" + } + ], + "title": "GPU Clock", + "type": "gauge" + }, + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "description": "Current frequency of memory clock / Maximum frequency of memory clock", + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "mappings": [], + "max": 1, + "min": 0, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "blue", + "value": null + } + ] + }, + "unit": "percentunit" + }, + "overrides": [] + }, + "gridPos": { + "h": 5, + "w": 3, + "x": 9, + "y": 9 + }, + "id": 33, + "options": { + "minVizHeight": 75, + "minVizWidth": 75, + "orientation": "auto", + "reduceOptions": { + "calcs": [ + "last" + ], + "fields": "", + "values": false + }, + "showThresholdLabels": false, + "showThresholdMarkers": true, + "sizing": "auto", + "text": {} + }, + "pluginVersion": "11.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "exemplar": true, + "expr": "nvidia_smi_clocks_current_memory_clock_hz{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"} / nvidia_smi_clocks_max_memory_clock_hz{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}", + "interval": "", + "legendFormat": "", + "refId": "A" + } + ], + "title": "Memory Clock", + "type": "gauge" + }, + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "description": "Total memory allocated by active contexts / Total installed GPU memory.", + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "mappings": [], + "max": 1, + "min": 0, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "purple", + "value": null + }, + { + "color": "#F2CC0C", + "value": 0.85 + }, + { + "color": "#E02F44", + "value": 0.95 + } + ] + }, + "unit": "percentunit" + }, + "overrides": [] + }, + "gridPos": { + "h": 5, + "w": 3, + "x": 12, + "y": 9 + }, + "id": 25, + "options": { + "minVizHeight": 75, + "minVizWidth": 75, + "orientation": "auto", + "reduceOptions": { + "calcs": [ + "last" + ], + "fields": "", + "values": false + }, + "showThresholdLabels": false, + "showThresholdMarkers": true, + "sizing": "auto", + "text": {} + }, + "pluginVersion": "11.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "exemplar": true, + "expr": "nvidia_smi_memory_used_bytes{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"} / nvidia_smi_memory_total_bytes{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}", + "interval": "", + "legendFormat": "", + "refId": "A" + } + ], + "title": "Memory Alloc", + "type": "gauge" + }, + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "description": "Percent of time over the past sample period during which global (device) memory was being read or written.\nThe sample period may be between 1 second and 1/6 second depending on the product. MIG-enabled cards do not report a whole-GPU number, which shows as N/A; per-instance activity is in the MIG panels.", + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "mappings": [ + { + "type": "value", + "options": { + "-1": { + "text": "N/A", + "color": "#6E7B8B", + "index": 0 + } + } + } + ], + "max": 1, + "min": 0, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "percentunit" + }, + "overrides": [] + }, + "gridPos": { + "h": 5, + "w": 3, + "x": 15, + "y": 9 + }, + "id": 7, + "options": { + "minVizHeight": 75, + "minVizWidth": 75, + "orientation": "auto", + "reduceOptions": { + "calcs": [ + "last" + ], + "fields": "", + "values": false + }, + "showThresholdLabels": false, + "showThresholdMarkers": true, + "sizing": "auto", + "text": {} + }, + "pluginVersion": "11.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "exemplar": true, + "expr": "sum(nvidia_smi_utilization_memory_ratio{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}) or sum(nvidia_smi_gpu_info{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}) * 0 - 1", + "interval": "", + "legendFormat": "", + "refId": "A" + } + ], + "title": "Memory Util", + "type": "gauge" + }, + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "description": "Percent of time over the past sample period during which global (device) memory was being read or written.\nThe sample period may be between 1 second and 1/6 second depending on the product.", + "fieldConfig": { + "defaults": { + "color": { + "mode": "fixed", + "fixedColor": "green" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 10, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "max": 1, + "min": 0, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "percentunit", + "noValue": "Not reported by driver" + }, + "overrides": [] + }, + "gridPos": { + "h": 5, + "w": 6, + "x": 18, + "y": 4 + }, + "id": 11, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": false + }, + "tooltip": { + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "11.1.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "exemplar": true, + "expr": "nvidia_smi_utilization_memory_ratio{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}", + "interval": "", + "legendFormat": "{{uuid}}", + "refId": "A" + } + ], + "title": "Memory Utilization %", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "description": "Percent of time over the past sample period during which one or more kernels was executing on the GPU.\nThe sample period may be between 1 second and 1/6 second depending on the product.", + "fieldConfig": { + "defaults": { + "color": { + "mode": "fixed", + "fixedColor": "green" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 10, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "max": 1, + "min": 0, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "percentunit", + "noValue": "Not reported by driver" + }, + "overrides": [] + }, + "gridPos": { + "h": 5, + "w": 6, + "x": 18, + "y": 9 + }, + "id": 10, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": false + }, + "tooltip": { + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "11.1.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "exemplar": true, + "expr": "nvidia_smi_utilization_gpu_ratio{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}", + "interval": "", + "legendFormat": "", + "refId": "A" + } + ], + "title": "GPU Utilization %", + "type": "timeseries" + }, + { + "collapsed": false, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 15 + }, + "id": 50, + "panels": [], + "title": "State History", + "type": "row" + }, + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "description": "Throttle reasons over time. A colored band means the reason was active. Hardware slowdowns (thermal, power brake) are serious, software power cap is routine under sustained load, and Idle just means the GPU had nothing to do.", + "fieldConfig": { + "defaults": { + "color": { + "mode": "fixed", + "fixedColor": "text" + }, + "custom": { + "fillOpacity": 80, + "lineWidth": 0, + "spanNulls": false, + "insertNulls": false, + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + } + }, + "mappings": [ + { + "type": "value", + "options": { + "0": { + "text": "Not Active", + "color": "transparent", + "index": 0 + }, + "1": { + "text": "Active", + "color": "#6E7B8B", + "index": 1 + } + } + } + ], + "noValue": "-" + }, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "SW Power Cap" + }, + "properties": [ + { + "id": "mappings", + "value": [ + { + "type": "value", + "options": { + "0": { + "text": "Not Active", + "color": "transparent", + "index": 0 + }, + "1": { + "text": "Active", + "color": "#F2CC0C", + "index": 1 + } + } + } + ] + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "SW Thermal Slowdown" + }, + "properties": [ + { + "id": "mappings", + "value": [ + { + "type": "value", + "options": { + "0": { + "text": "Not Active", + "color": "transparent", + "index": 0 + }, + "1": { + "text": "Active", + "color": "#FF9830", + "index": 1 + } + } + } + ] + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "HW Thermal Slowdown" + }, + "properties": [ + { + "id": "mappings", + "value": [ + { + "type": "value", + "options": { + "0": { + "text": "Not Active", + "color": "transparent", + "index": 0 + }, + "1": { + "text": "Active", + "color": "#E02F44", + "index": 1 + } + } + } + ] + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "HW Power Brake" + }, + "properties": [ + { + "id": "mappings", + "value": [ + { + "type": "value", + "options": { + "0": { + "text": "Not Active", + "color": "transparent", + "index": 0 + }, + "1": { + "text": "Active", + "color": "#E02F44", + "index": 1 + } + } + } + ] + } + ] + } + ] + }, + "gridPos": { + "h": 6, + "w": 12, + "x": 0, + "y": 16 + }, + "id": 38, + "options": { + "alignValue": "left", + "legend": { + "displayMode": "list", + "placement": "bottom", + "showLegend": false + }, + "mergeValues": true, + "rowHeight": 0.85, + "showValue": "never", + "tooltip": { + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "11.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "expr": "nvidia_smi_clocks_event_reasons_gpu_idle{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"} or nvidia_smi_clocks_throttle_reasons_gpu_idle{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}", + "legendFormat": "Idle", + "refId": "A" + }, + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "expr": "nvidia_smi_clocks_event_reasons_sw_power_cap{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"} or nvidia_smi_clocks_throttle_reasons_sw_power_cap{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}", + "legendFormat": "SW Power Cap", + "refId": "B" + }, + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "expr": "nvidia_smi_clocks_event_reasons_sw_thermal_slowdown{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"} or nvidia_smi_clocks_throttle_reasons_sw_thermal_slowdown{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}", + "legendFormat": "SW Thermal Slowdown", + "refId": "C" + }, + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "expr": "nvidia_smi_clocks_event_reasons_hw_thermal_slowdown{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"} or nvidia_smi_clocks_throttle_reasons_hw_thermal_slowdown{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}", + "legendFormat": "HW Thermal Slowdown", + "refId": "D" + }, + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "expr": "nvidia_smi_clocks_event_reasons_hw_power_brake_slowdown{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"} or nvidia_smi_clocks_throttle_reasons_hw_power_brake_slowdown{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}", + "legendFormat": "HW Power Brake", + "refId": "E" + }, + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "expr": "nvidia_smi_clocks_event_reasons_applications_clocks_setting{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"} or nvidia_smi_clocks_throttle_reasons_applications_clocks_setting{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}", + "legendFormat": "App Clocks Setting", + "refId": "F" + }, + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "expr": "nvidia_smi_clocks_event_reasons_sync_boost{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"} or nvidia_smi_clocks_throttle_reasons_sync_boost{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}", + "legendFormat": "Sync Boost", + "refId": "G" + } + ], + "title": "Throttle Reasons (history)", + "type": "state-timeline" + }, + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "description": "Performance state over time. P0 is maximum performance, P15 minimum. Lower P means busier. This is operational state, not a health signal.", + "fieldConfig": { + "defaults": { + "color": { + "mode": "fixed", + "fixedColor": "text" + }, + "custom": { + "fillOpacity": 80, + "lineWidth": 0, + "spanNulls": false, + "insertNulls": false, + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + } + }, + "mappings": [ + { + "type": "value", + "options": { + "0": { + "text": "P0 \u00b7 Max perf", + "color": "#2B62D9", + "index": 0 + }, + "1": { + "text": "P1", + "color": "#3469DB", + "index": 1 + }, + "2": { + "text": "P2", + "color": "#3E70DD", + "index": 2 + }, + "3": { + "text": "P3", + "color": "#4777DE", + "index": 3 + }, + "4": { + "text": "P4", + "color": "#507FE0", + "index": 4 + }, + "5": { + "text": "P5", + "color": "#5986E2", + "index": 5 + }, + "6": { + "text": "P6", + "color": "#638DE4", + "index": 6 + }, + "7": { + "text": "P7", + "color": "#6C94E6", + "index": 7 + }, + "8": { + "text": "P8 \u00b7 Idle", + "color": "#759BE7", + "index": 8 + }, + "9": { + "text": "P9", + "color": "#7EA2E9", + "index": 9 + }, + "10": { + "text": "P10", + "color": "#88A9EB", + "index": 10 + }, + "11": { + "text": "P11", + "color": "#91B0ED", + "index": 11 + }, + "12": { + "text": "P12 \u00b7 Deep idle", + "color": "#9AB8EF", + "index": 12 + }, + "13": { + "text": "P13", + "color": "#A3BFF0", + "index": 13 + }, + "14": { + "text": "P14", + "color": "#ADC6F2", + "index": 14 + }, + "15": { + "text": "P15 \u00b7 Min", + "color": "#B6CDF4", + "index": 15 + } + } + } + ] + }, + "overrides": [] + }, + "gridPos": { + "h": 6, + "w": 12, + "x": 12, + "y": 16 + }, + "id": 39, + "options": { + "alignValue": "left", + "legend": { + "displayMode": "list", + "placement": "bottom", + "showLegend": false + }, + "mergeValues": true, + "rowHeight": 0.85, + "showValue": "auto", + "tooltip": { + "mode": "single", + "sort": "none" + } + }, + "pluginVersion": "11.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "expr": "nvidia_smi_pstate{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}", + "legendFormat": "P-State", + "refId": "A" + } + ], + "title": "P-State (history)", + "type": "state-timeline" + }, + { + "collapsed": false, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 22 + }, + "id": 45, + "panels": [], + "title": "Power & Thermals", + "type": "row" + }, + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "description": "Total memory allocated by active contexts.", + "fieldConfig": { + "defaults": { + "color": { + "mode": "fixed", + "fixedColor": "purple" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 10, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "min": 0, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "purple", + "value": null + } + ] + }, + "unit": "bytes" + }, + "overrides": [] + }, + "gridPos": { + "h": 5, + "w": 6, + "x": 0, + "y": 23 + }, + "id": 17, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": false + }, + "tooltip": { + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "11.1.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "exemplar": true, + "expr": "nvidia_smi_memory_used_bytes{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}", + "interval": "", + "legendFormat": "{{uuid}}", + "refId": "A" + } + ], + "title": "Memory Allocation", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "description": "Core GPU temperature. in degrees C.", + "fieldConfig": { + "defaults": { + "color": { + "mode": "fixed", + "fixedColor": "orange" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 10, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "line" + } + }, + "mappings": [], + "max": 100, + "min": 0, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "transparent", + "value": null + }, + { + "color": "#E02F44", + "value": 80 + } + ] + }, + "unit": "celsius" + }, + "overrides": [] + }, + "gridPos": { + "h": 5, + "w": 6, + "x": 6, + "y": 23 + }, + "id": 15, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": false + }, + "tooltip": { + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "11.1.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "exemplar": true, + "expr": "nvidia_smi_temperature_gpu{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}", + "interval": "", + "legendFormat": "{{uuid}}", + "refId": "A" + } + ], + "title": "Temperature", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "description": "The last measured power draw for the entire board, in watts. Only available if power management is supported; the reading is accurate to within +/- 5 watts.", + "fieldConfig": { + "defaults": { + "color": { + "mode": "fixed", + "fixedColor": "yellow" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 10, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "min": 0, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "yellow", + "value": null + } + ] + }, + "unit": "watt" + }, + "overrides": [] + }, + "gridPos": { + "h": 5, + "w": 6, + "x": 12, + "y": 23 + }, + "id": 8, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": false + }, + "tooltip": { + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "11.1.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "exemplar": true, + "expr": "nvidia_smi_power_draw_watts{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}", + "interval": "", + "legendFormat": "{{uuid}}", + "refId": "A" + } + ], + "title": "Power Draw", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "description": "The fan speed value is the percent of the product's maximum noise tolerance fan speed that the device's fan is currently intended to run at. This value may exceed 100% in certain cases. Note: The reported speed is the intended fan speed. If the fan is physically blocked and unable to spin, this output will not match the actual fan speed. Many parts do not report fan speeds because they rely on cooling via fans in the surrounding enclosure.\n", + "fieldConfig": { + "defaults": { + "color": { + "mode": "fixed", + "fixedColor": "orange" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 10, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "max": 1, + "min": 0, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "orange", + "value": null + } + ] + }, + "unit": "percentunit", + "noValue": "N/A" + }, + "overrides": [] + }, + "gridPos": { + "h": 5, + "w": 6, + "x": 18, + "y": 23 + }, + "id": 9, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": false + }, + "tooltip": { + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "11.1.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "exemplar": true, + "expr": "nvidia_smi_fan_speed_ratio{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}", + "interval": "", + "legendFormat": "{{uuid}}", + "refId": "A" + } + ], + "title": "Fan Speed %", + "type": "timeseries" + }, + { + "collapsed": true, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 28 + }, + "id": 46, + "panels": [ + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "description": "Current frequency of graphics (shader) clock.", + "fieldConfig": { + "defaults": { + "color": { + "mode": "fixed", + "fixedColor": "blue" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 10, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "min": 0, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "blue", + "value": null + } + ] + }, + "unit": "hertz" + }, + "overrides": [] + }, + "gridPos": { + "h": 5, + "w": 6, + "x": 0, + "y": 28 + }, + "id": 12, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": false + }, + "tooltip": { + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "11.1.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "exemplar": true, + "expr": "nvidia_smi_clocks_current_graphics_clock_hz{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}", + "format": "time_series", + "interval": "", + "legendFormat": "{{uuid}}", + "refId": "A" + } + ], + "title": "Graphics Clock Speed", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "description": "Current frequency of video encoder/decoder clock.", + "fieldConfig": { + "defaults": { + "color": { + "mode": "fixed", + "fixedColor": "blue" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 10, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "min": 0, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "blue", + "value": null + } + ] + }, + "unit": "hertz" + }, + "overrides": [] + }, + "gridPos": { + "h": 5, + "w": 6, + "x": 6, + "y": 28 + }, + "id": 19, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": false + }, + "tooltip": { + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "11.1.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "exemplar": true, + "expr": "nvidia_smi_clocks_current_video_clock_hz{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}", + "format": "time_series", + "interval": "", + "legendFormat": "{{uuid}}", + "refId": "A" + } + ], + "title": "Video Clock Speed", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "description": "Current frequency of SM (Streaming Multiprocessor) clock.", + "fieldConfig": { + "defaults": { + "color": { + "mode": "fixed", + "fixedColor": "blue" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 10, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "min": 0, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "blue", + "value": null + } + ] + }, + "unit": "hertz" + }, + "overrides": [] + }, + "gridPos": { + "h": 5, + "w": 6, + "x": 12, + "y": 28 + }, + "id": 24, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": false + }, + "tooltip": { + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "11.1.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "exemplar": true, + "expr": "nvidia_smi_clocks_current_sm_clock_hz{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}", + "format": "time_series", + "interval": "", + "legendFormat": "{{uuid}}", + "refId": "A" + } + ], + "title": "SM Clock Speed", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "description": "Current frequency of memory clock.", + "fieldConfig": { + "defaults": { + "color": { + "mode": "fixed", + "fixedColor": "blue" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 10, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "min": 0, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "blue", + "value": null + } + ] + }, + "unit": "hertz" + }, + "overrides": [] + }, + "gridPos": { + "h": 5, + "w": 6, + "x": 18, + "y": 28 + }, + "id": 18, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": false + }, + "tooltip": { + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "11.1.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "exemplar": true, + "expr": "nvidia_smi_clocks_current_memory_clock_hz{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}", + "format": "time_series", + "interval": "", + "legendFormat": "{{uuid}}", + "refId": "A" + } + ], + "title": "Memory Clock Speed", + "type": "timeseries" + } + ], + "title": "Clocks", + "type": "row" + }, + { + "collapsed": true, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 29 + }, + "id": 47, + "panels": [ + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "description": "Processes with a compute context on the selected GPU, with used memory and its share of total VRAM. Requires the exporter to run with --collect.compute-apps. A memory value of 0 B means the platform does not report per-process memory (e.g. Windows in WDDM mode).", + "fieldConfig": { + "defaults": { + "color": { + "mode": "fixed", + "fixedColor": "text" + }, + "custom": { + "align": "left", + "cellOptions": { + "type": "auto" + }, + "inspect": false, + "filterable": false + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "text", + "value": null + } + ] + } + }, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "PID" + }, + "properties": [ + { + "id": "custom.width", + "value": 70 + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Memory" + }, + "properties": [ + { + "id": "unit", + "value": "bytes" + }, + { + "id": "custom.width", + "value": 100 + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Share" + }, + "properties": [ + { + "id": "unit", + "value": "percentunit" + }, + { + "id": "min", + "value": 0 + }, + { + "id": "max", + "value": 1 + }, + { + "id": "custom.width", + "value": 140 + }, + { + "id": "custom.cellOptions", + "value": { + "type": "gauge", + "mode": "gradient" + } + }, + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "purple" + } + } + ] + } + ] + }, + "gridPos": { + "x": 0, + "y": 29, + "w": 9, + "h": 8 + }, + "id": 43, + "options": { + "showHeader": true, + "cellHeight": "sm", + "footer": { + "show": false, + "reducer": [ + "sum" + ], + "countRows": false, + "fields": "" + }, + "sortBy": [ + { + "displayName": "Memory", + "desc": true + } + ] + }, + "pluginVersion": "11.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "expr": "nvidia_smi_compute_app_used_memory_bytes{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"} or (nvidia_smi_compute_app_info{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"} * 0)", + "format": "table", + "instant": true, + "legendFormat": "", + "refId": "A" + }, + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "expr": "(nvidia_smi_compute_app_used_memory_bytes{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"} or (nvidia_smi_compute_app_info{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"} * 0)) / on(uuid) group_left() nvidia_smi_memory_total_bytes{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}", + "format": "table", + "instant": true, + "legendFormat": "", + "refId": "B" + } + ], + "transformations": [ + { + "id": "merge", + "options": {} + }, + { + "id": "organize", + "options": { + "excludeByName": { + "Time": true, + "uuid": true, + "__name__": true, + "instance": true, + "job": true + }, + "indexByName": { + "process_name": 0, + "pid": 1, + "Value #A": 2, + "Value #B": 3 + }, + "renameByName": { + "process_name": "Process", + "pid": "PID", + "Value #A": "Memory", + "Value #B": "Share" + } + } + } + ], + "title": "GPU Processes", + "type": "table" + }, + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "description": "GPU memory per process over time, stacked. Processes appear and disappear as they open and close compute contexts. Empty when --collect.compute-apps is off or the platform reports no per-process memory.", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "drawStyle": "line", + "lineWidth": 1, + "fillOpacity": 35, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "mode": "normal", + "group": "A" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "min": 0, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "transparent", + "value": null + } + ] + }, + "unit": "bytes", + "noValue": "No process data" + }, + "overrides": [] + }, + "gridPos": { + "x": 9, + "y": 29, + "w": 15, + "h": 11 + }, + "id": 51, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "desc" + } + }, + "pluginVersion": "11.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "expr": "nvidia_smi_compute_app_used_memory_bytes{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}", + "legendFormat": "{{process_name}} ({{pid}})", + "refId": "A" + } + ], + "title": "Process Memory Over Time", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "description": "Health of the per-process collection on the selected node. OK means the last collection succeeded, so an empty table really means no compute processes. Failing means nvidia-smi could not deliver per-process data. Not enabled means the exporter runs without --collect.compute-apps.", + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "decimals": 0, + "mappings": [ + { + "type": "value", + "options": { + "0": { + "text": "Failing", + "color": "#E02F44", + "index": 0 + }, + "1": { + "text": "OK", + "color": "#56A64B", + "index": 1 + } + } + } + ], + "noValue": "Not enabled (--collect.compute-apps)", + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "#6E7B8B", + "value": null + } + ] + }, + "unit": "none" + }, + "overrides": [] + }, + "gridPos": { + "x": 0, + "y": 37, + "w": 9, + "h": 3 + }, + "id": 52, + "options": { + "colorMode": "background_solid", + "graphMode": "none", + "justifyMode": "center", + "orientation": "auto", + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "textMode": "value", + "wideLayout": false, + "text": { + "valueSize": 24 + } + }, + "pluginVersion": "11.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "expr": "nvidia_smi_compute_apps_last_collect_success{instance=\"$node\", job=\"$job\"}", + "legendFormat": "", + "refId": "A" + } + ], + "title": "Process Collection", + "type": "stat" + } + ], + "title": "Processes", + "type": "row" + }, + { + "collapsed": true, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 30 + }, + "id": 48, + "panels": [ + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "description": "NVLink fabric registration state. Completed is the healthy steady state. In Progress is normal briefly at boot, but persistent In Progress or Not Started means the GPU is not registered with the fabric manager and NVLink workloads will fail.", + "fieldConfig": { + "defaults": { + "color": { + "mode": "fixed", + "fixedColor": "text" + }, + "decimals": 0, + "mappings": [ + { + "type": "value", + "options": { + "0": { + "text": "Not Supported", + "color": "#6E7B8B", + "index": 0 + }, + "1": { + "text": "Not Started", + "color": "#FF9830", + "index": 1 + }, + "2": { + "text": "In Progress", + "color": "#F2CC0C", + "index": 2 + }, + "3": { + "text": "Completed", + "color": "#56A64B", + "index": 3 + } + } + } + ], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "", + "noValue": "No NVLink fabric" + }, + "overrides": [] + }, + "gridPos": { + "h": 4, + "w": 6, + "x": 0, + "y": 30 + }, + "id": 37, + "options": { + "colorMode": "background", + "graphMode": "none", + "justifyMode": "auto", + "orientation": "auto", + "percentChangeColorMode": "standard", + "reduceOptions": { + "calcs": [ + "last" + ], + "fields": "", + "values": false + }, + "showPercentChange": false, + "text": {}, + "textMode": "value", + "wideLayout": true + }, + "pluginVersion": "11.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "editorMode": "code", + "expr": "nvidia_smi_fabric_state{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}", + "instant": false, + "legendFormat": "__auto", + "range": true, + "refId": "A" + } + ], + "title": "Fabric State", + "type": "stat" + }, + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "description": "All uncorrected ECC error counters (aggregate = lifetime, volatile = since boot, broken down by memory subsystem). Empty on GPUs without ECC memory.", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "drawStyle": "line", + "lineWidth": 1, + "fillOpacity": 0, + "showPoints": "auto", + "spanNulls": false + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "transparent", + "value": null + } + ] + }, + "unit": "none", + "noValue": "No ECC reported" + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 18, + "x": 6, + "y": 30 + }, + "id": 44, + "options": { + "legend": { + "calcs": [ + "lastNotNull" + ], + "displayMode": "table", + "placement": "right", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "desc" + } + }, + "pluginVersion": "11.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "expr": "label_replace({__name__=~\"nvidia_smi_ecc_errors_uncorrected_.+\", uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}, \"kind\", \"$1\", \"__name__\", \"nvidia_smi_ecc_errors_uncorrected_(.+)\")", + "legendFormat": "{{kind}}", + "refId": "A" + } + ], + "title": "ECC Uncorrected (detail)", + "type": "timeseries" + } + ], + "title": "Datacenter Health", + "type": "row" + }, + { + "collapsed": true, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 31 + }, + "id": 53, + "panels": [ + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "description": "Total XID errors observed on this GPU since the exporter started (earlier history cannot be replayed, so a restart resets the count). Any value above zero deserves a look at the per-code panel next to this one. Served only by the nvml and demo backends; absent under the default exec backend.", + "fieldConfig": { + "defaults": { + "color": { + "mode": "fixed", + "fixedColor": "text" + }, + "decimals": 0, + "mappings": [ + { + "type": "value", + "options": { + "0": { + "text": "None", + "color": "#56A64B", + "index": 0 + } + } + } + ], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "#6E7B8B", + "value": null + }, + { + "color": "#E02F44", + "value": 1 + } + ] + }, + "unit": "", + "noValue": "No data yet" + }, + "overrides": [] + }, + "gridPos": { + "h": 5, + "w": 4, + "x": 0, + "y": 31 + }, + "id": 54, + "options": { + "colorMode": "background", + "graphMode": "none", + "justifyMode": "auto", + "orientation": "auto", + "percentChangeColorMode": "standard", + "reduceOptions": { + "calcs": [ + "last" + ], + "fields": "", + "values": false + }, + "showPercentChange": false, + "text": {}, + "textMode": "value", + "wideLayout": true + }, + "pluginVersion": "11.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "expr": "sum(nvidia_smi_xid_errors_total{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}) or sum(nvidia_smi_gpu_info{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}) * 0", + "legendFormat": "", + "refId": "A" + } + ], + "title": "XID Errors", + "type": "stat" + }, + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "description": "Seconds since the most recent XID error on this GPU. 'Never' means none were observed since the exporter started. This is the signal to alert on: unlike increase() on the error counter, it also sees each series' first event (see METRICS.md). Served only by the nvml and demo backends; absent under the default exec backend.", + "fieldConfig": { + "defaults": { + "color": { + "mode": "fixed", + "fixedColor": "text" + }, + "decimals": 0, + "mappings": [ + { + "type": "value", + "options": { + "-1": { + "text": "Never", + "color": "#56A64B", + "index": 0 + } + } + } + ], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "#E02F44", + "value": null + }, + { + "color": "#EAB839", + "value": 3600 + }, + { + "color": "#56A64B", + "value": 86400 + } + ] + }, + "unit": "s", + "noValue": "No data yet" + }, + "overrides": [] + }, + "gridPos": { + "h": 5, + "w": 4, + "x": 4, + "y": 31 + }, + "id": 61, + "options": { + "colorMode": "background", + "graphMode": "none", + "justifyMode": "auto", + "orientation": "auto", + "percentChangeColorMode": "standard", + "reduceOptions": { + "calcs": [ + "last" + ], + "fields": "", + "values": false + }, + "showPercentChange": false, + "text": {}, + "textMode": "value", + "wideLayout": true + }, + "pluginVersion": "11.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "expr": "time() - max(nvidia_smi_xid_last_timestamp_seconds{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}) or (sum(nvidia_smi_gpu_info{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}) * 0 - 1)", + "legendFormat": "", + "refId": "A" + } + ], + "title": "Last XID", + "type": "stat" + }, + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "description": "Cumulative count per XID code since the exporter started. A series appears when its first event arrives, so an empty panel means no errors were observed yet. Do not alert on increase() of this counter: a series' first event is invisible to it, alert on the last-timestamp metric instead (see METRICS.md). Served only by the nvml and demo backends; absent under the default exec backend.", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 10, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "yellow", + "value": null + } + ] + }, + "unit": "none", + "decimals": 0, + "noValue": "No XID data" + }, + "overrides": [] + }, + "gridPos": { + "h": 5, + "w": 8, + "x": 8, + "y": 31 + }, + "id": 55, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "11.1.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "exemplar": true, + "expr": "nvidia_smi_xid_errors_total{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}", + "interval": "", + "legendFormat": "XID {{xid}}", + "refId": "A" + } + ], + "title": "XID Errors by Code", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "description": "PCIe throughput per direction, sampled by the driver over 20ms windows. Requires --collect.pcie-throughput on the nvml backend. Served only by the nvml and demo backends; absent under the default exec backend.", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 10, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "yellow", + "value": null + } + ] + }, + "unit": "Bps", + "noValue": "No PCIe data" + }, + "overrides": [] + }, + "gridPos": { + "h": 5, + "w": 8, + "x": 16, + "y": 31 + }, + "id": 57, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "11.1.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "exemplar": true, + "expr": "nvidia_smi_pcie_throughput_tx_bytes_per_second{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}", + "interval": "", + "legendFormat": "TX", + "refId": "A" + }, + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "exemplar": true, + "expr": "nvidia_smi_pcie_throughput_rx_bytes_per_second{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}", + "interval": "", + "legendFormat": "RX", + "refId": "B" + } + ], + "title": "PCIe Throughput", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "description": "Compares power integrated from the energy counter against the sampled power draw. The energy counter is NVML-only, so the default backend draws only the sampled line.", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 10, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "yellow", + "value": null + } + ] + }, + "unit": "watt" + }, + "overrides": [] + }, + "gridPos": { + "h": 6, + "w": 8, + "x": 0, + "y": 36 + }, + "id": 56, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "11.1.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "exemplar": true, + "expr": "rate(nvidia_smi_energy_joules_total{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}[$__rate_interval])", + "interval": "", + "legendFormat": "from energy counter", + "refId": "A" + }, + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "exemplar": true, + "expr": "nvidia_smi_power_draw_watts{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}", + "interval": "", + "legendFormat": "sampled power draw", + "refId": "B" + } + ], + "title": "Power (energy counter vs sampled)", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "description": "Energy the GPU consumed over the trailing 24 hours, from the driver's cumulative counter (consumption actually measured, not a projection of the current rate). Served only by the nvml and demo backends; absent under the default exec backend.", + "fieldConfig": { + "defaults": { + "color": { + "mode": "fixed", + "fixedColor": "text" + }, + "decimals": 2, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "#6E7B8B", + "value": null + } + ] + }, + "unit": "kwatth", + "noValue": "No energy data" + }, + "overrides": [] + }, + "gridPos": { + "h": 6, + "w": 4, + "x": 8, + "y": 36 + }, + "id": 62, + "options": { + "colorMode": "background", + "graphMode": "none", + "justifyMode": "auto", + "orientation": "auto", + "percentChangeColorMode": "standard", + "reduceOptions": { + "calcs": [ + "last" + ], + "fields": "", + "values": false + }, + "showPercentChange": false, + "text": {}, + "textMode": "value", + "wideLayout": true + }, + "pluginVersion": "11.2.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "expr": "sum(increase(nvidia_smi_energy_joules_total{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}[24h])) / 3.6e6", + "legendFormat": "", + "refId": "A" + } + ], + "title": "Energy (trailing 24h)", + "type": "stat" + }, + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "description": "Framebuffer memory used per MIG GPU instance (memory is a GPU-instance-level resource, shared by its compute instances). Served only by the nvml and demo backends; absent under the default exec backend.", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 10, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "yellow", + "value": null + } + ] + }, + "unit": "bytes", + "noValue": "No MIG memory data" + }, + "overrides": [] + }, + "gridPos": { + "h": 6, + "w": 12, + "x": 12, + "y": 36 + }, + "id": 60, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "11.1.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "exemplar": true, + "expr": "(nvidia_smi_mig_memory_used_bytes{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}) * on(uuid, gpu_instance_id) group_left(profile) (max by (uuid, gpu_instance_id, profile) (label_replace(nvidia_smi_mig_info{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}, \"profile\", \"$1\", \"profile\", \"^[0-9]+c\\\\.(.*)$\")))", + "interval": "", + "legendFormat": "GI {{gpu_instance_id}} ({{profile}})", + "refId": "A" + } + ], + "title": "MIG Memory Used", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "description": "SM activity per MIG GPU instance. Empty on GPUs without MIG partitions; the first collection that sees an instance serves nothing (the sampling needs a pair of collections). Served only by the nvml and demo backends; absent under the default exec backend.", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 10, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "yellow", + "value": null + } + ] + }, + "unit": "percentunit", + "noValue": "No MIG activity data" + }, + "overrides": [] + }, + "gridPos": { + "h": 6, + "w": 12, + "x": 0, + "y": 42 + }, + "id": 58, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "11.1.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "exemplar": true, + "expr": "(nvidia_smi_mig_sm_activity_ratio{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}) * on(uuid, gpu_instance_id) group_left(profile) (max by (uuid, gpu_instance_id, profile) (label_replace(nvidia_smi_mig_info{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}, \"profile\", \"$1\", \"profile\", \"^[0-9]+c\\\\.(.*)$\")))", + "interval": "", + "legendFormat": "GI {{gpu_instance_id}} ({{profile}})", + "refId": "A" + } + ], + "title": "MIG SM Activity", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "description": "Tensor pipe activity per MIG GPU instance, the signal that shows whether an inference workload actually exercises the tensor cores. Served only by the nvml and demo backends; absent under the default exec backend.", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 10, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "yellow", + "value": null + } + ] + }, + "unit": "percentunit", + "noValue": "No MIG activity data" + }, + "overrides": [] + }, + "gridPos": { + "h": 6, + "w": 12, + "x": 12, + "y": 42 + }, + "id": 59, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "none" + } + }, + "pluginVersion": "11.1.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "exemplar": true, + "expr": "(nvidia_smi_mig_tensor_activity_ratio{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}) * on(uuid, gpu_instance_id) group_left(profile) (max by (uuid, gpu_instance_id, profile) (label_replace(nvidia_smi_mig_info{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}, \"profile\", \"$1\", \"profile\", \"^[0-9]+c\\\\.(.*)$\")))", + "interval": "", + "legendFormat": "GI {{gpu_instance_id}} ({{profile}})", + "refId": "A" + } + ], + "title": "MIG Tensor Activity", + "type": "timeseries" + } + ], + "title": "XID / MIG / Power / PCIe (NVML mode only)", + "type": "row" + } + ], + "refresh": "30s", + "schemaVersion": 39, + "tags": [ + "gpu", + "project" + ], + "templating": { + "list": [ + { + "name": "project", + "label": "Project", + "type": "query", + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "definition": "label_values(nvidia_smi_index, project)", + "query": { + "qryType": 1, + "query": "label_values(nvidia_smi_index, project)", + "refId": "var" + }, + "refresh": 2, + "sort": 1, + "current": {}, + "options": [], + "hide": 0 + }, + { + "name": "env", + "label": "Environment", + "type": "query", + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "definition": "label_values(nvidia_smi_index{project=\"$project\"}, env)", + "query": { + "qryType": 1, + "query": "label_values(nvidia_smi_index{project=\"$project\"}, env)", + "refId": "var" + }, + "refresh": 2, + "sort": 1, + "current": {}, + "options": [], + "hide": 0 + }, + { + "name": "job", + "label": "Job", + "type": "query", + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "definition": "label_values(nvidia_smi_index{project=\"$project\", env=\"$env\"}, job)", + "query": { + "qryType": 1, + "query": "label_values(nvidia_smi_index{project=\"$project\", env=\"$env\"}, job)", + "refId": "var" + }, + "refresh": 2, + "sort": 1, + "current": {}, + "options": [], + "hide": 0 + }, + { + "name": "node", + "label": "Host", + "type": "query", + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "definition": "label_values(nvidia_smi_index{project=\"$project\", env=\"$env\", job=\"$job\"}, instance)", + "query": { + "qryType": 1, + "query": "label_values(nvidia_smi_index{project=\"$project\", env=\"$env\", job=\"$job\"}, instance)", + "refId": "var" + }, + "refresh": 2, + "sort": 1, + "current": {}, + "options": [], + "hide": 0 + }, + { + "current": {}, + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "definition": "query_result(label_join(label_replace(label_join(nvidia_smi_gpu_info{instance=\"$node\", job=\"$job\"}, \"ni\", \" #\", \"name\", \"index\"), \"us\", \"($1)\", \"uuid\", \"^(........).*\"), \"gpu_label\", \" \", \"ni\", \"us\"))", + "hide": 0, + "includeAll": false, + "label": "GPU", + "multi": false, + "name": "gpu", + "options": [], + "query": { + "query": "query_result(label_join(label_replace(label_join(nvidia_smi_gpu_info{instance=\"$node\", job=\"$job\"}, \"ni\", \" #\", \"name\", \"index\"), \"us\", \"($1)\", \"uuid\", \"^(........).*\"), \"gpu_label\", \" \", \"ni\", \"us\"))", + "refId": "StandardVariableQuery" + }, + "refresh": 1, + "regex": "/gpu_label=\"(?[^\"]+)|uuid=\"(?[^\"]+)/g", + "skipUrlSync": false, + "sort": 1, + "tagValuesQuery": "", + "tagsQuery": "", + "type": "query", + "useTags": false + } + ] + }, + "time": { + "from": "now-30m", + "to": "now" + }, + "timepicker": {}, + "timezone": "", + "title": "GPU", + "uid": "gpu", + "weekStart": "" +} diff --git a/dashboards/host-containers.json b/dashboards/host-containers.json new file mode 100644 index 0000000..988ca0b --- /dev/null +++ b/dashboards/host-containers.json @@ -0,0 +1,314 @@ +{ + "uid": "host-containers", + "title": "Host & Containers", + "tags": [ + "host", + "containers", + "project" + ], + "time": { + "from": "now-6h", + "to": "now" + }, + "refresh": "30s", + "schemaVersion": 39, + "editable": null, + "description": "Per-host and per-container resources for any onboarded project, from the Alloy agent's node-exporter and cAdvisor. Container series exist only where the agent runs with cgroup: host \u2014 without it cAdvisor reports the root cgroup alone and every container panel here is empty.", + "templating": { + "list": [ + { + "name": "project", + "label": "Project", + "type": "query", + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "definition": "label_values(node_uname_info, project)", + "query": { + "qryType": 1, + "query": "label_values(node_uname_info, project)", + "refId": "var" + }, + "refresh": 2, + "sort": 1, + "current": {}, + "options": [], + "hide": 0 + }, + { + "name": "env", + "label": "Environment", + "type": "query", + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "definition": "label_values(node_uname_info{project=\"$project\"}, env)", + "query": { + "qryType": 1, + "query": "label_values(node_uname_info{project=\"$project\"}, env)", + "refId": "var" + }, + "refresh": 2, + "sort": 1, + "current": {}, + "options": [], + "hide": 0 + }, + { + "name": "host", + "label": "Host", + "type": "query", + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "definition": "label_values(node_uname_info{project=\"$project\", env=\"$env\"}, host_name)", + "query": { + "qryType": 1, + "query": "label_values(node_uname_info{project=\"$project\", env=\"$env\"}, host_name)", + "refId": "var" + }, + "refresh": 2, + "sort": 1, + "current": {}, + "options": [], + "hide": 0 + } + ] + }, + "panels": [ + { + "type": "timeseries", + "title": "Host CPU", + "description": "All cores, aggregated.", + "gridPos": { + "h": 8, + "w": 8, + "x": 0, + "y": 0 + }, + "id": 1, + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "targets": [ + { + "expr": "100 * (1 - avg(rate(node_cpu_seconds_total{project=\"$project\", env=\"$env\", host_name=\"$host\", mode=\"idle\"}[$__rate_interval])))", + "legendFormat": "cpu", + "refId": "A" + } + ], + "fieldConfig": { + "defaults": { + "unit": "percent", + "custom": { + "lineWidth": 2, + "fillOpacity": 10 + } + }, + "overrides": [] + } + }, + { + "type": "timeseries", + "title": "Host Memory Used", + "description": "Available, not free \u2014 page cache is not pressure.", + "gridPos": { + "h": 8, + "w": 8, + "x": 8, + "y": 0 + }, + "id": 2, + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "targets": [ + { + "expr": "100 * (1 - node_memory_MemAvailable_bytes{project=\"$project\", env=\"$env\", host_name=\"$host\"} / node_memory_MemTotal_bytes{project=\"$project\", env=\"$env\", host_name=\"$host\"})", + "legendFormat": "memory", + "refId": "A" + } + ], + "fieldConfig": { + "defaults": { + "unit": "percent", + "custom": { + "lineWidth": 2, + "fillOpacity": 10 + } + }, + "overrides": [] + } + }, + { + "type": "timeseries", + "title": "Filesystem Used", + "description": "HostDiskSpaceLow fires at 80%.", + "gridPos": { + "h": 8, + "w": 8, + "x": 16, + "y": 0 + }, + "id": 3, + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "targets": [ + { + "expr": "100 * (1 - node_filesystem_avail_bytes{project=\"$project\", env=\"$env\", host_name=\"$host\", fstype!~\"tmpfs|ramfs|overlay\"} / node_filesystem_size_bytes{project=\"$project\", env=\"$env\", host_name=\"$host\", fstype!~\"tmpfs|ramfs|overlay\"})", + "legendFormat": "{{mountpoint}}", + "refId": "A" + } + ], + "fieldConfig": { + "defaults": { + "unit": "percent", + "custom": { + "lineWidth": 2, + "fillOpacity": 10 + } + }, + "overrides": [] + } + }, + { + "type": "timeseries", + "title": "Container CPU", + "description": "Per container, by name.", + "gridPos": { + "h": 8, + "w": 12, + "x": 0, + "y": 8 + }, + "id": 4, + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "targets": [ + { + "expr": "sum by (name) (rate(container_cpu_usage_seconds_total{project=\"$project\", env=\"$env\", host_name=\"$host\", name!=\"\"}[$__rate_interval]))", + "legendFormat": "{{name}}", + "refId": "A" + } + ], + "fieldConfig": { + "defaults": { + "unit": "percentunit", + "custom": { + "lineWidth": 2, + "fillOpacity": 10 + } + }, + "overrides": [] + } + }, + { + "type": "timeseries", + "title": "Container Memory", + "description": "Working set, which is what the OOM killer reads.", + "gridPos": { + "h": 8, + "w": 12, + "x": 12, + "y": 8 + }, + "id": 5, + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "targets": [ + { + "expr": "sum by (name) (container_memory_working_set_bytes{project=\"$project\", env=\"$env\", host_name=\"$host\", name!=\"\"})", + "legendFormat": "{{name}}", + "refId": "A" + } + ], + "fieldConfig": { + "defaults": { + "unit": "bytes", + "custom": { + "lineWidth": 2, + "fillOpacity": 10 + } + }, + "overrides": [] + } + }, + { + "type": "timeseries", + "title": "Container Restarts (1h)", + "description": "ContainerRestarting fires above 3. This is the 668-restart incident's detector, drawn.", + "gridPos": { + "h": 8, + "w": 12, + "x": 0, + "y": 16 + }, + "id": 6, + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "targets": [ + { + "expr": "changes(container_start_time_seconds{project=\"$project\", env=\"$env\", host_name=\"$host\", name!=\"\"}[1h])", + "legendFormat": "{{name}}", + "refId": "A" + } + ], + "fieldConfig": { + "defaults": { + "unit": "short", + "custom": { + "lineWidth": 2, + "fillOpacity": 10 + } + }, + "overrides": [] + } + }, + { + "type": "timeseries", + "title": "Host Network", + "description": "Receive and transmit, all non-loopback interfaces.", + "gridPos": { + "h": 8, + "w": 12, + "x": 12, + "y": 16 + }, + "id": 7, + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "targets": [ + { + "expr": "sum by (device) (rate(node_network_receive_bytes_total{project=\"$project\", env=\"$env\", host_name=\"$host\", device!=\"lo\"}[$__rate_interval]))", + "legendFormat": "rx {{device}}", + "refId": "A" + } + ], + "fieldConfig": { + "defaults": { + "unit": "Bps", + "custom": { + "lineWidth": 2, + "fillOpacity": 10 + } + }, + "overrides": [] + } + } + ] +} diff --git a/dashboards/infrastructure-logs.json b/dashboards/infrastructure-logs.json deleted file mode 100644 index 5b0c0e1..0000000 --- a/dashboards/infrastructure-logs.json +++ /dev/null @@ -1,98 +0,0 @@ -{ - "title": "RELab — Container Logs", - "uid": "relab-infra-logs", - "schemaVersion": 39, - "version": 1, - "refresh": "30s", - "time": { "from": "now-1h", "to": "now" }, - "graphTooltip": 1, - "tags": ["relab"], - "description": "Container logs via OTLP ingest. This stack's Loki index carries only the service_name label (everything else, service_instance_id included, is structured metadata); there is no env/service docker-driver label path exposed here, so filtering is by service_name only.", - "templating": { - "list": [ - { - "name": "service", - "label": "Service", - "type": "query", - "datasource": { "type": "loki", "uid": "loki" }, - "definition": "label_values(service_name)", - "query": { "type": "labelValues", "label": "service_name" }, - "refresh": 2, - "sort": 1, - "multi": true, - "includeAll": true, - "allValue": ".+", - "current": { "text": "All", "value": "$__all" }, - "options": [], - "hide": 0 - } - ] - }, - "panels": [ - { - "type": "timeseries", - "title": "Log Rate by Service", - "gridPos": { "h": 8, "w": 24, "x": 0, "y": 0 }, - "id": 1, - "datasource": { "type": "loki", "uid": "loki" }, - "targets": [ - { - "expr": "sum by (service_name) (rate({service_name=~\"$service\"}[$__auto]))", - "legendFormat": "{{service_name}}", - "refId": "A" - } - ], - "fieldConfig": { - "defaults": { - "unit": "reqps", - "custom": { "lineWidth": 2, "fillOpacity": 10 } - } - }, - "options": { - "tooltip": { "mode": "multi", "sort": "desc" }, - "legend": { "displayMode": "list", "placement": "bottom" } - } - }, - { - "type": "logs", - "title": "All Container Logs", - "gridPos": { "h": 12, "w": 24, "x": 0, "y": 8 }, - "id": 2, - "datasource": { "type": "loki", "uid": "loki" }, - "targets": [ - { - "expr": "{service_name=~\"$service\"}", - "refId": "A" - } - ], - "options": { - "dedupStrategy": "none", - "showLabels": true, - "wrapLogMessage": false, - "enableLogDetails": true, - "sortOrder": "Descending" - } - }, - { - "type": "logs", - "title": "Errors & Warnings", - "gridPos": { "h": 10, "w": 24, "x": 0, "y": 20 }, - "id": 3, - "datasource": { "type": "loki", "uid": "loki" }, - "targets": [ - { - "expr": "{service_name=~\"$service\"} |~ `(?i)(error|fatal|panic|exception|warn)`", - "refId": "A" - } - ], - "options": { - "dedupStrategy": "none", - "showLabels": true, - "wrapLogMessage": true, - "enableLogDetails": true, - "sortOrder": "Descending" - } - } - ], - "annotations": { "list": [] } -} diff --git a/dashboards/logs-overview.json b/dashboards/logs-overview.json deleted file mode 100644 index 90410e9..0000000 --- a/dashboards/logs-overview.json +++ /dev/null @@ -1,130 +0,0 @@ -{ - "title": "Logs Overview", - "uid": "logs-overview", - "schemaVersion": 39, - "version": 1, - "refresh": "30s", - "time": { "from": "now-3h", "to": "now" }, - "graphTooltip": 1, - "tags": ["logs", "otlp"], - "templating": { - "list": [ - { - "name": "service", - "label": "Service", - "type": "query", - "datasource": { "type": "loki", "uid": "loki" }, - "definition": "label_values(service_name)", - "query": { "type": "labelValues", "label": "service_name" }, - "refresh": 2, - "sort": 1, - "multi": true, - "includeAll": true, - "allValue": ".+", - "current": { "text": "All", "value": "$__all" }, - "options": [], - "hide": 0 - } - ] - }, - "panels": [ - { - "type": "timeseries", - "title": "Log Rate by Service", - "gridPos": { "h": 8, "w": 12, "x": 0, "y": 0 }, - "id": 1, - "datasource": { "type": "loki", "uid": "loki" }, - "targets": [ - { - "expr": "sum by (service_name) (rate({service_name=~\"$service\"}[$__auto]))", - "legendFormat": "{{service_name}}", - "refId": "A" - } - ], - "fieldConfig": { - "defaults": { - "unit": "logs/s", - "custom": { "lineWidth": 2, "fillOpacity": 10 } - }, - "overrides": [] - } - }, - { - "type": "timeseries", - "title": "Log Rate by Level", - "gridPos": { "h": 8, "w": 12, "x": 12, "y": 0 }, - "id": 2, - "datasource": { "type": "loki", "uid": "loki" }, - "targets": [ - { - "expr": "sum by (detected_level) (rate({service_name=~\"$service\"}[$__auto]))", - "legendFormat": "{{detected_level}}", - "refId": "A" - } - ], - "fieldConfig": { - "defaults": { - "unit": "logs/s", - "custom": { "lineWidth": 2, "fillOpacity": 10 } - }, - "overrides": [ - { - "matcher": { "id": "byRegexp", "options": "(?i)error" }, - "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "red" } }] - }, - { - "matcher": { "id": "byRegexp", "options": "(?i)(critical|fatal)" }, - "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "dark-red" } }] - }, - { - "matcher": { "id": "byRegexp", "options": "(?i)warn.*" }, - "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "orange" } }] - }, - { - "matcher": { "id": "byRegexp", "options": "(?i)info" }, - "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "green" } }] - } - ] - } - }, - { - "type": "logs", - "title": "Errors", - "description": "Error-and-worse log lines. Expand one and follow trace_id to the trace in Tempo.", - "gridPos": { "h": 9, "w": 24, "x": 0, "y": 8 }, - "id": 3, - "datasource": { "type": "loki", "uid": "loki" }, - "targets": [ - { - "expr": "{service_name=~\"$service\"} | detected_level=~\"(?i)(error|critical|fatal)\"", - "refId": "A" - } - ], - "options": { - "showTime": true, - "wrapLogMessage": true, - "enableLogDetails": true, - "sortOrder": "Descending" - } - }, - { - "type": "logs", - "title": "All Logs", - "gridPos": { "h": 10, "w": 24, "x": 0, "y": 17 }, - "id": 4, - "datasource": { "type": "loki", "uid": "loki" }, - "targets": [ - { - "expr": "{service_name=~\"$service\"}", - "refId": "A" - } - ], - "options": { - "showTime": true, - "wrapLogMessage": true, - "enableLogDetails": true, - "sortOrder": "Descending" - } - } - ] -} diff --git a/dashboards/logs.json b/dashboards/logs.json new file mode 100644 index 0000000..f8d2c4a --- /dev/null +++ b/dashboards/logs.json @@ -0,0 +1,195 @@ +{ + "uid": "logs", + "title": "Logs", + "tags": [ + "logs", + "project" + ], + "time": { + "from": "now-1h", + "to": "now" + }, + "refresh": "30s", + "schemaVersion": 39, + "editable": null, + "description": "Every project's container logs. Scoped by the project/env template variables, so a new project appears here the moment its first log line lands \u2014 nobody edits this JSON to onboard one.", + "templating": { + "list": [ + { + "name": "project", + "label": "Project", + "type": "query", + "datasource": { + "type": "loki", + "uid": "loki" + }, + "definition": "label_values({project=~\".+\"}, project)", + "query": "label_values({project=~\".+\"}, project)", + "refresh": 2, + "sort": 1, + "current": {}, + "options": [], + "hide": 0 + }, + { + "name": "env", + "label": "Environment", + "type": "query", + "datasource": { + "type": "loki", + "uid": "loki" + }, + "definition": "label_values({project=\"$project\"}, env)", + "query": "label_values({project=\"$project\"}, env)", + "refresh": 2, + "sort": 1, + "current": {}, + "options": [], + "hide": 0 + }, + { + "name": "service", + "label": "Service", + "type": "query", + "datasource": { + "type": "loki", + "uid": "loki" + }, + "definition": "label_values({project=\"$project\", env=\"$env\"}, service_name)", + "query": "label_values({project=\"$project\", env=\"$env\"}, service_name)", + "refresh": 2, + "sort": 1, + "current": { + "text": "All", + "value": "$__all" + }, + "options": [], + "hide": 0, + "includeAll": true, + "allValue": ".+" + } + ] + }, + "panels": [ + { + "type": "timeseries", + "title": "Log Rate by Service", + "description": "Which services are talking, and how much.", + "gridPos": { + "h": 8, + "w": 12, + "x": 0, + "y": 0 + }, + "id": 1, + "datasource": { + "type": "loki", + "uid": "loki" + }, + "targets": [ + { + "expr": "sum by (service_name) (rate({project=\"$project\", env=\"$env\", service_name=~\"$service\"}[$__auto]))", + "legendFormat": "{{service_name}}", + "refId": "A" + } + ], + "fieldConfig": { + "defaults": { + "unit": "logs/s", + "custom": { + "lineWidth": 2, + "fillOpacity": 10 + } + }, + "overrides": [] + } + }, + { + "type": "timeseries", + "title": "Log Rate by Level", + "description": "A rising error line is the cheapest early warning here.", + "gridPos": { + "h": 8, + "w": 12, + "x": 12, + "y": 0 + }, + "id": 2, + "datasource": { + "type": "loki", + "uid": "loki" + }, + "targets": [ + { + "expr": "sum by (detected_level) (rate({project=\"$project\", env=\"$env\", service_name=~\"$service\"}[$__auto]))", + "legendFormat": "{{detected_level}}", + "refId": "A" + } + ], + "fieldConfig": { + "defaults": { + "unit": "logs/s", + "custom": { + "lineWidth": 2, + "fillOpacity": 10 + } + }, + "overrides": [] + } + }, + { + "type": "logs", + "title": "Errors & Warnings", + "description": "Level is detected by Loki at ingest, not parsed here.", + "gridPos": { + "h": 10, + "w": 24, + "x": 0, + "y": 8 + }, + "id": 3, + "datasource": { + "type": "loki", + "uid": "loki" + }, + "targets": [ + { + "expr": "{project=\"$project\", env=\"$env\", service_name=~\"$service\"} | detected_level=~\"(?i)(error|critical|fatal|warn)\"", + "refId": "A" + } + ], + "options": { + "showTime": true, + "wrapLogMessage": true, + "sortOrder": "Descending" + } + }, + { + "type": "logs", + "title": "All Logs", + "description": "Everything, newest first.", + "gridPos": { + "h": 10, + "w": 24, + "x": 0, + "y": 18 + }, + "id": 4, + "datasource": { + "type": "loki", + "uid": "loki" + }, + "targets": [ + { + "expr": "{project=\"$project\", env=\"$env\", service_name=~\"$service\"}", + "refId": "A" + } + ], + "options": { + "showTime": true, + "wrapLogMessage": true, + "sortOrder": "Descending" + } + } + ] +} diff --git a/dashboards/service-health.json b/dashboards/service-health.json index d0d6c73..dfd3606 100644 --- a/dashboards/service-health.json +++ b/dashboards/service-health.json @@ -4,21 +4,74 @@ "schemaVersion": 39, "version": 1, "refresh": "30s", - "time": { "from": "now-1h", "to": "now" }, + "time": { + "from": "now-1h", + "to": "now" + }, "graphTooltip": 1, - "tags": ["red", "otlp"], + "tags": [ + "red", + "otlp" + ], "templating": { "list": [ + { + "name": "project", + "label": "Project", + "type": "query", + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "definition": "label_values(http_server_request_duration_seconds_count, project)", + "query": { + "qryType": 1, + "query": "label_values(http_server_request_duration_seconds_count, project)", + "refId": "var" + }, + "refresh": 2, + "sort": 1, + "current": {}, + "options": [], + "hide": 0 + }, + { + "name": "env", + "label": "Environment", + "type": "query", + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "definition": "label_values(http_server_request_duration_seconds_count{project=\"$project\"}, env)", + "query": { + "qryType": 1, + "query": "label_values(http_server_request_duration_seconds_count{project=\"$project\"}, env)", + "refId": "var" + }, + "refresh": 2, + "sort": 1, + "current": {}, + "options": [], + "hide": 0 + }, { "name": "service", "label": "Service", "type": "query", - "datasource": { "type": "prometheus", "uid": "prometheus" }, - "definition": "label_values(traces_spanmetrics_calls_total, service)", - "query": { "qryType": 1, "query": "label_values(traces_spanmetrics_calls_total, service)", "refId": "var" }, + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "definition": "label_values(http_server_request_duration_seconds_count{project=\"$project\", env=\"$env\"}, job)", + "query": { + "qryType": 1, + "query": "label_values(http_server_request_duration_seconds_count{project=\"$project\", env=\"$env\"}, job)", + "refId": "var" + }, "refresh": 2, "sort": 1, - "current": { "text": "demo-api", "value": "demo-api" }, + "current": {}, "options": [], "hide": 0 } @@ -27,22 +80,33 @@ "panels": [ { "type": "timeseries", - "title": "Request Rate by Endpoint", - "description": "Server spans per second, from Tempo span metrics — works for any service that sends traces.", - "gridPos": { "h": 8, "w": 12, "x": 0, "y": 0 }, + "title": "Request Rate by Route", + "description": "Requests per second per route, from the app's own OTLP metrics.", + "gridPos": { + "h": 8, + "w": 12, + "x": 0, + "y": 0 + }, "id": 1, - "datasource": { "type": "prometheus", "uid": "prometheus" }, + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, "targets": [ { - "expr": "sum by (span_name) (rate(traces_spanmetrics_calls_total{service=\"$service\", span_kind=\"SPAN_KIND_SERVER\"}[$__rate_interval]))", - "legendFormat": "{{span_name}}", + "expr": "sum by (http_route) (rate(http_server_request_duration_seconds_count{project=\"$project\", env=\"$env\", job=\"$service\"}[$__rate_interval]))", + "legendFormat": "{{http_route}}", "refId": "A" } ], "fieldConfig": { "defaults": { "unit": "reqps", - "custom": { "lineWidth": 2, "fillOpacity": 10 } + "custom": { + "lineWidth": 2, + "fillOpacity": 10 + } }, "overrides": [] } @@ -51,12 +115,20 @@ "type": "timeseries", "title": "Error Rate", "description": "Share of server spans with STATUS_CODE_ERROR.", - "gridPos": { "h": 8, "w": 12, "x": 12, "y": 0 }, + "gridPos": { + "h": 8, + "w": 12, + "x": 12, + "y": 0 + }, "id": 2, - "datasource": { "type": "prometheus", "uid": "prometheus" }, + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, "targets": [ { - "expr": "100 * sum(rate(traces_spanmetrics_calls_total{service=\"$service\", span_kind=\"SPAN_KIND_SERVER\", status_code=\"STATUS_CODE_ERROR\"}[$__rate_interval])) / sum(rate(traces_spanmetrics_calls_total{service=\"$service\", span_kind=\"SPAN_KIND_SERVER\"}[$__rate_interval]))", + "expr": "100 * sum(rate(http_server_request_duration_seconds_count{project=\"$project\", env=\"$env\", job=\"$service\", http_response_status_code=~\"5..\"}[$__rate_interval])) / sum(rate(http_server_request_duration_seconds_count{project=\"$project\", env=\"$env\", job=\"$service\"}[$__rate_interval]))", "legendFormat": "errors", "refId": "A" } @@ -65,35 +137,57 @@ "defaults": { "unit": "percent", "min": 0, - "color": { "mode": "fixed", "fixedColor": "red" }, - "custom": { "lineWidth": 2, "fillOpacity": 10 }, - "thresholds": { "mode": "absolute", "steps": [{ "color": "red", "value": null }] } + "color": { + "mode": "fixed", + "fixedColor": "red" + }, + "custom": { + "lineWidth": 2, + "fillOpacity": 10 + }, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "red", + "value": null + } + ] + } }, "overrides": [] } }, { "type": "timeseries", - "title": "Latency (from traces, with exemplars)", - "description": "Quantiles over Tempo span-metrics latency. Dots are exemplars — click one to open the exact trace in Tempo.", - "gridPos": { "h": 8, "w": 12, "x": 0, "y": 8 }, + "title": "Latency percentiles", + "description": "Quantiles over Tempo span-metrics latency. Dots are exemplars \u2014 click one to open the exact trace in Tempo.", + "gridPos": { + "h": 8, + "w": 12, + "x": 0, + "y": 8 + }, "id": 3, - "datasource": { "type": "prometheus", "uid": "prometheus" }, + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, "targets": [ { - "expr": "histogram_quantile(0.50, sum by (le) (rate(traces_spanmetrics_latency_bucket{service=\"$service\", span_kind=\"SPAN_KIND_SERVER\"}[$__rate_interval])))", + "expr": "histogram_quantile(0.5, sum by (le) (rate(http_server_request_duration_seconds_bucket{project=\"$project\", env=\"$env\", job=\"$service\"}[$__rate_interval])))", "legendFormat": "p50", "refId": "A", "exemplar": true }, { - "expr": "histogram_quantile(0.95, sum by (le) (rate(traces_spanmetrics_latency_bucket{service=\"$service\", span_kind=\"SPAN_KIND_SERVER\"}[$__rate_interval])))", + "expr": "histogram_quantile(0.95, sum by (le) (rate(http_server_request_duration_seconds_bucket{project=\"$project\", env=\"$env\", job=\"$service\"}[$__rate_interval])))", "legendFormat": "p95", "refId": "B", "exemplar": true }, { - "expr": "histogram_quantile(0.99, sum by (le) (rate(traces_spanmetrics_latency_bucket{service=\"$service\", span_kind=\"SPAN_KIND_SERVER\"}[$__rate_interval])))", + "expr": "histogram_quantile(0.99, sum by (le) (rate(http_server_request_duration_seconds_bucket{project=\"$project\", env=\"$env\", job=\"$service\"}[$__rate_interval])))", "legendFormat": "p99", "refId": "C", "exemplar": true @@ -102,21 +196,32 @@ "fieldConfig": { "defaults": { "unit": "s", - "custom": { "lineWidth": 2, "fillOpacity": 5 } + "custom": { + "lineWidth": 2, + "fillOpacity": 5 + } }, "overrides": [] } }, { "type": "timeseries", - "title": "HTTP Server Duration p95 (app SDK metrics)", - "description": "From the service's own OTLP metrics (OTel SDK), i.e. the collector → Prometheus OTLP path.", - "gridPos": { "h": 8, "w": 12, "x": 12, "y": 8 }, + "title": "Latency p95 by route", + "description": "From the service's own OTLP metrics (OTel SDK), i.e. the collector \u2192 Prometheus OTLP path.", + "gridPos": { + "h": 8, + "w": 12, + "x": 12, + "y": 8 + }, "id": 4, - "datasource": { "type": "prometheus", "uid": "prometheus" }, + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, "targets": [ { - "expr": "histogram_quantile(0.95, sum by (le, http_route) (rate(http_server_request_duration_seconds_bucket{job=\"$service\"}[$__rate_interval])))", + "expr": "histogram_quantile(0.95, sum by (le, http_route) (rate(http_server_request_duration_seconds_bucket{project=\"$project\", env=\"$env\", job=\"$service\"}[$__rate_interval])))", "legendFormat": "{{http_route}}", "refId": "A" } @@ -124,7 +229,10 @@ "fieldConfig": { "defaults": { "unit": "s", - "custom": { "lineWidth": 2, "fillOpacity": 10 } + "custom": { + "lineWidth": 2, + "fillOpacity": 10 + } }, "overrides": [] } @@ -133,12 +241,20 @@ "type": "logs", "title": "Logs", "description": "OTLP logs for the service. Expand a line and follow trace_id to jump to the trace in Tempo.", - "gridPos": { "h": 10, "w": 24, "x": 0, "y": 16 }, + "gridPos": { + "h": 10, + "w": 24, + "x": 0, + "y": 16 + }, "id": 5, - "datasource": { "type": "loki", "uid": "loki" }, + "datasource": { + "type": "loki", + "uid": "loki" + }, "targets": [ { - "expr": "{service_name=\"$service\"}", + "expr": "{project=\"$project\", env=\"$env\"}", "refId": "A" } ], @@ -149,5 +265,6 @@ "sortOrder": "Descending" } } - ] + ], + "description": "RED for any onboarded project, from the applications' own OTLP metrics (ADR 0002 moved this off Tempo's span-metrics generator). The project/env variables mean a new service appears here without anyone editing JSON." } diff --git a/dashboards/stack-health.json b/dashboards/stack-health.json index 422bfc0..362bc8f 100644 --- a/dashboards/stack-health.json +++ b/dashboards/stack-health.json @@ -4,17 +4,30 @@ "schemaVersion": 39, "version": 1, "refresh": "30s", - "time": { "from": "now-3h", "to": "now" }, + "time": { + "from": "now-3h", + "to": "now" + }, "graphTooltip": 1, - "tags": ["infra"], + "tags": [ + "infra" + ], "panels": [ { "type": "timeseries", "title": "Filesystem Used", - "description": "Loki and Tempo have no total-size cap — the HostDiskSpaceLow alert fires at 80%.", - "gridPos": { "h": 8, "w": 8, "x": 0, "y": 0 }, + "description": "Loki and Tempo have no total-size cap \u2014 the HostDiskSpaceLow alert fires at 80%.", + "gridPos": { + "h": 8, + "w": 8, + "x": 0, + "y": 0 + }, "id": 1, - "datasource": { "type": "prometheus", "uid": "prometheus" }, + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, "targets": [ { "expr": "100 * (1 - node_filesystem_avail_bytes{fstype!~\"tmpfs|ramfs|overlay\"} / node_filesystem_size_bytes{fstype!~\"tmpfs|ramfs|overlay\"})", @@ -27,8 +40,23 @@ "unit": "percent", "min": 0, "max": 100, - "custom": { "lineWidth": 2, "fillOpacity": 10 }, - "thresholds": { "mode": "absolute", "steps": [{ "color": "green", "value": null }, { "color": "red", "value": 80 }] } + "custom": { + "lineWidth": 2, + "fillOpacity": 10 + }, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 80 + } + ] + } }, "overrides": [] } @@ -36,9 +64,17 @@ { "type": "timeseries", "title": "Host CPU", - "gridPos": { "h": 8, "w": 8, "x": 8, "y": 0 }, + "gridPos": { + "h": 8, + "w": 8, + "x": 8, + "y": 0 + }, "id": 2, - "datasource": { "type": "prometheus", "uid": "prometheus" }, + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, "targets": [ { "expr": "100 * (1 - avg(rate(node_cpu_seconds_total{mode=\"idle\"}[$__rate_interval])))", @@ -51,7 +87,10 @@ "unit": "percent", "min": 0, "max": 100, - "custom": { "lineWidth": 2, "fillOpacity": 10 } + "custom": { + "lineWidth": 2, + "fillOpacity": 10 + } }, "overrides": [] } @@ -59,9 +98,17 @@ { "type": "timeseries", "title": "Host Memory Used", - "gridPos": { "h": 8, "w": 8, "x": 16, "y": 0 }, + "gridPos": { + "h": 8, + "w": 8, + "x": 16, + "y": 0 + }, "id": 3, - "datasource": { "type": "prometheus", "uid": "prometheus" }, + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, "targets": [ { "expr": "100 * (1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)", @@ -74,7 +121,10 @@ "unit": "percent", "min": 0, "max": 100, - "custom": { "lineWidth": 2, "fillOpacity": 10 } + "custom": { + "lineWidth": 2, + "fillOpacity": 10 + } }, "overrides": [] } @@ -83,9 +133,17 @@ "type": "timeseries", "title": "Collector Ingest Rate", "description": "Items accepted by the OTLP receivers, per signal.", - "gridPos": { "h": 8, "w": 12, "x": 0, "y": 8 }, + "gridPos": { + "h": 8, + "w": 12, + "x": 0, + "y": 8 + }, "id": 4, - "datasource": { "type": "prometheus", "uid": "prometheus" }, + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, "targets": [ { "expr": "sum(rate(otelcol_receiver_accepted_spans[$__rate_interval]))", @@ -106,7 +164,10 @@ "fieldConfig": { "defaults": { "unit": "ops", - "custom": { "lineWidth": 2, "fillOpacity": 10 } + "custom": { + "lineWidth": 2, + "fillOpacity": 10 + } }, "overrides": [] } @@ -114,10 +175,18 @@ { "type": "timeseries", "title": "Collector Export Failures", - "description": "Anything above zero means a backend is rejecting or unreachable — the OtelExportFailures alert covers this.", - "gridPos": { "h": 8, "w": 12, "x": 12, "y": 8 }, + "description": "Anything above zero means a backend is rejecting or unreachable \u2014 the OtelExportFailures alert covers this.", + "gridPos": { + "h": 8, + "w": 12, + "x": 12, + "y": 8 + }, "id": 5, - "datasource": { "type": "prometheus", "uid": "prometheus" }, + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, "targets": [ { "expr": "sum by (exporter) (rate(otelcol_exporter_send_failed_spans[$__rate_interval])) or sum by (exporter) (rate(otelcol_exporter_send_failed_metric_points[$__rate_interval])) or sum by (exporter) (rate(otelcol_exporter_send_failed_log_records[$__rate_interval]))", @@ -129,47 +198,42 @@ "defaults": { "unit": "ops", "min": 0, - "color": { "mode": "fixed", "fixedColor": "red" }, - "custom": { "lineWidth": 2, "fillOpacity": 10 } + "color": { + "mode": "fixed", + "fixedColor": "red" + }, + "custom": { + "lineWidth": 2, + "fillOpacity": 10 + } }, "overrides": [] } }, { - "type": "table", + "type": "alertlist", "title": "Active Alerts", - "description": "Pending and firing alerts (Watchdog excluded — it fires by design).", - "gridPos": { "h": 6, "w": 24, "x": 0, "y": 16 }, "id": 6, - "datasource": { "type": "prometheus", "uid": "prometheus" }, - "targets": [ - { - "expr": "ALERTS{alertname!=\"Watchdog\"}", - "instant": true, - "format": "table", - "refId": "A" + "description": "Grafana-managed alerts. The old panel queried Prometheus's ALERTS series, which only exists while Prometheus evaluates rules \u2014 it stopped existing when alerting moved to Grafana (ADR 0002).", + "gridPos": { + "h": 6, + "w": 24, + "x": 0, + "y": 16 + }, + "options": { + "viewMode": "list", + "groupMode": "default", + "maxItems": 20, + "sortOrder": 1, + "alertInstanceLabelFilter": "{alertname!=\"Watchdog\"}", + "stateFilter": { + "firing": true, + "pending": true, + "noData": false, + "normal": false, + "error": true } - ], - "transformations": [ - { - "id": "organize", - "options": { - "excludeByName": { "Time": true, "Value": true, "__name__": true }, - "indexByName": { "alertname": 0, "alertstate": 1, "severity": 2 } - } - } - ], - "fieldConfig": { - "defaults": {}, - "overrides": [ - { - "matcher": { "id": "byName", "options": "alertstate" }, - "properties": [ - { "id": "custom.cellOptions", "value": { "type": "color-text" } }, - { "id": "mappings", "value": [{ "type": "value", "options": { "firing": { "color": "red" }, "pending": { "color": "orange" } } }] } - ] - } - ] } } ] From 625bd3a9f5581d10b5c34c4a44febc84191ab2a3 Mon Sep 17 00:00:00 2001 From: Simon van Lierde Date: Tue, 1 Sep 2026 12:32:23 +0000 Subject: [PATCH 20/85] feat(onboarding): add bootstrap.sh and the vendored agent templates - one Alloy agent per host: container logs, host metrics, cAdvisor, GPU - bootstrap renders the keystone rule, regenerates the coverage backstop, creates healthchecks.io checks and prints the .env and vendoring curls - replace the ONBOARDING templates that pushed straight to Loki, which needed Loki exposed without authentication --- .env.example | 6 + bootstrap.sh | 187 ++++++++++ config/grafana/alerting/coverage.yaml | 50 +++ .../alerting/project-relab-staging.yaml | 57 +++ docs/ONBOARDING.md | 68 +--- templates/README.md | 131 +++++++ templates/alerting/project.yaml.tmpl | 57 +++ templates/alloy/config.alloy | 351 ++++++++++++++++++ templates/compose.telemetry.gpu.yml | 58 +++ templates/compose.telemetry.yml | 159 ++++++++ templates/run_scheduled.sh | 103 +++++ 11 files changed, 1171 insertions(+), 56 deletions(-) create mode 100755 bootstrap.sh create mode 100644 config/grafana/alerting/coverage.yaml create mode 100644 config/grafana/alerting/project-relab-staging.yaml create mode 100644 templates/README.md create mode 100644 templates/alerting/project.yaml.tmpl create mode 100644 templates/alloy/config.alloy create mode 100644 templates/compose.telemetry.gpu.yml create mode 100644 templates/compose.telemetry.yml create mode 100755 templates/run_scheduled.sh diff --git a/.env.example b/.env.example index f2d1f9a..518a51b 100644 --- a/.env.example +++ b/.env.example @@ -25,6 +25,12 @@ ALERT_WEBHOOK_URL= # alert posts here every 5m; alert externally when pings stop. HEARTBEAT_URL= +# Optional. Lets `./bootstrap.sh ` create a project's healthchecks.io +# checks and print their ping URLs instead of telling you to create them by hand. +# Must be the project's READ-WRITE API key (healthchecks.io -> Project Settings -> API +# keys): the read-only key cannot create checks, and a ping key only sends pings. +HEALTHCHECKS_API_KEY= + # Only needed for `just up-tunnel` (production exposure via Cloudflare Tunnel). # From: Cloudflare Zero Trust → Networks → Tunnels → → Configure → token CLOUDFLARE_TUNNEL_TOKEN= diff --git a/bootstrap.sh b/bootstrap.sh new file mode 100755 index 0000000..5ca0805 --- /dev/null +++ b/bootstrap.sh @@ -0,0 +1,187 @@ +#!/usr/bin/env bash +# Onboard one project/environment onto this monitoring stack. +# +# ./bootstrap.sh +# +# It does the four things a human would otherwise get subtly wrong: +# +# 1. renders the keystone ProjectTelemetrySilent rule and reloads Grafana; +# 2. regenerates the coverage rule, so a project that ships telemetry without ever +# being bootstrapped is itself alerted on; +# 3. creates the project's healthchecks.io checks (if an API key is present) and +# prints their ping URLs; +# 4. prints the `.env` block to paste on the project host, and the curl that vendors +# the templates at a pinned tag. +# +# Bootstrap is what creates the safety net, NOT the telemetry. A host can ship perfectly +# good telemetry and still be unmonitored, because the rule that notices its silence +# lives here. That asymmetry is the whole reason this script exists. +# +# Idempotent: re-running it re-renders the same files and reloads again. +set -euo pipefail + +project="${1:-}" +env_name="${2:-}" +if [[ -z "$project" || -z "$env_name" ]]; then + echo "usage: $0 " >&2 + exit 2 +fi +# These become Prometheus label values, a Grafana rule uid, and a filename. Keep them +# boring: a label value with a quote or a brace in it produces a rule that silently +# never matches, which is the failure mode this whole stack exists to avoid. +if [[ ! "$project" =~ ^[a-z0-9][a-z0-9-]*$ || ! "$env_name" =~ ^[a-z0-9][a-z0-9-]*$ ]]; then + echo "error: project and env must match [a-z0-9][a-z0-9-]* (lowercase, no spaces)" >&2 + exit 2 +fi + +root="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)" +cd "$root" + +# Read HEALTHCHECKS_API_KEY from .env if the environment does not already carry it. +# `just` recipes get .env via dotenv-load; a bare ./bootstrap.sh does not, and silently +# skipping check creation because of that is a confusing way to find out. Only this one +# key is read — sourcing the whole file would drag the stack's secrets into scope. +# +# It must be the project's READ-WRITE API key: the read-only key cannot POST, and a ping +# key only sends pings. +if [[ -z "${HEALTHCHECKS_API_KEY:-}" && -f .env ]]; then + HEALTHCHECKS_API_KEY="$(sed -n 's/^HEALTHCHECKS_API_KEY=//p' .env | tail -1)" + export HEALTHCHECKS_API_KEY +fi +# Flat, not a subdirectory: Grafana's alerting provisioner does not recurse, and it +# skips a directory with a warning rather than an error — so a nested layout looks like +# it worked and provisions nothing. The project- prefix keeps them grouped in `ls`. +out_dir="config/grafana/alerting" + +# Pinned tag for the vendoring curl. A moving ref would let two projects vendor two +# different agent configs and call it the same template. +tag="$(git -C "$root" describe --tags --abbrev=0 2>/dev/null || echo main)" +repo_raw="https://raw.githubusercontent.com/CMLPlatform/monitoring/${tag}/templates" + +# ---------------------------------------------------------------- 1. the keystone rule +rendered="${out_dir}/project-${project}-${env_name}.yaml" +sed -e "s/__PROJECT__/${project}/g" \ + -e "s/__ENV__/${env_name}/g" \ + -e "s/__UID__/proj-silent-${project}-${env_name}/g" \ + templates/alerting/project.yaml.tmpl > "$rendered" +echo "rendered $rendered" + +# ------------------------------------------------------------ 2. the coverage backstop +# Every project with a rendered rule file, derived from the directory rather than a list +# someone has to remember to update. +covered="$(find "$out_dir" -maxdepth 1 -name 'project-*.yaml' -exec basename {} .yaml \; \ + | sed -e 's/^project-//' -e 's/-[^-]*$//' | sort -u | paste -sd'|' -)" +: "${covered:=__none__}" +cat > "${out_dir}/coverage.yaml" < on the monitoring host." +COVERAGE +echo "rendered ${out_dir}/coverage.yaml (covering: ${covered})" + +# ------------------------------------------------------------------ 3. reload Grafana +if docker compose ps --status running --format '{{.Service}}' 2>/dev/null | grep -qx grafana; then + # Provisioned files are re-read on SIGHUP; no restart, no dropped evaluations. + # A restart, not SIGHUP: Grafana re-reads alert provisioning only at startup, and + # a SIGHUP reports success while changing nothing. + docker compose up -d --force-recreate grafana >/dev/null 2>&1 \ + && echo "reloaded grafana (restarted)" \ + || echo "WARNING: could not restart grafana; run 'docker compose up -d --force-recreate grafana'" >&2 +else + echo "note grafana is not running; the rules apply next time it starts" +fi + +# ------------------------------------------------------- 4. healthchecks.io + printout +hc_note="create these by hand at https://healthchecks.io and note their ping URLs" +if [[ -n "${HEALTHCHECKS_API_KEY:-}" ]]; then + hc_note="created via API" + for job in backup watchdog restore-check; do + curl -fsS -X POST https://healthchecks.io/api/v3/checks/ \ + -H "X-Api-Key: ${HEALTHCHECKS_API_KEY}" \ + -H "Content-Type: application/json" \ + -d "{\"name\":\"${project}-${env_name}-${job}\",\"slug\":\"${project}-${env_name}-${job}\",\"unique\":[\"name\"],\"timeout\":93600,\"grace\":3600,\"channels\":\"*\"}" \ + | sed -n 's/.*"ping_url": *"\([^"]*\)".*/ PING_'"$(echo "$job" | tr 'a-z-' 'A-Z_')"'=\1/p' + done +else + echo "note HEALTHCHECKS_API_KEY unset; ${hc_note}" +fi + +cat < +OTLP_AUTH_TOKEN= +TELEMETRY_EDGE_KEY= # only if the project's egress crosses a WAF +GPU_METRICS= # 1 on hosts with an NVIDIA card + +──────────────────────────────────────────────────────────────────────────── +Vendor the templates on the project host (pinned at ${tag}) +──────────────────────────────────────────────────────────────────────────── +mkdir -p deploy/alloy scripts +curl -fsSL -o deploy/alloy/config.alloy ${repo_raw}/alloy/config.alloy +curl -fsSL -o compose.telemetry.yml ${repo_raw}/compose.telemetry.yml +curl -fsSL -o compose.telemetry.gpu.yml ${repo_raw}/compose.telemetry.gpu.yml +curl -fsSL -o scripts/run_scheduled.sh ${repo_raw}/run_scheduled.sh +chmod +x scripts/run_scheduled.sh + +Then include the overlay and bring it up: + docker compose -f compose.yaml -f compose.telemetry.yml up -d + +The agent container needs 'cgroup: host' (the overlay sets it). Without it cAdvisor +reports the root cgroup only, and every container alert here matches nothing. + +Verify from the monitoring host, within ~2 minutes: + count({project="${project}",env="${env_name}"}) -> non-zero + count(container_start_time_seconds{project="${project}",name!=""}) -> one per container +SUMMARY diff --git a/config/grafana/alerting/coverage.yaml b/config/grafana/alerting/coverage.yaml new file mode 100644 index 0000000..6716e2e --- /dev/null +++ b/config/grafana/alerting/coverage.yaml @@ -0,0 +1,50 @@ +# GENERATED by bootstrap.sh — do not edit. Regenerated on every run. +# +# The trap this closes: ProjectTelemetrySilent only exists for projects bootstrap.sh was +# actually run for. A project that ships telemetry but skipped bootstrap is silently +# uncovered — the exact failure class the keystone closes, one level up. This fires on +# any series carrying a project label that has no rendered rule file. +# +# Covered right now: relab + +apiVersion: 1 + +groups: + - orgId: 1 + name: _coverage + folder: Stack alerts + interval: 5m + rules: + - uid: projects-uncovered + title: ProjectsUncovered + condition: FIRING + for: 15m + noDataState: OK + execErrState: Error + isPaused: false + data: + - refId: QUERY + datasourceUid: prometheus + relativeTimeRange: + from: 3600 + to: 0 + model: + refId: QUERY + instant: true + editorMode: code + expr: count by (project) ({project!~"relab", project!=""}) + - refId: FIRING + datasourceUid: __expr__ + model: + refId: FIRING + type: threshold + expression: QUERY + conditions: + - evaluator: + type: gt + params: [0] + labels: + severity: warning + annotations: + summary: "Project {{ $labels.project }} is sending telemetry but has no alert rules" + description: "Telemetry is arriving for a project bootstrap.sh was never run for, so nothing would notice if it went silent. Run ./bootstrap.sh {{ $labels.project }} on the monitoring host." diff --git a/config/grafana/alerting/project-relab-staging.yaml b/config/grafana/alerting/project-relab-staging.yaml new file mode 100644 index 0000000..6c0d384 --- /dev/null +++ b/config/grafana/alerting/project-relab-staging.yaml @@ -0,0 +1,57 @@ +# Rendered by bootstrap.sh into config/grafana/alerting/projects/. Do not edit the +# rendered files by hand — re-run bootstrap.sh, or edit this template so every project +# gets the fix. +# +# This is the keystone: nothing on a spoke can detect its own absence. A project that +# ships telemetry but never had bootstrap.sh run is silently uncovered — that is what +# the ProjectsUncovered rule in _coverage.yaml exists to catch, one level up. +# +# Placeholders: relab staging proj-silent-relab-staging + +apiVersion: 1 + +groups: + - orgId: 1 + name: project-relab-staging + folder: Stack alerts + interval: 1m + rules: + # A label-only selector, deliberately not target_info: agents attach the identity + # labels to the series themselves, so target_info carries none of them. absent() + # yields nothing while telemetry flows, so NoData is the HEALTHY state and must + # map to OK or the rule reads inverted. + - uid: proj-silent-relab-staging + title: ProjectTelemetrySilent + condition: FIRING + for: 15m + noDataState: OK + execErrState: Error + isPaused: false + data: + - refId: QUERY + datasourceUid: prometheus + relativeTimeRange: + from: 3600 + to: 0 + model: + refId: QUERY + instant: true + editorMode: code + expr: absent({project="relab",env="staging"}) + - refId: FIRING + datasourceUid: __expr__ + model: + refId: FIRING + type: threshold + expression: QUERY + conditions: + - evaluator: + type: gt + params: [0] + labels: + severity: critical + project: relab + env: staging + annotations: + summary: "No telemetry from relab/staging in 15m" + description: "Host down, Docker down, the agent down, tunnel down, token rotated wrong, or the collector is rejecting this project's data. Nothing on the relab side can detect this on its own. Expect this ~20m after the last sample: absent() needs the series to go stale first." diff --git a/docs/ONBOARDING.md b/docs/ONBOARDING.md index 83f25ef..78dcd8c 100644 --- a/docs/ONBOARDING.md +++ b/docs/ONBOARDING.md @@ -75,62 +75,18 @@ OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer " OTEL_RESOURCE_ATTRIBUTES=env=prod ``` -## Template 3 — Docker container logs (Loki driver) +## Container logs, host metrics, per-container metrics -This ships container stdout/stderr without touching the app. It needs a -Loki push URL, which **this stack does not expose by default**: Loki has -no authentication of its own, so a push hostname must first be added to -`infra/main.tf` and protected with a Cloudflare Access service token, or -reached over a private network path (VPN/WireGuard). If in doubt, use the -OTLP log path from Templates 1–2 instead. +Everything the application cannot report about itself — other containers' stdout, host +resources, container lifecycle — is shipped by one Grafana Alloy agent per host. It is +vendored from `templates/`, not written per project, and it rides the same OTLP endpoint +and token as Templates 1 and 2: no second hostname, no second credential. -```sh -# once per host -docker plugin install grafana/loki-docker-driver:latest --alias loki --grant-all-permissions -``` - -```yaml -# per service, in compose.yml -logging: - driver: loki - options: - loki-url: ${LOKI_URL} # e.g. https://logs.example.org/loki/api/v1/push - loki-external-labels: service={{.Name}},env=prod,host=myhost -``` - -## Template 4 — host or file logs (Grafana Alloy) - -For log files that live outside containers. (Promtail is end-of-life; -Alloy is its successor.) - -```alloy -// alloy/config.alloy -local.file_match "app" { - path_targets = [{ __path__ = "/var/log/myapp/*.log", service = "myapp", env = "prod", host = "myhost" }] -} - -loki.source.file "app" { - targets = local.file_match.app.targets - forward_to = [loki.write.central.receiver] -} - -loki.write "central" { - endpoint { - url = "https://logs.example.org/loki/api/v1/push" - } -} -``` - -```yaml -# compose service -alloy: - image: grafana/alloy:v1.13.0 - restart: unless-stopped - command: [ "run", "/etc/alloy/config.alloy" ] - volumes: - - ./alloy/config.alloy:/etc/alloy/config.alloy:ro - - /var/log/myapp:/var/log/myapp:ro -``` +Run `./bootstrap.sh ` on the monitoring host and follow what it prints. +See [templates/README.md](../templates/README.md). -The same caveat as Template 3 applies: the Loki push URL needs a protected -network path. +> Earlier revisions of this document carried two templates that pushed straight to Loki +> (the Docker `loki` log driver, and Alloy's `loki.write`). Both required exposing Loki, +> which this stack deliberately does not do because Loki has no authentication of its +> own, and undoing it later is more work than not starting. If you find those +> instructions in an old copy, they are wrong — use the agent above. diff --git a/templates/README.md b/templates/README.md new file mode 100644 index 0000000..0616cb3 --- /dev/null +++ b/templates/README.md @@ -0,0 +1,131 @@ +# Templates: onboarding a project onto this stack + +Four files get vendored onto a project host at a pinned tag. **No project edits them.** +Everything that differs between projects arrives as an environment variable, so the +agent config on every host is byte-identical and a fix here reaches all of them. + +| File | What it is | +| --- | --- | +| `alloy/config.alloy` | The one agent config: container logs, host metrics, cAdvisor, optional GPU | +| `compose.telemetry.yml` | The Alloy agent and its Docker socket proxy | +| `compose.telemetry.gpu.yml` | Opt-in overlay: `nvidia_gpu_exporter`, discovered automatically | +| `run_scheduled.sh` | Dead-man's-switch wrapper for scheduled jobs | + +`alerting/project.yaml.tmpl` is not vendored — it stays here and is rendered by +`bootstrap.sh` into this stack's own alert rules. + +## The whole checklist + +Run `./bootstrap.sh ` on the monitoring host. It prints the two blocks +you need: the `.env` variables to paste on the project host, and the `curl` commands +that vendor these files at a pinned tag. Then on the project host: + +```sh +docker compose -f compose.yaml -f compose.telemetry.yml up -d +``` + +Under an hour, most of it waiting for the first scrape. + +## Bootstrap creates the safety net, not the telemetry + +A host can ship perfect telemetry and still be unmonitored, because the rule that +notices its *silence* lives on this stack, not on the host. That asymmetry is why +`bootstrap.sh` exists and why skipping it is dangerous rather than merely untidy. + +The backstop for skipping it anyway is `ProjectsUncovered`, regenerated on every +bootstrap run: it fires on any series carrying a `project` label with no rendered rule +file. It is the keystone applied one level up — the detector for a missing detector. + +## Two things that silently produce nothing + +- **`cgroup: host` on the agent container.** The overlay sets it. cAdvisor finds + containers by walking the cgroup tree, not through the Docker API, so under Docker's + default private cgroup namespace it sees only its own cgroup, reports one root series + with no `name` label, and every container alert matches nothing. No error is logged. +- **`OTEL_SEMCONV_STABILITY_OPT_IN=http` in an instrumented app.** Without it the SDK + emits the legacy HTTP metric names, whose `http_target` label carries the raw request + path — unbounded series on any API with path parameters. The stable names use + `http_route`, and the central Service Health dashboard queries those. + +## GPU hosts + +Include `compose.telemetry.gpu.yml` as well and set `GPU_METRICS=1`. The agent config +already discovers the exporter by its Compose service label, so nothing else changes: +a GPU host is an ordinary host plus one overlay. + +`nvidia_gpu_exporter` rather than dcgm-exporter on purpose — DCGM's advantage is its +`DCGM_FI_PROF_*` profiling fields, which NVIDIA document as datacentre-only. On a +consumer card they are simply absent, which removes the reason to prefer DCGM while +keeping its `SYS_ADMIN` requirement. + +Three GPU rules are worth adding per GPU host. They are not provisioned by +`bootstrap.sh` yet: + +- `nvidia_smi_gpu_recovery_action > 0` — the driver is asking for a reset. The single + best GPU health signal. +- a thermal/power throttle flag. +- XID faults, with an explicit code allowlist rather than every code, since most XIDs + are application faults: + + ```promql + time() - nvidia_smi_xid_last_timestamp_seconds{xid=~"48|62|64|74|79|95|119|120"} < 300 + ``` + +XIDs are the GPU analogue of the crash-loop blind spot: a stuck kernel, an +uncorrectable memory fault, or a card that has fallen off the bus are all invisible to +utilisation graphs, and they are what silently kills a twelve-hour training run. + +For dashboards, import [14574](https://grafana.com/grafana/dashboards/14574) (revised +2026-08-04) and its multi-GPU companion 25547. Do **not** use the canonical DCGM +dashboard 12239: last revised 2021, and its panels lean on profiling fields consumer +cards cannot produce. For per-container resources, +[15798](https://grafana.com/grafana/dashboards/15798-docker-monitoring/) (revised +2025-07-12), with 19792 as the Compose-aware second choice. Avoid 10619 and 893 — +high download counts, untouched since 2019 and Grafana 4 respectively. Download counts +measure inertia, not maintenance. + +## Removing a project + +Deleting `config/grafana/alerting/project--.yaml` is **not** enough. +Grafana provisioning creates and updates rules from files but never deletes a rule +because its file vanished, and the API refuses to delete a provisioned rule (409, even +with `X-Disable-Provenance`). The orphan keeps evaluating and firing. + +Delete the file, then drop the rule explicitly with a one-off provisioning file: + +```yaml +# config/grafana/alerting/zz-delete.yaml — temporary +apiVersion: 1 +deleteRules: + - orgId: 1 + uid: proj-silent-- +``` + +Restart Grafana, confirm the group is gone, then remove `zz-delete.yaml` and restart +again. It has to be temporary: left in place it would delete the rule again the next +time `bootstrap.sh` renders it for that project. + +Finally re-run `bootstrap.sh` for a project that remains, so `coverage.yaml` stops +listing the removed one as covered. + +## Budgets to plan against + +- **healthchecks.io free tier is exactly 20 checks.** `bootstrap.sh` creates three per + project/environment, so that is the onboarding ceiling — about six environments. +- **Prometheus's OTLP receiver is documented as "not an efficient way of ingesting + samples"**, for "specific low-volume use cases". A few hundred series per host at 30s + is exactly the case that sentence carves out. The trigger for revisiting is volume: a + host past a few thousand active series should move infrastructure metrics to + `remote_write` behind an authenticating proxy, leaving app metrics on OTLP. + +## Verifying, from the monitoring host + +```promql +count({project="",env=""}) # non-zero within ~2 min +count(container_start_time_seconds{project="",name!=""}) # one per container +count({job=""}) # app's own SDK metrics +``` + +If the first is zero after five minutes, `ProjectTelemetrySilent` will tell you anyway +— that is the point of it. Expect it ~20 minutes after the last sample: `absent()` +needs the series to go stale before it reports. diff --git a/templates/alerting/project.yaml.tmpl b/templates/alerting/project.yaml.tmpl new file mode 100644 index 0000000..52cbf4a --- /dev/null +++ b/templates/alerting/project.yaml.tmpl @@ -0,0 +1,57 @@ +# Rendered by bootstrap.sh into config/grafana/alerting/projects/. Do not edit the +# rendered files by hand — re-run bootstrap.sh, or edit this template so every project +# gets the fix. +# +# This is the keystone: nothing on a spoke can detect its own absence. A project that +# ships telemetry but never had bootstrap.sh run is silently uncovered — that is what +# the ProjectsUncovered rule in _coverage.yaml exists to catch, one level up. +# +# Placeholders: __PROJECT__ __ENV__ __UID__ + +apiVersion: 1 + +groups: + - orgId: 1 + name: project-__PROJECT__-__ENV__ + folder: Stack alerts + interval: 1m + rules: + # A label-only selector, deliberately not target_info: agents attach the identity + # labels to the series themselves, so target_info carries none of them. absent() + # yields nothing while telemetry flows, so NoData is the HEALTHY state and must + # map to OK or the rule reads inverted. + - uid: __UID__ + title: ProjectTelemetrySilent + condition: FIRING + for: 15m + noDataState: OK + execErrState: Error + isPaused: false + data: + - refId: QUERY + datasourceUid: prometheus + relativeTimeRange: + from: 3600 + to: 0 + model: + refId: QUERY + instant: true + editorMode: code + expr: absent({project="__PROJECT__",env="__ENV__"}) + - refId: FIRING + datasourceUid: __expr__ + model: + refId: FIRING + type: threshold + expression: QUERY + conditions: + - evaluator: + type: gt + params: [0] + labels: + severity: critical + project: __PROJECT__ + env: __ENV__ + annotations: + summary: "No telemetry from __PROJECT__/__ENV__ in 15m" + description: "Host down, Docker down, the agent down, tunnel down, token rotated wrong, or the collector is rejecting this project's data. Nothing on the __PROJECT__ side can detect this on its own. Expect this ~20m after the last sample: absent() needs the series to go stale first." diff --git a/templates/alloy/config.alloy b/templates/alloy/config.alloy new file mode 100644 index 0000000..1dede27 --- /dev/null +++ b/templates/alloy/config.alloy @@ -0,0 +1,351 @@ +// SHARED AGENT CONFIG — vendored from the central monitoring repo at a pinned tag. +// Do not edit it per project: every deployment-specific value arrives as an environment +// variable (PROJECT, ENVIRONMENT, COMPOSE_PROJECT_NAME, OTEL_EXPORTER_OTLP_ENDPOINT, +// OTLP_AUTH_TOKEN, TELEMETRY_EDGE_KEY). If you find yourself editing this file to +// onboard a project, that is a bug in the template — report it upstream instead, so the +// next project does not hit the same wall. +// +// Ship this host's telemetry to the department OpenTelemetry collector: container +// stdout/stderr as logs, and host resource metrics. +// +// An instrumented application sends its own traces and metrics over OTLP directly. +// Everything else in the stack — databases, caches, proxies, frontends, batch jobs — +// only writes to stdout, so this agent picks those up and forwards them on the +// same OTLP endpoint with the same token. Nothing new is published at the edge: the +// monitoring stack exposes `otlp.` and deliberately does not expose Loki, which has no +// authentication of its own. + +// The host's own name, read from its /etc/hostname through the rootfs mount rather than +// taken as an operator input: one less thing to set per host, and one less thing to get +// wrong. The container's own hostname is a docker id and useless here. +local.file "hostname" { + filename = "/rootfs/etc/hostname" +} + +// Discovery and log reading both talk to the Docker API — through the socket proxy +// (compose.telemetry.yml), never the socket itself, so this container cannot +// mutate the daemon. The API is what makes container NAMES available; reading the +// on-disk json logs instead would label everything by container id. +discovery.docker "containers" { + host = "tcp://docker-socket-proxy:2375" + + // The default is 1m, and the backup one-shot runs for 60-90s: at the default a whole + // backup can start and exit inside one poll window and never be discovered, which + // would lose exactly the logs most worth having. Polling the local socket for a + // dozen containers is cheap; a missed short-lived container is not. + refresh_interval = "5s" +} + +discovery.relabel "containers" { + targets = discovery.docker.containers.targets + + // Only this stack. The host may run other Compose projects (monitoring's own, for + // one) that ship their own telemetry; picking them up here would double-report them. + rule { + source_labels = ["__meta_docker_container_label_com_docker_compose_project"] + regex = sys.env("COMPOSE_PROJECT_NAME") + action = "keep" + } + + // Never tail this pipeline's own containers. When the export path is broken, every + // failed batch writes an error line; tailing that line queues it, it fails, and + // writes another — an amplifying self-feed with no exit. The json-file driver note + // in the overlay only prevents the log-DRIVER loop; this prevents the discovery one. + rule { + source_labels = ["__meta_docker_container_label_com_docker_compose_service"] + regex = "alloy|docker-socket-proxy" + action = "drop" + } + + // The Compose service name is the stable per-deployable identifier the monitoring + // stack's conventions ask for: `api`, never `api-prod-2`. + rule { + source_labels = ["__meta_docker_container_label_com_docker_compose_service"] + target_label = "service_name" + } + + rule { + source_labels = ["__meta_docker_container_label_com_docker_compose_service"] + target_label = "container" + } +} + +loki.source.docker "containers" { + host = "tcp://docker-socket-proxy:2375" + targets = discovery.relabel.containers.output + labels = { + env = sys.env("ENVIRONMENT"), + project = sys.env("PROJECT"), + + // Without this every host collapses onto one identity: Alloy scrapes its OWN + // in-container exporter, so the address Prometheus derives `instance` from is + // identical on every machine. The moment a second host ships, the series + // interleave and every host-level alert becomes meaningless. The coalesce + // keeps a blank /etc/hostname from silently reintroducing exactly that: the + // container hostname is a poor identity, but it is a distinguishable one. + host_name = coalesce(string.trim_space(local.file.hostname.content), constants.hostname), + } + forward_to = [otelcol.receiver.loki.containers.receiver] +} + +// Loki-shaped entries in, OpenTelemetry logs out. This is the only reason Alloy is +// here rather than a plain OTel collector: loki.source.docker is the least fiddly way +// to read Docker's log stream with real container labels attached. +otelcol.receiver.loki "containers" { + output { + logs = [otelcol.processor.transform.resource_attributes.input] + } +} + +// Promote the identifiers the monitoring stack keys dashboards on from log-record +// attributes to RESOURCE attributes. error_mode = "ignore" is deliberate: if the +// conversion ever names these attributes differently, logs still arrive with the +// attributes unpromoted, rather than the pipeline dropping them. +otelcol.processor.transform "resource_attributes" { + error_mode = "ignore" + + log_statements { + context = "log" + + statements = [ + `set(resource.attributes["service.name"], attributes["service_name"]) where attributes["service_name"] != nil`, + `set(resource.attributes["env"], attributes["env"]) where attributes["env"] != nil`, + `set(resource.attributes["project"], attributes["project"]) where attributes["project"] != nil`, + `set(resource.attributes["host.name"], attributes["host_name"]) where attributes["host_name"] != nil`, + ] + } + + output { + logs = [otelcol.processor.batch.default.input] + } +} + +// --------------------------------------------------------------------------- +// Host metrics. node_exporter's collectors, scraped in-process and converted to OTLP +// so they ride the same endpoint and token as the logs above — no second pipeline, no +// second hostname, no second credential. +// +// set_collectors is an allowlist rather than the default set: fewer series, and each one +// is here for a reason. `hwmon` earns its place as a dashboard panel — thermal data on a +// physical box is cheap and occasionally telling — but it is NOT a control: fan RPM is a +// symptom of load, legitimate load spins fans too, and no threshold on it would be +// trustworthy. The crash loop that made this memorable is properly detected by container +// restart count, not by temperature. +prometheus.exporter.unix "host" { + // The exporter runs inside the container, so point it at the host's mounted /proc, + // /sys and /. + procfs_path = "/host/proc" + sysfs_path = "/host/sys" + rootfs_path = "/rootfs" + + set_collectors = [ + "cpu", + "diskstats", + "filesystem", + "hwmon", + "loadavg", + "meminfo", + "netdev", + "stat", + "uname", + ] + + filesystem { + // Pseudo-filesystems and per-container overlays: neither is disk anyone can free. + mount_points_exclude = "^/(dev|proc|sys|run)($|/)|^/rootfs/(dev|proc|sys|run)($|/)|^/var/lib/docker/" + } +} + +// --------------------------------------------------------------------------- +// Per-container resource metrics. This is what makes a crash loop visible: cAdvisor has +// no restart counter, but container_start_time_seconds moving repeatedly is the same +// signal, and container_oom_events_total says why. The incident that produced these +// runbooks — 668 restarts over 19 hours, every monitor green — is detected by +// +// changes(container_start_time_seconds{name!=""}[1h]) > 3 +// +// which is a rule on the central stack, not here. This side only has to ship the series. +// Shipping them at all needs `cgroup: host` on this container: see the note in +// compose.telemetry.yml, without which cAdvisor reports one root series and the +// rule above matches nothing. +prometheus.exporter.cadvisor "containers" { + docker_host = "tcp://docker-socket-proxy:2375" + + // Both defaults are cardinality traps. store_container_labels = true turns every + // container label AND environment variable into a Prometheus label, which on Compose + // smears com_docker_compose_* across every series; docker_only = false additionally + // emits raw systemd-slice cgroups nobody will ever read. With these two set, a + // ten-container host is a few hundred series. + store_container_labels = false + docker_only = true + + allowlisted_container_labels = [ + "com.docker.compose.project", + "com.docker.compose.service", + ] +} + +prometheus.scrape "cadvisor" { + targets = prometheus.exporter.cadvisor.containers.targets + forward_to = [prometheus.relabel.cadvisor_scope.receiver] + scrape_interval = "30s" + job_name = "cadvisor" +} + +// cAdvisor walks the host's cgroup tree, so it sees EVERY container on the box. The +// COMPOSE_PROJECT_NAME filter on discovery.docker above scopes log tailing only — it +// does not reach this exporter. On a host that also runs a dev stack, an unrelated +// tool, or the monitoring stack itself, those containers otherwise arrive stamped with +// THIS deployment's project and env, firing this project's alerts for something it does +// not own and double-reporting anything that ships its own telemetry. Keep only this +// project's containers, using the compose label the allowlist above preserves. +// +// The unlabelled root-cgroup series drops out with them, which is correct: it carries no +// `name`, so no container rule can match it, and node_exporter above already owns +// host-level totals. +prometheus.relabel "cadvisor_scope" { + forward_to = [prometheus.relabel.host.receiver] + + rule { + source_labels = ["container_label_com_docker_compose_project"] + regex = sys.env("COMPOSE_PROJECT_NAME") + action = "keep" + } +} + +prometheus.scrape "host" { + targets = prometheus.exporter.unix.host.targets + forward_to = [prometheus.relabel.host.receiver] + scrape_interval = "30s" + job_name = "host" +} + +// Same identifiers the logs carry, so host metrics and container logs can be filtered +// by the same env/project in one dashboard. +prometheus.relabel "host" { + forward_to = [otelcol.receiver.prometheus.host.receiver] + + rule { + target_label = "env" + replacement = sys.env("ENVIRONMENT") + action = "replace" + } + + rule { + target_label = "project" + replacement = sys.env("PROJECT") + action = "replace" + } + + // See the note on the log labels: `instance` is not a host identity here. + rule { + target_label = "host_name" + replacement = coalesce(string.trim_space(local.file.hostname.content), constants.hostname) + action = "replace" + } +} + +// --------------------------------------------------------------------------- +// Alloy's own health. The exporter below is the only place that counts what THIS agent +// drops (otelcol_exporter_send_failed_*): without shipping it, an outage's log loss is +// not just possible but uncounted, because the counters live on a loopback port nobody +// reads. Riding the same pipeline is fine — when the pipeline is down these metrics go +// dark too, and that absence is what the central ProjectTelemetrySilent rule fires on. +prometheus.exporter.self "alloy" { } + +prometheus.scrape "alloy" { + targets = prometheus.exporter.self.alloy.targets + forward_to = [prometheus.relabel.host.receiver] + scrape_interval = "30s" + job_name = "alloy" +} + +// --------------------------------------------------------------------------- +// GPU metrics, present only on hosts that run compose.telemetry.gpu.yml. No targets +// and costs nothing when the exporter container is absent, so the same config file is +// correct on every host — a GPU host is not a different design, it is one extra overlay. +// +// Alloy has no native GPU component (the whole prometheus.exporter.* list was checked, +// and OTel Collector contrib has no NVIDIA receiver either), so an exporter container +// scraped over the Compose network is the pattern. It still means no new agent, no new +// endpoint and no new credential. +discovery.relabel "gpu_exporter" { + targets = discovery.docker.containers.targets + + rule { + source_labels = ["__meta_docker_container_label_com_docker_compose_project"] + regex = sys.env("COMPOSE_PROJECT_NAME") + action = "keep" + } + + rule { + source_labels = ["__meta_docker_container_label_com_docker_compose_service"] + regex = "nvidia-gpu-exporter" + action = "keep" + } + + rule { + target_label = "service_name" + replacement = "gpu" + action = "replace" + } +} + +prometheus.scrape "gpu" { + targets = discovery.relabel.gpu_exporter.output + forward_to = [prometheus.relabel.host.receiver] + + // GPU metrics are cheap to over-scrape and the temptation is 5s. 30s matches the + // other scrapes here: thermal throttling and XID faults persist for far longer than + // one interval, and a training run's utilisation curve does not need sub-minute + // resolution to be readable. + scrape_interval = "30s" + job_name = "gpu" +} + +otelcol.receiver.prometheus "host" { + output { + metrics = [otelcol.processor.batch.default.input] + } +} + +otelcol.processor.batch "default" { + output { + logs = [otelcol.exporter.otlphttp.central.input] + metrics = [otelcol.exporter.otlphttp.central.input] + } +} + +otelcol.exporter.otlphttp "central" { + client { + // The tunnel routes HTTPS to the collector's HTTP receiver only; there is no + // public gRPC path, which is why this is otlphttp and not otlp. The endpoint + // MUST be https:// — an http:// value ships the bearer token and every log + // line in cleartext, and nothing here can tell (see .env.example). + endpoint = sys.env("OTEL_EXPORTER_OTLP_ENDPOINT") + + headers = { + Authorization = "Bearer " + sys.env("OTLP_AUTH_TOKEN"), + + // Second, weaker credential for projects whose egress crosses a WAF: the + // project's WAF-skip rule matches THIS header, so the bearer token itself + // never appears in a ruleset expression readable through the provider's + // API. It grants only the managed-security skip. Projects with no WAF in + // front of their egress can leave TELEMETRY_EDGE_KEY empty; the header is + // then sent empty and ignored. + "X-Telemetry-Key" = sys.env("TELEMETRY_EDGE_KEY"), + } + } + + // Explicit rather than collector defaults, so an outage's behaviour is a decision: + // retry for 15 minutes, hold up to 2000 batches in memory, then drop and COUNT the + // drop (shipped by prometheus.exporter.self above). The queue is the reason the + // overlay caps this container's memory — see mem_limit there before raising it. + sending_queue { + queue_size = 2000 + } + + retry_on_failure { + max_elapsed_time = "15m" + } +} diff --git a/templates/compose.telemetry.gpu.yml b/templates/compose.telemetry.gpu.yml new file mode 100644 index 0000000..695177e --- /dev/null +++ b/templates/compose.telemetry.gpu.yml @@ -0,0 +1,58 @@ +# compose.telemetry.gpu.yml — VENDORED from the central monitoring repo at a pinned tag. +# +# OPTIONAL OVERLAY — GPU metrics on a host with NVIDIA hardware. +# +# Adds a GPU exporter that the Alloy agent discovers and scrapes automatically. Include +# it only on hosts that have a card: +# +# docker compose ... -f compose.telemetry.yml -f compose.telemetry.gpu.yml up -d +# +# or set GPU_METRICS=1 in the host's root `.env` and let the deploy recipes include it. +# +# Why nvidia_gpu_exporter and not dcgm-exporter, which is the reflex answer: DCGM's +# advantage is its DCGM_FI_PROF_* profiling fields, and NVIDIA document those as +# datacentre-only hardware features. On a GeForce card they are simply absent, which +# removes the entire reason to prefer DCGM while keeping its SYS_ADMIN requirement and +# its 2021-vintage dashboard. Consumer cards also have no framebuffer ECC, so the ECC +# fields are empty too. What remains is the NVML field set, which is exactly what this +# exporter reads. +# +# If the department ever buys datacentre silicon (A100/H100/L40S), profiling, ECC, NVLink and MIG +# all light up and dcgm-exporter becomes strictly better. That is a one-container swap +# plus one dashboard — do not pre-build for it. + +services: + nvidia-gpu-exporter: + image: utkuozdemir/nvidia_gpu_exporter:1.14.0-nvml@sha256:82acc3fc60a5a709846ea9757bbccb170fd141039fa47e5899c55fe9a60f56fe + restart: unless-stopped + # The NVML backend is what produces XID error counters, which the nvidia-smi + # backend cannot report. XIDs are the signal that matters most here: a stuck kernel, + # an uncorrectable memory fault or a card that has fallen off the bus are invisible + # to utilisation graphs, and they are what silently kills a long training run. + environment: + NVIDIA_DRIVER_CAPABILITIES: utility + deploy: + resources: + reservations: + devices: + - driver: nvidia + count: all + capabilities: [gpu] + mem_limit: 256m + pids_limit: 256 + # The exporter only reads NVML and serves HTTP; the driver files arrive as runtime + # mounts, so a read-only root and an empty capability set cost it nothing. + read_only: true + cap_drop: + - ALL + security_opt: + - no-new-privileges:true + # No published ports and no host network: Alloy reaches it over the Compose network, + # discovers it by its Compose service label, and nothing else can. + networks: + - egress + logging: + driver: json-file + options: + max-size: "10m" + max-file: "3" diff --git a/templates/compose.telemetry.yml b/templates/compose.telemetry.yml new file mode 100644 index 0000000..3cce129 --- /dev/null +++ b/templates/compose.telemetry.yml @@ -0,0 +1,159 @@ +# compose.telemetry.yml — VENDORED from the central monitoring repo at a pinned tag. +# +# One Grafana Alloy agent per deploy host. Vendor it as-is; everything that differs +# between projects arrives as an environment variable. See templates/README.md. +# +# An instrumented application sends its own telemetry. This overlay adds a Grafana +# Alloy agent that collects three things the application cannot report on itself and +# forwards them to the same OTLP endpoint with the same token: +# +# - the stdout of every OTHER container in the stack, as logs; +# - host resource metrics — CPU, memory, load, disk, network, hwmon; +# - per-container resources and lifecycle from cAdvisor, which is what makes a crash +# loop visible centrally. +# +# It replaces the older Loki Docker-driver overlay. That approach needed a Loki push +# hostname exposed to the internet, which the monitoring stack deliberately does not +# publish because Loki has no authentication of its own, and it wrote the push +# credential into every container's HostConfig where `docker inspect` could read it. +# +# The root deploy recipes auto-include this overlay when OTEL_EXPORTER_OTLP_ENDPOINT is +# set in the host's root `.env` — the same switch that turns on the API's own exporter. +# Hosts without it ship nothing and keep `docker logs` as the only log path. + +services: + # Least-privilege gate between Alloy and the Docker API. Alloy never sees the socket: + # it talks HTTP to this proxy, which allows exactly the read-only endpoints discovery, + # log tailing and cAdvisor need (containers, events, info, version, ping) and refuses + # everything else — POST is off, so nothing behind it can start or exec a container. + # The proxy itself holds the socket, so IT is the root-equivalent component; it runs + # nothing but haproxy with a static allowlist, joins only the internal socket network, + # and is the single place to audit. + docker-socket-proxy: + image: tecnativa/docker-socket-proxy:v0.5.0@sha256:1f5038b54f06c3e18422902cf00ba21803d1c97805aae032e5e6673d532d3459 + restart: unless-stopped + environment: + CONTAINERS: 1 # list/inspect/logs — discovery.docker, loki.source.docker + NETWORKS: 1 # discovery.docker computes network labels; 403 here kills discovery + EVENTS: 1 # container lifecycle stream — cAdvisor watches it + INFO: 1 # machine info — cAdvisor + VERSION: 1 + PING: 1 + POST: 0 # no mutations, ever + volumes: + - /var/run/docker.sock:/var/run/docker.sock:ro + security_opt: + - no-new-privileges:true + mem_limit: 64m + pids_limit: 128 + networks: + - docker_socket + logging: + driver: json-file + options: + max-size: "10m" + max-file: "3" + + alloy: + image: grafana/alloy:v1.18.1@sha256:0f4434c92b3e6cdac38bb129b344e1790c246f7b6e2eaffcc16a5fa363240e33 + restart: unless-stopped + # Root for the host mounts below (/host/proc, /host/sys, /rootfs): node_exporter's + # collectors read files that are root-only on the host. The Docker API is reached + # through docker-socket-proxy above, so root here no longer implies control of the + # Docker daemon. + user: root + command: + - run + - /etc/alloy/config.alloy + - --storage.path=/var/lib/alloy/data + # Bound to loopback: the diagnostics UI has no auth and nothing needs to reach it + # from another container. + - --server.http.listen-addr=127.0.0.1:12345 + environment: + ENVIRONMENT: ${ENVIRONMENT:?telemetry overlay requires ENVIRONMENT} + # The project identity every signal is labelled with, and what the central + # ProjectTelemetrySilent rule keys on. Must match the value bootstrap.sh was run with. + PROJECT: ${PROJECT:?telemetry overlay requires PROJECT} + # Alloy only reads containers belonging to this Compose project, so it must know + # which one it is. Compose always injects COMPOSE_PROJECT_NAME into interpolation + # — the deploy directory's base name unless -p or .env overrides it — so this + # passes through whatever the label on the sibling containers actually says. A + # `:-` default here would be dead text. This is the COMPOSE project (docker's -p), + # not PROJECT above; rename the deploy directory and the filter follows it. + COMPOSE_PROJECT_NAME: ${COMPOSE_PROJECT_NAME} + OTEL_EXPORTER_OTLP_ENDPOINT: ${OTEL_EXPORTER_OTLP_ENDPOINT:?telemetry overlay requires OTEL_EXPORTER_OTLP_ENDPOINT} + OTLP_AUTH_TOKEN: ${OTLP_AUTH_TOKEN:?telemetry overlay requires OTLP_AUTH_TOKEN} + # Second, weaker credential: matched by the Cloudflare WAF-skip rule for the + # telemetry hostname (infra/cloudflare-zone), so the bearer token itself never + # appears in a Cloudflare ruleset expression. Rotates independently of the token. + # Optional: only projects whose egress crosses a WAF need this. Empty is fine. + TELEMETRY_EDGE_KEY: ${TELEMETRY_EDGE_KEY:-} + volumes: + - ./deploy/alloy/config.alloy:/etc/alloy/config.alloy:ro # vendored, never edited + # node_exporter's collectors read the host's own /proc, /sys and filesystems. + # All read-only. Note this does give the container read access to the host + # filesystem via /rootfs, which is the price of host disk-usage metrics from a + # container; it is no more than a host-installed node_exporter running as root + # already has, but it is worth knowing rather than discovering. + - /proc:/host/proc:ro + - /sys:/host/sys:ro + - /:/rootfs:ro + # cAdvisor's filesystem reads; container metadata comes over the socket proxy. + - /var/lib/docker:/var/lib/docker:ro + - /dev/disk:/dev/disk:ro + # Read positions survive restarts; without this every restart re-ships whatever + # the containers still have on disk. + - alloy_data:/var/lib/alloy/data + # cAdvisor finds containers by walking the cgroup hierarchy, NOT through the Docker + # API — the socket proxy only supplies names and metadata for cgroups it has already + # found. Under Docker's default private cgroup namespace this container sees just its + # own cgroup, as "/", so every container_* metric collapses to a single root series + # with no `name` label: the central ContainerRestarting rule matches zero series and + # ContainerOOMKilled cannot say which container died. Joining the host's namespace is + # the whole fix; mounting /sys/fs/cgroup as well is redundant, verified on cgroup v2. + # It grants read access to the host's cgroup tree, which is strictly less than the + # /rootfs and /var/lib/docker mounts above already give. + cgroup: host + # cAdvisor reads OOM-kill events from the kernel log; without the device AND + # CAP_SYSLOG (kernel.dmesg_restrict gates /dev/kmsg reads) container_oom_events_total + # never increments and the central ContainerOOMKilled rule can never fire ("Could + # not configure a source for OOM detection" in the agent log is this exact gap). + devices: + - /dev/kmsg:/dev/kmsg + # Root, but not root-with-everything: the default capability set is dropped like the + # other hardened services here (socket proxy above, backup). DAC_READ_SEARCH is what + # lets the collectors read root-only files under /host/proc, /host/sys and /rootfs; + # unlike the default set's DAC_OVERRIDE it grants no write-side permission bypass, + # and the rest (SETUID, MKNOD, NET_RAW, ...) is nothing a metrics agent needs. + cap_drop: + - ALL + cap_add: + - SYSLOG + - DAC_READ_SEARCH + security_opt: + - no-new-privileges:true + # Bounded like every other service on the host: during a collector outage the + # export queue grows in memory, and without a ceiling the OOM killer picks by RSS — + # the plausible victim being postgres or the API, i.e. the agent taking down the + # thing it observes. Keep in step with the sending_queue size in config.alloy. + mem_limit: 512m + pids_limit: 512 + networks: + - egress + - docker_socket + logging: + # Never through itself: Alloy reporting its own failure to ship logs, by shipping + # a log, is a loop with no exit. + driver: json-file + options: + max-size: "10m" + max-file: "3" + +volumes: + alloy_data: + +networks: + # Carries only the proxied Docker API between the proxy and Alloy. Internal: it has + # no gateway, so nothing on it can reach out and nothing outside can reach in. + docker_socket: + internal: true diff --git a/templates/run_scheduled.sh b/templates/run_scheduled.sh new file mode 100755 index 0000000..38ec8b4 --- /dev/null +++ b/templates/run_scheduled.sh @@ -0,0 +1,103 @@ +#!/usr/bin/env bash +# VENDORED from the central monitoring repo at a pinned tag. Generic: nothing here +# names a project or a job list. +# +# Run one scheduled job and report the result to a dead-man's switch. +# +# Usage: run_scheduled.sh +# job — any name; it selects the ping URL variable and the command below. +# env — the deployment environment, passed through to the command. +# +# Called by this host's systemd units. This is the ONE piece of monitoring +# that does not share fate with the observability stack, and that is its whole reason to +# exist. Everything else — logs, traces, host metrics — flows through Alloy to the +# department collector, so a dead host, a dead collector, a broken tunnel or an expired +# token all look identical from Grafana: silence. A push from the host to an external +# endpoint is the one signal that still arrives when that pipeline is the thing that +# broke, and its absence is itself the alarm. +# +# The ping lives here rather than in each unit so the reporting is written once. +# +# Ping URLs come from the host file loaded by the units, never from the repository — +# they are capability URLs. bootstrap.sh on the central host prints the PING_* block. An unset URL disables the ping +# for that job without failing it, so a host that has not been wired up yet still runs +# its jobs. +set -uo pipefail + +job="${1:-}" +env_name="${2:-}" + +if [[ -z "$job" || -z "$env_name" ]]; then + echo "usage: $0 " >&2 + exit 2 +fi +# Job and env become part of an environment-variable name below, so keep them to +# characters that can appear in one — a stray character would silently look up the +# wrong variable and disable the ping rather than fail. +if [[ ! "$job" =~ ^[A-Za-z0-9_-]+$ || ! "$env_name" =~ ^[A-Za-z0-9_-]+$ ]]; then + echo "error: job and env must match [A-Za-z0-9_-]+" >&2 + exit 2 +fi + +ROOT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/.." && pwd)" +cd "$ROOT_DIR" || exit 1 + +# `just` is resolved at unit-render time and passed in; fall back to PATH for manual runs. +JUST_BIN="${JUST_BIN:-just}" + +# One `just` recipe per job, named the same as the job — that identity is the contract, +# which is why there is no mapping table here. Projects that do not use `just` can point +# JUST_BIN at any runner with the same shape. +command=("$JUST_BIN" "$job" "$env_name") + +# One variable per job, so each gets its own check. Sharing a URL would let a frequent +# job's pings mask a rare one's silence — exactly the failure the rare job exists to catch. +url_var="PING_${job//-/_}" +url_var="${url_var^^}" +ping_url="${!url_var:-}" + +output_file="$(mktemp)" +trap 'rm -f "$output_file"' EXIT + +# A killed job must still report. systemd's TimeoutStartSec TERMs the whole cgroup: +# the job dies, and without this trap bash would die too — before the ping block — +# so a HUNG job would send neither success nor failure and its captured output would +# be lost. The job runs in the background so `wait` can be interrupted by the signal; +# the child has already received the same TERM from systemd (KillMode=control-group). +# shellcheck disable=SC2329 # invoked via the TERM/INT traps below +on_terminate() { + local sig="$1" + echo "run_scheduled: received SIG${sig}; job killed (likely a systemd timeout)" >>"$output_file" + cat "$output_file" + if [[ -n "$ping_url" ]]; then + curl -fsS -m 10 --retry 3 --data-binary "@${output_file}" "${ping_url}/fail" -o /dev/null \ + || echo "WARNING: failure ping to ${url_var} failed" >&2 + fi + rm -f "$output_file" + exit 143 +} +trap 'on_terminate TERM' TERM +trap 'on_terminate INT' INT + +status=0 +"${command[@]}" >"$output_file" 2>&1 & +wait $! || status=$? +cat "$output_file" + +if [[ -z "$ping_url" ]]; then + exit "$status" +fi + +if [[ "$status" -eq 0 ]]; then + curl -fsS -m 10 --retry 3 "$ping_url" -o /dev/null \ + || echo "WARNING: success ping to ${url_var} failed" >&2 +else + # Send the job's own output as the failure body: the alert then carries the reason, + # instead of only saying that something went wrong. Note this puts job output — + # hostnames, paths, restic summaries — in a third party's hands; it is why the ping + # carries no credentials and why the URL itself is the only secret. + curl -fsS -m 10 --retry 3 --data-binary "@${output_file}" "${ping_url}/fail" -o /dev/null \ + || echo "WARNING: failure ping to ${url_var} failed" >&2 +fi + +exit "$status" From 25a9d45bd1d38838f44e0b992f2a3659de08ae2f Mon Sep 17 00:00:00 2001 From: Simon van Lierde Date: Tue, 1 Sep 2026 12:32:42 +0000 Subject: [PATCH 21/85] docs: close out the hub-and-spoke migration - mark ADR 0002 complete and release the spoke-watchdog tripwire - delete HANDOVER.md: it described a transition, not a system - point onboarding at templates/README.md --- README.md | 31 +- docs/HANDOVER.md | 297 ------------------- docs/RUNBOOK.md | 31 +- docs/adr/0002-hub-and-spoke-observability.md | 11 +- 4 files changed, 46 insertions(+), 324 deletions(-) delete mode 100644 docs/HANDOVER.md diff --git a/README.md b/README.md index 9872684..d219219 100644 --- a/README.md +++ b/README.md @@ -77,7 +77,7 @@ distributed ingestion would add operational weight for no gain. The reasoning, and the alternatives we considered, are recorded in [ADR 0001](docs/adr/0001-observability-stack.md). The hub-and-spoke target for serving multiple CML projects is [ADR 0002](docs/adr/0002-hub-and-spoke-observability.md), -and the open migration work toward it lives in [docs/HANDOVER.md](docs/HANDOVER.md). +Onboarding a project onto it is [templates/README.md](templates/README.md). ## Run it for real @@ -103,8 +103,8 @@ OpenTofu configuration. Applying it produces the `CLOUDFLARE_TUNNEL_TOKEN` that `just up-tunnel` needs; bootstrap steps are at the top of [infra/main.tf](infra/main.tf). -`just check` validates compose files, Prometheus config and alert rules, the -collector, Alertmanager, Loki and Tempo configs, YAML, workflows, OpenTofu +`just check` validates compose files, Prometheus config, the collector, Loki +and Tempo configs, Grafana provisioning, YAML, workflows, OpenTofu formatting, and dashboard JSON. Every validator runs in a pinned container, so nothing needs to be installed on the host. CI runs the same command on every push and pull request, plus a smoke test that boots the stack, waits for @@ -124,22 +124,26 @@ Grafana Alloy for log files — are in ## Alerting -Prometheus evaluates the rules in `config/alerts/`: scrape target down, OTel -export failures, error rate above 5%, disk above 80%. Alertmanager delivers -them to whatever webhook you set in `ALERT_WEBHOOK_URL` (ntfy, Slack, and so -on). +Grafana both evaluates and delivers, from `config/grafana/alerting/`: telemetry +silent per project, container crash-looping, container OOM-killed, scrape target +down, OTel export failures, alert delivery failing, error rate above 5%, disk +above 80%. Notifications go to whatever webhook you set in `ALERT_WEBHOOK_URL` +(ntfy, Slack, and so on). There is no Alertmanager: Grafana rules can query Loki +as well as Prometheus, and one engine owning both means one answer to "who gets +told". One rule, `Watchdog`, fires permanently by design and posts to `HEARTBEAT_URL` every five minutes. Point that at a dead man's switch such as healthchecks.io — a service that alerts when the pings *stop* — and you will also hear about the -one failure the host cannot report itself: its own death. Both variables are -optional; leave them unset and alerts are simply visible in Grafana. +one failure the host cannot report itself: its own death. Set both: an unset +`ALERT_WEBHOOK_URL` drops every alert while the heartbeat keeps reporting +healthy, so `just up-tunnel` refuses to start without it. ## Storage Everything persists to local Docker volumes (`loki_data`, `tempo_data`, -`prometheus_data`, `grafana_data`, `alertmanager_data`), all of which `just -backup` captures. A sixth, `otel_queue`, holds the collector's on-disk export +`prometheus_data`, `grafana_data`), all of which `just +backup` captures. A fifth, `otel_queue`, holds the collector's on-disk export queue — seconds of in-flight telemetry, worthless by the time anyone restores, so it is deliberately left out. When local disk stops fitting, Loki and Tempo can move to any S3-compatible object store (Backblaze B2, Cloudflare R2, @@ -158,9 +162,8 @@ config/ loki.yaml # logs tempo.yaml # traces prometheus.yaml # metrics - alertmanager.yaml # alert routing (webhook + watchdog heartbeat) - alerts/ # Prometheus alert rules - grafana/ # provisioned datasources + dashboard loader + grafana/ # provisioned datasources, dashboards, and alerting + # alerting/ = rules, contact points, routing tree dashboards/ # drop JSON dashboards here; Grafana auto-loads them docs/ # runbook, onboarding templates, ADRs, screenshots infra/ # OpenTofu: Cloudflare tunnel, ingress routes, DNS diff --git a/docs/HANDOVER.md b/docs/HANDOVER.md deleted file mode 100644 index 8db2c71..0000000 --- a/docs/HANDOVER.md +++ /dev/null @@ -1,297 +0,0 @@ -# Handover: the central stack's half of the August 2026 review - -Owner: Simon van Lierde -Review date: 2026-09-19 - -Relab's side of an architecture review is done and merged (its repo carries the full -review in `deploy/MONITORING-DESIGN.md`; the target architecture is recorded here as -[ADR 0002](adr/0002-hub-and-spoke-observability.md)). This document is the actionable -half for THIS repo: what Relab now emits, what the central stack has to do with it, and -what should be deleted here. It is written to be actionable without reading the Relab -repo. - -**Delete this file once the work below is done.** It describes a transition, not a system. - -______________________________________________________________________ - -## Why this exists - -A backup container crash-looped 668 times over 19 hours. Every monitor read green the -whole time, because the only backup check was newest-snapshot age — which a crash loop -makes look *better* than healthy. It was noticed by ear, from fan noise. - -Nothing here is about adding tools. It is about the fact that several failures currently -have no detector at all, and one of them is the failure that already happened. - -______________________________________________________________________ - -## What Relab now emits - -One Grafana Alloy agent per deploy host, one OTLP/HTTP endpoint, one bearer token. No new -hostnames, no new credentials, nothing to configure centrally per host. - -| Signal | Source | Notes | -| ---------------------------- | --------------------------------------- | -------------------------------------------------------------------------------------------- | -| Application traces + metrics | the API's own OTel SDK | `service.name=relab-api` | -| Container logs | Alloy `loki.source.docker` | every container's stdout, labelled by Compose service | -| Host metrics | Alloy `prometheus.exporter.unix` | CPU, memory, load, disk, network, `hwmon` | -| Per-container metrics | Alloy `prometheus.exporter.cadvisor` | CPU, memory, network, disk I/O, `container_start_time_seconds`, `container_oom_events_total` | -| GPU metrics | `nvidia_gpu_exporter`, scraped by Alloy | opt-in overlay, present only on GPU hosts | - -Every signal carries four identity labels: **`project`**, **`env`**, **`service.name`**, -**`host_name`**. Enforced by the agent config, so a host cannot omit them. - -Verified live 2026-08-27, with two corrections. Alloy attaches these as **ordinary series -labels, not OTLP resource attributes** — so `target_info` carries none of them, and any rule -keyed on `target_info` is keyed on nothing. And Relab's `env` is **`staging`**, not -`production`; no `production` series exist. `config/prometheus.yaml` promotes the four -attributes anyway, for spokes whose app SDK sends them the resource-attribute way. - -Application logs are **no longer** exported by the API's SDK. Alloy ships that container's -stdout, so exporting them twice stored every line twice in two shapes. The stdout path is -the one kept, because it also carries what the SDK cannot report: the SDK's own export -failures. - -______________________________________________________________________ - -## The keystone alert, and the four behind it - -Nothing on the Relab side can detect its own absence. These rules are the deliverable — -without them the telemetry is a dashboard, not monitoring. - -| Rule | Shape | For | Catches | -| ---------------------------- | --------------------------------------------------------- | --- | -------------------------------------------------------------------------------------------------------------------- | -| **`ProjectTelemetrySilent`** | no logs or metrics with `project=X, env=Y` | 15m | Host down, Docker down, Alloy down, tunnel down, token rotated wrong, collector rejecting. **Write this one first.** | -| **`ContainerRestarting`** | `changes(container_start_time_seconds{name!=""}[1h]) > 3` | 10m | The 668-restart incident. A crash loop becomes louder than health instead of quieter. | -| `ContainerOOMKilled` | increase in `container_oom_events_total` | 0m | Distinct cause, distinct fix. | -| `HostDiskSpaceLow` | < 20% free, by `host_name` | 15m | Data already arrives; nothing reads it. | -| `OtelExportFailures` | collector send-failed rate > 0 | 5m | Backend rejecting data. | - -Keep the total under about ten rules. Past that the operator stops reading them. - -Only `ProjectTelemetrySilent` needs generating per project and environment. The rest are -generic once `host_name` and `project` labels are correct. - -For GPU hosts, three more: `nvidia_smi_gpu_recovery_action > 0` (the driver wants a -reset — the best single health alert), a thermal/power throttle flag, and XID faults with -an explicit code allowlist rather than every code, since most are application faults: - -```promql -time() - nvidia_smi_xid_last_timestamp_seconds{xid=~"48|62|64|74|79|95|119|120"} < 300 -``` - -XIDs are the GPU analogue of the crash-loop blind spot: a stuck kernel, an uncorrectable -memory fault or a card off the bus are invisible to utilisation graphs, and they are what -silently kills a twelve-hour training run. - -______________________________________________________________________ - -## Delete list for the monitoring repo - -- **Alertmanager**, its volume, its `url_file` entrypoint hack, its `amtool` check step and - its backup mount. Move every rule to Grafana-managed alert rules provisioned from YAML - under `/etc/grafana/provisioning/alerting/`. Three reasons, by weight: - - 1. **Grafana rules can query Loki; Prometheus rules cannot.** The alerts worth having are - log-shaped and metric-shaped at once. Splitting the rule engine means the most - valuable ones cannot be written at all. - 1. One place owns notification, instead of two answers to "who gets told". - 1. It deletes a container, a volume, a config file and a check step. - - Caveat that shapes the work: Grafana's built-in Alertmanager handles Grafana-managed - alerts only, and there is no documented endpoint for an external Prometheus to POST into - it. So the Prometheus rule files must genuinely *become* Grafana rules — this is not a - re-pointing of delivery. That is four rules; an afternoon. - - *The opposite choice is right if* you want alerting to survive Grafana being down. It - does not buy that today, because Alertmanager sits on the same host and dies with it. - The real answer to "alerting is down" is the heartbeat below. - -- **ONBOARDING templates 3 and 4** (Loki Docker driver, `loki.write` to a push hostname). - Both document a path that requires exposing Loki, which this stack deliberately does not - do because Loki has no authentication of its own. Relab followed template 3, and it cost - a day to undo. - -- **`compose.storage-s3.yml`** and **Tempo's span-metrics dependency** — neither is - load-bearing; move RED metrics onto native OTLP HTTP metrics instead. - -- **Grafana OnCall OSS** is not an option in 2026: maintenance mode March 2025, archived - read-only March 2026, paid Grafana Cloud IRM the only successor. If it comes up, that is - the answer. - -______________________________________________________________________ - -## The heartbeat, and why it is not optional - -Everything above travels through one collector over one tunnel. A dead host, a dead -collector, a broken tunnel and an expired token are indistinguishable from Grafana's point -of view: silence that looks like health. - -Relab's scheduled jobs each ping a per-job dead-man's switch (healthchecks.io) directly -from the host. That is the one signal that does not share fate with this stack, and it must -stay. On the central side, add a `Watchdog` rule (`vector(1)`, always firing) routed to a -heartbeat contact point on a short repeat interval — its *silence* is the alarm. - -healthchecks.io's free tier is exactly 20 checks. Relab uses three per environment; that is -the ceiling to plan against as projects onboard. - -______________________________________________________________________ - -## Dashboards: the actual scalability lever - -Today there is a per-project dashboard. N projects means N hand-maintained dashboards, and -that will stop this scaling long before storage does. - -Replace with three dashboards carrying a `project` template variable — *Service Health*, -*Logs*, *Host & Containers* — so a new project gets full dashboards the moment its first -telemetry lands, with nobody editing JSON. - -Import rather than author: - -- per-container resources: [15798](https://grafana.com/grafana/dashboards/15798-docker-monitoring/) - (revised 2025-07-12); [19792](https://grafana.com/grafana/dashboards/19792-cadvisor-dashboard/) - is the Compose-aware second choice. Avoid 10619 and 893 — high download counts, untouched - since 2019 and Grafana 4 respectively. Download counts measure inertia, not maintenance. -- GPU: [14574](https://grafana.com/grafana/dashboards/14574) (revised 2026-08-04) and its - multi-GPU companion 25547. Do **not** use the canonical DCGM dashboard 12239: last - revised 2021, and its panels lean on profiling fields consumer cards cannot produce. - -### The one experiment to run first - -Stock cAdvisor dashboards assume Prometheus *scraped* cAdvisor — they key on `job`, -`instance` and `name` as a scrape produces them. Relab's metrics arrive over OTLP, where -Prometheus reconstructs `job` from `service.name`, `instance` from `service.instance.id`, -and applies a translation strategy to metric names. - -**Run and answered (2026-08-27): names survive.** Confirmed twice — once against a -scratch Prometheus v3.13.2, once against the live stack, where Relab's own -`container_cpu_usage_seconds_total` and friends are present under their exact upstream -names. So: keep the single OTLP path, import 15798, and do not reach for the experimental -`otlp.translation_strategy: NoTranslation`. - -**But do not repoint 15798's variables at `host_name`/`project` yet, and do not import it -expecting per-container panels.** See the cAdvisor gap below — the per-container series the -dashboard is built on are not arriving at all. - -______________________________________________________________________ - -## Two settings that are easy to miss - -- **`out_of_order_time_window: 30m`** on Prometheus. The official OTel guide requires it for - OTLP ingestion; without it, late batches are silently dropped. "Silently" is the operative - word. -- **Prometheus's OTLP receiver is documented as "not an efficient way of ingesting - samples"**, for "specific low-volume use cases". CML is exactly the case that sentence - carves out — a few hundred series per host at 30s. The deciding factor for revisiting is - volume: if a host exceeds a few thousand active series, move infrastructure metrics to - `remote_write` behind an authenticating proxy and leave app metrics on OTLP. - -______________________________________________________________________ - -## Onboarding a second project - -This is the part worth doing well: it is the difference between a stack that serves CML and -one that serves Relab. - -The central repo should publish: - -1. `templates/alloy/config.alloy` — one shared agent config, parameterised only by - environment variables. **Relab's `deploy/alloy/config.alloy` is this file already**, - minus two hardcoded `"relab"` strings. No project should ever edit it. -1. `templates/compose.telemetry.yml` — Relab's `compose.logging.alloy.yaml` is ~90% of it. -1. `templates/compose.telemetry.gpu.yml` — Relab's `compose.gpu.yaml`, likewise. -1. `templates/run_scheduled.sh` — the dead-man's-switch wrapper; Relab's is generic apart - from job names. -1. `templates/alerts/project.yaml.tmpl` — the rules above, with `{{project}}` and `{{env}}` - substituted. -1. `bootstrap.sh ` — renders the alert rules and reloads Grafana, creates the - healthchecks.io checks and prints their URLs, prints the `.env` block to paste on the - project host, and prints the `curl` that vendors the templates at a pinned tag. - -A new project's whole checklist then is: vendor two files, add six `.env` variables, include -the overlay, run `bootstrap.sh`. Under an hour, most of it waiting. - -One trap to design out: `ProjectTelemetrySilent` only exists for projects whose -`bootstrap.sh` was actually run. A project that ships telemetry but skipped bootstrap is -silently uncovered — the exact failure class the rule closes, one level up. Cheapest -countermeasure: one standing rule that fires on any series whose `project` label has no -matching provisioned rule group; failing that, make the onboarding doc say plainly that -bootstrap is what creates the safety net, not the telemetry. - -Take the templates from Relab, then **re-vendor Relab from them**, so Relab is proof the -path works rather than a special case that drifted. - -______________________________________________________________________ - -## Sequencing - -**Tripwire first:** the owner and review date at the top of this file are load-bearing. -Relab's local watchdog checks (service health, snapshot age, timer state) are scheduled -for deletion *only after* steps 1–2 below are live and verified — if nobody has picked -this work up by the review date, the watchdog stays, and the deletion item in Relab's -`deploy/MONITORING-DESIGN.md` §2.1 must not proceed on optimism. Deleting a weak local -signal before its central replacement exists trades a weak signal for none. - -1. ~~`ProjectTelemetrySilent`~~ — done and **verified against live traffic**; - `config/alerts/projects.yaml`. Keyed on a label-only selector, `project`/`env` on the - series themselves, because `target_info` carries no identity labels here. -1. `ContainerRestarting` + `ContainerOOMKilled` — written (`config/alerts/stack.yaml`, - group `container-lifecycle`) but **blind: no data reaches them.** See below. This is - the open item; the incident that started all of this is still undetected. -1. ~~Run the metric-name experiment~~ — done, names survive. Import of 15798/14574 is - blocked behind the same cAdvisor gap. -1. `HostDiskSpaceLow`, `OtelExportFailures`, `Watchdog` heartbeat. -1. Grafana-managed alerting; delete Alertmanager. -1. Generic dashboards with a `project` variable. -1. Templates and `bootstrap.sh`; re-vendor Relab from them. - -Steps 1–2 convert this from telemetry into monitoring. Everything after is leverage. - -### What live verification found (2026-08-27) - -Checked against the running stack with Relab reporting. Step 1 holds. Step 2 does not. - -- **cAdvisor sends only the root cgroup.** `container_start_time_seconds`, - `container_oom_events_total` and every other `container_*` metric have exactly **one** - series each, `id="/"`, and there is no `name` label on any of them. So - `changes(container_start_time_seconds{name!=""}[1h]) > 3` matches zero series and - **`ContainerRestarting` can never fire**; `ContainerOOMKilled` sees only host-level OOM - and cannot say which container. The 668-restart incident would still be invisible today. - - The fix is on the spoke, not here: Relab's Alloy needs the mounts - `prometheus.exporter.cadvisor` requires to see container cgroups (Docker socket, - `/sys/fs/cgroup`, `/var/lib/docker`). Nothing in this repo can close it. **Until it is - closed, the tripwire holds and Relab's local watchdog checks stay.** - -- **30% of all OTLP metric writes were being rejected** — 19,585 HTTP 400s, 1,307 points - dropped every 30 seconds, silently, for as long as the counters go back. - `prometheus_tsdb_out_of_order_samples_total` matched the 400 count exactly: every - rejection was an out-of-order sample. Fixed by the `out_of_order_time_window` this branch - already carried; zero rejections since. - - Two operational notes worth keeping. `out_of_order_time_window` is **not applied by a - config reload** — SIGHUP logs a successful load and leaves the window at 0. It needs - `docker compose up -d --force-recreate prometheus`; plain `up -d` and `restart` are both - no-ops for a config-only change. And this failure mode is exactly what `OtelExportFailures` - exists to catch, which means it had been firing, unread, the whole time. - -- **`relab-api`'s own SDK metrics are not arriving.** The only jobs carrying - `project="relab"` are `gpu`, `integrations/self`, `integrations/unix` and - `integrations/cadvisor` — all Alloy. The emission table above claims app traces and - metrics under `service.name=relab-api`; the metrics half is not there. Traces were not - checked. - -______________________________________________________________________ - -## Be skeptical of these - -This came from an architecture review that verified claims against current documentation and -flagged what it could not. One is still worth checking rather than trusting: - -- the Cloudflare Zero Trust free-plan seat count — the widely-cited figure appears only in - third-party posts, never in Cloudflare's own docs; -- ~~the OTLP metric-name round trip~~ — run on 2026-08-27; names survive. See above. - -There is also **no official cadence recommendation** for `restic check --read-data-subset`; -the commonly repeated "1/12 monthly" is forum folklore. Pick a cadence, write down why, and -treat the number as arbitrary-but-declared. diff --git a/docs/RUNBOOK.md b/docs/RUNBOOK.md index b410c9f..e4d412b 100644 --- a/docs/RUNBOOK.md +++ b/docs/RUNBOOK.md @@ -14,13 +14,13 @@ panel is a backup/restore drill. Both procedures are below. ```sh just ps # what's running, what's restarting -just logs # follow logs (otel-collector, loki, tempo, prometheus, grafana, alertmanager) +just logs # follow logs (otel-collector, loki, tempo, prometheus, grafana) just restart ``` Two alerts point here. `TargetDown` fires after two minutes when Prometheus can't scrape a target, and it scrapes every service in the -stack — collector, node-exporter, Grafana, Loki, Tempo, Alertmanager, and +stack — collector, node-exporter, Grafana, Loki, Tempo, and itself — so the alert names whichever one went quiet. `OtelExportFailures` means the collector is up but a backend is rejecting its data, so look at that backend's logs, not the collector's. @@ -100,8 +100,10 @@ the host is the intended setup. ## Alert delivery -Prometheus evaluates the rules; Alertmanager delivers them. Two -environment variables control where: +Grafana evaluates the rules and delivers them; there is no Alertmanager. Rules, +contact points and the routing tree are provisioned from +`config/grafana/alerting/`, so the UI shows them read-only — edit the YAML. +Two environment variables control where notifications go: - `ALERT_WEBHOOK_URL` receives all alerts (any webhook: ntfy, Slack, …). - `HEARTBEAT_URL` receives the always-firing `Watchdog` every five @@ -109,13 +111,24 @@ environment variables control where: raises the alarm when pings **stop** — that is the "monitoring host is dead" signal nothing inside the host can send. -Leaving both empty is fine: nothing is delivered, Alertmanager logs one -notify error per cycle (expected, harmless), and alerts remain visible in -Grafana. After changing either variable, `docker compose up -d -alertmanager`. +Leaving them empty is not a safe default. Delivery then fails silently while +the heartbeat keeps pinging, so the dead man's switch reads healthy and every +real alert is dropped. `just up-tunnel` refuses to start without +`ALERT_WEBHOOK_URL`, and `AlertDeliveryFailing` fires on a failing notifier. + +After changing either variable, `docker compose up -d grafana`. Compose only +recreates a container when its own definition changes, so a `.env` edit needs +that command — a plain `restart` keeps the old environment, and the stale value +survives with no indication that it has. ## Changing the Cloudflare edge +> Unverified: the Cloudflare Zero Trust free-plan seat count that the Access policy +> below assumes. The widely-cited figure appears only in third-party posts, never in +> Cloudflare's own documentation. Check it against your plan before adding people, not +> after they cannot log in. + + The tunnel, its ingress rules, both DNS records, and the Cloudflare Access policy that fronts Grafana are all OpenTofu in `infra/`. Change them there, not in the Zero Trust dashboard: the next apply reverts anything clicked in @@ -154,7 +167,7 @@ cd infra && tofu apply ## Upgrading images Dependabot opens PRs that bump the pinned versions, and CI runs `just -check` on each one. The validators (promtool, otelcol, amtool) read their +check` on each one. The validators (promtool, otelcol) read their image versions from `compose.yml`, so every bump is checked with the exact binaries the stack will run — when a new version changes its config syntax, CI fails loudly before the change reaches the host. That is the diff --git a/docs/adr/0002-hub-and-spoke-observability.md b/docs/adr/0002-hub-and-spoke-observability.md index 1f8c5fa..f5022a5 100644 --- a/docs/adr/0002-hub-and-spoke-observability.md +++ b/docs/adr/0002-hub-and-spoke-observability.md @@ -1,7 +1,8 @@ # ADR 0002: Hub-and-spoke observability for CML projects -Date: 2026-08-20. Status: accepted (records the target architecture from the -August 2026 review; the migration work is tracked in [HANDOVER.md](../HANDOVER.md)). +Date: 2026-08-20. Status: accepted; **migration complete 2026-08-28**. The +transitional HANDOVER.md that tracked it has been deleted — it described a +transition, not a system. Onboarding is [templates/README.md](../../templates/README.md). Supersedes one decision from ADR 0001: RED metrics move off Tempo's span-metrics and onto the applications' native OTLP HTTP metrics. Everything else in ADR 0001 @@ -83,8 +84,10 @@ Contracts that make it scale: - Tempo demotes to trace storage only; deleting its metrics-generator dependency makes it disposable on its next breaking upgrade. - The spokes' local watchdog checks (service health, snapshot age, timer state) - are deletable only after `ProjectTelemetrySilent` and the container-lifecycle - rules are live here — see the tripwire in [HANDOVER.md](../HANDOVER.md). + were deletable only once `ProjectTelemetrySilent` and the container-lifecycle + rules were live here. Both are, and both have been seen working on real traffic: + the keystone fired on an actual spoke outage on 2026-08-27 and resolved when the + host came back. That tripwire has therefore been released. - Prometheus needs `out_of_order_time_window: 30m` for OTLP ingestion; without it late batches drop silently. The OTLP receiver is documented as a low-volume path — the revisit trigger is a host exceeding a few thousand From 8ff57fa463c7759a3ddc6f935837b4a3b766a475 Mon Sep 17 00:00:00 2001 From: Simon van Lierde Date: Tue, 1 Sep 2026 12:42:26 +0000 Subject: [PATCH 22/85] fix(alerting): drop the removed config/alerts mount, assert provisioned rule count - config/alerts/ went with Alertmanager, but Prometheus and the promtool validator still bind-mounted it, so Docker kept recreating it root-owned and promtool checked a config with no rules - assert in `just smoke` that as many alert rules provisioned as there are on disk; a malformed file provisions none of its own group, so a non-zero check would miss it --- README.md | 11 ++++++----- compose.yml | 1 - justfile | 3 ++- 3 files changed, 8 insertions(+), 7 deletions(-) diff --git a/README.md b/README.md index d219219..7b50d4e 100644 --- a/README.md +++ b/README.md @@ -104,11 +104,12 @@ OpenTofu configuration. Applying it produces the `CLOUDFLARE_TUNNEL_TOKEN` that [infra/main.tf](infra/main.tf). `just check` validates compose files, Prometheus config, the collector, Loki -and Tempo configs, Grafana provisioning, YAML, workflows, OpenTofu -formatting, and dashboard JSON. Every validator runs in a pinned container, so -nothing needs to be installed on the host. CI runs the same command on every -push and pull request, plus a smoke test that boots the stack, waits for -Grafana to come up healthy, and checks that every dashboard provisioned. +and Tempo configs, YAML, workflows, OpenTofu formatting, and dashboard JSON. +Every validator runs in a pinned container, so nothing needs to be installed on +the host. Grafana's alerting provisioning has no offline validator, so `just +smoke` covers it instead: it boots the stack, waits for Grafana to come up +healthy, and checks that every dashboard and every alert rule provisioned. CI +runs both on every push and pull request. ## Sending telemetry from a project diff --git a/compose.yml b/compose.yml index 73ca847..5549660 100644 --- a/compose.yml +++ b/compose.yml @@ -82,7 +82,6 @@ services: restart: unless-stopped volumes: - ./config/prometheus.yaml:/etc/prometheus/prometheus.yaml:ro - - ./config/alerts:/etc/prometheus/alerts:ro - prometheus_data:/prometheus command: - --config.file=/etc/prometheus/prometheus.yaml diff --git a/justfile b/justfile index a1f60dc..9057062 100644 --- a/justfile +++ b/justfile @@ -71,7 +71,7 @@ check: docker compose config -q CLOUDFLARE_TUNNEL_TOKEN=dummy docker compose -f compose.yml -f compose.tunnel.yml config -q docker compose -f compose.yml -f compose.demo.yml config -q - docker run --rm -v ./config/prometheus.yaml:/etc/prometheus/prometheus.yaml:ro -v ./config/alerts:/etc/prometheus/alerts:ro --entrypoint promtool $(docker compose config --images | grep prom/prometheus) check config /etc/prometheus/prometheus.yaml + docker run --rm -v ./config/prometheus.yaml:/etc/prometheus/prometheus.yaml:ro --entrypoint promtool $(docker compose config --images | grep prom/prometheus) check config /etc/prometheus/prometheus.yaml docker run --rm -e OTLP_AUTH_TOKEN=dummy -v ./config/otel-collector.yaml:/etc/otelcol/config.yaml:ro $(docker compose config --images | grep opentelemetry-collector) validate --config=/etc/otelcol/config.yaml docker run --rm --network none -v ./config/loki.yaml:/etc/loki/loki.yaml:ro $(docker compose config --images | grep grafana/loki) -config.file=/etc/loki/loki.yaml -verify-config docker run --rm --network none -v ./config/tempo.yaml:/etc/tempo/tempo.yaml:ro $(docker compose config --images | grep grafana/tempo) -config.file=/etc/tempo/tempo.yaml -config.verify=true @@ -116,5 +116,6 @@ smoke: _queue-volume docker compose up -d n=0; until curl -sf http://localhost:3000/api/health >/dev/null; do n=$((n+3)); [ $n -ge 120 ] && { echo "Grafana not healthy after 120s" >&2; exit 1; }; sleep 3; done @for uid in $(docker run --rm --network none -v ./dashboards:/dashboards:ro ghcr.io/jqlang/jq:1.8.1 -r .uid $(ls dashboards/*.json | sed 's|^dashboards|/dashboards|')); do n=0; until curl -sf -u "admin:${GRAFANA_ADMIN_PASSWORD}" "http://localhost:3000/api/dashboards/uid/$uid" >/dev/null; do n=$((n+3)); [ $n -ge 60 ] && { echo "error: dashboard $uid was not provisioned" >&2; exit 1; }; sleep 3; done; done + @want=$(grep -h '^ *title:' config/grafana/alerting/*.yaml | wc -l); got=$(curl -sf -u "admin:${GRAFANA_ADMIN_PASSWORD}" http://localhost:3000/api/v1/provisioning/alert-rules | docker run --rm -i ghcr.io/jqlang/jq:1.8.1 length); [ "$want" = "$got" ] || { echo "error: $want alert rules on disk, $got provisioned — a malformed file provisions none of its group. See just logs grafana" >&2; exit 1; } @[ -z "$(docker compose ps -q --status=restarting --status=exited)" ] || { echo "error: services not running:" >&2; docker compose ps >&2; exit 1; } @echo "Stack healthy" From 030a1f3e24a2e67c27a0b801476fe100e1b9d02b Mon Sep 17 00:00:00 2001 From: Simon van Lierde Date: Tue, 1 Sep 2026 12:42:44 +0000 Subject: [PATCH 23/85] fix(alerting): key the coverage backstop on project and env MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A project bootstrapped for one environment read as covered while its other environments had no keystone rule — the same silent gap the rule exists to close, one level in. - read the pair back from a COVERS marker in each rendered file rather than splitting the filename, which is ambiguous when either half has a dash - match with `unless on (project, env)` over the rendered pairs - correct the template header: rendered flat, and the file is coverage.yaml --- bootstrap.sh | 28 +++++++++++++------ config/grafana/alerting/coverage.yaml | 14 ++++++---- .../alerting/project-relab-staging.yaml | 13 +++++---- templates/alerting/project.yaml.tmpl | 13 +++++---- 4 files changed, 44 insertions(+), 24 deletions(-) diff --git a/bootstrap.sh b/bootstrap.sh index 5ca0805..a96eb89 100755 --- a/bootstrap.sh +++ b/bootstrap.sh @@ -67,18 +67,28 @@ sed -e "s/__PROJECT__/${project}/g" \ echo "rendered $rendered" # ------------------------------------------------------------ 2. the coverage backstop -# Every project with a rendered rule file, derived from the directory rather than a list -# someone has to remember to update. -covered="$(find "$out_dir" -maxdepth 1 -name 'project-*.yaml' -exec basename {} .yaml \; \ - | sed -e 's/^project-//' -e 's/-[^-]*$//' | sort -u | paste -sd'|' -)" -: "${covered:=__none__}" +# Every project/environment with a rendered rule file, read back from the COVERS marker +# each one carries rather than from a list someone has to remember to update. The pair +# matters, not just the project: a project bootstrapped for staging that also ships prod +# would otherwise read as covered while prod has no keystone rule at all — the same +# silent gap one level further in. +pairs="$(sed -n 's/^# COVERS: //p' "$out_dir"/project-*.yaml | sort -u)" +: "${pairs:=__none__ __none__}" +covered="$(echo "$pairs" | sed 's| |/|' | paste -sd',' - | sed 's/,/, /g')" +covered_expr="$(echo "$pairs" \ + | sed 's|^\([^ ]*\) \([^ ]*\)$|{project="\1",env="\2"}|' \ + | paste -sd'|' - | sed 's/|/ or /g')" cat > "${out_dir}/coverage.yaml" < on the monitoring host." + summary: "{{ \$labels.project }}/{{ \$labels.env }} is sending telemetry but has no alert rules" + description: "Telemetry is arriving for a project/environment bootstrap.sh was never run for, so nothing would notice if it went silent. Run ./bootstrap.sh {{ \$labels.project }} {{ \$labels.env }} on the monitoring host." COVERAGE echo "rendered ${out_dir}/coverage.yaml (covering: ${covered})" diff --git a/config/grafana/alerting/coverage.yaml b/config/grafana/alerting/coverage.yaml index 6716e2e..31096ef 100644 --- a/config/grafana/alerting/coverage.yaml +++ b/config/grafana/alerting/coverage.yaml @@ -3,9 +3,13 @@ # The trap this closes: ProjectTelemetrySilent only exists for projects bootstrap.sh was # actually run for. A project that ships telemetry but skipped bootstrap is silently # uncovered — the exact failure class the keystone closes, one level up. This fires on -# any series carrying a project label that has no rendered rule file. +# any series whose project/env pair has no rendered rule file. # -# Covered right now: relab +# The selector touches every series every project sends (~3k for one project). Cheap at +# this size, evaluated instantly every 5m; revisit if a project's active series reach +# five figures. +# +# Covered right now: relab/staging apiVersion: 1 @@ -32,7 +36,7 @@ groups: refId: QUERY instant: true editorMode: code - expr: count by (project) ({project!~"relab", project!=""}) + expr: count by (project, env) ({project!="", env!=""} unless on (project, env) ({project="relab",env="staging"})) - refId: FIRING datasourceUid: __expr__ model: @@ -46,5 +50,5 @@ groups: labels: severity: warning annotations: - summary: "Project {{ $labels.project }} is sending telemetry but has no alert rules" - description: "Telemetry is arriving for a project bootstrap.sh was never run for, so nothing would notice if it went silent. Run ./bootstrap.sh {{ $labels.project }} on the monitoring host." + summary: "{{ $labels.project }}/{{ $labels.env }} is sending telemetry but has no alert rules" + description: "Telemetry is arriving for a project/environment bootstrap.sh was never run for, so nothing would notice if it went silent. Run ./bootstrap.sh {{ $labels.project }} {{ $labels.env }} on the monitoring host." diff --git a/config/grafana/alerting/project-relab-staging.yaml b/config/grafana/alerting/project-relab-staging.yaml index 6c0d384..65c41c3 100644 --- a/config/grafana/alerting/project-relab-staging.yaml +++ b/config/grafana/alerting/project-relab-staging.yaml @@ -1,12 +1,15 @@ -# Rendered by bootstrap.sh into config/grafana/alerting/projects/. Do not edit the -# rendered files by hand — re-run bootstrap.sh, or edit this template so every project -# gets the fix. +# Rendered by bootstrap.sh into config/grafana/alerting/, flat — Grafana's alerting +# provisioner does not recurse. Do not edit the rendered files by hand: re-run +# bootstrap.sh, or edit this template so every project gets the fix. # # This is the keystone: nothing on a spoke can detect its own absence. A project that # ships telemetry but never had bootstrap.sh run is silently uncovered — that is what -# the ProjectsUncovered rule in _coverage.yaml exists to catch, one level up. +# the ProjectsUncovered rule in coverage.yaml exists to catch, one level up. # -# Placeholders: relab staging proj-silent-relab-staging +# The marker below is read back by bootstrap.sh to regenerate that coverage rule. It is +# the pair, not the filename, because both halves may contain a dash: `project-a-b-c` +# cannot be split back into the project and env that produced it. +# COVERS: relab staging apiVersion: 1 diff --git a/templates/alerting/project.yaml.tmpl b/templates/alerting/project.yaml.tmpl index 52cbf4a..f8e7ae9 100644 --- a/templates/alerting/project.yaml.tmpl +++ b/templates/alerting/project.yaml.tmpl @@ -1,12 +1,15 @@ -# Rendered by bootstrap.sh into config/grafana/alerting/projects/. Do not edit the -# rendered files by hand — re-run bootstrap.sh, or edit this template so every project -# gets the fix. +# Rendered by bootstrap.sh into config/grafana/alerting/, flat — Grafana's alerting +# provisioner does not recurse. Do not edit the rendered files by hand: re-run +# bootstrap.sh, or edit this template so every project gets the fix. # # This is the keystone: nothing on a spoke can detect its own absence. A project that # ships telemetry but never had bootstrap.sh run is silently uncovered — that is what -# the ProjectsUncovered rule in _coverage.yaml exists to catch, one level up. +# the ProjectsUncovered rule in coverage.yaml exists to catch, one level up. # -# Placeholders: __PROJECT__ __ENV__ __UID__ +# The marker below is read back by bootstrap.sh to regenerate that coverage rule. It is +# the pair, not the filename, because both halves may contain a dash: `project-a-b-c` +# cannot be split back into the project and env that produced it. +# COVERS: __PROJECT__ __ENV__ apiVersion: 1 From a40012b73f4b3f02dfd3710cad33448e97a50e40 Mon Sep 17 00:00:00 2001 From: Simon van Lierde Date: Tue, 1 Sep 2026 12:42:55 +0000 Subject: [PATCH 24/85] docs: correct claims the alerting and onboarding moves left stale - nothing detects an empty ALERT_WEBHOOK_URL: AlertDeliveryFailing routes to that same webhook and the heartbeat stays green, so the up-tunnel guard is the only control - ONBOARDING no longer carries the Loki-driver and Alloy templates - move the Cloudflare seat-count caveat to the access bullet it qualifies --- README.md | 13 +++++++------ config/grafana/alerting/contact-points.yaml | 7 +++++-- docs/RUNBOOK.md | 11 ++++------- 3 files changed, 16 insertions(+), 15 deletions(-) diff --git a/README.md b/README.md index 7b50d4e..4c43060 100644 --- a/README.md +++ b/README.md @@ -76,8 +76,8 @@ The stack deliberately runs on a single host. At CML's telemetry volume, distributed ingestion would add operational weight for no gain. The reasoning, and the alternatives we considered, are recorded in [ADR 0001](docs/adr/0001-observability-stack.md). The hub-and-spoke target for -serving multiple CML projects is [ADR 0002](docs/adr/0002-hub-and-spoke-observability.md), -Onboarding a project onto it is [templates/README.md](templates/README.md). +serving multiple CML projects is [ADR 0002](docs/adr/0002-hub-and-spoke-observability.md), and onboarding a project +onto it is [templates/README.md](templates/README.md). ## Run it for real @@ -114,10 +114,11 @@ runs both on every push and pull request. ## Sending telemetry from a project You need three things: the OTLP endpoint, the bearer token (`OTLP_AUTH_TOKEN`), -and a few naming conventions. Copy-paste templates for each route — zero-code -Python/FastAPI, plain OTLP environment variables, the Loki Docker driver, and -Grafana Alloy for log files — are in -**[docs/ONBOARDING.md](docs/ONBOARDING.md)**. +and a few naming conventions. Copy-paste templates for the two application +routes — zero-code Python/FastAPI, and plain OTLP environment variables — are in +**[docs/ONBOARDING.md](docs/ONBOARDING.md)**. Everything an application cannot +report about itself comes from the vendored agent in +**[templates/README.md](templates/README.md)**. > [!WARNING] > Never publish ports 4317/4318 to the internet. The compose file binds them to diff --git a/config/grafana/alerting/contact-points.yaml b/config/grafana/alerting/contact-points.yaml index feea705..c291bcd 100644 --- a/config/grafana/alerting/contact-points.yaml +++ b/config/grafana/alerting/contact-points.yaml @@ -2,8 +2,11 @@ # straight from the environment and no secret is written to disk. # # An unset variable leaves the URL empty and every notification fails — silently, from -# the outside. AlertDeliveryFailing and the heartbeat are what surface that; the guard -# in `just up-tunnel` is what prevents it. +# the outside. Nothing here catches that: AlertDeliveryFailing routes to this same +# webhook, so it fails exactly the way it is detecting, and the heartbeat rides its own +# contact point and stays green throughout. The guard in `just up-tunnel` is the only +# control. AlertDeliveryFailing earns its place for the other case — a URL that is set +# but whose receiver is rejecting or flapping, where a later retry does get through. apiVersion: 1 diff --git a/docs/RUNBOOK.md b/docs/RUNBOOK.md index e4d412b..88e8f3a 100644 --- a/docs/RUNBOOK.md +++ b/docs/RUNBOOK.md @@ -123,12 +123,6 @@ survives with no indication that it has. ## Changing the Cloudflare edge -> Unverified: the Cloudflare Zero Trust free-plan seat count that the Access policy -> below assumes. The widely-cited figure appears only in third-party posts, never in -> Cloudflare's own documentation. Check it against your plan before adding people, not -> after they cannot log in. - - The tunnel, its ingress rules, both DNS records, and the Cloudflare Access policy that fronts Grafana are all OpenTofu in `infra/`. Change them there, not in the Zero Trust dashboard: the next apply reverts anything clicked in @@ -145,7 +139,10 @@ cd infra && tofu apply removed address keeps working until their Access session expires (24h), so for an urgent revocation also revoke the session in Zero Trust. At least one address has to remain — the variable's validation rejects an - empty list, which would lock everyone out of Grafana. + empty list, which would lock everyone out of Grafana. Check your plan's Zero + Trust seat count before adding people: the free-plan figure ADR 0002 cites + appears only in third-party posts, never in Cloudflare's own documentation, + so confirm it before someone cannot log in rather than after. - **Per-user Grafana logins:** by default everyone who clears Access then shares the one admin password. Setting `GRAFANA_JWT_AUTH=true` and `CF_ACCESS_TEAM_DOMAIN=` in `.env` makes Grafana verify the Access From d649c3679989725032f4a777ae3bd3a87b32d4ef Mon Sep 17 00:00:00 2001 From: Simon van Lierde Date: Tue, 1 Sep 2026 13:17:59 +0000 Subject: [PATCH 25/85] feat: drive compose overlays from COMPOSE_FILE in .env - overlays are host config: every recipe (up, down, logs, ps, backup) acts on the same COMPOSE_FILE set, so tunnel services are visible to logs/ps on the hosts that run them - drop 'just up-tunnel': 'just up' runs the exposure guards automatically whenever the tunnel overlay is active - document the mechanism in .env.example, the README, and the runbook" --- .env.example | 15 +++- CHANGELOG.md | 45 ++++++---- README.md | 46 +++++----- compose.storage-s3.yml | 6 +- compose.tunnel.yml | 2 +- compose.yml | 5 +- config/grafana/alerting/contact-points.yaml | 2 +- docs/RUNBOOK.md | 97 ++++++++++----------- infra/main.tf | 2 +- justfile | 47 ++++++---- 10 files changed, 148 insertions(+), 119 deletions(-) diff --git a/.env.example b/.env.example index 518a51b..cd12f70 100644 --- a/.env.example +++ b/.env.example @@ -1,5 +1,12 @@ # Copy to .env and fill in. The .env file is gitignored. +# Which compose overlays this host runs. Compose and every `just` recipe read +# it, so `just up`, `logs`, `ps`, `backup` all act on the same set. Unset = +# the core stack only. +# production: COMPOSE_FILE=compose.yml:compose.tunnel.yml +# local demo: COMPOSE_FILE=compose.yml:compose.demo.yml +#COMPOSE_FILE= + GRAFANA_ADMIN_PASSWORD=change-me # Set to the public URL Cloudflare Tunnel exposes Grafana on, e.g.: @@ -7,7 +14,8 @@ GRAFANA_ADMIN_PASSWORD=change-me GRAFANA_ROOT_URL=http://localhost:3000 # Set to true whenever GRAFANA_ROOT_URL is https (i.e. behind the tunnel). -# Marks the session cookie Secure; `just up-tunnel` refuses to run without it. +# Marks the session cookie Secure; with the tunnel overlay in COMPOSE_FILE, +# `just up` refuses to run without it. # Keep false for plain-http localhost use, or logins break. GRAFANA_COOKIE_SECURE=false @@ -18,7 +26,8 @@ OTLP_AUTH_TOKEN=local-dev-token # Where Grafana delivers alert notifications (any webhook: ntfy, Slack, …). # Not optional. An empty value makes every alert fail delivery silently while the # heartbeat below keeps pinging, so the dead man's switch reads healthy and nothing -# reaches anyone. `just up-tunnel` refuses to start without it. +# reaches anyone. With the tunnel overlay active, `just up` refuses to start +# without it. ALERT_WEBHOOK_URL= # Dead man's switch ping target (e.g. https://hc-ping.com/). The Watchdog @@ -31,7 +40,7 @@ HEARTBEAT_URL= # keys): the read-only key cannot create checks, and a ping key only sends pings. HEALTHCHECKS_API_KEY= -# Only needed for `just up-tunnel` (production exposure via Cloudflare Tunnel). +# Only needed with the tunnel overlay (production exposure via Cloudflare Tunnel). # From: Cloudflare Zero Trust → Networks → Tunnels → → Configure → token CLOUDFLARE_TUNNEL_TOKEN= diff --git a/CHANGELOG.md b/CHANGELOG.md index 4ca187e..59a124c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,7 +1,8 @@ # Changelog -Notable changes to this stack. Format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/); -versions follow [SemVer](https://semver.org/). +Notable changes to this stack. Format follows +[Keep a Changelog](https://keepachangelog.com/en/1.1.0/); versions follow +[SemVer](https://semver.org/). ## [Unreleased] @@ -13,17 +14,17 @@ handful of queries that had been quietly measuring the wrong thing. - **Durable export queue**: the collector's send queues are file-backed on a new `otel_queue` volume, so telemetry buffered during a backend outage - survives a collector restart. `just up` (and `demo`, `smoke`, `up-tunnel`) + survives a collector restart. `just up` (and `demo`, `smoke`) prepares the volume's ownership; it is deliberately not backed up. -- **Full-stack scraping**: Prometheus now scrapes Grafana, Loki, Tempo, and - Alertmanager as well, so `TargetDown` covers every service. +- **Full-stack scraping**: Prometheus now scrapes Grafana, Loki, and Tempo as + well, so `TargetDown` covers every service. - **Per-user Grafana logins**, optional: `GRAFANA_JWT_AUTH` plus `CF_ACCESS_TEAM_DOMAIN` make Grafana verify the Cloudflare Access JWT instead of everyone sharing the admin password. - **Memory ceilings** (`mem_limit`) on every service, sized from observed usage, so one runaway component cannot OOM the host. -- **Wider validation**: `just check` also verifies the Alertmanager, Loki, - and Tempo configs and OpenTofu formatting; `just smoke` asserts every +- **Wider validation**: `just check` also verifies the Loki and Tempo configs + and OpenTofu formatting; `just smoke` asserts every dashboard actually provisioned; CI additionally runs `just infra-validate` and a new `just demo-build`. - Dependabot now watches the Cloudflare provider in `infra/`, and the runbook @@ -31,23 +32,34 @@ handful of queries that had been quietly measuring the wrong thing. ### Changed -- **Loki indexes only `service.name`.** Everything else, `service.instance.id` - included, is structured metadata now — one stream per service instead of one - per sender restart. Existing streams keep their old labels until they age - out (30 days). +- **Overlays are host config now**: `COMPOSE_FILE` in `.env` names the compose + file set, and every recipe — `up`, `logs`, `ps`, `backup` — acts on that + same set. `just up-tunnel` is gone; its exposure guards run automatically + whenever the tunnel overlay is active. +- **Loki indexes only the identity labels** (`service.name`, `project`, `env`, + `host.name` — the authoritative list lives in `config/loki.yaml`). Everything + else, `service.instance.id` included, is structured metadata now — one stream + per service instead of one per sender restart. Existing streams keep their + old labels until they age out (30 days). - **Dashboards are provisioned, not editable**: `dashboards/*.json` is mounted read-only and UI saves are off, making the files the source of truth. - The collector's `memory_limiter` is sized in absolute MiB against the container limit, and the queue and retry settings behind the runbook's "buffers for five minutes" are pinned rather than inherited from upstream. -- Tempo's metrics generator is capped at 50k active series, so a sender with - unrouted span names cannot mint Prometheus series without bound. +- **Tempo's metrics generator is removed**: RED comes from the applications' + own OTLP metrics now (ADR 0002), so Tempo stores traces and nothing else. - Dropped the `relab-api` dashboard: the `$service` picker on Service Health and Logs Overview covers it. - README rewritten for a broader CML audience. - Image bumps: Grafana 13.1.4, Loki 3.7.6, Tempo 3.0.3, Prometheus 3.13.2, - the collector 0.156.0, Alertmanager 0.33.1, node-exporter 1.12.1, - cloudflared 2026.8.2, and the demo's Python dependencies. + the collector 0.156.0, node-exporter 1.12.1, cloudflared 2026.8.2, and the + demo's Python dependencies. + +### Removed + +- **Alertmanager**: alerting is Grafana-managed now (ADR 0002) — rules are + provisioned from `config/grafana/alerting/`, and delivery still posts to + `ALERT_WEBHOOK_URL`. ### Fixed @@ -70,7 +82,8 @@ handful of queries that had been quietly measuring the wrong thing. - `no-new-privileges` on every service; the demo image runs as `nobody`. - `GRAFANA_COOKIE_SECURE` marks the session cookie Secure (with strict - SameSite), and `just up-tunnel` refuses to expose the stack without it, a + SameSite), and `just up` with the tunnel overlay refuses to expose the + stack without it, a non-localhost `GRAFANA_ROOT_URL`, and non-default credentials. - GitHub Actions are pinned to commit SHAs, and `just infra-validate` runs against a copy of the sources so state and tfvars never enter the container. diff --git a/README.md b/README.md index 4c43060..7d77c00 100644 --- a/README.md +++ b/README.md @@ -45,11 +45,10 @@ running. ## How it works Everything enters through a single gateway, the OpenTelemetry Collector. A -project only ever configures one endpoint, and we can swap a storage backend -later without touching any application. Tempo also derives request-rate, -error-rate, and duration ("RED") metrics from the traces it receives, so a -service that sends nothing but traces still gets a working dashboard and error -alerting. +project only ever configures one endpoint, and a storage backend can be +swapped later without touching any application. The Service Health dashboard +and the error-rate alert read the standard HTTP metrics that OpenTelemetry +auto-instrumentation emits; traces add per-request drill-down on top. ```mermaid flowchart LR @@ -63,7 +62,6 @@ flowchart LR otel -->|logs| loki["Loki"] otel -->|traces| tempo["Tempo"] otel -->|metrics| prom["Prometheus"] - tempo -->|"span metrics (RED)"| prom grafana["Grafana"] -. queries .-> loki & tempo & prom end ``` @@ -72,35 +70,37 @@ Solid arrows show telemetry being written; dotted arrows show Grafana reading at query time. Locally (`just up` or `just demo`) there is no tunnel involved: everything talks over the compose network and Grafana is at `localhost:3000`. -The stack deliberately runs on a single host. At CML's telemetry volume, -distributed ingestion would add operational weight for no gain. The reasoning, -and the alternatives we considered, are recorded in -[ADR 0001](docs/adr/0001-observability-stack.md). The hub-and-spoke target for -serving multiple CML projects is [ADR 0002](docs/adr/0002-hub-and-spoke-observability.md), and onboarding a project -onto it is [templates/README.md](templates/README.md). +The stack runs on a single host; at CML's telemetry volume, distributed +ingestion would add operational weight for no gain +([ADR 0001](docs/adr/0001-observability-stack.md) records the alternatives). +The hub-and-spoke design for serving multiple CML projects is +[ADR 0002](docs/adr/0002-hub-and-spoke-observability.md), and onboarding a +project onto it is [templates/README.md](templates/README.md). ## Run it for real ```sh cp .env.example .env # set GRAFANA_ADMIN_PASSWORD -just up # core stack, local only -just up-tunnel # production: core stack + Cloudflare Tunnel +just up # the stack; overlays come from COMPOSE_FILE in .env just check # validate every config in the repo ``` Grafana: (admin / whatever you set). -`up-tunnel` refuses to run until the settings that only matter once the stack -is reachable are real: a generated `OTLP_AUTH_TOKEN`, a changed -`GRAFANA_ADMIN_PASSWORD`, `GRAFANA_ROOT_URL` pointing at the tunnel hostname -rather than localhost, and `GRAFANA_COOKIE_SECURE=true` so the session cookie -is marked Secure. An empty `HEARTBEAT_URL` only warns. +`COMPOSE_FILE` in `.env` names the overlays a host runs. Set +`COMPOSE_FILE=compose.yml:compose.tunnel.yml` in the production `.env`, and +every recipe (`up`, `logs`, `ps`, `backup`) acts on that same set. With the +tunnel overlay active, `just up` refuses to run until the settings that only +matter once the stack is reachable are real: a generated `OTLP_AUTH_TOKEN`, a +changed `GRAFANA_ADMIN_PASSWORD`, `GRAFANA_ROOT_URL` pointing at the tunnel +hostname, and `GRAFANA_COOKIE_SECURE=true` so the session cookie is marked +Secure. An empty `HEARTBEAT_URL` only warns. In production the stack sits behind a Cloudflare Tunnel, and that edge is code too. The tunnel, its hostnames, DNS, and the Cloudflare Access rule that puts an email one-time-PIN in front of Grafana all live in `infra/` as a small OpenTofu configuration. Applying it produces the `CLOUDFLARE_TUNNEL_TOKEN` that -`just up-tunnel` needs; bootstrap steps are at the top of +the tunnel overlay needs; bootstrap steps are at the top of [infra/main.tf](infra/main.tf). `just check` validates compose files, Prometheus config, the collector, Loki @@ -139,15 +139,15 @@ every five minutes. Point that at a dead man's switch such as healthchecks.io a service that alerts when the pings *stop* — and you will also hear about the one failure the host cannot report itself: its own death. Set both: an unset `ALERT_WEBHOOK_URL` drops every alert while the heartbeat keeps reporting -healthy, so `just up-tunnel` refuses to start without it. +healthy, so `just up` with the tunnel overlay refuses to start without it. ## Storage Everything persists to local Docker volumes (`loki_data`, `tempo_data`, `prometheus_data`, `grafana_data`), all of which `just backup` captures. A fifth, `otel_queue`, holds the collector's on-disk export -queue — seconds of in-flight telemetry, worthless by the time anyone restores, -so it is deliberately left out. When local disk stops fitting, Loki and +queue: seconds of in-flight telemetry, worthless by the time anyone restores, +so backups skip it. When local disk stops fitting, Loki and Tempo can move to any S3-compatible object store (Backblaze B2, Cloudflare R2, Hetzner, MinIO); `compose.storage-s3.yml` documents the concrete shape of that change. diff --git a/compose.storage-s3.yml b/compose.storage-s3.yml index 653d86b..8dc99b1 100644 --- a/compose.storage-s3.yml +++ b/compose.storage-s3.yml @@ -4,7 +4,7 @@ # to any S3-compatible backend (Cloudflare R2, Backblaze B2, Hetzner, MinIO) # without touching the collector, Prometheus, or any client project. # -# This is deliberately a commented stub: it documents the concrete shape of +# This is a commented stub: it documents the concrete shape of # the change so the README's scale-out claim is real, but it is not meant to # be started until credentials and a bucket exist. To activate: # @@ -35,8 +35,8 @@ # overlay below re-declares each `command` with -config.expand-env=true. # (compose replaces `command` wholesale, it does not merge it.) # -# 2. Uncomment the overlay below and run: -# docker compose -f compose.yml -f compose.storage-s3.yml up -d +# 2. Uncomment the overlay below, add compose.storage-s3.yml to COMPOSE_FILE +# in .env, and `just up`. # # services: # loki: diff --git a/compose.tunnel.yml b/compose.tunnel.yml index 291daf4..8467eac 100644 --- a/compose.tunnel.yml +++ b/compose.tunnel.yml @@ -1,7 +1,7 @@ # Production overlay: expose Grafana (and the OTLP endpoints) via Cloudflare # Tunnel. Requires CLOUDFLARE_TUNNEL_TOKEN in .env. # -# just up-tunnel +# COMPOSE_FILE=compose.yml:compose.tunnel.yml in .env, then `just up` # # The tunnel, its public hostnames, and DNS are managed as code in infra/ # (OpenTofu); `tofu output -raw tunnel_token` yields the token for .env. diff --git a/compose.yml b/compose.yml index 5549660..6f97405 100644 --- a/compose.yml +++ b/compose.yml @@ -87,7 +87,7 @@ services: - --config.file=/etc/prometheus/prometheus.yaml - --storage.tsdb.path=/prometheus - --storage.tsdb.retention.time=30d - # Prometheus 3.x: OTLP ingestion is its own flag, no longer a feature flag. + # Prometheus 3.x: OTLP ingestion is its own flag, not an --enable-feature entry. - --web.enable-otlp-receiver - --enable-feature=native-histograms,exemplar-storage # Whichever hits first wins; the disk-space alert is the backstop. @@ -146,7 +146,8 @@ services: # Land on Stack Health instead of the empty welcome page. GF_DASHBOARDS_DEFAULT_HOME_DASHBOARD_PATH: /var/lib/grafana/dashboards/stack-health.json # Secure cookies would break plain-http localhost logins, so this is - # opt-in; `just up-tunnel` refuses to expose Grafana without it. + # opt-in; with the tunnel overlay active, `just up` refuses to expose + # Grafana without it. GF_SECURITY_COOKIE_SECURE: ${GRAFANA_COOKIE_SECURE:-false} GF_SECURITY_COOKIE_SAMESITE: strict # Opt-in per-user identity from Cloudflare Access, replacing the shared diff --git a/config/grafana/alerting/contact-points.yaml b/config/grafana/alerting/contact-points.yaml index c291bcd..89bbc9c 100644 --- a/config/grafana/alerting/contact-points.yaml +++ b/config/grafana/alerting/contact-points.yaml @@ -4,7 +4,7 @@ # An unset variable leaves the URL empty and every notification fails — silently, from # the outside. Nothing here catches that: AlertDeliveryFailing routes to this same # webhook, so it fails exactly the way it is detecting, and the heartbeat rides its own -# contact point and stays green throughout. The guard in `just up-tunnel` is the only +# contact point and stays green throughout. The exposure guard in `just up` is the only # control. AlertDeliveryFailing earns its place for the other case — a URL that is set # but whose receiver is rejecting or flapping, where a later retry does get through. diff --git a/docs/RUNBOOK.md b/docs/RUNBOOK.md index 88e8f3a..c92dc45 100644 --- a/docs/RUNBOOK.md +++ b/docs/RUNBOOK.md @@ -6,9 +6,9 @@ between them they answer most "what is wrong" questions. ![Stack Health dashboard](img/stack-health.png) -That capture is honest, not staged: the red export-failure spike is a real -Tempo outage after a bad major-version bump, and the gap in the ingest -panel is a backup/restore drill. Both procedures are below. +In that capture, the red export-failure spike is a Tempo outage and the +gap in the ingest panel is a backup/restore drill. Both procedures are +below. ## A service is down or misbehaving @@ -27,8 +27,7 @@ its data, so look at that backend's logs, not the collector's. ## Disk filling up (`HostDiskSpaceLow`) -Retention is only partially size-bounded, and that's by design rather than -oversight: +Retention is only partially size-bounded: | Data | Time limit | Size limit | | --- | --- | --- | @@ -37,14 +36,13 @@ oversight: | Loki chunks | 30d | none — Loki cannot cap total size | | Tempo blocks | 7d | none | -Loki and Tempo simply have no total-size knob, which is why the disk alert -at 80% is the real backstop. When it fires: check the Filesystem panel on -Stack Health, then either free space or shorten a retention window -(`retention_period` in `config/loki.yaml`, `block_retention` in -`config/tempo.yaml`, the `--storage.tsdb.retention.*` flags in -`compose.yml`) and restart the affected service. If disk pressure keeps -coming back, the durable fix is moving Loki and Tempo to object storage — -see `compose.storage-s3.yml`. +Loki and Tempo have no total-size knob, so the disk alert at 80% is the +real backstop. When it fires: check the Filesystem panel on Stack Health, +then either free space or shorten a retention window (`retention_period` +in `config/loki.yaml`, `block_retention` in `config/tempo.yaml`, the +`--storage.tsdb.retention.*` flags in `compose.yml`) and restart the +affected service. If disk pressure keeps coming back, the durable fix is +moving Loki and Tempo to object storage (see `compose.storage-s3.yml`). Memory is bounded per service instead: every service carries a `mem_limit` in `compose.yml`, sized from observed usage with headroom so that one @@ -64,17 +62,16 @@ from a power loss, which every component in the stack does cleanly via its write-ahead log. Two practical notes: - The tarball is mode 0600 and contains secrets (the Grafana database - among them). Copy it off-host over a private channel — a backup on the + among them). Copy it off-host over a private channel: a backup on the disk it protects is a decoration. - The tarball covers the docker volumes and nothing else. The OpenTofu - state for the Cloudflare edge is not in it — see below. + state for the Cloudflare edge is not in it (see below). - During the pause the collector keeps accepting telemetry and buffers it - for five minutes — the `retry_on_failure.max_elapsed_time` pinned on - every exporter in `config/otel-collector.yaml`, not an upstream default - that can move under you. The queue is file-backed, so restarting the - collector inside that window keeps the buffer; a backup that runs longer - than five minutes still drops data, so on large volumes run it at a - quiet hour. + for five minutes (`retry_on_failure.max_elapsed_time` on every exporter + in `config/otel-collector.yaml`). The queue is file-backed, so + restarting the collector inside that window keeps the buffer. A backup + that runs longer than five minutes still drops data, so on large + volumes run it at a quiet hour. How much history you can lose equals how often you run it. A daily cron on the host is the intended setup. @@ -87,14 +84,13 @@ the host is the intended setup. (visible as export errors on their side) until they're updated. A running demo overlay counts as a sender: re-run `just demo` to recreate it with the new token. -- **Tunnel token:** the tunnel is OpenTofu-managed, so read the token back - from there rather than copying it out of the dashboard. Rotate the tunnel - secret in Cloudflare Zero Trust, then `cd infra && tofu apply` (which - refreshes the token data source) and `tofu output -raw tunnel_token`. To - rotate entirely from code instead, `tofu apply -replace=cloudflare_zero_trust_tunnel_cloudflared.monitoring` - builds a new tunnel and repoints both CNAMEs at it — ingestion and Grafana - are unreachable for the minute or so that takes. Either way: new token into - `CLOUDFLARE_TUNNEL_TOKEN` in `.env`, then `just up-tunnel`. +- **Tunnel token:** the tunnel is OpenTofu-managed, so read the token back from there rather than + copying it out of the dashboard. Rotate the tunnel secret in Cloudflare Zero Trust, then + `cd infra && tofu apply` (which refreshes the token data source) and + `tofu output -raw tunnel_token`. To rotate entirely from code instead, + `tofu apply -replace=cloudflare_zero_trust_tunnel_cloudflared.monitoring` builds a new tunnel and + repoints both CNAMEs at it; ingestion and Grafana are unreachable for the minute or so that + takes. Either way: new token into `CLOUDFLARE_TUNNEL_TOKEN` in `.env`, then `just up`. - **Grafana admin password:** change `GRAFANA_ADMIN_PASSWORD` in `.env`, then `docker compose up -d grafana`. @@ -102,24 +98,25 @@ the host is the intended setup. Grafana evaluates the rules and delivers them; there is no Alertmanager. Rules, contact points and the routing tree are provisioned from -`config/grafana/alerting/`, so the UI shows them read-only — edit the YAML. +`config/grafana/alerting/`, so the UI shows them read-only; edit the YAML. Two environment variables control where notifications go: - `ALERT_WEBHOOK_URL` receives all alerts (any webhook: ntfy, Slack, …). - `HEARTBEAT_URL` receives the always-firing `Watchdog` every five minutes. Point it at a dead man's switch (e.g. healthchecks.io) that - raises the alarm when pings **stop** — that is the "monitoring host is + raises the alarm when pings **stop**: that is the "monitoring host is dead" signal nothing inside the host can send. Leaving them empty is not a safe default. Delivery then fails silently while the heartbeat keeps pinging, so the dead man's switch reads healthy and every -real alert is dropped. `just up-tunnel` refuses to start without -`ALERT_WEBHOOK_URL`, and `AlertDeliveryFailing` fires on a failing notifier. +real alert is dropped. With the tunnel overlay active, `just up` refuses to +start without `ALERT_WEBHOOK_URL`, and `AlertDeliveryFailing` fires on a +failing notifier. -After changing either variable, `docker compose up -d grafana`. Compose only -recreates a container when its own definition changes, so a `.env` edit needs -that command — a plain `restart` keeps the old environment, and the stale value -survives with no indication that it has. +After changing either variable, `docker compose up -d grafana`. A plain +`restart` keeps the old environment: compose only rebuilds a container's +environment on `up`, and the stale value survives with no indication that +it has. ## Changing the Cloudflare edge @@ -138,19 +135,18 @@ cd infra && tofu apply `infra/terraform.tfvars` and apply. That list is the entire allowlist. A removed address keeps working until their Access session expires (24h), so for an urgent revocation also revoke the session in Zero Trust. At - least one address has to remain — the variable's validation rejects an - empty list, which would lock everyone out of Grafana. Check your plan's Zero - Trust seat count before adding people: the free-plan figure ADR 0002 cites - appears only in third-party posts, never in Cloudflare's own documentation, - so confirm it before someone cannot log in rather than after. + least one address has to remain: the variable's validation rejects an + empty list, which would lock everyone out of Grafana. Check your plan's + Zero Trust seat limit in the Cloudflare dashboard before adding people; + Cloudflare's documentation does not state the free-plan figure. - **Per-user Grafana logins:** by default everyone who clears Access then shares the one admin password. Setting `GRAFANA_JWT_AUTH=true` and `CF_ACCESS_TEAM_DOMAIN=` in `.env` makes Grafana verify the Access JWT instead, so each address signs in as itself and new ones land on the org's default role (Viewer). The JWK set is team-wide, so if the Zero Trust team fronts more than one Access application, also pin - `GF_AUTH_JWT_EXPECT_CLAIMS` in `compose.yml` to this app's `aud` tag — - otherwise a token minted for any other app in the team is accepted here + `GF_AUTH_JWT_EXPECT_CLAIMS` in `compose.yml` to this app's `aud` tag. + Otherwise a token minted for any other app in the team is accepted here too. - **Adding a hostname:** add an `ingress` entry pointing at the service's container port, plus a matching `cloudflare_dns_record`. The catch-all @@ -164,20 +160,19 @@ cd infra && tofu apply ## Upgrading images Dependabot opens PRs that bump the pinned versions, and CI runs `just -check` on each one. The validators (promtool, otelcol) read their -image versions from `compose.yml`, so every bump is checked with the exact -binaries the stack will run — when a new version changes its config -syntax, CI fails loudly before the change reaches the host. That is the -point. +check` on each one. The validators (promtool, otelcol) read their image +versions from `compose.yml`, so every bump is checked with the exact +binaries the stack will run. When a new version changes its config +syntax, CI fails before the change reaches the host. Dependabot also watches the Cloudflare provider in `infra/`. Those PRs need one manual step: it bumps the constraint in `main.tf` but not the recorded hashes in `.terraform.lock.hcl`, so check the branch out and run `cd infra && tofu init -upgrade`, then `just infra-validate` and `tofu plan` -against the real account — validation proves the syntax parses, only a plan +against the real account. Validation proves the syntax parses; only a plan proves the provider still maps the config to the same resources. -Nothing watches the tool images pinned in the `justfile` — yamllint, +Nothing watches the tool images pinned in the `justfile`: yamllint, actionlint, OpenTofu, jq, and the alpine that backup, restore, and the queue-volume setup run in. No Dependabot ecosystem covers a justfile, so those are bumped by hand. diff --git a/infra/main.tf b/infra/main.tf index 9510b66..63617d7 100644 --- a/infra/main.tf +++ b/infra/main.tf @@ -129,7 +129,7 @@ data "cloudflare_zero_trust_tunnel_cloudflared_token" "monitoring" { } output "tunnel_token" { - description = "Set as CLOUDFLARE_TUNNEL_TOKEN in ../.env for just up-tunnel." + description = "Set as CLOUDFLARE_TUNNEL_TOKEN in ../.env for the tunnel overlay." value = data.cloudflare_zero_trust_tunnel_cloudflared_token.monitoring.token sensitive = true } diff --git a/justfile b/justfile index 9057062..40cca3c 100644 --- a/justfile +++ b/justfile @@ -5,6 +5,14 @@ set dotenv-load stateful := "grafana prometheus loki tempo" backup_mounts := "-v monitoring_grafana_data:/data/grafana -v monitoring_prometheus_data:/data/prometheus -v monitoring_loki_data:/data/loki -v monitoring_tempo_data:/data/tempo" +# Compose with the demo overlay, for the demo lifecycle recipes — explicit -f +# so `just demo` works regardless of what COMPOSE_FILE names. +compose_demo := "docker compose -f compose.yml -f compose.demo.yml" + +# dashboards/*.json as the paths they get when mounted at /dashboards, shared +# by check and smoke. +dash_paths := `ls dashboards/*.json | sed 's|^dashboards|/dashboards|' | tr '\n' ' '` + default: @just --list @@ -15,20 +23,22 @@ _queue-volume: @docker volume create monitoring_otel_queue > /dev/null @docker run --rm --network none -v monitoring_otel_queue:/q alpine:3.24 chown 10001:10001 /q -# Core stack (no tunnel; Grafana at http://localhost:3000) +# The stack (Grafana at http://localhost:3000). Overlays are host config: +# COMPOSE_FILE in .env names the file set (see .env.example), and every recipe +# here — up, down, logs, ps, backup — acts on that same set. With the tunnel +# overlay active, up refuses to start until the exposure guards pass. up: _queue-volume + @case "${COMPOSE_FILE:-}" in *compose.tunnel.yml*) just _expose-guards;; esac docker compose up -d -# Core stack + Cloudflare Tunnel (production; needs CLOUDFLARE_TUNNEL_TOKEN). # Refuses to expose the stack with the documented default credentials. -up-tunnel: _queue-volume +_expose-guards: @[ "${OTLP_AUTH_TOKEN:-}" != "local-dev-token" ] || { echo "error: OTLP_AUTH_TOKEN is still the local default; generate one (openssl rand -hex 32) before exposing ingestion" >&2; exit 1; } @[ "${GRAFANA_ADMIN_PASSWORD:-}" != "change-me" ] || { echo "error: GRAFANA_ADMIN_PASSWORD is still the documented default; change it before exposing Grafana" >&2; exit 1; } @[ "${GRAFANA_ROOT_URL:-}" != "http://localhost:3000" ] || { echo "error: GRAFANA_ROOT_URL is still the localhost default; set it to the tunnel hostname or every absolute URL Grafana generates breaks" >&2; exit 1; } @[ "${GRAFANA_COOKIE_SECURE:-false}" = "true" ] || { echo "error: GRAFANA_COOKIE_SECURE must be true when Grafana is served over HTTPS; set it in .env" >&2; exit 1; } @[ -n "${HEARTBEAT_URL:-}" ] || echo "WARNING: HEARTBEAT_URL is empty; the stack goes live without a dead-man's switch" >&2 @[ -n "${ALERT_WEBHOOK_URL:-}" ] || { echo "error: ALERT_WEBHOOK_URL is empty; every alert would fire into an empty url_file and be dropped. The heartbeat keeps pinging either way, so this failure looks healthy from the outside — set it, or comment out this guard deliberately" >&2; exit 1; } - docker compose -f compose.yml -f compose.tunnel.yml up -d down: docker compose down --remove-orphans @@ -36,24 +46,24 @@ down: # Core stack + a demo telemetry source (see compose.demo.yml), then look at # Grafana: http://localhost:3000 demo: _queue-volume - docker compose -f compose.yml -f compose.demo.yml up -d --build + {{compose_demo}} up -d --build # Build the demo image without starting anything. Used by CI to catch a broken # demo app before it merges. demo-build: - docker compose -f compose.yml -f compose.demo.yml build + {{compose_demo}} build # Remove only the demo services; the core stack keeps running. demo-down: - docker compose -f compose.yml -f compose.demo.yml rm -sf demo-api demo-load + {{compose_demo}} rm -sf demo-api demo-load logs service="": - docker compose -f compose.yml -f compose.demo.yml logs -f {{service}} + docker compose logs -f {{service}} ps: - docker compose -f compose.yml -f compose.demo.yml ps + docker compose ps -# Targets the base stack only; demo services are recreated with `just demo`. +# Targets the COMPOSE_FILE set; demo services are recreated with `just demo`. restart service: docker compose restart {{service}} @@ -62,7 +72,7 @@ pull: # Tail a service's logs as JSON, decoded. Useful before Grafana is set up. tail service: - docker compose -f compose.yml -f compose.demo.yml logs -f --no-log-prefix {{service}} | jq -R 'fromjson? // .' + docker compose logs -f --no-log-prefix {{service}} | jq -R 'fromjson? // .' # Validate everything. All validators run in containers — no host installs. # promtool/otelcol images are read from compose.yml so they can't @@ -70,15 +80,16 @@ tail service: check: docker compose config -q CLOUDFLARE_TUNNEL_TOKEN=dummy docker compose -f compose.yml -f compose.tunnel.yml config -q - docker compose -f compose.yml -f compose.demo.yml config -q - docker run --rm -v ./config/prometheus.yaml:/etc/prometheus/prometheus.yaml:ro --entrypoint promtool $(docker compose config --images | grep prom/prometheus) check config /etc/prometheus/prometheus.yaml - docker run --rm -e OTLP_AUTH_TOKEN=dummy -v ./config/otel-collector.yaml:/etc/otelcol/config.yaml:ro $(docker compose config --images | grep opentelemetry-collector) validate --config=/etc/otelcol/config.yaml - docker run --rm --network none -v ./config/loki.yaml:/etc/loki/loki.yaml:ro $(docker compose config --images | grep grafana/loki) -config.file=/etc/loki/loki.yaml -verify-config - docker run --rm --network none -v ./config/tempo.yaml:/etc/tempo/tempo.yaml:ro $(docker compose config --images | grep grafana/tempo) -config.file=/etc/tempo/tempo.yaml -config.verify=true + {{compose_demo}} config -q + images="$(docker compose config --images)" && \ + docker run --rm -v ./config/prometheus.yaml:/etc/prometheus/prometheus.yaml:ro --entrypoint promtool $(echo "$images" | grep prom/prometheus) check config /etc/prometheus/prometheus.yaml && \ + docker run --rm -e OTLP_AUTH_TOKEN=dummy -v ./config/otel-collector.yaml:/etc/otelcol/config.yaml:ro $(echo "$images" | grep opentelemetry-collector) validate --config=/etc/otelcol/config.yaml && \ + docker run --rm --network none -v ./config/loki.yaml:/etc/loki/loki.yaml:ro $(echo "$images" | grep grafana/loki) -config.file=/etc/loki/loki.yaml -verify-config && \ + docker run --rm --network none -v ./config/tempo.yaml:/etc/tempo/tempo.yaml:ro $(echo "$images" | grep grafana/tempo) -config.file=/etc/tempo/tempo.yaml -config.verify=true docker run --rm --network none -v .:/code:ro pipelinecomponents/yamllint:0.35.13 yamllint -d '{extends: relaxed, ignore: [.git/, backups/, infra/.terraform/]}' . docker run --rm --network none -v .:/repo:ro -w /repo rhysd/actionlint:1.7.12 -color docker run --rm --network none -v ./infra:/infra:ro -w /infra ghcr.io/opentofu/opentofu:1.12.3 fmt -check - docker run --rm -v ./dashboards:/dashboards:ro ghcr.io/jqlang/jq:1.8.1 empty $(ls dashboards/*.json | sed 's|^dashboards|/dashboards|') + docker run --rm -v ./dashboards:/dashboards:ro ghcr.io/jqlang/jq:1.8.1 empty {{dash_paths}} # Full OpenTofu validation (downloads the provider, so not part of `check`). # Runs against a copy of the sources only: state and tfvars never enter the @@ -115,7 +126,7 @@ restore file: smoke: _queue-volume docker compose up -d n=0; until curl -sf http://localhost:3000/api/health >/dev/null; do n=$((n+3)); [ $n -ge 120 ] && { echo "Grafana not healthy after 120s" >&2; exit 1; }; sleep 3; done - @for uid in $(docker run --rm --network none -v ./dashboards:/dashboards:ro ghcr.io/jqlang/jq:1.8.1 -r .uid $(ls dashboards/*.json | sed 's|^dashboards|/dashboards|')); do n=0; until curl -sf -u "admin:${GRAFANA_ADMIN_PASSWORD}" "http://localhost:3000/api/dashboards/uid/$uid" >/dev/null; do n=$((n+3)); [ $n -ge 60 ] && { echo "error: dashboard $uid was not provisioned" >&2; exit 1; }; sleep 3; done; done + @want="$(docker run --rm --network none -v ./dashboards:/dashboards:ro ghcr.io/jqlang/jq:1.8.1 -r .uid {{dash_paths}})"; n=0; while :; do have="$(curl -sf -u "admin:${GRAFANA_ADMIN_PASSWORD}" 'http://localhost:3000/api/search?type=dash-db&limit=5000' | docker run --rm -i ghcr.io/jqlang/jq:1.8.1 -r '.[].uid')"; missing=""; for uid in $want; do echo "$have" | grep -qx "$uid" || missing="$missing $uid"; done; [ -z "$missing" ] && break; n=$((n+3)); [ $n -ge 60 ] && { echo "error: dashboards not provisioned:$missing" >&2; exit 1; }; sleep 3; done @want=$(grep -h '^ *title:' config/grafana/alerting/*.yaml | wc -l); got=$(curl -sf -u "admin:${GRAFANA_ADMIN_PASSWORD}" http://localhost:3000/api/v1/provisioning/alert-rules | docker run --rm -i ghcr.io/jqlang/jq:1.8.1 length); [ "$want" = "$got" ] || { echo "error: $want alert rules on disk, $got provisioned — a malformed file provisions none of its group. See just logs grafana" >&2; exit 1; } @[ -z "$(docker compose ps -q --status=restarting --status=exited)" ] || { echo "error: services not running:" >&2; docker compose ps >&2; exit 1; } @echo "Stack healthy" From d072c6e3da0cb6bd82b49b116ba26066b737c764 Mon Sep 17 00:00:00 2001 From: Simon van Lierde Date: Tue, 1 Sep 2026 13:18:12 +0000 Subject: [PATCH 26/85] refactor: dedup configs and scripts, fix stale docs - render the ProjectsUncovered rule from templates/alerting/coverage.yaml.tmpl instead of a shell heredoc, and re-render every covered pair on each bootstrap run so template fixes propagate to all projects - collapse the identical FIRING threshold node (rules.yaml) and the exporter queue/retry blocks (otel-collector.yaml) into YAML anchors - share the compose file set and dashboard paths via justfile variables, run 'docker compose config --images' once in check, and verify dashboards in smoke with one search call instead of per-uid polling loops - scope the Service Health logs panel to the selected service and use the indexed label_values(project) in the logs dashboard - x-logging anchor in the telemetry overlay, ping_fail() helper in run_scheduled.sh, HC_JOBS override for the healthchecks job list - fix docs contradicting the config: Loki's four identity index labels, Tempo's removed metrics generator, the Alertmanager removal, and the now centrally provisioned GPU/container dashboards" --- bootstrap.sh | 102 +++++++------------------- compose.demo.yml | 3 +- config/grafana/alerting/coverage.yaml | 3 +- config/grafana/alerting/rules.yaml | 84 +++------------------ config/grafana/datasources.yaml | 9 +-- config/loki.yaml | 3 +- config/otel-collector.yaml | 34 +++------ config/tempo.yaml | 4 +- dashboards/logs.json | 4 +- dashboards/service-health.json | 2 +- docs/ONBOARDING.md | 32 ++++---- templates/README.md | 40 +++++----- templates/alerting/coverage.yaml.tmpl | 55 ++++++++++++++ templates/alloy/config.alloy | 7 +- templates/compose.telemetry.yml | 28 +++---- templates/run_scheduled.sh | 19 +++-- 16 files changed, 176 insertions(+), 253 deletions(-) create mode 100644 templates/alerting/coverage.yaml.tmpl diff --git a/bootstrap.sh b/bootstrap.sh index a96eb89..a1b0898 100755 --- a/bootstrap.sh +++ b/bootstrap.sh @@ -13,9 +13,9 @@ # 4. prints the `.env` block to paste on the project host, and the curl that vendors # the templates at a pinned tag. # -# Bootstrap is what creates the safety net, NOT the telemetry. A host can ship perfectly -# good telemetry and still be unmonitored, because the rule that notices its silence -# lives here. That asymmetry is the whole reason this script exists. +# Bootstrap creates the safety net, not the telemetry: a host can ship perfectly good +# telemetry and still be unmonitored, because the rule that notices its silence lives +# here. # # Idempotent: re-running it re-renders the same files and reloads again. set -euo pipefail @@ -58,87 +58,38 @@ out_dir="config/grafana/alerting" tag="$(git -C "$root" describe --tags --abbrev=0 2>/dev/null || echo main)" repo_raw="https://raw.githubusercontent.com/CMLPlatform/monitoring/${tag}/templates" -# ---------------------------------------------------------------- 1. the keystone rule -rendered="${out_dir}/project-${project}-${env_name}.yaml" -sed -e "s/__PROJECT__/${project}/g" \ - -e "s/__ENV__/${env_name}/g" \ - -e "s/__UID__/proj-silent-${project}-${env_name}/g" \ - templates/alerting/project.yaml.tmpl > "$rendered" -echo "rendered $rendered" +# --------------------------------------------------------------- 1. the keystone rules +# Every covered project/environment, read back from the COVERS marker each rendered file +# carries rather than from a list someone has to remember to update, plus the pair being +# bootstrapped now. The pair matters, not just the project: a project bootstrapped for +# staging that also ships prod would otherwise read as covered while prod has no +# keystone rule at all — the same silent gap one level further in. +pairs="$({ sed -n 's/^# COVERS: //p' "$out_dir"/project-*.yaml 2>/dev/null || true + echo "$project $env_name"; } | sort -u)" + +# ALL pairs are re-rendered, not just the invoked one, so a template fix propagates to +# every project on the next bootstrap run instead of waiting for a per-project re-run. +while read -r p e; do + rendered="${out_dir}/project-${p}-${e}.yaml" + sed -e "s/__PROJECT__/${p}/g" \ + -e "s/__ENV__/${e}/g" \ + -e "s/__UID__/proj-silent-${p}-${e}/g" \ + templates/alerting/project.yaml.tmpl > "$rendered" + echo "rendered $rendered" +done <<<"$pairs" # ------------------------------------------------------------ 2. the coverage backstop -# Every project/environment with a rendered rule file, read back from the COVERS marker -# each one carries rather than from a list someone has to remember to update. The pair -# matters, not just the project: a project bootstrapped for staging that also ships prod -# would otherwise read as covered while prod has no keystone rule at all — the same -# silent gap one level further in. -pairs="$(sed -n 's/^# COVERS: //p' "$out_dir"/project-*.yaml | sort -u)" -: "${pairs:=__none__ __none__}" covered="$(echo "$pairs" | sed 's| |/|' | paste -sd',' - | sed 's/,/, /g')" covered_expr="$(echo "$pairs" \ | sed 's|^\([^ ]*\) \([^ ]*\)$|{project="\1",env="\2"}|' \ | paste -sd'|' - | sed 's/|/ or /g')" -cat > "${out_dir}/coverage.yaml" < "${out_dir}/coverage.yaml" echo "rendered ${out_dir}/coverage.yaml (covering: ${covered})" # ------------------------------------------------------------------ 3. reload Grafana if docker compose ps --status running --format '{{.Service}}' 2>/dev/null | grep -qx grafana; then - # Provisioned files are re-read on SIGHUP; no restart, no dropped evaluations. # A restart, not SIGHUP: Grafana re-reads alert provisioning only at startup, and # a SIGHUP reports success while changing nothing. docker compose up -d --force-recreate grafana >/dev/null 2>&1 \ @@ -152,7 +103,8 @@ fi hc_note="create these by hand at https://healthchecks.io and note their ping URLs" if [[ -n "${HEALTHCHECKS_API_KEY:-}" ]]; then hc_note="created via API" - for job in backup watchdog restore-check; do + # Default job set; override per project with HC_JOBS="backup nightly-sync" etc. + for job in ${HC_JOBS:-backup watchdog restore-check}; do curl -fsS -X POST https://healthchecks.io/api/v3/checks/ \ -H "X-Api-Key: ${HEALTHCHECKS_API_KEY}" \ -H "Content-Type: application/json" \ diff --git a/compose.demo.yml b/compose.demo.yml index 8c09c9d..7db589d 100644 --- a/compose.demo.yml +++ b/compose.demo.yml @@ -1,6 +1,7 @@ # Demo overlay: a tiny auto-instrumented FastAPI service plus a load # generator, so the stack has real traces, metrics, and correlated logs -# to show. Not for production — no restart-across-reboots on purpose. +# to show. Not for production: restart policies are omitted, so it does not +# survive a reboot. # # just demo # core stack + this overlay # just demo-down # remove just the demo services diff --git a/config/grafana/alerting/coverage.yaml b/config/grafana/alerting/coverage.yaml index 31096ef..84a24d6 100644 --- a/config/grafana/alerting/coverage.yaml +++ b/config/grafana/alerting/coverage.yaml @@ -1,4 +1,5 @@ -# GENERATED by bootstrap.sh — do not edit. Regenerated on every run. +# GENERATED by bootstrap.sh from templates/alerting/coverage.yaml.tmpl — do not edit +# the rendered file. Regenerated on every run. # # The trap this closes: ProjectTelemetrySilent only exists for projects bootstrap.sh was # actually run for. A project that ships telemetry but skipped bootstrap is silently diff --git a/config/grafana/alerting/rules.yaml b/config/grafana/alerting/rules.yaml index 3a3294c..05afdff 100644 --- a/config/grafana/alerting/rules.yaml +++ b/config/grafana/alerting/rules.yaml @@ -37,7 +37,10 @@ groups: instant: true editorMode: code expr: vector(1) - - refId: FIRING + # Identical on every rule (see the shape note up top): defined once + # here, aliased as `- *firing` below. + - &firing + refId: FIRING datasourceUid: __expr__ model: refId: FIRING @@ -72,16 +75,7 @@ groups: instant: true editorMode: code expr: rate(grafana_alerting_notifications_failed_total[10m]) > 0 - - refId: FIRING - datasourceUid: __expr__ - model: - refId: FIRING - type: threshold - expression: QUERY - conditions: - - evaluator: - type: gt - params: [0] + - *firing labels: severity: critical annotations: @@ -111,16 +105,7 @@ groups: instant: true editorMode: code expr: up == 0 - - refId: FIRING - datasourceUid: __expr__ - model: - refId: FIRING - type: threshold - expression: QUERY - conditions: - - evaluator: - type: gt - params: [0] + - *firing labels: severity: critical annotations: @@ -147,16 +132,7 @@ groups: sum by (exporter) (rate(otelcol_exporter_send_failed_spans[5m])) > 0 or sum by (exporter) (rate(otelcol_exporter_send_failed_metric_points[5m])) > 0 or sum by (exporter) (rate(otelcol_exporter_send_failed_log_records[5m])) > 0 - - refId: FIRING - datasourceUid: __expr__ - model: - refId: FIRING - type: threshold - expression: QUERY - conditions: - - evaluator: - type: gt - params: [0] + - *firing labels: severity: warning annotations: @@ -188,22 +164,13 @@ groups: expr: > (node_filesystem_avail_bytes{fstype!~"tmpfs|ramfs|overlay"} / node_filesystem_size_bytes{fstype!~"tmpfs|ramfs|overlay"}) < 0.2 - - refId: FIRING - datasourceUid: __expr__ - model: - refId: FIRING - type: threshold - expression: QUERY - conditions: - - evaluator: - type: gt - params: [0] + - *firing labels: severity: warning annotations: summary: "Filesystem {{ $labels.mountpoint }} is over 80% full" description: "Loki and Tempo have no total-size cap, so this alert is the storage backstop. Free space or lower retention (see RUNBOOK)." - # The 668-restart incident: a crash loop must get louder, not quieter. + # A crash loop must get louder, not quieter. - orgId: 1 name: container-lifecycle folder: Stack alerts @@ -227,16 +194,7 @@ groups: instant: true editorMode: code expr: changes(container_start_time_seconds{name!=""}[1h]) > 3 - - refId: FIRING - datasourceUid: __expr__ - model: - refId: FIRING - type: threshold - expression: QUERY - conditions: - - evaluator: - type: gt - params: [0] + - *firing labels: severity: warning annotations: @@ -260,16 +218,7 @@ groups: instant: true editorMode: code expr: increase(container_oom_events_total[5m]) > 0 - - refId: FIRING - datasourceUid: __expr__ - model: - refId: FIRING - type: threshold - expression: QUERY - conditions: - - evaluator: - type: gt - params: [0] + - *firing labels: severity: warning annotations: @@ -301,16 +250,7 @@ groups: expr: > sum by (job) (rate(http_server_request_duration_seconds_count{http_response_status_code=~"5.."}[5m])) / sum by (job) (rate(http_server_request_duration_seconds_count[5m])) > 0.05 - - refId: FIRING - datasourceUid: __expr__ - model: - refId: FIRING - type: threshold - expression: QUERY - conditions: - - evaluator: - type: gt - params: [0] + - *firing labels: severity: warning annotations: diff --git a/config/grafana/datasources.yaml b/config/grafana/datasources.yaml index 565dde1..4feaca7 100644 --- a/config/grafana/datasources.yaml +++ b/config/grafana/datasources.yaml @@ -44,8 +44,7 @@ datasources: tags: - key: service.name value: service_name - # No tracesToMetrics/serviceMap/nodeGraph: all three read the series Tempo's - # metrics-generator used to write, and that generator is gone (ADR 0002 — RED - # comes from the apps' own OTLP metrics). Leaving them configured would render - # a permanently empty service graph, which reads as "no traffic" rather than - # "not wired". + # No tracesToMetrics/serviceMap/nodeGraph: all three need span-metrics series + # this stack does not generate (RED comes from the apps' own OTLP metrics, + # ADR 0002). Configured anyway, they render a permanently empty service graph + # that reads as "no traffic". diff --git a/config/loki.yaml b/config/loki.yaml index 7d07918..0cc65fa 100644 --- a/config/loki.yaml +++ b/config/loki.yaml @@ -39,7 +39,8 @@ limits_config: # Keep ingestion generous for a single-tenant homelab. ingestion_rate_mb: 16 ingestion_burst_size_mb: 32 - # Index only service.name. Loki's default OTLP mapping also indexes + # Index only the identity labels listed below — the authoritative set; the + # docs point here. Loki's default OTLP mapping also indexes # service.instance.id, which mints a fresh stream on every sender restart; # it is now structured metadata, so query it with `| service_instance_id=...`. otlp_config: diff --git a/config/otel-collector.yaml b/config/otel-collector.yaml index a797daf..154a70c 100644 --- a/config/otel-collector.yaml +++ b/config/otel-collector.yaml @@ -38,46 +38,34 @@ processors: timeout: 5s send_batch_size: 1024 -# The queue/retry settings repeated on each exporter below are pinned rather -# than left to upstream defaults, because the RUNBOOK's "buffers about five -# minutes" promise is exactly those numbers. (No YAML anchor: the collector -# rejects unknown top-level keys, so there is nowhere to hang one.) The queue -# is file-backed, so both a backend outage and a collector restart are ridden out. +# The queue/retry settings are pinned rather than left to upstream defaults, +# because the RUNBOOK's "buffers about five minutes" promise is exactly those +# numbers. Defined once on the loki exporter, aliased onto the other two. The +# queue is file-backed, so both a backend outage and a collector restart are +# ridden out. exporters: # Loki 3.x accepts OTLP natively — no Promtail / loki exporter needed. otlp_http/loki: endpoint: http://loki:3100/otlp - sending_queue: - enabled: true + sending_queue: &queue queue_size: 1000 storage: file_storage - retry_on_failure: - enabled: true + retry_on_failure: &retry max_elapsed_time: 300s otlp_grpc/tempo: endpoint: tempo:4317 tls: insecure: true - sending_queue: - enabled: true - queue_size: 1000 - storage: file_storage - retry_on_failure: - enabled: true - max_elapsed_time: 300s + sending_queue: *queue + retry_on_failure: *retry # Prometheus native OTLP receiver (needs --web.enable-otlp-receiver, set in # compose.yml; it is a real flag, not an --enable-feature entry). otlp_http/prometheus: endpoint: http://prometheus:9090/api/v1/otlp - sending_queue: - enabled: true - queue_size: 1000 - storage: file_storage - retry_on_failure: - enabled: true - max_elapsed_time: 300s + sending_queue: *queue + retry_on_failure: *retry service: extensions: [bearertokenauth, file_storage] diff --git a/config/tempo.yaml b/config/tempo.yaml index 75c1544..bbdac09 100644 --- a/config/tempo.yaml +++ b/config/tempo.yaml @@ -1,8 +1,6 @@ # Tempo 3.x single-binary. Traces arrive only from the collector # (collector → tempo:4317) so external apps never talk to Tempo directly. -# Migrated from 2.x: the ingester/compactor blocks and the local-blocks -# processor are gone (handled internally by the live-store/backend worker); -# block retention now lives under overrides. +# Block retention lives under overrides. server: http_listen_port: 3200 diff --git a/dashboards/logs.json b/dashboards/logs.json index f8d2c4a..cebc35b 100644 --- a/dashboards/logs.json +++ b/dashboards/logs.json @@ -23,8 +23,8 @@ "type": "loki", "uid": "loki" }, - "definition": "label_values({project=~\".+\"}, project)", - "query": "label_values({project=~\".+\"}, project)", + "definition": "label_values(project)", + "query": "label_values(project)", "refresh": 2, "sort": 1, "current": {}, diff --git a/dashboards/service-health.json b/dashboards/service-health.json index dfd3606..5d59f49 100644 --- a/dashboards/service-health.json +++ b/dashboards/service-health.json @@ -254,7 +254,7 @@ }, "targets": [ { - "expr": "{project=\"$project\", env=\"$env\"}", + "expr": "{project=\"$project\", env=\"$env\", service_name=~\"$service\"}", "refId": "A" } ], diff --git a/docs/ONBOARDING.md b/docs/ONBOARDING.md index 78dcd8c..0885ccb 100644 --- a/docs/ONBOARDING.md +++ b/docs/ONBOARDING.md @@ -28,18 +28,20 @@ directly. - **Keep labels low-cardinality.** Prometheus turns every distinct label value into a series. User IDs, request IDs, and timestamps therefore don't belong in resource attributes or metric labels. Put them in the log - line or in span attributes instead — you can still filter on them at - query time, without the storage blowing up. -- **In Loki, `service.name` is the only index label.** Every other - attribute, `service.instance.id` included, is stored as structured - metadata, so a query starts from the stream selector and filters after - it: `{service_name="my-service"} | env="prod"`. Streams written before - this change keep their old labels until they age out after 30 days. - -Traces are the most valuable signal to send. Tempo derives request-rate, -error-rate, and duration ("RED") metrics from them, so a service that -sends only traces already gets the Service Health dashboard and error -alerting. Start with traces; everything else is a bonus. + line or in span attributes instead; they stay filterable at query time + without the storage blowing up. +- **In Loki, only the identity labels are index labels**: `service.name`, + `project`, `env`, `host.name` (the authoritative list lives in + `config/loki.yaml`). Every other attribute, `service.instance.id` + included, is stored as structured metadata, so a query starts from the + stream selector and filters after it: + `{service_name="my-service", env="prod"} | service_instance_id="..."`. + +The Service Health dashboard and the error-rate alert key on the standard +HTTP server metrics (`http_server_request_duration_seconds`), which the +auto-instrumentation below emits out of the box. Traces add per-request +drill-down on top. Start with the templates below and you get all three +signals at once. ## Template 1 — Python/FastAPI, zero code changes @@ -84,9 +86,3 @@ and token as Templates 1 and 2: no second hostname, no second credential. Run `./bootstrap.sh ` on the monitoring host and follow what it prints. See [templates/README.md](../templates/README.md). - -> Earlier revisions of this document carried two templates that pushed straight to Loki -> (the Docker `loki` log driver, and Alloy's `loki.write`). Both required exposing Loki, -> which this stack deliberately does not do because Loki has no authentication of its -> own, and undoing it later is more work than not starting. If you find those -> instructions in an old copy, they are wrong — use the agent above. diff --git a/templates/README.md b/templates/README.md index 0616cb3..91b3ca2 100644 --- a/templates/README.md +++ b/templates/README.md @@ -11,7 +11,7 @@ agent config on every host is byte-identical and a fix here reaches all of them. | `compose.telemetry.gpu.yml` | Opt-in overlay: `nvidia_gpu_exporter`, discovered automatically | | `run_scheduled.sh` | Dead-man's-switch wrapper for scheduled jobs | -`alerting/project.yaml.tmpl` is not vendored — it stays here and is rendered by +`alerting/project.yaml.tmpl` is not vendored; it stays here and is rendered by `bootstrap.sh` into this stack's own alert rules. ## The whole checklist @@ -28,13 +28,13 @@ Under an hour, most of it waiting for the first scrape. ## Bootstrap creates the safety net, not the telemetry -A host can ship perfect telemetry and still be unmonitored, because the rule that -notices its *silence* lives on this stack, not on the host. That asymmetry is why -`bootstrap.sh` exists and why skipping it is dangerous rather than merely untidy. +A host can ship perfect telemetry and still be unmonitored: the rule that notices its +*silence* lives on this stack, not on the host. Skipping `bootstrap.sh` therefore +leaves a project uncovered, with no error anywhere. The backstop for skipping it anyway is `ProjectsUncovered`, regenerated on every bootstrap run: it fires on any series carrying a `project` label with no rendered rule -file. It is the keystone applied one level up — the detector for a missing detector. +file. It is the detector for a missing detector. ## Two things that silently produce nothing @@ -44,7 +44,7 @@ file. It is the keystone applied one level up — the detector for a missing det with no `name` label, and every container alert matches nothing. No error is logged. - **`OTEL_SEMCONV_STABILITY_OPT_IN=http` in an instrumented app.** Without it the SDK emits the legacy HTTP metric names, whose `http_target` label carries the raw request - path — unbounded series on any API with path parameters. The stable names use + path: unbounded series on any API with path parameters. The stable names use `http_route`, and the central Service Health dashboard queries those. ## GPU hosts @@ -53,15 +53,14 @@ Include `compose.telemetry.gpu.yml` as well and set `GPU_METRICS=1`. The agent c already discovers the exporter by its Compose service label, so nothing else changes: a GPU host is an ordinary host plus one overlay. -`nvidia_gpu_exporter` rather than dcgm-exporter on purpose — DCGM's advantage is its +The exporter is `nvidia_gpu_exporter`, not dcgm-exporter: DCGM's advantage is its `DCGM_FI_PROF_*` profiling fields, which NVIDIA document as datacentre-only. On a -consumer card they are simply absent, which removes the reason to prefer DCGM while -keeping its `SYS_ADMIN` requirement. +consumer card they are absent, and dcgm-exporter still requires `SYS_ADMIN`. Three GPU rules are worth adding per GPU host. They are not provisioned by `bootstrap.sh` yet: -- `nvidia_smi_gpu_recovery_action > 0` — the driver is asking for a reset. The single +- `nvidia_smi_gpu_recovery_action > 0`: the driver is asking for a reset. The single best GPU health signal. - a thermal/power throttle flag. - XID faults, with an explicit code allowlist rather than every code, since most XIDs @@ -75,14 +74,11 @@ XIDs are the GPU analogue of the crash-loop blind spot: a stuck kernel, an uncorrectable memory fault, or a card that has fallen off the bus are all invisible to utilisation graphs, and they are what silently kills a twelve-hour training run. -For dashboards, import [14574](https://grafana.com/grafana/dashboards/14574) (revised -2026-08-04) and its multi-GPU companion 25547. Do **not** use the canonical DCGM -dashboard 12239: last revised 2021, and its panels lean on profiling fields consumer -cards cannot produce. For per-container resources, -[15798](https://grafana.com/grafana/dashboards/15798-docker-monitoring/) (revised -2025-07-12), with 19792 as the Compose-aware second choice. Avoid 10619 and 893 — -high download counts, untouched since 2019 and Grafana 4 respectively. Download counts -measure inertia, not maintenance. +The dashboards for both are provisioned centrally on the monitoring stack: `dashboards/gpu.json` +(vendored from [14574](https://grafana.com/grafana/dashboards/14574); its multi-GPU +companion is 25547 if a host ever grows a second card) and +`dashboards/host-containers.json` for per-container resources. Nothing to import on +the project host. ## Removing a project @@ -111,7 +107,7 @@ listing the removed one as covered. ## Budgets to plan against - **healthchecks.io free tier is exactly 20 checks.** `bootstrap.sh` creates three per - project/environment, so that is the onboarding ceiling — about six environments. + project/environment, so that is the onboarding ceiling: about six environments. - **Prometheus's OTLP receiver is documented as "not an efficient way of ingesting samples"**, for "specific low-volume use cases". A few hundred series per host at 30s is exactly the case that sentence carves out. The trigger for revisiting is volume: a @@ -126,6 +122,6 @@ count(container_start_time_seconds{project="",name!=""}) # one per cont count({job=""}) # app's own SDK metrics ``` -If the first is zero after five minutes, `ProjectTelemetrySilent` will tell you anyway -— that is the point of it. Expect it ~20 minutes after the last sample: `absent()` -needs the series to go stale before it reports. +If the first is zero after five minutes, `ProjectTelemetrySilent` will tell you anyway. +Expect it ~20 minutes after the last sample: `absent()` needs the series to go stale +before it reports. diff --git a/templates/alerting/coverage.yaml.tmpl b/templates/alerting/coverage.yaml.tmpl new file mode 100644 index 0000000..73ea5e6 --- /dev/null +++ b/templates/alerting/coverage.yaml.tmpl @@ -0,0 +1,55 @@ +# GENERATED by bootstrap.sh from templates/alerting/coverage.yaml.tmpl — do not edit +# the rendered file. Regenerated on every run. +# +# The trap this closes: ProjectTelemetrySilent only exists for projects bootstrap.sh was +# actually run for. A project that ships telemetry but skipped bootstrap is silently +# uncovered — the exact failure class the keystone closes, one level up. This fires on +# any series whose project/env pair has no rendered rule file. +# +# The selector touches every series every project sends (~3k for one project). Cheap at +# this size, evaluated instantly every 5m; revisit if a project's active series reach +# five figures. +# +# Covered right now: __COVERED__ + +apiVersion: 1 + +groups: + - orgId: 1 + name: _coverage + folder: Stack alerts + interval: 5m + rules: + - uid: projects-uncovered + title: ProjectsUncovered + condition: FIRING + for: 15m + noDataState: OK + execErrState: Error + isPaused: false + data: + - refId: QUERY + datasourceUid: prometheus + relativeTimeRange: + from: 3600 + to: 0 + model: + refId: QUERY + instant: true + editorMode: code + expr: count by (project, env) ({project!="", env!=""} unless on (project, env) (__COVERED_EXPR__)) + - refId: FIRING + datasourceUid: __expr__ + model: + refId: FIRING + type: threshold + expression: QUERY + conditions: + - evaluator: + type: gt + params: [0] + labels: + severity: warning + annotations: + summary: "{{ $labels.project }}/{{ $labels.env }} is sending telemetry but has no alert rules" + description: "Telemetry is arriving for a project/environment bootstrap.sh was never run for, so nothing would notice if it went silent. Run ./bootstrap.sh {{ $labels.project }} {{ $labels.env }} on the monitoring host." diff --git a/templates/alloy/config.alloy b/templates/alloy/config.alloy index 1dede27..bca34f7 100644 --- a/templates/alloy/config.alloy +++ b/templates/alloy/config.alloy @@ -12,7 +12,7 @@ // Everything else in the stack — databases, caches, proxies, frontends, batch jobs — // only writes to stdout, so this agent picks those up and forwards them on the // same OTLP endpoint with the same token. Nothing new is published at the edge: the -// monitoring stack exposes `otlp.` and deliberately does not expose Loki, which has no +// monitoring stack exposes `otlp.` and does not expose Loki, which has no // authentication of its own. // The host's own name, read from its /etc/hostname through the rootfs mount rather than @@ -159,12 +159,11 @@ prometheus.exporter.unix "host" { // --------------------------------------------------------------------------- // Per-container resource metrics. This is what makes a crash loop visible: cAdvisor has // no restart counter, but container_start_time_seconds moving repeatedly is the same -// signal, and container_oom_events_total says why. The incident that produced these -// runbooks — 668 restarts over 19 hours, every monitor green — is detected by +// signal, and container_oom_events_total says why. The central stack alerts on // // changes(container_start_time_seconds{name!=""}[1h]) > 3 // -// which is a rule on the central stack, not here. This side only has to ship the series. +// so this side only has to ship the series. // Shipping them at all needs `cgroup: host` on this container: see the note in // compose.telemetry.yml, without which cAdvisor reports one root series and the // rule above matches nothing. diff --git a/templates/compose.telemetry.yml b/templates/compose.telemetry.yml index 3cce129..765ec87 100644 --- a/templates/compose.telemetry.yml +++ b/templates/compose.telemetry.yml @@ -12,15 +12,17 @@ # - per-container resources and lifecycle from cAdvisor, which is what makes a crash # loop visible centrally. # -# It replaces the older Loki Docker-driver overlay. That approach needed a Loki push -# hostname exposed to the internet, which the monitoring stack deliberately does not -# publish because Loki has no authentication of its own, and it wrote the push -# credential into every container's HostConfig where `docker inspect` could read it. -# # The root deploy recipes auto-include this overlay when OTEL_EXPORTER_OTLP_ENDPOINT is # set in the host's root `.env` — the same switch that turns on the API's own exporter. # Hosts without it ship nothing and keep `docker logs` as the only log path. +# Local json-file logging with rotation for the agent stack itself. +x-logging: &default-logging + driver: json-file + options: + max-size: "10m" + max-file: "3" + services: # Least-privilege gate between Alloy and the Docker API. Alloy never sees the socket: # it talks HTTP to this proxy, which allows exactly the read-only endpoints discovery, @@ -48,11 +50,7 @@ services: pids_limit: 128 networks: - docker_socket - logging: - driver: json-file - options: - max-size: "10m" - max-file: "3" + logging: *default-logging alloy: image: grafana/alloy:v1.18.1@sha256:0f4434c92b3e6cdac38bb129b344e1790c246f7b6e2eaffcc16a5fa363240e33 @@ -141,13 +139,9 @@ services: networks: - egress - docker_socket - logging: - # Never through itself: Alloy reporting its own failure to ship logs, by shipping - # a log, is a loop with no exit. - driver: json-file - options: - max-size: "10m" - max-file: "3" + # Never through itself: Alloy reporting its own failure to ship logs, by shipping + # a log, is a loop with no exit. + logging: *default-logging volumes: alloy_data: diff --git a/templates/run_scheduled.sh b/templates/run_scheduled.sh index 38ec8b4..bdc41e3 100755 --- a/templates/run_scheduled.sh +++ b/templates/run_scheduled.sh @@ -59,6 +59,15 @@ ping_url="${!url_var:-}" output_file="$(mktemp)" trap 'rm -f "$output_file"' EXIT +# Send the job's own output as the failure body: the alert then carries the reason, +# instead of only saying that something went wrong. Note this puts job output — +# hostnames, paths, restic summaries — in a third party's hands; it is why the ping +# carries no credentials and why the URL itself is the only secret. +ping_fail() { + curl -fsS -m 10 --retry 3 --data-binary "@${output_file}" "${ping_url}/fail" -o /dev/null \ + || echo "WARNING: failure ping to ${url_var} failed" >&2 +} + # A killed job must still report. systemd's TimeoutStartSec TERMs the whole cgroup: # the job dies, and without this trap bash would die too — before the ping block — # so a HUNG job would send neither success nor failure and its captured output would @@ -70,8 +79,7 @@ on_terminate() { echo "run_scheduled: received SIG${sig}; job killed (likely a systemd timeout)" >>"$output_file" cat "$output_file" if [[ -n "$ping_url" ]]; then - curl -fsS -m 10 --retry 3 --data-binary "@${output_file}" "${ping_url}/fail" -o /dev/null \ - || echo "WARNING: failure ping to ${url_var} failed" >&2 + ping_fail fi rm -f "$output_file" exit 143 @@ -92,12 +100,7 @@ if [[ "$status" -eq 0 ]]; then curl -fsS -m 10 --retry 3 "$ping_url" -o /dev/null \ || echo "WARNING: success ping to ${url_var} failed" >&2 else - # Send the job's own output as the failure body: the alert then carries the reason, - # instead of only saying that something went wrong. Note this puts job output — - # hostnames, paths, restic summaries — in a third party's hands; it is why the ping - # carries no credentials and why the URL itself is the only secret. - curl -fsS -m 10 --retry 3 --data-binary "@${output_file}" "${ping_url}/fail" -o /dev/null \ - || echo "WARNING: failure ping to ${url_var} failed" >&2 + ping_fail fi exit "$status" From 425d1b20245fc9652fe93960fc648ea601fdb0bd Mon Sep 17 00:00:00 2001 From: Simon van Lierde Date: Tue, 1 Sep 2026 13:34:26 +0000 Subject: [PATCH 27/85] docs: update comments and docs --- CHANGELOG.md | 17 ++-- README.md | 87 +++++++++-------- config/grafana/alerting/rules.yaml | 20 ++-- dashboards/host-containers.json | 4 +- docs/ONBOARDING.md | 32 +++---- docs/RUNBOOK.md | 144 ++++++++++++++--------------- templates/README.md | 27 +++--- templates/alloy/config.alloy | 102 ++++++++++---------- 8 files changed, 205 insertions(+), 228 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 59a124c..8d48c43 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,8 +7,8 @@ Notable changes to this stack. Format follows ## [Unreleased] A hardening pass over the whole stack: buffering that survives a restart, -self-monitoring that covers every service, tighter container defaults, and a -handful of queries that had been quietly measuring the wrong thing. +self-monitoring that covers every service, tighter container defaults, and +queries that had been measuring the wrong thing. ### Added @@ -24,9 +24,8 @@ handful of queries that had been quietly measuring the wrong thing. - **Memory ceilings** (`mem_limit`) on every service, sized from observed usage, so one runaway component cannot OOM the host. - **Wider validation**: `just check` also verifies the Loki and Tempo configs - and OpenTofu formatting; `just smoke` asserts every - dashboard actually provisioned; CI additionally runs `just infra-validate` - and a new `just demo-build`. + and OpenTofu formatting; `just smoke` asserts every dashboard provisioned; + CI also runs `just infra-validate` and a new `just demo-build`. - Dependabot now watches the Cloudflare provider in `infra/`, and the runbook covers OpenTofu-managed tunnel and Access changes. @@ -68,7 +67,8 @@ handful of queries that had been quietly measuring the wrong thing. so clicking a dot resolved to nothing. - **`HighErrorRate` measured the wrong denominator**: it counted all spans, so internal child spans diluted the ratio well below the real request error - rate. It now filters server spans, matching the Service Health dashboard. + rate. It now reads the applications' HTTP server metrics, matching the + Service Health dashboard. - The demo load generator hit a never-failing endpoint half the time, so the advertised one-in-ten error rate showed up as one in twenty. - The Infrastructure Logs dashboard queried `env` and `service` labels this @@ -82,9 +82,8 @@ handful of queries that had been quietly measuring the wrong thing. - `no-new-privileges` on every service; the demo image runs as `nobody`. - `GRAFANA_COOKIE_SECURE` marks the session cookie Secure (with strict - SameSite), and `just up` with the tunnel overlay refuses to expose the - stack without it, a - non-localhost `GRAFANA_ROOT_URL`, and non-default credentials. + SameSite), and `just up` with the tunnel overlay refuses to expose the stack + without it, a non-localhost `GRAFANA_ROOT_URL`, and non-default credentials. - GitHub Actions are pinned to commit SHAs, and `just infra-validate` runs against a copy of the sources so state and tfvars never enter the container. diff --git a/README.md b/README.md index 7d77c00..bcebe30 100644 --- a/README.md +++ b/README.md @@ -6,10 +6,10 @@ [![License: MIT](https://img.shields.io/badge/license-MIT-blue)](LICENSE) Central monitoring for CML's research software. One host runs Grafana, Loki, -Tempo, Prometheus, and an OpenTelemetry Collector, wired together so logs, -traces, and metrics cross-reference each other. Projects send their telemetry -here over OTLP (the OpenTelemetry protocol) and it lands in one place, -queryable side by side. +Tempo, Prometheus, and an OpenTelemetry Collector, wired so logs, traces, and +metrics cross-reference each other. Projects send telemetry here over OTLP +(the OpenTelemetry protocol) and it lands in one place, queryable side by +side. ## Try it in one command @@ -23,17 +23,16 @@ just demo ``` This starts the full stack plus a small FastAPI service under constant -artificial load (`compose.demo.yml`). The service is instrumented with -OpenTelemetry auto-instrumentation and fails about one request in ten, on -purpose. Give it a minute, then open Grafana at -(admin / change-me) and look at: +artificial load (`compose.demo.yml`). It uses OpenTelemetry +auto-instrumentation and fails about one request in ten, on purpose. Give it +a minute, then open Grafana at (admin / change-me): - **Dashboards → Service Health (RED)** — request rate, error rate, and latency. The dots on the latency panel are exemplars: click one and Grafana opens the exact trace behind that measurement. - **Dashboards → Logs Overview** — log volume by service and level, an error feed, and a live tail of everything arriving over OTLP. -- **Alerting → Alert rules** — the stack-health and error-rate rules Prometheus +- **Alerting → Alert rules** — the stack-health and error-rate rules Grafana is evaluating. `HighErrorRate` trips on the demo service after five minutes: one request in ten failing is twice the 5% threshold. @@ -44,9 +43,9 @@ running. ## How it works -Everything enters through a single gateway, the OpenTelemetry Collector. A -project only ever configures one endpoint, and a storage backend can be -swapped later without touching any application. The Service Health dashboard +Everything enters through one gateway, the OpenTelemetry Collector. A project +configures a single endpoint, and a storage backend can be swapped later +without touching any application. The Service Health dashboard and the error-rate alert read the standard HTTP metrics that OpenTelemetry auto-instrumentation emits; traces add per-request drill-down on top. @@ -67,15 +66,15 @@ flowchart LR ``` Solid arrows show telemetry being written; dotted arrows show Grafana reading -at query time. Locally (`just up` or `just demo`) there is no tunnel involved: +at query time. Locally (`just up` or `just demo`) there is no tunnel: everything talks over the compose network and Grafana is at `localhost:3000`. The stack runs on a single host; at CML's telemetry volume, distributed ingestion would add operational weight for no gain ([ADR 0001](docs/adr/0001-observability-stack.md) records the alternatives). The hub-and-spoke design for serving multiple CML projects is -[ADR 0002](docs/adr/0002-hub-and-spoke-observability.md), and onboarding a -project onto it is [templates/README.md](templates/README.md). +[ADR 0002](docs/adr/0002-hub-and-spoke-observability.md); onboarding a project +onto it is [templates/README.md](templates/README.md). ## Run it for real @@ -90,32 +89,31 @@ Grafana: (admin / whatever you set). `COMPOSE_FILE` in `.env` names the overlays a host runs. Set `COMPOSE_FILE=compose.yml:compose.tunnel.yml` in the production `.env`, and every recipe (`up`, `logs`, `ps`, `backup`) acts on that same set. With the -tunnel overlay active, `just up` refuses to run until the settings that only -matter once the stack is reachable are real: a generated `OTLP_AUTH_TOKEN`, a -changed `GRAFANA_ADMIN_PASSWORD`, `GRAFANA_ROOT_URL` pointing at the tunnel -hostname, and `GRAFANA_COOKIE_SECURE=true` so the session cookie is marked -Secure. An empty `HEARTBEAT_URL` only warns. +tunnel overlay active, `just up` refuses to run until the settings that matter +once the stack is reachable are real: a generated `OTLP_AUTH_TOKEN`, a changed +`GRAFANA_ADMIN_PASSWORD`, `GRAFANA_ROOT_URL` pointing at the tunnel hostname, +and `GRAFANA_COOKIE_SECURE=true`. An empty `HEARTBEAT_URL` only warns. In production the stack sits behind a Cloudflare Tunnel, and that edge is code too. The tunnel, its hostnames, DNS, and the Cloudflare Access rule that puts -an email one-time-PIN in front of Grafana all live in `infra/` as a small -OpenTofu configuration. Applying it produces the `CLOUDFLARE_TUNNEL_TOKEN` that -the tunnel overlay needs; bootstrap steps are at the top of +an email one-time-PIN in front of Grafana live in `infra/` as a small OpenTofu +configuration. Applying it produces the `CLOUDFLARE_TUNNEL_TOKEN` the tunnel +overlay needs; bootstrap steps are at the top of [infra/main.tf](infra/main.tf). `just check` validates compose files, Prometheus config, the collector, Loki and Tempo configs, YAML, workflows, OpenTofu formatting, and dashboard JSON. -Every validator runs in a pinned container, so nothing needs to be installed on -the host. Grafana's alerting provisioning has no offline validator, so `just -smoke` covers it instead: it boots the stack, waits for Grafana to come up -healthy, and checks that every dashboard and every alert rule provisioned. CI -runs both on every push and pull request. +Every validator runs in a pinned container, so nothing is installed on the +host. Grafana's alerting provisioning has no offline validator, so `just smoke` +covers it: it boots the stack, waits for Grafana to report healthy, and checks +that every dashboard and every alert rule provisioned. CI runs both on every +push and pull request. ## Sending telemetry from a project -You need three things: the OTLP endpoint, the bearer token (`OTLP_AUTH_TOKEN`), -and a few naming conventions. Copy-paste templates for the two application -routes — zero-code Python/FastAPI, and plain OTLP environment variables — are in +You need the OTLP endpoint, the bearer token (`OTLP_AUTH_TOKEN`), and a few +naming conventions. Copy-paste templates for the two application routes — +zero-code Python/FastAPI, and plain OTLP environment variables — are in **[docs/ONBOARDING.md](docs/ONBOARDING.md)**. Everything an application cannot report about itself comes from the vendored agent in **[templates/README.md](templates/README.md)**. @@ -136,21 +134,20 @@ told". One rule, `Watchdog`, fires permanently by design and posts to `HEARTBEAT_URL` every five minutes. Point that at a dead man's switch such as healthchecks.io — -a service that alerts when the pings *stop* — and you will also hear about the -one failure the host cannot report itself: its own death. Set both: an unset +a service that alerts when the pings *stop* — to hear about the one failure the +host cannot report itself: its own death. Set both. An unset `ALERT_WEBHOOK_URL` drops every alert while the heartbeat keeps reporting healthy, so `just up` with the tunnel overlay refuses to start without it. ## Storage Everything persists to local Docker volumes (`loki_data`, `tempo_data`, -`prometheus_data`, `grafana_data`), all of which `just -backup` captures. A fifth, `otel_queue`, holds the collector's on-disk export -queue: seconds of in-flight telemetry, worthless by the time anyone restores, -so backups skip it. When local disk stops fitting, Loki and -Tempo can move to any S3-compatible object store (Backblaze B2, Cloudflare R2, -Hetzner, MinIO); `compose.storage-s3.yml` documents the concrete shape of that -change. +`prometheus_data`, `grafana_data`), all captured by `just backup`. A fifth, +`otel_queue`, holds the collector's on-disk export queue: seconds of in-flight +telemetry, worthless by the time anyone restores, so backups skip it. When +local disk stops fitting, Loki and Tempo can move to any S3-compatible object +store (Backblaze B2, Cloudflare R2, Hetzner, MinIO); `compose.storage-s3.yml` +documents that change. ## Layout @@ -172,11 +169,11 @@ infra/ # OpenTofu: Cloudflare tunnel, ingress routes, DNS ``` `dashboards/*.json` is the source of truth for what Grafana shows: the -directory is mounted read-only and saving from the UI is disabled, so a change -made in the browser lasts until the page is reloaded. Edit the JSON and -provisioning picks it up within about 30 seconds; to keep something built -interactively, export the dashboard as JSON (or copy a single panel's JSON out -of *Inspect → Panel JSON*) and paste it back into the file. +directory is mounted read-only and UI saves are disabled, so a change made in +the browser lasts until the page reloads. Edit the JSON and provisioning picks +it up within about 30 seconds. To keep something built interactively, export +the dashboard as JSON (or copy one panel's JSON from *Inspect → Panel JSON*) +and paste it back into the file. ## Documentation diff --git a/config/grafana/alerting/rules.yaml b/config/grafana/alerting/rules.yaml index 05afdff..4869b24 100644 --- a/config/grafana/alerting/rules.yaml +++ b/config/grafana/alerting/rules.yaml @@ -2,8 +2,8 @@ # and Grafana rules can query Loki as well as Prometheus. # # Shape note, true for every rule here: the PromQL carries its own comparison, so it -# returns a series only when the alert should fire, and the threshold node just asks -# whether anything came back — no reduce/threshold pair to decode. +# returns a series only when the alert should fire, and the threshold node only asks +# whether anything came back. # # Keep the total under about ten. These are the generic rules; the per-project keystone # lives in project--.yaml beside this file, rendered by bootstrap.sh. @@ -37,9 +37,9 @@ groups: instant: true editorMode: code expr: vector(1) - # Identical on every rule (see the shape note up top): defined once - # here, aliased as `- *firing` below. - &firing + # Identical on every rule (see the shape note up top): defined once + # here, aliased as `- *firing` below. refId: FIRING datasourceUid: __expr__ model: @@ -56,7 +56,7 @@ groups: summary: "Alerting-pipeline heartbeat (always firing)" description: "Routed to a dead man's switch. Investigate if the pings stop." # Cannot page when the path it reports on is the broken one — that is what the - # heartbeat is for. Its job is to make the state visible instead of silent. + # heartbeat is for. It makes the state visible instead of silent. - uid: alert-delivery-failing title: AlertDeliveryFailing condition: FIRING @@ -129,9 +129,7 @@ groups: instant: true editorMode: code expr: > - sum by (exporter) (rate(otelcol_exporter_send_failed_spans[5m])) > 0 - or sum by (exporter) (rate(otelcol_exporter_send_failed_metric_points[5m])) > 0 - or sum by (exporter) (rate(otelcol_exporter_send_failed_log_records[5m])) > 0 + sum by (exporter) (rate(otelcol_exporter_send_failed_spans[5m])) > 0 or sum by (exporter) (rate(otelcol_exporter_send_failed_metric_points[5m])) > 0 or sum by (exporter) (rate(otelcol_exporter_send_failed_log_records[5m])) > 0 - *firing labels: severity: warning @@ -162,8 +160,7 @@ groups: instant: true editorMode: code expr: > - (node_filesystem_avail_bytes{fstype!~"tmpfs|ramfs|overlay"} - / node_filesystem_size_bytes{fstype!~"tmpfs|ramfs|overlay"}) < 0.2 + (node_filesystem_avail_bytes{fstype!~"tmpfs|ramfs|overlay"} / node_filesystem_size_bytes{fstype!~"tmpfs|ramfs|overlay"}) < 0.2 - *firing labels: severity: warning @@ -248,8 +245,7 @@ groups: instant: true editorMode: code expr: > - sum by (job) (rate(http_server_request_duration_seconds_count{http_response_status_code=~"5.."}[5m])) - / sum by (job) (rate(http_server_request_duration_seconds_count[5m])) > 0.05 + sum by (job) (rate(http_server_request_duration_seconds_count{http_response_status_code=~"5.."}[5m])) / sum by (job) (rate(http_server_request_duration_seconds_count[5m])) > 0.05 - *firing labels: severity: warning diff --git a/dashboards/host-containers.json b/dashboards/host-containers.json index 988ca0b..e065442 100644 --- a/dashboards/host-containers.json +++ b/dashboards/host-containers.json @@ -13,7 +13,7 @@ "refresh": "30s", "schemaVersion": 39, "editable": null, - "description": "Per-host and per-container resources for any onboarded project, from the Alloy agent's node-exporter and cAdvisor. Container series exist only where the agent runs with cgroup: host \u2014 without it cAdvisor reports the root cgroup alone and every container panel here is empty.", + "description": "Per-host and per-container resources for any onboarded project, from the Alloy agent's node-exporter and cAdvisor. Container panels are empty unless the agent runs with cgroup: host \u2014 without it cAdvisor reports the root cgroup alone.", "templating": { "list": [ { @@ -247,7 +247,7 @@ { "type": "timeseries", "title": "Container Restarts (1h)", - "description": "ContainerRestarting fires above 3. This is the 668-restart incident's detector, drawn.", + "description": "ContainerRestarting fires above 3.", "gridPos": { "h": 8, "w": 12, diff --git a/docs/ONBOARDING.md b/docs/ONBOARDING.md index 0885ccb..1e087a7 100644 --- a/docs/ONBOARDING.md +++ b/docs/ONBOARDING.md @@ -1,10 +1,9 @@ # Sending telemetry from your project This stack collects logs, traces, and metrics from CML projects and shows -them side by side in one Grafana. Getting your project in takes an -endpoint, a token, and a couple of naming conventions. Then pick the -template that matches how your project runs — if it's a Python/FastAPI -service, Template 1 needs no code changes at all. +them side by side in one Grafana. You need an endpoint, a token, and a +couple of naming conventions. Then pick the template that matches how your +project runs; a Python/FastAPI service needs no code changes at all. ## The endpoint @@ -26,10 +25,10 @@ directly. (`relab-api`, not `relab-api-prod-2`). Dashboards key on it. - **`env`** is `prod`, `staging`, or `dev`, set as a resource attribute. - **Keep labels low-cardinality.** Prometheus turns every distinct label - value into a series. User IDs, request IDs, and timestamps therefore - don't belong in resource attributes or metric labels. Put them in the log - line or in span attributes instead; they stay filterable at query time - without the storage blowing up. + value into a series, so user IDs, request IDs, and timestamps do not + belong in resource attributes or metric labels. Put them in the log line + or in span attributes: still filterable at query time, without the storage + blowing up. - **In Loki, only the identity labels are index labels**: `service.name`, `project`, `env`, `host.name` (the authoritative list lives in `config/loki.yaml`). Every other attribute, `service.instance.id` @@ -40,8 +39,7 @@ directly. The Service Health dashboard and the error-rate alert key on the standard HTTP server metrics (`http_server_request_duration_seconds`), which the auto-instrumentation below emits out of the box. Traces add per-request -drill-down on top. Start with the templates below and you get all three -signals at once. +drill-down on top. ## Template 1 — Python/FastAPI, zero code changes @@ -61,9 +59,9 @@ export OTEL_SEMCONV_STABILITY_OPT_IN=http opentelemetry-instrument uvicorn app:app --host 0.0.0.0 --port 8000 ``` -That's the whole integration: traces, RED metrics, and logs that carry -their trace context, without a line of OTel code in the app. The working -example is this repo's own [`demo/`](../demo/) service plus +That is the whole integration: traces, RED metrics, and logs carrying their +trace context, with no OTel code in the app. A working example is this +repo's [`demo/`](../demo/) service plus [`compose.demo.yml`](../compose.demo.yml). ## Template 2 — any language, plain OTLP @@ -79,10 +77,10 @@ OTEL_RESOURCE_ATTRIBUTES=env=prod ## Container logs, host metrics, per-container metrics -Everything the application cannot report about itself — other containers' stdout, host -resources, container lifecycle — is shipped by one Grafana Alloy agent per host. It is -vendored from `templates/`, not written per project, and it rides the same OTLP endpoint -and token as Templates 1 and 2: no second hostname, no second credential. +One Grafana Alloy agent per host ships everything the application cannot report about +itself: other containers' stdout, host resources, container lifecycle. It is vendored +from `templates/`, not written per project, and rides the same OTLP endpoint and token +as Templates 1 and 2 — no second hostname, no second credential. Run `./bootstrap.sh ` on the monitoring host and follow what it prints. See [templates/README.md](../templates/README.md). diff --git a/docs/RUNBOOK.md b/docs/RUNBOOK.md index c92dc45..b18fab2 100644 --- a/docs/RUNBOOK.md +++ b/docs/RUNBOOK.md @@ -1,14 +1,13 @@ # Runbook -How to operate this stack. All commands run from the repo root on the -monitoring host. Start with `just ps` and the **Stack Health** dashboard; -between them they answer most "what is wrong" questions. +All commands run from the repo root on the monitoring host. Start with +`just ps` and the **Stack Health** dashboard; between them they answer most +"what is wrong" questions. ![Stack Health dashboard](img/stack-health.png) In that capture, the red export-failure spike is a Tempo outage and the -gap in the ingest panel is a backup/restore drill. Both procedures are -below. +gap in the ingest panel is a backup/restore drill. ## A service is down or misbehaving @@ -19,11 +18,11 @@ just restart ``` Two alerts point here. `TargetDown` fires after two minutes when -Prometheus can't scrape a target, and it scrapes every service in the -stack — collector, node-exporter, Grafana, Loki, Tempo, and -itself — so the alert names whichever one went quiet. -`OtelExportFailures` means the collector is up but a backend is rejecting -its data, so look at that backend's logs, not the collector's. +Prometheus cannot scrape a target. It scrapes every service — collector, +node-exporter, Grafana, Loki, Tempo, and itself — so the alert names +whichever one went quiet. `OtelExportFailures` means the collector is up but +a backend is rejecting its data: read that backend's logs, not the +collector's. ## Disk filling up (`HostDiskSpaceLow`) @@ -37,17 +36,17 @@ Retention is only partially size-bounded: | Tempo blocks | 7d | none | Loki and Tempo have no total-size knob, so the disk alert at 80% is the -real backstop. When it fires: check the Filesystem panel on Stack Health, -then either free space or shorten a retention window (`retention_period` -in `config/loki.yaml`, `block_retention` in `config/tempo.yaml`, the -`--storage.tsdb.retention.*` flags in `compose.yml`) and restart the -affected service. If disk pressure keeps coming back, the durable fix is -moving Loki and Tempo to object storage (see `compose.storage-s3.yml`). +backstop. When it fires, check the Filesystem panel on Stack Health. Then +free space, or shorten a retention window and restart the affected service +(`retention_period` in `config/loki.yaml`, `block_retention` in +`config/tempo.yaml`, the `--storage.tsdb.retention.*` flags in +`compose.yml`). If disk pressure keeps returning, move Loki and Tempo to +object storage (see `compose.storage-s3.yml`). Memory is bounded per service instead: every service carries a `mem_limit` -in `compose.yml`, sized from observed usage with headroom so that one -runaway component cannot take the host down with it. If a component -legitimately grows into its ceiling, raise it there. +in `compose.yml`, sized from observed usage so one runaway component cannot +take the host down. Raise it there if a component legitimately grows into +its ceiling. ## Backup and restore @@ -57,48 +56,47 @@ just restore backups/monitoring-.tar.gz # stops the stack, wipes volumes, just up ``` -Backups are crash-consistent: restoring one is equivalent to recovering -from a power loss, which every component in the stack does cleanly via its -write-ahead log. Two practical notes: +Backups are crash-consistent: restoring one is like recovering from a power +loss, which every component does cleanly via its write-ahead log. Notes: - The tarball is mode 0600 and contains secrets (the Grafana database among them). Copy it off-host over a private channel: a backup on the disk it protects is a decoration. -- The tarball covers the docker volumes and nothing else. The OpenTofu - state for the Cloudflare edge is not in it (see below). +- It covers the docker volumes and nothing else. The OpenTofu state for the + Cloudflare edge is not in it (see below). - During the pause the collector keeps accepting telemetry and buffers it for five minutes (`retry_on_failure.max_elapsed_time` on every exporter in `config/otel-collector.yaml`). The queue is file-backed, so restarting the collector inside that window keeps the buffer. A backup - that runs longer than five minutes still drops data, so on large - volumes run it at a quiet hour. + that runs longer than five minutes still drops data: on large volumes, run + it at a quiet hour. -How much history you can lose equals how often you run it. A daily cron on -the host is the intended setup. +Worst-case loss equals the interval between backups. A daily cron on the +host is the intended setup. ## Rotating secrets -- **OTLP token:** set the new `OTLP_AUTH_TOKEN` in `.env`, then +- **OTLP token:** set the new `OTLP_AUTH_TOKEN` in `.env`, run `docker compose up -d otel-collector`, then update every sender's `OTEL_EXPORTER_OTLP_HEADERS`. Senders still on the old token get 401s - (visible as export errors on their side) until they're updated. A - running demo overlay counts as a sender: re-run `just demo` to recreate - it with the new token. -- **Tunnel token:** the tunnel is OpenTofu-managed, so read the token back from there rather than - copying it out of the dashboard. Rotate the tunnel secret in Cloudflare Zero Trust, then - `cd infra && tofu apply` (which refreshes the token data source) and - `tofu output -raw tunnel_token`. To rotate entirely from code instead, - `tofu apply -replace=cloudflare_zero_trust_tunnel_cloudflared.monitoring` builds a new tunnel and - repoints both CNAMEs at it; ingestion and Grafana are unreachable for the minute or so that - takes. Either way: new token into `CLOUDFLARE_TUNNEL_TOKEN` in `.env`, then `just up`. + (export errors on their side) until updated. A running demo overlay counts + as a sender: re-run `just demo` to recreate it with the new token. +- **Tunnel token:** the tunnel is OpenTofu-managed, so read the token back + from there, not from the dashboard. Rotate the tunnel secret in Cloudflare + Zero Trust, then `cd infra && tofu apply` (which refreshes the token data + source) and `tofu output -raw tunnel_token`. To rotate entirely from code, + `tofu apply -replace=cloudflare_zero_trust_tunnel_cloudflared.monitoring` + builds a new tunnel and repoints both CNAMEs at it; ingestion and Grafana + are unreachable for the minute or so that takes. Either way: new token into + `CLOUDFLARE_TUNNEL_TOKEN` in `.env`, then `just up`. - **Grafana admin password:** change `GRAFANA_ADMIN_PASSWORD` in `.env`, then `docker compose up -d grafana`. ## Alert delivery -Grafana evaluates the rules and delivers them; there is no Alertmanager. Rules, +Grafana evaluates and delivers the rules; there is no Alertmanager. Rules, contact points and the routing tree are provisioned from -`config/grafana/alerting/`, so the UI shows them read-only; edit the YAML. +`config/grafana/alerting/`, so the UI shows them read-only: edit the YAML. Two environment variables control where notifications go: - `ALERT_WEBHOOK_URL` receives all alerts (any webhook: ntfy, Slack, …). @@ -107,16 +105,14 @@ Two environment variables control where notifications go: raises the alarm when pings **stop**: that is the "monitoring host is dead" signal nothing inside the host can send. -Leaving them empty is not a safe default. Delivery then fails silently while -the heartbeat keeps pinging, so the dead man's switch reads healthy and every -real alert is dropped. With the tunnel overlay active, `just up` refuses to -start without `ALERT_WEBHOOK_URL`, and `AlertDeliveryFailing` fires on a -failing notifier. +Leaving them empty is not a safe default: delivery fails silently while the +heartbeat keeps pinging, so the switch reads healthy and every real alert is +dropped. With the tunnel overlay active, `just up` refuses to start without +`ALERT_WEBHOOK_URL`, and `AlertDeliveryFailing` fires on a failing notifier. -After changing either variable, `docker compose up -d grafana`. A plain -`restart` keeps the old environment: compose only rebuilds a container's -environment on `up`, and the stale value survives with no indication that -it has. +After changing either variable, run `docker compose up -d grafana`. A plain +`restart` silently keeps the old environment: compose rebuilds a container's +environment only on `up`. ## Changing the Cloudflare edge @@ -135,39 +131,37 @@ cd infra && tofu apply `infra/terraform.tfvars` and apply. That list is the entire allowlist. A removed address keeps working until their Access session expires (24h), so for an urgent revocation also revoke the session in Zero Trust. At - least one address has to remain: the variable's validation rejects an - empty list, which would lock everyone out of Grafana. Check your plan's - Zero Trust seat limit in the Cloudflare dashboard before adding people; - Cloudflare's documentation does not state the free-plan figure. -- **Per-user Grafana logins:** by default everyone who clears Access then - shares the one admin password. Setting `GRAFANA_JWT_AUTH=true` and - `CF_ACCESS_TEAM_DOMAIN=` in `.env` makes Grafana verify the Access - JWT instead, so each address signs in as itself and new ones land on the - org's default role (Viewer). The JWK set is team-wide, so if the Zero - Trust team fronts more than one Access application, also pin - `GF_AUTH_JWT_EXPECT_CLAIMS` in `compose.yml` to this app's `aud` tag. - Otherwise a token minted for any other app in the team is accepted here - too. + least one address has to remain; the variable's validation rejects an empty + list, which would lock everyone out. Before adding people, check your plan's + Zero Trust seat limit in the Cloudflare dashboard — Cloudflare does not + document the free-plan figure. +- **Per-user Grafana logins:** by default everyone who clears Access shares + the one admin password. Set `GRAFANA_JWT_AUTH=true` and + `CF_ACCESS_TEAM_DOMAIN=` in `.env` to make Grafana verify the Access + JWT instead: each address signs in as itself, and new ones land on the org's + default role (Viewer). The JWK set is team-wide. If the Zero Trust team + fronts more than one Access application, also pin + `GF_AUTH_JWT_EXPECT_CLAIMS` in `compose.yml` to this app's `aud` tag, or a + token minted for any other app in the team is accepted here too. - **Adding a hostname:** add an `ingress` entry pointing at the service's container port, plus a matching `cloudflare_dns_record`. The catch-all `http_status:404` entry stays last, or it swallows everything after it. - **State lives on this host only.** `infra/terraform.tfstate` is gitignored and `just backup` does not touch it. Copy it off-host next to the backups. - Losing it orphans the Cloudflare resources: they keep running, but the - next apply tries to create duplicates and you get them back only by hand - with `tofu import`. + Losing it orphans the Cloudflare resources: they keep running, but the next + apply creates duplicates, and recovery is `tofu import` by hand. ## Upgrading images Dependabot opens PRs that bump the pinned versions, and CI runs `just check` on each one. The validators (promtool, otelcol) read their image versions from `compose.yml`, so every bump is checked with the exact -binaries the stack will run. When a new version changes its config -syntax, CI fails before the change reaches the host. +binaries the stack will run. A new version that changes its config syntax +fails CI before it reaches the host. Dependabot also watches the Cloudflare provider in `infra/`. Those PRs need one manual step: it bumps the constraint in `main.tf` but not the recorded -hashes in `.terraform.lock.hcl`, so check the branch out and run +hashes in `.terraform.lock.hcl`. Check the branch out, run `cd infra && tofu init -upgrade`, then `just infra-validate` and `tofu plan` against the real account. Validation proves the syntax parses; only a plan proves the provider still maps the config to the same resources. @@ -175,10 +169,10 @@ proves the provider still maps the config to the same resources. Nothing watches the tool images pinned in the `justfile`: yamllint, actionlint, OpenTofu, jq, and the alpine that backup, restore, and the queue-volume setup run in. No Dependabot ecosystem covers a justfile, so -those are bumped by hand. +bump those by hand. -After merging: on the host, `git pull && just pull && just up`. Bring the -stack up through `just up` rather than `docker compose up -d`: the recipe -first chowns the collector's queue volume to uid 10001, and on a host where -that volume is new a raw compose up leaves it root-owned and the collector -crash-looping on a queue directory it cannot write. +After merging, on the host: `git pull && just pull && just up`. Use `just +up`, not `docker compose up -d`: the recipe first chowns the collector's +queue volume to uid 10001. Where that volume is new, a raw compose up leaves +it root-owned and the collector crash-looping on a queue directory it cannot +write. diff --git a/templates/README.md b/templates/README.md index 91b3ca2..1e1bf16 100644 --- a/templates/README.md +++ b/templates/README.md @@ -29,12 +29,12 @@ Under an hour, most of it waiting for the first scrape. ## Bootstrap creates the safety net, not the telemetry A host can ship perfect telemetry and still be unmonitored: the rule that notices its -*silence* lives on this stack, not on the host. Skipping `bootstrap.sh` therefore -leaves a project uncovered, with no error anywhere. +*silence* lives on this stack, not on the host. Skipping `bootstrap.sh` leaves a +project uncovered, with no error anywhere. -The backstop for skipping it anyway is `ProjectsUncovered`, regenerated on every -bootstrap run: it fires on any series carrying a `project` label with no rendered rule -file. It is the detector for a missing detector. +The backstop is `ProjectsUncovered`, regenerated on every bootstrap run: it fires on +any series carrying a `project` label with no rendered rule file. It is the detector +for a missing detector. ## Two things that silently produce nothing @@ -72,13 +72,12 @@ Three GPU rules are worth adding per GPU host. They are not provisioned by XIDs are the GPU analogue of the crash-loop blind spot: a stuck kernel, an uncorrectable memory fault, or a card that has fallen off the bus are all invisible to -utilisation graphs, and they are what silently kills a twelve-hour training run. +utilisation graphs, and any of them silently kills a twelve-hour training run. -The dashboards for both are provisioned centrally on the monitoring stack: `dashboards/gpu.json` -(vendored from [14574](https://grafana.com/grafana/dashboards/14574); its multi-GPU -companion is 25547 if a host ever grows a second card) and -`dashboards/host-containers.json` for per-container resources. Nothing to import on -the project host. +Both dashboards are provisioned centrally: `dashboards/gpu.json` (vendored from +[14574](https://grafana.com/grafana/dashboards/14574); its multi-GPU companion is +25547 if a host ever grows a second card) and `dashboards/host-containers.json` for +per-container resources. Nothing to import on the project host. ## Removing a project @@ -110,9 +109,9 @@ listing the removed one as covered. project/environment, so that is the onboarding ceiling: about six environments. - **Prometheus's OTLP receiver is documented as "not an efficient way of ingesting samples"**, for "specific low-volume use cases". A few hundred series per host at 30s - is exactly the case that sentence carves out. The trigger for revisiting is volume: a - host past a few thousand active series should move infrastructure metrics to - `remote_write` behind an authenticating proxy, leaving app metrics on OTLP. + is exactly that case. Revisit on volume: a host past a few thousand active series + should move infrastructure metrics to `remote_write` behind an authenticating proxy, + leaving app metrics on OTLP. ## Verifying, from the monitoring host diff --git a/templates/alloy/config.alloy b/templates/alloy/config.alloy index bca34f7..3a2d963 100644 --- a/templates/alloy/config.alloy +++ b/templates/alloy/config.alloy @@ -1,23 +1,21 @@ // SHARED AGENT CONFIG — vendored from the central monitoring repo at a pinned tag. // Do not edit it per project: every deployment-specific value arrives as an environment // variable (PROJECT, ENVIRONMENT, COMPOSE_PROJECT_NAME, OTEL_EXPORTER_OTLP_ENDPOINT, -// OTLP_AUTH_TOKEN, TELEMETRY_EDGE_KEY). If you find yourself editing this file to -// onboard a project, that is a bug in the template — report it upstream instead, so the -// next project does not hit the same wall. +// OTLP_AUTH_TOKEN, TELEMETRY_EDGE_KEY). Needing to edit this file to onboard a project +// is a bug in the template — report it upstream. // -// Ship this host's telemetry to the department OpenTelemetry collector: container +// It ships this host's telemetry to the department OpenTelemetry collector: container // stdout/stderr as logs, and host resource metrics. // // An instrumented application sends its own traces and metrics over OTLP directly. -// Everything else in the stack — databases, caches, proxies, frontends, batch jobs — -// only writes to stdout, so this agent picks those up and forwards them on the -// same OTLP endpoint with the same token. Nothing new is published at the edge: the -// monitoring stack exposes `otlp.` and does not expose Loki, which has no -// authentication of its own. +// Everything else — databases, caches, proxies, frontends, batch jobs — only writes to +// stdout, so this agent picks those up and forwards them on the same OTLP endpoint with +// the same token. Nothing new is published at the edge: the monitoring stack exposes +// `otlp.` and does not expose Loki, which has no authentication of its own. // The host's own name, read from its /etc/hostname through the rootfs mount rather than -// taken as an operator input: one less thing to set per host, and one less thing to get -// wrong. The container's own hostname is a docker id and useless here. +// taken as an operator input: one less thing to set per host. The container's own +// hostname is a docker id and useless here. local.file "hostname" { filename = "/rootfs/etc/hostname" } @@ -29,10 +27,10 @@ local.file "hostname" { discovery.docker "containers" { host = "tcp://docker-socket-proxy:2375" - // The default is 1m, and the backup one-shot runs for 60-90s: at the default a whole - // backup can start and exit inside one poll window and never be discovered, which - // would lose exactly the logs most worth having. Polling the local socket for a - // dozen containers is cheap; a missed short-lived container is not. + // The default is 1m and the backup one-shot runs for 60-90s, so a whole backup can + // start and exit inside one poll window and never be discovered — losing exactly the + // logs most worth having. Polling the local socket for a dozen containers is cheap; + // a missed short-lived container is not. refresh_interval = "5s" } @@ -79,10 +77,10 @@ loki.source.docker "containers" { // Without this every host collapses onto one identity: Alloy scrapes its OWN // in-container exporter, so the address Prometheus derives `instance` from is - // identical on every machine. The moment a second host ships, the series - // interleave and every host-level alert becomes meaningless. The coalesce - // keeps a blank /etc/hostname from silently reintroducing exactly that: the - // container hostname is a poor identity, but it is a distinguishable one. + // identical on every machine. Once a second host ships, the series interleave + // and every host-level alert is meaningless. The coalesce keeps a blank + // /etc/hostname from reintroducing that: the container hostname is a poor + // identity, but a distinguishable one. host_name = coalesce(string.trim_space(local.file.hostname.content), constants.hostname), } forward_to = [otelcol.receiver.loki.containers.receiver] @@ -125,12 +123,11 @@ otelcol.processor.transform "resource_attributes" { // so they ride the same endpoint and token as the logs above — no second pipeline, no // second hostname, no second credential. // -// set_collectors is an allowlist rather than the default set: fewer series, and each one -// is here for a reason. `hwmon` earns its place as a dashboard panel — thermal data on a -// physical box is cheap and occasionally telling — but it is NOT a control: fan RPM is a -// symptom of load, legitimate load spins fans too, and no threshold on it would be -// trustworthy. The crash loop that made this memorable is properly detected by container -// restart count, not by temperature. +// set_collectors is an allowlist rather than the default set: fewer series, each one here +// for a reason. `hwmon` earns a dashboard panel — thermal data on a physical box is cheap +// and occasionally telling — but it is NOT a control: fan RPM is a symptom of load, +// legitimate load spins fans too, and no threshold on it would be trustworthy. A crash +// loop is detected by container restart count, not by temperature. prometheus.exporter.unix "host" { // The exporter runs inside the container, so point it at the host's mounted /proc, // /sys and /. @@ -163,10 +160,9 @@ prometheus.exporter.unix "host" { // // changes(container_start_time_seconds{name!=""}[1h]) > 3 // -// so this side only has to ship the series. -// Shipping them at all needs `cgroup: host` on this container: see the note in -// compose.telemetry.yml, without which cAdvisor reports one root series and the -// rule above matches nothing. +// so this side only has to ship the series. Shipping them at all needs `cgroup: host` +// on this container (see the note in compose.telemetry.yml); without it cAdvisor +// reports one root series and the rule above matches nothing. prometheus.exporter.cadvisor "containers" { docker_host = "tcp://docker-socket-proxy:2375" @@ -193,11 +189,11 @@ prometheus.scrape "cadvisor" { // cAdvisor walks the host's cgroup tree, so it sees EVERY container on the box. The // COMPOSE_PROJECT_NAME filter on discovery.docker above scopes log tailing only — it -// does not reach this exporter. On a host that also runs a dev stack, an unrelated -// tool, or the monitoring stack itself, those containers otherwise arrive stamped with -// THIS deployment's project and env, firing this project's alerts for something it does -// not own and double-reporting anything that ships its own telemetry. Keep only this -// project's containers, using the compose label the allowlist above preserves. +// does not reach this exporter. Without this rule, containers from a dev stack, an +// unrelated tool, or the monitoring stack itself arrive stamped with THIS deployment's +// project and env, firing this project's alerts for something it does not own and +// double-reporting anything that ships its own telemetry. So keep only this project's +// containers, using the compose label the allowlist above preserves. // // The unlabelled root-cgroup series drops out with them, which is correct: it carries no // `name`, so no container rule can match it, and node_exporter above already owns @@ -246,10 +242,10 @@ prometheus.relabel "host" { // --------------------------------------------------------------------------- // Alloy's own health. The exporter below is the only place that counts what THIS agent -// drops (otelcol_exporter_send_failed_*): without shipping it, an outage's log loss is -// not just possible but uncounted, because the counters live on a loopback port nobody -// reads. Riding the same pipeline is fine — when the pipeline is down these metrics go -// dark too, and that absence is what the central ProjectTelemetrySilent rule fires on. +// drops (otelcol_exporter_send_failed_*); unshipped, those counters live on a loopback +// port nobody reads and an outage's log loss goes uncounted. Riding the same pipeline is +// fine: when the pipeline is down these metrics go dark too, and that absence is what the +// central ProjectTelemetrySilent rule fires on. prometheus.exporter.self "alloy" { } prometheus.scrape "alloy" { @@ -260,14 +256,13 @@ prometheus.scrape "alloy" { } // --------------------------------------------------------------------------- -// GPU metrics, present only on hosts that run compose.telemetry.gpu.yml. No targets -// and costs nothing when the exporter container is absent, so the same config file is -// correct on every host — a GPU host is not a different design, it is one extra overlay. +// GPU metrics, present only on hosts that run compose.telemetry.gpu.yml. With the +// exporter container absent this finds no targets and costs nothing, so the same config +// file is correct on every host — a GPU host is one extra overlay, not a new design. // -// Alloy has no native GPU component (the whole prometheus.exporter.* list was checked, -// and OTel Collector contrib has no NVIDIA receiver either), so an exporter container -// scraped over the Compose network is the pattern. It still means no new agent, no new -// endpoint and no new credential. +// Alloy has no native GPU component, and OTel Collector contrib has no NVIDIA receiver, +// so an exporter container scraped over the Compose network is the pattern. Still no new +// agent, no new endpoint and no new credential. discovery.relabel "gpu_exporter" { targets = discovery.docker.containers.targets @@ -294,10 +289,9 @@ prometheus.scrape "gpu" { targets = discovery.relabel.gpu_exporter.output forward_to = [prometheus.relabel.host.receiver] - // GPU metrics are cheap to over-scrape and the temptation is 5s. 30s matches the - // other scrapes here: thermal throttling and XID faults persist for far longer than - // one interval, and a training run's utilisation curve does not need sub-minute - // resolution to be readable. + // 30s matches the other scrapes here: thermal throttling and XID faults persist for + // far longer than one interval, and a training run's utilisation curve is readable + // without sub-minute resolution. scrape_interval = "30s" job_name = "gpu" } @@ -327,11 +321,11 @@ otelcol.exporter.otlphttp "central" { Authorization = "Bearer " + sys.env("OTLP_AUTH_TOKEN"), // Second, weaker credential for projects whose egress crosses a WAF: the - // project's WAF-skip rule matches THIS header, so the bearer token itself - // never appears in a ruleset expression readable through the provider's - // API. It grants only the managed-security skip. Projects with no WAF in - // front of their egress can leave TELEMETRY_EDGE_KEY empty; the header is - // then sent empty and ignored. + // project's WAF-skip rule matches THIS header, so the bearer token never + // appears in a ruleset expression readable through the provider's API. It + // grants only the managed-security skip. Projects with no WAF in front of + // their egress leave TELEMETRY_EDGE_KEY empty; the header is then sent + // empty and ignored. "X-Telemetry-Key" = sys.env("TELEMETRY_EDGE_KEY"), } } From 7e74925cdfdaf820e60837e127c8f200b120961b Mon Sep 17 00:00:00 2001 From: Simon van Lierde Date: Tue, 1 Sep 2026 15:02:54 +0000 Subject: [PATCH 28/85] fix(security): fail closed on JWT auth and harden containers - drop the JWK-set fallback domain: GRAFANA_JWT_AUTH now requires CF_ACCESS_TEAM_DOMAIN and CF_ACCESS_AUD, with the aud pinned via GF_AUTH_JWT_EXPECT_CLAIMS and exported from infra as grafana_access_aud - pass the tunnel token to cloudflared via TUNNEL_TOKEN env, not argv - cap_drop ALL on every service; node-exporter read-only with a pids limit - digest-pin all hub images, matching the client templates - bootstrap: healthchecks API key via curl config-from-stdin, vendoring refuses a tag without templates/ and prints sha256sum -c lines - grafana healthcheck so compose --wait has something to block on --- .env.example | 6 ++++- bootstrap.sh | 32 +++++++++++++++++++++----- compose.tunnel.yml | 11 ++++++--- compose.yml | 56 +++++++++++++++++++++++++++++++++------------- infra/main.tf | 5 +++++ 5 files changed, 84 insertions(+), 26 deletions(-) diff --git a/.env.example b/.env.example index cd12f70..6aaeb7f 100644 --- a/.env.example +++ b/.env.example @@ -45,7 +45,11 @@ HEALTHCHECKS_API_KEY= CLOUDFLARE_TUNNEL_TOKEN= # Set to true to let Cloudflare Access sign users into Grafana individually, -# instead of everyone sharing the admin login. Needs CF_ACCESS_TEAM_DOMAIN. +# instead of everyone sharing the admin login. Needs both values below; with +# the tunnel overlay active, `just up` refuses to start without them. GRAFANA_JWT_AUTH=false # Your Zero Trust team name — the in https://.cloudflareaccess.com. CF_ACCESS_TEAM_DOMAIN= +# The Grafana Access application's aud tag, so tokens minted for other apps in +# the same team are rejected: cd infra && tofu output -raw grafana_access_aud +CF_ACCESS_AUD= diff --git a/bootstrap.sh b/bootstrap.sh index a1b0898..f963bb3 100755 --- a/bootstrap.sh +++ b/bootstrap.sh @@ -54,8 +54,12 @@ fi out_dir="config/grafana/alerting" # Pinned tag for the vendoring curl. A moving ref would let two projects vendor two -# different agent configs and call it the same template. -tag="$(git -C "$root" describe --tags --abbrev=0 2>/dev/null || echo main)" +# different agent configs and call it the same template — so no fallback to a branch. +tag="$(git -C "$root" describe --tags --abbrev=0 2>/dev/null)" \ + || { echo "error: no release tag to pin the vendoring curls to; tag a release first" >&2; exit 1; } +# ...and the tag must actually contain the templates, or every curl 404s. +git -C "$root" rev-parse -q --verify "${tag}:templates/alloy/config.alloy" >/dev/null \ + || { echo "error: tag ${tag} predates templates/; tag a new release before onboarding" >&2; exit 1; } repo_raw="https://raw.githubusercontent.com/CMLPlatform/monitoring/${tag}/templates" # --------------------------------------------------------------- 1. the keystone rules @@ -89,7 +93,7 @@ sed -e "s@__COVERED__@${covered}@" \ echo "rendered ${out_dir}/coverage.yaml (covering: ${covered})" # ------------------------------------------------------------------ 3. reload Grafana -if docker compose ps --status running --format '{{.Service}}' 2>/dev/null | grep -qx grafana; then +if docker compose ps --status running --services 2>/dev/null | grep -qx grafana; then # A restart, not SIGHUP: Grafana re-reads alert provisioning only at startup, and # a SIGHUP reports success while changing nothing. docker compose up -d --force-recreate grafana >/dev/null 2>&1 \ @@ -105,8 +109,10 @@ if [[ -n "${HEALTHCHECKS_API_KEY:-}" ]]; then hc_note="created via API" # Default job set; override per project with HC_JOBS="backup nightly-sync" etc. for job in ${HC_JOBS:-backup watchdog restore-check}; do - curl -fsS -X POST https://healthchecks.io/api/v3/checks/ \ - -H "X-Api-Key: ${HEALTHCHECKS_API_KEY}" \ + # API key via curl's stdin config, not -H: argv is readable in `ps` by + # any local user, and this is a read-write key. + printf 'header = "X-Api-Key: %s"\n' "$HEALTHCHECKS_API_KEY" \ + | curl -fsS -K - -X POST https://healthchecks.io/api/v3/checks/ \ -H "Content-Type: application/json" \ -d "{\"name\":\"${project}-${env_name}-${job}\",\"slug\":\"${project}-${env_name}-${job}\",\"unique\":[\"name\"],\"timeout\":93600,\"grace\":3600,\"channels\":\"*\"}" \ | sed -n 's/.*"ping_url": *"\([^"]*\)".*/ PING_'"$(echo "$job" | tr 'a-z-' 'A-Z_')"'=\1/p' @@ -135,10 +141,24 @@ curl -fsSL -o deploy/alloy/config.alloy ${repo_raw}/alloy/config.alloy curl -fsSL -o compose.telemetry.yml ${repo_raw}/compose.telemetry.yml curl -fsSL -o compose.telemetry.gpu.yml ${repo_raw}/compose.telemetry.gpu.yml curl -fsSL -o scripts/run_scheduled.sh ${repo_raw}/run_scheduled.sh + +Verify before executing anything — hashes taken from the ${tag} tag here, so a +repo compromise after tagging cannot silently change what project hosts run: +sha256sum -c <<'SUM' +$(for pair in "alloy/config.alloy deploy/alloy/config.alloy" \ + "compose.telemetry.yml compose.telemetry.yml" \ + "compose.telemetry.gpu.yml compose.telemetry.gpu.yml" \ + "run_scheduled.sh scripts/run_scheduled.sh"; do + # Deliberate word splitting: each pair is " ". + # shellcheck disable=SC2086 + set -- $pair + printf '%s %s\n' "$(git -C "$root" show "${tag}:templates/$1" | sha256sum | cut -d' ' -f1)" "$2" + done) +SUM chmod +x scripts/run_scheduled.sh Then include the overlay and bring it up: - docker compose -f compose.yaml -f compose.telemetry.yml up -d + docker compose -f compose.yml -f compose.telemetry.yml up -d The agent container needs 'cgroup: host' (the overlay sets it). Without it cAdvisor reports the root cgroup only, and every container alert here matches nothing. diff --git a/compose.tunnel.yml b/compose.tunnel.yml index 8467eac..b33b7e4 100644 --- a/compose.tunnel.yml +++ b/compose.tunnel.yml @@ -9,12 +9,17 @@ services: cloudflared: - image: cloudflare/cloudflared:2026.8.2 + image: cloudflare/cloudflared:2026.8.2@sha256:0aa26e284f05e6c77ae375b8c9c11d9eb6a448fb7bcd8d40f31cb6176189eb38 restart: unless-stopped - depends_on: [ grafana, otel-collector ] - command: tunnel --no-autoupdate run --token ${CLOUDFLARE_TUNNEL_TOKEN:?set CLOUDFLARE_TUNNEL_TOKEN in .env} + depends_on: [grafana, otel-collector] + # Token via env, not argv: --token would sit in the process table, readable + # from the host and from any container with pid:host (node-exporter has it). + command: tunnel --no-autoupdate run + environment: + TUNNEL_TOKEN: ${CLOUDFLARE_TUNNEL_TOKEN:?set CLOUDFLARE_TUNNEL_TOKEN in .env} # Ceiling sized from observed usage with headroom; see compose.yml. mem_limit: 256m + cap_drop: [ALL] security_opt: - no-new-privileges:true logging: diff --git a/compose.yml b/compose.yml index 6f97405..7ee2d0d 100644 --- a/compose.yml +++ b/compose.yml @@ -21,13 +21,17 @@ x-logging: &default-logging x-security: &default-security - no-new-privileges:true +# Every service also drops all capabilities: they all run unprivileged uids, +# write only to named volumes, and need none. (The client templates already +# ship this — see templates/compose.telemetry.yml.) + # The mem_limit on each service is sized from observed steady-state usage with # generous headroom for bursts. They are ceilings, not reservations: the point # is that one runaway component can't OOM a host that also runs production. services: otel-collector: - image: otel/opentelemetry-collector-contrib:0.156.0 + image: otel/opentelemetry-collector-contrib:0.156.0@sha256:125bdbeb7590cc1952c5b3430ecf14063568980c2c93d5b38676cc0446ed8108 restart: unless-stopped depends_on: loki: @@ -44,41 +48,44 @@ services: # File-backed exporter queue: telemetry buffered during a backend outage # survives a collector restart. Owned by uid 10001 — see justfile. - otel_queue:/var/lib/otelcol/queue - command: [ "--config=/etc/otelcol/config.yaml" ] + command: ["--config=/etc/otelcol/config.yaml"] environment: OTLP_AUTH_TOKEN: ${OTLP_AUTH_TOKEN:?set OTLP_AUTH_TOKEN in .env} # Keep in step with memory_limiter in config/otel-collector.yaml. mem_limit: 512m + cap_drop: [ALL] security_opt: *default-security logging: *default-logging loki: - image: grafana/loki:3.7.6 + image: grafana/loki:3.7.6@sha256:efd47c67f9bac88ca29bcf8cb997d9ab29d1848bd0aff579282295542a745952 restart: unless-stopped volumes: - ./config/loki.yaml:/etc/loki/loki.yaml:ro - loki_data:/loki - command: [ "-config.file=/etc/loki/loki.yaml" ] + command: ["-config.file=/etc/loki/loki.yaml"] # No healthcheck: the image is distroless, so there is no shell, wget or # curl to probe /ready with. Grafana's Stack Health dashboard covers it. mem_limit: 2g + cap_drop: [ALL] security_opt: *default-security logging: *default-logging tempo: - image: grafana/tempo:3.0.3 + image: grafana/tempo:3.0.3@sha256:0296560ac66f8a3600d7fb3014a52c189d4d9c3549ad6ff441bf2409855d68d5 restart: unless-stopped volumes: - ./config/tempo.yaml:/etc/tempo/tempo.yaml:ro - tempo_data:/var/tempo - command: [ "-config.file=/etc/tempo/tempo.yaml" ] + command: ["-config.file=/etc/tempo/tempo.yaml"] # Distroless too — see the note on loki. mem_limit: 2g + cap_drop: [ALL] security_opt: *default-security logging: *default-logging prometheus: - image: prom/prometheus:v3.13.2 + image: prom/prometheus:v3.13.2@sha256:508729e0e2d18e11fd742a5a5ca70e557b940a93948c3c95fd0123a6fd538b69 restart: unless-stopped volumes: - ./config/prometheus.yaml:/etc/prometheus/prometheus.yaml:ro @@ -93,21 +100,27 @@ services: # Whichever hits first wins; the disk-space alert is the backstop. - --storage.tsdb.retention.size=15GB healthcheck: - test: [ "CMD", "wget", "-q", "--spider", "http://localhost:9090/-/ready" ] + test: ["CMD", "wget", "-q", "--spider", "http://localhost:9090/-/ready"] interval: 15s timeout: 3s retries: 5 start_period: 30s mem_limit: 2g + cap_drop: [ALL] security_opt: *default-security logging: *default-logging node-exporter: - image: prom/node-exporter:v1.12.1 + image: prom/node-exporter:v1.12.1@sha256:1b4e4438faca4dd7e001dd445d161a4a2091b0fededa84093b3a8dfeae1f1be0 restart: unless-stopped - command: [ "--path.rootfs=/host" ] + command: ["--path.rootfs=/host"] pid: host mem_limit: 128m + # Hardened hardest of the set: it holds pid:host plus the whole host + # filesystem read-only, so a compromise here sees the most. + cap_drop: [ALL] + read_only: true + pids_limit: 64 security_opt: *default-security volumes: # No rslave: unsupported on Docker Desktop; only affects mounts added after start. @@ -117,7 +130,7 @@ services: logging: *default-logging grafana: - image: grafana/grafana:13.1.4 + image: grafana/grafana:13.1.4@sha256:9be3a3ccdb06bcbb127f888b0c4c1d151837443e478887897a63a27d7b348043 restart: unless-stopped depends_on: loki: @@ -151,17 +164,28 @@ services: GF_SECURITY_COOKIE_SECURE: ${GRAFANA_COOKIE_SECURE:-false} GF_SECURITY_COOKIE_SAMESITE: strict # Opt-in per-user identity from Cloudflare Access, replacing the shared - # admin login. Off until GRAFANA_JWT_AUTH=true and CF_ACCESS_TEAM_DOMAIN - # are set in .env; new users land on the org's default role (Viewer). - # NOTE: the JWK set is team-wide, so if the Zero Trust team fronts more - # than one app, also pin GF_AUTH_JWT_EXPECT_CLAIMS to this app's aud tag. + # admin login. Off until GRAFANA_JWT_AUTH=true plus CF_ACCESS_TEAM_DOMAIN + # and CF_ACCESS_AUD are set in .env (the exposure guards enforce the + # pair); new users land on the org's default role (Viewer). No fallback + # domain: an unset team name must fail closed, not fetch signing keys + # from a claimable cloudflareaccess.com subdomain. The aud pin matters + # because the JWK set is team-wide — without it a token minted for any + # other Access app in the team is accepted here too. GF_AUTH_JWT_ENABLED: ${GRAFANA_JWT_AUTH:-false} GF_AUTH_JWT_HEADER_NAME: Cf-Access-Jwt-Assertion - GF_AUTH_JWT_JWK_SET_URL: https://${CF_ACCESS_TEAM_DOMAIN:-example}.cloudflareaccess.com/cdn-cgi/access/certs + GF_AUTH_JWT_JWK_SET_URL: https://${CF_ACCESS_TEAM_DOMAIN:-}.cloudflareaccess.com/cdn-cgi/access/certs + GF_AUTH_JWT_EXPECT_CLAIMS: '{"aud":"${CF_ACCESS_AUD:-}"}' GF_AUTH_JWT_EMAIL_CLAIM: email GF_AUTH_JWT_USERNAME_CLAIM: email GF_AUTH_JWT_AUTO_SIGN_UP: "true" + healthcheck: + test: ["CMD", "wget", "-q", "--spider", "http://localhost:3000/api/health"] + interval: 15s + timeout: 3s + retries: 5 + start_period: 30s mem_limit: 1g + cap_drop: [ALL] security_opt: *default-security logging: *default-logging diff --git a/infra/main.tf b/infra/main.tf index 63617d7..7c3b6b7 100644 --- a/infra/main.tf +++ b/infra/main.tf @@ -128,6 +128,11 @@ data "cloudflare_zero_trust_tunnel_cloudflared_token" "monitoring" { tunnel_id = cloudflare_zero_trust_tunnel_cloudflared.monitoring.id } +output "grafana_access_aud" { + description = "Set as CF_ACCESS_AUD in ../.env so Grafana rejects tokens minted for other Access apps." + value = cloudflare_zero_trust_access_application.grafana.aud +} + output "tunnel_token" { description = "Set as CLOUDFLARE_TUNNEL_TOKEN in ../.env for the tunnel overlay." value = data.cloudflare_zero_trust_tunnel_cloudflared_token.monitoring.token From 79a01cf283a6b2ce02471da26899f3e88d469421 Mon Sep 17 00:00:00 2001 From: Simon van Lierde Date: Tue, 1 Sep 2026 15:03:04 +0000 Subject: [PATCH 29/85] fix(alerting): key HighErrorRate on project and env - aggregating by job alone let a healthy prod service dilute a broken staging one sharing the job name below the 5% threshold - HostDiskSpaceLow names the host via host_name or instance - wrap long exprs and descriptions as folded scalars --- config/grafana/alerting/rules.yaml | 38 +++++++++++++++++++++--------- 1 file changed, 27 insertions(+), 11 deletions(-) diff --git a/config/grafana/alerting/rules.yaml b/config/grafana/alerting/rules.yaml index 4869b24..fa531fd 100644 --- a/config/grafana/alerting/rules.yaml +++ b/config/grafana/alerting/rules.yaml @@ -7,9 +7,7 @@ # # Keep the total under about ten. These are the generic rules; the per-project keystone # lives in project--.yaml beside this file, rendered by bootstrap.sh. - apiVersion: 1 - groups: # The alerting pipeline's own proof of life, and the detector for it failing. - orgId: 1 @@ -80,7 +78,9 @@ groups: severity: critical annotations: summary: "Grafana cannot deliver {{ $labels.integration }} notifications" - description: "Alerts are firing and going nowhere. Most likely an unset ALERT_WEBHOOK_URL, otherwise the receiver is rejecting. Check `just logs grafana`." + description: >- + Alerts are firing and going nowhere. Most likely an unset ALERT_WEBHOOK_URL, + otherwise the receiver is rejecting. Check `just logs grafana`. # The central stack watching itself. - orgId: 1 name: stack-health @@ -129,13 +129,17 @@ groups: instant: true editorMode: code expr: > - sum by (exporter) (rate(otelcol_exporter_send_failed_spans[5m])) > 0 or sum by (exporter) (rate(otelcol_exporter_send_failed_metric_points[5m])) > 0 or sum by (exporter) (rate(otelcol_exporter_send_failed_log_records[5m])) > 0 + sum by (exporter) (rate(otelcol_exporter_send_failed_spans[5m])) > 0 + or sum by (exporter) (rate(otelcol_exporter_send_failed_metric_points[5m])) > 0 + or sum by (exporter) (rate(otelcol_exporter_send_failed_log_records[5m])) > 0 - *firing labels: severity: warning annotations: summary: "OTel Collector failing to export via {{ $labels.exporter }}" - description: "The collector has been failing to deliver telemetry to a backend for 5 minutes. Check `just logs otel-collector`." + description: >- + The collector has been failing to deliver telemetry to a backend for + 5 minutes. Check `just logs otel-collector`. # Storage backstop: Loki and Tempo have no total-size cap. - orgId: 1 name: capacity @@ -160,13 +164,18 @@ groups: instant: true editorMode: code expr: > - (node_filesystem_avail_bytes{fstype!~"tmpfs|ramfs|overlay"} / node_filesystem_size_bytes{fstype!~"tmpfs|ramfs|overlay"}) < 0.2 + (node_filesystem_avail_bytes{fstype!~"tmpfs|ramfs|overlay"} + / node_filesystem_size_bytes{fstype!~"tmpfs|ramfs|overlay"}) < 0.2 - *firing labels: severity: warning annotations: - summary: "Filesystem {{ $labels.mountpoint }} is over 80% full" - description: "Loki and Tempo have no total-size cap, so this alert is the storage backstop. Free space or lower retention (see RUNBOOK)." + # host_name comes on spoke series shipped via Alloy; the central + # node-exporter has only instance. `or` picks whichever exists. + summary: "Filesystem {{ $labels.mountpoint }} on {{ or $labels.host_name $labels.instance }} is over 80% full" + description: >- + Loki and Tempo have no total-size cap, so this alert is the storage + backstop. Free space or lower retention (see RUNBOOK). # A crash loop must get louder, not quieter. - orgId: 1 name: container-lifecycle @@ -244,11 +253,18 @@ groups: refId: QUERY instant: true editorMode: code + # Keyed on (job, project, env) like every other multi-tenant rule + # here: aggregating on job alone lets a healthy prod service dilute + # a broken staging one sharing the job name below the threshold. expr: > - sum by (job) (rate(http_server_request_duration_seconds_count{http_response_status_code=~"5.."}[5m])) / sum by (job) (rate(http_server_request_duration_seconds_count[5m])) > 0.05 + sum by (job, project, env) + (rate(http_server_request_duration_seconds_count{http_response_status_code=~"5.."}[5m])) + / sum by (job, project, env) (rate(http_server_request_duration_seconds_count[5m])) > 0.05 - *firing labels: severity: warning annotations: - summary: "{{ $labels.job }} error rate above 5%" - description: "More than 5% of requests to {{ $labels.job }} returned 5xx over the last 5 minutes." + summary: "{{ $labels.project }}/{{ $labels.env }} {{ $labels.job }} error rate above 5%" + description: >- + More than 5% of requests to {{ $labels.job }} + ({{ $labels.project }}/{{ $labels.env }}) returned 5xx over the last 5 minutes. From 0a59039be2b9739402972d6d320b116a168c73f0 Mon Sep 17 00:00:00 2001 From: Simon van Lierde Date: Tue, 1 Sep 2026 15:03:16 +0000 Subject: [PATCH 30/85] feat(telemetry): add memory limiter to the spoke Alloy agent - mirrors the hub collector: a long hub outage now sheds load instead of OOM-killing the agent together with its drop counters - shorten the OTLP endpoint guard message --- templates/alloy/config.alloy | 17 ++++++++++++++++- templates/compose.telemetry.yml | 4 ++-- 2 files changed, 18 insertions(+), 3 deletions(-) diff --git a/templates/alloy/config.alloy b/templates/alloy/config.alloy index 3a2d963..5d0f0dc 100644 --- a/templates/alloy/config.alloy +++ b/templates/alloy/config.alloy @@ -114,7 +114,7 @@ otelcol.processor.transform "resource_attributes" { } output { - logs = [otelcol.processor.batch.default.input] + logs = [otelcol.processor.memory_limiter.default.input] } } @@ -298,6 +298,21 @@ prometheus.scrape "gpu" { otelcol.receiver.prometheus "host" { output { + metrics = [otelcol.processor.memory_limiter.default.input] + } +} + +// Refuse work before the cgroup does it for us: without this, a long hub outage +// grows the sending queue until the OOM killer takes the agent AND the self-metrics +// that would have counted the loss. Mirrors the central collector; sized under the +// overlay's 512m mem_limit — keep the two in step. +otelcol.processor.memory_limiter "default" { + check_interval = "1s" + limit = "400MiB" + spike_limit = "100MiB" + + output { + logs = [otelcol.processor.batch.default.input] metrics = [otelcol.processor.batch.default.input] } } diff --git a/templates/compose.telemetry.yml b/templates/compose.telemetry.yml index 765ec87..f684577 100644 --- a/templates/compose.telemetry.yml +++ b/templates/compose.telemetry.yml @@ -79,7 +79,7 @@ services: # `:-` default here would be dead text. This is the COMPOSE project (docker's -p), # not PROJECT above; rename the deploy directory and the filter follows it. COMPOSE_PROJECT_NAME: ${COMPOSE_PROJECT_NAME} - OTEL_EXPORTER_OTLP_ENDPOINT: ${OTEL_EXPORTER_OTLP_ENDPOINT:?telemetry overlay requires OTEL_EXPORTER_OTLP_ENDPOINT} + OTEL_EXPORTER_OTLP_ENDPOINT: ${OTEL_EXPORTER_OTLP_ENDPOINT:?required by the telemetry overlay} OTLP_AUTH_TOKEN: ${OTLP_AUTH_TOKEN:?telemetry overlay requires OTLP_AUTH_TOKEN} # Second, weaker credential: matched by the Cloudflare WAF-skip rule for the # telemetry hostname (infra/cloudflare-zone), so the bearer token itself never @@ -87,7 +87,7 @@ services: # Optional: only projects whose egress crosses a WAF need this. Empty is fine. TELEMETRY_EDGE_KEY: ${TELEMETRY_EDGE_KEY:-} volumes: - - ./deploy/alloy/config.alloy:/etc/alloy/config.alloy:ro # vendored, never edited + - ./deploy/alloy/config.alloy:/etc/alloy/config.alloy:ro # vendored, never edited # node_exporter's collectors read the host's own /proc, /sys and filesystems. # All read-only. Note this does give the container read access to the host # filesystem via /rootfs, which is the price of host disk-usage metrics from a From 77414463931c406d982615f042e1130723f7ba28 Mon Sep 17 00:00:00 2001 From: Simon van Lierde Date: Tue, 1 Sep 2026 15:03:57 +0000 Subject: [PATCH 31/85] refactor(devx): isolate smoke and demo, fold linting into check - smoke and demo run under their own compose projects and Grafana ports (compose.sandbox.yml), so neither touches a stack already on the host - shellcheck, ruff, gitleaks history scan and alloy validate move into just check; the prek layer and just lint are gone - CI: concurrency cancellation, timeouts, full clone for the history scan, smoke logs on failure; dependabot bumps ungrouped - backup unpauses unconditionally; yamllint at 120 cols with generated rules ignored; .yamlfmt keeps folded scalars wrapped - justfile also carries the JWT exposure guard, alpine digest pins, and smoke's curl auth via config-from-stdin (pairs with the security commit) --- .github/dependabot.yml | 7 +- .github/workflows/ci.yml | 19 +++- .yamlfmt | 5 ++ compose.demo.yml | 4 +- compose.sandbox.yml | 17 ++++ demo/pyproject.toml | 19 ++-- justfile | 187 ++++++++++++++++++++++++++++----------- 7 files changed, 196 insertions(+), 62 deletions(-) create mode 100644 .yamlfmt create mode 100644 compose.sandbox.yml diff --git a/.github/dependabot.yml b/.github/dependabot.yml index 1f09af9..444dc1f 100644 --- a/.github/dependabot.yml +++ b/.github/dependabot.yml @@ -1,13 +1,12 @@ version: 2 updates: + # Deliberately ungrouped: a grouped bump that fails `just check` or `just + # smoke` gives no clue which of six images broke it, and bisecting a + # Dependabot PR by hand is worse than reviewing six small ones. - package-ecosystem: "docker-compose" directory: "/" schedule: interval: "weekly" - groups: - monitoring-stack-updates: - patterns: - - "*" - package-ecosystem: "docker" directory: "/demo" diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 2975e06..68c0281 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -10,11 +10,22 @@ on: permissions: contents: read +# A push to an open PR supersedes the run already in flight; both jobs boot a +# full stack, so leaving them stacked wastes a runner per push. +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: true + jobs: check: runs-on: ubuntu-latest + timeout-minutes: 15 steps: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + # `just check` scans git history for secrets; the default shallow clone + # would give it one commit and pass vacuously. + with: + fetch-depth: 0 - uses: extractions/setup-just@53165ef7e734c5c07cb06b3c8e7b647c5aa16db3 # v4.0.0 - run: cp .env.example .env - run: just check @@ -23,9 +34,15 @@ jobs: smoke: runs-on: ubuntu-latest + timeout-minutes: 15 steps: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - uses: extractions/setup-just@53165ef7e734c5c07cb06b3c8e7b647c5aa16db3 # v4.0.0 - run: cp .env.example .env - run: just smoke - - run: just down + # Every smoke failure message points at the logs, which are only useful if + # they outlive the job. + - if: failure() + run: just smoke-logs + - if: always() + run: just smoke-down diff --git a/.yamlfmt b/.yamlfmt new file mode 100644 index 0000000..db5abcf --- /dev/null +++ b/.yamlfmt @@ -0,0 +1,5 @@ +# Keep line breaks inside folded (>) block scalars: the default re-joins them +# onto one line, which un-wraps the long PromQL exprs and alert descriptions +# in config/grafana/alerting/ every time yamlfmt runs. +formatter: + scan_folded_as_literal: true diff --git a/compose.demo.yml b/compose.demo.yml index 7db589d..c7a5631 100644 --- a/compose.demo.yml +++ b/compose.demo.yml @@ -16,7 +16,7 @@ services: demo-api: build: ./demo restart: on-failure - depends_on: [ otel-collector ] + depends_on: [otel-collector] environment: OTEL_SERVICE_NAME: demo-api OTEL_RESOURCE_ATTRIBUTES: env=demo @@ -36,7 +36,7 @@ services: demo-load: image: curlimages/curl:8.21.0 restart: on-failure - depends_on: [ demo-api ] + depends_on: [demo-api] command: - sh - -c diff --git a/compose.sandbox.yml b/compose.sandbox.yml new file mode 100644 index 0000000..677f174 --- /dev/null +++ b/compose.sandbox.yml @@ -0,0 +1,17 @@ +# Isolation overlay shared by `just smoke` and `just demo`. +# +# Smoke runs the core stack under its own compose project name (see the justfile), +# which already gives it separate containers and volumes. The one thing a project +# name cannot separate is host port bindings: compose.yml publishes 3000, 4317 and +# 4318 on 127.0.0.1, so a smoke run on a host that is already serving the stack +# would fail to bind — or, worse on a host where the stack is down, quietly answer +# on the port something else expects to own. +# +# So: drop the ingestion ports entirely (nothing ships telemetry during a smoke +# run) and move Grafana to SANDBOX_PORT, which the assertions in `just smoke` and `just demo` probe. +services: + otel-collector: + ports: !override [] + + grafana: + ports: !override ["127.0.0.1:${SANDBOX_PORT:-3001}:3000"] diff --git a/demo/pyproject.toml b/demo/pyproject.toml index aa3ef02..4ff5f57 100644 --- a/demo/pyproject.toml +++ b/demo/pyproject.toml @@ -3,9 +3,18 @@ name = "demo" version = "0.1.0" requires-python = ">=3.14" dependencies = [ - "fastapi==0.141.1", - "uvicorn==0.52.4", - "opentelemetry-distro==0.65b0", - "opentelemetry-exporter-otlp==1.44.0", - "opentelemetry-instrumentation-fastapi==0.65b0", + "fastapi==0.141.1", + "uvicorn==0.52.4", + "opentelemetry-distro==0.65b0", + "opentelemetry-exporter-otlp==1.44.0", + "opentelemetry-instrumentation-fastapi==0.65b0", ] + +# Pinned so lint results do not depend on whatever ruff config the contributor +# has at home. target-version is explicit because requires-python is ahead of +# what released ruff knows about. +[tool.ruff] +target-version = "py313" + +[tool.ruff.lint] +select = ["E4", "E7", "E9", "F", "I", "UP", "B"] diff --git a/justfile b/justfile index 40cca3c..b871200 100644 --- a/justfile +++ b/justfile @@ -5,58 +5,99 @@ set dotenv-load stateful := "grafana prometheus loki tempo" backup_mounts := "-v monitoring_grafana_data:/data/grafana -v monitoring_prometheus_data:/data/prometheus -v monitoring_loki_data:/data/loki -v monitoring_tempo_data:/data/tempo" -# Compose with the demo overlay, for the demo lifecycle recipes — explicit -f -# so `just demo` works regardless of what COMPOSE_FILE names. -compose_demo := "docker compose -f compose.yml -f compose.demo.yml" +# `demo` and `smoke` each bring up a throwaway copy of the core stack, so both +# run under their own compose project and their own Grafana port (see +# compose.sandbox.yml). Neither can adopt, recreate or delete the containers and +# volumes of a stack already running on this host: `just demo` on the production +# box spins up its own Grafana next to the real one instead of joining it. The +# explicit -f list also keeps a host's COMPOSE_FILE (tunnel) out of both. +demo_project := "monitoring-demo" +demo_port := env("DEMO_PORT", "3002") +demo_url := "http://localhost:" + demo_port +compose_demo := "SANDBOX_PORT=" + demo_port + " docker compose -p " + demo_project + " -f compose.yml -f compose.demo.yml -f compose.sandbox.yml" + +# What compose will actually name the project for the up-paths, so the queue +# volume gets chowned where the collector will look for it. +core_project := env("COMPOSE_PROJECT_NAME", "monitoring") + +# Same isolation for smoke, on its own project and port so a smoke run and a +# demo stack can also coexist. +smoke_project := "monitoring-smoke" +smoke_port := env("SMOKE_PORT", "3001") +smoke_url := "http://localhost:" + smoke_port +compose_smoke := "SANDBOX_PORT=" + smoke_port + " docker compose -p " + smoke_project + " -f compose.yml -f compose.sandbox.yml" # dashboards/*.json as the paths they get when mounted at /dashboards, shared -# by check and smoke. -dash_paths := `ls dashboards/*.json | sed 's|^dashboards|/dashboards|' | tr '\n' ' '` +# by check and smoke. 2>/dev/null so an empty dashboards/ doesn't abort every +# recipe in this file, including the teardown ones. +dash_paths := `ls dashboards/*.json 2>/dev/null | sed 's|^dashboards|/dashboards|' | tr '\n' ' '` default: @just --list +# backup/restore mount volumes by literal name, which assumes compose's default +# project name (compose.yml sets `name: monitoring`). COMPOSE_PROJECT_NAME +# overrides it, so they would silently target volumes nothing uses. +_project-guard: + @[ -z "${COMPOSE_PROJECT_NAME:-}" ] || { echo "error: COMPOSE_PROJECT_NAME is set; this recipe expects the monitoring_* volume names" >&2; exit 1; } + # The collector's queue volume must be writable by the image's uid 10001, but # a fresh named volume is root-owned and the image is distroless (no chown at -# startup possible). Idempotent, so every up-path just runs it. -_queue-volume: - @docker volume create monitoring_otel_queue > /dev/null - @docker run --rm --network none -v monitoring_otel_queue:/q alpine:3.24 chown 10001:10001 /q - -# The stack (Grafana at http://localhost:3000). Overlays are host config: -# COMPOSE_FILE in .env names the file set (see .env.example), and every recipe -# here — up, down, logs, ps, backup — acts on that same set. With the tunnel -# overlay active, up refuses to start until the exposure guards pass. -up: _queue-volume - @case "${COMPOSE_FILE:-}" in *compose.tunnel.yml*) just _expose-guards;; esac +# startup possible). Idempotent, so every up-path just runs it. Takes the +# project name because smoke brings the stack up under a different one. +_queue-volume project: + @docker volume create {{project}}_otel_queue > /dev/null + @docker run --rm --network none -v {{project}}_otel_queue:/q alpine:3.24@sha256:28bd5fe8b56d1bd048e5babf5b10710ebe0bae67db86916198a6eec434943f8b chown 10001:10001 /q + +# Overlays are host config: COMPOSE_FILE in .env names the file set (see +# .env.example), and every recipe here — up, down, logs, ps, backup — acts on +# that same set. With the tunnel overlay active, this refuses to start until the +# exposure guards pass. +# Start the stack (Grafana at http://localhost:3000). +up: (_queue-volume core_project) _guard-if-exposed docker compose up -d +# The exposure guards, but only when the tunnel overlay is in play. Shared by +# every recipe that can put a service back on the tunnel. +_guard-if-exposed: + @case "${COMPOSE_FILE:-}" in *compose.tunnel.yml*) just _expose-guards;; esac + # Refuses to expose the stack with the documented default credentials. _expose-guards: @[ "${OTLP_AUTH_TOKEN:-}" != "local-dev-token" ] || { echo "error: OTLP_AUTH_TOKEN is still the local default; generate one (openssl rand -hex 32) before exposing ingestion" >&2; exit 1; } @[ "${GRAFANA_ADMIN_PASSWORD:-}" != "change-me" ] || { echo "error: GRAFANA_ADMIN_PASSWORD is still the documented default; change it before exposing Grafana" >&2; exit 1; } @[ "${GRAFANA_ROOT_URL:-}" != "http://localhost:3000" ] || { echo "error: GRAFANA_ROOT_URL is still the localhost default; set it to the tunnel hostname or every absolute URL Grafana generates breaks" >&2; exit 1; } @[ "${GRAFANA_COOKIE_SECURE:-false}" = "true" ] || { echo "error: GRAFANA_COOKIE_SECURE must be true when Grafana is served over HTTPS; set it in .env" >&2; exit 1; } + @[ "${GRAFANA_JWT_AUTH:-false}" != "true" ] || { [ -n "${CF_ACCESS_TEAM_DOMAIN:-}" ] && [ -n "${CF_ACCESS_AUD:-}" ]; } || { echo "error: GRAFANA_JWT_AUTH=true needs CF_ACCESS_TEAM_DOMAIN and CF_ACCESS_AUD in .env (cd infra && tofu output -raw grafana_access_aud)" >&2; exit 1; } @[ -n "${HEARTBEAT_URL:-}" ] || echo "WARNING: HEARTBEAT_URL is empty; the stack goes live without a dead-man's switch" >&2 @[ -n "${ALERT_WEBHOOK_URL:-}" ] || { echo "error: ALERT_WEBHOOK_URL is empty; every alert would fire into an empty url_file and be dropped. The heartbeat keeps pinging either way, so this failure looks healthy from the outside — set it, or comment out this guard deliberately" >&2; exit 1; } down: docker compose down --remove-orphans -# Core stack + a demo telemetry source (see compose.demo.yml), then look at -# Grafana: http://localhost:3000 -demo: _queue-volume +# See compose.demo.yml; watch it arrive at http://localhost:3002 (DEMO_PORT) — +# its own Grafana, not the one `just up` serves on :3000. +# Core stack plus a demo telemetry source, isolated from any running stack. +demo: (_queue-volume demo_project) {{compose_demo}} up -d --build -# Build the demo image without starting anything. Used by CI to catch a broken -# demo app before it merges. +# Used by CI to catch a broken demo app before it merges. +# Build the demo image without starting anything. demo-build: {{compose_demo}} build -# Remove only the demo services; the core stack keeps running. +# Removes only the demo services; the demo project's own core stack keeps +# running. `just demo-destroy` takes the whole thing down. +# Stop the demo telemetry source. demo-down: {{compose_demo}} rm -sf demo-api demo-load +# Safe: -p scopes it to the demo project, so it cannot touch a real stack on +# the same host. +# Tear down the whole demo stack and its throwaway volumes. +demo-destroy: + {{compose_demo}} down --remove-orphans --volumes + logs service="": docker compose logs -f {{service}} @@ -64,7 +105,8 @@ ps: docker compose ps # Targets the COMPOSE_FILE set; demo services are recreated with `just demo`. -restart service: +# Restart one service. +restart service: _guard-if-exposed docker compose restart {{service}} pull: @@ -74,26 +116,43 @@ pull: tail service: docker compose logs -f --no-log-prefix {{service}} | jq -R 'fromjson? // .' -# Validate everything. All validators run in containers — no host installs. -# promtool/otelcol images are read from compose.yml so they can't -# drift from the versions the stack actually runs. +# All validators run in containers — no host installs, no network. The +# promtool/otelcol images are read from compose.yml so they can't drift from +# the versions the stack actually runs. +# Validate every config in the repo. check: docker compose config -q CLOUDFLARE_TUNNEL_TOKEN=dummy docker compose -f compose.yml -f compose.tunnel.yml config -q {{compose_demo}} config -q + {{compose_smoke}} config -q images="$(docker compose config --images)" && \ - docker run --rm -v ./config/prometheus.yaml:/etc/prometheus/prometheus.yaml:ro --entrypoint promtool $(echo "$images" | grep prom/prometheus) check config /etc/prometheus/prometheus.yaml && \ - docker run --rm -e OTLP_AUTH_TOKEN=dummy -v ./config/otel-collector.yaml:/etc/otelcol/config.yaml:ro $(echo "$images" | grep opentelemetry-collector) validate --config=/etc/otelcol/config.yaml && \ + docker run --rm --network none -v ./config/prometheus.yaml:/etc/prometheus/prometheus.yaml:ro --entrypoint promtool $(echo "$images" | grep prom/prometheus) check config /etc/prometheus/prometheus.yaml && \ + docker run --rm --network none -e OTLP_AUTH_TOKEN=dummy -v ./config/otel-collector.yaml:/etc/otelcol/config.yaml:ro $(echo "$images" | grep opentelemetry-collector) validate --config=/etc/otelcol/config.yaml && \ docker run --rm --network none -v ./config/loki.yaml:/etc/loki/loki.yaml:ro $(echo "$images" | grep grafana/loki) -config.file=/etc/loki/loki.yaml -verify-config && \ docker run --rm --network none -v ./config/tempo.yaml:/etc/tempo/tempo.yaml:ro $(echo "$images" | grep grafana/tempo) -config.file=/etc/tempo/tempo.yaml -config.verify=true - docker run --rm --network none -v .:/code:ro pipelinecomponents/yamllint:0.35.13 yamllint -d '{extends: relaxed, ignore: [.git/, backups/, infra/.terraform/]}' . + # line-length at 120, not the default 80: digest-pinned image refs need + # ~130 but count as non-breakable mappings. The rendered project-*/coverage + # rules are ignored — machine-written by bootstrap.sh, their expr lines + # grow with every onboarded project. + docker run --rm --network none -v .:/code:ro pipelinecomponents/yamllint:0.35.13 yamllint -d '{extends: relaxed, rules: {line-length: {max: 120, allow-non-breakable-inline-mappings: true}}, ignore: [.git/, backups/, infra/.terraform/, config/grafana/alerting/project-*.yaml, config/grafana/alerting/coverage.yaml]}' . docker run --rm --network none -v .:/repo:ro -w /repo rhysd/actionlint:1.7.12 -color + docker run --rm --network none -v .:/mnt:ro koalaman/shellcheck:v0.11.0 bootstrap.sh templates/run_scheduled.sh + docker run --rm --network none -v ./demo:/demo:ro ghcr.io/astral-sh/ruff:0.14.2 check --no-cache /demo docker run --rm --network none -v ./infra:/infra:ro -w /infra ghcr.io/opentofu/opentofu:1.12.3 fmt -check - docker run --rm -v ./dashboards:/dashboards:ro ghcr.io/jqlang/jq:1.8.1 empty {{dash_paths}} + docker run --rm --network none -v ./dashboards:/dashboards:ro ghcr.io/jqlang/jq:1.8.1 empty {{dash_paths}} + # The one config vendored verbatim onto every project host; a syntax error + # here otherwise first surfaces as a crash-looping agent on a client machine. + # Image ref matches templates/compose.telemetry.yml — keep them in step. + # Secrets that reached git history. Scans commits, not the working tree, so + # it sees exactly what is in the repo and never the gitignored .env — a + # working-tree scan flags .env's real tokens and fails on every dev machine. + # Needs full history: a shallow CI clone has one commit and passes vacuously. + docker run --rm --network none -v .:/repo:ro zricethezav/gitleaks:v8.30.1@sha256:c00b6bd0aeb3071cbcb79009cb16a60dd9e0a7c60e2be9ab65d25e6bc8abbb7f git --redact --no-banner /repo + docker run --rm --network none -v ./templates/alloy/config.alloy:/etc/alloy/config.alloy:ro -e COMPOSE_PROJECT_NAME=dummy -e ENVIRONMENT=dummy -e PROJECT=dummy -e OTEL_EXPORTER_OTLP_ENDPOINT=https://dummy -e OTLP_AUTH_TOKEN=dummy -e TELEMETRY_EDGE_KEY= grafana/alloy:v1.18.1@sha256:0f4434c92b3e6cdac38bb129b344e1790c246f7b6e2eaffcc16a5fa363240e33 validate /etc/alloy/config.alloy -# Full OpenTofu validation (downloads the provider, so not part of `check`). # Runs against a copy of the sources only: state and tfvars never enter the # container, which needs network access to fetch the provider. +# Full OpenTofu validation (downloads the provider, so not part of `check`). infra-validate: @d=$(mktemp -d) && cp infra/main.tf infra/.terraform.lock.hcl "$d"/ && docker run --rm --entrypoint sh -v "$d":/src:ro ghcr.io/opentofu/opentofu:1.12.3 -c 'mkdir /work && cp /src/main.tf /src/.terraform.lock.hcl /work && cd /work && tofu init -backend=false -input=false >/dev/null && tofu validate'; rc=$?; rm -rf "$d"; exit $rc @@ -101,32 +160,60 @@ infra-validate: fmt: yamlfmt . -# Snapshot all stateful volumes to backups/.tar.gz (mode 0600 — -# it contains the Grafana DB and webhook secrets; copy it off-host, keep it -# private). Services are paused during the copy, and unpaused even if the -# copy fails. -backup: - @[ -z "${COMPOSE_PROJECT_NAME:-}" ] || { echo "error: COMPOSE_PROJECT_NAME is set; backup expects the monitoring_* volume names" >&2; exit 1; } +# The tarball is mode 0600 — it contains the Grafana DB and webhook secrets; +# copy it off-host, keep it private. Services are paused during the copy and +# unpaused unconditionally afterwards: `pause` is per-container and can fail +# halfway, so an unpause reached only on success would leave the stack frozen. +# Snapshot all stateful volumes to backups/.tar.gz. +backup: _project-guard mkdir -p backups -docker compose unpause {{stateful}} 2>/dev/null - docker compose pause {{stateful}} && { docker run --rm {{backup_mounts}} -v ./backups:/backups alpine:3.24 sh -c 'umask 077 && tar czf /backups/monitoring-$(date +%Y%m%d-%H%M%S).tar.gz -C /data .'; rc=$?; docker compose unpause {{stateful}}; exit $rc; } + rc=0; docker compose pause {{stateful}} && docker run --rm {{backup_mounts}} -v ./backups:/backups alpine:3.24@sha256:28bd5fe8b56d1bd048e5babf5b10710ebe0bae67db86916198a6eec434943f8b sh -c 'umask 077 && tar czf /backups/monitoring-$(date +%Y%m%d-%H%M%S).tar.gz -C /data .' || rc=$?; docker compose unpause {{stateful}}; exit $rc @ls -lh backups/ | tail -1 +# The wipe is unrecoverable, so the current state is snapshotted first: if the +# extract dies halfway (full disk, wrong volume set) the volumes are left +# partial, and backups/pre-restore-*.tar.gz is the only way back. # Restore a backup tarball into the volumes (stops the stack; wipes current state). -restore file: +restore file: _project-guard @[ -f "{{file}}" ] || { echo "error: {{file}} not found" >&2; exit 1; } - docker run --rm -v {{absolute_path(file)}}:/backup.tar.gz:ro alpine:3.24 tar tzf /backup.tar.gz > /dev/null + docker run --rm --network none -v {{absolute_path(file)}}:/backup.tar.gz:ro alpine:3.24@sha256:28bd5fe8b56d1bd048e5babf5b10710ebe0bae67db86916198a6eec434943f8b tar tzf /backup.tar.gz > /dev/null docker compose down --remove-orphans - docker run --rm {{backup_mounts}} -v {{absolute_path(file)}}:/backup.tar.gz:ro alpine:3.24 sh -c 'for d in /data/*; do find "$d" -mindepth 1 -delete; done && tar xzf /backup.tar.gz -C /data' + mkdir -p backups + docker run --rm --network none {{backup_mounts}} -v ./backups:/backups alpine:3.24@sha256:28bd5fe8b56d1bd048e5babf5b10710ebe0bae67db86916198a6eec434943f8b sh -c 'umask 077 && tar czf /backups/pre-restore-$(date +%Y%m%d-%H%M%S).tar.gz -C /data .' + docker run --rm --network none {{backup_mounts}} -v {{absolute_path(file)}}:/backup.tar.gz:ro alpine:3.24@sha256:28bd5fe8b56d1bd048e5babf5b10710ebe0bae67db86916198a6eec434943f8b sh -c 'for d in /data/*; do find "$d" -mindepth 1 -delete; done && tar xzf /backup.tar.gz -C /data' @echo "Restored {{file}} — run 'just up' to start the stack." -# Boot the core stack, wait until Grafana reports healthy, assert every -# dashboard in dashboards/ actually provisioned (Grafana skips broken ones -# silently), and fail if any service is crash-looping. Used by CI. -smoke: _queue-volume - docker compose up -d - n=0; until curl -sf http://localhost:3000/api/health >/dev/null; do n=$((n+3)); [ $n -ge 120 ] && { echo "Grafana not healthy after 120s" >&2; exit 1; }; sleep 3; done - @want="$(docker run --rm --network none -v ./dashboards:/dashboards:ro ghcr.io/jqlang/jq:1.8.1 -r .uid {{dash_paths}})"; n=0; while :; do have="$(curl -sf -u "admin:${GRAFANA_ADMIN_PASSWORD}" 'http://localhost:3000/api/search?type=dash-db&limit=5000' | docker run --rm -i ghcr.io/jqlang/jq:1.8.1 -r '.[].uid')"; missing=""; for uid in $want; do echo "$have" | grep -qx "$uid" || missing="$missing $uid"; done; [ -z "$missing" ] && break; n=$((n+3)); [ $n -ge 60 ] && { echo "error: dashboards not provisioned:$missing" >&2; exit 1; }; sleep 3; done - @want=$(grep -h '^ *title:' config/grafana/alerting/*.yaml | wc -l); got=$(curl -sf -u "admin:${GRAFANA_ADMIN_PASSWORD}" http://localhost:3000/api/v1/provisioning/alert-rules | docker run --rm -i ghcr.io/jqlang/jq:1.8.1 length); [ "$want" = "$got" ] || { echo "error: $want alert rules on disk, $got provisioned — a malformed file provisions none of its group. See just logs grafana" >&2; exit 1; } - @[ -z "$(docker compose ps -q --status=restarting --status=exited)" ] || { echo "error: services not running:" >&2; docker compose ps >&2; exit 1; } +# `--wait` does the readiness and crash-loop work: it blocks on the grafana and +# prometheus healthchecks in compose.yml and fails if any container exits, so +# there is no poll loop or `ps --status=exited` check to hand-roll here. +# Measured, because it is not obvious: a service with no healthcheck of its own +# (otel-collector, loki, tempo) still fails the wait while it is restarting, so +# a crash-looping collector is caught. It costs the full --wait-timeout to +# report, where the old explicit check failed immediately. +# What it cannot see is provisioning — Grafana answers /api/health long before +# it has read the provisioning dirs, and skips a broken dashboard or a malformed +# alert group silently — so that part is asserted below, on a single loop +# because both land asynchronously. +# Boot an isolated copy of the core stack and assert it provisioned everything. +smoke: (_queue-volume smoke_project) + {{compose_smoke}} up -d --wait --wait-timeout 120 + @want_dash="$(docker run --rm --network none -v ./dashboards:/dashboards:ro ghcr.io/jqlang/jq:1.8.1 -r .uid {{dash_paths}})"; want_rules=$(grep -h '^ *title:' config/grafana/alerting/*.yaml | wc -l); auth="user = \"admin:${GRAFANA_ADMIN_PASSWORD}\""; n=0; while :; do \ + search=$(printf '%s\n' "$auth" | curl -sf -K - '{{smoke_url}}/api/search?type=dash-db&limit=5000') && rules=$(printf '%s\n' "$auth" | curl -sf -K - {{smoke_url}}/api/v1/provisioning/alert-rules) || { echo "error: Grafana API request failed — wrong GRAFANA_ADMIN_PASSWORD, or Grafana is not answering on {{smoke_url}}" >&2; exit 1; }; \ + have=$(printf '%s' "$search" | docker run --rm -i ghcr.io/jqlang/jq:1.8.1 -r '.[].uid'); got=$(printf '%s' "$rules" | docker run --rm -i ghcr.io/jqlang/jq:1.8.1 length); \ + missing=""; for uid in $want_dash; do printf '%s' "$have" | grep -qx "$uid" || missing="$missing $uid"; done; \ + [ -z "$missing" ] && [ "$want_rules" = "$got" ] && break; \ + n=$((n+3)); [ $n -ge 60 ] && { echo "error: not provisioned after 60s — dashboards missing:${missing:- none}; alert rules $got/$want_rules (a malformed file provisions none of its group). See just smoke-logs" >&2; exit 1; }; \ + sleep 3; \ + done @echo "Stack healthy" + +# Logs from the smoke stack (its own project, so `just logs` will not show it). +smoke-logs: + {{compose_smoke}} logs --no-color --tail=200 + +# Safe: -p scopes it to the smoke project, so it cannot touch a real stack on +# the same host. +# Tear down the smoke stack and its throwaway volumes. +smoke-down: + {{compose_smoke}} down --remove-orphans --volumes From b16516496abdf3400191fc927ba22d4529d91f62 Mon Sep 17 00:00:00 2001 From: Simon van Lierde Date: Tue, 1 Sep 2026 15:04:12 +0000 Subject: [PATCH 32/85] docs: update for the hardening pass, fold S3 stub into ADR 0001 - compose.storage-s3.yml was 61 lines of commented stub; the escape hatch is an ADR 0001 appendix now, references updated - README names the Logs dashboard correctly and lists check's full scope - RUNBOOK: JWT setup needs CF_ACCESS_AUD, shared-token trust model stated, hand-bumped tool image list extended - changelog entries for the hardening pass - datasources comment moved to a yamlfmt-stable position --- CHANGELOG.md | 40 +++++++++++++++---- README.md | 30 +++++++++----- compose.storage-s3.yml | 60 ---------------------------- config/grafana/datasources.yaml | 8 ++-- config/tempo.yaml | 2 +- docs/ONBOARDING.md | 6 +-- docs/RUNBOOK.md | 29 +++++++++----- docs/adr/0001-observability-stack.md | 44 +++++++++++++++++++- templates/README.md | 2 +- 9 files changed, 123 insertions(+), 98 deletions(-) delete mode 100644 compose.storage-s3.yml diff --git a/CHANGELOG.md b/CHANGELOG.md index 8d48c43..0f1652c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -19,13 +19,29 @@ queries that had been measuring the wrong thing. - **Full-stack scraping**: Prometheus now scrapes Grafana, Loki, and Tempo as well, so `TargetDown` covers every service. - **Per-user Grafana logins**, optional: `GRAFANA_JWT_AUTH` plus - `CF_ACCESS_TEAM_DOMAIN` make Grafana verify the Cloudflare Access JWT - instead of everyone sharing the admin password. + `CF_ACCESS_TEAM_DOMAIN` and `CF_ACCESS_AUD` make Grafana verify the + Cloudflare Access JWT — pinned to this app's `aud` tag, both values enforced + by the exposure guards — instead of everyone sharing the admin password. - **Memory ceilings** (`mem_limit`) on every service, sized from observed - usage, so one runaway component cannot OOM the host. -- **Wider validation**: `just check` also verifies the Loki and Tempo configs - and OpenTofu formatting; `just smoke` asserts every dashboard provisioned; - CI also runs `just infra-validate` and a new `just demo-build`. + usage, so one runaway component cannot OOM the host; the spoke Alloy agent + gains a matching in-pipeline memory limiter so a long hub outage sheds load + instead of OOM-killing the agent and its loss counters with it. +- **Tighter container defaults**: every service drops all capabilities, + node-exporter (which holds `pid: host` and the host filesystem) runs + read-only with a pids limit, and the hub images are digest-pinned like the + client templates. The tunnel token reaches cloudflared via environment, not + argv. +- **Wider validation**: `just check` also verifies the Loki and Tempo configs, + the vendored Alloy config, shell scripts, the demo app's Python, OpenTofu + formatting, and git history for secrets; `just smoke` asserts every + dashboard and alert rule provisioned; CI also runs `just infra-validate` + and a new `just demo-build`. +- **Isolated smoke and demo**: each runs under its own compose project and + Grafana port (`compose.sandbox.yml`), so neither can adopt or recreate a + stack already running on the host. +- `bootstrap.sh` refuses to run from a release tag that lacks `templates/` + and prints `sha256sum -c` lines for the files it tells a project host to + vendor. - Dependabot now watches the Cloudflare provider in `infra/`, and the runbook covers OpenTofu-managed tunnel and Access changes. @@ -48,7 +64,7 @@ queries that had been measuring the wrong thing. - **Tempo's metrics generator is removed**: RED comes from the applications' own OTLP metrics now (ADR 0002), so Tempo stores traces and nothing else. - Dropped the `relab-api` dashboard: the `$service` picker on Service Health - and Logs Overview covers it. + and Logs covers it. - README rewritten for a broader CML audience. - Image bumps: Grafana 13.1.4, Loki 3.7.6, Tempo 3.0.3, Prometheus 3.13.2, the collector 0.156.0, node-exporter 1.12.1, cloudflared 2026.8.2, and the @@ -59,6 +75,8 @@ queries that had been measuring the wrong thing. - **Alertmanager**: alerting is Grafana-managed now (ADR 0002) — rules are provisioned from `config/grafana/alerting/`, and delivery still posts to `ALERT_WEBHOOK_URL`. +- The commented `compose.storage-s3.yml` stub: the S3 escape hatch lives as + an appendix of ADR 0001 instead. ### Fixed @@ -77,6 +95,14 @@ queries that had been measuring the wrong thing. of racing it on a cold start. - The S3 overlay documents the `-config.expand-env=true` that Loki and Tempo need before `${...}` in their configs expands at all. +- **`HighErrorRate` merged environments**: aggregating by job alone let a + healthy prod service dilute a broken staging one sharing the job name below + the threshold; it now keys on job, project and env like the other + multi-tenant rules, and `HostDiskSpaceLow` says whose disk is filling. +- With `GRAFANA_JWT_AUTH=true` but no team domain set, Grafana fetched its + JWT signing keys from a placeholder `cloudflareaccess.com` subdomain any + Cloudflare customer could claim; the fallback is gone and the guards refuse + to start without the real values. ### Security diff --git a/README.md b/README.md index bcebe30..31dbc5b 100644 --- a/README.md +++ b/README.md @@ -25,12 +25,12 @@ just demo This starts the full stack plus a small FastAPI service under constant artificial load (`compose.demo.yml`). It uses OpenTelemetry auto-instrumentation and fails about one request in ten, on purpose. Give it -a minute, then open Grafana at (admin / change-me): +a minute, then open Grafana at (admin / change-me): - **Dashboards → Service Health (RED)** — request rate, error rate, and latency. The dots on the latency panel are exemplars: click one and Grafana opens the exact trace behind that measurement. -- **Dashboards → Logs Overview** — log volume by service and level, an error +- **Dashboards → Logs** — log volume by service and level, an error feed, and a live tail of everything arriving over OTLP. - **Alerting → Alert rules** — the stack-health and error-rate rules Grafana is evaluating. `HighErrorRate` trips on the demo service after five minutes: @@ -38,8 +38,12 @@ a minute, then open Grafana at (admin / change-me): ![Service Health (RED) dashboard](docs/img/service-health.png) -`just demo-down` removes the demo services again; the rest of the stack keeps -running. +`just demo-down` removes the demo services again; the rest of the demo stack +keeps running, and `just demo-destroy` takes the whole thing down. + +The demo runs under its own compose project on its own port, so it never joins +or disturbs a stack already running on the host — safe on the production box. +Same for `just smoke`, on :3001. Override with `DEMO_PORT` / `SMOKE_PORT`. ## How it works @@ -66,8 +70,9 @@ flowchart LR ``` Solid arrows show telemetry being written; dotted arrows show Grafana reading -at query time. Locally (`just up` or `just demo`) there is no tunnel: -everything talks over the compose network and Grafana is at `localhost:3000`. +at query time. Locally there is no tunnel: everything talks over the compose +network, and Grafana is at `localhost:3000` for `just up`, `localhost:3002` for +the isolated `just demo` stack. The stack runs on a single host; at CML's telemetry volume, distributed ingestion would add operational weight for no gain @@ -102,12 +107,16 @@ overlay needs; bootstrap steps are at the top of [infra/main.tf](infra/main.tf). `just check` validates compose files, Prometheus config, the collector, Loki -and Tempo configs, YAML, workflows, OpenTofu formatting, and dashboard JSON. +and Tempo configs, the vendored Alloy config, YAML, workflows, shell scripts, +the demo app's Python, OpenTofu formatting, dashboard JSON, and git history +for leaked secrets. Every validator runs in a pinned container, so nothing is installed on the host. Grafana's alerting provisioning has no offline validator, so `just smoke` covers it: it boots the stack, waits for Grafana to report healthy, and checks -that every dashboard and every alert rule provisioned. CI runs both on every -push and pull request. +that every dashboard and every alert rule provisioned. It runs under its own +compose project on its own ports, so it cannot disturb a stack already running +on the host — `just smoke` is safe on the production box, and `just smoke-down` +cleans it up. CI runs both on every push and pull request. ## Sending telemetry from a project @@ -146,7 +155,7 @@ Everything persists to local Docker volumes (`loki_data`, `tempo_data`, `otel_queue`, holds the collector's on-disk export queue: seconds of in-flight telemetry, worthless by the time anyone restores, so backups skip it. When local disk stops fitting, Loki and Tempo can move to any S3-compatible object -store (Backblaze B2, Cloudflare R2, Hetzner, MinIO); `compose.storage-s3.yml` +store (Backblaze B2, Cloudflare R2, Hetzner, MinIO); the appendix of ADR 0001 documents that change. ## Layout @@ -155,6 +164,7 @@ documents that change. compose.yml # core services compose.tunnel.yml # production overlay: Cloudflare Tunnel compose.demo.yml # demo overlay: sample telemetry source +compose.sandbox.yml # isolation overlay for `just demo` and `just smoke` demo/ # the demo FastAPI service config/ otel-collector.yaml # ingestion gateway (OTLP in → Loki/Tempo/Prometheus out) diff --git a/compose.storage-s3.yml b/compose.storage-s3.yml deleted file mode 100644 index 8dc99b1..0000000 --- a/compose.storage-s3.yml +++ /dev/null @@ -1,60 +0,0 @@ -# STUB — the storage scale-out path (not wired up; see ADR 0001). -# -# When local volumes stop fitting, Loki and Tempo move their object storage -# to any S3-compatible backend (Cloudflare R2, Backblaze B2, Hetzner, MinIO) -# without touching the collector, Prometheus, or any client project. -# -# This is a commented stub: it documents the concrete shape of -# the change so the README's scale-out claim is real, but it is not meant to -# be started until credentials and a bucket exist. To activate: -# -# 1. Create s3 variants of the configs, e.g. config/loki.s3.yaml replacing -# `common.storage.filesystem` with: -# -# common: -# storage: -# s3: -# endpoint: ${S3_ENDPOINT} # e.g. .r2.cloudflarestorage.com -# bucketnames: cml-loki -# access_key_id: ${S3_ACCESS_KEY_ID} -# secret_access_key: ${S3_SECRET_ACCESS_KEY} -# s3forcepathstyle: true -# -# and config/tempo.s3.yaml replacing `storage.trace.backend: local` with: -# -# storage: -# trace: -# backend: s3 -# s3: -# endpoint: ${S3_ENDPOINT} -# bucket: cml-tempo -# access_key: ${S3_ACCESS_KEY_ID} -# secret_key: ${S3_SECRET_ACCESS_KEY} -# -# Neither Loki nor Tempo expands ${...} in its config by default, so the -# overlay below re-declares each `command` with -config.expand-env=true. -# (compose replaces `command` wholesale, it does not merge it.) -# -# 2. Uncomment the overlay below, add compose.storage-s3.yml to COMPOSE_FILE -# in .env, and `just up`. -# -# services: -# loki: -# volumes: -# - ./config/loki.s3.yaml:/etc/loki/loki.yaml:ro -# - loki_data:/loki -# command: [ "-config.file=/etc/loki/loki.yaml", "-config.expand-env=true" ] -# environment: -# S3_ENDPOINT: ${S3_ENDPOINT:?set S3_ENDPOINT in .env} -# S3_ACCESS_KEY_ID: ${S3_ACCESS_KEY_ID:?set S3_ACCESS_KEY_ID in .env} -# S3_SECRET_ACCESS_KEY: ${S3_SECRET_ACCESS_KEY:?set S3_SECRET_ACCESS_KEY in .env} -# -# tempo: -# volumes: -# - ./config/tempo.s3.yaml:/etc/tempo/tempo.yaml:ro -# - tempo_data:/var/tempo -# command: [ "-config.file=/etc/tempo/tempo.yaml", "-config.expand-env=true" ] -# environment: -# S3_ENDPOINT: ${S3_ENDPOINT:?set S3_ENDPOINT in .env} -# S3_ACCESS_KEY_ID: ${S3_ACCESS_KEY_ID:?set S3_ACCESS_KEY_ID in .env} -# S3_SECRET_ACCESS_KEY: ${S3_SECRET_ACCESS_KEY:?set S3_SECRET_ACCESS_KEY in .env} diff --git a/config/grafana/datasources.yaml b/config/grafana/datasources.yaml index 4feaca7..3c38b73 100644 --- a/config/grafana/datasources.yaml +++ b/config/grafana/datasources.yaml @@ -38,13 +38,13 @@ datasources: access: proxy url: http://tempo:3200 jsonData: + # No tracesToMetrics/serviceMap/nodeGraph: all three need span-metrics series + # this stack does not generate (RED comes from the apps' own OTLP metrics, + # ADR 0002). Configured anyway, they render a permanently empty service graph + # that reads as "no traffic". tracesToLogsV2: datasourceUid: loki filterByTraceID: true tags: - key: service.name value: service_name - # No tracesToMetrics/serviceMap/nodeGraph: all three need span-metrics series - # this stack does not generate (RED comes from the apps' own OTLP metrics, - # ADR 0002). Configured anyway, they render a permanently empty service graph - # that reads as "no traffic". diff --git a/config/tempo.yaml b/config/tempo.yaml index bbdac09..5e9a1ce 100644 --- a/config/tempo.yaml +++ b/config/tempo.yaml @@ -30,7 +30,7 @@ storage: overrides: defaults: compaction: - block_retention: 168h # 7 days — traces are bulky, tune to disk budget + block_retention: 168h # 7 days — traces are bulky, tune to disk budget usage_report: reporting_enabled: false diff --git a/docs/ONBOARDING.md b/docs/ONBOARDING.md index 1e087a7..5a9fa98 100644 --- a/docs/ONBOARDING.md +++ b/docs/ONBOARDING.md @@ -9,7 +9,7 @@ project runs; a Python/FastAPI service needs no code changes at all. | | | | --- | --- | -| Production (via tunnel) | `https://otlp.` — OTLP **HTTP** (`http/protobuf`) only | +| Production (via tunnel) | `https://otel.` — OTLP **HTTP** (`http/protobuf`) only | | Private network / same host | `:4317` (gRPC) or `:4318` (HTTP) | | Auth | `Authorization: Bearer ` (ask the stack operator) | @@ -50,7 +50,7 @@ pip install opentelemetry-distro opentelemetry-exporter-otlp opentelemetry-instr ```sh export OTEL_SERVICE_NAME=my-service export OTEL_RESOURCE_ATTRIBUTES=env=prod -export OTEL_EXPORTER_OTLP_ENDPOINT=https://otlp.example.org +export OTEL_EXPORTER_OTLP_ENDPOINT=https://otel.example.org export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer " export OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED=true @@ -70,7 +70,7 @@ Every OpenTelemetry SDK understands the same four environment variables: ```sh OTEL_SERVICE_NAME=my-service -OTEL_EXPORTER_OTLP_ENDPOINT=https://otlp.example.org +OTEL_EXPORTER_OTLP_ENDPOINT=https://otel.example.org OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer " OTEL_RESOURCE_ATTRIBUTES=env=prod ``` diff --git a/docs/RUNBOOK.md b/docs/RUNBOOK.md index b18fab2..fd510fc 100644 --- a/docs/RUNBOOK.md +++ b/docs/RUNBOOK.md @@ -41,7 +41,7 @@ free space, or shorten a retention window and restart the affected service (`retention_period` in `config/loki.yaml`, `block_retention` in `config/tempo.yaml`, the `--storage.tsdb.retention.*` flags in `compose.yml`). If disk pressure keeps returning, move Loki and Tempo to -object storage (see `compose.storage-s3.yml`). +object storage (see the appendix of ADR 0001). Memory is bounded per service instead: every service carries a `mem_limit` in `compose.yml`, sized from observed usage so one runaway component cannot @@ -81,6 +81,12 @@ host is the intended setup. `OTEL_EXPORTER_OTLP_HEADERS`. Senders still on the old token get 401s (export errors on their side) until updated. A running demo overlay counts as a sender: re-run `just demo` to recreate it with the new token. + The token is shared and the collector does not verify `project`/`env` + against the sender, so every project host is trusted with every other + project's telemetry identity: a compromised host could spoof another + project's labels (and so quiet its silence alarm). Per-project tokens with + a collector-side identity check are the upgrade if that trust ever stops + being acceptable. - **Tunnel token:** the tunnel is OpenTofu-managed, so read the token back from there, not from the dashboard. Rotate the tunnel secret in Cloudflare Zero Trust, then `cd infra && tofu apply` (which refreshes the token data @@ -136,13 +142,14 @@ cd infra && tofu apply Zero Trust seat limit in the Cloudflare dashboard — Cloudflare does not document the free-plan figure. - **Per-user Grafana logins:** by default everyone who clears Access shares - the one admin password. Set `GRAFANA_JWT_AUTH=true` and - `CF_ACCESS_TEAM_DOMAIN=` in `.env` to make Grafana verify the Access - JWT instead: each address signs in as itself, and new ones land on the org's - default role (Viewer). The JWK set is team-wide. If the Zero Trust team - fronts more than one Access application, also pin - `GF_AUTH_JWT_EXPECT_CLAIMS` in `compose.yml` to this app's `aud` tag, or a - token minted for any other app in the team is accepted here too. + the one admin password. Set `GRAFANA_JWT_AUTH=true`, + `CF_ACCESS_TEAM_DOMAIN=`, and `CF_ACCESS_AUD` (from + `tofu output -raw grafana_access_aud`) in `.env` to make Grafana verify the + Access JWT instead: each address signs in as itself, and new ones land on + the org's default role (Viewer). The aud pin is required because the JWK + set is team-wide — without it a token minted for any other Access app in + the team would be accepted here too; the `just up` exposure guards refuse + to start JWT auth without both values. - **Adding a hostname:** add an `ingress` entry pointing at the service's container port, plus a matching `cloudflare_dns_record`. The catch-all `http_status:404` entry stays last, or it swallows everything after it. @@ -167,9 +174,9 @@ against the real account. Validation proves the syntax parses; only a plan proves the provider still maps the config to the same resources. Nothing watches the tool images pinned in the `justfile`: yamllint, -actionlint, OpenTofu, jq, and the alpine that backup, restore, and the -queue-volume setup run in. No Dependabot ecosystem covers a justfile, so -bump those by hand. +actionlint, shellcheck, ruff, gitleaks, OpenTofu, jq, the Alloy validator, +and the alpine that backup, restore, and the queue-volume setup run in. No +Dependabot ecosystem covers a justfile, so bump those by hand. After merging, on the host: `git pull && just pull && just up`. Use `just up`, not `docker compose up -d`: the recipe first chowns the collector's diff --git a/docs/adr/0001-observability-stack.md b/docs/adr/0001-observability-stack.md index bf0357c..a01e322 100644 --- a/docs/adr/0001-observability-stack.md +++ b/docs/adr/0001-observability-stack.md @@ -43,6 +43,48 @@ endpoints via Cloudflare Tunnel; bind everything else to `127.0.0.1`. fire-and-forget) and the stack rebuilds from this repo in minutes. - Local disk bounds retention (30d logs/metrics, 7d traces). The escape hatch, reached before any move to distributed ingest, is S3-compatible storage for - Loki and Tempo — see `compose.storage-s3.yml`. + Loki and Tempo — see the appendix below. - Every image is pinned and validated by `just check` in CI, so the stack stays reproducible. + +## Appendix: the S3 storage escape hatch + +When local volumes stop fitting, Loki and Tempo move their object storage to +any S3-compatible backend (Cloudflare R2, Backblaze B2, Hetzner, MinIO) +without touching the collector, Prometheus, or any client project. Not wired +up — don't start until credentials and a bucket exist. The concrete shape: + +1. Create s3 variants of the configs. `config/loki.s3.yaml` replaces + `common.storage.filesystem` with: + + ```yaml + common: + storage: + s3: + endpoint: ${S3_ENDPOINT} # e.g. .r2.cloudflarestorage.com + bucketnames: cml-loki + access_key_id: ${S3_ACCESS_KEY_ID} + secret_access_key: ${S3_SECRET_ACCESS_KEY} + s3forcepathstyle: true + ``` + + and `config/tempo.s3.yaml` replaces `storage.trace.backend: local` with: + + ```yaml + storage: + trace: + backend: s3 + s3: + endpoint: ${S3_ENDPOINT} + bucket: cml-tempo + access_key: ${S3_ACCESS_KEY_ID} + secret_key: ${S3_SECRET_ACCESS_KEY} + ``` + +2. Add a `compose.storage-s3.yml` overlay to `COMPOSE_FILE` in `.env` that + mounts the s3 config variants over the originals and re-declares each + service's `command` with `-config.expand-env=true` appended — neither Loki + nor Tempo expands `${...}` in its config by default, and compose replaces + `command` wholesale rather than merging it. Pass `S3_ENDPOINT`, + `S3_ACCESS_KEY_ID`, and `S3_SECRET_ACCESS_KEY` through each service's + `environment` with `:?` guards, then `just up`. diff --git a/templates/README.md b/templates/README.md index 1e1bf16..e217e4d 100644 --- a/templates/README.md +++ b/templates/README.md @@ -21,7 +21,7 @@ you need: the `.env` variables to paste on the project host, and the `curl` comm that vendor these files at a pinned tag. Then on the project host: ```sh -docker compose -f compose.yaml -f compose.telemetry.yml up -d +docker compose -f compose.yml -f compose.telemetry.yml up -d ``` Under an hour, most of it waiting for the first scrape. From cd03c9af88be2ceb096bb11faa75f2941a7cc44a Mon Sep 17 00:00:00 2001 From: Simon van Lierde Date: Sat, 5 Sep 2026 16:53:36 +0000 Subject: [PATCH 33/85] feat(edge): rename the ingestion hostname to otel. - point the tunnel ingress and DNS record at otel., one department-wide name for machine telemetry beside grafana. - rename the record with a moved block, so the CNAME is updated in place instead of destroyed and recreated with a resolution gap in between --- infra/main.tf | 17 ++++++++++++----- infra/terraform.tfvars.example | 2 +- templates/alloy/config.alloy | 2 +- 3 files changed, 14 insertions(+), 7 deletions(-) diff --git a/infra/main.tf b/infra/main.tf index 7c3b6b7..02acfc0 100644 --- a/infra/main.tf +++ b/infra/main.tf @@ -37,7 +37,7 @@ variable "zone_id" { variable "domain" { type = string - description = "Apex domain, e.g. example.org → grafana.example.org, otlp.example.org." + description = "Apex domain, e.g. example.org → grafana.example.org, otel.example.org." } variable "grafana_allowed_emails" { @@ -67,7 +67,7 @@ resource "cloudflare_zero_trust_tunnel_cloudflared_config" "monitoring" { }, { # OTLP HTTP ingestion; the collector enforces bearer-token auth. - hostname = "otlp.${var.domain}" + hostname = "otel.${var.domain}" service = "http://otel-collector:4318" }, { @@ -87,9 +87,16 @@ resource "cloudflare_dns_record" "grafana" { ttl = 1 } -resource "cloudflare_dns_record" "otlp" { +# Renamed from `otlp` (2026-09): `moved` keeps the existing record and renames it +# in place, instead of destroying and recreating it with a gap in between. +moved { + from = cloudflare_dns_record.otlp + to = cloudflare_dns_record.otel +} + +resource "cloudflare_dns_record" "otel" { zone_id = var.zone_id - name = "otlp.${var.domain}" + name = "otel.${var.domain}" type = "CNAME" content = "${cloudflare_zero_trust_tunnel_cloudflared.monitoring.id}.cfargotunnel.com" proxied = true @@ -98,7 +105,7 @@ resource "cloudflare_dns_record" "otlp" { # Cloudflare Access in front of Grafana: email one-time-PIN at the edge, so # the public hostname never reaches Grafana's login page unauthenticated. -# The OTLP hostname is NOT behind Access — machines authenticate with the +# The ingestion hostname is NOT behind Access — machines authenticate with the # bearer token instead. resource "cloudflare_zero_trust_access_application" "grafana" { account_id = var.account_id diff --git a/infra/terraform.tfvars.example b/infra/terraform.tfvars.example index 5ef4f33..62a40de 100644 --- a/infra/terraform.tfvars.example +++ b/infra/terraform.tfvars.example @@ -8,7 +8,7 @@ account_id = "your-account-id" # Zone ID of the domain the hostnames live under (same overview sidebar). zone_id = "your-zone-id" -# Apex domain → grafana., otlp.. +# Apex domain → grafana., otel.. domain = "example.org" # Emails allowed through Cloudflare Access to Grafana (one-time PIN). diff --git a/templates/alloy/config.alloy b/templates/alloy/config.alloy index 5d0f0dc..c4073f3 100644 --- a/templates/alloy/config.alloy +++ b/templates/alloy/config.alloy @@ -11,7 +11,7 @@ // Everything else — databases, caches, proxies, frontends, batch jobs — only writes to // stdout, so this agent picks those up and forwards them on the same OTLP endpoint with // the same token. Nothing new is published at the edge: the monitoring stack exposes -// `otlp.` and does not expose Loki, which has no authentication of its own. +// `otel.` and does not expose Loki, which has no authentication of its own. // The host's own name, read from its /etc/hostname through the rootfs mount rather than // taken as an operator input: one less thing to set per host. The container's own From 2917479bcc9773b6b5a1a757d01ee73459c106e6 Mon Sep 17 00:00:00 2001 From: Simon van Lierde Date: Sat, 5 Sep 2026 16:53:42 +0000 Subject: [PATCH 34/85] feat(telemetry): label every signal with its department - stamp department at the gateway from DEPARTMENT in .env, upserted so a spoke cannot ship telemetry attributed to another department - promote it to a Prometheus label and index it as a Loki stream label, alongside the identity labels already carried - pass DEPARTMENT to the collector's config validation in just check --- .env.example | 4 ++++ compose.yml | 1 + config/loki.yaml | 8 +++++--- config/otel-collector.yaml | 17 ++++++++++++++--- config/prometheus.yaml | 2 +- justfile | 2 +- 6 files changed, 26 insertions(+), 8 deletions(-) diff --git a/.env.example b/.env.example index 6aaeb7f..72678fc 100644 --- a/.env.example +++ b/.env.example @@ -19,6 +19,10 @@ GRAFANA_ROOT_URL=http://localhost:3000 # Keep false for plain-http localhost use, or logins break. GRAFANA_COOKIE_SECURE=false +# Department this stack collects for. Stamped onto every signal as the `department` +# label, so telemetry stays attributable once a second department runs its own stack. +DEPARTMENT=cml + # Bearer token every telemetry sender must present (Authorization: Bearer ). # The default only suits local use — generate a real one for production, e.g.: openssl rand -hex 32 OTLP_AUTH_TOKEN=local-dev-token diff --git a/compose.yml b/compose.yml index 7ee2d0d..1b803da 100644 --- a/compose.yml +++ b/compose.yml @@ -51,6 +51,7 @@ services: command: ["--config=/etc/otelcol/config.yaml"] environment: OTLP_AUTH_TOKEN: ${OTLP_AUTH_TOKEN:?set OTLP_AUTH_TOKEN in .env} + DEPARTMENT: ${DEPARTMENT:-cml} # Keep in step with memory_limiter in config/otel-collector.yaml. mem_limit: 512m cap_drop: [ALL] diff --git a/config/loki.yaml b/config/loki.yaml index 0cc65fa..539aa3d 100644 --- a/config/loki.yaml +++ b/config/loki.yaml @@ -46,16 +46,18 @@ limits_config: otlp_config: resource_attributes: # ignore_defaults keeps Loki from indexing every resource attribute it is - # handed. These four are the identity labels ADR 0002 requires on every + # handed. These five are the identity labels ADR 0002 requires on every # signal, and they have to be real stream LABELS, not structured metadata: # a `project` template variable is a stream selector, and label_values() - # cannot see structured metadata. All four are bounded — one value per - # project, environment and host — so indexing them costs nothing. + # cannot see structured metadata. All five are bounded — one value per + # department, project, environment and host — so indexing them costs + # nothing. ignore_defaults: true attributes_config: - action: index_label attributes: - service.name + - department - project - env - host.name diff --git a/config/otel-collector.yaml b/config/otel-collector.yaml index 154a70c..9dcf399 100644 --- a/config/otel-collector.yaml +++ b/config/otel-collector.yaml @@ -27,6 +27,17 @@ receivers: authenticator: bearertokenauth processors: + # The department every signal belongs to. Stamped here rather than at the app + # edge: it is the same value for everything this gateway accepts, and upsert + # means a sender cannot claim someone else's department. project, env and + # service.name still come from the sender, which is the only thing that knows + # them. + resource/department: + attributes: + - key: department + value: ${env:DEPARTMENT} + action: upsert + # Absolute, not percentage: percentages are read against the cgroup, and an # unbounded container makes that the whole host. Sized to the container's # mem_limit (512m in compose.yml) — change both together. @@ -72,15 +83,15 @@ service: pipelines: logs: receivers: [otlp] - processors: [memory_limiter, batch] + processors: [memory_limiter, resource/department, batch] exporters: [otlp_http/loki] traces: receivers: [otlp] - processors: [memory_limiter, batch] + processors: [memory_limiter, resource/department, batch] exporters: [otlp_grpc/tempo] metrics: receivers: [otlp] - processors: [memory_limiter, batch] + processors: [memory_limiter, resource/department, batch] exporters: [otlp_http/prometheus] telemetry: metrics: diff --git a/config/prometheus.yaml b/config/prometheus.yaml index 4e634a4..73f19f4 100644 --- a/config/prometheus.yaml +++ b/config/prometheus.yaml @@ -15,7 +15,7 @@ otlp: # so alerts and dashboards see no project/env/host. Promoting them puts the # identity labels ADR 0002 requires on every series. service.name and # service.instance.id already become job/instance without being listed. - promote_resource_attributes: [project, env, host.name] + promote_resource_attributes: [department, project, env, host.name] # No rule_files and no alerting block: Grafana owns both evaluation and # notification (ADR 0002). Prometheus is a store and a query engine here. diff --git a/justfile b/justfile index b871200..9bb73a0 100644 --- a/justfile +++ b/justfile @@ -127,7 +127,7 @@ check: {{compose_smoke}} config -q images="$(docker compose config --images)" && \ docker run --rm --network none -v ./config/prometheus.yaml:/etc/prometheus/prometheus.yaml:ro --entrypoint promtool $(echo "$images" | grep prom/prometheus) check config /etc/prometheus/prometheus.yaml && \ - docker run --rm --network none -e OTLP_AUTH_TOKEN=dummy -v ./config/otel-collector.yaml:/etc/otelcol/config.yaml:ro $(echo "$images" | grep opentelemetry-collector) validate --config=/etc/otelcol/config.yaml && \ + docker run --rm --network none -e OTLP_AUTH_TOKEN=dummy -e DEPARTMENT=dummy -v ./config/otel-collector.yaml:/etc/otelcol/config.yaml:ro $(echo "$images" | grep opentelemetry-collector) validate --config=/etc/otelcol/config.yaml && \ docker run --rm --network none -v ./config/loki.yaml:/etc/loki/loki.yaml:ro $(echo "$images" | grep grafana/loki) -config.file=/etc/loki/loki.yaml -verify-config && \ docker run --rm --network none -v ./config/tempo.yaml:/etc/tempo/tempo.yaml:ro $(echo "$images" | grep grafana/tempo) -config.file=/etc/tempo/tempo.yaml -config.verify=true # line-length at 120, not the default 80: digest-pinned image refs need From 02436dbefd01dd1e3b33d2907860ed5b288c83b4 Mon Sep 17 00:00:00 2001 From: Simon van Lierde Date: Sat, 5 Sep 2026 16:53:49 +0000 Subject: [PATCH 35/85] docs: record the otel. hostname and the department label - ADR 0002 now lists five identity labels, noting department comes from the hub rather than the agent config - update the onboarding endpoint and Loki's index-label list --- CHANGELOG.md | 19 ++++++++++++++----- docs/ONBOARDING.md | 2 +- docs/adr/0002-hub-and-spoke-observability.md | 8 +++++--- 3 files changed, 20 insertions(+), 9 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 0f1652c..3d3d4c4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -47,15 +47,24 @@ queries that had been measuring the wrong thing. ### Changed +- **Ingestion hostname is `otel.`**, not `otlp.` — one + department-wide name for machine telemetry alongside `grafana.` for + humans. The tunnel ingress and the DNS record move together (a `moved` block + renames the record in place); every spoke's `OTEL_EXPORTER_OTLP_ENDPOINT` and + any edge rule matching the old host have to follow. +- **`department` on every signal**: the gateway collector stamps it from + `DEPARTMENT` in `.env` and Prometheus and Loki carry it as an identity label. + It is set at the hub rather than by the sender, so a spoke cannot ship + telemetry attributed to someone else. - **Overlays are host config now**: `COMPOSE_FILE` in `.env` names the compose file set, and every recipe — `up`, `logs`, `ps`, `backup` — acts on that same set. `just up-tunnel` is gone; its exposure guards run automatically whenever the tunnel overlay is active. -- **Loki indexes only the identity labels** (`service.name`, `project`, `env`, - `host.name` — the authoritative list lives in `config/loki.yaml`). Everything - else, `service.instance.id` included, is structured metadata now — one stream - per service instead of one per sender restart. Existing streams keep their - old labels until they age out (30 days). +- **Loki indexes only the identity labels** (`service.name`, `department`, + `project`, `env`, `host.name` — the authoritative list lives in + `config/loki.yaml`). Everything else, `service.instance.id` included, is + structured metadata now — one stream per service instead of one per sender + restart. Existing streams keep their old labels until they age out (30 days). - **Dashboards are provisioned, not editable**: `dashboards/*.json` is mounted read-only and UI saves are off, making the files the source of truth. - The collector's `memory_limiter` is sized in absolute MiB against the diff --git a/docs/ONBOARDING.md b/docs/ONBOARDING.md index 5a9fa98..451c7d9 100644 --- a/docs/ONBOARDING.md +++ b/docs/ONBOARDING.md @@ -30,7 +30,7 @@ directly. or in span attributes: still filterable at query time, without the storage blowing up. - **In Loki, only the identity labels are index labels**: `service.name`, - `project`, `env`, `host.name` (the authoritative list lives in + `department`, `project`, `env`, `host.name` (the authoritative list lives in `config/loki.yaml`). Every other attribute, `service.instance.id` included, is stored as structured metadata, so a query starts from the stream selector and filters after it: diff --git a/docs/adr/0002-hub-and-spoke-observability.md b/docs/adr/0002-hub-and-spoke-observability.md index f5022a5..b08a4dc 100644 --- a/docs/adr/0002-hub-and-spoke-observability.md +++ b/docs/adr/0002-hub-and-spoke-observability.md @@ -40,9 +40,11 @@ Three tiers, each owning distinct signals: Contracts that make it scale: -- **Four identity labels on every signal**, enforced by the shared agent config: - `project`, `env`, `service.name`, `host_name` (Prometheus form; OTel form is - `host.name`). Cardinality rule: user ids, request ids and timestamps go in +- **Five identity labels on every signal**: `project`, `env`, `service.name` and + `host_name` (Prometheus form; OTel form is `host.name`) come from the shared + agent config, and `department` is stamped by the gateway collector, which is + the only component that knows it and the only one a sender cannot override. + Cardinality rule: user ids, request ids and timestamps go in bodies and span attributes, never labels. - **Each signal has exactly one producer.** Alloy owns all container logs (SDK log exporters stay off); native app metrics own RED; cAdvisor owns container From 1f5a3103b1b32d02e2f93902e0763c0eaea7781c Mon Sep 17 00:00:00 2001 From: Simon van Lierde Date: Sun, 6 Sep 2026 13:08:22 +0000 Subject: [PATCH 36/85] feat(infra): add script to generate OpenTofu import blocks for existing Cloudflare resources - infra/generate-imports.sh emits import blocks for the tunnel, its config, both DNS records, and the Access app and policy when they exist - main.tf: update comments for adopting an existing tunnel and renaming the ingestion DNS record - gitignore the throwaway infra/imports.tf --- .gitignore | 1 + CHANGELOG.md | 12 ++- infra/generate-imports.sh | 181 ++++++++++++++++++++++++++++++++++++++ infra/main.tf | 19 ++-- justfile | 2 +- 5 files changed, 202 insertions(+), 13 deletions(-) create mode 100755 infra/generate-imports.sh diff --git a/.gitignore b/.gitignore index 6968e63..666d257 100644 --- a/.gitignore +++ b/.gitignore @@ -4,5 +4,6 @@ backups/ infra/.terraform/ infra/terraform.tfstate* +infra/imports.tf infra/*.tfvars !infra/*.tfvars.example diff --git a/CHANGELOG.md b/CHANGELOG.md index 3d3d4c4..d6c6a8a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -45,13 +45,19 @@ queries that had been measuring the wrong thing. - Dependabot now watches the Cloudflare provider in `infra/`, and the runbook covers OpenTofu-managed tunnel and Access changes. +### Added + +- **`infra/generate-imports.sh`**: emits OpenTofu `import` blocks for the edge + built by hand in the Zero Trust dashboard. Without it the first plan against + an empty state reads "create" for objects already serving traffic, and + applying it mints a second tunnel and a duplicate Access app. + ### Changed - **Ingestion hostname is `otel.`**, not `otlp.` — one department-wide name for machine telemetry alongside `grafana.` for - humans. The tunnel ingress and the DNS record move together (a `moved` block - renames the record in place); every spoke's `OTEL_EXPORTER_OTLP_ENDPOINT` and - any edge rule matching the old host have to follow. + humans. Every spoke's `OTEL_EXPORTER_OTLP_ENDPOINT` and any edge rule matching + the old host have to follow. - **`department` on every signal**: the gateway collector stamps it from `DEPARTMENT` in `.env` and Prometheus and Loki carry it as an identity label. It is set at the hub rather than by the sender, so a spoke cannot ship diff --git a/infra/generate-imports.sh b/infra/generate-imports.sh new file mode 100755 index 0000000..0b9c808 --- /dev/null +++ b/infra/generate-imports.sh @@ -0,0 +1,181 @@ +#!/usr/bin/env bash +# Emit OpenTofu `import` blocks for the Cloudflare resources that already exist. +# +# This root was added after the edge already existed, so its first plan against an +# empty state says "create" for objects that are already serving traffic. Applying +# that plan mints a SECOND tunnel beside `cml-monitoring` and DNS records that fight +# the live ones. Import blocks make the adoption reviewable: you read the generated +# file, then the plan, and only then apply. +# +# The ingestion record is the one asymmetry. Live, it is still `otlp.`; this +# root now calls it `otel.`. The block below imports that record as +# `cloudflare_dns_record.otel`, so the apply RENAMES the record in place instead of +# creating a second one — which is also why there is no `moved` block in main.tf: +# nothing was ever in state under the old resource name. +# +# The generated imports.tf is a throwaway, NOT something to commit: it names one +# account's resource ids and is meaningless after the apply that consumes it. Delete +# it once the apply has succeeded — import blocks are a one-time instruction, and +# leaving them in place re-runs them on every plan. +# +# Usage (run in this directory, with terraform.tfvars already filled in): +# export CLOUDFLARE_API_TOKEN=... # Tunnel:Read, DNS:Read, Access: Apps and Policies:Read +# ./generate-imports.sh > imports.tf +# tofu plan # expect "0 to add", only in-place changes +# tofu apply && rm imports.tf +set -euo pipefail + +cd -- "$(dirname -- "${BASH_SOURCE[0]}")" + +die() { + echo "error: $*" >&2 + exit 1 +} + +api() { + # The token rides a curl config read from stdin, not argv: /proc//cmdline is + # readable by every local user for the life of each call. + curl -fsS --config - "https://api.cloudflare.com/client/v4/$1" \ + <<<"header = \"Authorization: Bearer ${CLOUDFLARE_API_TOKEN}\"" +} + +command -v jq >/dev/null || die "jq is required" +: "${CLOUDFLARE_API_TOKEN:?is not set}" + +# Read the ids from terraform.tfvars rather than asking for them again as TF_VAR_*: +# they are already there, and a second source of the same value is a second chance to +# get it wrong. An environment variable still wins if one is exported. +[[ -f terraform.tfvars ]] || die "no terraform.tfvars here; copy terraform.tfvars.example and fill it in" +tfvar() { + sed -n "s/^[[:space:]]*$1[[:space:]]*=[[:space:]]*\"\([^\"]*\)\".*/\1/p" terraform.tfvars | tail -1 +} +account="${TF_VAR_account_id:-$(tfvar account_id)}" +zone="${TF_VAR_zone_id:-$(tfvar zone_id)}" +domain="${TF_VAR_domain:-$(tfvar domain)}" + +# The example file's placeholders are valid-looking strings, and a plan run against +# them is what makes a fresh-create plan look plausible. Catch them here, where the +# message can say which line to edit. +for pair in "account_id:$account" "zone_id:$zone" "domain:$domain"; do + value="${pair#*:}" + [[ -n "$value" ]] || die "${pair%%:*} is empty in terraform.tfvars" + case "$value" in + your-* | example.org) die "${pair%%:*} is still the placeholder from terraform.tfvars.example" ;; + esac +done + +# One lookup per resource kind. Each fails loudly when the resource is absent: a +# silently skipped import comes back as a "create" in the plan, which is the exact +# outcome this script exists to prevent. +lookup_dns_record() { + local hostname="$1" id + # Type-filtered: this root manages CNAMEs, and a name can also carry TXT records. + # An unfiltered .result[0] could bind one of those, and the apply would rewrite it + # into a proxied CNAME — destroying a TXT record and leaving the CNAME unmanaged. + id="$(api "zones/$zone/dns_records?name=$hostname&type=CNAME" | jq -r '.result[0].id // empty')" + [[ -n "$id" ]] || return 1 + printf '%s' "$id" +} + +tunnel_name="${MONITORING_TUNNEL_NAME:-cml-monitoring}" +tunnels="$(api "accounts/$account/cfd_tunnel?is_deleted=false")" +tunnel_id="$(jq -r --arg name "$tunnel_name" '.result[] | select(.name == $name) | .id' <<<"$tunnels" | head -1)" +if [[ -z "$tunnel_id" ]]; then + # Listing what IS there turns a wrong guess about the name from a dead end into a + # one-line fix: the tunnel is rarely absent, it is just called something else. + echo "error: no tunnel named $tunnel_name in account $account" >&2 + echo "tunnels that do exist in this account:" >&2 + jq -r '.result[]? | " \(.name)\t\(.id)\tconnections=\(.connections | length)"' <<<"$tunnels" >&2 + echo "Re-run with MONITORING_TUNNEL_NAME='' once you know which one serves monitoring." >&2 + exit 1 +fi + +grafana_record="$(lookup_dns_record "grafana.$domain")" || die "no CNAME found for grafana.$domain" + +# Still otlp. before the rename, otel. after it (or on a re-run). Try the new name +# first so a second run is a no-op rather than a resurrection of the old record. +for host in "otel.$domain" "otlp.$domain"; do + if otel_record="$(lookup_dns_record "$host")"; then + ingestion_host="$host" + break + fi +done +[[ -n "${ingestion_host:-}" ]] || die "no CNAME found for otel.$domain or otlp.$domain" + +# Unlike the tunnel and the DNS records, the Access app may legitimately not exist — +# and then the apply SHOULD create it, because an unprotected Grafana hostname is the +# thing this root exists to close. So: warn, omit the import, let the plan create it. +# +# Checked at both scopes. Apps predating account-scoped Access live under the zone, +# and one found there cannot be adopted by this resource as written (it is configured +# with account_id), so that case gets its own message rather than a silent create. +access_apps="$(api "accounts/$account/access/apps?per_page=100")" +access_app_id="$(jq -r --arg d "grafana.$domain" '.result[] | select(.domain == $d) | .id' <<<"$access_apps" | head -1)" +if [[ -z "$access_app_id" ]]; then + zone_app_id="$(api "zones/$zone/access/apps?per_page=100" \ + | jq -r --arg d "grafana.$domain" '.result[]? | select(.domain == $d) | .id' | head -1)" || zone_app_id="" + if [[ -n "$zone_app_id" ]]; then + die "the Access app on grafana.$domain is zone-scoped ($zone_app_id); this root declares an + account-scoped one. Recreate it at the account level, or give the resource + zone_id instead of account_id, before importing." + fi + echo "note: no Access application on grafana.$domain — omitting its import block so the" >&2 + echo " apply CREATES it. Until that apply lands, Grafana's hostname is protected only" >&2 + echo " by its own login. Access apps that do exist in this account:" >&2 + jq -r '.result[]? | " \(.name)\t\(.domain)"' <<<"$access_apps" >&2 +fi + +# The policy is the one resource that may legitimately be absent: an app built in the +# dashboard usually carries an INLINE policy, which has no id to import. The apply +# then creates the reusable policy this root declares and reattaches the app to it. +access_policies="$(api "accounts/$account/access/policies")" +access_policy_id="$(jq -r '.result[] | select(.name == "monitoring: allowed emails") | .id' <<<"$access_policies" | head -1)" + +echo "# Generated by generate-imports.sh on $(date -u +%Y-%m-%dT%H:%M:%SZ). Delete after a successful apply." +echo "# Ingestion record adopted from $ingestion_host as cloudflare_dns_record.otel." +cat <&2 + echo " import block so the apply creates it. Check the plan's include{} list against" >&2 + echo " the emails the live app admits before applying." >&2 +fi diff --git a/infra/main.tf b/infra/main.tf index 02acfc0..501bb2f 100644 --- a/infra/main.tf +++ b/infra/main.tf @@ -5,7 +5,10 @@ # Bootstrap (owner-run, once): # cp terraform.tfvars.example terraform.tfvars # then fill it in # export CLOUDFLARE_API_TOKEN=... # needs Tunnel:Edit, DNS:Edit, Access:Edit -# cd infra && tofu init && tofu apply +# cd infra && tofu init +# ./generate-imports.sh > imports.tf # the edge already exists: adopt it first +# tofu plan # expect "0 to add" — see the script's header +# tofu apply && rm imports.tf # tofu output -raw tunnel_token # → CLOUDFLARE_TUNNEL_TOKEN in ../.env # # State is local (infra/terraform.tfstate, gitignored) — one host, one @@ -51,7 +54,9 @@ variable "grafana_allowed_emails" { resource "cloudflare_zero_trust_tunnel_cloudflared" "monitoring" { account_id = var.account_id - name = "monitoring" + # Must match the live tunnel's name: this root adopts it rather than creating it + # (see generate-imports.sh), and a different name here renames it on apply. + name = "cml-monitoring" config_src = "cloudflare" } @@ -87,13 +92,9 @@ resource "cloudflare_dns_record" "grafana" { ttl = 1 } -# Renamed from `otlp` (2026-09): `moved` keeps the existing record and renames it -# in place, instead of destroying and recreating it with a gap in between. -moved { - from = cloudflare_dns_record.otlp - to = cloudflare_dns_record.otel -} - +# Renamed from `otlp` (2026-09). There is no `moved` block because nothing was ever +# in state under the old name: the live record is adopted straight into this address +# by ./generate-imports.sh, and the apply then renames it in place. resource "cloudflare_dns_record" "otel" { zone_id = var.zone_id name = "otel.${var.domain}" diff --git a/justfile b/justfile index 9bb73a0..3922f19 100644 --- a/justfile +++ b/justfile @@ -136,7 +136,7 @@ check: # grow with every onboarded project. docker run --rm --network none -v .:/code:ro pipelinecomponents/yamllint:0.35.13 yamllint -d '{extends: relaxed, rules: {line-length: {max: 120, allow-non-breakable-inline-mappings: true}}, ignore: [.git/, backups/, infra/.terraform/, config/grafana/alerting/project-*.yaml, config/grafana/alerting/coverage.yaml]}' . docker run --rm --network none -v .:/repo:ro -w /repo rhysd/actionlint:1.7.12 -color - docker run --rm --network none -v .:/mnt:ro koalaman/shellcheck:v0.11.0 bootstrap.sh templates/run_scheduled.sh + docker run --rm --network none -v .:/mnt:ro koalaman/shellcheck:v0.11.0 bootstrap.sh templates/run_scheduled.sh infra/generate-imports.sh docker run --rm --network none -v ./demo:/demo:ro ghcr.io/astral-sh/ruff:0.14.2 check --no-cache /demo docker run --rm --network none -v ./infra:/infra:ro -w /infra ghcr.io/opentofu/opentofu:1.12.3 fmt -check docker run --rm --network none -v ./dashboards:/dashboards:ro ghcr.io/jqlang/jq:1.8.1 empty {{dash_paths}} From a39904bb20cfecfdd1fff726f67254a05cc7bd57 Mon Sep 17 00:00:00 2001 From: Simon van Lierde Date: Sun, 6 Sep 2026 14:12:22 +0000 Subject: [PATCH 37/85] fix(templates): declare the egress network the spoke overlay joins --- templates/compose.telemetry.yml | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/templates/compose.telemetry.yml b/templates/compose.telemetry.yml index f684577..d1797ff 100644 --- a/templates/compose.telemetry.yml +++ b/templates/compose.telemetry.yml @@ -147,6 +147,10 @@ volumes: alloy_data: networks: + # Alloy's way out to the hub, and where the GPU exporter overlay attaches. Declared + # here so the overlay stands on its own; a project that already defines `egress` in + # its compose.yml keeps its definition, Compose merges the two by name. + egress: # Carries only the proxied Docker API between the proxy and Alloy. Internal: it has # no gateway, so nothing on it can reach out and nothing outside can reach in. docker_socket: From 00987bb796f74a369c19d87938113f8064eb1b22 Mon Sep 17 00:00:00 2001 From: Simon van Lierde Date: Sun, 6 Sep 2026 14:13:36 +0000 Subject: [PATCH 38/85] fix(infra): scope the Access app import id to accounts/ --- infra/generate-imports.sh | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/infra/generate-imports.sh b/infra/generate-imports.sh index 0b9c808..06d0328 100755 --- a/infra/generate-imports.sh +++ b/infra/generate-imports.sh @@ -159,9 +159,10 @@ EOF if [[ -n "$access_app_id" ]]; then cat <//'. import { to = cloudflare_zero_trust_access_application.grafana - id = "$account/$access_app_id" + id = "accounts/$account/$access_app_id" } EOF fi From d919a3bf470ca6f24f771078c9edfec13b359bee Mon Sep 17 00:00:00 2001 From: Simon van Lierde Date: Sun, 6 Sep 2026 14:13:37 +0000 Subject: [PATCH 39/85] feat(security): isolate the backends on an internal network, add pids limits - Loki, Tempo, Prometheus and node-exporter move to an internal backend network; Grafana and the collector bridge it, cloudflared stays on default - pids_limit on every hub service and on cloudflared - Dependabot watches the spoke images pinned in templates/ --- .github/dependabot.yml | 7 +++++++ compose.tunnel.yml | 1 + compose.yml | 26 +++++++++++++++++++++++--- 3 files changed, 31 insertions(+), 3 deletions(-) diff --git a/.github/dependabot.yml b/.github/dependabot.yml index 444dc1f..eaa3791 100644 --- a/.github/dependabot.yml +++ b/.github/dependabot.yml @@ -8,6 +8,13 @@ updates: schedule: interval: "weekly" + # The spoke images every project host runs (Alloy, socket proxy, GPU + # exporter). Digest pins keep them reproducible, not patched. + - package-ecosystem: "docker-compose" + directory: "/templates" + schedule: + interval: "weekly" + - package-ecosystem: "docker" directory: "/demo" schedule: diff --git a/compose.tunnel.yml b/compose.tunnel.yml index b33b7e4..fe02955 100644 --- a/compose.tunnel.yml +++ b/compose.tunnel.yml @@ -19,6 +19,7 @@ services: TUNNEL_TOKEN: ${CLOUDFLARE_TUNNEL_TOKEN:?set CLOUDFLARE_TUNNEL_TOKEN in .env} # Ceiling sized from observed usage with headroom; see compose.yml. mem_limit: 256m + pids_limit: 256 cap_drop: [ALL] security_opt: - no-new-privileges:true diff --git a/compose.yml b/compose.yml index 1b803da..7b48cdf 100644 --- a/compose.yml +++ b/compose.yml @@ -21,9 +21,9 @@ x-logging: &default-logging x-security: &default-security - no-new-privileges:true -# Every service also drops all capabilities: they all run unprivileged uids, -# write only to named volumes, and need none. (The client templates already -# ship this — see templates/compose.telemetry.yml.) +# Every service also drops all capabilities and carries a pids limit: they all +# run unprivileged uids, write only to named volumes, and need none. The client +# templates go further (read-only roots) — see templates/compose.telemetry.yml. # The mem_limit on each service is sized from observed steady-state usage with # generous headroom for bursts. They are ceilings, not reservations: the point @@ -54,8 +54,10 @@ services: DEPARTMENT: ${DEPARTMENT:-cml} # Keep in step with memory_limiter in config/otel-collector.yaml. mem_limit: 512m + pids_limit: 512 cap_drop: [ALL] security_opt: *default-security + networks: [default, backend] logging: *default-logging loki: @@ -68,8 +70,10 @@ services: # No healthcheck: the image is distroless, so there is no shell, wget or # curl to probe /ready with. Grafana's Stack Health dashboard covers it. mem_limit: 2g + pids_limit: 1024 cap_drop: [ALL] security_opt: *default-security + networks: [backend] logging: *default-logging tempo: @@ -81,8 +85,10 @@ services: command: ["-config.file=/etc/tempo/tempo.yaml"] # Distroless too — see the note on loki. mem_limit: 2g + pids_limit: 1024 cap_drop: [ALL] security_opt: *default-security + networks: [backend] logging: *default-logging prometheus: @@ -107,8 +113,10 @@ services: retries: 5 start_period: 30s mem_limit: 2g + pids_limit: 1024 cap_drop: [ALL] security_opt: *default-security + networks: [backend] logging: *default-logging node-exporter: @@ -128,6 +136,7 @@ services: # No network_mode:host either, so node_network_* describes the container # veth, not host NICs — fine while nothing alerts on network metrics. - /:/host:ro + networks: [backend] logging: *default-logging grafana: @@ -186,10 +195,21 @@ services: retries: 5 start_period: 30s mem_limit: 1g + pids_limit: 1024 cap_drop: [ALL] security_opt: *default-security + networks: [default, backend] logging: *default-logging +# Loki, Prometheus and Tempo have no authentication of their own, so they live +# on a network with no gateway that only Grafana and the collector can reach. +# cloudflared (compose.tunnel.yml) stays on `default`: its ingress list is +# fetched from Cloudflare at runtime, and this is what keeps an edited ingress +# from publishing a backend the token gate on the collector never sees. +networks: + backend: + internal: true + volumes: loki_data: tempo_data: From 6cbc62ea353969d2833066c7fcf78d6047f161f0 Mon Sep 17 00:00:00 2001 From: Simon van Lierde Date: Sun, 6 Sep 2026 14:13:37 +0000 Subject: [PATCH 40/85] fix(dashboards): key the GPU host picker on host_name, refresh variables on load --- dashboards/gpu.json | 114 ++++++++++++++++---------------- dashboards/host-containers.json | 6 +- dashboards/logs.json | 6 +- dashboards/service-health.json | 6 +- 4 files changed, 66 insertions(+), 66 deletions(-) diff --git a/dashboards/gpu.json b/dashboards/gpu.json index 3585d30..90f570d 100644 --- a/dashboards/gpu.json +++ b/dashboards/gpu.json @@ -130,7 +130,7 @@ "uid": "prometheus" }, "editorMode": "code", - "expr": "nvidia_smi_gpu_recovery_action{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}", + "expr": "nvidia_smi_gpu_recovery_action{uuid=\"$gpu\", host_name=\"$node\", job=\"$job\"}", "instant": false, "legendFormat": "__auto", "range": true, @@ -230,7 +230,7 @@ "type": "prometheus", "uid": "prometheus" }, - "expr": "(sum by(uuid) (nvidia_smi_clocks_event_reasons_sw_power_cap{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"} or nvidia_smi_clocks_throttle_reasons_sw_power_cap{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}) or (sum by(uuid) (nvidia_smi_gpu_info{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}) * 0)) + (sum by(uuid) (nvidia_smi_clocks_event_reasons_sw_thermal_slowdown{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"} or nvidia_smi_clocks_throttle_reasons_sw_thermal_slowdown{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}) or (sum by(uuid) (nvidia_smi_gpu_info{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}) * 0)) + (sum by(uuid) (nvidia_smi_clocks_event_reasons_hw_thermal_slowdown{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"} or nvidia_smi_clocks_throttle_reasons_hw_thermal_slowdown{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}) or (sum by(uuid) (nvidia_smi_gpu_info{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}) * 0)) + (sum by(uuid) (nvidia_smi_clocks_event_reasons_hw_power_brake_slowdown{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"} or nvidia_smi_clocks_throttle_reasons_hw_power_brake_slowdown{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}) or (sum by(uuid) (nvidia_smi_gpu_info{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}) * 0))", + "expr": "(sum by(uuid) (nvidia_smi_clocks_event_reasons_sw_power_cap{uuid=\"$gpu\", host_name=\"$node\", job=\"$job\"} or nvidia_smi_clocks_throttle_reasons_sw_power_cap{uuid=\"$gpu\", host_name=\"$node\", job=\"$job\"}) or (sum by(uuid) (nvidia_smi_gpu_info{uuid=\"$gpu\", host_name=\"$node\", job=\"$job\"}) * 0)) + (sum by(uuid) (nvidia_smi_clocks_event_reasons_sw_thermal_slowdown{uuid=\"$gpu\", host_name=\"$node\", job=\"$job\"} or nvidia_smi_clocks_throttle_reasons_sw_thermal_slowdown{uuid=\"$gpu\", host_name=\"$node\", job=\"$job\"}) or (sum by(uuid) (nvidia_smi_gpu_info{uuid=\"$gpu\", host_name=\"$node\", job=\"$job\"}) * 0)) + (sum by(uuid) (nvidia_smi_clocks_event_reasons_hw_thermal_slowdown{uuid=\"$gpu\", host_name=\"$node\", job=\"$job\"} or nvidia_smi_clocks_throttle_reasons_hw_thermal_slowdown{uuid=\"$gpu\", host_name=\"$node\", job=\"$job\"}) or (sum by(uuid) (nvidia_smi_gpu_info{uuid=\"$gpu\", host_name=\"$node\", job=\"$job\"}) * 0)) + (sum by(uuid) (nvidia_smi_clocks_event_reasons_hw_power_brake_slowdown{uuid=\"$gpu\", host_name=\"$node\", job=\"$job\"} or nvidia_smi_clocks_throttle_reasons_hw_power_brake_slowdown{uuid=\"$gpu\", host_name=\"$node\", job=\"$job\"}) or (sum by(uuid) (nvidia_smi_gpu_info{uuid=\"$gpu\", host_name=\"$node\", job=\"$job\"}) * 0))", "legendFormat": "", "refId": "A" } @@ -313,7 +313,7 @@ "type": "prometheus", "uid": "prometheus" }, - "expr": "sum by(uuid) (nvidia_smi_ecc_errors_uncorrected_aggregate_total{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}) or sum by(uuid) ({__name__=~\"nvidia_smi_ecc_errors_uncorrected_aggregate_.+\", uuid=\"$gpu\", instance=\"$node\", job=\"$job\"})", + "expr": "sum by(uuid) (nvidia_smi_ecc_errors_uncorrected_aggregate_total{uuid=\"$gpu\", host_name=\"$node\", job=\"$job\"}) or sum by(uuid) ({__name__=~\"nvidia_smi_ecc_errors_uncorrected_aggregate_.+\", uuid=\"$gpu\", host_name=\"$node\", job=\"$job\"})", "legendFormat": "", "refId": "A" } @@ -482,7 +482,7 @@ "uid": "prometheus" }, "exemplar": true, - "expr": "nvidia_smi_pstate{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}", + "expr": "nvidia_smi_pstate{uuid=\"$gpu\", host_name=\"$node\", job=\"$job\"}", "interval": "", "legendFormat": "", "refId": "A" @@ -576,7 +576,7 @@ "type": "prometheus", "uid": "prometheus" }, - "expr": "nvidia_smi_gpu_info{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}", + "expr": "nvidia_smi_gpu_info{uuid=\"$gpu\", host_name=\"$node\", job=\"$job\"}", "format": "table", "instant": true, "legendFormat": "", @@ -714,7 +714,7 @@ "uid": "prometheus" }, "editorMode": "code", - "expr": "nvidia_smi_compute_mode{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}", + "expr": "nvidia_smi_compute_mode{uuid=\"$gpu\", host_name=\"$node\", job=\"$job\"}", "instant": false, "legendFormat": "__auto", "range": true, @@ -793,7 +793,7 @@ "uid": "prometheus" }, "exemplar": true, - "expr": "sum(nvidia_smi_utilization_gpu_ratio{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}) or sum(nvidia_smi_gpu_info{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}) * 0 - 1", + "expr": "sum(nvidia_smi_utilization_gpu_ratio{uuid=\"$gpu\", host_name=\"$node\", job=\"$job\"}) or sum(nvidia_smi_gpu_info{uuid=\"$gpu\", host_name=\"$node\", job=\"$job\"}) * 0 - 1", "interval": "", "legendFormat": "{{uuid}}", "refId": "A" @@ -860,7 +860,7 @@ "uid": "prometheus" }, "exemplar": true, - "expr": "nvidia_smi_power_draw_watts{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"} / (nvidia_smi_enforced_power_limit_watts{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"} or nvidia_smi_power_limit_watts{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"} or nvidia_smi_power_default_limit_watts{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"})", + "expr": "nvidia_smi_power_draw_watts{uuid=\"$gpu\", host_name=\"$node\", job=\"$job\"} / (nvidia_smi_enforced_power_limit_watts{uuid=\"$gpu\", host_name=\"$node\", job=\"$job\"} or nvidia_smi_power_limit_watts{uuid=\"$gpu\", host_name=\"$node\", job=\"$job\"} or nvidia_smi_power_default_limit_watts{uuid=\"$gpu\", host_name=\"$node\", job=\"$job\"})", "interval": "", "legendFormat": "", "refId": "A" @@ -938,7 +938,7 @@ "uid": "prometheus" }, "exemplar": true, - "expr": "sum(nvidia_smi_fan_speed_ratio{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}) or sum(nvidia_smi_gpu_info{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}) * 0 - 1", + "expr": "sum(nvidia_smi_fan_speed_ratio{uuid=\"$gpu\", host_name=\"$node\", job=\"$job\"}) or sum(nvidia_smi_gpu_info{uuid=\"$gpu\", host_name=\"$node\", job=\"$job\"}) * 0 - 1", "interval": "", "legendFormat": "", "refId": "A" @@ -1013,7 +1013,7 @@ "uid": "prometheus" }, "exemplar": true, - "expr": "nvidia_smi_temperature_gpu{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}", + "expr": "nvidia_smi_temperature_gpu{uuid=\"$gpu\", host_name=\"$node\", job=\"$job\"}", "interval": "", "legendFormat": "{{uuid}}", "refId": "A" @@ -1080,7 +1080,7 @@ "uid": "prometheus" }, "exemplar": true, - "expr": "nvidia_smi_clocks_current_graphics_clock_hz{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"} / nvidia_smi_clocks_max_graphics_clock_hz{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}", + "expr": "nvidia_smi_clocks_current_graphics_clock_hz{uuid=\"$gpu\", host_name=\"$node\", job=\"$job\"} / nvidia_smi_clocks_max_graphics_clock_hz{uuid=\"$gpu\", host_name=\"$node\", job=\"$job\"}", "interval": "", "legendFormat": "", "refId": "A" @@ -1147,7 +1147,7 @@ "uid": "prometheus" }, "exemplar": true, - "expr": "nvidia_smi_clocks_current_memory_clock_hz{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"} / nvidia_smi_clocks_max_memory_clock_hz{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}", + "expr": "nvidia_smi_clocks_current_memory_clock_hz{uuid=\"$gpu\", host_name=\"$node\", job=\"$job\"} / nvidia_smi_clocks_max_memory_clock_hz{uuid=\"$gpu\", host_name=\"$node\", job=\"$job\"}", "interval": "", "legendFormat": "", "refId": "A" @@ -1222,7 +1222,7 @@ "uid": "prometheus" }, "exemplar": true, - "expr": "nvidia_smi_memory_used_bytes{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"} / nvidia_smi_memory_total_bytes{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}", + "expr": "nvidia_smi_memory_used_bytes{uuid=\"$gpu\", host_name=\"$node\", job=\"$job\"} / nvidia_smi_memory_total_bytes{uuid=\"$gpu\", host_name=\"$node\", job=\"$job\"}", "interval": "", "legendFormat": "", "refId": "A" @@ -1300,7 +1300,7 @@ "uid": "prometheus" }, "exemplar": true, - "expr": "sum(nvidia_smi_utilization_memory_ratio{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}) or sum(nvidia_smi_gpu_info{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}) * 0 - 1", + "expr": "sum(nvidia_smi_utilization_memory_ratio{uuid=\"$gpu\", host_name=\"$node\", job=\"$job\"}) or sum(nvidia_smi_gpu_info{uuid=\"$gpu\", host_name=\"$node\", job=\"$job\"}) * 0 - 1", "interval": "", "legendFormat": "", "refId": "A" @@ -1398,7 +1398,7 @@ "uid": "prometheus" }, "exemplar": true, - "expr": "nvidia_smi_utilization_memory_ratio{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}", + "expr": "nvidia_smi_utilization_memory_ratio{uuid=\"$gpu\", host_name=\"$node\", job=\"$job\"}", "interval": "", "legendFormat": "{{uuid}}", "refId": "A" @@ -1496,7 +1496,7 @@ "uid": "prometheus" }, "exemplar": true, - "expr": "nvidia_smi_utilization_gpu_ratio{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}", + "expr": "nvidia_smi_utilization_gpu_ratio{uuid=\"$gpu\", host_name=\"$node\", job=\"$job\"}", "interval": "", "legendFormat": "", "refId": "A" @@ -1704,7 +1704,7 @@ "type": "prometheus", "uid": "prometheus" }, - "expr": "nvidia_smi_clocks_event_reasons_gpu_idle{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"} or nvidia_smi_clocks_throttle_reasons_gpu_idle{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}", + "expr": "nvidia_smi_clocks_event_reasons_gpu_idle{uuid=\"$gpu\", host_name=\"$node\", job=\"$job\"} or nvidia_smi_clocks_throttle_reasons_gpu_idle{uuid=\"$gpu\", host_name=\"$node\", job=\"$job\"}", "legendFormat": "Idle", "refId": "A" }, @@ -1713,7 +1713,7 @@ "type": "prometheus", "uid": "prometheus" }, - "expr": "nvidia_smi_clocks_event_reasons_sw_power_cap{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"} or nvidia_smi_clocks_throttle_reasons_sw_power_cap{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}", + "expr": "nvidia_smi_clocks_event_reasons_sw_power_cap{uuid=\"$gpu\", host_name=\"$node\", job=\"$job\"} or nvidia_smi_clocks_throttle_reasons_sw_power_cap{uuid=\"$gpu\", host_name=\"$node\", job=\"$job\"}", "legendFormat": "SW Power Cap", "refId": "B" }, @@ -1722,7 +1722,7 @@ "type": "prometheus", "uid": "prometheus" }, - "expr": "nvidia_smi_clocks_event_reasons_sw_thermal_slowdown{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"} or nvidia_smi_clocks_throttle_reasons_sw_thermal_slowdown{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}", + "expr": "nvidia_smi_clocks_event_reasons_sw_thermal_slowdown{uuid=\"$gpu\", host_name=\"$node\", job=\"$job\"} or nvidia_smi_clocks_throttle_reasons_sw_thermal_slowdown{uuid=\"$gpu\", host_name=\"$node\", job=\"$job\"}", "legendFormat": "SW Thermal Slowdown", "refId": "C" }, @@ -1731,7 +1731,7 @@ "type": "prometheus", "uid": "prometheus" }, - "expr": "nvidia_smi_clocks_event_reasons_hw_thermal_slowdown{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"} or nvidia_smi_clocks_throttle_reasons_hw_thermal_slowdown{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}", + "expr": "nvidia_smi_clocks_event_reasons_hw_thermal_slowdown{uuid=\"$gpu\", host_name=\"$node\", job=\"$job\"} or nvidia_smi_clocks_throttle_reasons_hw_thermal_slowdown{uuid=\"$gpu\", host_name=\"$node\", job=\"$job\"}", "legendFormat": "HW Thermal Slowdown", "refId": "D" }, @@ -1740,7 +1740,7 @@ "type": "prometheus", "uid": "prometheus" }, - "expr": "nvidia_smi_clocks_event_reasons_hw_power_brake_slowdown{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"} or nvidia_smi_clocks_throttle_reasons_hw_power_brake_slowdown{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}", + "expr": "nvidia_smi_clocks_event_reasons_hw_power_brake_slowdown{uuid=\"$gpu\", host_name=\"$node\", job=\"$job\"} or nvidia_smi_clocks_throttle_reasons_hw_power_brake_slowdown{uuid=\"$gpu\", host_name=\"$node\", job=\"$job\"}", "legendFormat": "HW Power Brake", "refId": "E" }, @@ -1749,7 +1749,7 @@ "type": "prometheus", "uid": "prometheus" }, - "expr": "nvidia_smi_clocks_event_reasons_applications_clocks_setting{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"} or nvidia_smi_clocks_throttle_reasons_applications_clocks_setting{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}", + "expr": "nvidia_smi_clocks_event_reasons_applications_clocks_setting{uuid=\"$gpu\", host_name=\"$node\", job=\"$job\"} or nvidia_smi_clocks_throttle_reasons_applications_clocks_setting{uuid=\"$gpu\", host_name=\"$node\", job=\"$job\"}", "legendFormat": "App Clocks Setting", "refId": "F" }, @@ -1758,7 +1758,7 @@ "type": "prometheus", "uid": "prometheus" }, - "expr": "nvidia_smi_clocks_event_reasons_sync_boost{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"} or nvidia_smi_clocks_throttle_reasons_sync_boost{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}", + "expr": "nvidia_smi_clocks_event_reasons_sync_boost{uuid=\"$gpu\", host_name=\"$node\", job=\"$job\"} or nvidia_smi_clocks_throttle_reasons_sync_boost{uuid=\"$gpu\", host_name=\"$node\", job=\"$job\"}", "legendFormat": "Sync Boost", "refId": "G" } @@ -1908,7 +1908,7 @@ "type": "prometheus", "uid": "prometheus" }, - "expr": "nvidia_smi_pstate{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}", + "expr": "nvidia_smi_pstate{uuid=\"$gpu\", host_name=\"$node\", job=\"$job\"}", "legendFormat": "P-State", "refId": "A" } @@ -2016,7 +2016,7 @@ "uid": "prometheus" }, "exemplar": true, - "expr": "nvidia_smi_memory_used_bytes{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}", + "expr": "nvidia_smi_memory_used_bytes{uuid=\"$gpu\", host_name=\"$node\", job=\"$job\"}", "interval": "", "legendFormat": "{{uuid}}", "refId": "A" @@ -2117,7 +2117,7 @@ "uid": "prometheus" }, "exemplar": true, - "expr": "nvidia_smi_temperature_gpu{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}", + "expr": "nvidia_smi_temperature_gpu{uuid=\"$gpu\", host_name=\"$node\", job=\"$job\"}", "interval": "", "legendFormat": "{{uuid}}", "refId": "A" @@ -2213,7 +2213,7 @@ "uid": "prometheus" }, "exemplar": true, - "expr": "nvidia_smi_power_draw_watts{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}", + "expr": "nvidia_smi_power_draw_watts{uuid=\"$gpu\", host_name=\"$node\", job=\"$job\"}", "interval": "", "legendFormat": "{{uuid}}", "refId": "A" @@ -2311,7 +2311,7 @@ "uid": "prometheus" }, "exemplar": true, - "expr": "nvidia_smi_fan_speed_ratio{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}", + "expr": "nvidia_smi_fan_speed_ratio{uuid=\"$gpu\", host_name=\"$node\", job=\"$job\"}", "interval": "", "legendFormat": "{{uuid}}", "refId": "A" @@ -2417,7 +2417,7 @@ "uid": "prometheus" }, "exemplar": true, - "expr": "nvidia_smi_clocks_current_graphics_clock_hz{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}", + "expr": "nvidia_smi_clocks_current_graphics_clock_hz{uuid=\"$gpu\", host_name=\"$node\", job=\"$job\"}", "format": "time_series", "interval": "", "legendFormat": "{{uuid}}", @@ -2514,7 +2514,7 @@ "uid": "prometheus" }, "exemplar": true, - "expr": "nvidia_smi_clocks_current_video_clock_hz{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}", + "expr": "nvidia_smi_clocks_current_video_clock_hz{uuid=\"$gpu\", host_name=\"$node\", job=\"$job\"}", "format": "time_series", "interval": "", "legendFormat": "{{uuid}}", @@ -2611,7 +2611,7 @@ "uid": "prometheus" }, "exemplar": true, - "expr": "nvidia_smi_clocks_current_sm_clock_hz{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}", + "expr": "nvidia_smi_clocks_current_sm_clock_hz{uuid=\"$gpu\", host_name=\"$node\", job=\"$job\"}", "format": "time_series", "interval": "", "legendFormat": "{{uuid}}", @@ -2708,7 +2708,7 @@ "uid": "prometheus" }, "exemplar": true, - "expr": "nvidia_smi_clocks_current_memory_clock_hz{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}", + "expr": "nvidia_smi_clocks_current_memory_clock_hz{uuid=\"$gpu\", host_name=\"$node\", job=\"$job\"}", "format": "time_series", "interval": "", "legendFormat": "{{uuid}}", @@ -2864,7 +2864,7 @@ "type": "prometheus", "uid": "prometheus" }, - "expr": "nvidia_smi_compute_app_used_memory_bytes{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"} or (nvidia_smi_compute_app_info{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"} * 0)", + "expr": "nvidia_smi_compute_app_used_memory_bytes{uuid=\"$gpu\", host_name=\"$node\", job=\"$job\"} or (nvidia_smi_compute_app_info{uuid=\"$gpu\", host_name=\"$node\", job=\"$job\"} * 0)", "format": "table", "instant": true, "legendFormat": "", @@ -2875,7 +2875,7 @@ "type": "prometheus", "uid": "prometheus" }, - "expr": "(nvidia_smi_compute_app_used_memory_bytes{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"} or (nvidia_smi_compute_app_info{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"} * 0)) / on(uuid) group_left() nvidia_smi_memory_total_bytes{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}", + "expr": "(nvidia_smi_compute_app_used_memory_bytes{uuid=\"$gpu\", host_name=\"$node\", job=\"$job\"} or (nvidia_smi_compute_app_info{uuid=\"$gpu\", host_name=\"$node\", job=\"$job\"} * 0)) / on(uuid) group_left() nvidia_smi_memory_total_bytes{uuid=\"$gpu\", host_name=\"$node\", job=\"$job\"}", "format": "table", "instant": true, "legendFormat": "", @@ -2982,7 +2982,7 @@ "type": "prometheus", "uid": "prometheus" }, - "expr": "nvidia_smi_compute_app_used_memory_bytes{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}", + "expr": "nvidia_smi_compute_app_used_memory_bytes{uuid=\"$gpu\", host_name=\"$node\", job=\"$job\"}", "legendFormat": "{{process_name}} ({{pid}})", "refId": "A" } @@ -3065,7 +3065,7 @@ "type": "prometheus", "uid": "prometheus" }, - "expr": "nvidia_smi_compute_apps_last_collect_success{instance=\"$node\", job=\"$job\"}", + "expr": "nvidia_smi_compute_apps_last_collect_success{host_name=\"$node\", job=\"$job\"}", "legendFormat": "", "refId": "A" } @@ -3174,7 +3174,7 @@ "uid": "prometheus" }, "editorMode": "code", - "expr": "nvidia_smi_fabric_state{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}", + "expr": "nvidia_smi_fabric_state{uuid=\"$gpu\", host_name=\"$node\", job=\"$job\"}", "instant": false, "legendFormat": "__auto", "range": true, @@ -3245,7 +3245,7 @@ "type": "prometheus", "uid": "prometheus" }, - "expr": "label_replace({__name__=~\"nvidia_smi_ecc_errors_uncorrected_.+\", uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}, \"kind\", \"$1\", \"__name__\", \"nvidia_smi_ecc_errors_uncorrected_(.+)\")", + "expr": "label_replace({__name__=~\"nvidia_smi_ecc_errors_uncorrected_.+\", uuid=\"$gpu\", host_name=\"$node\", job=\"$job\"}, \"kind\", \"$1\", \"__name__\", \"nvidia_smi_ecc_errors_uncorrected_(.+)\")", "legendFormat": "{{kind}}", "refId": "A" } @@ -3342,7 +3342,7 @@ "type": "prometheus", "uid": "prometheus" }, - "expr": "sum(nvidia_smi_xid_errors_total{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}) or sum(nvidia_smi_gpu_info{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}) * 0", + "expr": "sum(nvidia_smi_xid_errors_total{uuid=\"$gpu\", host_name=\"$node\", job=\"$job\"}) or sum(nvidia_smi_gpu_info{uuid=\"$gpu\", host_name=\"$node\", job=\"$job\"}) * 0", "legendFormat": "", "refId": "A" } @@ -3429,7 +3429,7 @@ "type": "prometheus", "uid": "prometheus" }, - "expr": "time() - max(nvidia_smi_xid_last_timestamp_seconds{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}) or (sum(nvidia_smi_gpu_info{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}) * 0 - 1)", + "expr": "time() - max(nvidia_smi_xid_last_timestamp_seconds{uuid=\"$gpu\", host_name=\"$node\", job=\"$job\"}) or (sum(nvidia_smi_gpu_info{uuid=\"$gpu\", host_name=\"$node\", job=\"$job\"}) * 0 - 1)", "legendFormat": "", "refId": "A" } @@ -3524,7 +3524,7 @@ "uid": "prometheus" }, "exemplar": true, - "expr": "nvidia_smi_xid_errors_total{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}", + "expr": "nvidia_smi_xid_errors_total{uuid=\"$gpu\", host_name=\"$node\", job=\"$job\"}", "interval": "", "legendFormat": "XID {{xid}}", "refId": "A" @@ -3619,7 +3619,7 @@ "uid": "prometheus" }, "exemplar": true, - "expr": "nvidia_smi_pcie_throughput_tx_bytes_per_second{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}", + "expr": "nvidia_smi_pcie_throughput_tx_bytes_per_second{uuid=\"$gpu\", host_name=\"$node\", job=\"$job\"}", "interval": "", "legendFormat": "TX", "refId": "A" @@ -3630,7 +3630,7 @@ "uid": "prometheus" }, "exemplar": true, - "expr": "nvidia_smi_pcie_throughput_rx_bytes_per_second{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}", + "expr": "nvidia_smi_pcie_throughput_rx_bytes_per_second{uuid=\"$gpu\", host_name=\"$node\", job=\"$job\"}", "interval": "", "legendFormat": "RX", "refId": "B" @@ -3724,7 +3724,7 @@ "uid": "prometheus" }, "exemplar": true, - "expr": "rate(nvidia_smi_energy_joules_total{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}[$__rate_interval])", + "expr": "rate(nvidia_smi_energy_joules_total{uuid=\"$gpu\", host_name=\"$node\", job=\"$job\"}[$__rate_interval])", "interval": "", "legendFormat": "from energy counter", "refId": "A" @@ -3735,7 +3735,7 @@ "uid": "prometheus" }, "exemplar": true, - "expr": "nvidia_smi_power_draw_watts{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}", + "expr": "nvidia_smi_power_draw_watts{uuid=\"$gpu\", host_name=\"$node\", job=\"$job\"}", "interval": "", "legendFormat": "sampled power draw", "refId": "B" @@ -3804,7 +3804,7 @@ "type": "prometheus", "uid": "prometheus" }, - "expr": "sum(increase(nvidia_smi_energy_joules_total{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}[24h])) / 3.6e6", + "expr": "sum(increase(nvidia_smi_energy_joules_total{uuid=\"$gpu\", host_name=\"$node\", job=\"$job\"}[24h])) / 3.6e6", "legendFormat": "", "refId": "A" } @@ -3898,7 +3898,7 @@ "uid": "prometheus" }, "exemplar": true, - "expr": "(nvidia_smi_mig_memory_used_bytes{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}) * on(uuid, gpu_instance_id) group_left(profile) (max by (uuid, gpu_instance_id, profile) (label_replace(nvidia_smi_mig_info{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}, \"profile\", \"$1\", \"profile\", \"^[0-9]+c\\\\.(.*)$\")))", + "expr": "(nvidia_smi_mig_memory_used_bytes{uuid=\"$gpu\", host_name=\"$node\", job=\"$job\"}) * on(uuid, gpu_instance_id) group_left(profile) (max by (uuid, gpu_instance_id, profile) (label_replace(nvidia_smi_mig_info{uuid=\"$gpu\", host_name=\"$node\", job=\"$job\"}, \"profile\", \"$1\", \"profile\", \"^[0-9]+c\\\\.(.*)$\")))", "interval": "", "legendFormat": "GI {{gpu_instance_id}} ({{profile}})", "refId": "A" @@ -3993,7 +3993,7 @@ "uid": "prometheus" }, "exemplar": true, - "expr": "(nvidia_smi_mig_sm_activity_ratio{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}) * on(uuid, gpu_instance_id) group_left(profile) (max by (uuid, gpu_instance_id, profile) (label_replace(nvidia_smi_mig_info{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}, \"profile\", \"$1\", \"profile\", \"^[0-9]+c\\\\.(.*)$\")))", + "expr": "(nvidia_smi_mig_sm_activity_ratio{uuid=\"$gpu\", host_name=\"$node\", job=\"$job\"}) * on(uuid, gpu_instance_id) group_left(profile) (max by (uuid, gpu_instance_id, profile) (label_replace(nvidia_smi_mig_info{uuid=\"$gpu\", host_name=\"$node\", job=\"$job\"}, \"profile\", \"$1\", \"profile\", \"^[0-9]+c\\\\.(.*)$\")))", "interval": "", "legendFormat": "GI {{gpu_instance_id}} ({{profile}})", "refId": "A" @@ -4088,7 +4088,7 @@ "uid": "prometheus" }, "exemplar": true, - "expr": "(nvidia_smi_mig_tensor_activity_ratio{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}) * on(uuid, gpu_instance_id) group_left(profile) (max by (uuid, gpu_instance_id, profile) (label_replace(nvidia_smi_mig_info{uuid=\"$gpu\", instance=\"$node\", job=\"$job\"}, \"profile\", \"$1\", \"profile\", \"^[0-9]+c\\\\.(.*)$\")))", + "expr": "(nvidia_smi_mig_tensor_activity_ratio{uuid=\"$gpu\", host_name=\"$node\", job=\"$job\"}) * on(uuid, gpu_instance_id) group_left(profile) (max by (uuid, gpu_instance_id, profile) (label_replace(nvidia_smi_mig_info{uuid=\"$gpu\", host_name=\"$node\", job=\"$job\"}, \"profile\", \"$1\", \"profile\", \"^[0-9]+c\\\\.(.*)$\")))", "interval": "", "legendFormat": "GI {{gpu_instance_id}} ({{profile}})", "refId": "A" @@ -4124,7 +4124,7 @@ "query": "label_values(nvidia_smi_index, project)", "refId": "var" }, - "refresh": 2, + "refresh": 1, "sort": 1, "current": {}, "options": [], @@ -4144,7 +4144,7 @@ "query": "label_values(nvidia_smi_index{project=\"$project\"}, env)", "refId": "var" }, - "refresh": 2, + "refresh": 1, "sort": 1, "current": {}, "options": [], @@ -4164,7 +4164,7 @@ "query": "label_values(nvidia_smi_index{project=\"$project\", env=\"$env\"}, job)", "refId": "var" }, - "refresh": 2, + "refresh": 1, "sort": 1, "current": {}, "options": [], @@ -4178,13 +4178,13 @@ "type": "prometheus", "uid": "prometheus" }, - "definition": "label_values(nvidia_smi_index{project=\"$project\", env=\"$env\", job=\"$job\"}, instance)", + "definition": "label_values(nvidia_smi_index{project=\"$project\", env=\"$env\", job=\"$job\"}, host_name)", "query": { "qryType": 1, - "query": "label_values(nvidia_smi_index{project=\"$project\", env=\"$env\", job=\"$job\"}, instance)", + "query": "label_values(nvidia_smi_index{project=\"$project\", env=\"$env\", job=\"$job\"}, host_name)", "refId": "var" }, - "refresh": 2, + "refresh": 1, "sort": 1, "current": {}, "options": [], @@ -4196,7 +4196,7 @@ "type": "prometheus", "uid": "prometheus" }, - "definition": "query_result(label_join(label_replace(label_join(nvidia_smi_gpu_info{instance=\"$node\", job=\"$job\"}, \"ni\", \" #\", \"name\", \"index\"), \"us\", \"($1)\", \"uuid\", \"^(........).*\"), \"gpu_label\", \" \", \"ni\", \"us\"))", + "definition": "query_result(label_join(label_replace(label_join(nvidia_smi_gpu_info{host_name=\"$node\", job=\"$job\"}, \"ni\", \" #\", \"name\", \"index\"), \"us\", \"($1)\", \"uuid\", \"^(........).*\"), \"gpu_label\", \" \", \"ni\", \"us\"))", "hide": 0, "includeAll": false, "label": "GPU", @@ -4204,7 +4204,7 @@ "name": "gpu", "options": [], "query": { - "query": "query_result(label_join(label_replace(label_join(nvidia_smi_gpu_info{instance=\"$node\", job=\"$job\"}, \"ni\", \" #\", \"name\", \"index\"), \"us\", \"($1)\", \"uuid\", \"^(........).*\"), \"gpu_label\", \" \", \"ni\", \"us\"))", + "query": "query_result(label_join(label_replace(label_join(nvidia_smi_gpu_info{host_name=\"$node\", job=\"$job\"}, \"ni\", \" #\", \"name\", \"index\"), \"us\", \"($1)\", \"uuid\", \"^(........).*\"), \"gpu_label\", \" \", \"ni\", \"us\"))", "refId": "StandardVariableQuery" }, "refresh": 1, diff --git a/dashboards/host-containers.json b/dashboards/host-containers.json index e065442..e66e7aa 100644 --- a/dashboards/host-containers.json +++ b/dashboards/host-containers.json @@ -30,7 +30,7 @@ "query": "label_values(node_uname_info, project)", "refId": "var" }, - "refresh": 2, + "refresh": 1, "sort": 1, "current": {}, "options": [], @@ -50,7 +50,7 @@ "query": "label_values(node_uname_info{project=\"$project\"}, env)", "refId": "var" }, - "refresh": 2, + "refresh": 1, "sort": 1, "current": {}, "options": [], @@ -70,7 +70,7 @@ "query": "label_values(node_uname_info{project=\"$project\", env=\"$env\"}, host_name)", "refId": "var" }, - "refresh": 2, + "refresh": 1, "sort": 1, "current": {}, "options": [], diff --git a/dashboards/logs.json b/dashboards/logs.json index cebc35b..73f1caa 100644 --- a/dashboards/logs.json +++ b/dashboards/logs.json @@ -25,7 +25,7 @@ }, "definition": "label_values(project)", "query": "label_values(project)", - "refresh": 2, + "refresh": 1, "sort": 1, "current": {}, "options": [], @@ -41,7 +41,7 @@ }, "definition": "label_values({project=\"$project\"}, env)", "query": "label_values({project=\"$project\"}, env)", - "refresh": 2, + "refresh": 1, "sort": 1, "current": {}, "options": [], @@ -57,7 +57,7 @@ }, "definition": "label_values({project=\"$project\", env=\"$env\"}, service_name)", "query": "label_values({project=\"$project\", env=\"$env\"}, service_name)", - "refresh": 2, + "refresh": 1, "sort": 1, "current": { "text": "All", diff --git a/dashboards/service-health.json b/dashboards/service-health.json index 5d59f49..3dd9390 100644 --- a/dashboards/service-health.json +++ b/dashboards/service-health.json @@ -29,7 +29,7 @@ "query": "label_values(http_server_request_duration_seconds_count, project)", "refId": "var" }, - "refresh": 2, + "refresh": 1, "sort": 1, "current": {}, "options": [], @@ -49,7 +49,7 @@ "query": "label_values(http_server_request_duration_seconds_count{project=\"$project\"}, env)", "refId": "var" }, - "refresh": 2, + "refresh": 1, "sort": 1, "current": {}, "options": [], @@ -69,7 +69,7 @@ "query": "label_values(http_server_request_duration_seconds_count{project=\"$project\", env=\"$env\"}, job)", "refId": "var" }, - "refresh": 2, + "refresh": 1, "sort": 1, "current": {}, "options": [], From 25f0804803e59233ec8d05d4cefd66de6d211dd4 Mon Sep 17 00:00:00 2001 From: Simon van Lierde Date: Sun, 6 Sep 2026 14:13:37 +0000 Subject: [PATCH 41/85] test(check): exercise the exposure guards and spoke overlays, prove delivery in smoke - check runs the exposure guards positively and against each documented default, renders the tunnel set with JWT on and both spoke overlays - smoke asserts contact points were expanded and pushes one OTLP log through bearer auth, read back from Loki with the department label - GRAFANA_ROOT_URL guard requires https://; stale url_file wording dropped --- justfile | 33 +++++++++++++++++++++++++++++++-- 1 file changed, 31 insertions(+), 2 deletions(-) diff --git a/justfile b/justfile index 3922f19..9af5aa9 100644 --- a/justfile +++ b/justfile @@ -66,11 +66,11 @@ _guard-if-exposed: _expose-guards: @[ "${OTLP_AUTH_TOKEN:-}" != "local-dev-token" ] || { echo "error: OTLP_AUTH_TOKEN is still the local default; generate one (openssl rand -hex 32) before exposing ingestion" >&2; exit 1; } @[ "${GRAFANA_ADMIN_PASSWORD:-}" != "change-me" ] || { echo "error: GRAFANA_ADMIN_PASSWORD is still the documented default; change it before exposing Grafana" >&2; exit 1; } - @[ "${GRAFANA_ROOT_URL:-}" != "http://localhost:3000" ] || { echo "error: GRAFANA_ROOT_URL is still the localhost default; set it to the tunnel hostname or every absolute URL Grafana generates breaks" >&2; exit 1; } + @case "${GRAFANA_ROOT_URL:-}" in https://*) ;; *) echo "error: GRAFANA_ROOT_URL must be the https:// tunnel hostname (got '${GRAFANA_ROOT_URL:-}'); every absolute URL Grafana generates comes from it" >&2; exit 1;; esac @[ "${GRAFANA_COOKIE_SECURE:-false}" = "true" ] || { echo "error: GRAFANA_COOKIE_SECURE must be true when Grafana is served over HTTPS; set it in .env" >&2; exit 1; } @[ "${GRAFANA_JWT_AUTH:-false}" != "true" ] || { [ -n "${CF_ACCESS_TEAM_DOMAIN:-}" ] && [ -n "${CF_ACCESS_AUD:-}" ]; } || { echo "error: GRAFANA_JWT_AUTH=true needs CF_ACCESS_TEAM_DOMAIN and CF_ACCESS_AUD in .env (cd infra && tofu output -raw grafana_access_aud)" >&2; exit 1; } @[ -n "${HEARTBEAT_URL:-}" ] || echo "WARNING: HEARTBEAT_URL is empty; the stack goes live without a dead-man's switch" >&2 - @[ -n "${ALERT_WEBHOOK_URL:-}" ] || { echo "error: ALERT_WEBHOOK_URL is empty; every alert would fire into an empty url_file and be dropped. The heartbeat keeps pinging either way, so this failure looks healthy from the outside — set it, or comment out this guard deliberately" >&2; exit 1; } + @[ -n "${ALERT_WEBHOOK_URL:-}" ] || { echo "error: ALERT_WEBHOOK_URL is empty; every alert would fire into an empty webhook URL and be dropped. The heartbeat keeps pinging either way, so this failure looks healthy from the outside — set it, or comment out this guard deliberately" >&2; exit 1; } down: docker compose down --remove-orphans @@ -125,6 +125,21 @@ check: CLOUDFLARE_TUNNEL_TOKEN=dummy docker compose -f compose.yml -f compose.tunnel.yml config -q {{compose_demo}} config -q {{compose_smoke}} config -q + # The tunnel set with JWT auth on, so the JWK URL and claims interpolation + # in compose.tunnel.yml is at least schema-checked; nothing else enables it. + CLOUDFLARE_TUNNEL_TOKEN=dummy GRAFANA_JWT_AUTH=true CF_ACCESS_TEAM_DOMAIN=dummy CF_ACCESS_AUD=dummy docker compose -f compose.yml -f compose.tunnel.yml config -q + # The spoke overlays every project host layers onto its own compose.yml; a + # bad interpolation or a network they forget to declare otherwise first + # fails on a client machine, after vendoring. + ENVIRONMENT=dummy PROJECT=dummy COMPOSE_PROJECT_NAME=dummy OTEL_EXPORTER_OTLP_ENDPOINT=https://dummy OTLP_AUTH_TOKEN=dummy docker compose -f templates/compose.telemetry.yml -f templates/compose.telemetry.gpu.yml config -q + # The exposure guards, both ways: a fully set .env must pass, and each + # documented default must be refused on its own. Nothing else runs them — + # CI never sets COMPOSE_FILE to the tunnel overlay. + @good="OTLP_AUTH_TOKEN=t GRAFANA_ADMIN_PASSWORD=p GRAFANA_ROOT_URL=https://g.example GRAFANA_COOKIE_SECURE=true GRAFANA_JWT_AUTH=true CF_ACCESS_TEAM_DOMAIN=d CF_ACCESS_AUD=a HEARTBEAT_URL=https://h ALERT_WEBHOOK_URL=https://w"; \ + env $good just _expose-guards; \ + for bad in OTLP_AUTH_TOKEN=local-dev-token GRAFANA_ADMIN_PASSWORD=change-me GRAFANA_ROOT_URL=http://g.example GRAFANA_COOKIE_SECURE=false CF_ACCESS_AUD= ALERT_WEBHOOK_URL=; do \ + ! env $good $bad just _expose-guards 2>/dev/null || { echo "error: exposure guards accepted $bad" >&2; exit 1; }; \ + done images="$(docker compose config --images)" && \ docker run --rm --network none -v ./config/prometheus.yaml:/etc/prometheus/prometheus.yaml:ro --entrypoint promtool $(echo "$images" | grep prom/prometheus) check config /etc/prometheus/prometheus.yaml && \ docker run --rm --network none -e OTLP_AUTH_TOKEN=dummy -e DEPARTMENT=dummy -v ./config/otel-collector.yaml:/etc/otelcol/config.yaml:ro $(echo "$images" | grep opentelemetry-collector) validate --config=/etc/otelcol/config.yaml && \ @@ -206,6 +221,20 @@ smoke: (_queue-volume smoke_project) n=$((n+3)); [ $n -ge 60 ] && { echo "error: not provisioned after 60s — dashboards missing:${missing:- none}; alert rules $got/$want_rules (a malformed file provisions none of its group). See just smoke-logs" >&2; exit 1; }; \ sleep 3; \ done + # Provisioning proved the rules exist; this proves they can be delivered + # and that the data path works. Grafana expands $VAR in the alerting + # provisioning files — a Grafana that stopped doing so would store the + # literal name and every notification would fail silently (see + # contact-points.yaml). Then one OTLP log through the collector's bearer + # auth, asserted back out of Loki with the department label the collector + # stamps: the label chain the keystone rules key on, end to end. + @auth="user = \"admin:${GRAFANA_ADMIN_PASSWORD}\""; \ + printf '%s\n' "$auth" | curl -sf -K - {{smoke_url}}/api/v1/provisioning/contact-points | docker run --rm -i ghcr.io/jqlang/jq:1.8.1 -e '[.[] | select(.uid | startswith("cp-")) | .settings.url] | all(startswith("$") | not)' >/dev/null || { echo "error: a contact point still carries a literal \$VAR — Grafana did not expand the alerting provisioning file" >&2; exit 1; }; \ + ts="$(date +%s)000000000"; body='{"resourceLogs":[{"resource":{"attributes":[{"key":"service.name","value":{"stringValue":"smoke"}},{"key":"project","value":{"stringValue":"smoke"}},{"key":"env","value":{"stringValue":"ci"}}]},"scopeLogs":[{"logRecords":[{"timeUnixNano":"'"$ts"'","body":{"stringValue":"smoke"}}]}]}]}'; \ + docker run --rm --network {{smoke_project}}_default alpine:3.24@sha256:28bd5fe8b56d1bd048e5babf5b10710ebe0bae67db86916198a6eec434943f8b wget -qO- --header="Authorization: Bearer ${OTLP_AUTH_TOKEN}" --header='Content-Type: application/json' --post-data="$body" http://otel-collector:4318/v1/logs >/dev/null || { echo "error: the collector refused an OTLP log with the .env token" >&2; exit 1; }; \ + n=0; until printf '%s\n' "$auth" | curl -sf -K - -G --data-urlencode "query={project=\"smoke\",env=\"ci\",department=\"${DEPARTMENT:-cml}\"}" '{{smoke_url}}/api/datasources/proxy/uid/loki/loki/api/v1/query_range' | grep -q '"smoke"'; do \ + n=$((n+3)); [ $n -ge 60 ] && { echo "error: the smoke log never reached Loki with its department label — see just smoke-logs" >&2; exit 1; }; sleep 3; \ + done @echo "Stack healthy" # Logs from the smoke stack (its own project, so `just logs` will not show it). From 299e475a8168a14271f0b84add8c20dc15f05be5 Mon Sep 17 00:00:00 2001 From: Simon van Lierde Date: Sun, 6 Sep 2026 14:13:37 +0000 Subject: [PATCH 42/85] docs: upgrade steps for the import script and the Alertmanager volume --- CHANGELOG.md | 28 ++++++++++++++++---- docs/RUNBOOK.md | 11 +++++++- docs/adr/0002-hub-and-spoke-observability.md | 10 +++---- 3 files changed, 38 insertions(+), 11 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index d6c6a8a..0ed1635 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -44,9 +44,6 @@ queries that had been measuring the wrong thing. vendor. - Dependabot now watches the Cloudflare provider in `infra/`, and the runbook covers OpenTofu-managed tunnel and Access changes. - -### Added - - **`infra/generate-imports.sh`**: emits OpenTofu `import` blocks for the edge built by hand in the Zero Trust dashboard. Without it the first plan against an empty state reads "create" for objects already serving traffic, and @@ -89,12 +86,25 @@ queries that had been measuring the wrong thing. - **Alertmanager**: alerting is Grafana-managed now (ADR 0002) — rules are provisioned from `config/grafana/alerting/`, and delivery still posts to - `ALERT_WEBHOOK_URL`. + `ALERT_WEBHOOK_URL`. Its `alertmanager_data` volume is left behind on an + upgraded host; the runbook says when to remove it. - The commented `compose.storage-s3.yml` stub: the S3 escape hatch lives as an appendix of ADR 0001 instead. ### Fixed +- **Spoke overlay declared no `egress` network**: `compose.telemetry.yml` + joined it without defining it, so the documented `up -d` failed on any host + whose own compose file did not happen to name one. `just check` now renders + both spoke overlays. +- **GPU dashboard host picker keyed on `instance`**, which is the same + in-container address on every host; it uses `host_name` like the rest. + Dashboard variables refresh on load, not on every 30s tick. +- **Access app import id**: `generate-imports.sh` emitted it without the + `accounts/` scope the 5.x provider requires. +- **`just smoke` proves delivery and the data path**: contact points must be + expanded (no literal `$VAR`), and one OTLP log through the collector's + bearer auth must come back out of Loki carrying the `department` label. - **Trace links from the latency panel**: the Prometheus datasource pointed exemplars at a `trace_id` label, but span-metrics exemplars carry `traceID`, so clicking a dot resolved to nothing. @@ -124,7 +134,15 @@ queries that had been measuring the wrong thing. - `no-new-privileges` on every service; the demo image runs as `nobody`. - `GRAFANA_COOKIE_SECURE` marks the session cookie Secure (with strict SameSite), and `just up` with the tunnel overlay refuses to expose the stack - without it, a non-localhost `GRAFANA_ROOT_URL`, and non-default credentials. + without it, an `https://` `GRAFANA_ROOT_URL`, and non-default credentials. + `just check` runs the guards both ways, so a guard that silently accepts a + default fails CI rather than the production start. +- Loki, Tempo, Prometheus and node-exporter sit on an internal `backend` + network only Grafana and the collector join. cloudflared stays on `default`, + so an ingress edited in the Cloudflare dashboard cannot reach a backend that + has no authentication of its own. +- Every hub service carries a `pids_limit`; Dependabot also watches the spoke + images pinned in `templates/`. - GitHub Actions are pinned to commit SHAs, and `just infra-validate` runs against a copy of the sources so state and tfvars never enter the container. diff --git a/docs/RUNBOOK.md b/docs/RUNBOOK.md index fd510fc..0cda42e 100644 --- a/docs/RUNBOOK.md +++ b/docs/RUNBOOK.md @@ -153,6 +153,12 @@ cd infra && tofu apply - **Adding a hostname:** add an `ingress` entry pointing at the service's container port, plus a matching `cloudflare_dns_record`. The catch-all `http_status:404` entry stays last, or it swallows everything after it. +- **First apply against an edge built by hand** (a tunnel, DNS records, or + Access app that already exist in the dashboard): an empty state plans them + as "create", and applying that mints a second tunnel and a duplicate Access + app. Run `infra/generate-imports.sh > infra/imports.tf` first, check the + plan reads 0 to add for the imported resources, apply, then delete + `imports.tf`; it is a one-time instruction and gitignored. - **State lives on this host only.** `infra/terraform.tfstate` is gitignored and `just backup` does not touch it. Copy it off-host next to the backups. Losing it orphans the Cloudflare resources: they keep running, but the next @@ -178,7 +184,10 @@ actionlint, shellcheck, ruff, gitleaks, OpenTofu, jq, the Alloy validator, and the alpine that backup, restore, and the queue-volume setup run in. No Dependabot ecosystem covers a justfile, so bump those by hand. -After merging, on the host: `git pull && just pull && just up`. Use `just +After merging, on the host: `git pull && just pull && just up`. Coming from +a release that still ran Alertmanager, its volume outlives the service: +`docker volume rm monitoring_alertmanager_data` once the new stack is up, and +`config/alertmanager.yaml` can go with it. Use `just up`, not `docker compose up -d`: the recipe first chowns the collector's queue volume to uid 10001. Where that volume is new, a raw compose up leaves it root-owned and the collector crash-looping on a queue directory it cannot diff --git a/docs/adr/0002-hub-and-spoke-observability.md b/docs/adr/0002-hub-and-spoke-observability.md index b08a4dc..497c8c5 100644 --- a/docs/adr/0002-hub-and-spoke-observability.md +++ b/docs/adr/0002-hub-and-spoke-observability.md @@ -55,11 +55,11 @@ Contracts that make it scale: on a spoke can detect its own absence. Templated and provisioned by `bootstrap.sh`, which is also what creates a project's healthchecks and prints its `.env` block. Bootstrap is what creates the safety net, not the telemetry. -- **Onboarding is a copy, not a port:** vendor two template files at a pinned - tag, add six `.env` variables, include the overlay, run `bootstrap.sh`. A GPU - host is an ordinary host plus one opt-in overlay (`nvidia_gpu_exporter` - scraped by Alloy — not dcgm-exporter, whose profiling fields are - datacentre-only) and three GPU alert rules. +- **Onboarding is a copy, not a port:** vendor the template files at a pinned + tag (three, plus one for a GPU host), add six `.env` variables, include the + overlay, run `bootstrap.sh`. A GPU host is an ordinary host plus one opt-in + overlay (`nvidia_gpu_exporter` scraped by Alloy — not dcgm-exporter, whose + profiling fields are datacentre-only) and three GPU alert rules. ## Alternatives considered From 242cb1ba7d91eabe0ae93bbbf4db0424f6639fbd Mon Sep 17 00:00:00 2001 From: Simon van Lierde Date: Sun, 6 Sep 2026 14:28:07 +0000 Subject: [PATCH 43/85] feat(security): read-only root for cloudflared --- compose.tunnel.yml | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/compose.tunnel.yml b/compose.tunnel.yml index fe02955..9c220ea 100644 --- a/compose.tunnel.yml +++ b/compose.tunnel.yml @@ -6,7 +6,6 @@ # The tunnel, its public hostnames, and DNS are managed as code in infra/ # (OpenTofu); `tofu output -raw tunnel_token` yields the token for .env. # See infra/main.tf for the bootstrap steps. - services: cloudflared: image: cloudflare/cloudflared:2026.8.2@sha256:0aa26e284f05e6c77ae375b8c9c11d9eb6a448fb7bcd8d40f31cb6176189eb38 @@ -20,6 +19,8 @@ services: # Ceiling sized from observed usage with headroom; see compose.yml. mem_limit: 256m pids_limit: 256 + # Nothing to write: config comes from Cloudflare, the token from env. + read_only: true cap_drop: [ALL] security_opt: - no-new-privileges:true From b3f8633d439f5798ae9ece74040531ec81a10e2d Mon Sep 17 00:00:00 2001 From: Simon van Lierde Date: Sun, 6 Sep 2026 14:42:33 +0000 Subject: [PATCH 44/85] docs(runbook): inventory every secret, rotate the rest, mark tfstate as sensitive --- docs/RUNBOOK.md | 39 ++++++++++++++++++++++++++++++++++++--- 1 file changed, 36 insertions(+), 3 deletions(-) diff --git a/docs/RUNBOOK.md b/docs/RUNBOOK.md index 0cda42e..2201ac9 100644 --- a/docs/RUNBOOK.md +++ b/docs/RUNBOOK.md @@ -97,6 +97,37 @@ host is the intended setup. `CLOUDFLARE_TUNNEL_TOKEN` in `.env`, then `just up`. - **Grafana admin password:** change `GRAFANA_ADMIN_PASSWORD` in `.env`, then `docker compose up -d grafana`. +- **Alert webhook and heartbeat URLs:** both are capability URLs, so the URL + is the credential. Mint a new topic or check at the provider, put it in + `ALERT_WEBHOOK_URL` / `HEARTBEAT_URL`, then `docker compose up -d grafana`; + Grafana reads the contact points only at startup. +- **healthchecks.io API key:** regenerate it in the project's settings, set + `HEALTHCHECKS_API_KEY` in `.env`. Only `bootstrap.sh` reads it, so nothing + needs a restart. +- **Cloudflare API token:** it is never stored here; create a new one with the + same three permissions, revoke the old one, export the new value before the + next `tofu` run. + +### Where every secret lives + +Three files hold everything, all gitignored, none backed up by `just backup`. +Copy `.env` and `infra/terraform.tfstate` off-host together with the backups +and treat the copies the way you treat the originals. + +| Secret | Lives in | Comes from | +| --- | --- | --- | +| `OTLP_AUTH_TOKEN` | `.env` | `openssl rand -hex 32` | +| `GRAFANA_ADMIN_PASSWORD` | `.env` | you | +| `ALERT_WEBHOOK_URL`, `HEARTBEAT_URL` | `.env` | the notification provider | +| `HEALTHCHECKS_API_KEY` | `.env` | healthchecks.io project settings | +| `CLOUDFLARE_TUNNEL_TOKEN` | `.env` | `tofu output -raw tunnel_token` | +| `CF_ACCESS_AUD` | `.env` (not secret, but paired) | `tofu output -raw grafana_access_aud` | +| Tunnel secret, API responses | `infra/terraform.tfstate` | written by every `tofu apply` | +| `CLOUDFLARE_API_TOKEN` | your shell, per session | Cloudflare dashboard | + +`infra/terraform.tfvars` holds identifiers only (account, zone, domain, the +Access email list) and is gitignored for privacy, not because it holds a +credential. ## Alert delivery @@ -159,9 +190,11 @@ cd infra && tofu apply app. Run `infra/generate-imports.sh > infra/imports.tf` first, check the plan reads 0 to add for the imported resources, apply, then delete `imports.tf`; it is a one-time instruction and gitignored. -- **State lives on this host only.** `infra/terraform.tfstate` is gitignored - and `just backup` does not touch it. Copy it off-host next to the backups. - Losing it orphans the Cloudflare resources: they keep running, but the next +- **State lives on this host only, and it is a secret.** `infra/terraform.tfstate` + is gitignored and `just backup` does not touch it. Copy it off-host next to + the backups, with the same care as `.env`: state stores the tunnel secret + and every API response in plain text, so whoever can read it can run the + tunnel. Losing it orphans the Cloudflare resources: they keep running, but the next apply creates duplicates, and recovery is `tofu import` by hand. ## Upgrading images From 423b71920ed8dfd61b304f3310390f78e51bf2aa Mon Sep 17 00:00:00 2001 From: Simon van Lierde Date: Sun, 6 Sep 2026 14:53:25 +0000 Subject: [PATCH 45/85] ci(dependabot): group patch bumps per directory and the demo into one PR - hub and spoke images: patches grouped, minors and majors one per PR so a red PR names its image - demo pip and docker updates: one monthly group --- .github/dependabot.yml | 25 +++++++++++++++++-------- CHANGELOG.md | 4 +++- docs/RUNBOOK.md | 4 +++- 3 files changed, 23 insertions(+), 10 deletions(-) diff --git a/.github/dependabot.yml b/.github/dependabot.yml index eaa3791..cfdb73e 100644 --- a/.github/dependabot.yml +++ b/.github/dependabot.yml @@ -1,37 +1,46 @@ version: 2 updates: - # Deliberately ungrouped: a grouped bump that fails `just check` or `just - # smoke` gives no clue which of six images broke it, and bisecting a - # Dependabot PR by hand is worse than reviewing six small ones. + # Hub images: patch bumps ride together, minors and majors come one per PR. + # Patches rarely change config syntax, so one CI run covers a batch; a + # minor or major that fails `just check` or `just smoke` then names its own + # image instead of leaving six to bisect. - package-ecosystem: "docker-compose" directory: "/" schedule: interval: "weekly" - + groups: + hub-patches: + update-types: ["patch"] # The spoke images every project host runs (Alloy, socket proxy, GPU # exporter). Digest pins keep them reproducible, not patched. - package-ecosystem: "docker-compose" directory: "/templates" schedule: interval: "weekly" - + groups: + spoke-patches: + update-types: ["patch"] + # The demo is a toy: one PR per month for all of it. - package-ecosystem: "docker" directory: "/demo" schedule: interval: "monthly" - + groups: + demo: + patterns: ["*"] - package-ecosystem: "pip" directory: "/demo" schedule: interval: "monthly" - + groups: + demo: + patterns: ["*"] # Cloudflare provider for infra/. Dependabot bumps the constraint in # main.tf but not the hashes in .terraform.lock.hcl — see the runbook. - package-ecosystem: "terraform" directory: "/infra" schedule: interval: "monthly" - - package-ecosystem: "github-actions" directory: "/" schedule: diff --git a/CHANGELOG.md b/CHANGELOG.md index 0ed1635..6fa46a0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -142,7 +142,9 @@ queries that had been measuring the wrong thing. so an ingress edited in the Cloudflare dashboard cannot reach a backend that has no authentication of its own. - Every hub service carries a `pids_limit`; Dependabot also watches the spoke - images pinned in `templates/`. + images pinned in `templates/`. Patch bumps are grouped per directory and + the demo's dependencies into one monthly PR; minors and majors stay one + per PR. - GitHub Actions are pinned to commit SHAs, and `just infra-validate` runs against a copy of the sources so state and tfvars never enter the container. diff --git a/docs/RUNBOOK.md b/docs/RUNBOOK.md index 2201ac9..533c80c 100644 --- a/docs/RUNBOOK.md +++ b/docs/RUNBOOK.md @@ -203,7 +203,9 @@ Dependabot opens PRs that bump the pinned versions, and CI runs `just check` on each one. The validators (promtool, otelcol) read their image versions from `compose.yml`, so every bump is checked with the exact binaries the stack will run. A new version that changes its config syntax -fails CI before it reaches the host. +fails CI before it reaches the host. Patch bumps arrive grouped, one PR for +the hub images and one for the spoke images; a minor or major comes on its +own, so a red PR names the one image that broke. Dependabot also watches the Cloudflare provider in `infra/`. Those PRs need one manual step: it bumps the constraint in `main.tf` but not the recorded From 72d9e786b599210cc46f538322e59a1eb7bc0233 Mon Sep 17 00:00:00 2001 From: Simon van Lierde Date: Sun, 6 Sep 2026 15:04:51 +0000 Subject: [PATCH 46/85] docs: update phrasing across repo --- .env.example | 13 +-- .github/dependabot.yml | 7 +- .yamlfmt | 1 + CHANGELOG.md | 32 +++--- README.md | 83 +++++++------- bootstrap.sh | 46 ++++---- compose.sandbox.yml | 15 +-- compose.yml | 54 +++++----- config/grafana/alerting/contact-points.yaml | 8 +- config/grafana/alerting/coverage.yaml | 10 +- .../alerting/notification-policies.yaml | 7 +- .../alerting/project-relab-staging.yaml | 16 +-- config/grafana/alerting/rules.yaml | 8 +- config/loki.yaml | 9 +- config/otel-collector.yaml | 19 ++-- config/prometheus.yaml | 4 +- config/tempo.yaml | 11 +- demo/app.py | 4 +- demo/pyproject.toml | 37 ++++--- docs/ONBOARDING.md | 16 +-- docs/RUNBOOK.md | 69 ++++++------ docs/adr/0001-observability-stack.md | 18 ++-- docs/adr/0002-hub-and-spoke-observability.md | 35 +++--- infra/generate-imports.sh | 62 ++++++----- infra/main.tf | 8 +- infra/terraform.tfvars.example | 2 +- justfile | 54 +++++----- templates/README.md | 26 ++--- templates/alerting/coverage.yaml.tmpl | 10 +- templates/alerting/project.yaml.tmpl | 16 +-- templates/alloy/config.alloy | 102 +++++++++--------- templates/compose.telemetry.gpu.yml | 23 ++-- templates/compose.telemetry.yml | 82 +++++++------- templates/run_scheduled.sh | 60 +++++------ 34 files changed, 495 insertions(+), 472 deletions(-) diff --git a/.env.example b/.env.example index 72678fc..d590426 100644 --- a/.env.example +++ b/.env.example @@ -24,14 +24,15 @@ GRAFANA_COOKIE_SECURE=false DEPARTMENT=cml # Bearer token every telemetry sender must present (Authorization: Bearer ). -# The default only suits local use — generate a real one for production, e.g.: openssl rand -hex 32 +# The default only suits local use. Generate a real one for production: +# openssl rand -hex 32 OTLP_AUTH_TOKEN=local-dev-token # Where Grafana delivers alert notifications (any webhook: ntfy, Slack, …). -# Not optional. An empty value makes every alert fail delivery silently while the -# heartbeat below keeps pinging, so the dead man's switch reads healthy and nothing -# reaches anyone. With the tunnel overlay active, `just up` refuses to start -# without it. +# Not optional. An empty value makes every alert fail delivery silently. The +# heartbeat below keeps pinging, so the dead man's switch reads healthy while +# nothing reaches anyone. With the tunnel overlay active, `just up` refuses to +# start without it. ALERT_WEBHOOK_URL= # Dead man's switch ping target (e.g. https://hc-ping.com/). The Watchdog @@ -52,7 +53,7 @@ CLOUDFLARE_TUNNEL_TOKEN= # instead of everyone sharing the admin login. Needs both values below; with # the tunnel overlay active, `just up` refuses to start without them. GRAFANA_JWT_AUTH=false -# Your Zero Trust team name — the in https://.cloudflareaccess.com. +# Your Zero Trust team name: the in https://.cloudflareaccess.com. CF_ACCESS_TEAM_DOMAIN= # The Grafana Access application's aud tag, so tokens minted for other apps in # the same team are rejected: cd infra && tofu output -raw grafana_access_aud diff --git a/.github/dependabot.yml b/.github/dependabot.yml index cfdb73e..8941593 100644 --- a/.github/dependabot.yml +++ b/.github/dependabot.yml @@ -11,6 +11,7 @@ updates: groups: hub-patches: update-types: ["patch"] + # The spoke images every project host runs (Alloy, socket proxy, GPU # exporter). Digest pins keep them reproducible, not patched. - package-ecosystem: "docker-compose" @@ -20,6 +21,7 @@ updates: groups: spoke-patches: update-types: ["patch"] + # The demo is a toy: one PR per month for all of it. - package-ecosystem: "docker" directory: "/demo" @@ -28,6 +30,7 @@ updates: groups: demo: patterns: ["*"] + - package-ecosystem: "pip" directory: "/demo" schedule: @@ -35,12 +38,14 @@ updates: groups: demo: patterns: ["*"] + # Cloudflare provider for infra/. Dependabot bumps the constraint in - # main.tf but not the hashes in .terraform.lock.hcl — see the runbook. + # main.tf but not the hashes in .terraform.lock.hcl; see the runbook. - package-ecosystem: "terraform" directory: "/infra" schedule: interval: "monthly" + - package-ecosystem: "github-actions" directory: "/" schedule: diff --git a/.yamlfmt b/.yamlfmt index db5abcf..7ac61d8 100644 --- a/.yamlfmt +++ b/.yamlfmt @@ -3,3 +3,4 @@ # in config/grafana/alerting/ every time yamlfmt runs. formatter: scan_folded_as_literal: true + retain_line_breaks_single: true diff --git a/CHANGELOG.md b/CHANGELOG.md index 6fa46a0..0cef9fc 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,17 +14,19 @@ queries that had been measuring the wrong thing. - **Durable export queue**: the collector's send queues are file-backed on a new `otel_queue` volume, so telemetry buffered during a backend outage - survives a collector restart. `just up` (and `demo`, `smoke`) - prepares the volume's ownership; it is deliberately not backed up. + survives a collector restart. `just up` (and `demo`, `smoke`) prepares the + volume's ownership. `just backup` skips the volume, whose contents are + worthless by the time anyone restores. - **Full-stack scraping**: Prometheus now scrapes Grafana, Loki, and Tempo as well, so `TargetDown` covers every service. - **Per-user Grafana logins**, optional: `GRAFANA_JWT_AUTH` plus `CF_ACCESS_TEAM_DOMAIN` and `CF_ACCESS_AUD` make Grafana verify the - Cloudflare Access JWT — pinned to this app's `aud` tag, both values enforced - by the exposure guards — instead of everyone sharing the admin password. + Cloudflare Access JWT (pinned to this app's `aud` tag, with both values + enforced by the exposure guards) instead of everyone sharing the admin + password. - **Memory ceilings** (`mem_limit`) on every service, sized from observed - usage, so one runaway component cannot OOM the host; the spoke Alloy agent - gains a matching in-pipeline memory limiter so a long hub outage sheds load + usage, so one runaway component cannot OOM the host. The spoke Alloy agent + gains a matching in-pipeline memory limiter, so a long hub outage sheds load instead of OOM-killing the agent and its loss counters with it. - **Tighter container defaults**: every service drops all capabilities, node-exporter (which holds `pid: host` and the host filesystem) runs @@ -51,7 +53,7 @@ queries that had been measuring the wrong thing. ### Changed -- **Ingestion hostname is `otel.`**, not `otlp.` — one +- **Ingestion hostname is `otel.`**, not `otlp.`: one department-wide name for machine telemetry alongside `grafana.` for humans. Every spoke's `OTEL_EXPORTER_OTLP_ENDPOINT` and any edge rule matching the old host have to follow. @@ -60,13 +62,13 @@ queries that had been measuring the wrong thing. It is set at the hub rather than by the sender, so a spoke cannot ship telemetry attributed to someone else. - **Overlays are host config now**: `COMPOSE_FILE` in `.env` names the compose - file set, and every recipe — `up`, `logs`, `ps`, `backup` — acts on that - same set. `just up-tunnel` is gone; its exposure guards run automatically + file set, and every recipe (`up`, `logs`, `ps`, `backup`) acts on that same + set. `just up-tunnel` is gone; its exposure guards run automatically whenever the tunnel overlay is active. - **Loki indexes only the identity labels** (`service.name`, `department`, - `project`, `env`, `host.name` — the authoritative list lives in + `project`, `env`, `host.name`; the authoritative list lives in `config/loki.yaml`). Everything else, `service.instance.id` included, is - structured metadata now — one stream per service instead of one per sender + structured metadata now: one stream per service instead of one per sender restart. Existing streams keep their old labels until they age out (30 days). - **Dashboards are provisioned, not editable**: `dashboards/*.json` is mounted read-only and UI saves are off, making the files the source of truth. @@ -84,7 +86,7 @@ queries that had been measuring the wrong thing. ### Removed -- **Alertmanager**: alerting is Grafana-managed now (ADR 0002) — rules are +- **Alertmanager**: alerting is Grafana-managed now (ADR 0002). Rules are provisioned from `config/grafana/alerting/`, and delivery still posts to `ALERT_WEBHOOK_URL`. Its `alertmanager_data` volume is left behind on an upgraded host; the runbook says when to remove it. @@ -122,8 +124,8 @@ queries that had been measuring the wrong thing. need before `${...}` in their configs expands at all. - **`HighErrorRate` merged environments**: aggregating by job alone let a healthy prod service dilute a broken staging one sharing the job name below - the threshold; it now keys on job, project and env like the other - multi-tenant rules, and `HostDiskSpaceLow` says whose disk is filling. + the threshold. It now keys on job, project and env like the other + multi-tenant rules. `HostDiskSpaceLow` says whose disk is filling. - With `GRAFANA_JWT_AUTH=true` but no team domain set, Grafana fetched its JWT signing keys from a placeholder `cloudflareaccess.com` subdomain any Cloudflare customer could claim; the fallback is gone and the guards refuse @@ -184,7 +186,7 @@ First tagged release: the stack is runnable, demoable, and validated in CI. ### Added -- `just demo`: one-command demo: an auto-instrumented FastAPI service under +- `just demo`: a one-command demo. An auto-instrumented FastAPI service under constant load populates Grafana with correlated traces, metrics, and logs. - `just check`: validation gate (compose syntax, Prometheus config + alert rules, collector config, YAML, workflows, dashboard JSON), all in pinned diff --git a/README.md b/README.md index 31dbc5b..64f3ee3 100644 --- a/README.md +++ b/README.md @@ -23,16 +23,17 @@ just demo ``` This starts the full stack plus a small FastAPI service under constant -artificial load (`compose.demo.yml`). It uses OpenTelemetry -auto-instrumentation and fails about one request in ten, on purpose. Give it -a minute, then open Grafana at (admin / change-me): +artificial load (`compose.demo.yml`). The service uses OpenTelemetry +auto-instrumentation and fails about one request in ten, which gives the error +panels and the error-rate alert something to show. Give it a minute, then open +Grafana at (admin / change-me): -- **Dashboards → Service Health (RED)** — request rate, error rate, and +- **Dashboards → Service Health (RED)**: request rate, error rate, and latency. The dots on the latency panel are exemplars: click one and Grafana opens the exact trace behind that measurement. -- **Dashboards → Logs** — log volume by service and level, an error - feed, and a live tail of everything arriving over OTLP. -- **Alerting → Alert rules** — the stack-health and error-rate rules Grafana +- **Dashboards → Logs**: log volume by service and level, an error feed, and + a live tail of everything arriving over OTLP. +- **Alerting → Alert rules**: the stack-health and error-rate rules Grafana is evaluating. `HighErrorRate` trips on the demo service after five minutes: one request in ten failing is twice the 5% threshold. @@ -42,8 +43,9 @@ a minute, then open Grafana at (admin / change-me): keeps running, and `just demo-destroy` takes the whole thing down. The demo runs under its own compose project on its own port, so it never joins -or disturbs a stack already running on the host — safe on the production box. -Same for `just smoke`, on :3001. Override with `DEMO_PORT` / `SMOKE_PORT`. +or disturbs a stack already running on the host. It is safe to run on the +production box. Same for `just smoke`, on :3001. Override with `DEMO_PORT` / +`SMOKE_PORT`. ## How it works @@ -60,7 +62,7 @@ flowchart LR cf --> otel cf --> grafana - subgraph host["Monitoring host — Docker Compose, ports bound to 127.0.0.1"] + subgraph host["Monitoring host: Docker Compose, ports bound to 127.0.0.1"] otel["OTel Collector
(ingestion gateway)"] otel -->|logs| loki["Loki"] otel -->|traces| tempo["Tempo"] @@ -74,7 +76,7 @@ at query time. Locally there is no tunnel: everything talks over the compose network, and Grafana is at `localhost:3000` for `just up`, `localhost:3002` for the isolated `just demo` stack. -The stack runs on a single host; at CML's telemetry volume, distributed +The stack runs on a single host. At CML's telemetry volume, distributed ingestion would add operational weight for no gain ([ADR 0001](docs/adr/0001-observability-stack.md) records the alternatives). The hub-and-spoke design for serving multiple CML projects is @@ -93,11 +95,13 @@ Grafana: (admin / whatever you set). `COMPOSE_FILE` in `.env` names the overlays a host runs. Set `COMPOSE_FILE=compose.yml:compose.tunnel.yml` in the production `.env`, and -every recipe (`up`, `logs`, `ps`, `backup`) acts on that same set. With the -tunnel overlay active, `just up` refuses to run until the settings that matter -once the stack is reachable are real: a generated `OTLP_AUTH_TOKEN`, a changed -`GRAFANA_ADMIN_PASSWORD`, `GRAFANA_ROOT_URL` pointing at the tunnel hostname, -and `GRAFANA_COOKIE_SECURE=true`. An empty `HEARTBEAT_URL` only warns. +every recipe (`up`, `logs`, `ps`, `backup`) acts on that same set. + +With the tunnel overlay active, `just up` refuses to run until four settings are +real: a generated `OTLP_AUTH_TOKEN`, a changed `GRAFANA_ADMIN_PASSWORD`, +`GRAFANA_ROOT_URL` pointing at the tunnel hostname, and +`GRAFANA_COOKIE_SECURE=true`. These are the settings that matter once the stack +is reachable. An empty `HEARTBEAT_URL` only warns. In production the stack sits behind a Cloudflare Tunnel, and that edge is code too. The tunnel, its hostnames, DNS, and the Cloudflare Access rule that puts @@ -111,19 +115,21 @@ and Tempo configs, the vendored Alloy config, YAML, workflows, shell scripts, the demo app's Python, OpenTofu formatting, dashboard JSON, and git history for leaked secrets. Every validator runs in a pinned container, so nothing is installed on the -host. Grafana's alerting provisioning has no offline validator, so `just smoke` -covers it: it boots the stack, waits for Grafana to report healthy, and checks -that every dashboard and every alert rule provisioned. It runs under its own -compose project on its own ports, so it cannot disturb a stack already running -on the host — `just smoke` is safe on the production box, and `just smoke-down` -cleans it up. CI runs both on every push and pull request. +host. + +Grafana's alerting provisioning has no offline validator. `just smoke` covers +it: it boots the stack, waits for Grafana to report healthy, and checks that +every dashboard and every alert rule provisioned. It runs under its own compose +project on its own ports, so it cannot disturb a stack already running on the +host. `just smoke` is safe on the production box, and `just smoke-down` cleans +it up. CI runs `just check` and `just smoke` on every push and pull request. ## Sending telemetry from a project You need the OTLP endpoint, the bearer token (`OTLP_AUTH_TOKEN`), and a few -naming conventions. Copy-paste templates for the two application routes — -zero-code Python/FastAPI, and plain OTLP environment variables — are in -**[docs/ONBOARDING.md](docs/ONBOARDING.md)**. Everything an application cannot +naming conventions. **[docs/ONBOARDING.md](docs/ONBOARDING.md)** holds +copy-paste templates for the two application routes: zero-code Python/FastAPI, +and plain OTLP environment variables. Everything an application cannot report about itself comes from the vendored agent in **[templates/README.md](templates/README.md)**. @@ -141,22 +147,25 @@ above 80%. Notifications go to whatever webhook you set in `ALERT_WEBHOOK_URL` as well as Prometheus, and one engine owning both means one answer to "who gets told". -One rule, `Watchdog`, fires permanently by design and posts to `HEARTBEAT_URL` -every five minutes. Point that at a dead man's switch such as healthchecks.io — -a service that alerts when the pings *stop* — to hear about the one failure the -host cannot report itself: its own death. Set both. An unset -`ALERT_WEBHOOK_URL` drops every alert while the heartbeat keeps reporting -healthy, so `just up` with the tunnel overlay refuses to start without it. +One rule, `Watchdog`, fires permanently and posts to `HEARTBEAT_URL` every +five minutes. Point that at a dead man's switch such as healthchecks.io, a +service that alerts when the pings *stop*. That covers the one failure the host +cannot report itself: its own death. + +Set both variables. An unset `ALERT_WEBHOOK_URL` drops every alert while the +heartbeat keeps reporting healthy, so `just up` with the tunnel overlay refuses +to start without it. ## Storage Everything persists to local Docker volumes (`loki_data`, `tempo_data`, -`prometheus_data`, `grafana_data`), all captured by `just backup`. A fifth, -`otel_queue`, holds the collector's on-disk export queue: seconds of in-flight -telemetry, worthless by the time anyone restores, so backups skip it. When -local disk stops fitting, Loki and Tempo can move to any S3-compatible object -store (Backblaze B2, Cloudflare R2, Hetzner, MinIO); the appendix of ADR 0001 -documents that change. +`prometheus_data`, `grafana_data`), all captured by `just backup`. Backups skip +a fifth volume, `otel_queue`. It holds the collector's on-disk export queue: +seconds of in-flight telemetry, worthless by the time anyone restores. + +When local disk stops fitting, Loki and Tempo can move to any S3-compatible +object store (Backblaze B2, Cloudflare R2, Hetzner, MinIO). The appendix of +ADR 0001 documents that change. ## Layout diff --git a/bootstrap.sh b/bootstrap.sh index f963bb3..dc1970b 100755 --- a/bootstrap.sh +++ b/bootstrap.sh @@ -3,7 +3,7 @@ # # ./bootstrap.sh # -# It does the four things a human would otherwise get subtly wrong: +# It does four things: # # 1. renders the keystone ProjectTelemetrySilent rule and reloads Grafana; # 2. regenerates the coverage rule, so a project that ships telemetry without ever @@ -13,9 +13,8 @@ # 4. prints the `.env` block to paste on the project host, and the curl that vendors # the templates at a pinned tag. # -# Bootstrap creates the safety net, not the telemetry: a host can ship perfectly good -# telemetry and still be unmonitored, because the rule that notices its silence lives -# here. +# Bootstrap creates the safety net, not the telemetry. A host can ship good telemetry +# and still be unmonitored, because the rule that notices its silence lives here. # # Idempotent: re-running it re-renders the same files and reloads again. set -euo pipefail @@ -26,9 +25,8 @@ if [[ -z "$project" || -z "$env_name" ]]; then echo "usage: $0 " >&2 exit 2 fi -# These become Prometheus label values, a Grafana rule uid, and a filename. Keep them -# boring: a label value with a quote or a brace in it produces a rule that silently -# never matches, which is the failure mode this whole stack exists to avoid. +# These become Prometheus label values, a Grafana rule uid, and a filename. A quote or a +# brace in a label value produces a rule that silently never matches. if [[ ! "$project" =~ ^[a-z0-9][a-z0-9-]*$ || ! "$env_name" =~ ^[a-z0-9][a-z0-9-]*$ ]]; then echo "error: project and env must match [a-z0-9][a-z0-9-]* (lowercase, no spaces)" >&2 exit 2 @@ -38,36 +36,36 @@ root="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)" cd "$root" # Read HEALTHCHECKS_API_KEY from .env if the environment does not already carry it. -# `just` recipes get .env via dotenv-load; a bare ./bootstrap.sh does not, and silently -# skipping check creation because of that is a confusing way to find out. Only this one -# key is read — sourcing the whole file would drag the stack's secrets into scope. +# `just` recipes get .env via dotenv-load; a bare ./bootstrap.sh does not, and would then +# skip check creation without saying why. Only this one key is read: sourcing the whole +# file would drag the stack's secrets into scope. # -# It must be the project's READ-WRITE API key: the read-only key cannot POST, and a ping +# It must be the project's READ-WRITE API key. The read-only key cannot POST, and a ping # key only sends pings. if [[ -z "${HEALTHCHECKS_API_KEY:-}" && -f .env ]]; then HEALTHCHECKS_API_KEY="$(sed -n 's/^HEALTHCHECKS_API_KEY=//p' .env | tail -1)" export HEALTHCHECKS_API_KEY fi -# Flat, not a subdirectory: Grafana's alerting provisioner does not recurse, and it -# skips a directory with a warning rather than an error — so a nested layout looks like -# it worked and provisions nothing. The project- prefix keeps them grouped in `ls`. +# Flat, not a subdirectory: Grafana's alerting provisioner does not recurse. It skips a +# nested directory with a warning, not an error, so that layout looks like it worked and +# provisions nothing. The project- prefix keeps the files grouped in `ls`. out_dir="config/grafana/alerting" -# Pinned tag for the vendoring curl. A moving ref would let two projects vendor two -# different agent configs and call it the same template — so no fallback to a branch. +# Pinned tag for the vendoring curl, with no fallback to a branch. A moving ref would let +# two projects vendor two different agent configs and call it the same template. tag="$(git -C "$root" describe --tags --abbrev=0 2>/dev/null)" \ || { echo "error: no release tag to pin the vendoring curls to; tag a release first" >&2; exit 1; } -# ...and the tag must actually contain the templates, or every curl 404s. +# The tag must also contain the templates, or every curl 404s. git -C "$root" rev-parse -q --verify "${tag}:templates/alloy/config.alloy" >/dev/null \ || { echo "error: tag ${tag} predates templates/; tag a new release before onboarding" >&2; exit 1; } repo_raw="https://raw.githubusercontent.com/CMLPlatform/monitoring/${tag}/templates" # --------------------------------------------------------------- 1. the keystone rules -# Every covered project/environment, read back from the COVERS marker each rendered file -# carries rather than from a list someone has to remember to update, plus the pair being -# bootstrapped now. The pair matters, not just the project: a project bootstrapped for -# staging that also ships prod would otherwise read as covered while prod has no -# keystone rule at all — the same silent gap one level further in. +# Every covered project/environment, read back from the COVERS marker in each rendered +# file, plus the pair being bootstrapped now. No hand-maintained list to fall behind. +# +# The pair matters, not just the project. A project bootstrapped for staging that also +# ships prod would otherwise read as covered while prod has no keystone rule at all. pairs="$({ sed -n 's/^# COVERS: //p' "$out_dir"/project-*.yaml 2>/dev/null || true echo "$project $env_name"; } | sort -u)" @@ -142,14 +140,14 @@ curl -fsSL -o compose.telemetry.yml ${repo_raw}/compose.telemetry.yml curl -fsSL -o compose.telemetry.gpu.yml ${repo_raw}/compose.telemetry.gpu.yml curl -fsSL -o scripts/run_scheduled.sh ${repo_raw}/run_scheduled.sh -Verify before executing anything — hashes taken from the ${tag} tag here, so a +Verify before executing anything. The hashes come from the ${tag} tag, so a repo compromise after tagging cannot silently change what project hosts run: sha256sum -c <<'SUM' $(for pair in "alloy/config.alloy deploy/alloy/config.alloy" \ "compose.telemetry.yml compose.telemetry.yml" \ "compose.telemetry.gpu.yml compose.telemetry.gpu.yml" \ "run_scheduled.sh scripts/run_scheduled.sh"; do - # Deliberate word splitting: each pair is " ". + # Word splitting is intended here: each pair is " ". # shellcheck disable=SC2086 set -- $pair printf '%s %s\n' "$(git -C "$root" show "${tag}:templates/$1" | sha256sum | cut -d' ' -f1)" "$2" diff --git a/compose.sandbox.yml b/compose.sandbox.yml index 677f174..185d4d6 100644 --- a/compose.sandbox.yml +++ b/compose.sandbox.yml @@ -1,14 +1,15 @@ # Isolation overlay shared by `just smoke` and `just demo`. # # Smoke runs the core stack under its own compose project name (see the justfile), -# which already gives it separate containers and volumes. The one thing a project -# name cannot separate is host port bindings: compose.yml publishes 3000, 4317 and -# 4318 on 127.0.0.1, so a smoke run on a host that is already serving the stack -# would fail to bind — or, worse on a host where the stack is down, quietly answer -# on the port something else expects to own. +# which already gives it separate containers and volumes. Host port bindings are +# the one thing a project name cannot separate: compose.yml publishes 3000, 4317 +# and 4318 on 127.0.0.1. A smoke run on a host that already serves the stack +# would fail to bind, and on a host where the stack is down it would quietly +# answer on a port something else expects to own. # -# So: drop the ingestion ports entirely (nothing ships telemetry during a smoke -# run) and move Grafana to SANDBOX_PORT, which the assertions in `just smoke` and `just demo` probe. +# This overlay drops the ingestion ports entirely (nothing ships telemetry during +# a smoke run) and moves Grafana to SANDBOX_PORT, which the assertions in +# `just smoke` and `just demo` probe. services: otel-collector: ports: !override [] diff --git a/compose.yml b/compose.yml index 7b48cdf..5a3d1b6 100644 --- a/compose.yml +++ b/compose.yml @@ -8,9 +8,9 @@ name: monitoring # Grafana → UI # # Projects ship telemetry to this host via OTLP. Do NOT publish :4317/:4318 -# directly to the public internet — expose via Cloudflare Tunnel, Tailscale, -# WireGuard, or similar. The port bindings below are bound to 127.0.0.1 so -# they are only reachable from the host / tunnel sidecar. +# directly to the public internet. Expose them via Cloudflare Tunnel, Tailscale, +# WireGuard, or similar. The ports below bind to 127.0.0.1, so only the host and +# the tunnel sidecar reach them. x-logging: &default-logging driver: json-file @@ -23,11 +23,11 @@ x-security: &default-security # Every service also drops all capabilities and carries a pids limit: they all # run unprivileged uids, write only to named volumes, and need none. The client -# templates go further (read-only roots) — see templates/compose.telemetry.yml. +# templates go further, with read-only roots (templates/compose.telemetry.yml). -# The mem_limit on each service is sized from observed steady-state usage with -# generous headroom for bursts. They are ceilings, not reservations: the point -# is that one runaway component can't OOM a host that also runs production. +# Each mem_limit is sized from observed steady-state usage with headroom for +# bursts. They are ceilings, not reservations: they stop one runaway component +# from OOMing a host that also runs production. services: otel-collector: @@ -46,7 +46,8 @@ services: volumes: - ./config/otel-collector.yaml:/etc/otelcol/config.yaml:ro # File-backed exporter queue: telemetry buffered during a backend outage - # survives a collector restart. Owned by uid 10001 — see justfile. + # survives a collector restart. The volume must be owned by uid 10001; + # `just up` chowns it. - otel_queue:/var/lib/otelcol/queue command: ["--config=/etc/otelcol/config.yaml"] environment: @@ -83,7 +84,7 @@ services: - ./config/tempo.yaml:/etc/tempo/tempo.yaml:ro - tempo_data:/var/tempo command: ["-config.file=/etc/tempo/tempo.yaml"] - # Distroless too — see the note on loki. + # Distroless too, so no healthcheck (see loki). mem_limit: 2g pids_limit: 1024 cap_drop: [ALL] @@ -125,8 +126,8 @@ services: command: ["--path.rootfs=/host"] pid: host mem_limit: 128m - # Hardened hardest of the set: it holds pid:host plus the whole host - # filesystem read-only, so a compromise here sees the most. + # Holds pid:host plus the whole host filesystem, so a compromise here sees + # more than in any other service: hence the tightest limits of the set. cap_drop: [ALL] read_only: true pids_limit: 64 @@ -134,7 +135,7 @@ services: volumes: # No rslave: unsupported on Docker Desktop; only affects mounts added after start. # No network_mode:host either, so node_network_* describes the container - # veth, not host NICs — fine while nothing alerts on network metrics. + # veth, not host NICs. Fine while nothing alerts on network metrics. - /:/host:ro networks: [backend] logging: *default-logging @@ -159,28 +160,27 @@ services: - grafana_data:/var/lib/grafana environment: GF_SECURITY_ADMIN_PASSWORD: ${GRAFANA_ADMIN_PASSWORD:?set GRAFANA_ADMIN_PASSWORD in .env} - # Expanded by Grafana into the provisioned contact points. Alertmanager needed - # a url_file written by an entrypoint because it cannot read env vars; Grafana - # can, so that hack is gone along with its tmpfs. + # Expanded by Grafana into the provisioned contact points. ALERT_WEBHOOK_URL: ${ALERT_WEBHOOK_URL:-} HEARTBEAT_URL: ${HEARTBEAT_URL:-} GF_USERS_ALLOW_SIGN_UP: "false" GF_SERVER_ROOT_URL: ${GRAFANA_ROOT_URL:-http://localhost:3000} # Land on Stack Health instead of the empty welcome page. GF_DASHBOARDS_DEFAULT_HOME_DASHBOARD_PATH: /var/lib/grafana/dashboards/stack-health.json - # Secure cookies would break plain-http localhost logins, so this is - # opt-in; with the tunnel overlay active, `just up` refuses to expose - # Grafana without it. + # Opt-in: secure cookies would break plain-http localhost logins. With + # the tunnel overlay active, `just up` refuses to expose Grafana without + # it. GF_SECURITY_COOKIE_SECURE: ${GRAFANA_COOKIE_SECURE:-false} GF_SECURITY_COOKIE_SAMESITE: strict # Opt-in per-user identity from Cloudflare Access, replacing the shared # admin login. Off until GRAFANA_JWT_AUTH=true plus CF_ACCESS_TEAM_DOMAIN # and CF_ACCESS_AUD are set in .env (the exposure guards enforce the - # pair); new users land on the org's default role (Viewer). No fallback - # domain: an unset team name must fail closed, not fetch signing keys - # from a claimable cloudflareaccess.com subdomain. The aud pin matters - # because the JWK set is team-wide — without it a token minted for any - # other Access app in the team is accepted here too. + # pair). New users land on the org's default role (Viewer). + # No fallback domain: an unset team name must fail closed, not fetch + # signing keys from a claimable cloudflareaccess.com subdomain. + # The aud pin matters because the JWK set is team-wide. Without it, a + # token minted for any other Access app in the team is accepted here + # too. GF_AUTH_JWT_ENABLED: ${GRAFANA_JWT_AUTH:-false} GF_AUTH_JWT_HEADER_NAME: Cf-Access-Jwt-Assertion GF_AUTH_JWT_JWK_SET_URL: https://${CF_ACCESS_TEAM_DOMAIN:-}.cloudflareaccess.com/cdn-cgi/access/certs @@ -204,8 +204,8 @@ services: # Loki, Prometheus and Tempo have no authentication of their own, so they live # on a network with no gateway that only Grafana and the collector can reach. # cloudflared (compose.tunnel.yml) stays on `default`: its ingress list is -# fetched from Cloudflare at runtime, and this is what keeps an edited ingress -# from publishing a backend the token gate on the collector never sees. +# fetched from Cloudflare at runtime, so an edited ingress cannot publish a +# backend that skips the collector's token gate. networks: backend: internal: true @@ -215,6 +215,6 @@ volumes: tempo_data: prometheus_data: grafana_data: - # Deliberately not in `just backup`: queue contents are seconds of in-flight - # telemetry, worthless by restore time. + # Not in `just backup`: queue contents are seconds of in-flight telemetry, + # worthless by restore time. otel_queue: diff --git a/config/grafana/alerting/contact-points.yaml b/config/grafana/alerting/contact-points.yaml index 89bbc9c..99b98cb 100644 --- a/config/grafana/alerting/contact-points.yaml +++ b/config/grafana/alerting/contact-points.yaml @@ -1,12 +1,12 @@ # Notification targets. Grafana expands $VAR in provisioning files, so the URLs come # straight from the environment and no secret is written to disk. # -# An unset variable leaves the URL empty and every notification fails — silently, from +# An unset variable leaves the URL empty and every notification fails, silently from # the outside. Nothing here catches that: AlertDeliveryFailing routes to this same -# webhook, so it fails exactly the way it is detecting, and the heartbeat rides its own +# webhook, so it fails the same way it is detecting, and the heartbeat rides its own # contact point and stays green throughout. The exposure guard in `just up` is the only -# control. AlertDeliveryFailing earns its place for the other case — a URL that is set -# but whose receiver is rejecting or flapping, where a later retry does get through. +# control. AlertDeliveryFailing covers the other case: a URL that is set but whose +# receiver rejects or flaps, where a later retry does get through. apiVersion: 1 diff --git a/config/grafana/alerting/coverage.yaml b/config/grafana/alerting/coverage.yaml index 84a24d6..2e5d97c 100644 --- a/config/grafana/alerting/coverage.yaml +++ b/config/grafana/alerting/coverage.yaml @@ -1,10 +1,10 @@ -# GENERATED by bootstrap.sh from templates/alerting/coverage.yaml.tmpl — do not edit -# the rendered file. Regenerated on every run. +# GENERATED by bootstrap.sh from templates/alerting/coverage.yaml.tmpl. Do not edit the +# rendered file; it is regenerated on every run. # -# The trap this closes: ProjectTelemetrySilent only exists for projects bootstrap.sh was +# The gap this closes: ProjectTelemetrySilent only exists for projects bootstrap.sh was # actually run for. A project that ships telemetry but skipped bootstrap is silently -# uncovered — the exact failure class the keystone closes, one level up. This fires on -# any series whose project/env pair has no rendered rule file. +# uncovered, one level up from the failure the keystone rule closes. This fires on any +# series whose project/env pair has no rendered rule file. # # The selector touches every series every project sends (~3k for one project). Cheap at # this size, evaluated instantly every 5m; revisit if a project's active series reach diff --git a/config/grafana/alerting/notification-policies.yaml b/config/grafana/alerting/notification-policies.yaml index 79ba2a7..7754eb7 100644 --- a/config/grafana/alerting/notification-policies.yaml +++ b/config/grafana/alerting/notification-policies.yaml @@ -1,6 +1,7 @@ -# Routing tree. Everything reaches the webhook except Watchdog, whose delivery -# IS the heartbeat: it fires permanently and its silence is the alarm, so it -# goes to the dead man's switch on a short repeat and never groups or waits. +# Routing tree. Everything reaches the webhook except Watchdog. Watchdog's +# delivery is itself the heartbeat: it fires permanently and its silence is the +# alarm, so it goes to the dead man's switch on a short repeat and never groups +# or waits. apiVersion: 1 diff --git a/config/grafana/alerting/project-relab-staging.yaml b/config/grafana/alerting/project-relab-staging.yaml index 65c41c3..b1712b9 100644 --- a/config/grafana/alerting/project-relab-staging.yaml +++ b/config/grafana/alerting/project-relab-staging.yaml @@ -1,12 +1,12 @@ -# Rendered by bootstrap.sh into config/grafana/alerting/, flat — Grafana's alerting -# provisioner does not recurse. Do not edit the rendered files by hand: re-run +# Rendered by bootstrap.sh into config/grafana/alerting/, flat: Grafana's alerting +# provisioner does not recurse. Do not edit the rendered files by hand. Re-run # bootstrap.sh, or edit this template so every project gets the fix. # -# This is the keystone: nothing on a spoke can detect its own absence. A project that -# ships telemetry but never had bootstrap.sh run is silently uncovered — that is what -# the ProjectsUncovered rule in coverage.yaml exists to catch, one level up. +# This is the keystone: nothing on a project host can detect its own absence. A project +# that ships telemetry but never had bootstrap.sh run is silently uncovered, which is +# what the ProjectsUncovered rule in coverage.yaml catches one level up. # -# The marker below is read back by bootstrap.sh to regenerate that coverage rule. It is +# bootstrap.sh reads the marker below back to regenerate that coverage rule. It carries # the pair, not the filename, because both halves may contain a dash: `project-a-b-c` # cannot be split back into the project and env that produced it. # COVERS: relab staging @@ -19,8 +19,8 @@ groups: folder: Stack alerts interval: 1m rules: - # A label-only selector, deliberately not target_info: agents attach the identity - # labels to the series themselves, so target_info carries none of them. absent() + # A label-only selector, not target_info: agents attach the identity labels to + # the series themselves, so target_info carries none of them. absent() # yields nothing while telemetry flows, so NoData is the HEALTHY state and must # map to OK or the rule reads inverted. - uid: proj-silent-relab-staging diff --git a/config/grafana/alerting/rules.yaml b/config/grafana/alerting/rules.yaml index fa531fd..3c16139 100644 --- a/config/grafana/alerting/rules.yaml +++ b/config/grafana/alerting/rules.yaml @@ -15,8 +15,8 @@ groups: folder: Stack alerts interval: 1m rules: - # Always firing on purpose; its ARRIVAL is the proof. execErrState Error is - # deliberate: if the query breaks the heartbeat stops and the switch alarms. + # Always firing; its arrival is the proof. execErrState is Error so that a + # broken query stops the heartbeat and the switch alarms. - uid: watchdog-heartbeat title: Watchdog condition: FIRING @@ -53,8 +53,8 @@ groups: annotations: summary: "Alerting-pipeline heartbeat (always firing)" description: "Routed to a dead man's switch. Investigate if the pings stop." - # Cannot page when the path it reports on is the broken one — that is what the - # heartbeat is for. It makes the state visible instead of silent. + # Cannot page when the path it reports on is the broken one; the heartbeat + # covers that case. It makes the state visible instead of silent. - uid: alert-delivery-failing title: AlertDeliveryFailing condition: FIRING diff --git a/config/loki.yaml b/config/loki.yaml index 539aa3d..a60faee 100644 --- a/config/loki.yaml +++ b/config/loki.yaml @@ -39,8 +39,8 @@ limits_config: # Keep ingestion generous for a single-tenant homelab. ingestion_rate_mb: 16 ingestion_burst_size_mb: 32 - # Index only the identity labels listed below — the authoritative set; the - # docs point here. Loki's default OTLP mapping also indexes + # Index only the identity labels listed below. That list is the authoritative + # set, and the docs point here. Loki's default OTLP mapping also indexes # service.instance.id, which mints a fresh stream on every sender restart; # it is now structured metadata, so query it with `| service_instance_id=...`. otlp_config: @@ -49,9 +49,8 @@ limits_config: # handed. These five are the identity labels ADR 0002 requires on every # signal, and they have to be real stream LABELS, not structured metadata: # a `project` template variable is a stream selector, and label_values() - # cannot see structured metadata. All five are bounded — one value per - # department, project, environment and host — so indexing them costs - # nothing. + # cannot see structured metadata. All five are bounded (one value per + # department, project, environment and host), so indexing them is cheap. ignore_defaults: true attributes_config: - action: index_label diff --git a/config/otel-collector.yaml b/config/otel-collector.yaml index 9dcf399..0281b6d 100644 --- a/config/otel-collector.yaml +++ b/config/otel-collector.yaml @@ -1,6 +1,6 @@ # Ingestion gateway. Receives OTLP from all projects and fans out to the -# appropriate backend. Keep this config dumb — do shaping/enrichment at the -# app edge (per-project collector) where it has context. +# appropriate backend. Keep this config dumb. Shaping and enrichment belong at +# the app edge (per-project collector), which has the context for them. # All senders must present "Authorization: Bearer ". The # OTLP hostnames are public behind the tunnel, so ingestion needs auth. @@ -9,7 +9,7 @@ extensions: token: ${env:OTLP_AUTH_TOKEN} # Backs the exporter queues below so buffered telemetry survives collector # restarts (every image bump is one). The volume must be writable by the - # image's uid 10001 — `just up` handles that (see _queue-volume). + # image's uid 10001; `just up` chowns it (see _queue-volume in the justfile). file_storage: directory: /var/lib/otelcol/queue create_directory: true @@ -40,7 +40,7 @@ processors: # Absolute, not percentage: percentages are read against the cgroup, and an # unbounded container makes that the whole host. Sized to the container's - # mem_limit (512m in compose.yml) — change both together. + # mem_limit (512m in compose.yml); keep the two in step. memory_limiter: check_interval: 2s limit_mib: 400 @@ -49,13 +49,12 @@ processors: timeout: 5s send_batch_size: 1024 -# The queue/retry settings are pinned rather than left to upstream defaults, -# because the RUNBOOK's "buffers about five minutes" promise is exactly those -# numbers. Defined once on the loki exporter, aliased onto the other two. The -# queue is file-backed, so both a backend outage and a collector restart are -# ridden out. +# The queue and retry settings are pinned: the RUNBOOK's "buffers about five +# minutes" promise is exactly these numbers. Defined once on the loki exporter +# and aliased onto the other two. The queue is file-backed, so it rides out both +# a backend outage and a collector restart. exporters: - # Loki 3.x accepts OTLP natively — no Promtail / loki exporter needed. + # Loki 3.x accepts OTLP natively, so no Promtail or loki exporter is needed. otlp_http/loki: endpoint: http://loki:3100/otlp sending_queue: &queue diff --git a/config/prometheus.yaml b/config/prometheus.yaml index 73f19f4..2d34317 100644 --- a/config/prometheus.yaml +++ b/config/prometheus.yaml @@ -6,8 +6,8 @@ global: storage: tsdb: - # OTel's guide requires this for OTLP ingestion — without it, late - # batches are silently dropped. + # Required for OTLP ingestion: without it, late batches are silently + # dropped. out_of_order_time_window: 30m otlp: diff --git a/config/tempo.yaml b/config/tempo.yaml index 5e9a1ce..9447b5e 100644 --- a/config/tempo.yaml +++ b/config/tempo.yaml @@ -21,16 +21,15 @@ storage: local: path: /var/tempo/blocks -# No metrics_generator: RED comes from the applications' own OTLP metrics (ADR 0002). -# Deriving it from traces as well meant two producers for one number, a Prometheus -# remote-write path that existed for nothing else, and a cardinality backstop to -# maintain. Tempo is trace storage now, which makes it disposable on its next -# breaking upgrade. +# No metrics_generator: RED comes from the applications' own OTLP metrics (ADR +# 0002). Deriving it from traces too would give two producers for one number. +# Tempo stores traces and nothing else. overrides: defaults: compaction: - block_retention: 168h # 7 days — traces are bulky, tune to disk budget + # 7 days. Traces are bulky; tune this to the disk budget. + block_retention: 168h usage_report: reporting_enabled: false diff --git a/demo/app.py b/demo/app.py index a43c6ac..ad0db1d 100644 --- a/demo/app.py +++ b/demo/app.py @@ -1,7 +1,7 @@ """Minimal FastAPI service for the demo overlay. All telemetry (traces, metrics, logs with trace context) comes from OTel -auto-instrumentation — see compose.demo.yml. No OTel code needed here. +auto-instrumentation; see compose.demo.yml. No OTel code needed here. """ import logging @@ -26,7 +26,7 @@ def root() -> dict[str, bool]: @app.get("/work") def work() -> dict[str, bool]: """Simulate variable-latency work that sometimes fails.""" - time.sleep(random.uniform(0.02, 0.3)) # noqa: S311 — not crypto, just jitter + time.sleep(random.uniform(0.02, 0.3)) # noqa: S311 (not crypto, just jitter) if random.random() < ERROR_RATE: # noqa: S311 log.error("work failed: upstream flaked") raise HTTPException(status_code=500, detail="upstream flaked") diff --git a/demo/pyproject.toml b/demo/pyproject.toml index 4ff5f57..75aed8f 100644 --- a/demo/pyproject.toml +++ b/demo/pyproject.toml @@ -1,20 +1,25 @@ [project] -name = "demo" -version = "0.1.0" -requires-python = ">=3.14" -dependencies = [ - "fastapi==0.141.1", - "uvicorn==0.52.4", - "opentelemetry-distro==0.65b0", - "opentelemetry-exporter-otlp==1.44.0", - "opentelemetry-instrumentation-fastapi==0.65b0", -] + dependencies = [ + "fastapi==0.141.1", + "uvicorn==0.52.4", + "opentelemetry-distro==0.65b0", + "opentelemetry-exporter-otlp==1.44.0", + "opentelemetry-instrumentation-fastapi==0.65b0", + ] + name = "demo" + requires-python = ">=3.14" + version = "0.1.0" -# Pinned so lint results do not depend on whatever ruff config the contributor -# has at home. target-version is explicit because requires-python is ahead of -# what released ruff knows about. +# The ruff config lives here so lint results do not depend on whatever config the +# contributor has at home. target-version is explicit because requires-python is +# ahead of what released ruff knows about. [tool.ruff] -target-version = "py313" + target-version = "py313" -[tool.ruff.lint] -select = ["E4", "E7", "E9", "F", "I", "UP", "B"] + [tool.ruff.lint] + ignore = [ + "D203", # line too long + "E203", # whitespace before ':' + "W503", # line break before binary operator + ] + select = ["ALL"] diff --git a/docs/ONBOARDING.md b/docs/ONBOARDING.md index 451c7d9..7bd8c9a 100644 --- a/docs/ONBOARDING.md +++ b/docs/ONBOARDING.md @@ -9,7 +9,7 @@ project runs; a Python/FastAPI service needs no code changes at all. | | | | --- | --- | -| Production (via tunnel) | `https://otel.` — OTLP **HTTP** (`http/protobuf`) only | +| Production (via tunnel) | `https://otel.`, OTLP **HTTP** (`http/protobuf`) only | | Private network / same host | `:4317` (gRPC) or `:4318` (HTTP) | | Auth | `Authorization: Bearer ` (ask the stack operator) | @@ -41,7 +41,7 @@ HTTP server metrics (`http_server_request_duration_seconds`), which the auto-instrumentation below emits out of the box. Traces add per-request drill-down on top. -## Template 1 — Python/FastAPI, zero code changes +## Template 1: Python/FastAPI, zero code changes ```sh pip install opentelemetry-distro opentelemetry-exporter-otlp opentelemetry-instrumentation-fastapi @@ -64,7 +64,7 @@ trace context, with no OTel code in the app. A working example is this repo's [`demo/`](../demo/) service plus [`compose.demo.yml`](../compose.demo.yml). -## Template 2 — any language, plain OTLP +## Template 2: any language, plain OTLP Every OpenTelemetry SDK understands the same four environment variables: @@ -78,9 +78,9 @@ OTEL_RESOURCE_ATTRIBUTES=env=prod ## Container logs, host metrics, per-container metrics One Grafana Alloy agent per host ships everything the application cannot report about -itself: other containers' stdout, host resources, container lifecycle. It is vendored -from `templates/`, not written per project, and rides the same OTLP endpoint and token -as Templates 1 and 2 — no second hostname, no second credential. +itself: other containers' stdout, host resources, container lifecycle. The agent config +is vendored from `templates/`, not written per project. It rides the same OTLP endpoint +and token as Templates 1 and 2: no second hostname, no second credential. -Run `./bootstrap.sh ` on the monitoring host and follow what it prints. -See [templates/README.md](../templates/README.md). +Run `./bootstrap.sh ` on the monitoring host. Follow what it prints. See +[templates/README.md](../templates/README.md). diff --git a/docs/RUNBOOK.md b/docs/RUNBOOK.md index 533c80c..c19bf08 100644 --- a/docs/RUNBOOK.md +++ b/docs/RUNBOOK.md @@ -18,10 +18,10 @@ just restart ``` Two alerts point here. `TargetDown` fires after two minutes when -Prometheus cannot scrape a target. It scrapes every service — collector, -node-exporter, Grafana, Loki, Tempo, and itself — so the alert names -whichever one went quiet. `OtelExportFailures` means the collector is up but -a backend is rejecting its data: read that backend's logs, not the +Prometheus cannot scrape a target. Prometheus scrapes every service +(collector, node-exporter, Grafana, Loki, Tempo, and itself), so the alert +names whichever one went quiet. `OtelExportFailures` means the collector is up +but a backend is rejecting its data: read that backend's logs, not the collector's. ## Disk filling up (`HostDiskSpaceLow`) @@ -30,9 +30,9 @@ Retention is only partially size-bounded: | Data | Time limit | Size limit | | --- | --- | --- | -| Container stdout logs | — | json-file 10m × 3 per service | +| Container stdout logs | none | json-file 10m × 3 per service | | Prometheus TSDB | 30d | 15GB (`--storage.tsdb.retention.size`) | -| Loki chunks | 30d | none — Loki cannot cap total size | +| Loki chunks | 30d | none (Loki cannot cap total size) | | Tempo blocks | 7d | none | Loki and Tempo have no total-size knob, so the disk alert at 80% is the @@ -81,20 +81,21 @@ host is the intended setup. `OTEL_EXPORTER_OTLP_HEADERS`. Senders still on the old token get 401s (export errors on their side) until updated. A running demo overlay counts as a sender: re-run `just demo` to recreate it with the new token. - The token is shared and the collector does not verify `project`/`env` - against the sender, so every project host is trusted with every other - project's telemetry identity: a compromised host could spoof another - project's labels (and so quiet its silence alarm). Per-project tokens with - a collector-side identity check are the upgrade if that trust ever stops - being acceptable. + The token is shared, and the collector does not check `project`/`env` + against the sender. Every project host is therefore trusted with every + other project's telemetry identity. A compromised host could spoof another + project's labels, and so quiet that project's silence alarm. Per-project + tokens with a collector-side identity check are the upgrade if that trust + ever stops being acceptable. - **Tunnel token:** the tunnel is OpenTofu-managed, so read the token back from there, not from the dashboard. Rotate the tunnel secret in Cloudflare - Zero Trust, then `cd infra && tofu apply` (which refreshes the token data - source) and `tofu output -raw tunnel_token`. To rotate entirely from code, + Zero Trust. Then run `cd infra && tofu apply`, which refreshes the token + data source. Then read the new value with `tofu output -raw tunnel_token`. + To rotate entirely from code instead, expect ingestion and Grafana to be + unreachable for the minute or so it takes. `tofu apply -replace=cloudflare_zero_trust_tunnel_cloudflared.monitoring` - builds a new tunnel and repoints both CNAMEs at it; ingestion and Grafana - are unreachable for the minute or so that takes. Either way: new token into - `CLOUDFLARE_TUNNEL_TOKEN` in `.env`, then `just up`. + builds a new tunnel and repoints both CNAMEs at it. Either way: new token + into `CLOUDFLARE_TUNNEL_TOKEN` in `.env`, then `just up`. - **Grafana admin password:** change `GRAFANA_ADMIN_PASSWORD` in `.env`, then `docker compose up -d grafana`. - **Alert webhook and heartbeat URLs:** both are capability URLs, so the URL @@ -170,7 +171,7 @@ cd infra && tofu apply so for an urgent revocation also revoke the session in Zero Trust. At least one address has to remain; the variable's validation rejects an empty list, which would lock everyone out. Before adding people, check your plan's - Zero Trust seat limit in the Cloudflare dashboard — Cloudflare does not + Zero Trust seat limit in the Cloudflare dashboard. Cloudflare does not document the free-plan figure. - **Per-user Grafana logins:** by default everyone who clears Access shares the one admin password. Set `GRAFANA_JWT_AUTH=true`, @@ -178,8 +179,8 @@ cd infra && tofu apply `tofu output -raw grafana_access_aud`) in `.env` to make Grafana verify the Access JWT instead: each address signs in as itself, and new ones land on the org's default role (Viewer). The aud pin is required because the JWK - set is team-wide — without it a token minted for any other Access app in - the team would be accepted here too; the `just up` exposure guards refuse + set is team-wide. Without it, a token minted for any other Access app in + the team would be accepted here too. The `just up` exposure guards refuse to start JWT auth without both values. - **Adding a hostname:** add an `ingress` entry pointing at the service's container port, plus a matching `cloudflare_dns_record`. The catch-all @@ -190,17 +191,18 @@ cd infra && tofu apply app. Run `infra/generate-imports.sh > infra/imports.tf` first, check the plan reads 0 to add for the imported resources, apply, then delete `imports.tf`; it is a one-time instruction and gitignored. -- **State lives on this host only, and it is a secret.** `infra/terraform.tfstate` - is gitignored and `just backup` does not touch it. Copy it off-host next to - the backups, with the same care as `.env`: state stores the tunnel secret - and every API response in plain text, so whoever can read it can run the - tunnel. Losing it orphans the Cloudflare resources: they keep running, but the next - apply creates duplicates, and recovery is `tofu import` by hand. +- **State lives on this host only, and it is a secret.** State stores the + tunnel secret and every API response in plain text, so whoever can read it + can run the tunnel. `infra/terraform.tfstate` is gitignored, and + `just backup` does not touch it. Copy it off-host next to the backups, with + the same care as `.env`. Losing it orphans the Cloudflare resources: they + keep running, but the next apply creates duplicates, and recovery is + `tofu import` by hand. ## Upgrading images -Dependabot opens PRs that bump the pinned versions, and CI runs `just -check` on each one. The validators (promtool, otelcol) read their image +Dependabot opens PRs that bump the pinned versions, and CI runs +`just check` on each one. The validators (promtool, otelcol) read their image versions from `compose.yml`, so every bump is checked with the exact binaries the stack will run. A new version that changes its config syntax fails CI before it reaches the host. Patch bumps arrive grouped, one PR for @@ -222,8 +224,9 @@ Dependabot ecosystem covers a justfile, so bump those by hand. After merging, on the host: `git pull && just pull && just up`. Coming from a release that still ran Alertmanager, its volume outlives the service: `docker volume rm monitoring_alertmanager_data` once the new stack is up, and -`config/alertmanager.yaml` can go with it. Use `just -up`, not `docker compose up -d`: the recipe first chowns the collector's -queue volume to uid 10001. Where that volume is new, a raw compose up leaves -it root-owned and the collector crash-looping on a queue directory it cannot -write. +`config/alertmanager.yaml` can go with it. + +Use `just up`, not `docker compose up -d`. The recipe first chowns the +collector's queue volume to uid 10001. Where that volume is new, a raw compose +up leaves it root-owned and the collector crash-looping on a queue directory it +cannot write. diff --git a/docs/adr/0001-observability-stack.md b/docs/adr/0001-observability-stack.md index a01e322..73ff9a5 100644 --- a/docs/adr/0001-observability-stack.md +++ b/docs/adr/0001-observability-stack.md @@ -5,9 +5,9 @@ Date: 2026-07-03. Status: accepted (records a decision already in production). ## Context CML runs several long-lived research platforms (RELab and others) that need -their logs, traces, and metrics in one place. The volume is modest — a handful -of services at single-digit requests per second — but one small team maintains -all of it, so whatever we run must stay cheap and auditable. +their logs, traces, and metrics in one place. The volume is modest: a handful +of services at single-digit requests per second. One small team maintains all of +it, so whatever we run must stay cheap and auditable. ## Decision @@ -38,12 +38,12 @@ endpoints via Cloudflare Tunnel; bind everything else to `127.0.0.1`. ## Consequences -- The host is a single point of failure — acceptable, because the monitored +- The host is a single point of failure. That is acceptable: the monitored platforms degrade gracefully when telemetry stops (OTLP export is - fire-and-forget) and the stack rebuilds from this repo in minutes. + fire-and-forget), and the stack rebuilds from this repo in minutes. - Local disk bounds retention (30d logs/metrics, 7d traces). The escape hatch, reached before any move to distributed ingest, is S3-compatible storage for - Loki and Tempo — see the appendix below. + Loki and Tempo (see the appendix below). - Every image is pinned and validated by `just check` in CI, so the stack stays reproducible. @@ -51,8 +51,8 @@ endpoints via Cloudflare Tunnel; bind everything else to `127.0.0.1`. When local volumes stop fitting, Loki and Tempo move their object storage to any S3-compatible backend (Cloudflare R2, Backblaze B2, Hetzner, MinIO) -without touching the collector, Prometheus, or any client project. Not wired -up — don't start until credentials and a bucket exist. The concrete shape: +without touching the collector, Prometheus, or any client project. It is not +wired up: don't start until credentials and a bucket exist. The concrete shape: 1. Create s3 variants of the configs. `config/loki.s3.yaml` replaces `common.storage.filesystem` with: @@ -83,7 +83,7 @@ up — don't start until credentials and a bucket exist. The concrete shape: 2. Add a `compose.storage-s3.yml` overlay to `COMPOSE_FILE` in `.env` that mounts the s3 config variants over the originals and re-declares each - service's `command` with `-config.expand-env=true` appended — neither Loki + service's `command` with `-config.expand-env=true` appended. Neither Loki nor Tempo expands `${...}` in its config by default, and compose replaces `command` wholesale rather than merging it. Pass `S3_ENDPOINT`, `S3_ACCESS_KEY_ID`, and `S3_SECRET_ACCESS_KEY` through each service's diff --git a/docs/adr/0002-hub-and-spoke-observability.md b/docs/adr/0002-hub-and-spoke-observability.md index 497c8c5..7b0d247 100644 --- a/docs/adr/0002-hub-and-spoke-observability.md +++ b/docs/adr/0002-hub-and-spoke-observability.md @@ -1,7 +1,7 @@ # ADR 0002: Hub-and-spoke observability for CML projects Date: 2026-08-20. Status: accepted; **migration complete 2026-08-28**. The -transitional HANDOVER.md that tracked it has been deleted — it described a +transitional HANDOVER.md that tracked it has been deleted: it described a transition, not a system. Onboarding is [templates/README.md](../../templates/README.md). Supersedes one decision from ADR 0001: RED metrics move off Tempo's span-metrics @@ -14,25 +14,25 @@ This stack was built for RELab and must now serve multiple CML projects at very different maturity levels, including GPU hosts for computer-vision work. One part-time operator, Docker Compose everywhere, zero budget. The formative incident: a backup container crash-looped 668 times over 19 hours while every -monitor read green — the failure modes that matter are the ones with no detector +monitor read green. The failure modes that matter are the ones with no detector at all. ## Decision Three tiers, each owning distinct signals: -- **Per-project host (spoke):** one Grafana Alloy agent per host — container - stdout, host metrics (node exporter), and container lifecycle/resources - (cAdvisor) — plus the application's own OTel SDK for traces and app metrics. - The agent config is one shared file published by this repo, parameterised only - by environment variables; no project ever edits it. systemd timers run - scheduled jobs (backups, checks) through a wrapper that pings a per-job - dead-man's switch. +- **Per-project host (spoke):** one Grafana Alloy agent per host, covering + container stdout, host metrics (node exporter), and container + lifecycle/resources (cAdvisor). The application's own OTel SDK adds traces and + app metrics. The agent config is one shared file published by this repo, + parameterised only by environment variables; no project ever edits it. systemd + timers run scheduled jobs (backups, checks) through a wrapper that pings a + per-job dead-man's switch. - **Central host (hub):** this stack. One OTLP/HTTP endpoint, one bearer token, no per-backend hostnames or credentials, ever. Grafana is the *single* home for alert rules and notification (Alertmanager and Prometheus rule files go - away — Grafana-managed rules can query Loki, which the most valuable alerts - need). + away, because Grafana-managed rules can query Loki, which the most valuable + alerts need). - **Outside everything:** healthchecks.io as the per-job dead-man's switch, and an external HTTP prober for public reachability. These are the only detectors whose default state is alarm; everything else fails silent, and silence is @@ -51,14 +51,14 @@ Contracts that make it scale: lifecycle; healthchecks.io owns "did the job run"; a host-local drift script owns "is the deployed code the code we think"; nothing derives metrics from logs or from traces. -- **`ProjectTelemetrySilent` per project/env is the keystone alert** — nothing +- **`ProjectTelemetrySilent` per project/env is the keystone alert**: nothing on a spoke can detect its own absence. Templated and provisioned by `bootstrap.sh`, which is also what creates a project's healthchecks and prints its `.env` block. Bootstrap is what creates the safety net, not the telemetry. - **Onboarding is a copy, not a port:** vendor the template files at a pinned tag (three, plus one for a GPU host), add six `.env` variables, include the overlay, run `bootstrap.sh`. A GPU host is an ordinary host plus one opt-in - overlay (`nvidia_gpu_exporter` scraped by Alloy — not dcgm-exporter, whose + overlay (`nvidia_gpu_exporter` scraped by Alloy, not dcgm-exporter, whose profiling fields are datacentre-only) and three GPU alert rules. ## Alternatives considered @@ -78,8 +78,9 @@ Contracts that make it scale: machine-level batch jobs use node_exporter's textfile collector plus a dead-man's-switch check. - **A second alerting engine, SLOs, paging rotations, per-project dashboards, - long retention:** all rejected — one operator, alert count capped around ten, - dashboards carry a `project` template variable instead of per-project copies. + long retention:** all rejected. One operator, alert count capped around ten, + and dashboards carry a `project` template variable instead of per-project + copies. ## Consequences @@ -92,8 +93,8 @@ Contracts that make it scale: host came back. That tripwire has therefore been released. - Prometheus needs `out_of_order_time_window: 30m` for OTLP ingestion; without it late batches drop silently. The OTLP receiver is documented as a - low-volume path — the revisit trigger is a host exceeding a few thousand - active series. + low-volume path. The revisit trigger is a host exceeding a few thousand active + series. - Unverified at decision time: whether cAdvisor metric names survive the OTLP round trip (a 15-minute experiment decides between importing dashboard 15798 and adding a second ingestion path); Cloudflare free-plan Zero Trust seat diff --git a/infra/generate-imports.sh b/infra/generate-imports.sh index 06d0328..7327de0 100755 --- a/infra/generate-imports.sh +++ b/infra/generate-imports.sh @@ -3,20 +3,20 @@ # # This root was added after the edge already existed, so its first plan against an # empty state says "create" for objects that are already serving traffic. Applying -# that plan mints a SECOND tunnel beside `cml-monitoring` and DNS records that fight -# the live ones. Import blocks make the adoption reviewable: you read the generated -# file, then the plan, and only then apply. +# that plan mints a SECOND tunnel beside `cml-monitoring`, and DNS records that fight +# the live ones. Import blocks make the adoption reviewable: read the generated file, +# then the plan, and only then apply. # # The ingestion record is the one asymmetry. Live, it is still `otlp.`; this # root now calls it `otel.`. The block below imports that record as # `cloudflare_dns_record.otel`, so the apply RENAMES the record in place instead of -# creating a second one — which is also why there is no `moved` block in main.tf: -# nothing was ever in state under the old resource name. +# creating a second one. That is also why main.tf has no `moved` block: nothing was ever +# in state under the old resource name. # # The generated imports.tf is a throwaway, NOT something to commit: it names one -# account's resource ids and is meaningless after the apply that consumes it. Delete -# it once the apply has succeeded — import blocks are a one-time instruction, and -# leaving them in place re-runs them on every plan. +# account's resource ids and is meaningless after the apply that consumes it. Delete it +# once the apply has succeeded. Import blocks are a one-time instruction, and leaving +# them in place re-runs them on every plan. # # Usage (run in this directory, with terraform.tfvars already filled in): # export CLOUDFLARE_API_TOKEN=... # Tunnel:Read, DNS:Read, Access: Apps and Policies:Read @@ -42,9 +42,9 @@ api() { command -v jq >/dev/null || die "jq is required" : "${CLOUDFLARE_API_TOKEN:?is not set}" -# Read the ids from terraform.tfvars rather than asking for them again as TF_VAR_*: -# they are already there, and a second source of the same value is a second chance to -# get it wrong. An environment variable still wins if one is exported. +# Read the ids from terraform.tfvars, which already holds them. A second source of the +# same value is a second chance to get it wrong. An exported environment variable still +# wins. [[ -f terraform.tfvars ]] || die "no terraform.tfvars here; copy terraform.tfvars.example and fill it in" tfvar() { sed -n "s/^[[:space:]]*$1[[:space:]]*=[[:space:]]*\"\([^\"]*\)\".*/\1/p" terraform.tfvars | tail -1 @@ -53,9 +53,8 @@ account="${TF_VAR_account_id:-$(tfvar account_id)}" zone="${TF_VAR_zone_id:-$(tfvar zone_id)}" domain="${TF_VAR_domain:-$(tfvar domain)}" -# The example file's placeholders are valid-looking strings, and a plan run against -# them is what makes a fresh-create plan look plausible. Catch them here, where the -# message can say which line to edit. +# The example file's placeholders are valid-looking strings, so a plan run against them +# looks plausible. Catch them here, where the message can name the line to edit. for pair in "account_id:$account" "zone_id:$zone" "domain:$domain"; do value="${pair#*:}" [[ -n "$value" ]] || die "${pair%%:*} is empty in terraform.tfvars" @@ -64,14 +63,13 @@ for pair in "account_id:$account" "zone_id:$zone" "domain:$domain"; do esac done -# One lookup per resource kind. Each fails loudly when the resource is absent: a -# silently skipped import comes back as a "create" in the plan, which is the exact -# outcome this script exists to prevent. +# One lookup per resource kind. Each fails loudly when the resource is absent: a silently +# skipped import comes back as a "create" in the plan. lookup_dns_record() { local hostname="$1" id # Type-filtered: this root manages CNAMEs, and a name can also carry TXT records. # An unfiltered .result[0] could bind one of those, and the apply would rewrite it - # into a proxied CNAME — destroying a TXT record and leaving the CNAME unmanaged. + # into a proxied CNAME, destroying the TXT record and leaving the CNAME unmanaged. id="$(api "zones/$zone/dns_records?name=$hostname&type=CNAME" | jq -r '.result[0].id // empty')" [[ -n "$id" ]] || return 1 printf '%s' "$id" @@ -81,8 +79,8 @@ tunnel_name="${MONITORING_TUNNEL_NAME:-cml-monitoring}" tunnels="$(api "accounts/$account/cfd_tunnel?is_deleted=false")" tunnel_id="$(jq -r --arg name "$tunnel_name" '.result[] | select(.name == $name) | .id' <<<"$tunnels" | head -1)" if [[ -z "$tunnel_id" ]]; then - # Listing what IS there turns a wrong guess about the name from a dead end into a - # one-line fix: the tunnel is rarely absent, it is just called something else. + # List what IS there. The tunnel is rarely absent; it is usually called something + # else, and then the fix is one flag. echo "error: no tunnel named $tunnel_name in account $account" >&2 echo "tunnels that do exist in this account:" >&2 jq -r '.result[]? | " \(.name)\t\(.id)\tconnections=\(.connections | length)"' <<<"$tunnels" >&2 @@ -92,8 +90,8 @@ fi grafana_record="$(lookup_dns_record "grafana.$domain")" || die "no CNAME found for grafana.$domain" -# Still otlp. before the rename, otel. after it (or on a re-run). Try the new name -# first so a second run is a no-op rather than a resurrection of the old record. +# Still otlp. before the rename, otel. after it (or on a re-run). Try the new name first, +# so a second run is a no-op instead of resurrecting the old record. for host in "otel.$domain" "otlp.$domain"; do if otel_record="$(lookup_dns_record "$host")"; then ingestion_host="$host" @@ -102,13 +100,13 @@ for host in "otel.$domain" "otlp.$domain"; do done [[ -n "${ingestion_host:-}" ]] || die "no CNAME found for otel.$domain or otlp.$domain" -# Unlike the tunnel and the DNS records, the Access app may legitimately not exist — -# and then the apply SHOULD create it, because an unprotected Grafana hostname is the -# thing this root exists to close. So: warn, omit the import, let the plan create it. +# Unlike the tunnel and the DNS records, the Access app may legitimately not exist, and +# then the apply SHOULD create it: an unprotected Grafana hostname is what this root +# exists to close. So warn, omit the import, and let the plan create it. # -# Checked at both scopes. Apps predating account-scoped Access live under the zone, -# and one found there cannot be adopted by this resource as written (it is configured -# with account_id), so that case gets its own message rather than a silent create. +# Checked at both scopes. Apps predating account-scoped Access live under the zone, and +# one found there cannot be adopted by this resource as written (it is configured with +# account_id), so that case gets its own message instead of a silent create. access_apps="$(api "accounts/$account/access/apps?per_page=100")" access_app_id="$(jq -r --arg d "grafana.$domain" '.result[] | select(.domain == $d) | .id' <<<"$access_apps" | head -1)" if [[ -z "$access_app_id" ]]; then @@ -119,15 +117,15 @@ if [[ -z "$access_app_id" ]]; then account-scoped one. Recreate it at the account level, or give the resource zone_id instead of account_id, before importing." fi - echo "note: no Access application on grafana.$domain — omitting its import block so the" >&2 + echo "note: no Access application on grafana.$domain; omitting its import block so the" >&2 echo " apply CREATES it. Until that apply lands, Grafana's hostname is protected only" >&2 echo " by its own login. Access apps that do exist in this account:" >&2 jq -r '.result[]? | " \(.name)\t\(.domain)"' <<<"$access_apps" >&2 fi -# The policy is the one resource that may legitimately be absent: an app built in the -# dashboard usually carries an INLINE policy, which has no id to import. The apply -# then creates the reusable policy this root declares and reattaches the app to it. +# The policy may also be absent: an app built in the dashboard usually carries an INLINE +# policy, which has no id to import. The apply then creates the reusable policy this root +# declares and reattaches the app to it. access_policies="$(api "accounts/$account/access/policies")" access_policy_id="$(jq -r '.result[] | select(.name == "monitoring: allowed emails") | .id' <<<"$access_policies" | head -1)" diff --git a/infra/main.tf b/infra/main.tf index 501bb2f..ce25f21 100644 --- a/infra/main.tf +++ b/infra/main.tf @@ -7,12 +7,12 @@ # export CLOUDFLARE_API_TOKEN=... # needs Tunnel:Edit, DNS:Edit, Access:Edit # cd infra && tofu init # ./generate-imports.sh > imports.tf # the edge already exists: adopt it first -# tofu plan # expect "0 to add" — see the script's header +# tofu plan # expect "0 to add"; see the script's header # tofu apply && rm imports.tf # tofu output -raw tunnel_token # → CLOUDFLARE_TUNNEL_TOKEN in ../.env # -# State is local (infra/terraform.tfstate, gitignored) — one host, one -# operator; move it to R2 the day a second operator exists. +# State is local (infra/terraform.tfstate, gitignored): one host, one operator. +# Move it to R2 the day a second operator exists. terraform { required_version = ">= 1.8" @@ -106,7 +106,7 @@ resource "cloudflare_dns_record" "otel" { # Cloudflare Access in front of Grafana: email one-time-PIN at the edge, so # the public hostname never reaches Grafana's login page unauthenticated. -# The ingestion hostname is NOT behind Access — machines authenticate with the +# The ingestion hostname is NOT behind Access; machines authenticate with the # bearer token instead. resource "cloudflare_zero_trust_access_application" "grafana" { account_id = var.account_id diff --git a/infra/terraform.tfvars.example b/infra/terraform.tfvars.example index 62a40de..9ddd21c 100644 --- a/infra/terraform.tfvars.example +++ b/infra/terraform.tfvars.example @@ -1,5 +1,5 @@ # Copy to terraform.tfvars (gitignored) and fill in. The Cloudflare API token -# is NOT set here — export it: `export CLOUDFLARE_API_TOKEN=...` +# is NOT set here. Export it: `export CLOUDFLARE_API_TOKEN=...` # (needs Tunnel:Edit, DNS:Edit, and Access: Apps and Policies:Edit). # Cloudflare account ID (dash.cloudflare.com → any domain → overview sidebar). diff --git a/justfile b/justfile index 9af5aa9..278d0e1 100644 --- a/justfile +++ b/justfile @@ -50,7 +50,7 @@ _queue-volume project: @docker run --rm --network none -v {{project}}_otel_queue:/q alpine:3.24@sha256:28bd5fe8b56d1bd048e5babf5b10710ebe0bae67db86916198a6eec434943f8b chown 10001:10001 /q # Overlays are host config: COMPOSE_FILE in .env names the file set (see -# .env.example), and every recipe here — up, down, logs, ps, backup — acts on +# .env.example), and every recipe here (up, down, logs, ps, backup) acts on # that same set. With the tunnel overlay active, this refuses to start until the # exposure guards pass. # Start the stack (Grafana at http://localhost:3000). @@ -70,12 +70,12 @@ _expose-guards: @[ "${GRAFANA_COOKIE_SECURE:-false}" = "true" ] || { echo "error: GRAFANA_COOKIE_SECURE must be true when Grafana is served over HTTPS; set it in .env" >&2; exit 1; } @[ "${GRAFANA_JWT_AUTH:-false}" != "true" ] || { [ -n "${CF_ACCESS_TEAM_DOMAIN:-}" ] && [ -n "${CF_ACCESS_AUD:-}" ]; } || { echo "error: GRAFANA_JWT_AUTH=true needs CF_ACCESS_TEAM_DOMAIN and CF_ACCESS_AUD in .env (cd infra && tofu output -raw grafana_access_aud)" >&2; exit 1; } @[ -n "${HEARTBEAT_URL:-}" ] || echo "WARNING: HEARTBEAT_URL is empty; the stack goes live without a dead-man's switch" >&2 - @[ -n "${ALERT_WEBHOOK_URL:-}" ] || { echo "error: ALERT_WEBHOOK_URL is empty; every alert would fire into an empty webhook URL and be dropped. The heartbeat keeps pinging either way, so this failure looks healthy from the outside — set it, or comment out this guard deliberately" >&2; exit 1; } + @[ -n "${ALERT_WEBHOOK_URL:-}" ] || { echo "error: ALERT_WEBHOOK_URL is empty; every alert would fire into an empty webhook URL and be dropped. The heartbeat keeps pinging either way, so this failure looks healthy from the outside. Set it, or comment out this guard" >&2; exit 1; } down: docker compose down --remove-orphans -# See compose.demo.yml; watch it arrive at http://localhost:3002 (DEMO_PORT) — +# See compose.demo.yml; watch it arrive at http://localhost:3002 (DEMO_PORT): # its own Grafana, not the one `just up` serves on :3000. # Core stack plus a demo telemetry source, isolated from any running stack. demo: (_queue-volume demo_project) @@ -116,7 +116,7 @@ pull: tail service: docker compose logs -f --no-log-prefix {{service}} | jq -R 'fromjson? // .' -# All validators run in containers — no host installs, no network. The +# All validators run in containers: no host installs, no network. The # promtool/otelcol images are read from compose.yml so they can't drift from # the versions the stack actually runs. # Validate every config in the repo. @@ -133,7 +133,7 @@ check: # fails on a client machine, after vendoring. ENVIRONMENT=dummy PROJECT=dummy COMPOSE_PROJECT_NAME=dummy OTEL_EXPORTER_OTLP_ENDPOINT=https://dummy OTLP_AUTH_TOKEN=dummy docker compose -f templates/compose.telemetry.yml -f templates/compose.telemetry.gpu.yml config -q # The exposure guards, both ways: a fully set .env must pass, and each - # documented default must be refused on its own. Nothing else runs them — + # documented default must be refused on its own. Nothing else runs them: # CI never sets COMPOSE_FILE to the tunnel overlay. @good="OTLP_AUTH_TOKEN=t GRAFANA_ADMIN_PASSWORD=p GRAFANA_ROOT_URL=https://g.example GRAFANA_COOKIE_SECURE=true GRAFANA_JWT_AUTH=true CF_ACCESS_TEAM_DOMAIN=d CF_ACCESS_AUD=a HEARTBEAT_URL=https://h ALERT_WEBHOOK_URL=https://w"; \ env $good just _expose-guards; \ @@ -147,22 +147,22 @@ check: docker run --rm --network none -v ./config/tempo.yaml:/etc/tempo/tempo.yaml:ro $(echo "$images" | grep grafana/tempo) -config.file=/etc/tempo/tempo.yaml -config.verify=true # line-length at 120, not the default 80: digest-pinned image refs need # ~130 but count as non-breakable mappings. The rendered project-*/coverage - # rules are ignored — machine-written by bootstrap.sh, their expr lines - # grow with every onboarded project. + # rules are ignored: bootstrap.sh writes them, and their expr lines grow + # with every onboarded project. docker run --rm --network none -v .:/code:ro pipelinecomponents/yamllint:0.35.13 yamllint -d '{extends: relaxed, rules: {line-length: {max: 120, allow-non-breakable-inline-mappings: true}}, ignore: [.git/, backups/, infra/.terraform/, config/grafana/alerting/project-*.yaml, config/grafana/alerting/coverage.yaml]}' . docker run --rm --network none -v .:/repo:ro -w /repo rhysd/actionlint:1.7.12 -color docker run --rm --network none -v .:/mnt:ro koalaman/shellcheck:v0.11.0 bootstrap.sh templates/run_scheduled.sh infra/generate-imports.sh docker run --rm --network none -v ./demo:/demo:ro ghcr.io/astral-sh/ruff:0.14.2 check --no-cache /demo docker run --rm --network none -v ./infra:/infra:ro -w /infra ghcr.io/opentofu/opentofu:1.12.3 fmt -check docker run --rm --network none -v ./dashboards:/dashboards:ro ghcr.io/jqlang/jq:1.8.1 empty {{dash_paths}} - # The one config vendored verbatim onto every project host; a syntax error - # here otherwise first surfaces as a crash-looping agent on a client machine. - # Image ref matches templates/compose.telemetry.yml — keep them in step. # Secrets that reached git history. Scans commits, not the working tree, so - # it sees exactly what is in the repo and never the gitignored .env — a + # it sees exactly what is in the repo and never the gitignored .env. A # working-tree scan flags .env's real tokens and fails on every dev machine. # Needs full history: a shallow CI clone has one commit and passes vacuously. docker run --rm --network none -v .:/repo:ro zricethezav/gitleaks:v8.30.1@sha256:c00b6bd0aeb3071cbcb79009cb16a60dd9e0a7c60e2be9ab65d25e6bc8abbb7f git --redact --no-banner /repo + # The one config vendored verbatim onto every project host; a syntax error + # here otherwise first surfaces as a crash-looping agent on a client machine. + # Image ref matches templates/compose.telemetry.yml; keep them in step. docker run --rm --network none -v ./templates/alloy/config.alloy:/etc/alloy/config.alloy:ro -e COMPOSE_PROJECT_NAME=dummy -e ENVIRONMENT=dummy -e PROJECT=dummy -e OTEL_EXPORTER_OTLP_ENDPOINT=https://dummy -e OTLP_AUTH_TOKEN=dummy -e TELEMETRY_EDGE_KEY= grafana/alloy:v1.18.1@sha256:0f4434c92b3e6cdac38bb129b344e1790c246f7b6e2eaffcc16a5fa363240e33 validate /etc/alloy/config.alloy # Runs against a copy of the sources only: state and tfvars never enter the @@ -175,8 +175,8 @@ infra-validate: fmt: yamlfmt . -# The tarball is mode 0600 — it contains the Grafana DB and webhook secrets; -# copy it off-host, keep it private. Services are paused during the copy and +# The tarball is mode 0600: it contains the Grafana DB and webhook secrets, so +# copy it off-host and keep it private. Services are paused during the copy and # unpaused unconditionally afterwards: `pause` is per-container and can fail # halfway, so an unpause reached only on success would leave the stack frozen. # Snapshot all stateful volumes to backups/.tar.gz. @@ -197,43 +197,43 @@ restore file: _project-guard mkdir -p backups docker run --rm --network none {{backup_mounts}} -v ./backups:/backups alpine:3.24@sha256:28bd5fe8b56d1bd048e5babf5b10710ebe0bae67db86916198a6eec434943f8b sh -c 'umask 077 && tar czf /backups/pre-restore-$(date +%Y%m%d-%H%M%S).tar.gz -C /data .' docker run --rm --network none {{backup_mounts}} -v {{absolute_path(file)}}:/backup.tar.gz:ro alpine:3.24@sha256:28bd5fe8b56d1bd048e5babf5b10710ebe0bae67db86916198a6eec434943f8b sh -c 'for d in /data/*; do find "$d" -mindepth 1 -delete; done && tar xzf /backup.tar.gz -C /data' - @echo "Restored {{file}} — run 'just up' to start the stack." + @echo "Restored {{file}}. Run 'just up' to start the stack." # `--wait` does the readiness and crash-loop work: it blocks on the grafana and # prometheus healthchecks in compose.yml and fails if any container exits, so # there is no poll loop or `ps --status=exited` check to hand-roll here. -# Measured, because it is not obvious: a service with no healthcheck of its own -# (otel-collector, loki, tempo) still fails the wait while it is restarting, so -# a crash-looping collector is caught. It costs the full --wait-timeout to -# report, where the old explicit check failed immediately. -# What it cannot see is provisioning — Grafana answers /api/health long before -# it has read the provisioning dirs, and skips a broken dashboard or a malformed -# alert group silently — so that part is asserted below, on a single loop -# because both land asynchronously. +# A service with no healthcheck of its own (otel-collector, loki, tempo) still +# fails the wait while it is restarting, so a crash-looping collector is caught. +# That costs the full --wait-timeout to report, where an explicit exit check +# failed immediately. +# What `--wait` cannot see is provisioning. Grafana answers /api/health long +# before it has read the provisioning dirs, and skips a broken dashboard or a +# malformed alert group silently, so the assertions below cover that. They share +# one loop because both land asynchronously. # Boot an isolated copy of the core stack and assert it provisioned everything. smoke: (_queue-volume smoke_project) {{compose_smoke}} up -d --wait --wait-timeout 120 @want_dash="$(docker run --rm --network none -v ./dashboards:/dashboards:ro ghcr.io/jqlang/jq:1.8.1 -r .uid {{dash_paths}})"; want_rules=$(grep -h '^ *title:' config/grafana/alerting/*.yaml | wc -l); auth="user = \"admin:${GRAFANA_ADMIN_PASSWORD}\""; n=0; while :; do \ - search=$(printf '%s\n' "$auth" | curl -sf -K - '{{smoke_url}}/api/search?type=dash-db&limit=5000') && rules=$(printf '%s\n' "$auth" | curl -sf -K - {{smoke_url}}/api/v1/provisioning/alert-rules) || { echo "error: Grafana API request failed — wrong GRAFANA_ADMIN_PASSWORD, or Grafana is not answering on {{smoke_url}}" >&2; exit 1; }; \ + search=$(printf '%s\n' "$auth" | curl -sf -K - '{{smoke_url}}/api/search?type=dash-db&limit=5000') && rules=$(printf '%s\n' "$auth" | curl -sf -K - {{smoke_url}}/api/v1/provisioning/alert-rules) || { echo "error: Grafana API request failed. Check GRAFANA_ADMIN_PASSWORD, and that Grafana answers on {{smoke_url}}" >&2; exit 1; }; \ have=$(printf '%s' "$search" | docker run --rm -i ghcr.io/jqlang/jq:1.8.1 -r '.[].uid'); got=$(printf '%s' "$rules" | docker run --rm -i ghcr.io/jqlang/jq:1.8.1 length); \ missing=""; for uid in $want_dash; do printf '%s' "$have" | grep -qx "$uid" || missing="$missing $uid"; done; \ [ -z "$missing" ] && [ "$want_rules" = "$got" ] && break; \ - n=$((n+3)); [ $n -ge 60 ] && { echo "error: not provisioned after 60s — dashboards missing:${missing:- none}; alert rules $got/$want_rules (a malformed file provisions none of its group). See just smoke-logs" >&2; exit 1; }; \ + n=$((n+3)); [ $n -ge 60 ] && { echo "error: not provisioned after 60s. Dashboards missing:${missing:- none}; alert rules $got/$want_rules (a malformed file provisions none of its group). See just smoke-logs" >&2; exit 1; }; \ sleep 3; \ done # Provisioning proved the rules exist; this proves they can be delivered # and that the data path works. Grafana expands $VAR in the alerting - # provisioning files — a Grafana that stopped doing so would store the + # provisioning files. A Grafana that stopped doing so would store the # literal name and every notification would fail silently (see # contact-points.yaml). Then one OTLP log through the collector's bearer # auth, asserted back out of Loki with the department label the collector # stamps: the label chain the keystone rules key on, end to end. @auth="user = \"admin:${GRAFANA_ADMIN_PASSWORD}\""; \ - printf '%s\n' "$auth" | curl -sf -K - {{smoke_url}}/api/v1/provisioning/contact-points | docker run --rm -i ghcr.io/jqlang/jq:1.8.1 -e '[.[] | select(.uid | startswith("cp-")) | .settings.url] | all(startswith("$") | not)' >/dev/null || { echo "error: a contact point still carries a literal \$VAR — Grafana did not expand the alerting provisioning file" >&2; exit 1; }; \ + printf '%s\n' "$auth" | curl -sf -K - {{smoke_url}}/api/v1/provisioning/contact-points | docker run --rm -i ghcr.io/jqlang/jq:1.8.1 -e '[.[] | select(.uid | startswith("cp-")) | .settings.url] | all(startswith("$") | not)' >/dev/null || { echo "error: a contact point still carries a literal \$VAR; Grafana did not expand the alerting provisioning file" >&2; exit 1; }; \ ts="$(date +%s)000000000"; body='{"resourceLogs":[{"resource":{"attributes":[{"key":"service.name","value":{"stringValue":"smoke"}},{"key":"project","value":{"stringValue":"smoke"}},{"key":"env","value":{"stringValue":"ci"}}]},"scopeLogs":[{"logRecords":[{"timeUnixNano":"'"$ts"'","body":{"stringValue":"smoke"}}]}]}]}'; \ docker run --rm --network {{smoke_project}}_default alpine:3.24@sha256:28bd5fe8b56d1bd048e5babf5b10710ebe0bae67db86916198a6eec434943f8b wget -qO- --header="Authorization: Bearer ${OTLP_AUTH_TOKEN}" --header='Content-Type: application/json' --post-data="$body" http://otel-collector:4318/v1/logs >/dev/null || { echo "error: the collector refused an OTLP log with the .env token" >&2; exit 1; }; \ n=0; until printf '%s\n' "$auth" | curl -sf -K - -G --data-urlencode "query={project=\"smoke\",env=\"ci\",department=\"${DEPARTMENT:-cml}\"}" '{{smoke_url}}/api/datasources/proxy/uid/loki/loki/api/v1/query_range' | grep -q '"smoke"'; do \ - n=$((n+3)); [ $n -ge 60 ] && { echo "error: the smoke log never reached Loki with its department label — see just smoke-logs" >&2; exit 1; }; sleep 3; \ + n=$((n+3)); [ $n -ge 60 ] && { echo "error: the smoke log never reached Loki with its department label; see just smoke-logs" >&2; exit 1; }; sleep 3; \ done @echo "Stack healthy" diff --git a/templates/README.md b/templates/README.md index e217e4d..690be5b 100644 --- a/templates/README.md +++ b/templates/README.md @@ -49,7 +49,7 @@ for a missing detector. ## GPU hosts -Include `compose.telemetry.gpu.yml` as well and set `GPU_METRICS=1`. The agent config +Include `compose.telemetry.gpu.yml` as well. Set `GPU_METRICS=1`. The agent config already discovers the exporter by its Compose service label, so nothing else changes: a GPU host is an ordinary host plus one overlay. @@ -70,9 +70,9 @@ Three GPU rules are worth adding per GPU host. They are not provisioned by time() - nvidia_smi_xid_last_timestamp_seconds{xid=~"48|62|64|74|79|95|119|120"} < 300 ``` -XIDs are the GPU analogue of the crash-loop blind spot: a stuck kernel, an -uncorrectable memory fault, or a card that has fallen off the bus are all invisible to -utilisation graphs, and any of them silently kills a twelve-hour training run. +XIDs are the GPU analogue of the crash-loop blind spot. A stuck kernel, an +uncorrectable memory fault, and a card that has fallen off the bus are all invisible to +utilisation graphs. Any of them silently kills a twelve-hour training run. Both dashboards are provisioned centrally: `dashboards/gpu.json` (vendored from [14574](https://grafana.com/grafana/dashboards/14574); its multi-GPU companion is @@ -82,26 +82,28 @@ per-container resources. Nothing to import on the project host. ## Removing a project Deleting `config/grafana/alerting/project--.yaml` is **not** enough. -Grafana provisioning creates and updates rules from files but never deletes a rule -because its file vanished, and the API refuses to delete a provisioned rule (409, even +Grafana provisioning creates and updates rules from files, but it never deletes a rule +because its file vanished. The API refuses to delete a provisioned rule too (409, even with `X-Disable-Provenance`). The orphan keeps evaluating and firing. Delete the file, then drop the rule explicitly with a one-off provisioning file: ```yaml -# config/grafana/alerting/zz-delete.yaml — temporary +# config/grafana/alerting/zz-delete.yaml (temporary) apiVersion: 1 deleteRules: - orgId: 1 uid: proj-silent-- ``` -Restart Grafana, confirm the group is gone, then remove `zz-delete.yaml` and restart -again. It has to be temporary: left in place it would delete the rule again the next -time `bootstrap.sh` renders it for that project. +The file has to be temporary. Left in place, it would delete the rule again the next +time `bootstrap.sh` renders it for that project. So: -Finally re-run `bootstrap.sh` for a project that remains, so `coverage.yaml` stops -listing the removed one as covered. +1. Restart Grafana. +2. Confirm the rule group is gone. +3. Remove `zz-delete.yaml` and restart Grafana again. +4. Re-run `bootstrap.sh` for a project that remains, so `coverage.yaml` stops listing + the removed one as covered. ## Budgets to plan against diff --git a/templates/alerting/coverage.yaml.tmpl b/templates/alerting/coverage.yaml.tmpl index 73ea5e6..0b2e195 100644 --- a/templates/alerting/coverage.yaml.tmpl +++ b/templates/alerting/coverage.yaml.tmpl @@ -1,10 +1,10 @@ -# GENERATED by bootstrap.sh from templates/alerting/coverage.yaml.tmpl — do not edit -# the rendered file. Regenerated on every run. +# GENERATED by bootstrap.sh from templates/alerting/coverage.yaml.tmpl. Do not edit the +# rendered file; it is regenerated on every run. # -# The trap this closes: ProjectTelemetrySilent only exists for projects bootstrap.sh was +# The gap this closes: ProjectTelemetrySilent only exists for projects bootstrap.sh was # actually run for. A project that ships telemetry but skipped bootstrap is silently -# uncovered — the exact failure class the keystone closes, one level up. This fires on -# any series whose project/env pair has no rendered rule file. +# uncovered, one level up from the failure the keystone rule closes. This fires on any +# series whose project/env pair has no rendered rule file. # # The selector touches every series every project sends (~3k for one project). Cheap at # this size, evaluated instantly every 5m; revisit if a project's active series reach diff --git a/templates/alerting/project.yaml.tmpl b/templates/alerting/project.yaml.tmpl index f8e7ae9..fb23bb2 100644 --- a/templates/alerting/project.yaml.tmpl +++ b/templates/alerting/project.yaml.tmpl @@ -1,12 +1,12 @@ -# Rendered by bootstrap.sh into config/grafana/alerting/, flat — Grafana's alerting -# provisioner does not recurse. Do not edit the rendered files by hand: re-run +# Rendered by bootstrap.sh into config/grafana/alerting/, flat: Grafana's alerting +# provisioner does not recurse. Do not edit the rendered files by hand. Re-run # bootstrap.sh, or edit this template so every project gets the fix. # -# This is the keystone: nothing on a spoke can detect its own absence. A project that -# ships telemetry but never had bootstrap.sh run is silently uncovered — that is what -# the ProjectsUncovered rule in coverage.yaml exists to catch, one level up. +# This is the keystone: nothing on a project host can detect its own absence. A project +# that ships telemetry but never had bootstrap.sh run is silently uncovered, which is +# what the ProjectsUncovered rule in coverage.yaml catches one level up. # -# The marker below is read back by bootstrap.sh to regenerate that coverage rule. It is +# bootstrap.sh reads the marker below back to regenerate that coverage rule. It carries # the pair, not the filename, because both halves may contain a dash: `project-a-b-c` # cannot be split back into the project and env that produced it. # COVERS: __PROJECT__ __ENV__ @@ -19,8 +19,8 @@ groups: folder: Stack alerts interval: 1m rules: - # A label-only selector, deliberately not target_info: agents attach the identity - # labels to the series themselves, so target_info carries none of them. absent() + # A label-only selector, not target_info: agents attach the identity labels to + # the series themselves, so target_info carries none of them. absent() # yields nothing while telemetry flows, so NoData is the HEALTHY state and must # map to OK or the rule reads inverted. - uid: __UID__ diff --git a/templates/alloy/config.alloy b/templates/alloy/config.alloy index c4073f3..94eb62f 100644 --- a/templates/alloy/config.alloy +++ b/templates/alloy/config.alloy @@ -1,36 +1,36 @@ -// SHARED AGENT CONFIG — vendored from the central monitoring repo at a pinned tag. +// SHARED AGENT CONFIG, vendored from the CML monitoring repository at a pinned tag. // Do not edit it per project: every deployment-specific value arrives as an environment // variable (PROJECT, ENVIRONMENT, COMPOSE_PROJECT_NAME, OTEL_EXPORTER_OTLP_ENDPOINT, -// OTLP_AUTH_TOKEN, TELEMETRY_EDGE_KEY). Needing to edit this file to onboard a project -// is a bug in the template — report it upstream. +// OTLP_AUTH_TOKEN, TELEMETRY_EDGE_KEY). If you need to edit this file to onboard a +// project, that is a bug in the template. Report it upstream. // -// It ships this host's telemetry to the department OpenTelemetry collector: container -// stdout/stderr as logs, and host resource metrics. +// It ships this host's telemetry to the shared OpenTelemetry collector run by the +// central monitoring stack: container stdout/stderr as logs, and host resource metrics. // // An instrumented application sends its own traces and metrics over OTLP directly. -// Everything else — databases, caches, proxies, frontends, batch jobs — only writes to +// Everything else (databases, caches, proxies, frontends, batch jobs) only writes to // stdout, so this agent picks those up and forwards them on the same OTLP endpoint with // the same token. Nothing new is published at the edge: the monitoring stack exposes // `otel.` and does not expose Loki, which has no authentication of its own. -// The host's own name, read from its /etc/hostname through the rootfs mount rather than -// taken as an operator input: one less thing to set per host. The container's own -// hostname is a docker id and useless here. +// The host's own name, read from its /etc/hostname through the rootfs mount: one less +// thing to set per host. The container's own hostname is a docker id, useless as an +// identity. local.file "hostname" { filename = "/rootfs/etc/hostname" } -// Discovery and log reading both talk to the Docker API — through the socket proxy -// (compose.telemetry.yml), never the socket itself, so this container cannot -// mutate the daemon. The API is what makes container NAMES available; reading the -// on-disk json logs instead would label everything by container id. +// Discovery and log reading both talk to the Docker API through the socket proxy +// (compose.telemetry.yml), never the socket itself, so this container cannot mutate the +// daemon. The API is what makes container NAMES available; reading the on-disk json logs +// instead would label everything by container id. discovery.docker "containers" { host = "tcp://docker-socket-proxy:2375" - // The default is 1m and the backup one-shot runs for 60-90s, so a whole backup can - // start and exit inside one poll window and never be discovered — losing exactly the - // logs most worth having. Polling the local socket for a dozen containers is cheap; - // a missed short-lived container is not. + // The default is 1m and a backup one-shot runs for 60-90s, so a whole backup can + // start and exit inside one poll window and never be discovered, losing the logs + // most worth having. Polling the local socket for a dozen containers is cheap; a + // missed short-lived container is not. refresh_interval = "5s" } @@ -47,8 +47,8 @@ discovery.relabel "containers" { // Never tail this pipeline's own containers. When the export path is broken, every // failed batch writes an error line; tailing that line queues it, it fails, and - // writes another — an amplifying self-feed with no exit. The json-file driver note - // in the overlay only prevents the log-DRIVER loop; this prevents the discovery one. + // writes another: an amplifying self-feed with no exit. The json-file driver note in + // the overlay only prevents the log-DRIVER loop; this prevents the discovery one. rule { source_labels = ["__meta_docker_container_label_com_docker_compose_service"] regex = "alloy|docker-socket-proxy" @@ -56,7 +56,7 @@ discovery.relabel "containers" { } // The Compose service name is the stable per-deployable identifier the monitoring - // stack's conventions ask for: `api`, never `api-prod-2`. + // stack keys on: `api`, never `api-prod-2`. rule { source_labels = ["__meta_docker_container_label_com_docker_compose_service"] target_label = "service_name" @@ -86,9 +86,9 @@ loki.source.docker "containers" { forward_to = [otelcol.receiver.loki.containers.receiver] } -// Loki-shaped entries in, OpenTelemetry logs out. This is the only reason Alloy is -// here rather than a plain OTel collector: loki.source.docker is the least fiddly way -// to read Docker's log stream with real container labels attached. +// Loki-shaped entries in, OpenTelemetry logs out. This is the only reason the agent is +// Alloy and not a plain OTel collector: loki.source.docker reads Docker's log stream +// with real container labels attached. otelcol.receiver.loki "containers" { output { logs = [otelcol.processor.transform.resource_attributes.input] @@ -96,9 +96,9 @@ otelcol.receiver.loki "containers" { } // Promote the identifiers the monitoring stack keys dashboards on from log-record -// attributes to RESOURCE attributes. error_mode = "ignore" is deliberate: if the -// conversion ever names these attributes differently, logs still arrive with the -// attributes unpromoted, rather than the pipeline dropping them. +// attributes to RESOURCE attributes. Keep error_mode = "ignore": if the conversion ever +// names these attributes differently, logs still arrive unpromoted instead of being +// dropped. otelcol.processor.transform "resource_attributes" { error_mode = "ignore" @@ -119,15 +119,15 @@ otelcol.processor.transform "resource_attributes" { } // --------------------------------------------------------------------------- -// Host metrics. node_exporter's collectors, scraped in-process and converted to OTLP -// so they ride the same endpoint and token as the logs above — no second pipeline, no -// second hostname, no second credential. +// Host metrics. node_exporter's collectors, scraped in-process and converted to OTLP so +// they ride the same endpoint and token as the logs above: no second pipeline, no second +// hostname, no second credential. // -// set_collectors is an allowlist rather than the default set: fewer series, each one here -// for a reason. `hwmon` earns a dashboard panel — thermal data on a physical box is cheap -// and occasionally telling — but it is NOT a control: fan RPM is a symptom of load, -// legitimate load spins fans too, and no threshold on it would be trustworthy. A crash -// loop is detected by container restart count, not by temperature. +// set_collectors is an allowlist, not the default set, which keeps the series count down. +// `hwmon` feeds a dashboard panel, since thermal data on a physical box is cheap and +// occasionally telling, but it is NOT a control: fan RPM is a symptom of load, legitimate +// load spins fans too, and no threshold on it would be trustworthy. A crash loop is +// detected by container restart count, not by temperature. prometheus.exporter.unix "host" { // The exporter runs inside the container, so point it at the host's mounted /proc, // /sys and /. @@ -156,7 +156,7 @@ prometheus.exporter.unix "host" { // --------------------------------------------------------------------------- // Per-container resource metrics. This is what makes a crash loop visible: cAdvisor has // no restart counter, but container_start_time_seconds moving repeatedly is the same -// signal, and container_oom_events_total says why. The central stack alerts on +// signal, and container_oom_events_total says why. The monitoring stack alerts on // // changes(container_start_time_seconds{name!=""}[1h]) > 3 // @@ -188,8 +188,8 @@ prometheus.scrape "cadvisor" { } // cAdvisor walks the host's cgroup tree, so it sees EVERY container on the box. The -// COMPOSE_PROJECT_NAME filter on discovery.docker above scopes log tailing only — it -// does not reach this exporter. Without this rule, containers from a dev stack, an +// COMPOSE_PROJECT_NAME filter on discovery.docker above scopes log tailing only; it does +// not reach this exporter. Without this rule, containers from a dev stack, an // unrelated tool, or the monitoring stack itself arrive stamped with THIS deployment's // project and env, firing this project's alerts for something it does not own and // double-reporting anything that ships its own telemetry. So keep only this project's @@ -245,7 +245,7 @@ prometheus.relabel "host" { // drops (otelcol_exporter_send_failed_*); unshipped, those counters live on a loopback // port nobody reads and an outage's log loss goes uncounted. Riding the same pipeline is // fine: when the pipeline is down these metrics go dark too, and that absence is what the -// central ProjectTelemetrySilent rule fires on. +// ProjectTelemetrySilent rule on the monitoring host fires on. prometheus.exporter.self "alloy" { } prometheus.scrape "alloy" { @@ -258,10 +258,10 @@ prometheus.scrape "alloy" { // --------------------------------------------------------------------------- // GPU metrics, present only on hosts that run compose.telemetry.gpu.yml. With the // exporter container absent this finds no targets and costs nothing, so the same config -// file is correct on every host — a GPU host is one extra overlay, not a new design. +// file is correct on every host: a GPU host is one extra overlay, not a new design. // // Alloy has no native GPU component, and OTel Collector contrib has no NVIDIA receiver, -// so an exporter container scraped over the Compose network is the pattern. Still no new +// so the pattern is an exporter container scraped over the Compose network. Still no new // agent, no new endpoint and no new credential. discovery.relabel "gpu_exporter" { targets = discovery.docker.containers.targets @@ -302,10 +302,10 @@ otelcol.receiver.prometheus "host" { } } -// Refuse work before the cgroup does it for us: without this, a long hub outage -// grows the sending queue until the OOM killer takes the agent AND the self-metrics -// that would have counted the loss. Mirrors the central collector; sized under the -// overlay's 512m mem_limit — keep the two in step. +// Refuse work before the cgroup does it for us. Without this, a long outage of the +// central monitoring stack grows the sending queue until the OOM killer takes the agent +// AND the self-metrics that would have counted the loss. Sized under the overlay's 512m +// mem_limit; keep the two in step. otelcol.processor.memory_limiter "default" { check_interval = "1s" limit = "400MiB" @@ -327,9 +327,9 @@ otelcol.processor.batch "default" { otelcol.exporter.otlphttp "central" { client { // The tunnel routes HTTPS to the collector's HTTP receiver only; there is no - // public gRPC path, which is why this is otlphttp and not otlp. The endpoint - // MUST be https:// — an http:// value ships the bearer token and every log - // line in cleartext, and nothing here can tell (see .env.example). + // public gRPC path, so this is otlphttp and not otlp. The endpoint MUST be + // https://. An http:// value ships the bearer token and every log line in + // cleartext, and nothing here can tell (see .env.example). endpoint = sys.env("OTEL_EXPORTER_OTLP_ENDPOINT") headers = { @@ -345,10 +345,10 @@ otelcol.exporter.otlphttp "central" { } } - // Explicit rather than collector defaults, so an outage's behaviour is a decision: - // retry for 15 minutes, hold up to 2000 batches in memory, then drop and COUNT the - // drop (shipped by prometheus.exporter.self above). The queue is the reason the - // overlay caps this container's memory — see mem_limit there before raising it. + // Set here rather than left to the collector defaults: retry for 15 minutes, hold up + // to 2000 batches in memory, then drop and COUNT the drop (shipped by + // prometheus.exporter.self above). This queue is why the overlay caps this + // container's memory; see mem_limit there before raising queue_size. sending_queue { queue_size = 2000 } diff --git a/templates/compose.telemetry.gpu.yml b/templates/compose.telemetry.gpu.yml index 695177e..5e55a86 100644 --- a/templates/compose.telemetry.gpu.yml +++ b/templates/compose.telemetry.gpu.yml @@ -1,6 +1,6 @@ -# compose.telemetry.gpu.yml — VENDORED from the central monitoring repo at a pinned tag. +# compose.telemetry.gpu.yml, VENDORED from the CML monitoring repository at a pinned tag. # -# OPTIONAL OVERLAY — GPU metrics on a host with NVIDIA hardware. +# OPTIONAL OVERLAY: GPU metrics on a host with NVIDIA hardware. # # Adds a GPU exporter that the Alloy agent discovers and scrapes automatically. Include # it only on hosts that have a card: @@ -9,17 +9,16 @@ # # or set GPU_METRICS=1 in the host's root `.env` and let the deploy recipes include it. # -# Why nvidia_gpu_exporter and not dcgm-exporter, which is the reflex answer: DCGM's -# advantage is its DCGM_FI_PROF_* profiling fields, and NVIDIA document those as -# datacentre-only hardware features. On a GeForce card they are simply absent, which -# removes the entire reason to prefer DCGM while keeping its SYS_ADMIN requirement and -# its 2021-vintage dashboard. Consumer cards also have no framebuffer ECC, so the ECC -# fields are empty too. What remains is the NVML field set, which is exactly what this -# exporter reads. +# Why nvidia_gpu_exporter and not dcgm-exporter: DCGM's advantage is its DCGM_FI_PROF_* +# profiling fields, which NVIDIA document as datacentre-only hardware features. On a +# GeForce card they are absent, which removes the reason to prefer DCGM while keeping its +# SYS_ADMIN requirement and its 2021-vintage dashboard. Consumer cards also have no +# framebuffer ECC, so the ECC fields are empty too. What remains is the NVML field set, +# which is what this exporter reads. # -# If the department ever buys datacentre silicon (A100/H100/L40S), profiling, ECC, NVLink and MIG -# all light up and dcgm-exporter becomes strictly better. That is a one-container swap -# plus one dashboard — do not pre-build for it. +# On datacentre silicon (A100/H100/L40S), profiling, ECC, NVLink and MIG all light up and +# dcgm-exporter becomes the better choice. That is a one-container swap plus one +# dashboard; do not pre-build for it. services: nvidia-gpu-exporter: diff --git a/templates/compose.telemetry.yml b/templates/compose.telemetry.yml index d1797ff..73f4306 100644 --- a/templates/compose.telemetry.yml +++ b/templates/compose.telemetry.yml @@ -1,4 +1,4 @@ -# compose.telemetry.yml — VENDORED from the central monitoring repo at a pinned tag. +# compose.telemetry.yml, VENDORED from the CML monitoring repository at a pinned tag. # # One Grafana Alloy agent per deploy host. Vendor it as-is; everything that differs # between projects arrives as an environment variable. See templates/README.md. @@ -8,12 +8,12 @@ # forwards them to the same OTLP endpoint with the same token: # # - the stdout of every OTHER container in the stack, as logs; -# - host resource metrics — CPU, memory, load, disk, network, hwmon; +# - host resource metrics: CPU, memory, load, disk, network, hwmon; # - per-container resources and lifecycle from cAdvisor, which is what makes a crash -# loop visible centrally. +# loop visible on the central monitoring host. # # The root deploy recipes auto-include this overlay when OTEL_EXPORTER_OTLP_ENDPOINT is -# set in the host's root `.env` — the same switch that turns on the API's own exporter. +# set in the host's root `.env`, the same switch that turns on the API's own exporter. # Hosts without it ship nothing and keep `docker logs` as the only log path. # Local json-file logging with rotation for the agent stack itself. @@ -25,9 +25,9 @@ x-logging: &default-logging services: # Least-privilege gate between Alloy and the Docker API. Alloy never sees the socket: - # it talks HTTP to this proxy, which allows exactly the read-only endpoints discovery, + # it talks HTTP to this proxy, which allows only the read-only endpoints discovery, # log tailing and cAdvisor need (containers, events, info, version, ping) and refuses - # everything else — POST is off, so nothing behind it can start or exec a container. + # everything else. POST is off, so nothing behind it can start or exec a container. # The proxy itself holds the socket, so IT is the root-equivalent component; it runs # nothing but haproxy with a static allowlist, joins only the internal socket network, # and is the single place to audit. @@ -35,10 +35,10 @@ services: image: tecnativa/docker-socket-proxy:v0.5.0@sha256:1f5038b54f06c3e18422902cf00ba21803d1c97805aae032e5e6673d532d3459 restart: unless-stopped environment: - CONTAINERS: 1 # list/inspect/logs — discovery.docker, loki.source.docker + CONTAINERS: 1 # list/inspect/logs, for discovery.docker and loki.source.docker NETWORKS: 1 # discovery.docker computes network labels; 403 here kills discovery - EVENTS: 1 # container lifecycle stream — cAdvisor watches it - INFO: 1 # machine info — cAdvisor + EVENTS: 1 # container lifecycle stream, watched by cAdvisor + INFO: 1 # machine info, read by cAdvisor VERSION: 1 PING: 1 POST: 0 # no mutations, ever @@ -57,7 +57,7 @@ services: restart: unless-stopped # Root for the host mounts below (/host/proc, /host/sys, /rootfs): node_exporter's # collectors read files that are root-only on the host. The Docker API is reached - # through docker-socket-proxy above, so root here no longer implies control of the + # through docker-socket-proxy above, so root here does not imply control of the # Docker daemon. user: root command: @@ -69,30 +69,31 @@ services: - --server.http.listen-addr=127.0.0.1:12345 environment: ENVIRONMENT: ${ENVIRONMENT:?telemetry overlay requires ENVIRONMENT} - # The project identity every signal is labelled with, and what the central - # ProjectTelemetrySilent rule keys on. Must match the value bootstrap.sh was run with. + # The project identity every signal is labelled with, and what the + # ProjectTelemetrySilent rule on the monitoring host keys on. It must match the + # value the monitoring host's onboarding script (bootstrap.sh) was run with. PROJECT: ${PROJECT:?telemetry overlay requires PROJECT} # Alloy only reads containers belonging to this Compose project, so it must know # which one it is. Compose always injects COMPOSE_PROJECT_NAME into interpolation - # — the deploy directory's base name unless -p or .env overrides it — so this - # passes through whatever the label on the sibling containers actually says. A - # `:-` default here would be dead text. This is the COMPOSE project (docker's -p), - # not PROJECT above; rename the deploy directory and the filter follows it. + # (the deploy directory's base name unless -p or .env overrides it), so this passes + # through whatever the label on the sibling containers says. A `:-` default here + # would be dead text. This is the COMPOSE project (docker's -p), not PROJECT above; + # rename the deploy directory and the filter follows it. COMPOSE_PROJECT_NAME: ${COMPOSE_PROJECT_NAME} OTEL_EXPORTER_OTLP_ENDPOINT: ${OTEL_EXPORTER_OTLP_ENDPOINT:?required by the telemetry overlay} OTLP_AUTH_TOKEN: ${OTLP_AUTH_TOKEN:?telemetry overlay requires OTLP_AUTH_TOKEN} - # Second, weaker credential: matched by the Cloudflare WAF-skip rule for the - # telemetry hostname (infra/cloudflare-zone), so the bearer token itself never - # appears in a Cloudflare ruleset expression. Rotates independently of the token. - # Optional: only projects whose egress crosses a WAF need this. Empty is fine. + # Second, weaker credential: the Cloudflare WAF-skip rule for the telemetry + # hostname matches this header, so the bearer token itself never appears in a + # Cloudflare ruleset expression. It rotates independently of the token. Optional: + # only projects whose egress crosses a WAF need it. Empty is fine. TELEMETRY_EDGE_KEY: ${TELEMETRY_EDGE_KEY:-} volumes: - ./deploy/alloy/config.alloy:/etc/alloy/config.alloy:ro # vendored, never edited # node_exporter's collectors read the host's own /proc, /sys and filesystems. - # All read-only. Note this does give the container read access to the host - # filesystem via /rootfs, which is the price of host disk-usage metrics from a - # container; it is no more than a host-installed node_exporter running as root - # already has, but it is worth knowing rather than discovering. + # All read-only. This does give the container read access to the whole host + # filesystem through /rootfs, the price of host disk-usage metrics from a + # container. It is no more than a host-installed node_exporter running as root + # already has, but know it rather than discover it. - /proc:/host/proc:ro - /sys:/host/sys:ro - /:/rootfs:ro @@ -103,14 +104,14 @@ services: # the containers still have on disk. - alloy_data:/var/lib/alloy/data # cAdvisor finds containers by walking the cgroup hierarchy, NOT through the Docker - # API — the socket proxy only supplies names and metadata for cgroups it has already + # API; the socket proxy only supplies names and metadata for cgroups it has already # found. Under Docker's default private cgroup namespace this container sees just its # own cgroup, as "/", so every container_* metric collapses to a single root series - # with no `name` label: the central ContainerRestarting rule matches zero series and + # with no `name` label: the ContainerRestarting rule matches zero series and # ContainerOOMKilled cannot say which container died. Joining the host's namespace is # the whole fix; mounting /sys/fs/cgroup as well is redundant, verified on cgroup v2. - # It grants read access to the host's cgroup tree, which is strictly less than the - # /rootfs and /var/lib/docker mounts above already give. + # It grants read access to the host's cgroup tree, which is less than the /rootfs and + # /var/lib/docker mounts above already give. cgroup: host # cAdvisor reads OOM-kill events from the kernel log; without the device AND # CAP_SYSLOG (kernel.dmesg_restrict gates /dev/kmsg reads) container_oom_events_total @@ -118,11 +119,11 @@ services: # not configure a source for OOM detection" in the agent log is this exact gap). devices: - /dev/kmsg:/dev/kmsg - # Root, but not root-with-everything: the default capability set is dropped like the - # other hardened services here (socket proxy above, backup). DAC_READ_SEARCH is what - # lets the collectors read root-only files under /host/proc, /host/sys and /rootfs; - # unlike the default set's DAC_OVERRIDE it grants no write-side permission bypass, - # and the rest (SETUID, MKNOD, NET_RAW, ...) is nothing a metrics agent needs. + # Root, but not root-with-everything: the default capability set is dropped, as on + # the socket proxy above. DAC_READ_SEARCH is what lets the collectors read root-only + # files under /host/proc, /host/sys and /rootfs; unlike the default set's + # DAC_OVERRIDE it grants no write-side permission bypass. The rest (SETUID, MKNOD, + # NET_RAW, ...) is nothing a metrics agent needs. cap_drop: - ALL cap_add: @@ -130,10 +131,10 @@ services: - DAC_READ_SEARCH security_opt: - no-new-privileges:true - # Bounded like every other service on the host: during a collector outage the - # export queue grows in memory, and without a ceiling the OOM killer picks by RSS — - # the plausible victim being postgres or the API, i.e. the agent taking down the - # thing it observes. Keep in step with the sending_queue size in config.alloy. + # Bounded like every other service on the host: during a collector outage the export + # queue grows in memory, and without a ceiling the OOM killer picks by RSS. The + # plausible victim is then postgres or the API, so the agent takes down the thing it + # observes. Keep in step with the sending_queue size in config.alloy. mem_limit: 512m pids_limit: 512 networks: @@ -147,9 +148,10 @@ volumes: alloy_data: networks: - # Alloy's way out to the hub, and where the GPU exporter overlay attaches. Declared - # here so the overlay stands on its own; a project that already defines `egress` in - # its compose.yml keeps its definition, Compose merges the two by name. + # Alloy's way out to the central collector, and where the GPU exporter overlay + # attaches. Declared here so the overlay stands on its own; a project that already + # defines `egress` in its compose.yml keeps its definition, Compose merges the two by + # name. egress: # Carries only the proxied Docker API between the proxy and Alloy. Internal: it has # no gateway, so nothing on it can reach out and nothing outside can reach in. diff --git a/templates/run_scheduled.sh b/templates/run_scheduled.sh index bdc41e3..6066e4d 100755 --- a/templates/run_scheduled.sh +++ b/templates/run_scheduled.sh @@ -1,27 +1,26 @@ #!/usr/bin/env bash -# VENDORED from the central monitoring repo at a pinned tag. Generic: nothing here +# VENDORED from the CML monitoring repository at a pinned tag. Generic: nothing here # names a project or a job list. # -# Run one scheduled job and report the result to a dead-man's switch. +# Run one scheduled job and report the result to a dead-man's switch (healthchecks.io). # # Usage: run_scheduled.sh -# job — any name; it selects the ping URL variable and the command below. -# env — the deployment environment, passed through to the command. +# job: any name; it selects the ping URL variable and the command below. +# env: the deployment environment, passed through to the command. # -# Called by this host's systemd units. This is the ONE piece of monitoring -# that does not share fate with the observability stack, and that is its whole reason to -# exist. Everything else — logs, traces, host metrics — flows through Alloy to the -# department collector, so a dead host, a dead collector, a broken tunnel or an expired -# token all look identical from Grafana: silence. A push from the host to an external -# endpoint is the one signal that still arrives when that pipeline is the thing that -# broke, and its absence is itself the alarm. +# Called by this host's systemd units. This is the ONE piece of monitoring that does not +# share fate with the observability stack. Everything else (logs, traces, host metrics) +# flows through the Alloy agent to the shared OpenTelemetry collector, so a dead host, a +# dead collector, a broken tunnel and an expired token all look identical from Grafana: +# silence. A push from this host to an external endpoint still arrives when that pipeline +# is the thing that broke, and its absence is itself the alarm. # -# The ping lives here rather than in each unit so the reporting is written once. +# The ping lives here, not in each systemd unit, so the reporting is written once. # -# Ping URLs come from the host file loaded by the units, never from the repository — -# they are capability URLs. bootstrap.sh on the central host prints the PING_* block. An unset URL disables the ping -# for that job without failing it, so a host that has not been wired up yet still runs -# its jobs. +# Ping URLs come from the host file the units load, never from the repository: they are +# capability URLs. The monitoring host's onboarding script (bootstrap.sh) prints the +# PING_* block. An unset URL disables the ping for that job without failing it, so a host +# that has not been wired up yet still runs its jobs. set -uo pipefail job="${1:-}" @@ -32,8 +31,8 @@ if [[ -z "$job" || -z "$env_name" ]]; then exit 2 fi # Job and env become part of an environment-variable name below, so keep them to -# characters that can appear in one — a stray character would silently look up the -# wrong variable and disable the ping rather than fail. +# characters that can appear in one. A stray character would look up the wrong variable +# and disable the ping instead of failing. if [[ ! "$job" =~ ^[A-Za-z0-9_-]+$ || ! "$env_name" =~ ^[A-Za-z0-9_-]+$ ]]; then echo "error: job and env must match [A-Za-z0-9_-]+" >&2 exit 2 @@ -45,13 +44,13 @@ cd "$ROOT_DIR" || exit 1 # `just` is resolved at unit-render time and passed in; fall back to PATH for manual runs. JUST_BIN="${JUST_BIN:-just}" -# One `just` recipe per job, named the same as the job — that identity is the contract, -# which is why there is no mapping table here. Projects that do not use `just` can point -# JUST_BIN at any runner with the same shape. +# One `just` recipe per job, named the same as the job. That identity is the contract, so +# there is no mapping table here. Projects that do not use `just` can point JUST_BIN at +# any runner with the same shape. command=("$JUST_BIN" "$job" "$env_name") # One variable per job, so each gets its own check. Sharing a URL would let a frequent -# job's pings mask a rare one's silence — exactly the failure the rare job exists to catch. +# job's pings mask a rare one's silence, the failure the rare job exists to catch. url_var="PING_${job//-/_}" url_var="${url_var^^}" ping_url="${!url_var:-}" @@ -59,20 +58,19 @@ ping_url="${!url_var:-}" output_file="$(mktemp)" trap 'rm -f "$output_file"' EXIT -# Send the job's own output as the failure body: the alert then carries the reason, -# instead of only saying that something went wrong. Note this puts job output — -# hostnames, paths, restic summaries — in a third party's hands; it is why the ping -# carries no credentials and why the URL itself is the only secret. +# Send the job's own output as the failure body, so the alert carries the reason. Note +# that this puts job output (hostnames, paths, backup summaries) in a third party's +# hands. The ping therefore carries no credentials, and the URL itself is the only secret. ping_fail() { curl -fsS -m 10 --retry 3 --data-binary "@${output_file}" "${ping_url}/fail" -o /dev/null \ || echo "WARNING: failure ping to ${url_var} failed" >&2 } -# A killed job must still report. systemd's TimeoutStartSec TERMs the whole cgroup: -# the job dies, and without this trap bash would die too — before the ping block — -# so a HUNG job would send neither success nor failure and its captured output would -# be lost. The job runs in the background so `wait` can be interrupted by the signal; -# the child has already received the same TERM from systemd (KillMode=control-group). +# A killed job must still report. systemd's TimeoutStartSec TERMs the whole cgroup: the +# job dies, and without this trap bash would die too, before the ping block, so a HUNG job +# would send neither success nor failure and its captured output would be lost. The job +# runs in the background so `wait` can be interrupted by the signal; the child has already +# received the same TERM from systemd (KillMode=control-group). # shellcheck disable=SC2329 # invoked via the TERM/INT traps below on_terminate() { local sig="$1" From 5e62b552ec772b0190a05203f65dab71cf031213 Mon Sep 17 00:00:00 2001 From: Simon van Lierde Date: Sun, 6 Sep 2026 15:14:20 +0000 Subject: [PATCH 47/85] fix(lint): update ruff configuration to remove ignored rules and set pydocstyle convention to google --- demo/pyproject.toml | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-) diff --git a/demo/pyproject.toml b/demo/pyproject.toml index 75aed8f..f6a8ff5 100644 --- a/demo/pyproject.toml +++ b/demo/pyproject.toml @@ -17,9 +17,5 @@ target-version = "py313" [tool.ruff.lint] - ignore = [ - "D203", # line too long - "E203", # whitespace before ':' - "W503", # line break before binary operator - ] - select = ["ALL"] + pydocstyle.convention = "google" + select = ["ALL"] From a29d25baf98239176f81839a74476c1a3a96f6d3 Mon Sep 17 00:00:00 2001 From: Simon van Lierde Date: Sun, 6 Sep 2026 15:14:24 +0000 Subject: [PATCH 48/85] style(logging): format logging configuration for better readability --- demo/app.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/demo/app.py b/demo/app.py index ad0db1d..802908d 100644 --- a/demo/app.py +++ b/demo/app.py @@ -12,7 +12,9 @@ ERROR_RATE = 0.1 # fixed error rate, enough to light up RED panels -logging.basicConfig(level=logging.INFO) # root logger defaults to WARNING; we want the INFO lines too +logging.basicConfig( + level=logging.INFO, +) # root logger defaults to WARNING; we want the INFO lines too log = logging.getLogger("demo-api") app = FastAPI() From 2d7e3de636ba7e8bd6f226e1dba64c3698084f4a Mon Sep 17 00:00:00 2001 From: Simon van Lierde Date: Sun, 6 Sep 2026 16:19:14 +0000 Subject: [PATCH 49/85] fix(infra): treat a failed DNS API call as an error, not an absent record --- infra/generate-imports.sh | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/infra/generate-imports.sh b/infra/generate-imports.sh index 7327de0..8277d06 100755 --- a/infra/generate-imports.sh +++ b/infra/generate-imports.sh @@ -70,7 +70,12 @@ lookup_dns_record() { # Type-filtered: this root manages CNAMEs, and a name can also carry TXT records. # An unfiltered .result[0] could bind one of those, and the apply would rewrite it # into a proxied CNAME, destroying the TXT record and leaving the CNAME unmanaged. - id="$(api "zones/$zone/dns_records?name=$hostname&type=CNAME" | jq -r '.result[0].id // empty')" + # Two statements, not one pipeline: this function only runs inside `if`/`||`, where + # errexit is off, so a failed API call would otherwise fall through and be reported + # as "no record", the one answer that sends the operator off to create a duplicate. + local body + body="$(api "zones/$zone/dns_records?name=$hostname&type=CNAME")" || die "DNS lookup for $hostname failed (token lacks DNS:Read?)" + id="$(jq -r '.result[0].id // empty' <<<"$body")" [[ -n "$id" ]] || return 1 printf '%s' "$id" } From 796c2de90572f392e6f3189bbe601b0e31669edd Mon Sep 17 00:00:00 2001 From: Simon van Lierde Date: Sun, 6 Sep 2026 16:19:14 +0000 Subject: [PATCH 50/85] ci: split lint from validate, assert the data paths in smoke, add git hooks - `just check` is now `lint` (static, small tool images pulled in parallel) plus `validate` (the stack's own images, looked up per service); CI runs lint on its own job and validate next to smoke, which pulls those images anyway - smoke boots with JWT auth on and its assertions move to scripts/smoke.sh: dashboards and alert rules compared uid for uid with no rule paused, contact points checked against the exact URLs given, JWT settings read back and a forged Access token refused, every scrape target up, and a metric and a log round-tripped through the collector into Prometheus and Loki with the promoted project/env/department labels - `just restore-check` round-trips backup and restore on the smoke volumes; `_backup`/`_restore` take the compose project and derive the volume mounts, so `_project-guard` goes; backup fails when the unpause fails and compresses with gzip -1 to shorten the pause - lint renders the alert templates through bootstrap.sh and yamllints the result, checks dashboard datasource uids against datasources.yaml, enforces yamlfmt and ruff formatting, pins compose.yml on the first config line so a tunnel host's COMPOSE_FILE cannot change what it validates, and no longer loses the status of the exposure-guard self-test's positive half - bootstrap.sh: hashes computed before the heredoc, every vendored template checked in the tag, the rendered rule read back from Grafana after the restart, and BOOTSTRAP_OUT_DIR for the lint render - prek hooks via `just hooks`: gitleaks on the staged diff and lint at commit, a Conventional Commits check on the message, validate at push, infra-validate at push when infra/ changed - `just fmt` runs yamlfmt in a container; `just demo-build` removed; healthchecks probe every 2s while starting; smoke-down stops with -t 1 --- .github/workflows/ci.yml | 18 ++-- .pre-commit-config.yaml | 51 +++++++++ CHANGELOG.md | 29 ++++- README.md | 35 ++++-- bootstrap.sh | 67 ++++++++---- compose.sandbox.yml | 6 +- compose.yml | 6 ++ demo/pyproject.toml | 2 + docs/RUNBOOK.md | 2 +- justfile | 228 ++++++++++++++++++++++----------------- scripts/smoke.sh | 101 +++++++++++++++++ 11 files changed, 404 insertions(+), 141 deletions(-) create mode 100644 .pre-commit-config.yaml create mode 100755 scripts/smoke.sh diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 68c0281..a463532 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -1,5 +1,7 @@ # Thin wrapper around the `just` contract: everything CI runs, you can run -# locally with the same command. +# locally with the same command. `just check` locally is `lint` + `validate`; +# here they split so a lint failure reports in seconds without pulling the +# stack images, which the smoke job pulls anyway. name: CI on: @@ -10,27 +12,26 @@ on: permissions: contents: read -# A push to an open PR supersedes the run already in flight; both jobs boot a -# full stack, so leaving them stacked wastes a runner per push. +# A push to an open PR supersedes the run already in flight; the smoke job +# boots a full stack, so leaving them stacked wastes a runner per push. concurrency: group: ${{ github.workflow }}-${{ github.ref }} cancel-in-progress: true jobs: - check: + lint: runs-on: ubuntu-latest - timeout-minutes: 15 + timeout-minutes: 10 steps: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - # `just check` scans git history for secrets; the default shallow clone + # `just lint` scans git history for secrets; the default shallow clone # would give it one commit and pass vacuously. with: fetch-depth: 0 - uses: extractions/setup-just@53165ef7e734c5c07cb06b3c8e7b647c5aa16db3 # v4.0.0 - run: cp .env.example .env - - run: just check + - run: just lint - run: just infra-validate - - run: just demo-build smoke: runs-on: ubuntu-latest @@ -39,6 +40,7 @@ jobs: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - uses: extractions/setup-just@53165ef7e734c5c07cb06b3c8e7b647c5aa16db3 # v4.0.0 - run: cp .env.example .env + - run: just validate - run: just smoke # Every smoke failure message points at the logs, which are only useful if # they outlive the job. diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml new file mode 100644 index 0000000..eb124cb --- /dev/null +++ b/.pre-commit-config.yaml @@ -0,0 +1,51 @@ +# Installed by `just hooks` (prek). Stages sized to their cost: +# pre-commit gitleaks on the staged diff (~0.5s; a secret is cheapest to +# catch before it is in history), then `just lint` (~6s warm). +# commit-msg Conventional Commits shape on the first line. +# pre-push `just validate` (the stack's images; seconds on a host that +# runs the stack) and `just infra-validate`, only when infra/ +# is in the push: it downloads the provider every time. +# `prek run` runs the pre-commit stage on demand; `--hook-stage pre-push` the rest. +repos: + - repo: local + hooks: + - id: gitleaks + name: gitleaks (staged) + entry: >- + docker run --rm --network none -v .:/repo + zricethezav/gitleaks:v8.30.1@sha256:c00b6bd0aeb3071cbcb79009cb16a60dd9e0a7c60e2be9ab65d25e6bc8abbb7f + git --pre-commit --staged --redact --no-banner /repo + language: system + pass_filenames: false + always_run: true + stages: [pre-commit] + - id: lint + name: just lint + entry: just lint + language: system + pass_filenames: false + always_run: true + stages: [pre-commit] + - id: commit-msg + name: conventional commit message + entry: >- + sh -c 'head -n1 "$1" + | grep -Eq "^((build|chore|ci|docs|feat|fix|perf|refactor|revert|style|test)(\([a-z0-9._/-]+\))?!?: + [a-z].{0,70}|(Merge|Revert|fixup!|squash!) .*)$" + || { echo "commit message must be type(scope): lower-case summary, at most 72 chars" >&2; exit 1; }' -- + language: system + stages: [commit-msg] + - id: validate + name: just validate + entry: just validate + language: system + pass_filenames: false + always_run: true + stages: [pre-push] + - id: infra-validate + name: just infra-validate + entry: just infra-validate + language: system + pass_filenames: false + files: ^infra/ + stages: [pre-push] diff --git a/CHANGELOG.md b/CHANGELOG.md index 0cef9fc..6072a5f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -35,9 +35,24 @@ queries that had been measuring the wrong thing. argv. - **Wider validation**: `just check` also verifies the Loki and Tempo configs, the vendored Alloy config, shell scripts, the demo app's Python, OpenTofu - formatting, and git history for secrets; `just smoke` asserts every - dashboard and alert rule provisioned; CI also runs `just infra-validate` - and a new `just demo-build`. + formatting, the datasource uids dashboards reference, the alert templates + `bootstrap.sh` renders, and git history for secrets; CI also runs + `just infra-validate`; `lint` also enforces yamlfmt and ruff formatting. + `check` splits into `lint` (static) and `validate` (the stack's own images); + CI runs the two on separate jobs so a lint failure never waits on the stack + images. `just hooks` installs git hooks via prek: gitleaks on the staged + diff and `lint` at commit, a Conventional Commits check on the message, + `validate` at push, `infra-validate` at push when `infra/` changed. +- **`just smoke` boots the production shape and asserts the data paths**: JWT + auth on with a forged Access token refused, dashboards and alert rules + compared uid for uid (not by count) with no rule paused, contact points + checked against the exact URLs given, every scrape target up, and a metric + and a log round-tripped through the collector into Prometheus and Loki with + the promoted `project`/`env`/`department` labels. The assertions moved to + `scripts/smoke.sh` (functions, shellcheck). `just restore-check` rehearses + backup and restore on the smoke volumes. `just demo-build` is gone: the + demo is a local fixture (`just demo`), not a CI step. +- `just fmt` runs yamlfmt in a container like every other tool. - **Isolated smoke and demo**: each runs under its own compose project and Grafana port (`compose.sandbox.yml`), so neither can adopt or recreate a stack already running on the host. @@ -95,6 +110,14 @@ queries that had been measuring the wrong thing. ### Fixed +- `just backup` fails (instead of exiting 0) when the unpause after the copy + fails and the stack is left paused. `just lint`'s exposure-guard self-test + lost the status of its positive half; a guard that rejected every valid + `.env` passed CI. The smoke provisioning check could pass with an empty + dashboard list if reading the uids failed. `bootstrap.sh` printed the hash + of empty input for a template missing from the tag, and re-reads its rule + from Grafana after the restart instead of trusting the restart. + `infra/generate-imports.sh` reported a failed DNS API call as "no record". - **Spoke overlay declared no `egress` network**: `compose.telemetry.yml` joined it without defining it, so the documented `up -d` failed on any host whose own compose file did not happen to name one. `just check` now renders diff --git a/README.md b/README.md index 64f3ee3..e659c44 100644 --- a/README.md +++ b/README.md @@ -110,19 +110,31 @@ configuration. Applying it produces the `CLOUDFLARE_TUNNEL_TOKEN` the tunnel overlay needs; bootstrap steps are at the top of [infra/main.tf](infra/main.tf). -`just check` validates compose files, Prometheus config, the collector, Loki -and Tempo configs, the vendored Alloy config, YAML, workflows, shell scripts, -the demo app's Python, OpenTofu formatting, dashboard JSON, and git history -for leaked secrets. -Every validator runs in a pinned container, so nothing is installed on the -host. +`just check` is two halves. `just lint` is the static half: compose files, +the rendered alert templates, YAML, workflows, shell scripts, the demo app's +Python, OpenTofu formatting, dashboard JSON and the datasources it names, and +git history for leaked secrets. `just validate` runs the Prometheus, collector, +Loki, Tempo and Alloy configs through the exact images the stack runs. Every +validator runs in a pinned container, so nothing is installed on the host; +`just fmt` (yamlfmt) is containerized the same way, and `lint` enforces it. +`just hooks` installs the git hooks via [prek](https://github.com/j178/prek): +gitleaks on the staged diff and `just lint` at commit, a Conventional Commits +check on the message, `just validate` at push, and `just infra-validate` at +push when `infra/` changed. Grafana's alerting provisioning has no offline validator. `just smoke` covers -it: it boots the stack, waits for Grafana to report healthy, and checks that -every dashboard and every alert rule provisioned. It runs under its own compose -project on its own ports, so it cannot disturb a stack already running on the -host. `just smoke` is safe on the production box, and `just smoke-down` cleans -it up. CI runs `just check` and `just smoke` on every push and pull request. +it and everything else that only shows once the stack runs: it boots the stack +in production shape (JWT auth on) and asserts, uid for uid, that every +dashboard and alert rule provisioned; that the contact points carry the URLs +from the environment; that Grafana honours the JWT settings and refuses a +forged token; that every scrape target is up; and that one metric and one log +posted through the collector's bearer auth come back out of Prometheus and +Loki with the promoted project/env/department labels. The assertions live in +[scripts/smoke.sh](scripts/smoke.sh). It runs under its own compose project on +its own port, so it cannot disturb a stack already running on the host: +`just smoke` is safe on the production box, `just restore-check` rehearses +backup and restore on its volumes, and `just smoke-down` cleans up. CI runs +`lint` on one job and `validate` plus `smoke` on another, on every push and pull request. ## Sending telemetry from a project @@ -175,6 +187,7 @@ compose.tunnel.yml # production overlay: Cloudflare Tunnel compose.demo.yml # demo overlay: sample telemetry source compose.sandbox.yml # isolation overlay for `just demo` and `just smoke` demo/ # the demo FastAPI service +scripts/smoke.sh # what `just smoke` asserts against the booted stack config/ otel-collector.yaml # ingestion gateway (OTLP in → Loki/Tempo/Prometheus out) loki.yaml # logs diff --git a/bootstrap.sh b/bootstrap.sh index dc1970b..8c9cfef 100755 --- a/bootstrap.sh +++ b/bootstrap.sh @@ -49,16 +49,26 @@ fi # Flat, not a subdirectory: Grafana's alerting provisioner does not recurse. It skips a # nested directory with a warning, not an error, so that layout looks like it worked and # provisions nothing. The project- prefix keeps the files grouped in `ls`. -out_dir="config/grafana/alerting" - -# Pinned tag for the vendoring curl, with no fallback to a branch. A moving ref would let -# two projects vendor two different agent configs and call it the same template. -tag="$(git -C "$root" describe --tags --abbrev=0 2>/dev/null)" \ - || { echo "error: no release tag to pin the vendoring curls to; tag a release first" >&2; exit 1; } -# The tag must also contain the templates, or every curl 404s. -git -C "$root" rev-parse -q --verify "${tag}:templates/alloy/config.alloy" >/dev/null \ - || { echo "error: tag ${tag} predates templates/; tag a new release before onboarding" >&2; exit 1; } -repo_raw="https://raw.githubusercontent.com/CMLPlatform/monitoring/${tag}/templates" +# BOOTSTRAP_OUT_DIR is for `just lint`: it renders the templates into a scratch dir and +# lints the result, so a template edit that Grafana would reject fails in CI instead of +# on the next onboarding. With it set, nothing past the rendering runs. +out_dir="${BOOTSTRAP_OUT_DIR:-config/grafana/alerting}" +# The vendored templates and the paths they are printed under, used twice below. +templates="alloy/config.alloy:deploy/alloy/config.alloy compose.telemetry.yml:compose.telemetry.yml compose.telemetry.gpu.yml:compose.telemetry.gpu.yml run_scheduled.sh:scripts/run_scheduled.sh" + +if [[ -z "${BOOTSTRAP_OUT_DIR:-}" ]]; then + # Pinned tag for the vendoring curl, with no fallback to a branch. A moving ref would + # let two projects vendor two different agent configs and call it the same template. + tag="$(git -C "$root" describe --tags --abbrev=0 2>/dev/null)" \ + || { echo "error: no release tag to pin the vendoring curls to; tag a release first" >&2; exit 1; } + # The tag must also contain every template, or a curl 404s and the hash printed for + # it below is the hash of nothing. + for pair in $templates; do + git -C "$root" rev-parse -q --verify "${tag}:templates/${pair%%:*}" >/dev/null \ + || { echo "error: tag ${tag} predates templates/${pair%%:*}; tag a new release before onboarding" >&2; exit 1; } + done + repo_raw="https://raw.githubusercontent.com/CMLPlatform/monitoring/${tag}/templates" +fi # --------------------------------------------------------------- 1. the keystone rules # Every covered project/environment, read back from the COVERS marker in each rendered @@ -89,6 +99,7 @@ sed -e "s@__COVERED__@${covered}@" \ -e "s@__COVERED_EXPR__@${covered_expr}@" \ templates/alerting/coverage.yaml.tmpl > "${out_dir}/coverage.yaml" echo "rendered ${out_dir}/coverage.yaml (covering: ${covered})" +[[ -z "${BOOTSTRAP_OUT_DIR:-}" ]] || exit 0 # ------------------------------------------------------------------ 3. reload Grafana if docker compose ps --status running --services 2>/dev/null | grep -qx grafana; then @@ -97,6 +108,23 @@ if docker compose ps --status running --services 2>/dev/null | grep -qx grafana; docker compose up -d --force-recreate grafana >/dev/null 2>&1 \ && echo "reloaded grafana (restarted)" \ || echo "WARNING: could not restart grafana; run 'docker compose up -d --force-recreate grafana'" >&2 + # A restart is not proof: Grafana skips a malformed alert group with a log line and + # comes up healthy without it. Read the rule back. Same single-key read as the + # healthchecks key above, for the same reason. + if [[ -z "${GRAFANA_ADMIN_PASSWORD:-}" && -f .env ]]; then + GRAFANA_ADMIN_PASSWORD="$(sed -n 's/^GRAFANA_ADMIN_PASSWORD=//p' .env | tail -1)" + fi + uid="proj-silent-${project}-${env_name}" + for _ in $(seq 30); do + sleep 2 + if printf 'user = "admin:%s"\n' "$GRAFANA_ADMIN_PASSWORD" \ + | curl -sf -K - "http://localhost:3000/api/v1/provisioning/alert-rules/${uid}" >/dev/null; then + echo "verified rule ${uid} is provisioned" + uid="" + break + fi + done + [[ -z "$uid" ]] || { echo "error: grafana restarted but rule ${uid} is not provisioned; check 'just logs grafana' for the rejected file" >&2; exit 1; } else echo "note grafana is not running; the rules apply next time it starts" fi @@ -119,6 +147,15 @@ else echo "note HEALTHCHECKS_API_KEY unset; ${hc_note}" fi +# Computed before the heredoc: a substitution inside `cat < ". - # shellcheck disable=SC2086 - set -- $pair - printf '%s %s\n' "$(git -C "$root" show "${tag}:templates/$1" | sha256sum | cut -d' ' -f1)" "$2" - done) +${hashes} SUM chmod +x scripts/run_scheduled.sh diff --git a/compose.sandbox.yml b/compose.sandbox.yml index 185d4d6..fb0628c 100644 --- a/compose.sandbox.yml +++ b/compose.sandbox.yml @@ -7,9 +7,9 @@ # would fail to bind, and on a host where the stack is down it would quietly # answer on a port something else expects to own. # -# This overlay drops the ingestion ports entirely (nothing ships telemetry during -# a smoke run) and moves Grafana to SANDBOX_PORT, which the assertions in -# `just smoke` and `just demo` probe. +# This overlay drops the ingestion ports entirely (the demo app and the smoke +# assertions reach the collector over the project network) and moves Grafana to +# SANDBOX_PORT, which `just smoke` probes and `just demo` serves. services: otel-collector: ports: !override [] diff --git a/compose.yml b/compose.yml index 5a3d1b6..68016cd 100644 --- a/compose.yml +++ b/compose.yml @@ -113,6 +113,9 @@ services: timeout: 3s retries: 5 start_period: 30s + # Probe often while starting: `--wait` (just up/smoke) returns on the + # first pass, and both are ready well inside one 15s interval. + start_interval: 2s mem_limit: 2g pids_limit: 1024 cap_drop: [ALL] @@ -194,6 +197,9 @@ services: timeout: 3s retries: 5 start_period: 30s + # Probe often while starting: `--wait` (just up/smoke) returns on the + # first pass, and both are ready well inside one 15s interval. + start_interval: 2s mem_limit: 1g pids_limit: 1024 cap_drop: [ALL] diff --git a/demo/pyproject.toml b/demo/pyproject.toml index f6a8ff5..97bea80 100644 --- a/demo/pyproject.toml +++ b/demo/pyproject.toml @@ -19,3 +19,5 @@ [tool.ruff.lint] pydocstyle.convention = "google" select = ["ALL"] + # COM812 fights the formatter (`ruff format --check` runs in `just lint`). + ignore = ["COM812"] diff --git a/docs/RUNBOOK.md b/docs/RUNBOOK.md index c19bf08..b9ab856 100644 --- a/docs/RUNBOOK.md +++ b/docs/RUNBOOK.md @@ -202,7 +202,7 @@ cd infra && tofu apply ## Upgrading images Dependabot opens PRs that bump the pinned versions, and CI runs -`just check` on each one. The validators (promtool, otelcol) read their image +`just validate` and `just smoke` on each one. The validators (promtool, otelcol) read their image versions from `compose.yml`, so every bump is checked with the exact binaries the stack will run. A new version that changes its config syntax fails CI before it reaches the host. Patch bumps arrive grouped, one PR for diff --git a/justfile b/justfile index 278d0e1..0700ed4 100644 --- a/justfile +++ b/justfile @@ -1,9 +1,21 @@ set dotenv-load -# Stateful services and their volumes, shared by backup/restore. The volume -# names assume the compose project name "monitoring" (see guard in backup). +# Stateful services. Their volumes are __data, which is how +# _backup and _restore find them for whichever compose project they act on. stateful := "grafana prometheus loki tempo" -backup_mounts := "-v monitoring_grafana_data:/data/grafana -v monitoring_prometheus_data:/data/prometheus -v monitoring_loki_data:/data/loki -v monitoring_tempo_data:/data/tempo" + +# Helper and lint images, pinned once. `lint` pulls the lint set in parallel +# up front: serial first-use pulls are most of a cold run. +alpine := "alpine:3.24@sha256:28bd5fe8b56d1bd048e5babf5b10710ebe0bae67db86916198a6eec434943f8b" +jq := "ghcr.io/jqlang/jq:1.8.1" +yamllint := "pipelinecomponents/yamllint:0.35.13" +yamlfmt := "ghcr.io/google/yamlfmt:0.17.2" +actionlint := "rhysd/actionlint:1.7.12" +shellcheck := "koalaman/shellcheck:v0.11.0" +ruff := "ghcr.io/astral-sh/ruff:0.14.2" +tofu := "ghcr.io/opentofu/opentofu:1.12.3" +gitleaks := "zricethezav/gitleaks:v8.30.1@sha256:c00b6bd0aeb3071cbcb79009cb16a60dd9e0a7c60e2be9ab65d25e6bc8abbb7f" +lint_images := jq + " " + yamllint + " " + yamlfmt + " " + actionlint + " " + shellcheck + " " + ruff + " " + tofu + " " + gitleaks # `demo` and `smoke` each bring up a throwaway copy of the core stack, so both # run under their own compose project and their own Grafana port (see @@ -13,41 +25,39 @@ backup_mounts := "-v monitoring_grafana_data:/data/grafana -v monitoring_prometh # explicit -f list also keeps a host's COMPOSE_FILE (tunnel) out of both. demo_project := "monitoring-demo" demo_port := env("DEMO_PORT", "3002") -demo_url := "http://localhost:" + demo_port compose_demo := "SANDBOX_PORT=" + demo_port + " docker compose -p " + demo_project + " -f compose.yml -f compose.demo.yml -f compose.sandbox.yml" # What compose will actually name the project for the up-paths, so the queue -# volume gets chowned where the collector will look for it. +# volume gets chowned where the collector will look for it, and backup/restore +# mount the volumes the running stack uses. core_project := env("COMPOSE_PROJECT_NAME", "monitoring") # Same isolation for smoke, on its own project and port so a smoke run and a -# demo stack can also coexist. +# demo stack can also coexist. The smoke stack boots in production shape: +# Grafana with JWT auth on (the Cloudflare Access path, otherwise only ever +# schema-checked) and fixed notification URLs so the contact-point assertion +# can check exact values instead of "not empty". smoke_project := "monitoring-smoke" smoke_port := env("SMOKE_PORT", "3001") -smoke_url := "http://localhost:" + smoke_port -compose_smoke := "SANDBOX_PORT=" + smoke_port + " docker compose -p " + smoke_project + " -f compose.yml -f compose.sandbox.yml" +smoke_env := "SANDBOX_PORT=" + smoke_port + " GRAFANA_JWT_AUTH=true CF_ACCESS_TEAM_DOMAIN=smoke CF_ACCESS_AUD=smoke ALERT_WEBHOOK_URL=https://smoke.invalid/alerts HEARTBEAT_URL=https://smoke.invalid/heartbeat" +compose_smoke := smoke_env + " docker compose -p " + smoke_project + " -f compose.yml -f compose.sandbox.yml" -# dashboards/*.json as the paths they get when mounted at /dashboards, shared -# by check and smoke. 2>/dev/null so an empty dashboards/ doesn't abort every -# recipe in this file, including the teardown ones. +# dashboards/*.json as the paths they get when mounted at /dashboards. 2>/dev/null +# so an empty dashboards/ doesn't abort every recipe in this file, including the +# teardown ones; `lint` refuses the empty list instead. dash_paths := `ls dashboards/*.json 2>/dev/null | sed 's|^dashboards|/dashboards|' | tr '\n' ' '` +# List the recipes. default: @just --list -# backup/restore mount volumes by literal name, which assumes compose's default -# project name (compose.yml sets `name: monitoring`). COMPOSE_PROJECT_NAME -# overrides it, so they would silently target volumes nothing uses. -_project-guard: - @[ -z "${COMPOSE_PROJECT_NAME:-}" ] || { echo "error: COMPOSE_PROJECT_NAME is set; this recipe expects the monitoring_* volume names" >&2; exit 1; } - # The collector's queue volume must be writable by the image's uid 10001, but # a fresh named volume is root-owned and the image is distroless (no chown at # startup possible). Idempotent, so every up-path just runs it. Takes the # project name because smoke brings the stack up under a different one. _queue-volume project: @docker volume create {{project}}_otel_queue > /dev/null - @docker run --rm --network none -v {{project}}_otel_queue:/q alpine:3.24@sha256:28bd5fe8b56d1bd048e5babf5b10710ebe0bae67db86916198a6eec434943f8b chown 10001:10001 /q + @docker run --rm --network none -v {{project}}_otel_queue:/q {{alpine}} chown 10001:10001 /q # Overlays are host config: COMPOSE_FILE in .env names the file set (see # .env.example), and every recipe here (up, down, logs, ps, backup) acts on @@ -72,6 +82,7 @@ _expose-guards: @[ -n "${HEARTBEAT_URL:-}" ] || echo "WARNING: HEARTBEAT_URL is empty; the stack goes live without a dead-man's switch" >&2 @[ -n "${ALERT_WEBHOOK_URL:-}" ] || { echo "error: ALERT_WEBHOOK_URL is empty; every alert would fire into an empty webhook URL and be dropped. The heartbeat keeps pinging either way, so this failure looks healthy from the outside. Set it, or comment out this guard" >&2; exit 1; } +# Stop the stack; volumes stay. down: docker compose down --remove-orphans @@ -81,11 +92,6 @@ down: demo: (_queue-volume demo_project) {{compose_demo}} up -d --build -# Used by CI to catch a broken demo app before it merges. -# Build the demo image without starting anything. -demo-build: - {{compose_demo}} build - # Removes only the demo services; the demo project's own core stack keeps # running. `just demo-destroy` takes the whole thing down. # Stop the demo telemetry source. @@ -98,9 +104,11 @@ demo-down: demo-destroy: {{compose_demo}} down --remove-orphans --volumes +# Follow logs, optionally of one service. logs service="": docker compose logs -f {{service}} +# Container status of the stack. ps: docker compose ps @@ -109,6 +117,7 @@ ps: restart service: _guard-if-exposed docker compose restart {{service}} +# Pull the pinned images. pull: docker compose pull @@ -116,18 +125,25 @@ pull: tail service: docker compose logs -f --no-log-prefix {{service}} | jq -R 'fromjson? // .' -# All validators run in containers: no host installs, no network. The -# promtool/otelcol images are read from compose.yml so they can't drift from -# the versions the stack actually runs. +# Every check in this file runs in a container: no host installs, no network. +# `lint` is the static half in small tool images; `validate` runs the stack's +# own images against its configs. Both are seconds warm, so the pre-push hook +# (`just hooks`) runs both; CI puts `lint` on its own job so a lint failure +# never waits on the stack images, which the smoke job pulls anyway. # Validate every config in the repo. -check: - docker compose config -q +check: lint validate + +# Static checks in small tool images (~130 MB cold, seconds warm). +lint: + @printf '%s\n' {{lint_images}} | xargs -P 8 -n 1 docker pull -q >/dev/null + # `-f compose.yml` on the first line, not the host's COMPOSE_FILE: on a + # tunnel host that would validate the tunnel set with no dummy token, so + # `just lint` would mean something different there than in CI. + docker compose -f compose.yml config -q CLOUDFLARE_TUNNEL_TOKEN=dummy docker compose -f compose.yml -f compose.tunnel.yml config -q {{compose_demo}} config -q + # Also the JWT interpolation in compose.yml: compose_smoke turns it on. {{compose_smoke}} config -q - # The tunnel set with JWT auth on, so the JWK URL and claims interpolation - # in compose.tunnel.yml is at least schema-checked; nothing else enables it. - CLOUDFLARE_TUNNEL_TOKEN=dummy GRAFANA_JWT_AUTH=true CF_ACCESS_TEAM_DOMAIN=dummy CF_ACCESS_AUD=dummy docker compose -f compose.yml -f compose.tunnel.yml config -q # The spoke overlays every project host layers onto its own compose.yml; a # bad interpolation or a network they forget to declare otherwise first # fails on a client machine, after vendoring. @@ -136,30 +152,52 @@ check: # documented default must be refused on its own. Nothing else runs them: # CI never sets COMPOSE_FILE to the tunnel overlay. @good="OTLP_AUTH_TOKEN=t GRAFANA_ADMIN_PASSWORD=p GRAFANA_ROOT_URL=https://g.example GRAFANA_COOKIE_SECURE=true GRAFANA_JWT_AUTH=true CF_ACCESS_TEAM_DOMAIN=d CF_ACCESS_AUD=a HEARTBEAT_URL=https://h ALERT_WEBHOOK_URL=https://w"; \ - env $good just _expose-guards; \ + env $good just _expose-guards || { echo "error: exposure guards rejected a fully set environment" >&2; exit 1; }; \ for bad in OTLP_AUTH_TOKEN=local-dev-token GRAFANA_ADMIN_PASSWORD=change-me GRAFANA_ROOT_URL=http://g.example GRAFANA_COOKIE_SECURE=false CF_ACCESS_AUD= ALERT_WEBHOOK_URL=; do \ ! env $good $bad just _expose-guards 2>/dev/null || { echo "error: exposure guards accepted $bad" >&2; exit 1; }; \ done - images="$(docker compose config --images)" && \ - docker run --rm --network none -v ./config/prometheus.yaml:/etc/prometheus/prometheus.yaml:ro --entrypoint promtool $(echo "$images" | grep prom/prometheus) check config /etc/prometheus/prometheus.yaml && \ - docker run --rm --network none -e OTLP_AUTH_TOKEN=dummy -e DEPARTMENT=dummy -v ./config/otel-collector.yaml:/etc/otelcol/config.yaml:ro $(echo "$images" | grep opentelemetry-collector) validate --config=/etc/otelcol/config.yaml && \ - docker run --rm --network none -v ./config/loki.yaml:/etc/loki/loki.yaml:ro $(echo "$images" | grep grafana/loki) -config.file=/etc/loki/loki.yaml -verify-config && \ - docker run --rm --network none -v ./config/tempo.yaml:/etc/tempo/tempo.yaml:ro $(echo "$images" | grep grafana/tempo) -config.file=/etc/tempo/tempo.yaml -config.verify=true # line-length at 120, not the default 80: digest-pinned image refs need # ~130 but count as non-breakable mappings. The rendered project-*/coverage # rules are ignored: bootstrap.sh writes them, and their expr lines grow - # with every onboarded project. - docker run --rm --network none -v .:/code:ro pipelinecomponents/yamllint:0.35.13 yamllint -d '{extends: relaxed, rules: {line-length: {max: 120, allow-non-breakable-inline-mappings: true}}, ignore: [.git/, backups/, infra/.terraform/, config/grafana/alerting/project-*.yaml, config/grafana/alerting/coverage.yaml]}' . - docker run --rm --network none -v .:/repo:ro -w /repo rhysd/actionlint:1.7.12 -color - docker run --rm --network none -v .:/mnt:ro koalaman/shellcheck:v0.11.0 bootstrap.sh templates/run_scheduled.sh infra/generate-imports.sh - docker run --rm --network none -v ./demo:/demo:ro ghcr.io/astral-sh/ruff:0.14.2 check --no-cache /demo - docker run --rm --network none -v ./infra:/infra:ro -w /infra ghcr.io/opentofu/opentofu:1.12.3 fmt -check - docker run --rm --network none -v ./dashboards:/dashboards:ro ghcr.io/jqlang/jq:1.8.1 empty {{dash_paths}} + # with every onboarded project. The templates they come from are checked + # by rendering them through bootstrap.sh itself into a scratch dir, so a + # template edit Grafana would reject fails here and not on the next + # onboarding. + docker run --rm --network none -v .:/code:ro {{yamllint}} yamllint -d '{extends: relaxed, rules: {line-length: {max: 120, allow-non-breakable-inline-mappings: true}}, ignore: [.git/, backups/, infra/.terraform/, config/grafana/alerting/project-*.yaml, config/grafana/alerting/coverage.yaml]}' . + @d=$(mktemp -d) && BOOTSTRAP_OUT_DIR="$d" ./bootstrap.sh dummy dummy >/dev/null && docker run --rm --network none -v "$d":/code:ro {{yamllint}} yamllint -d '{extends: relaxed, rules: {line-length: disable}}' .; rc=$?; rm -rf "$d"; exit $rc + docker run --rm --network none -v .:/repo:ro -w /repo {{actionlint}} -color + docker run --rm --network none -v .:/mnt:ro {{shellcheck}} bootstrap.sh scripts/smoke.sh templates/run_scheduled.sh infra/generate-imports.sh + docker run --rm --network none -v ./demo:/demo:ro {{ruff}} check --no-cache /demo + docker run --rm --network none -v ./demo:/demo:ro {{ruff}} format --check --no-cache /demo + # Formatting is enforced, not just linted: `just fmt` is the fix. + docker run --rm --network none -v .:/code:ro -w /code {{yamlfmt}} -lint . + docker run --rm --network none -v ./infra:/infra:ro -w /infra {{tofu}} fmt -check + # Dashboards: valid JSON, and every datasource they name is one that + # datasources.yaml provisions (plus Grafana's built-in). A typo here + # provisions fine and renders empty panels. + @[ -n "{{dash_paths}}" ] || { echo "error: no dashboards/*.json to check" >&2; exit 1; } + docker run --rm --network none -v ./dashboards:/dashboards:ro {{jq}} empty {{dash_paths}} + @bad=$(docker run --rm --network none -v ./dashboards:/dashboards:ro {{jq}} -r '.. | objects | select(has("datasource")) | .datasource | (if type == "object" then .uid else . end) | strings' {{dash_paths}} | sort -u | grep -vxF "$(sed -n 's/^ *uid: *//p' config/grafana/datasources.yaml; echo grafana)"); \ + [ -z "$bad" ] || { echo "error: dashboards reference datasource uids that are not provisioned:" $bad >&2; exit 1; } # Secrets that reached git history. Scans commits, not the working tree, so # it sees exactly what is in the repo and never the gitignored .env. A # working-tree scan flags .env's real tokens and fails on every dev machine. # Needs full history: a shallow CI clone has one commit and passes vacuously. - docker run --rm --network none -v .:/repo:ro zricethezav/gitleaks:v8.30.1@sha256:c00b6bd0aeb3071cbcb79009cb16a60dd9e0a7c60e2be9ab65d25e6bc8abbb7f git --redact --no-banner /repo + docker run --rm --network none -v .:/repo:ro {{gitleaks}} git --redact --no-banner /repo + +# Image ref of one compose.yml service, so the validators below can't drift +# from the versions the stack runs. (`config --images ` also lists the +# service's dependencies, hence the json route.) +_image service: + @docker compose -f compose.yml config --format json | docker run --rm -i {{jq}} -er '.services["{{service}}"].image // error("no service {{service}} in compose.yml")' + +# The validators run in the images the stack itself runs. ~300 MB cold. +# Run each config through the binary that will load it. +validate: + docker run --rm --network none -v ./config/prometheus.yaml:/etc/prometheus/prometheus.yaml:ro --entrypoint promtool $(just _image prometheus) check config /etc/prometheus/prometheus.yaml + docker run --rm --network none -e OTLP_AUTH_TOKEN=dummy -e DEPARTMENT=dummy -v ./config/otel-collector.yaml:/etc/otelcol/config.yaml:ro $(just _image otel-collector) validate --config=/etc/otelcol/config.yaml + docker run --rm --network none -v ./config/loki.yaml:/etc/loki/loki.yaml:ro $(just _image loki) -config.file=/etc/loki/loki.yaml -verify-config + docker run --rm --network none -v ./config/tempo.yaml:/etc/tempo/tempo.yaml:ro $(just _image tempo) -config.file=/etc/tempo/tempo.yaml -config.verify=true # The one config vendored verbatim onto every project host; a syntax error # here otherwise first surfaces as a crash-looping agent on a client machine. # Image ref matches templates/compose.telemetry.yml; keep them in step. @@ -169,35 +207,57 @@ check: # container, which needs network access to fetch the provider. # Full OpenTofu validation (downloads the provider, so not part of `check`). infra-validate: - @d=$(mktemp -d) && cp infra/main.tf infra/.terraform.lock.hcl "$d"/ && docker run --rm --entrypoint sh -v "$d":/src:ro ghcr.io/opentofu/opentofu:1.12.3 -c 'mkdir /work && cp /src/main.tf /src/.terraform.lock.hcl /work && cd /work && tofu init -backend=false -input=false >/dev/null && tofu validate'; rc=$?; rm -rf "$d"; exit $rc + @d=$(mktemp -d) && cp infra/main.tf infra/.terraform.lock.hcl "$d"/ && docker run --rm --entrypoint sh -v "$d":/src:ro {{tofu}} -c 'mkdir /work && cp /src/main.tf /src/.terraform.lock.hcl /work && cd /work && tofu init -backend=false -input=false >/dev/null && tofu validate'; rc=$?; rm -rf "$d"; exit $rc -# Format YAML in place (needs yamlfmt on the host; optional). +# Same container-only rule as `lint`; --user so the rewritten files stay yours. +# Format YAML in place. fmt: - yamlfmt . + docker run --rm --network none --user "$(id -u):$(id -g)" -v .:/code -w /code {{yamlfmt}} . + +# Install the git hooks: gitleaks + lint on commit, validate on push (see .pre-commit-config.yaml). +hooks: + prek install --hook-type pre-commit --hook-type commit-msg --hook-type pre-push # The tarball is mode 0600: it contains the Grafana DB and webhook secrets, so -# copy it off-host and keep it private. Services are paused during the copy and -# unpaused unconditionally afterwards: `pause` is per-container and can fail -# halfway, so an unpause reached only on success would leave the stack frozen. +# copy it off-host and keep it private. # Snapshot all stateful volumes to backups/.tar.gz. -backup: _project-guard - mkdir -p backups - -docker compose unpause {{stateful}} 2>/dev/null - rc=0; docker compose pause {{stateful}} && docker run --rm {{backup_mounts}} -v ./backups:/backups alpine:3.24@sha256:28bd5fe8b56d1bd048e5babf5b10710ebe0bae67db86916198a6eec434943f8b sh -c 'umask 077 && tar czf /backups/monitoring-$(date +%Y%m%d-%H%M%S).tar.gz -C /data .' || rc=$?; docker compose unpause {{stateful}}; exit $rc - @ls -lh backups/ | tail -1 +backup: (_backup core_project "backups") + +# gzip -1: the stack is paused for as long as the tar runs, and the TSDB +# chunks are already compressed, so the higher levels cost time for nothing. +# Services are paused during the copy and unpaused unconditionally afterwards: +# `pause` is per-container and can fail halfway, so an unpause reached only on +# success would leave the stack frozen. A failed unpause fails the recipe for +# the same reason: a green exit with the stack still SIGSTOPped is worse. +_backup project dir: + mkdir -p {{dir}} + @-docker compose -p {{project}} unpause {{stateful}} >/dev/null 2>&1 + @rc=0; m=$(for s in {{stateful}}; do printf -- '-v {{project}}_%s_data:/data/%s ' $s $s; done); docker compose -p {{project}} pause {{stateful}} && docker run --rm --network none $m -v {{absolute_path(dir)}}:/backups {{alpine}} sh -c 'umask 077 && tar cf - -C /data . | gzip -1 > /backups/monitoring-$(date +%Y%m%d-%H%M%S).tar.gz' || rc=$?; docker compose -p {{project}} unpause {{stateful}} || { echo "error: unpause failed; the stack is still paused" >&2; rc=1; }; exit $rc + @ls -lh {{dir}}/ | tail -1 + +# Restore a backup tarball into the volumes (stops the stack; wipes current state). +restore file: (_restore core_project file "backups") + @echo "Restored {{file}}. Run 'just up' to start the stack." # The wipe is unrecoverable, so the current state is snapshotted first: if the # extract dies halfway (full disk, wrong volume set) the volumes are left -# partial, and backups/pre-restore-*.tar.gz is the only way back. -# Restore a backup tarball into the volumes (stops the stack; wipes current state). -restore file: _project-guard +# partial, and /pre-restore-*.tar.gz is the only way back. +_restore project file dir: @[ -f "{{file}}" ] || { echo "error: {{file}} not found" >&2; exit 1; } - docker run --rm --network none -v {{absolute_path(file)}}:/backup.tar.gz:ro alpine:3.24@sha256:28bd5fe8b56d1bd048e5babf5b10710ebe0bae67db86916198a6eec434943f8b tar tzf /backup.tar.gz > /dev/null - docker compose down --remove-orphans - mkdir -p backups - docker run --rm --network none {{backup_mounts}} -v ./backups:/backups alpine:3.24@sha256:28bd5fe8b56d1bd048e5babf5b10710ebe0bae67db86916198a6eec434943f8b sh -c 'umask 077 && tar czf /backups/pre-restore-$(date +%Y%m%d-%H%M%S).tar.gz -C /data .' - docker run --rm --network none {{backup_mounts}} -v {{absolute_path(file)}}:/backup.tar.gz:ro alpine:3.24@sha256:28bd5fe8b56d1bd048e5babf5b10710ebe0bae67db86916198a6eec434943f8b sh -c 'for d in /data/*; do find "$d" -mindepth 1 -delete; done && tar xzf /backup.tar.gz -C /data' - @echo "Restored {{file}}. Run 'just up' to start the stack." + docker run --rm --network none -v {{absolute_path(file)}}:/backup.tar.gz:ro {{alpine}} tar tzf /backup.tar.gz > /dev/null + docker compose -p {{project}} down --remove-orphans + mkdir -p {{dir}} + m=$(for s in {{stateful}}; do printf -- '-v {{project}}_%s_data:/data/%s ' $s $s; done); docker run --rm --network none $m -v {{absolute_path(dir)}}:/backups {{alpine}} sh -c 'umask 077 && tar czf /backups/pre-restore-$(date +%Y%m%d-%H%M%S).tar.gz -C /data .' + m=$(for s in {{stateful}}; do printf -- '-v {{project}}_%s_data:/data/%s ' $s $s; done); docker run --rm --network none $m -v {{absolute_path(file)}}:/backup.tar.gz:ro {{alpine}} sh -c 'for d in /data/*; do find "$d" -mindepth 1 -delete; done && tar xzf /backup.tar.gz -C /data' + +# The only rehearsal of the one recipe that wipes state: backs up the smoke +# stack's volumes, restores them, and asserts Grafana's database came back. +# Needs a booted smoke stack (`just smoke`); `just smoke-down` cleans up after. +# Not in CI: it is a rehearsal for an operator, and 35s per push buys nothing a +# run after touching _backup/_restore does not. +# Round-trip backup and restore on the smoke stack. +restore-check: + @d=$(mktemp -d) && just _backup {{smoke_project}} "$d" && f=$(ls "$d"/monitoring-*.tar.gz) && just _restore {{smoke_project}} "$f" "$d" && docker run --rm --network none -v {{smoke_project}}_grafana_data:/g:ro {{alpine}} test -s /g/grafana.db && echo "Backup round-trip ok"; rc=$?; rm -rf "$d"; exit $rc # `--wait` does the readiness and crash-loop work: it blocks on the grafana and # prometheus healthchecks in compose.yml and fails if any container exits, so @@ -205,37 +265,13 @@ restore file: _project-guard # A service with no healthcheck of its own (otel-collector, loki, tempo) still # fails the wait while it is restarting, so a crash-looping collector is caught. # That costs the full --wait-timeout to report, where an explicit exit check -# failed immediately. -# What `--wait` cannot see is provisioning. Grafana answers /api/health long -# before it has read the provisioning dirs, and skips a broken dashboard or a -# malformed alert group silently, so the assertions below cover that. They share -# one loop because both land asynchronously. -# Boot an isolated copy of the core stack and assert it provisioned everything. +# failed immediately. What `--wait` cannot see is everything that lands after +# /api/health answers: provisioning, scrapes, the data paths. scripts/smoke.sh +# asserts those. +# Boot an isolated copy of the core stack and assert it works end to end. smoke: (_queue-volume smoke_project) {{compose_smoke}} up -d --wait --wait-timeout 120 - @want_dash="$(docker run --rm --network none -v ./dashboards:/dashboards:ro ghcr.io/jqlang/jq:1.8.1 -r .uid {{dash_paths}})"; want_rules=$(grep -h '^ *title:' config/grafana/alerting/*.yaml | wc -l); auth="user = \"admin:${GRAFANA_ADMIN_PASSWORD}\""; n=0; while :; do \ - search=$(printf '%s\n' "$auth" | curl -sf -K - '{{smoke_url}}/api/search?type=dash-db&limit=5000') && rules=$(printf '%s\n' "$auth" | curl -sf -K - {{smoke_url}}/api/v1/provisioning/alert-rules) || { echo "error: Grafana API request failed. Check GRAFANA_ADMIN_PASSWORD, and that Grafana answers on {{smoke_url}}" >&2; exit 1; }; \ - have=$(printf '%s' "$search" | docker run --rm -i ghcr.io/jqlang/jq:1.8.1 -r '.[].uid'); got=$(printf '%s' "$rules" | docker run --rm -i ghcr.io/jqlang/jq:1.8.1 length); \ - missing=""; for uid in $want_dash; do printf '%s' "$have" | grep -qx "$uid" || missing="$missing $uid"; done; \ - [ -z "$missing" ] && [ "$want_rules" = "$got" ] && break; \ - n=$((n+3)); [ $n -ge 60 ] && { echo "error: not provisioned after 60s. Dashboards missing:${missing:- none}; alert rules $got/$want_rules (a malformed file provisions none of its group). See just smoke-logs" >&2; exit 1; }; \ - sleep 3; \ - done - # Provisioning proved the rules exist; this proves they can be delivered - # and that the data path works. Grafana expands $VAR in the alerting - # provisioning files. A Grafana that stopped doing so would store the - # literal name and every notification would fail silently (see - # contact-points.yaml). Then one OTLP log through the collector's bearer - # auth, asserted back out of Loki with the department label the collector - # stamps: the label chain the keystone rules key on, end to end. - @auth="user = \"admin:${GRAFANA_ADMIN_PASSWORD}\""; \ - printf '%s\n' "$auth" | curl -sf -K - {{smoke_url}}/api/v1/provisioning/contact-points | docker run --rm -i ghcr.io/jqlang/jq:1.8.1 -e '[.[] | select(.uid | startswith("cp-")) | .settings.url] | all(startswith("$") | not)' >/dev/null || { echo "error: a contact point still carries a literal \$VAR; Grafana did not expand the alerting provisioning file" >&2; exit 1; }; \ - ts="$(date +%s)000000000"; body='{"resourceLogs":[{"resource":{"attributes":[{"key":"service.name","value":{"stringValue":"smoke"}},{"key":"project","value":{"stringValue":"smoke"}},{"key":"env","value":{"stringValue":"ci"}}]},"scopeLogs":[{"logRecords":[{"timeUnixNano":"'"$ts"'","body":{"stringValue":"smoke"}}]}]}]}'; \ - docker run --rm --network {{smoke_project}}_default alpine:3.24@sha256:28bd5fe8b56d1bd048e5babf5b10710ebe0bae67db86916198a6eec434943f8b wget -qO- --header="Authorization: Bearer ${OTLP_AUTH_TOKEN}" --header='Content-Type: application/json' --post-data="$body" http://otel-collector:4318/v1/logs >/dev/null || { echo "error: the collector refused an OTLP log with the .env token" >&2; exit 1; }; \ - n=0; until printf '%s\n' "$auth" | curl -sf -K - -G --data-urlencode "query={project=\"smoke\",env=\"ci\",department=\"${DEPARTMENT:-cml}\"}" '{{smoke_url}}/api/datasources/proxy/uid/loki/loki/api/v1/query_range' | grep -q '"smoke"'; do \ - n=$((n+3)); [ $n -ge 60 ] && { echo "error: the smoke log never reached Loki with its department label; see just smoke-logs" >&2; exit 1; }; sleep 3; \ - done - @echo "Stack healthy" + {{smoke_env}} SMOKE_URL=http://localhost:{{smoke_port}} SMOKE_PROJECT={{smoke_project}} scripts/smoke.sh # Logs from the smoke stack (its own project, so `just logs` will not show it). smoke-logs: @@ -243,6 +279,6 @@ smoke-logs: # Safe: -p scopes it to the smoke project, so it cannot touch a real stack on # the same host. -# Tear down the smoke stack and its throwaway volumes. +# Tear down the smoke stack and its throwaway volumes (-t 1: nothing in it is worth a graceful stop). smoke-down: - {{compose_smoke}} down --remove-orphans --volumes + {{compose_smoke}} down --remove-orphans --volumes -t 1 diff --git a/scripts/smoke.sh b/scripts/smoke.sh new file mode 100755 index 0000000..c873e74 --- /dev/null +++ b/scripts/smoke.sh @@ -0,0 +1,101 @@ +#!/usr/bin/env bash +# Assertions `just smoke` runs against the stack it just booted. A script rather +# than justfile one-liners so it can have functions and go through shellcheck. +# Host jq, not the container: it is part of the declared toolchain here and +# preinstalled on the CI runner, and a container start per poll adds up. +# +# Everything asserted here lands asynchronously after Grafana's /api/health +# answers, so each assertion polls (`poll`, one second, up to a minute). +set -euo pipefail + +url="${SMOKE_URL:?}" +project="${SMOKE_PROJECT:?}" +dept="${DEPARTMENT:-cml}" +command -v jq >/dev/null || { echo "error: jq is required" >&2; exit 1; } + +die() { echo "error: $*" >&2; exit 1; } +# Admin basic auth through curl's stdin config: argv is world-readable in ps. +gf() { printf 'user = "admin:%s"\n' "$GRAFANA_ADMIN_PASSWORD" | curl -sf -K - "$@"; } +promq() { gf -G --data-urlencode "query=$1" "$url/api/datasources/proxy/uid/prometheus/api/v1/query"; } +poll() { local n=0; until "$@"; do n=$((n + 1)); [[ $n -lt 60 ]] || return 1; sleep 1; done; } + +# ------------------------------------------------------------------ provisioning +# Grafana skips a broken dashboard or a malformed alert group silently, so the +# provisioned sets must equal what the repo holds, uid for uid. Both sides come +# from the repo, so adding a dashboard or a rule never breaks this. A missing +# dashboards/ glob fails the jq call, which is the right answer. +want_dash="$(jq -r .uid dashboards/*.json | sort)" +rule_files=() +for f in config/grafana/alerting/*.yaml; do + case "$f" in */contact-points.yaml | */notification-policies.yaml) ;; *) rule_files+=("$f") ;; esac +done +want_rules="$(sed -n 's/^ *- uid: *//p' "${rule_files[@]}" | sort)" +[[ -n "$want_dash" && -n "$want_rules" ]] || die "no dashboards or alert rules in the repo to assert against" + +provisioned() { + { have_dash="$(gf "$url/api/search?type=dash-db&limit=5000" | jq -r '.[].uid' | sort)" \ + && rules="$(gf "$url/api/v1/provisioning/alert-rules")"; } \ + || die "Grafana API request failed. Check GRAFANA_ADMIN_PASSWORD, and that Grafana answers on $url" + have_rules="$(jq -r '.[].uid' <<<"$rules" | sort)" + [[ "$have_dash" == "$want_dash" && "$have_rules" == "$want_rules" ]] +} +poll provisioned || die "not provisioned after 60s (a malformed file provisions none of its group). See just smoke-logs +dashboards, want vs have: $(diff <(echo "$want_dash") <(echo "$have_dash") | grep '^[<>]' | tr '\n' ' ') +alert rules, want vs have: $(diff <(echo "$want_rules") <(echo "$have_rules") | grep '^[<>]' | tr '\n' ' ')" +jq -e 'all(.isPaused | not)' <<<"$rules" >/dev/null \ + || die "paused alert rules never fire: $(jq -r '.[] | select(.isPaused) | .uid' <<<"$rules" | tr '\n' ' ')" + +# --------------------------------------------------------------- contact points +# Grafana expands $VAR in the alerting provisioning files. The exact URLs the +# stack was given must come back, not merely "something that is not literally +# $ALERT_WEBHOOK_URL": an empty value passes that test and drops every alert. +gf "$url/api/v1/provisioning/contact-points" \ + | jq -e --arg a "$ALERT_WEBHOOK_URL" --arg h "$HEARTBEAT_URL" \ + '[.[] | select(.uid | startswith("cp-")) | .settings.url] | sort == ([$a, $h] | sort)' >/dev/null \ + || die "contact points do not carry ALERT_WEBHOOK_URL and HEARTBEAT_URL; Grafana did not expand the provisioning file, or a receiver is missing" + +# ------------------------------------------------------------------- JWT auth +# The stack booted with GRAFANA_JWT_AUTH=true. Grafana ignores an env key it +# no longer knows, so read the parsed setting back, and check that a forged +# Access header is refused rather than auto-signed-up. +gf "$url/api/admin/settings" | jq -e '.["auth.jwt"].enabled == "true"' >/dev/null \ + || die "Grafana did not enable JWT auth from GF_AUTH_JWT_*; the Cloudflare Access path is broken" +code="$(curl -s -o /dev/null -w '%{http_code}' -H 'Cf-Access-Jwt-Assertion: not-a-jwt' "$url/api/dashboards/home")" +[[ "$code" == 401 ]] || die "a forged Access token got HTTP $code from Grafana, expected 401" + +# ------------------------------------------------------------- scrape targets +# promtool only checks syntax; a renamed service leaves its scrape job silently +# empty, and TargetDown (up == 0) then matches nothing. Every job in +# prometheus.yaml must have scraped its target successfully. +n_jobs="$(grep -c '^ *- job_name:' config/prometheus.yaml)" +all_up() { [[ "$(promq 'count(up == 1)' | jq -r '.data.result[0].value[1] // 0')" == "$n_jobs" ]]; } +poll all_up || die "not all $n_jobs scrape targets are up: $(promq up | jq -r '.data.result[] | "\(.metric.job)=\(.value[1])"' | tr '\n' ' ')" + +# ------------------------------------------------------------- data paths +# One metric and one log through the collector's bearer auth, read back out of +# Prometheus and Loki with the identity labels the alert rules key on: project +# and env promoted from the resource, department stamped by the collector. A +# collector pipeline or prometheus.yaml's promote_resource_attributes can break +# with both configs still valid; this is the only place that shows. Posted from +# inside the stack's network (the sandbox overlay publishes no ingestion ports) +# with Grafana's wget: it is on the same network and needs no extra image. +ts="$(date +%s)000000000" +res='{"attributes":[{"key":"service.name","value":{"stringValue":"smoke"}},{"key":"project","value":{"stringValue":"smoke"}},{"key":"env","value":{"stringValue":"ci"}}]}' +otlp() { + docker compose -p "$project" exec -T grafana wget -qO- --header="Authorization: Bearer ${OTLP_AUTH_TOKEN}" \ + --header='Content-Type: application/json' --post-data="$2" "http://otel-collector:4318/v1/$1" >/dev/null +} +otlp metrics '{"resourceMetrics":[{"resource":'"$res"',"scopeMetrics":[{"metrics":[{"name":"smoke_up","gauge":{"dataPoints":[{"asInt":"1","timeUnixNano":"'"$ts"'"}]}}]}]}]}' \ + || die "the collector refused an OTLP metric with the .env token" +otlp logs '{"resourceLogs":[{"resource":'"$res"',"scopeLogs":[{"logRecords":[{"timeUnixNano":"'"$ts"'","body":{"stringValue":"smoke"}}]}]}]}' \ + || die "the collector refused an OTLP log with the .env token" +smoke_metric() { promq "smoke_up{job=\"smoke\",project=\"smoke\",env=\"ci\",department=\"$dept\"}" | jq -e '.data.result | length > 0' >/dev/null; } +smoke_log() { + gf -G --data-urlencode "query={project=\"smoke\",env=\"ci\",department=\"$dept\"}" \ + "$url/api/datasources/proxy/uid/loki/loki/api/v1/query_range" \ + | jq -e 'any(.data.result[].values[][1]; . == "smoke")' >/dev/null +} +poll smoke_metric || die "the smoke metric never reached Prometheus with its project/env/department labels; see just smoke-logs" +poll smoke_log || die "the smoke log never reached Loki with its department label; see just smoke-logs" + +echo "Stack healthy" From 93a99a6465db9a2c5cd6f1f5741620bad694cbff Mon Sep 17 00:00:00 2001 From: Simon van Lierde Date: Sun, 6 Sep 2026 16:25:13 +0000 Subject: [PATCH 51/85] feat(alerting): warn on disk fill rate and series cardinality - HostDiskFilling: a 6h linear fit projects a filesystem (over 10G) full within 3 days; fires ahead of the 20% HostDiskSpaceLow floor - PrometheusCardinalityHigh: over 100k active series, ~7x the current baseline; series count is what grows the TSDB, not time --- CHANGELOG.md | 3 ++ config/grafana/alerting/rules.yaml | 68 +++++++++++++++++++++++++++++- docs/RUNBOOK.md | 6 ++- 3 files changed, 75 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 6072a5f..7ad6e3a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -19,6 +19,9 @@ queries that had been measuring the wrong thing. worthless by the time anyone restores. - **Full-stack scraping**: Prometheus now scrapes Grafana, Loki, and Tempo as well, so `TargetDown` covers every service. +- **Early storage warnings**: `HostDiskFilling` (a filesystem is full within + 3 days at the current rate) and `PrometheusCardinalityHigh` (over 100k + active series) fire while there is still time, ahead of the 80% backstop. - **Per-user Grafana logins**, optional: `GRAFANA_JWT_AUTH` plus `CF_ACCESS_TEAM_DOMAIN` and `CF_ACCESS_AUD` make Grafana verify the Cloudflare Access JWT (pinned to this app's `aud` tag, with both values diff --git a/config/grafana/alerting/rules.yaml b/config/grafana/alerting/rules.yaml index 3c16139..55bcd28 100644 --- a/config/grafana/alerting/rules.yaml +++ b/config/grafana/alerting/rules.yaml @@ -5,7 +5,7 @@ # returns a series only when the alert should fire, and the threshold node only asks # whether anything came back. # -# Keep the total under about ten. These are the generic rules; the per-project keystone +# Keep the total around a dozen. These are the generic rules; the per-project keystone # lives in project--.yaml beside this file, rendered by bootstrap.sh. apiVersion: 1 groups: @@ -176,6 +176,72 @@ groups: description: >- Loki and Tempo have no total-size cap, so this alert is the storage backstop. Free space or lower retention (see RUNBOOK). + # The 20% floor above is a last line; this one fires while there is still + # time to act. Linear fit over 6h, projected 3 days out. Root and data + # mounts only: a sub-10G filesystem (boot, EFI, snap) crosses zero on + # noise and is not where telemetry lands. + - uid: host-disk-filling + title: HostDiskFilling + condition: FIRING + for: 30m + noDataState: OK + execErrState: Error + isPaused: false + data: + - refId: QUERY + datasourceUid: prometheus + relativeTimeRange: + from: 3600 + to: 0 + model: + refId: QUERY + instant: true + editorMode: code + expr: > + predict_linear(node_filesystem_avail_bytes{fstype!~"tmpfs|ramfs|overlay"}[6h], 3 * 86400) < 0 + and node_filesystem_size_bytes{fstype!~"tmpfs|ramfs|overlay"} > 10e9 + - *firing + labels: + severity: warning + annotations: + summary: "Filesystem {{ $labels.mountpoint }} on {{ or $labels.host_name $labels.instance }} fills up within 3 days at the current rate" + description: >- + Extrapolated from the last 6 hours. Find what is growing (a spoke + shipping more than before, a log loop, a backup that stopped + rotating) before HostDiskSpaceLow makes it urgent. + # TSDB growth is series count, not time. A spoke that starts labelling by + # request id shows here days before it reaches the 15GB retention cap, + # and long before HostDiskSpaceLow. The threshold is ~7x the baseline + # of ~15k series with one spoke; raise it as spokes are onboarded. + - uid: prometheus-cardinality + title: PrometheusCardinalityHigh + condition: FIRING + for: 30m + noDataState: OK + execErrState: Error + isPaused: false + data: + - refId: QUERY + datasourceUid: prometheus + relativeTimeRange: + from: 3600 + to: 0 + model: + refId: QUERY + instant: true + editorMode: code + expr: prometheus_tsdb_head_series > 100000 + - *firing + labels: + severity: warning + annotations: + summary: "Prometheus holds {{ $values.QUERY }} active series, over the 100k ceiling" + description: >- + Find the label that exploded with + `topk(10, count by (__name__) ({__name__=~".+"}))` and + `topk(10, count by (job) ({__name__=~".+"}))`, then drop it at the + spoke's Alloy config or the collector. Retention is 15GB; at this + rate it fills. # A crash loop must get louder, not quieter. - orgId: 1 name: container-lifecycle diff --git a/docs/RUNBOOK.md b/docs/RUNBOOK.md index b9ab856..10aa66a 100644 --- a/docs/RUNBOOK.md +++ b/docs/RUNBOOK.md @@ -36,7 +36,11 @@ Retention is only partially size-bounded: | Tempo blocks | 7d | none | Loki and Tempo have no total-size knob, so the disk alert at 80% is the -backstop. When it fires, check the Filesystem panel on Stack Health. Then +backstop. Two earlier warnings sit in front of it: `HostDiskFilling` (a 6h +linear fit says a filesystem is full within 3 days) and +`PrometheusCardinalityHigh` (active series over 100k, ~7x the baseline; +series count, not time, is what grows the TSDB). When one fires, check the +Filesystem panel on Stack Health. Then free space, or shorten a retention window and restart the affected service (`retention_period` in `config/loki.yaml`, `block_retention` in `config/tempo.yaml`, the `--storage.tsdb.retention.*` flags in From fd0459fb8115935ec013c2bb36c3e86d0b85c567 Mon Sep 17 00:00:00 2001 From: Simon van Lierde Date: Sun, 6 Sep 2026 16:32:50 +0000 Subject: [PATCH 52/85] ci: gate by path, move lint to the push hook, group security updates - commit hook is gitleaks on the staged diff only (~0.3s); `just check` moves to pre-push, where a batch of commits pays once - prose changes (Markdown, docs/, LICENSE) skip CI; infra/ gets its own workflow that runs `just infra-validate` only when infra/ changes; `workflow_dispatch` on both - Alloy validation moves from `validate` to `lint`: the lint job has its own runner, the smoke job pulled 150MB for a one-second check - dependabot: a security-updates group for the demo's pip deps, so a batch of advisories lands as one PR instead of one per package --- .github/dependabot.yml | 6 ++++++ .github/workflows/ci.yml | 6 +++++- .github/workflows/infra.yml | 28 ++++++++++++++++++++++++++++ .pre-commit-config.yaml | 23 ++++++++--------------- config/grafana/alerting/rules.yaml | 4 +++- justfile | 19 +++++++++++-------- 6 files changed, 61 insertions(+), 25 deletions(-) create mode 100644 .github/workflows/infra.yml diff --git a/.github/dependabot.yml b/.github/dependabot.yml index 8941593..bbe2252 100644 --- a/.github/dependabot.yml +++ b/.github/dependabot.yml @@ -38,6 +38,12 @@ updates: groups: demo: patterns: ["*"] + # `groups` only cover version updates; without this, every advisory + # opens its own PR and CI run. Only pip: Docker, compose, Actions and + # Terraform get no Dependabot security advisories. + demo-security: + applies-to: security-updates + patterns: ["*"] # Cloudflare provider for infra/. Dependabot bumps the constraint in # main.tf but not the hashes in .terraform.lock.hcl; see the runbook. diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index a463532..4a3da7b 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -4,10 +4,15 @@ # stack images, which the smoke job pulls anyway. name: CI +# Prose changes skip the whole workflow: nothing here reads Markdown. infra/ +# has its own workflow (infra.yml) that only runs when infra/ changes. on: push: branches: [main] + paths-ignore: &docs ["**.md", "docs/**", "LICENSE", "infra/**"] pull_request: + paths-ignore: *docs + workflow_dispatch: permissions: contents: read @@ -31,7 +36,6 @@ jobs: - uses: extractions/setup-just@53165ef7e734c5c07cb06b3c8e7b647c5aa16db3 # v4.0.0 - run: cp .env.example .env - run: just lint - - run: just infra-validate smoke: runs-on: ubuntu-latest diff --git a/.github/workflows/infra.yml b/.github/workflows/infra.yml new file mode 100644 index 0000000..e12ed55 --- /dev/null +++ b/.github/workflows/infra.yml @@ -0,0 +1,28 @@ +# OpenTofu validation for infra/, on its own trigger: it downloads the +# provider every run and nothing outside infra/ can change its result. +name: infra + +on: + push: + branches: [main] + paths: &infra ["infra/**"] + pull_request: + paths: *infra + workflow_dispatch: + +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: true + +jobs: + validate: + runs-on: ubuntu-latest + timeout-minutes: 10 + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + - uses: extractions/setup-just@53165ef7e734c5c07cb06b3c8e7b647c5aa16db3 # v4.0.0 + - run: cp .env.example .env + - run: just infra-validate diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index eb124cb..577a338 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -1,10 +1,10 @@ # Installed by `just hooks` (prek). Stages sized to their cost: -# pre-commit gitleaks on the staged diff (~0.5s; a secret is cheapest to -# catch before it is in history), then `just lint` (~6s warm). +# pre-commit gitleaks on the staged diff, ~0.5s: a secret is cheapest to +# catch before it is in history. Nothing slower runs per commit. # commit-msg Conventional Commits shape on the first line. -# pre-push `just validate` (the stack's images; seconds on a host that -# runs the stack) and `just infra-validate`, only when infra/ -# is in the push: it downloads the provider every time. +# pre-push `just check` (lint + validate; ~10s warm on a host that runs +# the stack) and `just infra-validate`, only when infra/ is in +# the push: it downloads the provider every time. # `prek run` runs the pre-commit stage on demand; `--hook-stage pre-push` the rest. repos: - repo: local @@ -19,13 +19,6 @@ repos: pass_filenames: false always_run: true stages: [pre-commit] - - id: lint - name: just lint - entry: just lint - language: system - pass_filenames: false - always_run: true - stages: [pre-commit] - id: commit-msg name: conventional commit message entry: >- @@ -35,9 +28,9 @@ repos: || { echo "commit message must be type(scope): lower-case summary, at most 72 chars" >&2; exit 1; }' -- language: system stages: [commit-msg] - - id: validate - name: just validate - entry: just validate + - id: check + name: just check + entry: just check language: system pass_filenames: false always_run: true diff --git a/config/grafana/alerting/rules.yaml b/config/grafana/alerting/rules.yaml index 55bcd28..81225db 100644 --- a/config/grafana/alerting/rules.yaml +++ b/config/grafana/alerting/rules.yaml @@ -204,7 +204,9 @@ groups: labels: severity: warning annotations: - summary: "Filesystem {{ $labels.mountpoint }} on {{ or $labels.host_name $labels.instance }} fills up within 3 days at the current rate" + summary: >- + Filesystem {{ $labels.mountpoint }} on {{ or $labels.host_name $labels.instance }} + fills up within 3 days at the current rate description: >- Extrapolated from the last 6 hours. Find what is growing (a spoke shipping more than before, a log loop, a backup that stopped diff --git a/justfile b/justfile index 0700ed4..2bc899d 100644 --- a/justfile +++ b/justfile @@ -15,7 +15,8 @@ shellcheck := "koalaman/shellcheck:v0.11.0" ruff := "ghcr.io/astral-sh/ruff:0.14.2" tofu := "ghcr.io/opentofu/opentofu:1.12.3" gitleaks := "zricethezav/gitleaks:v8.30.1@sha256:c00b6bd0aeb3071cbcb79009cb16a60dd9e0a7c60e2be9ab65d25e6bc8abbb7f" -lint_images := jq + " " + yamllint + " " + yamlfmt + " " + actionlint + " " + shellcheck + " " + ruff + " " + tofu + " " + gitleaks +alloy := "grafana/alloy:v1.18.1@sha256:0f4434c92b3e6cdac38bb129b344e1790c246f7b6e2eaffcc16a5fa363240e33" +lint_images := jq + " " + yamllint + " " + yamlfmt + " " + actionlint + " " + shellcheck + " " + ruff + " " + tofu + " " + gitleaks + " " + alloy # `demo` and `smoke` each bring up a throwaway copy of the core stack, so both # run under their own compose project and their own Grafana port (see @@ -133,7 +134,7 @@ tail service: # Validate every config in the repo. check: lint validate -# Static checks in small tool images (~130 MB cold, seconds warm). +# Static checks in tool images (~280 MB cold, seconds warm). lint: @printf '%s\n' {{lint_images}} | xargs -P 8 -n 1 docker pull -q >/dev/null # `-f compose.yml` on the first line, not the host's COMPOSE_FILE: on a @@ -184,6 +185,11 @@ lint: # working-tree scan flags .env's real tokens and fails on every dev machine. # Needs full history: a shallow CI clone has one commit and passes vacuously. docker run --rm --network none -v .:/repo:ro {{gitleaks}} git --redact --no-banner /repo + # The one config vendored verbatim onto every project host; a syntax error + # here otherwise first surfaces as a crash-looping agent on a client machine. + # Image ref matches templates/compose.telemetry.yml; keep them in step. In + # `lint`, not `validate`: the smoke job pulls nothing else this size. + docker run --rm --network none -v ./templates/alloy/config.alloy:/etc/alloy/config.alloy:ro -e COMPOSE_PROJECT_NAME=dummy -e ENVIRONMENT=dummy -e PROJECT=dummy -e OTEL_EXPORTER_OTLP_ENDPOINT=https://dummy -e OTLP_AUTH_TOKEN=dummy -e TELEMETRY_EDGE_KEY= {{alloy}} validate /etc/alloy/config.alloy # Image ref of one compose.yml service, so the validators below can't drift # from the versions the stack runs. (`config --images ` also lists the @@ -191,17 +197,14 @@ lint: _image service: @docker compose -f compose.yml config --format json | docker run --rm -i {{jq}} -er '.services["{{service}}"].image // error("no service {{service}} in compose.yml")' -# The validators run in the images the stack itself runs. ~300 MB cold. +# The validators run in the images the stack itself runs. ~290 MB cold, but +# the smoke job has them anyway. # Run each config through the binary that will load it. validate: docker run --rm --network none -v ./config/prometheus.yaml:/etc/prometheus/prometheus.yaml:ro --entrypoint promtool $(just _image prometheus) check config /etc/prometheus/prometheus.yaml docker run --rm --network none -e OTLP_AUTH_TOKEN=dummy -e DEPARTMENT=dummy -v ./config/otel-collector.yaml:/etc/otelcol/config.yaml:ro $(just _image otel-collector) validate --config=/etc/otelcol/config.yaml docker run --rm --network none -v ./config/loki.yaml:/etc/loki/loki.yaml:ro $(just _image loki) -config.file=/etc/loki/loki.yaml -verify-config docker run --rm --network none -v ./config/tempo.yaml:/etc/tempo/tempo.yaml:ro $(just _image tempo) -config.file=/etc/tempo/tempo.yaml -config.verify=true - # The one config vendored verbatim onto every project host; a syntax error - # here otherwise first surfaces as a crash-looping agent on a client machine. - # Image ref matches templates/compose.telemetry.yml; keep them in step. - docker run --rm --network none -v ./templates/alloy/config.alloy:/etc/alloy/config.alloy:ro -e COMPOSE_PROJECT_NAME=dummy -e ENVIRONMENT=dummy -e PROJECT=dummy -e OTEL_EXPORTER_OTLP_ENDPOINT=https://dummy -e OTLP_AUTH_TOKEN=dummy -e TELEMETRY_EDGE_KEY= grafana/alloy:v1.18.1@sha256:0f4434c92b3e6cdac38bb129b344e1790c246f7b6e2eaffcc16a5fa363240e33 validate /etc/alloy/config.alloy # Runs against a copy of the sources only: state and tfvars never enter the # container, which needs network access to fetch the provider. @@ -214,7 +217,7 @@ infra-validate: fmt: docker run --rm --network none --user "$(id -u):$(id -g)" -v .:/code -w /code {{yamlfmt}} . -# Install the git hooks: gitleaks + lint on commit, validate on push (see .pre-commit-config.yaml). +# Install the git hooks: gitleaks on commit, `just check` on push (see .pre-commit-config.yaml). hooks: prek install --hook-type pre-commit --hook-type commit-msg --hook-type pre-push From a3e45a5326d3ebc702266463538c6c8a5b590bf2 Mon Sep 17 00:00:00 2001 From: Simon van Lierde Date: Sun, 6 Sep 2026 17:00:24 +0000 Subject: [PATCH 53/85] ci: run on pull requests only; main advances through them --- .github/workflows/ci.yml | 11 +++++------ .github/workflows/infra.yml | 5 +---- 2 files changed, 6 insertions(+), 10 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 4a3da7b..f2843c5 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -4,14 +4,13 @@ # stack images, which the smoke job pulls anyway. name: CI -# Prose changes skip the whole workflow: nothing here reads Markdown. infra/ -# has its own workflow (infra.yml) that only runs when infra/ changes. +# Pull requests only: main moves through PRs (the protect-main ruleset +# requires one), so a push run would repeat the check that passed on the PR +# head. Prose changes skip the whole workflow: nothing here reads Markdown. +# infra/ has its own workflow (infra.yml) that only runs when infra/ changes. on: - push: - branches: [main] - paths-ignore: &docs ["**.md", "docs/**", "LICENSE", "infra/**"] pull_request: - paths-ignore: *docs + paths-ignore: ["**.md", "docs/**", "LICENSE", "infra/**"] workflow_dispatch: permissions: diff --git a/.github/workflows/infra.yml b/.github/workflows/infra.yml index e12ed55..871a4c5 100644 --- a/.github/workflows/infra.yml +++ b/.github/workflows/infra.yml @@ -3,11 +3,8 @@ name: infra on: - push: - branches: [main] - paths: &infra ["infra/**"] pull_request: - paths: *infra + paths: ["infra/**"] workflow_dispatch: permissions: From f54507bbc54f2237de679691e8f8950880676e2d Mon Sep 17 00:00:00 2001 From: Simon van Lierde Date: Sun, 6 Sep 2026 17:05:28 +0000 Subject: [PATCH 54/85] docs: describe the pull-request-only CI --- CHANGELOG.md | 3 ++- README.md | 3 ++- 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 7ad6e3a..1c06085 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -43,7 +43,8 @@ queries that had been measuring the wrong thing. `just infra-validate`; `lint` also enforces yamlfmt and ruff formatting. `check` splits into `lint` (static) and `validate` (the stack's own images); CI runs the two on separate jobs so a lint failure never waits on the stack - images. `just hooks` installs git hooks via prek: gitleaks on the staged + images, on pull requests only (`main` requires one), skipping prose-only + changes; `infra/` validates on its own workflow when `infra/` changes. `just hooks` installs git hooks via prek: gitleaks on the staged diff and `lint` at commit, a Conventional Commits check on the message, `validate` at push, `infra-validate` at push when `infra/` changed. - **`just smoke` boots the production shape and asserts the data paths**: JWT diff --git a/README.md b/README.md index e659c44..a617e05 100644 --- a/README.md +++ b/README.md @@ -134,7 +134,8 @@ Loki with the promoted project/env/department labels. The assertions live in its own port, so it cannot disturb a stack already running on the host: `just smoke` is safe on the production box, `just restore-check` rehearses backup and restore on its volumes, and `just smoke-down` cleans up. CI runs -`lint` on one job and `validate` plus `smoke` on another, on every push and pull request. +`lint` on one job and `validate` plus `smoke` on another, on every pull +request; `main` only moves through pull requests, so nothing runs twice. ## Sending telemetry from a project From d4c2726f28d9e72e41d7ebbd7546d859461ee100 Mon Sep 17 00:00:00 2001 From: Simon van Lierde Date: Sun, 6 Sep 2026 17:14:42 +0000 Subject: [PATCH 55/85] docs(changelog): consolidate the unreleased notes into 1.0.0 --- CHANGELOG.md | 261 +++++++++++++++++++++------------------------------ 1 file changed, 105 insertions(+), 156 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 1c06085..cd525c5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,178 +4,127 @@ Notable changes to this stack. Format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/); versions follow [SemVer](https://semver.org/). -## [Unreleased] - -A hardening pass over the whole stack: buffering that survives a restart, -self-monitoring that covers every service, tighter container defaults, and -queries that had been measuring the wrong thing. +## [1.0.0] - 2026-09-06 + +The hub now runs the department's telemetry in production, and the contract a +spoke vendors (hostname, labels, templates pinned by tag) is the one we intend +to keep. This release hardens the stack around that contract and makes the +checks prove the data paths, not just the config syntax. + +### Upgrade + +- Point every spoke's `OTEL_EXPORTER_OTLP_ENDPOINT` at `otel.`. The + old `otlp.` name is gone, and so is any edge rule that matched it. +- Set `DEPARTMENT` in the hub's `.env`. The collector stamps it on every + signal; the alert rules and dashboards key on it. +- Set `COMPOSE_FILE` in `.env` to name the overlays this host runs + (`compose.yml:compose.tunnel.yml` in production). `just up-tunnel` is gone; + `just up` runs the exposure guards whenever the tunnel overlay is active. +- Remove the orphaned `alertmanager_data` volume when convenient. Alerting is + Grafana-managed (ADR 0002); delivery still posts to `ALERT_WEBHOOK_URL`. +- Re-vendor the templates on each spoke at `v1.0.0` (`bootstrap.sh` prints + the commands): the Alloy agent gains an in-pipeline memory limiter, and the + overlay declares the `egress` network it joins. +- Loki streams keep their old label set until they age out (30 days). ### Added -- **Durable export queue**: the collector's send queues are file-backed on a - new `otel_queue` volume, so telemetry buffered during a backend outage - survives a collector restart. `just up` (and `demo`, `smoke`) prepares the - volume's ownership. `just backup` skips the volume, whose contents are - worthless by the time anyone restores. -- **Full-stack scraping**: Prometheus now scrapes Grafana, Loki, and Tempo as - well, so `TargetDown` covers every service. -- **Early storage warnings**: `HostDiskFilling` (a filesystem is full within - 3 days at the current rate) and `PrometheusCardinalityHigh` (over 100k - active series) fire while there is still time, ahead of the 80% backstop. -- **Per-user Grafana logins**, optional: `GRAFANA_JWT_AUTH` plus - `CF_ACCESS_TEAM_DOMAIN` and `CF_ACCESS_AUD` make Grafana verify the - Cloudflare Access JWT (pinned to this app's `aud` tag, with both values - enforced by the exposure guards) instead of everyone sharing the admin - password. -- **Memory ceilings** (`mem_limit`) on every service, sized from observed - usage, so one runaway component cannot OOM the host. The spoke Alloy agent - gains a matching in-pipeline memory limiter, so a long hub outage sheds load - instead of OOM-killing the agent and its loss counters with it. -- **Tighter container defaults**: every service drops all capabilities, - node-exporter (which holds `pid: host` and the host filesystem) runs - read-only with a pids limit, and the hub images are digest-pinned like the - client templates. The tunnel token reaches cloudflared via environment, not - argv. -- **Wider validation**: `just check` also verifies the Loki and Tempo configs, - the vendored Alloy config, shell scripts, the demo app's Python, OpenTofu - formatting, the datasource uids dashboards reference, the alert templates - `bootstrap.sh` renders, and git history for secrets; CI also runs - `just infra-validate`; `lint` also enforces yamlfmt and ruff formatting. - `check` splits into `lint` (static) and `validate` (the stack's own images); - CI runs the two on separate jobs so a lint failure never waits on the stack - images, on pull requests only (`main` requires one), skipping prose-only - changes; `infra/` validates on its own workflow when `infra/` changes. `just hooks` installs git hooks via prek: gitleaks on the staged - diff and `lint` at commit, a Conventional Commits check on the message, - `validate` at push, `infra-validate` at push when `infra/` changed. -- **`just smoke` boots the production shape and asserts the data paths**: JWT - auth on with a forged Access token refused, dashboards and alert rules - compared uid for uid (not by count) with no rule paused, contact points - checked against the exact URLs given, every scrape target up, and a metric - and a log round-tripped through the collector into Prometheus and Loki with - the promoted `project`/`env`/`department` labels. The assertions moved to - `scripts/smoke.sh` (functions, shellcheck). `just restore-check` rehearses - backup and restore on the smoke volumes. `just demo-build` is gone: the - demo is a local fixture (`just demo`), not a CI step. -- `just fmt` runs yamlfmt in a container like every other tool. -- **Isolated smoke and demo**: each runs under its own compose project and - Grafana port (`compose.sandbox.yml`), so neither can adopt or recreate a - stack already running on the host. -- `bootstrap.sh` refuses to run from a release tag that lacks `templates/` - and prints `sha256sum -c` lines for the files it tells a project host to - vendor. -- Dependabot now watches the Cloudflare provider in `infra/`, and the runbook - covers OpenTofu-managed tunnel and Access changes. -- **`infra/generate-imports.sh`**: emits OpenTofu `import` blocks for the edge - built by hand in the Zero Trust dashboard. Without it the first plan against - an empty state reads "create" for objects already serving traffic, and - applying it mints a second tunnel and a duplicate Access app. +- **Per-user Grafana logins** through Cloudflare Access: `GRAFANA_JWT_AUTH` + with `CF_ACCESS_TEAM_DOMAIN` and `CF_ACCESS_AUD`. The exposure guards + enforce the pair, and the smoke test boots with it on. +- **Durable export queue**: the collector's send queues live on a file-backed + `otel_queue` volume, so telemetry buffered during an outage survives a + collector restart. +- **Self-monitoring covers every service**: Prometheus scrapes Grafana, Loki + and Tempo too, so `TargetDown` sees them. `HostDiskFilling` (full within 3 + days at the current rate) and `PrometheusCardinalityHigh` (over 100k active + series) warn ahead of the 80% disk backstop. +- **`infra/generate-imports.sh`** emits OpenTofu `import` blocks for the + tunnel, DNS records and Access app built by hand, so the first plan does + not create duplicates of objects already serving traffic. +- **`just smoke` proves the stack works, not that it boots**: every dashboard + and alert rule provisioned, uid for uid, none paused; contact points carry + the exact URLs given; Grafana honours the JWT settings and refuses a forged + token; every scrape target is up; a metric and a log round-trip through + the collector into Prometheus and Loki with the `project`, `env` and + `department` labels. `just restore-check` rehearses backup and restore on + the smoke volumes. +- **`just check` is `lint` plus `validate`**. `lint` covers compose files, + the rendered alert templates, YAML and its formatting, workflows, shell + scripts, the demo's Python, OpenTofu formatting, dashboard JSON and the + datasource uids it names, and git history for secrets. `validate` runs each + stack config through the exact image the stack uses. `just hooks` installs + git hooks via prek: gitleaks at commit, a Conventional Commits check on the + message, `check` at push. +- Dependabot watches the spoke images in `templates/` and the Cloudflare + provider in `infra/`; patch bumps arrive grouped, the demo's advisories in + one PR. ### Changed -- **Ingestion hostname is `otel.`**, not `otlp.`: one - department-wide name for machine telemetry alongside `grafana.` for - humans. Every spoke's `OTEL_EXPORTER_OTLP_ENDPOINT` and any edge rule matching - the old host have to follow. -- **`department` on every signal**: the gateway collector stamps it from - `DEPARTMENT` in `.env` and Prometheus and Loki carry it as an identity label. - It is set at the hub rather than by the sender, so a spoke cannot ship - telemetry attributed to someone else. -- **Overlays are host config now**: `COMPOSE_FILE` in `.env` names the compose - file set, and every recipe (`up`, `logs`, `ps`, `backup`) acts on that same - set. `just up-tunnel` is gone; its exposure guards run automatically - whenever the tunnel overlay is active. - **Loki indexes only the identity labels** (`service.name`, `department`, - `project`, `env`, `host.name`; the authoritative list lives in - `config/loki.yaml`). Everything else, `service.instance.id` included, is - structured metadata now: one stream per service instead of one per sender - restart. Existing streams keep their old labels until they age out (30 days). -- **Dashboards are provisioned, not editable**: `dashboards/*.json` is mounted - read-only and UI saves are off, making the files the source of truth. -- The collector's `memory_limiter` is sized in absolute MiB against the - container limit, and the queue and retry settings behind the runbook's - "buffers for five minutes" are pinned rather than inherited from upstream. -- **Tempo's metrics generator is removed**: RED comes from the applications' - own OTLP metrics now (ADR 0002), so Tempo stores traces and nothing else. -- Dropped the `relab-api` dashboard: the `$service` picker on Service Health - and Logs covers it. -- README rewritten for a broader CML audience. + `project`, `env`, `host.name`; the list lives in `config/loki.yaml`). + Everything else is structured metadata, so a sender restart no longer + mints a new stream. +- **Dashboards are provisioned, not editable**: `dashboards/*.json` is the + source of truth; UI saves are off. +- **Tempo stores traces and nothing else**: its metrics generator is gone. + RED comes from the applications' own OTLP metrics (ADR 0002). +- CI runs on pull requests only (`main` requires one), skips prose-only + changes, and validates `infra/` on its own workflow. `lint` and + `validate` + `smoke` run on separate jobs, so a lint failure never waits + on the stack images. `just demo-build` is gone; the demo is a local + fixture. +- `just smoke` and `just demo` each run under their own compose project and + Grafana port (`compose.sandbox.yml`), so neither can touch a running stack. - Image bumps: Grafana 13.1.4, Loki 3.7.6, Tempo 3.0.3, Prometheus 3.13.2, the collector 0.156.0, node-exporter 1.12.1, cloudflared 2026.8.2, and the - demo's Python dependencies. - -### Removed - -- **Alertmanager**: alerting is Grafana-managed now (ADR 0002). Rules are - provisioned from `config/grafana/alerting/`, and delivery still posts to - `ALERT_WEBHOOK_URL`. Its `alertmanager_data` volume is left behind on an - upgraded host; the runbook says when to remove it. -- The commented `compose.storage-s3.yml` stub: the S3 escape hatch lives as - an appendix of ADR 0001 instead. + demo's Python dependencies. The `relab-api` dashboard is dropped; the + `$service` picker on Service Health covers it. ### Fixed -- `just backup` fails (instead of exiting 0) when the unpause after the copy - fails and the stack is left paused. `just lint`'s exposure-guard self-test - lost the status of its positive half; a guard that rejected every valid - `.env` passed CI. The smoke provisioning check could pass with an empty - dashboard list if reading the uids failed. `bootstrap.sh` printed the hash - of empty input for a template missing from the tag, and re-reads its rule - from Grafana after the restart instead of trusting the restart. - `infra/generate-imports.sh` reported a failed DNS API call as "no record". -- **Spoke overlay declared no `egress` network**: `compose.telemetry.yml` - joined it without defining it, so the documented `up -d` failed on any host - whose own compose file did not happen to name one. `just check` now renders - both spoke overlays. -- **GPU dashboard host picker keyed on `instance`**, which is the same - in-container address on every host; it uses `host_name` like the rest. - Dashboard variables refresh on load, not on every 30s tick. -- **Access app import id**: `generate-imports.sh` emitted it without the - `accounts/` scope the 5.x provider requires. -- **`just smoke` proves delivery and the data path**: contact points must be - expanded (no literal `$VAR`), and one OTLP log through the collector's - bearer auth must come back out of Loki carrying the `department` label. -- **Trace links from the latency panel**: the Prometheus datasource pointed - exemplars at a `trace_id` label, but span-metrics exemplars carry `traceID`, - so clicking a dot resolved to nothing. -- **`HighErrorRate` measured the wrong denominator**: it counted all spans, so - internal child spans diluted the ratio well below the real request error - rate. It now reads the applications' HTTP server metrics, matching the - Service Health dashboard. -- The demo load generator hit a never-failing endpoint half the time, so the - advertised one-in-ten error rate showed up as one in twenty. -- The Infrastructure Logs dashboard queried `env` and `service` labels this - stack does not set, and so was always empty. -- Grafana and the collector now wait for Prometheus to report ready, instead - of racing it on a cold start. -- The S3 overlay documents the `-config.expand-env=true` that Loki and Tempo - need before `${...}` in their configs expands at all. -- **`HighErrorRate` merged environments**: aggregating by job alone let a - healthy prod service dilute a broken staging one sharing the job name below - the threshold. It now keys on job, project and env like the other - multi-tenant rules. `HostDiskSpaceLow` says whose disk is filling. -- With `GRAFANA_JWT_AUTH=true` but no team domain set, Grafana fetched its - JWT signing keys from a placeholder `cloudflareaccess.com` subdomain any - Cloudflare customer could claim; the fallback is gone and the guards refuse - to start without the real values. +- **`HighErrorRate` measured the wrong thing twice**: it counted all spans, + so child spans diluted the ratio; and it aggregated by job alone, so a + healthy prod service masked a broken staging one. It reads the HTTP server + metrics now, keyed on job, project and env. +- Trace links from the latency panel resolved to nothing: exemplars carry + `traceID`, the datasource looked for `trace_id`. +- The Infrastructure Logs dashboard queried labels this stack never set. The + GPU dashboard's host picker keyed on `instance`, identical on every host. +- The spoke overlay joined an `egress` network it never declared, so the + documented `up -d` failed on a host without one. `just lint` renders both + spoke overlays now. +- With `GRAFANA_JWT_AUTH=true` and no team domain, Grafana fetched signing + keys from a placeholder `cloudflareaccess.com` subdomain any customer could + claim. There is no fallback now, and the guards refuse to start. +- Checks that could pass without checking: the exposure-guard self-test + dropped the status of its positive half; the smoke provisioning check + accepted an empty dashboard list; `just backup` exited 0 with the stack + still paused when the unpause failed; `bootstrap.sh` printed the hash of + empty input for a template missing from the tag; `generate-imports.sh` + reported a failed DNS API call as "no record". `bootstrap.sh` now reads + its rule back from Grafana after the restart instead of trusting it. +- Grafana and the collector wait for Prometheus to report ready instead of + racing it on a cold start. The demo load generator's error rate matched + its advertised one in ten. ### Security -- `no-new-privileges` on every service; the demo image runs as `nobody`. -- `GRAFANA_COOKIE_SECURE` marks the session cookie Secure (with strict - SameSite), and `just up` with the tunnel overlay refuses to expose the stack - without it, an `https://` `GRAFANA_ROOT_URL`, and non-default credentials. - `just check` runs the guards both ways, so a guard that silently accepts a - default fails CI rather than the production start. +- Every service drops all capabilities, runs with `no-new-privileges`, and + carries `mem_limit` and `pids_limit` sized from observed usage. + node-exporter and cloudflared run read-only; the demo runs as `nobody`. - Loki, Tempo, Prometheus and node-exporter sit on an internal `backend` - network only Grafana and the collector join. cloudflared stays on `default`, - so an ingress edited in the Cloudflare dashboard cannot reach a backend that - has no authentication of its own. -- Every hub service carries a `pids_limit`; Dependabot also watches the spoke - images pinned in `templates/`. Patch bumps are grouped per directory and - the demo's dependencies into one monthly PR; minors and majors stay one - per PR. -- GitHub Actions are pinned to commit SHAs, and `just infra-validate` runs - against a copy of the sources so state and tfvars never enter the container. + network only Grafana and the collector join. cloudflared stays outside it, + so an ingress edited in the Cloudflare dashboard cannot reach a backend + with no authentication of its own. +- `GRAFANA_COOKIE_SECURE` marks the session cookie Secure with strict + SameSite. With the tunnel overlay, `just up` refuses to start without it, + an `https://` root URL, non-default credentials, and a webhook URL; + `just lint` runs those guards both ways. +- Hub images are digest-pinned, GitHub Actions are pinned to commit SHAs, the + tunnel token reaches cloudflared via environment rather than argv, and + `bootstrap.sh` prints `sha256sum -c` lines for the files a spoke vendors. ## [0.2.0] - 2026-07-05 From 5f8f83e014ae3b025c896cc23421d4807d444cec Mon Sep 17 00:00:00 2001 From: Simon van Lierde Date: Sun, 6 Sep 2026 17:21:30 +0000 Subject: [PATCH 56/85] fix(security): keep the ingest token off argv, guard bootstrap's Grafana restart - smoke.sh passes OTLP_AUTH_TOKEN to wget over stdin; on the hub it is the production token and argv is readable by any local user - bootstrap.sh runs the exposure guards before recreating Grafana, which on a tunnel host is an exposed service - the smoke stack's Access team domain is smoke.invalid, a name no Zero Trust team can register - runbook: backups contain the webhook and heartbeat URLs via Grafana's database --- bootstrap.sh | 3 +++ docs/RUNBOOK.md | 8 +++++--- justfile | 6 ++++-- scripts/smoke.sh | 7 +++++-- 4 files changed, 17 insertions(+), 7 deletions(-) diff --git a/bootstrap.sh b/bootstrap.sh index 8c9cfef..af4b9a0 100755 --- a/bootstrap.sh +++ b/bootstrap.sh @@ -103,6 +103,9 @@ echo "rendered ${out_dir}/coverage.yaml (covering: ${covered})" # ------------------------------------------------------------------ 3. reload Grafana if docker compose ps --status running --services 2>/dev/null | grep -qx grafana; then + # Same guards as `just up`: with the tunnel overlay in COMPOSE_FILE this recreates + # an exposed Grafana, and a drifted .env must refuse here too. + just _guard-if-exposed || exit 1 # A restart, not SIGHUP: Grafana re-reads alert provisioning only at startup, and # a SIGHUP reports success while changing nothing. docker compose up -d --force-recreate grafana >/dev/null 2>&1 \ diff --git a/docs/RUNBOOK.md b/docs/RUNBOOK.md index 10aa66a..f68fc15 100644 --- a/docs/RUNBOOK.md +++ b/docs/RUNBOOK.md @@ -115,9 +115,11 @@ host is the intended setup. ### Where every secret lives -Three files hold everything, all gitignored, none backed up by `just backup`. -Copy `.env` and `infra/terraform.tfstate` off-host together with the backups -and treat the copies the way you treat the originals. +Three files hold everything, all gitignored. `just backup` archives none of +them, but the tarball does contain Grafana's database, and Grafana stores the +expanded contact points there: the webhook and heartbeat URLs are inside every +backup. Copy `.env` and `infra/terraform.tfstate` off-host together with the +backups and treat all three the way you treat the originals. | Secret | Lives in | Comes from | | --- | --- | --- | diff --git a/justfile b/justfile index 2bc899d..399f261 100644 --- a/justfile +++ b/justfile @@ -37,10 +37,12 @@ core_project := env("COMPOSE_PROJECT_NAME", "monitoring") # demo stack can also coexist. The smoke stack boots in production shape: # Grafana with JWT auth on (the Cloudflare Access path, otherwise only ever # schema-checked) and fixed notification URLs so the contact-point assertion -# can check exact values instead of "not empty". +# can check exact values instead of "not empty". The team domain has a dot, +# which no real Zero Trust team name can, so the JWK URL can never resolve to +# a team someone registers. smoke_project := "monitoring-smoke" smoke_port := env("SMOKE_PORT", "3001") -smoke_env := "SANDBOX_PORT=" + smoke_port + " GRAFANA_JWT_AUTH=true CF_ACCESS_TEAM_DOMAIN=smoke CF_ACCESS_AUD=smoke ALERT_WEBHOOK_URL=https://smoke.invalid/alerts HEARTBEAT_URL=https://smoke.invalid/heartbeat" +smoke_env := "SANDBOX_PORT=" + smoke_port + " GRAFANA_JWT_AUTH=true CF_ACCESS_TEAM_DOMAIN=smoke.invalid CF_ACCESS_AUD=smoke ALERT_WEBHOOK_URL=https://smoke.invalid/alerts HEARTBEAT_URL=https://smoke.invalid/heartbeat" compose_smoke := smoke_env + " docker compose -p " + smoke_project + " -f compose.yml -f compose.sandbox.yml" # dashboards/*.json as the paths they get when mounted at /dashboards. 2>/dev/null diff --git a/scripts/smoke.sh b/scripts/smoke.sh index c873e74..795c755 100755 --- a/scripts/smoke.sh +++ b/scripts/smoke.sh @@ -81,9 +81,12 @@ poll all_up || die "not all $n_jobs scrape targets are up: $(promq up | jq -r '. # with Grafana's wget: it is on the same network and needs no extra image. ts="$(date +%s)000000000" res='{"attributes":[{"key":"service.name","value":{"stringValue":"smoke"}},{"key":"project","value":{"stringValue":"smoke"}},{"key":"env","value":{"stringValue":"ci"}}]}' +# The token goes in on stdin: on the production host this is the real +# OTLP_AUTH_TOKEN (dotenv), and argv is world-readable in ps. otlp() { - docker compose -p "$project" exec -T grafana wget -qO- --header="Authorization: Bearer ${OTLP_AUTH_TOKEN}" \ - --header='Content-Type: application/json' --post-data="$2" "http://otel-collector:4318/v1/$1" >/dev/null + docker compose -p "$project" exec -T grafana sh -c \ + 'read -r t; wget -qO- --header="Authorization: Bearer $t" --header="Content-Type: application/json" --post-data="$1" "http://otel-collector:4318/v1/$2"' \ + _ "$2" "$1" <<<"$OTLP_AUTH_TOKEN" >/dev/null } otlp metrics '{"resourceMetrics":[{"resource":'"$res"',"scopeMetrics":[{"metrics":[{"name":"smoke_up","gauge":{"dataPoints":[{"asInt":"1","timeUnixNano":"'"$ts"'"}]}}]}]}]}' \ || die "the collector refused an OTLP metric with the .env token" From ed76cef456c3e769357893e661b7758e3276c745 Mon Sep 17 00:00:00 2001 From: Simon van Lierde Date: Sun, 6 Sep 2026 17:29:17 +0000 Subject: [PATCH 57/85] docs(changelog): release as 0.3.0, not 1.0.0 --- CHANGELOG.md | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index cd525c5..ade8c3c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,12 +4,13 @@ Notable changes to this stack. Format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/); versions follow [SemVer](https://semver.org/). -## [1.0.0] - 2026-09-06 +## [0.3.0] - 2026-09-06 -The hub now runs the department's telemetry in production, and the contract a -spoke vendors (hostname, labels, templates pinned by tag) is the one we intend -to keep. This release hardens the stack around that contract and makes the -checks prove the data paths, not just the config syntax. +The hub runs the department's telemetry in production now, with a spoke +contract (hostname, labels, templates pinned by tag) that one project uses. +This release hardens the stack around that contract and makes the checks +prove the data paths, not just the config syntax. 1.0 waits until a second +consumer has confirmed the contract. ### Upgrade @@ -22,7 +23,7 @@ checks prove the data paths, not just the config syntax. `just up` runs the exposure guards whenever the tunnel overlay is active. - Remove the orphaned `alertmanager_data` volume when convenient. Alerting is Grafana-managed (ADR 0002); delivery still posts to `ALERT_WEBHOOK_URL`. -- Re-vendor the templates on each spoke at `v1.0.0` (`bootstrap.sh` prints +- Re-vendor the templates on each spoke at `v0.3.0` (`bootstrap.sh` prints the commands): the Alloy agent gains an in-pipeline memory limiter, and the overlay declares the `egress` network it joins. - Loki streams keep their old label set until they age out (30 days). From be203643ef083f2d0907ec909147009df53d0329 Mon Sep 17 00:00:00 2001 From: Simon van Lierde Date: Sun, 6 Sep 2026 17:32:14 +0000 Subject: [PATCH 58/85] docs(infra): the ingestion rename is applied; say so in the import script --- infra/generate-imports.sh | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/infra/generate-imports.sh b/infra/generate-imports.sh index 8277d06..bff8a44 100755 --- a/infra/generate-imports.sh +++ b/infra/generate-imports.sh @@ -7,11 +7,11 @@ # the live ones. Import blocks make the adoption reviewable: read the generated file, # then the plan, and only then apply. # -# The ingestion record is the one asymmetry. Live, it is still `otlp.`; this -# root now calls it `otel.`. The block below imports that record as -# `cloudflare_dns_record.otel`, so the apply RENAMES the record in place instead of -# creating a second one. That is also why main.tf has no `moved` block: nothing was ever -# in state under the old resource name. +# The ingestion record was renamed from `otlp.` to `otel.` (2026-09). +# On a state that predates the rename, the block below imports the live record as +# `cloudflare_dns_record.otel`, so the apply RENAMES it in place instead of creating a +# second one. That is also why main.tf has no `moved` block: nothing was ever in state +# under the old resource name. A current state already holds `otel.`. # # The generated imports.tf is a throwaway, NOT something to commit: it names one # account's resource ids and is meaningless after the apply that consumes it. Delete it From 7ef9e20a3c526a752c40858ca98163746289fe6b Mon Sep 17 00:00:00 2001 From: Simon van Lierde Date: Sun, 6 Sep 2026 17:50:35 +0000 Subject: [PATCH 59/85] docs: update repo documentation --- .env.example | 39 +-- .github/dependabot.yml | 16 +- .github/workflows/ci.yml | 21 +- .pre-commit-config.yaml | 11 +- CHANGELOG.md | 65 ++--- README.md | 209 +++++++------- bootstrap.sh | 85 ++---- compose.demo.yml | 13 +- compose.sandbox.yml | 14 +- compose.tunnel.yml | 14 +- compose.yml | 76 ++--- config/grafana/alerting/contact-points.yaml | 15 +- config/grafana/alerting/coverage.yaml | 14 +- .../alerting/notification-policies.yaml | 6 +- .../alerting/project-relab-staging.yaml | 20 +- config/grafana/alerting/rules.yaml | 48 ++-- config/grafana/datasources.yaml | 8 +- config/loki.yaml | 20 +- config/otel-collector.yaml | 32 +-- config/prometheus.yaml | 15 +- config/tempo.yaml | 12 +- demo/app.py | 11 +- docs/ONBOARDING.md | 64 ++--- docs/RUNBOOK.md | 266 ++++++++---------- infra/generate-imports.sh | 66 ++--- infra/main.tf | 22 +- justfile | 167 ++++------- scripts/smoke.sh | 42 +-- templates/README.md | 161 +++++------ templates/alerting/coverage.yaml.tmpl | 14 +- templates/alerting/project.yaml.tmpl | 20 +- templates/alloy/config.alloy | 165 ++++------- templates/compose.telemetry.gpu.yml | 30 +- templates/compose.telemetry.yml | 116 +++----- templates/run_scheduled.sh | 55 ++-- 35 files changed, 741 insertions(+), 1211 deletions(-) diff --git a/.env.example b/.env.example index d590426..fc2a252 100644 --- a/.env.example +++ b/.env.example @@ -1,8 +1,7 @@ # Copy to .env and fill in. The .env file is gitignored. -# Which compose overlays this host runs. Compose and every `just` recipe read -# it, so `just up`, `logs`, `ps`, `backup` all act on the same set. Unset = -# the core stack only. +# Which compose overlays this host runs; every `just` recipe acts on the same +# set. Unset = the core stack only. # production: COMPOSE_FILE=compose.yml:compose.tunnel.yml # local demo: COMPOSE_FILE=compose.yml:compose.demo.yml #COMPOSE_FILE= @@ -13,14 +12,12 @@ GRAFANA_ADMIN_PASSWORD=change-me # GRAFANA_ROOT_URL=https://grafana.example.com GRAFANA_ROOT_URL=http://localhost:3000 -# Set to true whenever GRAFANA_ROOT_URL is https (i.e. behind the tunnel). -# Marks the session cookie Secure; with the tunnel overlay in COMPOSE_FILE, -# `just up` refuses to run without it. -# Keep false for plain-http localhost use, or logins break. +# Set to true whenever GRAFANA_ROOT_URL is https; `just up` with the tunnel +# overlay refuses to run without it. Keep false for plain-http localhost use, +# or logins break. GRAFANA_COOKIE_SECURE=false -# Department this stack collects for. Stamped onto every signal as the `department` -# label, so telemetry stays attributable once a second department runs its own stack. +# Stamped onto every signal as the `department` label. DEPARTMENT=cml # Bearer token every telemetry sender must present (Authorization: Bearer ). @@ -29,32 +26,26 @@ DEPARTMENT=cml OTLP_AUTH_TOKEN=local-dev-token # Where Grafana delivers alert notifications (any webhook: ntfy, Slack, …). -# Not optional. An empty value makes every alert fail delivery silently. The -# heartbeat below keeps pinging, so the dead man's switch reads healthy while -# nothing reaches anyone. With the tunnel overlay active, `just up` refuses to -# start without it. +# Not optional: an empty value drops every alert silently while the heartbeat +# below keeps reporting healthy. With the tunnel overlay active, `just up` +# refuses to start without it. ALERT_WEBHOOK_URL= # Dead man's switch ping target (e.g. https://hc-ping.com/). The Watchdog # alert posts here every 5m; alert externally when pings stop. HEARTBEAT_URL= -# Optional. Lets `./bootstrap.sh ` create a project's healthchecks.io -# checks and print their ping URLs instead of telling you to create them by hand. -# Must be the project's READ-WRITE API key (healthchecks.io -> Project Settings -> API -# keys): the read-only key cannot create checks, and a ping key only sends pings. +# Optional. Lets `./bootstrap.sh` create a project's healthchecks.io checks. +# Must be the project's READ-WRITE API key (Project Settings -> API keys). HEALTHCHECKS_API_KEY= -# Only needed with the tunnel overlay (production exposure via Cloudflare Tunnel). -# From: Cloudflare Zero Trust → Networks → Tunnels → → Configure → token +# Only needed with the tunnel overlay: cd infra && tofu output -raw tunnel_token CLOUDFLARE_TUNNEL_TOKEN= -# Set to true to let Cloudflare Access sign users into Grafana individually, -# instead of everyone sharing the admin login. Needs both values below; with -# the tunnel overlay active, `just up` refuses to start without them. +# Set to true to let Cloudflare Access sign users into Grafana individually. +# Needs both values below; `just up` with the tunnel overlay enforces that. GRAFANA_JWT_AUTH=false # Your Zero Trust team name: the in https://.cloudflareaccess.com. CF_ACCESS_TEAM_DOMAIN= -# The Grafana Access application's aud tag, so tokens minted for other apps in -# the same team are rejected: cd infra && tofu output -raw grafana_access_aud +# The Grafana Access application's aud tag: cd infra && tofu output -raw grafana_access_aud CF_ACCESS_AUD= diff --git a/.github/dependabot.yml b/.github/dependabot.yml index bbe2252..f0e0153 100644 --- a/.github/dependabot.yml +++ b/.github/dependabot.yml @@ -1,9 +1,7 @@ version: 2 updates: - # Hub images: patch bumps ride together, minors and majors come one per PR. - # Patches rarely change config syntax, so one CI run covers a batch; a - # minor or major that fails `just check` or `just smoke` then names its own - # image instead of leaving six to bisect. + # Hub images: patch bumps ride together; a minor or major comes on its own, + # so a red PR names the one image that broke. - package-ecosystem: "docker-compose" directory: "/" schedule: @@ -12,8 +10,7 @@ updates: hub-patches: update-types: ["patch"] - # The spoke images every project host runs (Alloy, socket proxy, GPU - # exporter). Digest pins keep them reproducible, not patched. + # The spoke images every project host runs. - package-ecosystem: "docker-compose" directory: "/templates" schedule: @@ -22,7 +19,7 @@ updates: spoke-patches: update-types: ["patch"] - # The demo is a toy: one PR per month for all of it. + # The demo: one PR per month for all of it. - package-ecosystem: "docker" directory: "/demo" schedule: @@ -38,9 +35,8 @@ updates: groups: demo: patterns: ["*"] - # `groups` only cover version updates; without this, every advisory - # opens its own PR and CI run. Only pip: Docker, compose, Actions and - # Terraform get no Dependabot security advisories. + # `groups` only cover version updates; without this every advisory opens + # its own PR. Only pip gets Dependabot security advisories here. demo-security: applies-to: security-updates patterns: ["*"] diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index f2843c5..607bfec 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -1,13 +1,9 @@ -# Thin wrapper around the `just` contract: everything CI runs, you can run -# locally with the same command. `just check` locally is `lint` + `validate`; -# here they split so a lint failure reports in seconds without pulling the -# stack images, which the smoke job pulls anyway. +# Everything CI runs, you can run locally with the same `just` command. `lint` +# has its own job so a lint failure reports without pulling the stack images. name: CI -# Pull requests only: main moves through PRs (the protect-main ruleset -# requires one), so a push run would repeat the check that passed on the PR -# head. Prose changes skip the whole workflow: nothing here reads Markdown. -# infra/ has its own workflow (infra.yml) that only runs when infra/ changes. +# Pull requests only: main moves through PRs, so a push run would repeat the +# check. Prose changes skip the workflow. infra/ has its own (infra.yml). on: pull_request: paths-ignore: ["**.md", "docs/**", "LICENSE", "infra/**"] @@ -16,8 +12,7 @@ on: permissions: contents: read -# A push to an open PR supersedes the run already in flight; the smoke job -# boots a full stack, so leaving them stacked wastes a runner per push. +# A push to an open PR supersedes the run in flight. concurrency: group: ${{ github.workflow }}-${{ github.ref }} cancel-in-progress: true @@ -28,8 +23,7 @@ jobs: timeout-minutes: 10 steps: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - # `just lint` scans git history for secrets; the default shallow clone - # would give it one commit and pass vacuously. + # gitleaks scans git history; a shallow clone would pass vacuously. with: fetch-depth: 0 - uses: extractions/setup-just@53165ef7e734c5c07cb06b3c8e7b647c5aa16db3 # v4.0.0 @@ -45,8 +39,7 @@ jobs: - run: cp .env.example .env - run: just validate - run: just smoke - # Every smoke failure message points at the logs, which are only useful if - # they outlive the job. + # The smoke failure messages point at these logs. - if: failure() run: just smoke-logs - if: always() diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index 577a338..62405a0 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -1,10 +1,7 @@ -# Installed by `just hooks` (prek). Stages sized to their cost: -# pre-commit gitleaks on the staged diff, ~0.5s: a secret is cheapest to -# catch before it is in history. Nothing slower runs per commit. -# commit-msg Conventional Commits shape on the first line. -# pre-push `just check` (lint + validate; ~10s warm on a host that runs -# the stack) and `just infra-validate`, only when infra/ is in -# the push: it downloads the provider every time. +# Installed by `just hooks` (prek). +# pre-commit gitleaks on the staged diff +# commit-msg Conventional Commits shape on the first line +# pre-push `just check`, plus `just infra-validate` when infra/ changed # `prek run` runs the pre-commit stage on demand; `--hook-stage pre-push` the rest. repos: - repo: local diff --git a/CHANGELOG.md b/CHANGELOG.md index ade8c3c..f91803c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,26 +6,25 @@ Notable changes to this stack. Format follows ## [0.3.0] - 2026-09-06 -The hub runs the department's telemetry in production now, with a spoke -contract (hostname, labels, templates pinned by tag) that one project uses. -This release hardens the stack around that contract and makes the checks -prove the data paths, not just the config syntax. 1.0 waits until a second -consumer has confirmed the contract. +The hub runs the department's telemetry in production, with one spoke on the +contract (hostname, labels, templates pinned by tag). This release hardens the +stack around that contract and makes the checks prove the data paths, not just +the config syntax. 1.0 waits for a second consumer to confirm the contract. ### Upgrade - Point every spoke's `OTEL_EXPORTER_OTLP_ENDPOINT` at `otel.`. The - old `otlp.` name is gone, and so is any edge rule that matched it. + old `otlp.` name is gone. - Set `DEPARTMENT` in the hub's `.env`. The collector stamps it on every - signal; the alert rules and dashboards key on it. + signal. - Set `COMPOSE_FILE` in `.env` to name the overlays this host runs (`compose.yml:compose.tunnel.yml` in production). `just up-tunnel` is gone; `just up` runs the exposure guards whenever the tunnel overlay is active. - Remove the orphaned `alertmanager_data` volume when convenient. Alerting is - Grafana-managed (ADR 0002); delivery still posts to `ALERT_WEBHOOK_URL`. + Grafana-managed (ADR 0002). - Re-vendor the templates on each spoke at `v0.3.0` (`bootstrap.sh` prints - the commands): the Alloy agent gains an in-pipeline memory limiter, and the - overlay declares the `egress` network it joins. + the commands): the Alloy agent gains a memory limiter, and the overlay + declares the `egress` network it joins. - Loki streams keep their old label set until they age out (30 days). ### Added @@ -37,12 +36,12 @@ consumer has confirmed the contract. `otel_queue` volume, so telemetry buffered during an outage survives a collector restart. - **Self-monitoring covers every service**: Prometheus scrapes Grafana, Loki - and Tempo too, so `TargetDown` sees them. `HostDiskFilling` (full within 3 - days at the current rate) and `PrometheusCardinalityHigh` (over 100k active - series) warn ahead of the 80% disk backstop. + and Tempo too. `HostDiskFilling` (full within 3 days at the current rate) + and `PrometheusCardinalityHigh` (over 100k active series) warn ahead of the + 80% disk backstop. - **`infra/generate-imports.sh`** emits OpenTofu `import` blocks for the tunnel, DNS records and Access app built by hand, so the first plan does - not create duplicates of objects already serving traffic. + not create duplicates. - **`just smoke` proves the stack works, not that it boots**: every dashboard and alert rule provisioned, uid for uid, none paused; contact points carry the exact URLs given; Grafana honours the JWT settings and refuses a forged @@ -51,31 +50,27 @@ consumer has confirmed the contract. `department` labels. `just restore-check` rehearses backup and restore on the smoke volumes. - **`just check` is `lint` plus `validate`**. `lint` covers compose files, - the rendered alert templates, YAML and its formatting, workflows, shell - scripts, the demo's Python, OpenTofu formatting, dashboard JSON and the - datasource uids it names, and git history for secrets. `validate` runs each - stack config through the exact image the stack uses. `just hooks` installs - git hooks via prek: gitleaks at commit, a Conventional Commits check on the + the rendered alert templates, YAML formatting, workflows, shell scripts, + the demo's Python, OpenTofu formatting, dashboard JSON and the datasource + uids it names, and git history for secrets. `validate` runs each stack + config through the exact image the stack uses. `just hooks` installs git + hooks via prek: gitleaks at commit, a Conventional Commits check on the message, `check` at push. - Dependabot watches the spoke images in `templates/` and the Cloudflare - provider in `infra/`; patch bumps arrive grouped, the demo's advisories in - one PR. + provider in `infra/`. Patch bumps arrive grouped. ### Changed - **Loki indexes only the identity labels** (`service.name`, `department`, - `project`, `env`, `host.name`; the list lives in `config/loki.yaml`). - Everything else is structured metadata, so a sender restart no longer - mints a new stream. + `project`, `env`, `host.name`). Everything else is structured metadata, so + a sender restart no longer mints a new stream. - **Dashboards are provisioned, not editable**: `dashboards/*.json` is the source of truth; UI saves are off. - **Tempo stores traces and nothing else**: its metrics generator is gone. RED comes from the applications' own OTLP metrics (ADR 0002). -- CI runs on pull requests only (`main` requires one), skips prose-only - changes, and validates `infra/` on its own workflow. `lint` and - `validate` + `smoke` run on separate jobs, so a lint failure never waits - on the stack images. `just demo-build` is gone; the demo is a local - fixture. +- CI runs on pull requests only, skips prose-only changes, and validates + `infra/` on its own workflow. `lint` and `validate` + `smoke` run on + separate jobs. `just demo-build` is gone. - `just smoke` and `just demo` each run under their own compose project and Grafana port (`compose.sandbox.yml`), so neither can touch a running stack. - Image bumps: Grafana 13.1.4, Loki 3.7.6, Tempo 3.0.3, Prometheus 3.13.2, @@ -86,9 +81,9 @@ consumer has confirmed the contract. ### Fixed - **`HighErrorRate` measured the wrong thing twice**: it counted all spans, - so child spans diluted the ratio; and it aggregated by job alone, so a - healthy prod service masked a broken staging one. It reads the HTTP server - metrics now, keyed on job, project and env. + so child spans diluted the ratio, and it aggregated by job alone, so a + healthy prod service masked a broken staging one. It now reads the HTTP + server metrics, keyed on job, project and env. - Trace links from the latency panel resolved to nothing: exemplars carry `traceID`, the datasource looked for `trace_id`. - The Infrastructure Logs dashboard queried labels this stack never set. The @@ -105,9 +100,9 @@ consumer has confirmed the contract. still paused when the unpause failed; `bootstrap.sh` printed the hash of empty input for a template missing from the tag; `generate-imports.sh` reported a failed DNS API call as "no record". `bootstrap.sh` now reads - its rule back from Grafana after the restart instead of trusting it. + its rule back from Grafana after the restart. - Grafana and the collector wait for Prometheus to report ready instead of - racing it on a cold start. The demo load generator's error rate matched + racing it on a cold start. The demo load generator's error rate matches its advertised one in ten. ### Security @@ -121,7 +116,7 @@ consumer has confirmed the contract. with no authentication of its own. - `GRAFANA_COOKIE_SECURE` marks the session cookie Secure with strict SameSite. With the tunnel overlay, `just up` refuses to start without it, - an `https://` root URL, non-default credentials, and a webhook URL; + an `https://` root URL, non-default credentials, and a webhook URL. `just lint` runs those guards both ways. - Hub images are digest-pinned, GitHub Actions are pinned to commit SHAs, the tunnel token reaches cloudflared via environment rather than argv, and diff --git a/README.md b/README.md index a617e05..f73b4d0 100644 --- a/README.md +++ b/README.md @@ -6,15 +6,14 @@ [![License: MIT](https://img.shields.io/badge/license-MIT-blue)](LICENSE) Central monitoring for CML's research software. One host runs Grafana, Loki, -Tempo, Prometheus, and an OpenTelemetry Collector, wired so logs, traces, and -metrics cross-reference each other. Projects send telemetry here over OTLP -(the OpenTelemetry protocol) and it lands in one place, queryable side by -side. +Tempo, Prometheus, and an OpenTelemetry Collector. Projects send logs, traces, +and metrics here over OTLP (the OpenTelemetry protocol), and Grafana shows +them side by side. ## Try it in one command You need Docker with the [Compose plugin](https://docs.docker.com/compose/install/) -and [`just`](https://github.com/casey/just#installation); everything else runs +and [`just`](https://github.com/casey/just#installation). Everything else runs in containers. ```sh @@ -22,38 +21,34 @@ cp .env.example .env just demo ``` -This starts the full stack plus a small FastAPI service under constant -artificial load (`compose.demo.yml`). The service uses OpenTelemetry -auto-instrumentation and fails about one request in ten, which gives the error -panels and the error-rate alert something to show. Give it a minute, then open -Grafana at (admin / change-me): +This starts the stack plus a small FastAPI service under constant load +(`compose.demo.yml`). The service is auto-instrumented and fails one request +in ten, so the error panels and the error-rate alert have something to show. +Give it a minute, then open Grafana at +(admin / change-me): - **Dashboards → Service Health (RED)**: request rate, error rate, and - latency. The dots on the latency panel are exemplars: click one and Grafana - opens the exact trace behind that measurement. + latency. The dots on the latency panel are exemplars. Click one to open the + trace behind that measurement. - **Dashboards → Logs**: log volume by service and level, an error feed, and - a live tail of everything arriving over OTLP. -- **Alerting → Alert rules**: the stack-health and error-rate rules Grafana - is evaluating. `HighErrorRate` trips on the demo service after five minutes: - one request in ten failing is twice the 5% threshold. + a live tail. +- **Alerting → Alert rules**: the rules Grafana evaluates. `HighErrorRate` + trips on the demo service after five minutes. ![Service Health (RED) dashboard](docs/img/service-health.png) -`just demo-down` removes the demo services again; the rest of the demo stack -keeps running, and `just demo-destroy` takes the whole thing down. +`just demo-down` removes the demo services. `just demo-destroy` removes the +whole demo stack. -The demo runs under its own compose project on its own port, so it never joins -or disturbs a stack already running on the host. It is safe to run on the -production box. Same for `just smoke`, on :3001. Override with `DEMO_PORT` / -`SMOKE_PORT`. +The demo runs under its own compose project on its own port, so it never +touches a stack already running on the host. The same holds for `just smoke` +on :3001. Override the ports with `DEMO_PORT` and `SMOKE_PORT`. ## How it works Everything enters through one gateway, the OpenTelemetry Collector. A project configures a single endpoint, and a storage backend can be swapped later -without touching any application. The Service Health dashboard -and the error-rate alert read the standard HTTP metrics that OpenTelemetry -auto-instrumentation emits; traces add per-request drill-down on top. +without touching any application. ```mermaid flowchart LR @@ -71,114 +66,100 @@ flowchart LR end ``` -Solid arrows show telemetry being written; dotted arrows show Grafana reading -at query time. Locally there is no tunnel: everything talks over the compose -network, and Grafana is at `localhost:3000` for `just up`, `localhost:3002` for -the isolated `just demo` stack. +Solid arrows write telemetry. Dotted arrows are Grafana reading at query +time. Locally there is no tunnel, and Grafana is at `localhost:3000` for +`just up` or `localhost:3002` for `just demo`. -The stack runs on a single host. At CML's telemetry volume, distributed -ingestion would add operational weight for no gain -([ADR 0001](docs/adr/0001-observability-stack.md) records the alternatives). -The hub-and-spoke design for serving multiple CML projects is -[ADR 0002](docs/adr/0002-hub-and-spoke-observability.md); onboarding a project -onto it is [templates/README.md](templates/README.md). +The stack runs on a single host. [ADR 0001](docs/adr/0001-observability-stack.md) +records why, and the alternatives. [ADR 0002](docs/adr/0002-hub-and-spoke-observability.md) +describes the hub-and-spoke design that serves multiple projects. ## Run it for real ```sh cp .env.example .env # set GRAFANA_ADMIN_PASSWORD -just up # the stack; overlays come from COMPOSE_FILE in .env +just up just check # validate every config in the repo ``` -Grafana: (admin / whatever you set). +Grafana: . -`COMPOSE_FILE` in `.env` names the overlays a host runs. Set -`COMPOSE_FILE=compose.yml:compose.tunnel.yml` in the production `.env`, and -every recipe (`up`, `logs`, `ps`, `backup`) acts on that same set. +`COMPOSE_FILE` in `.env` names the overlays a host runs. In production, set +`COMPOSE_FILE=compose.yml:compose.tunnel.yml`. Every recipe (`up`, `logs`, +`ps`, `backup`) then acts on that set. -With the tunnel overlay active, `just up` refuses to run until four settings are -real: a generated `OTLP_AUTH_TOKEN`, a changed `GRAFANA_ADMIN_PASSWORD`, +With the tunnel overlay active, `just up` refuses to start until four settings +are real: a generated `OTLP_AUTH_TOKEN`, a changed `GRAFANA_ADMIN_PASSWORD`, `GRAFANA_ROOT_URL` pointing at the tunnel hostname, and -`GRAFANA_COOKIE_SECURE=true`. These are the settings that matter once the stack -is reachable. An empty `HEARTBEAT_URL` only warns. - -In production the stack sits behind a Cloudflare Tunnel, and that edge is code -too. The tunnel, its hostnames, DNS, and the Cloudflare Access rule that puts -an email one-time-PIN in front of Grafana live in `infra/` as a small OpenTofu -configuration. Applying it produces the `CLOUDFLARE_TUNNEL_TOKEN` the tunnel -overlay needs; bootstrap steps are at the top of -[infra/main.tf](infra/main.tf). - -`just check` is two halves. `just lint` is the static half: compose files, -the rendered alert templates, YAML, workflows, shell scripts, the demo app's -Python, OpenTofu formatting, dashboard JSON and the datasources it names, and -git history for leaked secrets. `just validate` runs the Prometheus, collector, -Loki, Tempo and Alloy configs through the exact images the stack runs. Every -validator runs in a pinned container, so nothing is installed on the host; -`just fmt` (yamlfmt) is containerized the same way, and `lint` enforces it. -`just hooks` installs the git hooks via [prek](https://github.com/j178/prek): -gitleaks on the staged diff and `just lint` at commit, a Conventional Commits -check on the message, `just validate` at push, and `just infra-validate` at -push when `infra/` changed. - -Grafana's alerting provisioning has no offline validator. `just smoke` covers -it and everything else that only shows once the stack runs: it boots the stack -in production shape (JWT auth on) and asserts, uid for uid, that every -dashboard and alert rule provisioned; that the contact points carry the URLs -from the environment; that Grafana honours the JWT settings and refuses a -forged token; that every scrape target is up; and that one metric and one log -posted through the collector's bearer auth come back out of Prometheus and -Loki with the promoted project/env/department labels. The assertions live in -[scripts/smoke.sh](scripts/smoke.sh). It runs under its own compose project on -its own port, so it cannot disturb a stack already running on the host: -`just smoke` is safe on the production box, `just restore-check` rehearses -backup and restore on its volumes, and `just smoke-down` cleans up. CI runs -`lint` on one job and `validate` plus `smoke` on another, on every pull -request; `main` only moves through pull requests, so nothing runs twice. +`GRAFANA_COOKIE_SECURE=true`. An empty `HEARTBEAT_URL` only warns. + +The Cloudflare edge is code too. The tunnel, its hostnames, DNS, and the +Access rule that puts an email one-time PIN in front of Grafana live in +`infra/` as an OpenTofu configuration. Applying it produces the +`CLOUDFLARE_TUNNEL_TOKEN` the overlay needs. The bootstrap steps are at the +top of [infra/main.tf](infra/main.tf). + +### Checks + +| Recipe | What it does | +| --- | --- | +| `just lint` | Static checks: compose files, rendered alert templates, YAML, workflows, shell, Python, OpenTofu formatting, dashboard JSON, git history for secrets | +| `just validate` | Runs each stack config through the exact image the stack uses | +| `just check` | `lint` plus `validate` | +| `just smoke` | Boots an isolated copy of the stack and asserts it works end to end | +| `just restore-check` | Rehearses backup and restore on the smoke stack | +| `just hooks` | Installs the git hooks: gitleaks at commit, a Conventional Commits check on the message, `check` at push | + +Every check runs in a pinned container. Nothing is installed on the host. + +`just smoke` covers what has no offline validator. It boots the stack in +production shape with JWT auth on, then asserts that every dashboard and +alert rule provisioned, that the contact points carry the URLs from the +environment, that Grafana refuses a forged token, that every scrape target is +up, and that a metric and a log posted through the collector come back out of +Prometheus and Loki with their labels. The assertions are in +[scripts/smoke.sh](scripts/smoke.sh). + +CI runs `lint` on one job and `validate` plus `smoke` on another, on every +pull request. ## Sending telemetry from a project -You need the OTLP endpoint, the bearer token (`OTLP_AUTH_TOKEN`), and a few -naming conventions. **[docs/ONBOARDING.md](docs/ONBOARDING.md)** holds -copy-paste templates for the two application routes: zero-code Python/FastAPI, -and plain OTLP environment variables. Everything an application cannot -report about itself comes from the vendored agent in -**[templates/README.md](templates/README.md)**. +You need the OTLP endpoint, the bearer token, and a few naming conventions. +[docs/ONBOARDING.md](docs/ONBOARDING.md) has copy-paste templates for +applications. [templates/README.md](templates/README.md) covers the host +agent that ships container logs and host metrics. > [!WARNING] > Never publish ports 4317/4318 to the internet. The compose file binds them to -> `127.0.0.1`; the tunnel is the way in. +> `127.0.0.1`. The tunnel is the way in. ## Alerting -Grafana both evaluates and delivers, from `config/grafana/alerting/`: telemetry -silent per project, container crash-looping, container OOM-killed, scrape target -down, OTel export failures, alert delivery failing, error rate above 5%, disk -above 80%. Notifications go to whatever webhook you set in `ALERT_WEBHOOK_URL` -(ntfy, Slack, and so on). There is no Alertmanager: Grafana rules can query Loki -as well as Prometheus, and one engine owning both means one answer to "who gets -told". +Grafana evaluates and delivers the rules in `config/grafana/alerting/`: +telemetry silent per project, container crash-looping or OOM-killed, scrape +target down, OTel export failures, alert delivery failing, error rate above +5%, disk above 80%. There is no Alertmanager. Grafana rules can query Loki as +well as Prometheus, and one engine means one answer to "who gets told". -One rule, `Watchdog`, fires permanently and posts to `HEARTBEAT_URL` every -five minutes. Point that at a dead man's switch such as healthchecks.io, a -service that alerts when the pings *stop*. That covers the one failure the host -cannot report itself: its own death. +Notifications go to the webhook in `ALERT_WEBHOOK_URL` (ntfy, Slack, and so +on). One rule, `Watchdog`, fires permanently and posts to `HEARTBEAT_URL` +every five minutes. Point that at a dead man's switch such as healthchecks.io, +which alerts when the pings stop. That is the only way to notice the host +itself dying. Set both variables. An unset `ALERT_WEBHOOK_URL` drops every alert while the -heartbeat keeps reporting healthy, so `just up` with the tunnel overlay refuses +heartbeat keeps reporting healthy. With the tunnel overlay, `just up` refuses to start without it. ## Storage -Everything persists to local Docker volumes (`loki_data`, `tempo_data`, -`prometheus_data`, `grafana_data`), all captured by `just backup`. Backups skip -a fifth volume, `otel_queue`. It holds the collector's on-disk export queue: -seconds of in-flight telemetry, worthless by the time anyone restores. +Everything persists to local Docker volumes, captured by `just backup`. The +one exception is `otel_queue`, the collector's on-disk export queue. It holds +seconds of in-flight telemetry and is not worth restoring. When local disk stops fitting, Loki and Tempo can move to any S3-compatible -object store (Backblaze B2, Cloudflare R2, Hetzner, MinIO). The appendix of -ADR 0001 documents that change. +object store. The appendix of ADR 0001 documents that change. ## Layout @@ -195,25 +176,25 @@ config/ tempo.yaml # traces prometheus.yaml # metrics grafana/ # provisioned datasources, dashboards, and alerting - # alerting/ = rules, contact points, routing tree -dashboards/ # drop JSON dashboards here; Grafana auto-loads them -docs/ # runbook, onboarding templates, ADRs, screenshots +dashboards/ # JSON dashboards; Grafana loads them from here +docs/ # runbook, onboarding, ADRs, screenshots infra/ # OpenTofu: Cloudflare tunnel, ingress routes, DNS +templates/ # files a project host vendors: agent config, overlays ``` -`dashboards/*.json` is the source of truth for what Grafana shows: the -directory is mounted read-only and UI saves are disabled, so a change made in -the browser lasts until the page reloads. Edit the JSON and provisioning picks -it up within about 30 seconds. To keep something built interactively, export -the dashboard as JSON (or copy one panel's JSON from *Inspect → Panel JSON*) -and paste it back into the file. +`dashboards/*.json` is the source of truth. The directory is mounted read-only +and UI saves are disabled, so a change made in the browser lasts until the +page reloads. Edit the JSON and provisioning picks it up within 30 seconds. To +keep something built interactively, export it as JSON and paste it into the +file. ## Documentation | Document | What it covers | | --- | --- | -| [docs/ONBOARDING.md](docs/ONBOARDING.md) | Connecting a project: endpoint, token, copy-paste templates | -| [docs/RUNBOOK.md](docs/RUNBOOK.md) | Day-to-day ops: rotating tokens, disk pressure, backup and restore | +| [docs/ONBOARDING.md](docs/ONBOARDING.md) | Connecting an application: endpoint, token, copy-paste templates | +| [templates/README.md](templates/README.md) | Onboarding a project host: the agent, GPU hosts, removing a project | +| [docs/RUNBOOK.md](docs/RUNBOOK.md) | Operations: rotating secrets, disk pressure, backup and restore | | [docs/adr/](docs/adr/) | Why the stack looks like this | | [CHANGELOG.md](CHANGELOG.md) | Release history | diff --git a/bootstrap.sh b/bootstrap.sh index af4b9a0..8bfa2ee 100755 --- a/bootstrap.sh +++ b/bootstrap.sh @@ -3,20 +3,13 @@ # # ./bootstrap.sh # -# It does four things: +# 1. renders the ProjectTelemetrySilent rule and reloads Grafana; +# 2. regenerates the coverage rule that catches projects never bootstrapped; +# 3. creates the project's healthchecks.io checks (if an API key is present); +# 4. prints the `.env` block for the project host and the curls that vendor the +# templates at a pinned tag. # -# 1. renders the keystone ProjectTelemetrySilent rule and reloads Grafana; -# 2. regenerates the coverage rule, so a project that ships telemetry without ever -# being bootstrapped is itself alerted on; -# 3. creates the project's healthchecks.io checks (if an API key is present) and -# prints their ping URLs; -# 4. prints the `.env` block to paste on the project host, and the curl that vendors -# the templates at a pinned tag. -# -# Bootstrap creates the safety net, not the telemetry. A host can ship good telemetry -# and still be unmonitored, because the rule that notices its silence lives here. -# -# Idempotent: re-running it re-renders the same files and reloads again. +# Idempotent. set -euo pipefail project="${1:-}" @@ -25,8 +18,8 @@ if [[ -z "$project" || -z "$env_name" ]]; then echo "usage: $0 " >&2 exit 2 fi -# These become Prometheus label values, a Grafana rule uid, and a filename. A quote or a -# brace in a label value produces a rule that silently never matches. +# These become label values, a rule uid, and a filename; a quote or brace in a +# label value produces a rule that silently never matches. if [[ ! "$project" =~ ^[a-z0-9][a-z0-9-]*$ || ! "$env_name" =~ ^[a-z0-9][a-z0-9-]*$ ]]; then echo "error: project and env must match [a-z0-9][a-z0-9-]* (lowercase, no spaces)" >&2 exit 2 @@ -35,34 +28,26 @@ fi root="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)" cd "$root" -# Read HEALTHCHECKS_API_KEY from .env if the environment does not already carry it. -# `just` recipes get .env via dotenv-load; a bare ./bootstrap.sh does not, and would then -# skip check creation without saying why. Only this one key is read: sourcing the whole -# file would drag the stack's secrets into scope. -# -# It must be the project's READ-WRITE API key. The read-only key cannot POST, and a ping -# key only sends pings. +# A bare ./bootstrap.sh does not load .env, so read this one key from it. +# Sourcing the whole file would drag the stack's secrets into scope. if [[ -z "${HEALTHCHECKS_API_KEY:-}" && -f .env ]]; then HEALTHCHECKS_API_KEY="$(sed -n 's/^HEALTHCHECKS_API_KEY=//p' .env | tail -1)" export HEALTHCHECKS_API_KEY fi -# Flat, not a subdirectory: Grafana's alerting provisioner does not recurse. It skips a -# nested directory with a warning, not an error, so that layout looks like it worked and -# provisions nothing. The project- prefix keeps the files grouped in `ls`. -# BOOTSTRAP_OUT_DIR is for `just lint`: it renders the templates into a scratch dir and -# lints the result, so a template edit that Grafana would reject fails in CI instead of -# on the next onboarding. With it set, nothing past the rendering runs. +# Flat, not a subdirectory: Grafana's alerting provisioner does not recurse, and +# skips a nested directory with only a warning. BOOTSTRAP_OUT_DIR is for `just +# lint`; with it set, nothing past the rendering runs. out_dir="${BOOTSTRAP_OUT_DIR:-config/grafana/alerting}" -# The vendored templates and the paths they are printed under, used twice below. +# The vendored templates and the paths they are printed under. templates="alloy/config.alloy:deploy/alloy/config.alloy compose.telemetry.yml:compose.telemetry.yml compose.telemetry.gpu.yml:compose.telemetry.gpu.yml run_scheduled.sh:scripts/run_scheduled.sh" if [[ -z "${BOOTSTRAP_OUT_DIR:-}" ]]; then - # Pinned tag for the vendoring curl, with no fallback to a branch. A moving ref would - # let two projects vendor two different agent configs and call it the same template. + # Pinned tag, no fallback to a branch: two projects must never vendor two + # different configs under the same name. tag="$(git -C "$root" describe --tags --abbrev=0 2>/dev/null)" \ || { echo "error: no release tag to pin the vendoring curls to; tag a release first" >&2; exit 1; } - # The tag must also contain every template, or a curl 404s and the hash printed for - # it below is the hash of nothing. + # The tag must contain every template, or a curl 404s and the hash printed + # for it below is the hash of nothing. for pair in $templates; do git -C "$root" rev-parse -q --verify "${tag}:templates/${pair%%:*}" >/dev/null \ || { echo "error: tag ${tag} predates templates/${pair%%:*}; tag a new release before onboarding" >&2; exit 1; } @@ -71,16 +56,12 @@ if [[ -z "${BOOTSTRAP_OUT_DIR:-}" ]]; then fi # --------------------------------------------------------------- 1. the keystone rules -# Every covered project/environment, read back from the COVERS marker in each rendered -# file, plus the pair being bootstrapped now. No hand-maintained list to fall behind. -# -# The pair matters, not just the project. A project bootstrapped for staging that also -# ships prod would otherwise read as covered while prod has no keystone rule at all. +# Every covered project/env pair, read back from the COVERS marker in each +# rendered file, plus the pair being bootstrapped now. pairs="$({ sed -n 's/^# COVERS: //p' "$out_dir"/project-*.yaml 2>/dev/null || true echo "$project $env_name"; } | sort -u)" -# ALL pairs are re-rendered, not just the invoked one, so a template fix propagates to -# every project on the next bootstrap run instead of waiting for a per-project re-run. +# All pairs are re-rendered, so a template fix reaches every project. while read -r p e; do rendered="${out_dir}/project-${p}-${e}.yaml" sed -e "s/__PROJECT__/${p}/g" \ @@ -103,17 +84,14 @@ echo "rendered ${out_dir}/coverage.yaml (covering: ${covered})" # ------------------------------------------------------------------ 3. reload Grafana if docker compose ps --status running --services 2>/dev/null | grep -qx grafana; then - # Same guards as `just up`: with the tunnel overlay in COMPOSE_FILE this recreates - # an exposed Grafana, and a drifted .env must refuse here too. + # This recreates an exposed Grafana, so the `just up` guards apply here too. just _guard-if-exposed || exit 1 - # A restart, not SIGHUP: Grafana re-reads alert provisioning only at startup, and - # a SIGHUP reports success while changing nothing. + # A restart, not SIGHUP: Grafana re-reads alert provisioning only at startup. docker compose up -d --force-recreate grafana >/dev/null 2>&1 \ && echo "reloaded grafana (restarted)" \ || echo "WARNING: could not restart grafana; run 'docker compose up -d --force-recreate grafana'" >&2 - # A restart is not proof: Grafana skips a malformed alert group with a log line and - # comes up healthy without it. Read the rule back. Same single-key read as the - # healthchecks key above, for the same reason. + # Grafana skips a malformed alert group with only a log line, so read the + # rule back. if [[ -z "${GRAFANA_ADMIN_PASSWORD:-}" && -f .env ]]; then GRAFANA_ADMIN_PASSWORD="$(sed -n 's/^GRAFANA_ADMIN_PASSWORD=//p' .env | tail -1)" fi @@ -138,8 +116,7 @@ if [[ -n "${HEALTHCHECKS_API_KEY:-}" ]]; then hc_note="created via API" # Default job set; override per project with HC_JOBS="backup nightly-sync" etc. for job in ${HC_JOBS:-backup watchdog restore-check}; do - # API key via curl's stdin config, not -H: argv is readable in `ps` by - # any local user, and this is a read-write key. + # Key via curl's stdin config, not -H: argv is readable in `ps`. printf 'header = "X-Api-Key: %s"\n' "$HEALTHCHECKS_API_KEY" \ | curl -fsS -K - -X POST https://healthchecks.io/api/v3/checks/ \ -H "Content-Type: application/json" \ @@ -150,8 +127,8 @@ else echo "note HEALTHCHECKS_API_KEY unset; ${hc_note}" fi -# Computed before the heredoc: a substitution inside `cat < non-zero count(container_start_time_seconds{project="${project}",name!=""}) -> one per container diff --git a/compose.demo.yml b/compose.demo.yml index c7a5631..861f976 100644 --- a/compose.demo.yml +++ b/compose.demo.yml @@ -1,7 +1,6 @@ -# Demo overlay: a tiny auto-instrumented FastAPI service plus a load -# generator, so the stack has real traces, metrics, and correlated logs -# to show. Not for production: restart policies are omitted, so it does not -# survive a reboot. +# Demo overlay: an auto-instrumented FastAPI service plus a load generator, so +# the stack has traces, metrics, and correlated logs to show. Does not survive a +# reboot. # # just demo # core stack + this overlay # just demo-down # remove just the demo services @@ -23,13 +22,11 @@ services: OTEL_EXPORTER_OTLP_ENDPOINT: http://otel-collector:4318 OTEL_EXPORTER_OTLP_PROTOCOL: http/protobuf OTEL_EXPORTER_OTLP_HEADERS: Authorization=Bearer ${OTLP_AUTH_TOKEN:?set OTLP_AUTH_TOKEN in .env} - # Traces/metrics/logs exporters all default to otlp via opentelemetry-distro. # Ship Python log records via OTLP with trace context attached. OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED: "true" OTEL_METRIC_EXPORT_INTERVAL: "10000" - # Emit the *stable* HTTP semconv (http.server.request.duration in seconds, - # http.route) rather than the legacy names. Stable metric names are frozen, - # so the Service Health dashboard's app-SDK panel survives library bumps. + # Stable HTTP semconv (http.server.request.duration, http.route), which the + # Service Health dashboard queries. OTEL_SEMCONV_STABILITY_OPT_IN: http logging: *demo-logging diff --git a/compose.sandbox.yml b/compose.sandbox.yml index fb0628c..41c2670 100644 --- a/compose.sandbox.yml +++ b/compose.sandbox.yml @@ -1,15 +1,9 @@ # Isolation overlay shared by `just smoke` and `just demo`. # -# Smoke runs the core stack under its own compose project name (see the justfile), -# which already gives it separate containers and volumes. Host port bindings are -# the one thing a project name cannot separate: compose.yml publishes 3000, 4317 -# and 4318 on 127.0.0.1. A smoke run on a host that already serves the stack -# would fail to bind, and on a host where the stack is down it would quietly -# answer on a port something else expects to own. -# -# This overlay drops the ingestion ports entirely (the demo app and the smoke -# assertions reach the collector over the project network) and moves Grafana to -# SANDBOX_PORT, which `just smoke` probes and `just demo` serves. +# A separate compose project gives them their own containers and volumes, but +# not their own host ports. This overlay drops the ingestion ports (the demo app +# and the smoke assertions reach the collector over the project network) and +# moves Grafana to SANDBOX_PORT, so neither can collide with a running stack. services: otel-collector: ports: !override [] diff --git a/compose.tunnel.yml b/compose.tunnel.yml index 9c220ea..fa3934d 100644 --- a/compose.tunnel.yml +++ b/compose.tunnel.yml @@ -1,25 +1,21 @@ -# Production overlay: expose Grafana (and the OTLP endpoints) via Cloudflare -# Tunnel. Requires CLOUDFLARE_TUNNEL_TOKEN in .env. +# Production overlay: expose Grafana and the OTLP endpoint via Cloudflare Tunnel. # # COMPOSE_FILE=compose.yml:compose.tunnel.yml in .env, then `just up` # -# The tunnel, its public hostnames, and DNS are managed as code in infra/ -# (OpenTofu); `tofu output -raw tunnel_token` yields the token for .env. -# See infra/main.tf for the bootstrap steps. +# The tunnel, its hostnames, and DNS are OpenTofu in infra/; +# `tofu output -raw tunnel_token` yields CLOUDFLARE_TUNNEL_TOKEN for .env. services: cloudflared: image: cloudflare/cloudflared:2026.8.2@sha256:0aa26e284f05e6c77ae375b8c9c11d9eb6a448fb7bcd8d40f31cb6176189eb38 restart: unless-stopped depends_on: [grafana, otel-collector] - # Token via env, not argv: --token would sit in the process table, readable - # from the host and from any container with pid:host (node-exporter has it). + # Token via env, not --token: argv is readable from the host and from any + # container with pid:host (node-exporter has it). command: tunnel --no-autoupdate run environment: TUNNEL_TOKEN: ${CLOUDFLARE_TUNNEL_TOKEN:?set CLOUDFLARE_TUNNEL_TOKEN in .env} - # Ceiling sized from observed usage with headroom; see compose.yml. mem_limit: 256m pids_limit: 256 - # Nothing to write: config comes from Cloudflare, the token from env. read_only: true cap_drop: [ALL] security_opt: diff --git a/compose.yml b/compose.yml index 68016cd..c1c7c68 100644 --- a/compose.yml +++ b/compose.yml @@ -3,14 +3,12 @@ name: monitoring # Central observability stack. # OTel Collector → ingestion gateway (OTLP gRPC 4317 / HTTP 4318) # Loki → logs -# Tempo → traces (OTLP-native; Grafana has first-class support) -# Prometheus → metrics (native OTLP remote-write-receiver) +# Tempo → traces +# Prometheus → metrics # Grafana → UI # -# Projects ship telemetry to this host via OTLP. Do NOT publish :4317/:4318 -# directly to the public internet. Expose them via Cloudflare Tunnel, Tailscale, -# WireGuard, or similar. The ports below bind to 127.0.0.1, so only the host and -# the tunnel sidecar reach them. +# Do NOT publish :4317/:4318 to the public internet. The ports below bind to +# 127.0.0.1; expose them via Cloudflare Tunnel, Tailscale, WireGuard, or similar. x-logging: &default-logging driver: json-file @@ -21,13 +19,9 @@ x-logging: &default-logging x-security: &default-security - no-new-privileges:true -# Every service also drops all capabilities and carries a pids limit: they all -# run unprivileged uids, write only to named volumes, and need none. The client -# templates go further, with read-only roots (templates/compose.telemetry.yml). - -# Each mem_limit is sized from observed steady-state usage with headroom for -# bursts. They are ceilings, not reservations: they stop one runaway component -# from OOMing a host that also runs production. +# Every service drops all capabilities and carries a pids limit. Each mem_limit +# is a ceiling sized from observed usage, so one runaway component cannot OOM a +# host that also runs production. services: otel-collector: @@ -45,9 +39,7 @@ services: - "127.0.0.1:4318:4318" # OTLP HTTP volumes: - ./config/otel-collector.yaml:/etc/otelcol/config.yaml:ro - # File-backed exporter queue: telemetry buffered during a backend outage - # survives a collector restart. The volume must be owned by uid 10001; - # `just up` chowns it. + # File-backed exporter queue. Must be owned by uid 10001; `just up` chowns it. - otel_queue:/var/lib/otelcol/queue command: ["--config=/etc/otelcol/config.yaml"] environment: @@ -68,8 +60,7 @@ services: - ./config/loki.yaml:/etc/loki/loki.yaml:ro - loki_data:/loki command: ["-config.file=/etc/loki/loki.yaml"] - # No healthcheck: the image is distroless, so there is no shell, wget or - # curl to probe /ready with. Grafana's Stack Health dashboard covers it. + # No healthcheck: the image is distroless, with no shell or wget to probe with. mem_limit: 2g pids_limit: 1024 cap_drop: [ALL] @@ -84,7 +75,7 @@ services: - ./config/tempo.yaml:/etc/tempo/tempo.yaml:ro - tempo_data:/var/tempo command: ["-config.file=/etc/tempo/tempo.yaml"] - # Distroless too, so no healthcheck (see loki). + # Distroless too, so no healthcheck. mem_limit: 2g pids_limit: 1024 cap_drop: [ALL] @@ -105,7 +96,7 @@ services: # Prometheus 3.x: OTLP ingestion is its own flag, not an --enable-feature entry. - --web.enable-otlp-receiver - --enable-feature=native-histograms,exemplar-storage - # Whichever hits first wins; the disk-space alert is the backstop. + # Whichever limit hits first wins; the disk-space alert is the backstop. - --storage.tsdb.retention.size=15GB healthcheck: test: ["CMD", "wget", "-q", "--spider", "http://localhost:9090/-/ready"] @@ -113,8 +104,7 @@ services: timeout: 3s retries: 5 start_period: 30s - # Probe often while starting: `--wait` (just up/smoke) returns on the - # first pass, and both are ready well inside one 15s interval. + # Probe often while starting so `up --wait` returns on the first pass. start_interval: 2s mem_limit: 2g pids_limit: 1024 @@ -129,16 +119,14 @@ services: command: ["--path.rootfs=/host"] pid: host mem_limit: 128m - # Holds pid:host plus the whole host filesystem, so a compromise here sees - # more than in any other service: hence the tightest limits of the set. + # Holds pid:host plus the whole host filesystem, hence the tightest limits. cap_drop: [ALL] read_only: true pids_limit: 64 security_opt: *default-security volumes: - # No rslave: unsupported on Docker Desktop; only affects mounts added after start. - # No network_mode:host either, so node_network_* describes the container - # veth, not host NICs. Fine while nothing alerts on network metrics. + # No rslave (unsupported on Docker Desktop) and no network_mode:host, so + # node_network_* describes the container veth, not host NICs. - /:/host:ro networks: [backend] logging: *default-logging @@ -168,22 +156,16 @@ services: HEARTBEAT_URL: ${HEARTBEAT_URL:-} GF_USERS_ALLOW_SIGN_UP: "false" GF_SERVER_ROOT_URL: ${GRAFANA_ROOT_URL:-http://localhost:3000} - # Land on Stack Health instead of the empty welcome page. GF_DASHBOARDS_DEFAULT_HOME_DASHBOARD_PATH: /var/lib/grafana/dashboards/stack-health.json - # Opt-in: secure cookies would break plain-http localhost logins. With - # the tunnel overlay active, `just up` refuses to expose Grafana without - # it. + # Secure cookies break plain-http localhost logins, so opt-in. With the + # tunnel overlay active, `just up` refuses to start without it. GF_SECURITY_COOKIE_SECURE: ${GRAFANA_COOKIE_SECURE:-false} GF_SECURITY_COOKIE_SAMESITE: strict - # Opt-in per-user identity from Cloudflare Access, replacing the shared - # admin login. Off until GRAFANA_JWT_AUTH=true plus CF_ACCESS_TEAM_DOMAIN - # and CF_ACCESS_AUD are set in .env (the exposure guards enforce the - # pair). New users land on the org's default role (Viewer). - # No fallback domain: an unset team name must fail closed, not fetch - # signing keys from a claimable cloudflareaccess.com subdomain. - # The aud pin matters because the JWK set is team-wide. Without it, a - # token minted for any other Access app in the team is accepted here - # too. + # Per-user identity from Cloudflare Access. Off until GRAFANA_JWT_AUTH=true + # plus CF_ACCESS_TEAM_DOMAIN and CF_ACCESS_AUD are set (the exposure guards + # enforce the pair). No fallback team domain: an unset name must fail + # closed, not fetch signing keys from a claimable cloudflareaccess.com + # subdomain. The aud pin is required because the JWK set is team-wide. GF_AUTH_JWT_ENABLED: ${GRAFANA_JWT_AUTH:-false} GF_AUTH_JWT_HEADER_NAME: Cf-Access-Jwt-Assertion GF_AUTH_JWT_JWK_SET_URL: https://${CF_ACCESS_TEAM_DOMAIN:-}.cloudflareaccess.com/cdn-cgi/access/certs @@ -197,8 +179,6 @@ services: timeout: 3s retries: 5 start_period: 30s - # Probe often while starting: `--wait` (just up/smoke) returns on the - # first pass, and both are ready well inside one 15s interval. start_interval: 2s mem_limit: 1g pids_limit: 1024 @@ -207,11 +187,10 @@ services: networks: [default, backend] logging: *default-logging -# Loki, Prometheus and Tempo have no authentication of their own, so they live -# on a network with no gateway that only Grafana and the collector can reach. -# cloudflared (compose.tunnel.yml) stays on `default`: its ingress list is -# fetched from Cloudflare at runtime, so an edited ingress cannot publish a -# backend that skips the collector's token gate. +# Loki, Prometheus and Tempo have no authentication of their own, so only +# Grafana and the collector can reach them. cloudflared stays on `default`: its +# ingress list comes from Cloudflare at runtime, so an edited ingress cannot +# publish a backend that skips the collector's token gate. networks: backend: internal: true @@ -221,6 +200,5 @@ volumes: tempo_data: prometheus_data: grafana_data: - # Not in `just backup`: queue contents are seconds of in-flight telemetry, - # worthless by restore time. + # Not in `just backup`: seconds of in-flight telemetry, worthless by restore time. otel_queue: diff --git a/config/grafana/alerting/contact-points.yaml b/config/grafana/alerting/contact-points.yaml index 99b98cb..621bdb2 100644 --- a/config/grafana/alerting/contact-points.yaml +++ b/config/grafana/alerting/contact-points.yaml @@ -1,12 +1,8 @@ -# Notification targets. Grafana expands $VAR in provisioning files, so the URLs come -# straight from the environment and no secret is written to disk. +# Notification targets. Grafana expands $VAR from the environment. # -# An unset variable leaves the URL empty and every notification fails, silently from -# the outside. Nothing here catches that: AlertDeliveryFailing routes to this same -# webhook, so it fails the same way it is detecting, and the heartbeat rides its own -# contact point and stays green throughout. The exposure guard in `just up` is the only -# control. AlertDeliveryFailing covers the other case: a URL that is set but whose -# receiver rejects or flaps, where a later retry does get through. +# An unset variable leaves the URL empty and every notification fails silently: +# AlertDeliveryFailing routes to this same webhook, and the heartbeat stays +# green. The exposure guard in `just up` is the only control. apiVersion: 1 @@ -26,6 +22,5 @@ contactPoints: type: webhook settings: url: $HEARTBEAT_URL - # A resolved-notification would ping the dead man's switch too, which - # would mask the very outage the switch exists to report. + # A resolved-notification would ping the switch and mask an outage. disableResolveMessage: true diff --git a/config/grafana/alerting/coverage.yaml b/config/grafana/alerting/coverage.yaml index 2e5d97c..865dbfb 100644 --- a/config/grafana/alerting/coverage.yaml +++ b/config/grafana/alerting/coverage.yaml @@ -1,14 +1,8 @@ -# GENERATED by bootstrap.sh from templates/alerting/coverage.yaml.tmpl. Do not edit the -# rendered file; it is regenerated on every run. +# GENERATED by bootstrap.sh from templates/alerting/coverage.yaml.tmpl. Do not edit +# the rendered file; it is regenerated on every run. # -# The gap this closes: ProjectTelemetrySilent only exists for projects bootstrap.sh was -# actually run for. A project that ships telemetry but skipped bootstrap is silently -# uncovered, one level up from the failure the keystone rule closes. This fires on any -# series whose project/env pair has no rendered rule file. -# -# The selector touches every series every project sends (~3k for one project). Cheap at -# this size, evaluated instantly every 5m; revisit if a project's active series reach -# five figures. +# Fires on any series whose project/env pair has no rendered rule file: a project that +# ships telemetry but was never bootstrapped has no ProjectTelemetrySilent rule. # # Covered right now: relab/staging diff --git a/config/grafana/alerting/notification-policies.yaml b/config/grafana/alerting/notification-policies.yaml index 7754eb7..dda18e5 100644 --- a/config/grafana/alerting/notification-policies.yaml +++ b/config/grafana/alerting/notification-policies.yaml @@ -1,7 +1,5 @@ -# Routing tree. Everything reaches the webhook except Watchdog. Watchdog's -# delivery is itself the heartbeat: it fires permanently and its silence is the -# alarm, so it goes to the dead man's switch on a short repeat and never groups -# or waits. +# Everything reaches the webhook except Watchdog, whose delivery is the +# heartbeat: it goes to the dead man's switch on a short repeat and never waits. apiVersion: 1 diff --git a/config/grafana/alerting/project-relab-staging.yaml b/config/grafana/alerting/project-relab-staging.yaml index b1712b9..0e5a4fe 100644 --- a/config/grafana/alerting/project-relab-staging.yaml +++ b/config/grafana/alerting/project-relab-staging.yaml @@ -1,14 +1,10 @@ -# Rendered by bootstrap.sh into config/grafana/alerting/, flat: Grafana's alerting -# provisioner does not recurse. Do not edit the rendered files by hand. Re-run -# bootstrap.sh, or edit this template so every project gets the fix. +# Rendered by bootstrap.sh into config/grafana/alerting/. Do not edit the rendered +# files by hand; edit this template so every project gets the fix. # -# This is the keystone: nothing on a project host can detect its own absence. A project -# that ships telemetry but never had bootstrap.sh run is silently uncovered, which is -# what the ProjectsUncovered rule in coverage.yaml catches one level up. +# Nothing on a project host can detect its own absence, so this rule lives here. # -# bootstrap.sh reads the marker below back to regenerate that coverage rule. It carries -# the pair, not the filename, because both halves may contain a dash: `project-a-b-c` -# cannot be split back into the project and env that produced it. +# bootstrap.sh reads the marker below to regenerate coverage.yaml. It carries the pair, +# not the filename, because both halves may contain a dash. # COVERS: relab staging apiVersion: 1 @@ -19,10 +15,8 @@ groups: folder: Stack alerts interval: 1m rules: - # A label-only selector, not target_info: agents attach the identity labels to - # the series themselves, so target_info carries none of them. absent() - # yields nothing while telemetry flows, so NoData is the HEALTHY state and must - # map to OK or the rule reads inverted. + # absent() yields nothing while telemetry flows, so NoData is the HEALTHY state + # and must map to OK. - uid: proj-silent-relab-staging title: ProjectTelemetrySilent condition: FIRING diff --git a/config/grafana/alerting/rules.yaml b/config/grafana/alerting/rules.yaml index 81225db..6b53433 100644 --- a/config/grafana/alerting/rules.yaml +++ b/config/grafana/alerting/rules.yaml @@ -1,22 +1,18 @@ -# Grafana-managed alert rules (ADR 0002): one engine owns evaluation and notification, -# and Grafana rules can query Loki as well as Prometheus. +# Grafana-managed alert rules (ADR 0002). Every rule has the same shape: the +# PromQL carries its own comparison and returns a series only when the alert +# should fire; the threshold node only asks whether anything came back. # -# Shape note, true for every rule here: the PromQL carries its own comparison, so it -# returns a series only when the alert should fire, and the threshold node only asks -# whether anything came back. -# -# Keep the total around a dozen. These are the generic rules; the per-project keystone -# lives in project--.yaml beside this file, rendered by bootstrap.sh. +# Keep the total around a dozen. The per-project rules beside this file are +# rendered by bootstrap.sh. apiVersion: 1 groups: - # The alerting pipeline's own proof of life, and the detector for it failing. - orgId: 1 name: meta folder: Stack alerts interval: 1m rules: - # Always firing; its arrival is the proof. execErrState is Error so that a - # broken query stops the heartbeat and the switch alarms. + # Always firing. execErrState is Error so a broken query stops the + # heartbeat and the switch alarms. - uid: watchdog-heartbeat title: Watchdog condition: FIRING @@ -36,8 +32,7 @@ groups: editorMode: code expr: vector(1) - &firing - # Identical on every rule (see the shape note up top): defined once - # here, aliased as `- *firing` below. + # Identical on every rule; aliased as `- *firing` below. refId: FIRING datasourceUid: __expr__ model: @@ -53,8 +48,7 @@ groups: annotations: summary: "Alerting-pipeline heartbeat (always firing)" description: "Routed to a dead man's switch. Investigate if the pings stop." - # Cannot page when the path it reports on is the broken one; the heartbeat - # covers that case. It makes the state visible instead of silent. + # Cannot page when the webhook itself is broken; the heartbeat covers that. - uid: alert-delivery-failing title: AlertDeliveryFailing condition: FIRING @@ -81,7 +75,6 @@ groups: description: >- Alerts are firing and going nowhere. Most likely an unset ALERT_WEBHOOK_URL, otherwise the receiver is rejecting. Check `just logs grafana`. - # The central stack watching itself. - orgId: 1 name: stack-health folder: Stack alerts @@ -140,7 +133,6 @@ groups: description: >- The collector has been failing to deliver telemetry to a backend for 5 minutes. Check `just logs otel-collector`. - # Storage backstop: Loki and Tempo have no total-size cap. - orgId: 1 name: capacity folder: Stack alerts @@ -170,16 +162,13 @@ groups: labels: severity: warning annotations: - # host_name comes on spoke series shipped via Alloy; the central - # node-exporter has only instance. `or` picks whichever exists. + # Spoke series carry host_name; the central node-exporter only instance. summary: "Filesystem {{ $labels.mountpoint }} on {{ or $labels.host_name $labels.instance }} is over 80% full" description: >- Loki and Tempo have no total-size cap, so this alert is the storage backstop. Free space or lower retention (see RUNBOOK). - # The 20% floor above is a last line; this one fires while there is still - # time to act. Linear fit over 6h, projected 3 days out. Root and data - # mounts only: a sub-10G filesystem (boot, EFI, snap) crosses zero on - # noise and is not where telemetry lands. + # Linear fit over 6h, projected 3 days out. Filesystems under 10G (boot, + # EFI, snap) are skipped: they cross zero on noise. - uid: host-disk-filling title: HostDiskFilling condition: FIRING @@ -211,10 +200,8 @@ groups: Extrapolated from the last 6 hours. Find what is growing (a spoke shipping more than before, a log loop, a backup that stopped rotating) before HostDiskSpaceLow makes it urgent. - # TSDB growth is series count, not time. A spoke that starts labelling by - # request id shows here days before it reaches the 15GB retention cap, - # and long before HostDiskSpaceLow. The threshold is ~7x the baseline - # of ~15k series with one spoke; raise it as spokes are onboarded. + # ~7x the baseline of ~15k series with one spoke; raise it as spokes are + # onboarded. - uid: prometheus-cardinality title: PrometheusCardinalityHigh condition: FIRING @@ -244,7 +231,6 @@ groups: `topk(10, count by (job) ({__name__=~".+"}))`, then drop it at the spoke's Alloy config or the collector. Retention is 15GB; at this rate it fills. - # A crash loop must get louder, not quieter. - orgId: 1 name: container-lifecycle folder: Stack alerts @@ -298,7 +284,6 @@ groups: annotations: summary: "Container {{ $labels.name }} was OOM-killed" description: "{{ $labels.name }} on {{ $labels.host_name }} hit an OOM kill in the last 5 minutes." - # RED from the applications' own OTLP metrics, not Tempo's span-metrics. - orgId: 1 name: service-red folder: Stack alerts @@ -321,9 +306,8 @@ groups: refId: QUERY instant: true editorMode: code - # Keyed on (job, project, env) like every other multi-tenant rule - # here: aggregating on job alone lets a healthy prod service dilute - # a broken staging one sharing the job name below the threshold. + # Keyed on (job, project, env): on job alone, a healthy prod + # service dilutes a broken staging one below the threshold. expr: > sum by (job, project, env) (rate(http_server_request_duration_seconds_count{http_response_status_code=~"5.."}[5m])) diff --git a/config/grafana/datasources.yaml b/config/grafana/datasources.yaml index 3c38b73..daefee2 100644 --- a/config/grafana/datasources.yaml +++ b/config/grafana/datasources.yaml @@ -19,7 +19,7 @@ datasources: access: proxy url: http://loki:3100 jsonData: - # Click a trace_id in a log line → jump to the trace in Tempo. + # trace_id in a log line links to the trace in Tempo. derivedFields: - name: trace_id matcherRegex: "[tT]race[_-]?[iI][dD][\"=:\\s]+([A-Fa-f0-9]+)" @@ -38,10 +38,8 @@ datasources: access: proxy url: http://tempo:3200 jsonData: - # No tracesToMetrics/serviceMap/nodeGraph: all three need span-metrics series - # this stack does not generate (RED comes from the apps' own OTLP metrics, - # ADR 0002). Configured anyway, they render a permanently empty service graph - # that reads as "no traffic". + # No tracesToMetrics/serviceMap/nodeGraph: they need span-metrics this + # stack does not generate, and would render an empty service graph. tracesToLogsV2: datasourceUid: loki filterByTraceID: true diff --git a/config/loki.yaml b/config/loki.yaml index a60faee..faff492 100644 --- a/config/loki.yaml +++ b/config/loki.yaml @@ -1,5 +1,4 @@ -# Single-binary Loki with filesystem storage. Swap `common.storage.filesystem` -# for `s3:` (B2 / R2 / Hetzner / MinIO) when you outgrow local disk. +# Single-binary Loki with filesystem storage. auth_enabled: false @@ -34,23 +33,16 @@ limits_config: retention_period: 30d allow_structured_metadata: true volume_enabled: true - # Most log lines under ~100KB. Bump if you log large payloads. max_line_size: 256kb - # Keep ingestion generous for a single-tenant homelab. ingestion_rate_mb: 16 ingestion_burst_size_mb: 32 - # Index only the identity labels listed below. That list is the authoritative - # set, and the docs point here. Loki's default OTLP mapping also indexes - # service.instance.id, which mints a fresh stream on every sender restart; - # it is now structured metadata, so query it with `| service_instance_id=...`. + # Index only the identity labels below; everything else is structured + # metadata. Loki's default mapping also indexes service.instance.id, which + # mints a fresh stream on every sender restart. These five must be real + # stream labels: dashboard template variables use label_values(), which + # cannot see structured metadata. otlp_config: resource_attributes: - # ignore_defaults keeps Loki from indexing every resource attribute it is - # handed. These five are the identity labels ADR 0002 requires on every - # signal, and they have to be real stream LABELS, not structured metadata: - # a `project` template variable is a stream selector, and label_values() - # cannot see structured metadata. All five are bounded (one value per - # department, project, environment and host), so indexing them is cheap. ignore_defaults: true attributes_config: - action: index_label diff --git a/config/otel-collector.yaml b/config/otel-collector.yaml index 0281b6d..ec6e886 100644 --- a/config/otel-collector.yaml +++ b/config/otel-collector.yaml @@ -1,15 +1,11 @@ -# Ingestion gateway. Receives OTLP from all projects and fans out to the -# appropriate backend. Keep this config dumb. Shaping and enrichment belong at -# the app edge (per-project collector), which has the context for them. +# Ingestion gateway: receives OTLP from all projects and fans out to the +# backends. Keep it dumb; shaping and enrichment belong at the app edge. -# All senders must present "Authorization: Bearer ". The -# OTLP hostnames are public behind the tunnel, so ingestion needs auth. extensions: bearertokenauth: token: ${env:OTLP_AUTH_TOKEN} - # Backs the exporter queues below so buffered telemetry survives collector - # restarts (every image bump is one). The volume must be writable by the - # image's uid 10001; `just up` chowns it (see _queue-volume in the justfile). + # Backs the exporter queues so buffered telemetry survives a collector + # restart. The directory must be writable by uid 10001; `just up` chowns it. file_storage: directory: /var/lib/otelcol/queue create_directory: true @@ -27,20 +23,14 @@ receivers: authenticator: bearertokenauth processors: - # The department every signal belongs to. Stamped here rather than at the app - # edge: it is the same value for everything this gateway accepts, and upsert - # means a sender cannot claim someone else's department. project, env and - # service.name still come from the sender, which is the only thing that knows - # them. + # Stamped here with upsert, so a sender cannot claim another department. resource/department: attributes: - key: department value: ${env:DEPARTMENT} action: upsert - # Absolute, not percentage: percentages are read against the cgroup, and an - # unbounded container makes that the whole host. Sized to the container's - # mem_limit (512m in compose.yml); keep the two in step. + # Sized to the container's mem_limit (512m in compose.yml); keep the two in step. memory_limiter: check_interval: 2s limit_mib: 400 @@ -49,12 +39,9 @@ processors: timeout: 5s send_batch_size: 1024 -# The queue and retry settings are pinned: the RUNBOOK's "buffers about five -# minutes" promise is exactly these numbers. Defined once on the loki exporter -# and aliased onto the other two. The queue is file-backed, so it rides out both -# a backend outage and a collector restart. +# The RUNBOOK's "buffers about five minutes" promise is max_elapsed_time below. +# Defined once on the loki exporter and aliased onto the other two. exporters: - # Loki 3.x accepts OTLP natively, so no Promtail or loki exporter is needed. otlp_http/loki: endpoint: http://loki:3100/otlp sending_queue: &queue @@ -70,8 +57,7 @@ exporters: sending_queue: *queue retry_on_failure: *retry - # Prometheus native OTLP receiver (needs --web.enable-otlp-receiver, set in - # compose.yml; it is a real flag, not an --enable-feature entry). + # Needs --web.enable-otlp-receiver in compose.yml. otlp_http/prometheus: endpoint: http://prometheus:9090/api/v1/otlp sending_queue: *queue diff --git a/config/prometheus.yaml b/config/prometheus.yaml index 2d34317..f82687b 100644 --- a/config/prometheus.yaml +++ b/config/prometheus.yaml @@ -6,21 +6,17 @@ global: storage: tsdb: - # Required for OTLP ingestion: without it, late batches are silently - # dropped. + # Required for OTLP ingestion: without it, late batches are silently dropped. out_of_order_time_window: 30m otlp: - # Prometheus otherwise keeps these resource attributes on target_info only, - # so alerts and dashboards see no project/env/host. Promoting them puts the - # identity labels ADR 0002 requires on every series. service.name and - # service.instance.id already become job/instance without being listed. + # Otherwise these stay on target_info only and no alert or dashboard sees + # them. service.name and service.instance.id become job/instance regardless. promote_resource_attributes: [department, project, env, host.name] -# No rule_files and no alerting block: Grafana owns both evaluation and -# notification (ADR 0002). Prometheus is a store and a query engine here. +# No rule_files and no alerting block: Grafana owns both (ADR 0002). -# Apps push metrics via OTLP (through the collector). Scrape only infra we host. +# Apps push metrics via OTLP. Scrape only what this stack hosts. scrape_configs: - job_name: prometheus static_configs: @@ -34,7 +30,6 @@ scrape_configs: static_configs: - targets: ["node-exporter:9100"] - # Self-metrics of the remaining stack services, so TargetDown covers them too. - job_name: grafana static_configs: - targets: ["grafana:3000"] diff --git a/config/tempo.yaml b/config/tempo.yaml index 9447b5e..20d260b 100644 --- a/config/tempo.yaml +++ b/config/tempo.yaml @@ -1,6 +1,4 @@ -# Tempo 3.x single-binary. Traces arrive only from the collector -# (collector → tempo:4317) so external apps never talk to Tempo directly. -# Block retention lives under overrides. +# Tempo single-binary. Traces arrive only from the collector. server: http_listen_port: 3200 @@ -9,7 +7,6 @@ distributor: receivers: otlp: protocols: - # gRPC only: the collector is the sole sender (otlp_grpc/tempo). grpc: endpoint: 0.0.0.0:4317 @@ -21,15 +18,12 @@ storage: local: path: /var/tempo/blocks -# No metrics_generator: RED comes from the applications' own OTLP metrics (ADR -# 0002). Deriving it from traces too would give two producers for one number. -# Tempo stores traces and nothing else. +# No metrics_generator: RED comes from the applications' own OTLP metrics (ADR 0002). overrides: defaults: compaction: - # 7 days. Traces are bulky; tune this to the disk budget. - block_retention: 168h + block_retention: 168h # 7 days usage_report: reporting_enabled: false diff --git a/demo/app.py b/demo/app.py index 802908d..43ad079 100644 --- a/demo/app.py +++ b/demo/app.py @@ -1,7 +1,6 @@ """Minimal FastAPI service for the demo overlay. -All telemetry (traces, metrics, logs with trace context) comes from OTel -auto-instrumentation; see compose.demo.yml. No OTel code needed here. +All telemetry comes from OTel auto-instrumentation; see compose.demo.yml. """ import logging @@ -10,11 +9,9 @@ from fastapi import FastAPI, HTTPException -ERROR_RATE = 0.1 # fixed error rate, enough to light up RED panels +ERROR_RATE = 0.1 -logging.basicConfig( - level=logging.INFO, -) # root logger defaults to WARNING; we want the INFO lines too +logging.basicConfig(level=logging.INFO) log = logging.getLogger("demo-api") app = FastAPI() @@ -28,7 +25,7 @@ def root() -> dict[str, bool]: @app.get("/work") def work() -> dict[str, bool]: """Simulate variable-latency work that sometimes fails.""" - time.sleep(random.uniform(0.02, 0.3)) # noqa: S311 (not crypto, just jitter) + time.sleep(random.uniform(0.02, 0.3)) # noqa: S311 if random.random() < ERROR_RATE: # noqa: S311 log.error("work failed: upstream flaked") raise HTTPException(status_code=500, detail="upstream flaked") diff --git a/docs/ONBOARDING.md b/docs/ONBOARDING.md index 7bd8c9a..f4fb320 100644 --- a/docs/ONBOARDING.md +++ b/docs/ONBOARDING.md @@ -1,9 +1,8 @@ -# Sending telemetry from your project +# Sending telemetry from your application -This stack collects logs, traces, and metrics from CML projects and shows -them side by side in one Grafana. You need an endpoint, a token, and a -couple of naming conventions. Then pick the template that matches how your -project runs; a Python/FastAPI service needs no code changes at all. +You need an endpoint, a token, and a few naming conventions. Then pick the +template that matches how your application runs. A Python/FastAPI service +needs no code changes. ## The endpoint @@ -13,11 +12,10 @@ project runs; a Python/FastAPI service needs no code changes at all. | Private network / same host | `:4317` (gRPC) or `:4318` (HTTP) | | Auth | `Authorization: Bearer ` (ask the stack operator) | -The tunnel only routes HTTPS to the collector's HTTP receiver; there is no -public gRPC path. When sending through it, set -`OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf`. gRPC works on private paths -only (VPN/WireGuard, or the same Docker network). Never expose 4317/4318 -directly. +The tunnel routes HTTPS to the collector's HTTP receiver only. Set +`OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf` when you send through it. gRPC +works on private paths only (VPN, WireGuard, or the same Docker network). +Never expose 4317/4318 directly. ## The conventions @@ -25,21 +23,16 @@ directly. (`relab-api`, not `relab-api-prod-2`). Dashboards key on it. - **`env`** is `prod`, `staging`, or `dev`, set as a resource attribute. - **Keep labels low-cardinality.** Prometheus turns every distinct label - value into a series, so user IDs, request IDs, and timestamps do not - belong in resource attributes or metric labels. Put them in the log line - or in span attributes: still filterable at query time, without the storage - blowing up. -- **In Loki, only the identity labels are index labels**: `service.name`, - `department`, `project`, `env`, `host.name` (the authoritative list lives in - `config/loki.yaml`). Every other attribute, `service.instance.id` - included, is stored as structured metadata, so a query starts from the - stream selector and filters after it: - `{service_name="my-service", env="prod"} | service_instance_id="..."`. - -The Service Health dashboard and the error-rate alert key on the standard -HTTP server metrics (`http_server_request_duration_seconds`), which the -auto-instrumentation below emits out of the box. Traces add per-request -drill-down on top. + value into a series. User IDs, request IDs, and timestamps belong in the + log line or in span attributes, not in resource attributes or metric labels. +- **Loki indexes only the identity labels**: `service.name`, `department`, + `project`, `env`, `host.name` (the list is in `config/loki.yaml`). Every + other attribute is structured metadata. Select the stream first, then + filter: `{service_name="my-service", env="prod"} | service_instance_id="..."`. + +The Service Health dashboard and the error-rate alert read the standard HTTP +server metrics (`http_server_request_duration_seconds`). The +auto-instrumentation below emits them. ## Template 1: Python/FastAPI, zero code changes @@ -59,14 +52,13 @@ export OTEL_SEMCONV_STABILITY_OPT_IN=http opentelemetry-instrument uvicorn app:app --host 0.0.0.0 --port 8000 ``` -That is the whole integration: traces, RED metrics, and logs carrying their -trace context, with no OTel code in the app. A working example is this -repo's [`demo/`](../demo/) service plus -[`compose.demo.yml`](../compose.demo.yml). +That gives traces, RED metrics, and logs with trace context, with no OTel +code in the app. A working example is this repo's [`demo/`](../demo/) service +plus [`compose.demo.yml`](../compose.demo.yml). ## Template 2: any language, plain OTLP -Every OpenTelemetry SDK understands the same four environment variables: +Every OpenTelemetry SDK reads the same four environment variables: ```sh OTEL_SERVICE_NAME=my-service @@ -75,12 +67,10 @@ OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer " OTEL_RESOURCE_ATTRIBUTES=env=prod ``` -## Container logs, host metrics, per-container metrics - -One Grafana Alloy agent per host ships everything the application cannot report about -itself: other containers' stdout, host resources, container lifecycle. The agent config -is vendored from `templates/`, not written per project. It rides the same OTLP endpoint -and token as Templates 1 and 2: no second hostname, no second credential. +## Container logs and host metrics -Run `./bootstrap.sh ` on the monitoring host. Follow what it prints. See +An application cannot report other containers' stdout, host resources, or its +own crash loops. One Grafana Alloy agent per host ships those, over the same +endpoint and token. Run `./bootstrap.sh ` on the monitoring +host and follow what it prints. Details are in [templates/README.md](../templates/README.md). diff --git a/docs/RUNBOOK.md b/docs/RUNBOOK.md index f68fc15..f842ef8 100644 --- a/docs/RUNBOOK.md +++ b/docs/RUNBOOK.md @@ -1,13 +1,13 @@ # Runbook All commands run from the repo root on the monitoring host. Start with -`just ps` and the **Stack Health** dashboard; between them they answer most +`just ps` and the **Stack Health** dashboard. Between them they answer most "what is wrong" questions. ![Stack Health dashboard](img/stack-health.png) -In that capture, the red export-failure spike is a Tempo outage and the -gap in the ingest panel is a backup/restore drill. +In that capture, the red export-failure spike is a Tempo outage and the gap in +the ingest panel is a backup/restore drill. ## A service is down or misbehaving @@ -17,12 +17,10 @@ just logs # follow logs (otel-collector, loki, tempo, prometheu just restart ``` -Two alerts point here. `TargetDown` fires after two minutes when -Prometheus cannot scrape a target. Prometheus scrapes every service -(collector, node-exporter, Grafana, Loki, Tempo, and itself), so the alert -names whichever one went quiet. `OtelExportFailures` means the collector is up -but a backend is rejecting its data: read that backend's logs, not the -collector's. +Two alerts point here. `TargetDown` fires after two minutes when Prometheus +cannot scrape a service. It scrapes every service, so the alert names the one +that went quiet. `OtelExportFailures` means the collector is up but a backend +is rejecting its data. Read that backend's logs, not the collector's. ## Disk filling up (`HostDiskSpaceLow`) @@ -32,25 +30,28 @@ Retention is only partially size-bounded: | --- | --- | --- | | Container stdout logs | none | json-file 10m × 3 per service | | Prometheus TSDB | 30d | 15GB (`--storage.tsdb.retention.size`) | -| Loki chunks | 30d | none (Loki cannot cap total size) | +| Loki chunks | 30d | none | | Tempo blocks | 7d | none | -Loki and Tempo have no total-size knob, so the disk alert at 80% is the -backstop. Two earlier warnings sit in front of it: `HostDiskFilling` (a 6h -linear fit says a filesystem is full within 3 days) and -`PrometheusCardinalityHigh` (active series over 100k, ~7x the baseline; -series count, not time, is what grows the TSDB). When one fires, check the -Filesystem panel on Stack Health. Then -free space, or shorten a retention window and restart the affected service -(`retention_period` in `config/loki.yaml`, `block_retention` in -`config/tempo.yaml`, the `--storage.tsdb.retention.*` flags in -`compose.yml`). If disk pressure keeps returning, move Loki and Tempo to -object storage (see the appendix of ADR 0001). - -Memory is bounded per service instead: every service carries a `mem_limit` -in `compose.yml`, sized from observed usage so one runaway component cannot -take the host down. Raise it there if a component legitimately grows into -its ceiling. +Loki and Tempo cannot cap their total size, so the disk alert at 80% is the +backstop. Two warnings fire earlier. `HostDiskFilling` means a 6-hour linear +fit says a filesystem is full within 3 days. `PrometheusCardinalityHigh` +means active series passed 100k, about 7x the baseline. Series count, not +time, is what grows the TSDB. + +When one fires: + +1. Check the Filesystem panel on Stack Health. +2. Free space, or shorten a retention window and restart the service: + `retention_period` in `config/loki.yaml`, `block_retention` in + `config/tempo.yaml`, or the `--storage.tsdb.retention.*` flags in + `compose.yml`. +3. If disk pressure keeps returning, move Loki and Tempo to object storage. + The appendix of ADR 0001 describes the change. + +Memory is bounded per service: every service has a `mem_limit` in +`compose.yml`, sized from observed usage. Raise it there if a component +legitimately grows into its ceiling. ## Backup and restore @@ -60,66 +61,57 @@ just restore backups/monitoring-.tar.gz # stops the stack, wipes volumes, just up ``` -Backups are crash-consistent: restoring one is like recovering from a power -loss, which every component does cleanly via its write-ahead log. Notes: +Backups are crash-consistent. Restoring one is like recovering from a power +loss, which every component handles through its write-ahead log. -- The tarball is mode 0600 and contains secrets (the Grafana database - among them). Copy it off-host over a private channel: a backup on the - disk it protects is a decoration. -- It covers the docker volumes and nothing else. The OpenTofu state for the - Cloudflare edge is not in it (see below). -- During the pause the collector keeps accepting telemetry and buffers it - for five minutes (`retry_on_failure.max_elapsed_time` on every exporter - in `config/otel-collector.yaml`). The queue is file-backed, so - restarting the collector inside that window keeps the buffer. A backup - that runs longer than five minutes still drops data: on large volumes, run - it at a quiet hour. +- The tarball is mode 0600 and contains secrets, including the Grafana + database. Copy it off-host over a private channel. +- It covers the docker volumes only. The OpenTofu state for the Cloudflare + edge is not in it. See "Where every secret lives" below. +- During the pause the collector buffers incoming telemetry for five minutes + (`retry_on_failure.max_elapsed_time` in `config/otel-collector.yaml`). A + backup that runs longer drops data. On large volumes, run it at a quiet + hour. -Worst-case loss equals the interval between backups. A daily cron on the -host is the intended setup. +Worst-case loss equals the interval between backups. Run it from a daily cron +on the host. ## Rotating secrets - **OTLP token:** set the new `OTLP_AUTH_TOKEN` in `.env`, run `docker compose up -d otel-collector`, then update every sender's - `OTEL_EXPORTER_OTLP_HEADERS`. Senders still on the old token get 401s - (export errors on their side) until updated. A running demo overlay counts - as a sender: re-run `just demo` to recreate it with the new token. - The token is shared, and the collector does not check `project`/`env` - against the sender. Every project host is therefore trusted with every - other project's telemetry identity. A compromised host could spoof another - project's labels, and so quiet that project's silence alarm. Per-project - tokens with a collector-side identity check are the upgrade if that trust - ever stops being acceptable. -- **Tunnel token:** the tunnel is OpenTofu-managed, so read the token back - from there, not from the dashboard. Rotate the tunnel secret in Cloudflare - Zero Trust. Then run `cd infra && tofu apply`, which refreshes the token - data source. Then read the new value with `tofu output -raw tunnel_token`. - To rotate entirely from code instead, expect ingestion and Grafana to be - unreachable for the minute or so it takes. - `tofu apply -replace=cloudflare_zero_trust_tunnel_cloudflared.monitoring` - builds a new tunnel and repoints both CNAMEs at it. Either way: new token - into `CLOUDFLARE_TUNNEL_TOKEN` in `.env`, then `just up`. + `OTEL_EXPORTER_OTLP_HEADERS`. Senders on the old token get 401s until + updated. A running demo overlay is a sender too: re-run `just demo`. + The token is shared, and the collector does not check `project` or `env` + against the sender. Every project host can therefore spoof another + project's labels. +- **Tunnel token:** rotate the tunnel secret in Cloudflare Zero Trust, then + run `cd infra && tofu apply` to refresh the token data source, then read + the new value with `tofu output -raw tunnel_token`. To rotate from code + instead, run `tofu apply -replace=cloudflare_zero_trust_tunnel_cloudflared.monitoring`. + That builds a new tunnel and repoints both CNAMEs at it, and ingestion and + Grafana are unreachable for about a minute. Either way, put the new token + in `CLOUDFLARE_TUNNEL_TOKEN` in `.env`, then `just up`. - **Grafana admin password:** change `GRAFANA_ADMIN_PASSWORD` in `.env`, then `docker compose up -d grafana`. -- **Alert webhook and heartbeat URLs:** both are capability URLs, so the URL - is the credential. Mint a new topic or check at the provider, put it in - `ALERT_WEBHOOK_URL` / `HEARTBEAT_URL`, then `docker compose up -d grafana`; - Grafana reads the contact points only at startup. -- **healthchecks.io API key:** regenerate it in the project's settings, set - `HEALTHCHECKS_API_KEY` in `.env`. Only `bootstrap.sh` reads it, so nothing - needs a restart. -- **Cloudflare API token:** it is never stored here; create a new one with the - same three permissions, revoke the old one, export the new value before the - next `tofu` run. +- **Alert webhook and heartbeat URLs:** the URL is the credential. Mint a new + topic or check at the provider, put it in `ALERT_WEBHOOK_URL` or + `HEARTBEAT_URL`, then `docker compose up -d grafana`. Grafana reads the + contact points only at startup. +- **healthchecks.io API key:** regenerate it in the project's settings and + set `HEALTHCHECKS_API_KEY` in `.env`. Only `bootstrap.sh` reads it, so + nothing restarts. +- **Cloudflare API token:** it is never stored here. Create a new one with + the same three permissions, revoke the old one, and export the new value + before the next `tofu` run. ### Where every secret lives -Three files hold everything, all gitignored. `just backup` archives none of -them, but the tarball does contain Grafana's database, and Grafana stores the -expanded contact points there: the webhook and heartbeat URLs are inside every +Three gitignored files hold everything. `just backup` archives none of them, +but the tarball contains Grafana's database, and Grafana stores the expanded +contact points there. So the webhook and heartbeat URLs are inside every backup. Copy `.env` and `infra/terraform.tfstate` off-host together with the -backups and treat all three the way you treat the originals. +backups and treat all three the same way. | Secret | Lives in | Comes from | | --- | --- | --- | @@ -133,37 +125,33 @@ backups and treat all three the way you treat the originals. | `CLOUDFLARE_API_TOKEN` | your shell, per session | Cloudflare dashboard | `infra/terraform.tfvars` holds identifiers only (account, zone, domain, the -Access email list) and is gitignored for privacy, not because it holds a +Access email list). It is gitignored for privacy, not because it holds a credential. ## Alert delivery -Grafana evaluates and delivers the rules; there is no Alertmanager. Rules, -contact points and the routing tree are provisioned from -`config/grafana/alerting/`, so the UI shows them read-only: edit the YAML. -Two environment variables control where notifications go: +Grafana evaluates and delivers the rules. Rules, contact points, and the +routing tree are provisioned from `config/grafana/alerting/`, so the UI shows +them read-only. Edit the YAML. -- `ALERT_WEBHOOK_URL` receives all alerts (any webhook: ntfy, Slack, …). -- `HEARTBEAT_URL` receives the always-firing `Watchdog` every five - minutes. Point it at a dead man's switch (e.g. healthchecks.io) that - raises the alarm when pings **stop**: that is the "monitoring host is - dead" signal nothing inside the host can send. +- `ALERT_WEBHOOK_URL` receives all alerts. +- `HEARTBEAT_URL` receives the always-firing `Watchdog` every five minutes. + Point it at a dead man's switch that raises the alarm when pings stop. -Leaving them empty is not a safe default: delivery fails silently while the -heartbeat keeps pinging, so the switch reads healthy and every real alert is -dropped. With the tunnel overlay active, `just up` refuses to start without +Leaving either empty is not safe. Delivery fails silently while the heartbeat +keeps pinging, so the switch reads healthy and every real alert is dropped. +With the tunnel overlay active, `just up` refuses to start without `ALERT_WEBHOOK_URL`, and `AlertDeliveryFailing` fires on a failing notifier. After changing either variable, run `docker compose up -d grafana`. A plain -`restart` silently keeps the old environment: compose rebuilds a container's +`restart` keeps the old environment. Compose rebuilds a container's environment only on `up`. ## Changing the Cloudflare edge -The tunnel, its ingress rules, both DNS records, and the Cloudflare Access -policy that fronts Grafana are all OpenTofu in `infra/`. Change them there, -not in the Zero Trust dashboard: the next apply reverts anything clicked in -by hand. +The tunnel, its ingress rules, both DNS records, and the Access policy in +front of Grafana are all OpenTofu in `infra/`. Change them there, not in the +Zero Trust dashboard. The next apply reverts anything clicked in by hand. ```sh just infra-validate # tofu init + validate, in a container @@ -173,66 +161,60 @@ cd infra && tofu apply - **Granting or revoking Grafana access:** edit `grafana_allowed_emails` in `infra/terraform.tfvars` and apply. That list is the entire allowlist. A - removed address keeps working until their Access session expires (24h), - so for an urgent revocation also revoke the session in Zero Trust. At - least one address has to remain; the variable's validation rejects an empty - list, which would lock everyone out. Before adding people, check your plan's - Zero Trust seat limit in the Cloudflare dashboard. Cloudflare does not - document the free-plan figure. + removed address keeps working until its Access session expires (24h). For + an urgent revocation, also revoke the session in Zero Trust. The variable + rejects an empty list, which would lock everyone out. Before adding people, + check your plan's Zero Trust seat limit in the Cloudflare dashboard. - **Per-user Grafana logins:** by default everyone who clears Access shares - the one admin password. Set `GRAFANA_JWT_AUTH=true`, + the admin password. Set `GRAFANA_JWT_AUTH=true`, `CF_ACCESS_TEAM_DOMAIN=`, and `CF_ACCESS_AUD` (from - `tofu output -raw grafana_access_aud`) in `.env` to make Grafana verify the - Access JWT instead: each address signs in as itself, and new ones land on - the org's default role (Viewer). The aud pin is required because the JWK - set is team-wide. Without it, a token minted for any other Access app in - the team would be accepted here too. The `just up` exposure guards refuse - to start JWT auth without both values. + `tofu output -raw grafana_access_aud`) in `.env`. Grafana then verifies + the Access JWT, and each address signs in as itself with the Viewer role. + The aud pin is required. The JWK set is team-wide, so without it a token + minted for any other Access app in the team would be accepted here. - **Adding a hostname:** add an `ingress` entry pointing at the service's container port, plus a matching `cloudflare_dns_record`. The catch-all `http_status:404` entry stays last, or it swallows everything after it. -- **First apply against an edge built by hand** (a tunnel, DNS records, or - Access app that already exist in the dashboard): an empty state plans them - as "create", and applying that mints a second tunnel and a duplicate Access - app. Run `infra/generate-imports.sh > infra/imports.tf` first, check the - plan reads 0 to add for the imported resources, apply, then delete - `imports.tf`; it is a one-time instruction and gitignored. -- **State lives on this host only, and it is a secret.** State stores the - tunnel secret and every API response in plain text, so whoever can read it - can run the tunnel. `infra/terraform.tfstate` is gitignored, and - `just backup` does not touch it. Copy it off-host next to the backups, with - the same care as `.env`. Losing it orphans the Cloudflare resources: they - keep running, but the next apply creates duplicates, and recovery is - `tofu import` by hand. +- **First apply against an edge built by hand:** an empty state plans the + existing tunnel, DNS records, and Access app as "create", and applying + that mints duplicates. Run `infra/generate-imports.sh > infra/imports.tf` + first, check the plan reads 0 to add for the imported resources, apply, + then delete `imports.tf`. +- **State lives on this host only, and it is a secret.** It stores the + tunnel secret and every API response in plain text. Copy + `infra/terraform.tfstate` off-host next to the backups. Losing it orphans + the Cloudflare resources. They keep running, but the next apply creates + duplicates, and recovery is `tofu import` by hand. ## Upgrading images -Dependabot opens PRs that bump the pinned versions, and CI runs -`just validate` and `just smoke` on each one. The validators (promtool, otelcol) read their image -versions from `compose.yml`, so every bump is checked with the exact -binaries the stack will run. A new version that changes its config syntax -fails CI before it reaches the host. Patch bumps arrive grouped, one PR for -the hub images and one for the spoke images; a minor or major comes on its -own, so a red PR names the one image that broke. +Dependabot opens PRs that bump the pinned versions, and CI runs `just validate` +and `just smoke` on each one. The validators read their image versions from +`compose.yml`, so every bump is checked with the exact binaries the stack +will run. Patch bumps arrive grouped. A minor or major comes on its own, so a +red PR names the one image that broke. Dependabot also watches the Cloudflare provider in `infra/`. Those PRs need -one manual step: it bumps the constraint in `main.tf` but not the recorded -hashes in `.terraform.lock.hcl`. Check the branch out, run -`cd infra && tofu init -upgrade`, then `just infra-validate` and `tofu plan` -against the real account. Validation proves the syntax parses; only a plan -proves the provider still maps the config to the same resources. - -Nothing watches the tool images pinned in the `justfile`: yamllint, -actionlint, shellcheck, ruff, gitleaks, OpenTofu, jq, the Alloy validator, -and the alpine that backup, restore, and the queue-volume setup run in. No -Dependabot ecosystem covers a justfile, so bump those by hand. - -After merging, on the host: `git pull && just pull && just up`. Coming from -a release that still ran Alertmanager, its volume outlives the service: -`docker volume rm monitoring_alertmanager_data` once the new stack is up, and -`config/alertmanager.yaml` can go with it. +one manual step: it bumps the constraint in `main.tf` but not the hashes in +`.terraform.lock.hcl`. Check the branch out, run `cd infra && tofu init +-upgrade`, then `just infra-validate` and `tofu plan` against the real +account. Only a plan proves the provider still maps the config to the same +resources. + +Nothing watches the tool images pinned in the `justfile` (yamllint, +actionlint, shellcheck, ruff, gitleaks, OpenTofu, jq, Alloy, alpine). Bump +those by hand. + +After merging, on the host: + +```sh +git pull && just pull && just up +``` Use `just up`, not `docker compose up -d`. The recipe first chowns the -collector's queue volume to uid 10001. Where that volume is new, a raw compose -up leaves it root-owned and the collector crash-looping on a queue directory it -cannot write. +collector's queue volume to uid 10001. A raw compose up leaves a new volume +root-owned and the collector crash-looping. + +Coming from a release that still ran Alertmanager, its volume outlives the +service: run `docker volume rm monitoring_alertmanager_data` once the new +stack is up. diff --git a/infra/generate-imports.sh b/infra/generate-imports.sh index bff8a44..501b3ae 100755 --- a/infra/generate-imports.sh +++ b/infra/generate-imports.sh @@ -1,22 +1,17 @@ #!/usr/bin/env bash # Emit OpenTofu `import` blocks for the Cloudflare resources that already exist. # -# This root was added after the edge already existed, so its first plan against an -# empty state says "create" for objects that are already serving traffic. Applying -# that plan mints a SECOND tunnel beside `cml-monitoring`, and DNS records that fight -# the live ones. Import blocks make the adoption reviewable: read the generated file, -# then the plan, and only then apply. +# A first plan against an empty state says "create" for objects already serving +# traffic, and applying it mints a second tunnel and duplicate DNS records. Import +# blocks make the adoption reviewable: read the generated file, then the plan, then +# apply. # # The ingestion record was renamed from `otlp.` to `otel.` (2026-09). -# On a state that predates the rename, the block below imports the live record as -# `cloudflare_dns_record.otel`, so the apply RENAMES it in place instead of creating a -# second one. That is also why main.tf has no `moved` block: nothing was ever in state -# under the old resource name. A current state already holds `otel.`. +# On a state that predates the rename, the live record is imported as +# `cloudflare_dns_record.otel`, so the apply renames it in place. # -# The generated imports.tf is a throwaway, NOT something to commit: it names one -# account's resource ids and is meaningless after the apply that consumes it. Delete it -# once the apply has succeeded. Import blocks are a one-time instruction, and leaving -# them in place re-runs them on every plan. +# The generated imports.tf is a throwaway: delete it after the apply, or every plan +# re-runs the imports. # # Usage (run in this directory, with terraform.tfvars already filled in): # export CLOUDFLARE_API_TOKEN=... # Tunnel:Read, DNS:Read, Access: Apps and Policies:Read @@ -33,8 +28,7 @@ die() { } api() { - # The token rides a curl config read from stdin, not argv: /proc//cmdline is - # readable by every local user for the life of each call. + # Token via curl config on stdin, not argv: /proc//cmdline is world-readable. curl -fsS --config - "https://api.cloudflare.com/client/v4/$1" \ <<<"header = \"Authorization: Bearer ${CLOUDFLARE_API_TOKEN}\"" } @@ -42,9 +36,7 @@ api() { command -v jq >/dev/null || die "jq is required" : "${CLOUDFLARE_API_TOKEN:?is not set}" -# Read the ids from terraform.tfvars, which already holds them. A second source of the -# same value is a second chance to get it wrong. An exported environment variable still -# wins. +# Read the ids from terraform.tfvars; an exported TF_VAR_* still wins. [[ -f terraform.tfvars ]] || die "no terraform.tfvars here; copy terraform.tfvars.example and fill it in" tfvar() { sed -n "s/^[[:space:]]*$1[[:space:]]*=[[:space:]]*\"\([^\"]*\)\".*/\1/p" terraform.tfvars | tail -1 @@ -53,8 +45,7 @@ account="${TF_VAR_account_id:-$(tfvar account_id)}" zone="${TF_VAR_zone_id:-$(tfvar zone_id)}" domain="${TF_VAR_domain:-$(tfvar domain)}" -# The example file's placeholders are valid-looking strings, so a plan run against them -# looks plausible. Catch them here, where the message can name the line to edit. +# The example file's placeholders are valid-looking strings; catch them here. for pair in "account_id:$account" "zone_id:$zone" "domain:$domain"; do value="${pair#*:}" [[ -n "$value" ]] || die "${pair%%:*} is empty in terraform.tfvars" @@ -63,16 +54,14 @@ for pair in "account_id:$account" "zone_id:$zone" "domain:$domain"; do esac done -# One lookup per resource kind. Each fails loudly when the resource is absent: a silently -# skipped import comes back as a "create" in the plan. +# A silently skipped import comes back as a "create" in the plan, so each lookup +# fails loudly when the resource is absent. lookup_dns_record() { local hostname="$1" id - # Type-filtered: this root manages CNAMEs, and a name can also carry TXT records. - # An unfiltered .result[0] could bind one of those, and the apply would rewrite it - # into a proxied CNAME, destroying the TXT record and leaving the CNAME unmanaged. - # Two statements, not one pipeline: this function only runs inside `if`/`||`, where - # errexit is off, so a failed API call would otherwise fall through and be reported - # as "no record", the one answer that sends the operator off to create a duplicate. + # Type-filtered: an unfiltered .result[0] could bind a TXT record on the same + # name, and the apply would rewrite it into a CNAME. Two statements, not one + # pipeline: this runs inside `if`/`||`, where errexit is off, and a failed API + # call must not be reported as "no record". local body body="$(api "zones/$zone/dns_records?name=$hostname&type=CNAME")" || die "DNS lookup for $hostname failed (token lacks DNS:Read?)" id="$(jq -r '.result[0].id // empty' <<<"$body")" @@ -84,8 +73,7 @@ tunnel_name="${MONITORING_TUNNEL_NAME:-cml-monitoring}" tunnels="$(api "accounts/$account/cfd_tunnel?is_deleted=false")" tunnel_id="$(jq -r --arg name "$tunnel_name" '.result[] | select(.name == $name) | .id' <<<"$tunnels" | head -1)" if [[ -z "$tunnel_id" ]]; then - # List what IS there. The tunnel is rarely absent; it is usually called something - # else, and then the fix is one flag. + # The tunnel is usually present under another name; list what is there. echo "error: no tunnel named $tunnel_name in account $account" >&2 echo "tunnels that do exist in this account:" >&2 jq -r '.result[]? | " \(.name)\t\(.id)\tconnections=\(.connections | length)"' <<<"$tunnels" >&2 @@ -95,8 +83,7 @@ fi grafana_record="$(lookup_dns_record "grafana.$domain")" || die "no CNAME found for grafana.$domain" -# Still otlp. before the rename, otel. after it (or on a re-run). Try the new name first, -# so a second run is a no-op instead of resurrecting the old record. +# otel. after the rename, otlp. before it. New name first, so a re-run is a no-op. for host in "otel.$domain" "otlp.$domain"; do if otel_record="$(lookup_dns_record "$host")"; then ingestion_host="$host" @@ -105,13 +92,9 @@ for host in "otel.$domain" "otlp.$domain"; do done [[ -n "${ingestion_host:-}" ]] || die "no CNAME found for otel.$domain or otlp.$domain" -# Unlike the tunnel and the DNS records, the Access app may legitimately not exist, and -# then the apply SHOULD create it: an unprotected Grafana hostname is what this root -# exists to close. So warn, omit the import, and let the plan create it. -# -# Checked at both scopes. Apps predating account-scoped Access live under the zone, and -# one found there cannot be adopted by this resource as written (it is configured with -# account_id), so that case gets its own message instead of a silent create. +# The Access app may legitimately not exist; then the apply should create it. A +# zone-scoped app (predating account-scoped Access) cannot be adopted by the +# account-scoped resource, so that case gets its own message. access_apps="$(api "accounts/$account/access/apps?per_page=100")" access_app_id="$(jq -r --arg d "grafana.$domain" '.result[] | select(.domain == $d) | .id' <<<"$access_apps" | head -1)" if [[ -z "$access_app_id" ]]; then @@ -128,9 +111,8 @@ if [[ -z "$access_app_id" ]]; then jq -r '.result[]? | " \(.name)\t\(.domain)"' <<<"$access_apps" >&2 fi -# The policy may also be absent: an app built in the dashboard usually carries an INLINE -# policy, which has no id to import. The apply then creates the reusable policy this root -# declares and reattaches the app to it. +# An app built in the dashboard usually carries an inline policy with no id to +# import; the apply then creates the reusable one and reattaches the app. access_policies="$(api "accounts/$account/access/policies")" access_policy_id="$(jq -r '.result[] | select(.name == "monitoring: allowed emails") | .id' <<<"$access_policies" | head -1)" diff --git a/infra/main.tf b/infra/main.tf index ce25f21..b083128 100644 --- a/infra/main.tf +++ b/infra/main.tf @@ -1,6 +1,5 @@ -# Cloudflare edge for the monitoring stack: the tunnel, its ingress rules, -# and DNS. This is the only part of the stack that otherwise lives as -# click-ops in the Zero Trust dashboard. +# Cloudflare edge for the monitoring stack: the tunnel, its ingress rules, DNS, +# and the Access policy in front of Grafana. # # Bootstrap (owner-run, once): # cp terraform.tfvars.example terraform.tfvars # then fill it in @@ -11,8 +10,7 @@ # tofu apply && rm imports.tf # tofu output -raw tunnel_token # → CLOUDFLARE_TUNNEL_TOKEN in ../.env # -# State is local (infra/terraform.tfstate, gitignored): one host, one operator. -# Move it to R2 the day a second operator exists. +# State is local (infra/terraform.tfstate, gitignored) and holds the tunnel secret. terraform { required_version = ">= 1.8" @@ -54,8 +52,7 @@ variable "grafana_allowed_emails" { resource "cloudflare_zero_trust_tunnel_cloudflared" "monitoring" { account_id = var.account_id - # Must match the live tunnel's name: this root adopts it rather than creating it - # (see generate-imports.sh), and a different name here renames it on apply. + # Must match the live tunnel's name; a different name here renames it on apply. name = "cml-monitoring" config_src = "cloudflare" } @@ -92,9 +89,8 @@ resource "cloudflare_dns_record" "grafana" { ttl = 1 } -# Renamed from `otlp` (2026-09). There is no `moved` block because nothing was ever -# in state under the old name: the live record is adopted straight into this address -# by ./generate-imports.sh, and the apply then renames it in place. +# Renamed from `otlp` (2026-09). No `moved` block: nothing was ever in state under +# the old name, and generate-imports.sh adopts the live record straight into this one. resource "cloudflare_dns_record" "otel" { zone_id = var.zone_id name = "otel.${var.domain}" @@ -104,10 +100,8 @@ resource "cloudflare_dns_record" "otel" { ttl = 1 } -# Cloudflare Access in front of Grafana: email one-time-PIN at the edge, so -# the public hostname never reaches Grafana's login page unauthenticated. -# The ingestion hostname is NOT behind Access; machines authenticate with the -# bearer token instead. +# Email one-time PIN in front of Grafana. The ingestion hostname is not behind +# Access; machines authenticate with the bearer token. resource "cloudflare_zero_trust_access_application" "grafana" { account_id = var.account_id name = "Grafana (monitoring)" diff --git a/justfile b/justfile index 399f261..5f53162 100644 --- a/justfile +++ b/justfile @@ -1,11 +1,9 @@ set dotenv-load -# Stateful services. Their volumes are __data, which is how -# _backup and _restore find them for whichever compose project they act on. +# Stateful services; their volumes are __data. stateful := "grafana prometheus loki tempo" -# Helper and lint images, pinned once. `lint` pulls the lint set in parallel -# up front: serial first-use pulls are most of a cold run. +# Helper and lint images, pinned once. alpine := "alpine:3.24@sha256:28bd5fe8b56d1bd048e5babf5b10710ebe0bae67db86916198a6eec434943f8b" jq := "ghcr.io/jqlang/jq:1.8.1" yamllint := "pipelinecomponents/yamllint:0.35.13" @@ -18,60 +16,48 @@ gitleaks := "zricethezav/gitleaks:v8.30.1@sha256:c00b6bd0aeb3071cbcb79009cb16a60 alloy := "grafana/alloy:v1.18.1@sha256:0f4434c92b3e6cdac38bb129b344e1790c246f7b6e2eaffcc16a5fa363240e33" lint_images := jq + " " + yamllint + " " + yamlfmt + " " + actionlint + " " + shellcheck + " " + ruff + " " + tofu + " " + gitleaks + " " + alloy -# `demo` and `smoke` each bring up a throwaway copy of the core stack, so both -# run under their own compose project and their own Grafana port (see -# compose.sandbox.yml). Neither can adopt, recreate or delete the containers and -# volumes of a stack already running on this host: `just demo` on the production -# box spins up its own Grafana next to the real one instead of joining it. The -# explicit -f list also keeps a host's COMPOSE_FILE (tunnel) out of both. +# `demo` and `smoke` each run a throwaway copy of the core stack under their own +# compose project and Grafana port (compose.sandbox.yml), so neither can touch a +# stack already running on this host. The explicit -f list keeps the host's +# COMPOSE_FILE out of both. demo_project := "monitoring-demo" demo_port := env("DEMO_PORT", "3002") compose_demo := "SANDBOX_PORT=" + demo_port + " docker compose -p " + demo_project + " -f compose.yml -f compose.demo.yml -f compose.sandbox.yml" -# What compose will actually name the project for the up-paths, so the queue -# volume gets chowned where the collector will look for it, and backup/restore -# mount the volumes the running stack uses. +# The project name compose will use, so the queue volume and backups target +# the running stack's volumes. core_project := env("COMPOSE_PROJECT_NAME", "monitoring") -# Same isolation for smoke, on its own project and port so a smoke run and a -# demo stack can also coexist. The smoke stack boots in production shape: -# Grafana with JWT auth on (the Cloudflare Access path, otherwise only ever -# schema-checked) and fixed notification URLs so the contact-point assertion -# can check exact values instead of "not empty". The team domain has a dot, -# which no real Zero Trust team name can, so the JWK URL can never resolve to -# a team someone registers. +# The smoke stack boots in production shape: JWT auth on and fixed notification +# URLs, so smoke.sh can assert exact values. The team domain has a dot, which no +# real Zero Trust team name can, so the JWK URL can never resolve to a team +# someone registers. smoke_project := "monitoring-smoke" smoke_port := env("SMOKE_PORT", "3001") smoke_env := "SANDBOX_PORT=" + smoke_port + " GRAFANA_JWT_AUTH=true CF_ACCESS_TEAM_DOMAIN=smoke.invalid CF_ACCESS_AUD=smoke ALERT_WEBHOOK_URL=https://smoke.invalid/alerts HEARTBEAT_URL=https://smoke.invalid/heartbeat" compose_smoke := smoke_env + " docker compose -p " + smoke_project + " -f compose.yml -f compose.sandbox.yml" -# dashboards/*.json as the paths they get when mounted at /dashboards. 2>/dev/null -# so an empty dashboards/ doesn't abort every recipe in this file, including the -# teardown ones; `lint` refuses the empty list instead. +# dashboards/*.json as mounted at /dashboards. 2>/dev/null so an empty +# dashboards/ doesn't abort every recipe; `lint` refuses the empty list instead. dash_paths := `ls dashboards/*.json 2>/dev/null | sed 's|^dashboards|/dashboards|' | tr '\n' ' '` # List the recipes. default: @just --list -# The collector's queue volume must be writable by the image's uid 10001, but -# a fresh named volume is root-owned and the image is distroless (no chown at -# startup possible). Idempotent, so every up-path just runs it. Takes the -# project name because smoke brings the stack up under a different one. +# A fresh named volume is root-owned and the collector image is distroless, so +# chown its queue volume to uid 10001 here. Idempotent; every up-path runs it. _queue-volume project: @docker volume create {{project}}_otel_queue > /dev/null @docker run --rm --network none -v {{project}}_otel_queue:/q {{alpine}} chown 10001:10001 /q -# Overlays are host config: COMPOSE_FILE in .env names the file set (see -# .env.example), and every recipe here (up, down, logs, ps, backup) acts on -# that same set. With the tunnel overlay active, this refuses to start until the -# exposure guards pass. +# COMPOSE_FILE in .env names the overlays. With the tunnel overlay active, this +# refuses to start until the exposure guards pass. # Start the stack (Grafana at http://localhost:3000). up: (_queue-volume core_project) _guard-if-exposed docker compose up -d -# The exposure guards, but only when the tunnel overlay is in play. Shared by -# every recipe that can put a service back on the tunnel. +# The exposure guards, only when the tunnel overlay is in play. _guard-if-exposed: @case "${COMPOSE_FILE:-}" in *compose.tunnel.yml*) just _expose-guards;; esac @@ -89,20 +75,14 @@ _expose-guards: down: docker compose down --remove-orphans -# See compose.demo.yml; watch it arrive at http://localhost:3002 (DEMO_PORT): -# its own Grafana, not the one `just up` serves on :3000. -# Core stack plus a demo telemetry source, isolated from any running stack. +# Core stack plus a demo telemetry source, isolated from any running stack (:3002). demo: (_queue-volume demo_project) {{compose_demo}} up -d --build -# Removes only the demo services; the demo project's own core stack keeps -# running. `just demo-destroy` takes the whole thing down. -# Stop the demo telemetry source. +# Stop the demo telemetry source; the demo project's core stack keeps running. demo-down: {{compose_demo}} rm -sf demo-api demo-load -# Safe: -p scopes it to the demo project, so it cannot touch a real stack on -# the same host. # Tear down the whole demo stack and its throwaway volumes. demo-destroy: {{compose_demo}} down --remove-orphans --volumes @@ -115,7 +95,6 @@ logs service="": ps: docker compose ps -# Targets the COMPOSE_FILE set; demo services are recreated with `just demo`. # Restart one service. restart service: _guard-if-exposed docker compose restart {{service}} @@ -128,79 +107,58 @@ pull: tail service: docker compose logs -f --no-log-prefix {{service}} | jq -R 'fromjson? // .' -# Every check in this file runs in a container: no host installs, no network. -# `lint` is the static half in small tool images; `validate` runs the stack's -# own images against its configs. Both are seconds warm, so the pre-push hook -# (`just hooks`) runs both; CI puts `lint` on its own job so a lint failure -# never waits on the stack images, which the smoke job pulls anyway. +# Every check runs in a container: no host installs, no network. # Validate every config in the repo. check: lint validate # Static checks in tool images (~280 MB cold, seconds warm). lint: @printf '%s\n' {{lint_images}} | xargs -P 8 -n 1 docker pull -q >/dev/null - # `-f compose.yml` on the first line, not the host's COMPOSE_FILE: on a - # tunnel host that would validate the tunnel set with no dummy token, so - # `just lint` would mean something different there than in CI. + # Explicit -f, not the host's COMPOSE_FILE, so lint means the same here as in CI. docker compose -f compose.yml config -q CLOUDFLARE_TUNNEL_TOKEN=dummy docker compose -f compose.yml -f compose.tunnel.yml config -q {{compose_demo}} config -q - # Also the JWT interpolation in compose.yml: compose_smoke turns it on. + # compose_smoke turns the JWT interpolation on. {{compose_smoke}} config -q - # The spoke overlays every project host layers onto its own compose.yml; a - # bad interpolation or a network they forget to declare otherwise first - # fails on a client machine, after vendoring. + # The spoke overlays, which otherwise first fail on a project host after vendoring. ENVIRONMENT=dummy PROJECT=dummy COMPOSE_PROJECT_NAME=dummy OTEL_EXPORTER_OTLP_ENDPOINT=https://dummy OTLP_AUTH_TOKEN=dummy docker compose -f templates/compose.telemetry.yml -f templates/compose.telemetry.gpu.yml config -q - # The exposure guards, both ways: a fully set .env must pass, and each - # documented default must be refused on its own. Nothing else runs them: - # CI never sets COMPOSE_FILE to the tunnel overlay. + # The exposure guards, both ways: a fully set .env passes, and each + # documented default is refused on its own. @good="OTLP_AUTH_TOKEN=t GRAFANA_ADMIN_PASSWORD=p GRAFANA_ROOT_URL=https://g.example GRAFANA_COOKIE_SECURE=true GRAFANA_JWT_AUTH=true CF_ACCESS_TEAM_DOMAIN=d CF_ACCESS_AUD=a HEARTBEAT_URL=https://h ALERT_WEBHOOK_URL=https://w"; \ env $good just _expose-guards || { echo "error: exposure guards rejected a fully set environment" >&2; exit 1; }; \ for bad in OTLP_AUTH_TOKEN=local-dev-token GRAFANA_ADMIN_PASSWORD=change-me GRAFANA_ROOT_URL=http://g.example GRAFANA_COOKIE_SECURE=false CF_ACCESS_AUD= ALERT_WEBHOOK_URL=; do \ ! env $good $bad just _expose-guards 2>/dev/null || { echo "error: exposure guards accepted $bad" >&2; exit 1; }; \ done - # line-length at 120, not the default 80: digest-pinned image refs need - # ~130 but count as non-breakable mappings. The rendered project-*/coverage - # rules are ignored: bootstrap.sh writes them, and their expr lines grow - # with every onboarded project. The templates they come from are checked - # by rendering them through bootstrap.sh itself into a scratch dir, so a - # template edit Grafana would reject fails here and not on the next - # onboarding. + # The rendered project-*/coverage rules are skipped (their expr lines grow + # with every project); their templates are checked by rendering them into + # a scratch dir instead. docker run --rm --network none -v .:/code:ro {{yamllint}} yamllint -d '{extends: relaxed, rules: {line-length: {max: 120, allow-non-breakable-inline-mappings: true}}, ignore: [.git/, backups/, infra/.terraform/, config/grafana/alerting/project-*.yaml, config/grafana/alerting/coverage.yaml]}' . @d=$(mktemp -d) && BOOTSTRAP_OUT_DIR="$d" ./bootstrap.sh dummy dummy >/dev/null && docker run --rm --network none -v "$d":/code:ro {{yamllint}} yamllint -d '{extends: relaxed, rules: {line-length: disable}}' .; rc=$?; rm -rf "$d"; exit $rc docker run --rm --network none -v .:/repo:ro -w /repo {{actionlint}} -color docker run --rm --network none -v .:/mnt:ro {{shellcheck}} bootstrap.sh scripts/smoke.sh templates/run_scheduled.sh infra/generate-imports.sh docker run --rm --network none -v ./demo:/demo:ro {{ruff}} check --no-cache /demo docker run --rm --network none -v ./demo:/demo:ro {{ruff}} format --check --no-cache /demo - # Formatting is enforced, not just linted: `just fmt` is the fix. + # `just fmt` is the fix. docker run --rm --network none -v .:/code:ro -w /code {{yamlfmt}} -lint . docker run --rm --network none -v ./infra:/infra:ro -w /infra {{tofu}} fmt -check - # Dashboards: valid JSON, and every datasource they name is one that - # datasources.yaml provisions (plus Grafana's built-in). A typo here - # provisions fine and renders empty panels. + # Dashboards: valid JSON, and every datasource uid they name is provisioned. + # A typo provisions fine and renders empty panels. @[ -n "{{dash_paths}}" ] || { echo "error: no dashboards/*.json to check" >&2; exit 1; } docker run --rm --network none -v ./dashboards:/dashboards:ro {{jq}} empty {{dash_paths}} @bad=$(docker run --rm --network none -v ./dashboards:/dashboards:ro {{jq}} -r '.. | objects | select(has("datasource")) | .datasource | (if type == "object" then .uid else . end) | strings' {{dash_paths}} | sort -u | grep -vxF "$(sed -n 's/^ *uid: *//p' config/grafana/datasources.yaml; echo grafana)"); \ [ -z "$bad" ] || { echo "error: dashboards reference datasource uids that are not provisioned:" $bad >&2; exit 1; } - # Secrets that reached git history. Scans commits, not the working tree, so - # it sees exactly what is in the repo and never the gitignored .env. A - # working-tree scan flags .env's real tokens and fails on every dev machine. - # Needs full history: a shallow CI clone has one commit and passes vacuously. + # Secrets in git history. Scans commits, not the working tree, so the + # gitignored .env never trips it. Needs full history (see ci.yml). docker run --rm --network none -v .:/repo:ro {{gitleaks}} git --redact --no-banner /repo - # The one config vendored verbatim onto every project host; a syntax error - # here otherwise first surfaces as a crash-looping agent on a client machine. - # Image ref matches templates/compose.telemetry.yml; keep them in step. In - # `lint`, not `validate`: the smoke job pulls nothing else this size. + # The agent config every project host vendors. Image ref matches + # templates/compose.telemetry.yml; keep them in step. docker run --rm --network none -v ./templates/alloy/config.alloy:/etc/alloy/config.alloy:ro -e COMPOSE_PROJECT_NAME=dummy -e ENVIRONMENT=dummy -e PROJECT=dummy -e OTEL_EXPORTER_OTLP_ENDPOINT=https://dummy -e OTLP_AUTH_TOKEN=dummy -e TELEMETRY_EDGE_KEY= {{alloy}} validate /etc/alloy/config.alloy -# Image ref of one compose.yml service, so the validators below can't drift -# from the versions the stack runs. (`config --images ` also lists the -# service's dependencies, hence the json route.) +# Image ref of one compose.yml service. (`config --images ` also lists +# the service's dependencies, hence the json route.) _image service: @docker compose -f compose.yml config --format json | docker run --rm -i {{jq}} -er '.services["{{service}}"].image // error("no service {{service}} in compose.yml")' -# The validators run in the images the stack itself runs. ~290 MB cold, but -# the smoke job has them anyway. # Run each config through the binary that will load it. validate: docker run --rm --network none -v ./config/prometheus.yaml:/etc/prometheus/prometheus.yaml:ro --entrypoint promtool $(just _image prometheus) check config /etc/prometheus/prometheus.yaml @@ -208,14 +166,13 @@ validate: docker run --rm --network none -v ./config/loki.yaml:/etc/loki/loki.yaml:ro $(just _image loki) -config.file=/etc/loki/loki.yaml -verify-config docker run --rm --network none -v ./config/tempo.yaml:/etc/tempo/tempo.yaml:ro $(just _image tempo) -config.file=/etc/tempo/tempo.yaml -config.verify=true -# Runs against a copy of the sources only: state and tfvars never enter the -# container, which needs network access to fetch the provider. +# Runs against a copy of the sources: state and tfvars never enter the +# container, which has network access to fetch the provider. # Full OpenTofu validation (downloads the provider, so not part of `check`). infra-validate: @d=$(mktemp -d) && cp infra/main.tf infra/.terraform.lock.hcl "$d"/ && docker run --rm --entrypoint sh -v "$d":/src:ro {{tofu}} -c 'mkdir /work && cp /src/main.tf /src/.terraform.lock.hcl /work && cd /work && tofu init -backend=false -input=false >/dev/null && tofu validate'; rc=$?; rm -rf "$d"; exit $rc -# Same container-only rule as `lint`; --user so the rewritten files stay yours. -# Format YAML in place. +# Format YAML in place (--user so the rewritten files stay yours). fmt: docker run --rm --network none --user "$(id -u):$(id -g)" -v .:/code -w /code {{yamlfmt}} . @@ -223,17 +180,12 @@ fmt: hooks: prek install --hook-type pre-commit --hook-type commit-msg --hook-type pre-push -# The tarball is mode 0600: it contains the Grafana DB and webhook secrets, so -# copy it off-host and keep it private. -# Snapshot all stateful volumes to backups/.tar.gz. +# Snapshot all stateful volumes to backups/.tar.gz (mode 0600: it holds secrets). backup: (_backup core_project "backups") -# gzip -1: the stack is paused for as long as the tar runs, and the TSDB -# chunks are already compressed, so the higher levels cost time for nothing. -# Services are paused during the copy and unpaused unconditionally afterwards: -# `pause` is per-container and can fail halfway, so an unpause reached only on -# success would leave the stack frozen. A failed unpause fails the recipe for -# the same reason: a green exit with the stack still SIGSTOPped is worse. +# gzip -1: the stack is paused for as long as the tar runs, and the chunks are +# already compressed. The unpause runs unconditionally (`pause` can fail +# halfway), and a failed unpause fails the recipe. _backup project dir: mkdir -p {{dir}} @-docker compose -p {{project}} unpause {{stateful}} >/dev/null 2>&1 @@ -244,9 +196,8 @@ _backup project dir: restore file: (_restore core_project file "backups") @echo "Restored {{file}}. Run 'just up' to start the stack." -# The wipe is unrecoverable, so the current state is snapshotted first: if the -# extract dies halfway (full disk, wrong volume set) the volumes are left -# partial, and /pre-restore-*.tar.gz is the only way back. +# The current state is snapshotted to /pre-restore-*.tar.gz first: if the +# extract dies halfway, that is the only way back. _restore project file dir: @[ -f "{{file}}" ] || { echo "error: {{file}} not found" >&2; exit 1; } docker run --rm --network none -v {{absolute_path(file)}}:/backup.tar.gz:ro {{alpine}} tar tzf /backup.tar.gz > /dev/null @@ -255,24 +206,14 @@ _restore project file dir: m=$(for s in {{stateful}}; do printf -- '-v {{project}}_%s_data:/data/%s ' $s $s; done); docker run --rm --network none $m -v {{absolute_path(dir)}}:/backups {{alpine}} sh -c 'umask 077 && tar czf /backups/pre-restore-$(date +%Y%m%d-%H%M%S).tar.gz -C /data .' m=$(for s in {{stateful}}; do printf -- '-v {{project}}_%s_data:/data/%s ' $s $s; done); docker run --rm --network none $m -v {{absolute_path(file)}}:/backup.tar.gz:ro {{alpine}} sh -c 'for d in /data/*; do find "$d" -mindepth 1 -delete; done && tar xzf /backup.tar.gz -C /data' -# The only rehearsal of the one recipe that wipes state: backs up the smoke -# stack's volumes, restores them, and asserts Grafana's database came back. -# Needs a booted smoke stack (`just smoke`); `just smoke-down` cleans up after. -# Not in CI: it is a rehearsal for an operator, and 35s per push buys nothing a -# run after touching _backup/_restore does not. +# Needs a booted smoke stack (`just smoke`). Run it after touching _backup/_restore. # Round-trip backup and restore on the smoke stack. restore-check: @d=$(mktemp -d) && just _backup {{smoke_project}} "$d" && f=$(ls "$d"/monitoring-*.tar.gz) && just _restore {{smoke_project}} "$f" "$d" && docker run --rm --network none -v {{smoke_project}}_grafana_data:/g:ro {{alpine}} test -s /g/grafana.db && echo "Backup round-trip ok"; rc=$?; rm -rf "$d"; exit $rc -# `--wait` does the readiness and crash-loop work: it blocks on the grafana and -# prometheus healthchecks in compose.yml and fails if any container exits, so -# there is no poll loop or `ps --status=exited` check to hand-roll here. -# A service with no healthcheck of its own (otel-collector, loki, tempo) still -# fails the wait while it is restarting, so a crash-looping collector is caught. -# That costs the full --wait-timeout to report, where an explicit exit check -# failed immediately. What `--wait` cannot see is everything that lands after -# /api/health answers: provisioning, scrapes, the data paths. scripts/smoke.sh -# asserts those. +# `--wait` blocks on the healthchecks and fails if any container exits, so a +# crash-looping service is caught (after the full timeout). scripts/smoke.sh +# asserts what lands after that: provisioning, scrapes, the data paths. # Boot an isolated copy of the core stack and assert it works end to end. smoke: (_queue-volume smoke_project) {{compose_smoke}} up -d --wait --wait-timeout 120 @@ -282,8 +223,6 @@ smoke: (_queue-volume smoke_project) smoke-logs: {{compose_smoke}} logs --no-color --tail=200 -# Safe: -p scopes it to the smoke project, so it cannot touch a real stack on -# the same host. -# Tear down the smoke stack and its throwaway volumes (-t 1: nothing in it is worth a graceful stop). +# Tear down the smoke stack and its throwaway volumes. smoke-down: {{compose_smoke}} down --remove-orphans --volumes -t 1 diff --git a/scripts/smoke.sh b/scripts/smoke.sh index 795c755..369469a 100755 --- a/scripts/smoke.sh +++ b/scripts/smoke.sh @@ -1,11 +1,7 @@ #!/usr/bin/env bash -# Assertions `just smoke` runs against the stack it just booted. A script rather -# than justfile one-liners so it can have functions and go through shellcheck. -# Host jq, not the container: it is part of the declared toolchain here and -# preinstalled on the CI runner, and a container start per poll adds up. -# -# Everything asserted here lands asynchronously after Grafana's /api/health -# answers, so each assertion polls (`poll`, one second, up to a minute). +# Assertions `just smoke` runs against the stack it just booted. Everything +# here lands asynchronously after Grafana's /api/health answers, so each +# assertion polls (one second, up to a minute). set -euo pipefail url="${SMOKE_URL:?}" @@ -21,9 +17,7 @@ poll() { local n=0; until "$@"; do n=$((n + 1)); [[ $n -lt 60 ]] || return 1; sl # ------------------------------------------------------------------ provisioning # Grafana skips a broken dashboard or a malformed alert group silently, so the -# provisioned sets must equal what the repo holds, uid for uid. Both sides come -# from the repo, so adding a dashboard or a rule never breaks this. A missing -# dashboards/ glob fails the jq call, which is the right answer. +# provisioned sets must equal what the repo holds, uid for uid. want_dash="$(jq -r .uid dashboards/*.json | sort)" rule_files=() for f in config/grafana/alerting/*.yaml; do @@ -46,43 +40,35 @@ jq -e 'all(.isPaused | not)' <<<"$rules" >/dev/null \ || die "paused alert rules never fire: $(jq -r '.[] | select(.isPaused) | .uid' <<<"$rules" | tr '\n' ' ')" # --------------------------------------------------------------- contact points -# Grafana expands $VAR in the alerting provisioning files. The exact URLs the -# stack was given must come back, not merely "something that is not literally -# $ALERT_WEBHOOK_URL": an empty value passes that test and drops every alert. +# The exact URLs the stack was given must come back: an empty value would +# pass a "not literally $ALERT_WEBHOOK_URL" test and drop every alert. gf "$url/api/v1/provisioning/contact-points" \ | jq -e --arg a "$ALERT_WEBHOOK_URL" --arg h "$HEARTBEAT_URL" \ '[.[] | select(.uid | startswith("cp-")) | .settings.url] | sort == ([$a, $h] | sort)' >/dev/null \ || die "contact points do not carry ALERT_WEBHOOK_URL and HEARTBEAT_URL; Grafana did not expand the provisioning file, or a receiver is missing" # ------------------------------------------------------------------- JWT auth -# The stack booted with GRAFANA_JWT_AUTH=true. Grafana ignores an env key it -# no longer knows, so read the parsed setting back, and check that a forged -# Access header is refused rather than auto-signed-up. +# Grafana ignores an env key it no longer knows, so read the parsed setting +# back, and check that a forged Access header is refused. gf "$url/api/admin/settings" | jq -e '.["auth.jwt"].enabled == "true"' >/dev/null \ || die "Grafana did not enable JWT auth from GF_AUTH_JWT_*; the Cloudflare Access path is broken" code="$(curl -s -o /dev/null -w '%{http_code}' -H 'Cf-Access-Jwt-Assertion: not-a-jwt' "$url/api/dashboards/home")" [[ "$code" == 401 ]] || die "a forged Access token got HTTP $code from Grafana, expected 401" # ------------------------------------------------------------- scrape targets -# promtool only checks syntax; a renamed service leaves its scrape job silently -# empty, and TargetDown (up == 0) then matches nothing. Every job in -# prometheus.yaml must have scraped its target successfully. +# A renamed service leaves its scrape job silently empty, and TargetDown then +# matches nothing. Every job in prometheus.yaml must have scraped its target. n_jobs="$(grep -c '^ *- job_name:' config/prometheus.yaml)" all_up() { [[ "$(promq 'count(up == 1)' | jq -r '.data.result[0].value[1] // 0')" == "$n_jobs" ]]; } poll all_up || die "not all $n_jobs scrape targets are up: $(promq up | jq -r '.data.result[] | "\(.metric.job)=\(.value[1])"' | tr '\n' ' ')" # ------------------------------------------------------------- data paths -# One metric and one log through the collector's bearer auth, read back out of -# Prometheus and Loki with the identity labels the alert rules key on: project -# and env promoted from the resource, department stamped by the collector. A -# collector pipeline or prometheus.yaml's promote_resource_attributes can break -# with both configs still valid; this is the only place that shows. Posted from -# inside the stack's network (the sandbox overlay publishes no ingestion ports) -# with Grafana's wget: it is on the same network and needs no extra image. +# One metric and one log through the collector's bearer auth, read back with +# the identity labels the alert rules key on. Posted from inside the stack's +# network with Grafana's wget: the sandbox overlay publishes no ingestion ports. ts="$(date +%s)000000000" res='{"attributes":[{"key":"service.name","value":{"stringValue":"smoke"}},{"key":"project","value":{"stringValue":"smoke"}},{"key":"env","value":{"stringValue":"ci"}}]}' -# The token goes in on stdin: on the production host this is the real -# OTLP_AUTH_TOKEN (dotenv), and argv is world-readable in ps. +# Token on stdin: argv is world-readable in ps. otlp() { docker compose -p "$project" exec -T grafana sh -c \ 'read -r t; wget -qO- --header="Authorization: Bearer $t" --header="Content-Type: application/json" --post-data="$1" "http://otel-collector:4318/v1/$2"' \ diff --git a/templates/README.md b/templates/README.md index 690be5b..406f135 100644 --- a/templates/README.md +++ b/templates/README.md @@ -1,119 +1,109 @@ -# Templates: onboarding a project onto this stack +# Onboarding a project host -Four files get vendored onto a project host at a pinned tag. **No project edits them.** -Everything that differs between projects arrives as an environment variable, so the -agent config on every host is byte-identical and a fix here reaches all of them. +Four files get vendored onto a project host at a pinned tag. **No project +edits them.** Everything that differs between projects arrives as an +environment variable, so the agent config on every host is byte-identical and +a fix here reaches all of them. | File | What it is | | --- | --- | -| `alloy/config.alloy` | The one agent config: container logs, host metrics, cAdvisor, optional GPU | +| `alloy/config.alloy` | The agent config: container logs, host metrics, cAdvisor, optional GPU | | `compose.telemetry.yml` | The Alloy agent and its Docker socket proxy | | `compose.telemetry.gpu.yml` | Opt-in overlay: `nvidia_gpu_exporter`, discovered automatically | -| `run_scheduled.sh` | Dead-man's-switch wrapper for scheduled jobs | +| `run_scheduled.sh` | Dead man's switch wrapper for scheduled jobs | -`alerting/project.yaml.tmpl` is not vendored; it stays here and is rendered by -`bootstrap.sh` into this stack's own alert rules. +`alerting/*.tmpl` are not vendored. `bootstrap.sh` renders them into this +stack's own alert rules. -## The whole checklist +## The checklist -Run `./bootstrap.sh ` on the monitoring host. It prints the two blocks -you need: the `.env` variables to paste on the project host, and the `curl` commands -that vendor these files at a pinned tag. Then on the project host: +1. On the monitoring host, run `./bootstrap.sh `. It prints + the `.env` variables for the project host and the `curl` commands that + vendor the files at a pinned tag. +2. On the project host, paste the variables, run the curls, and bring the + overlay up: -```sh -docker compose -f compose.yml -f compose.telemetry.yml up -d -``` - -Under an hour, most of it waiting for the first scrape. - -## Bootstrap creates the safety net, not the telemetry + ```sh + docker compose -f compose.yml -f compose.telemetry.yml up -d + ``` -A host can ship perfect telemetry and still be unmonitored: the rule that notices its -*silence* lives on this stack, not on the host. Skipping `bootstrap.sh` leaves a -project uncovered, with no error anywhere. +3. Verify from the monitoring host (see below). -The backstop is `ProjectsUncovered`, regenerated on every bootstrap run: it fires on -any series carrying a `project` label with no rendered rule file. It is the detector -for a missing detector. +Skipping `bootstrap.sh` leaves a project unmonitored with no error anywhere. +The rule that notices a host's *silence* lives on this stack, not on the +host. The backstop is `ProjectsUncovered`, regenerated on every bootstrap +run: it fires on any series with a `project` label that has no rendered rule +file. -## Two things that silently produce nothing +## Two settings that silently produce nothing -- **`cgroup: host` on the agent container.** The overlay sets it. cAdvisor finds - containers by walking the cgroup tree, not through the Docker API, so under Docker's - default private cgroup namespace it sees only its own cgroup, reports one root series - with no `name` label, and every container alert matches nothing. No error is logged. -- **`OTEL_SEMCONV_STABILITY_OPT_IN=http` in an instrumented app.** Without it the SDK - emits the legacy HTTP metric names, whose `http_target` label carries the raw request - path: unbounded series on any API with path parameters. The stable names use - `http_route`, and the central Service Health dashboard queries those. +- **`cgroup: host` on the agent container.** The overlay sets it. cAdvisor + finds containers by walking the cgroup tree, so under Docker's default + private cgroup namespace it sees only its own cgroup. It then reports one + root series with no `name` label, and every container alert matches + nothing. No error is logged. +- **`OTEL_SEMCONV_STABILITY_OPT_IN=http` in an instrumented app.** Without + it the SDK emits the legacy HTTP metric names, whose `http_target` label + carries the raw request path: unbounded series on any API with path + parameters. The Service Health dashboard queries the stable names. ## GPU hosts -Include `compose.telemetry.gpu.yml` as well. Set `GPU_METRICS=1`. The agent config -already discovers the exporter by its Compose service label, so nothing else changes: -a GPU host is an ordinary host plus one overlay. +Include `compose.telemetry.gpu.yml` as well and set `GPU_METRICS=1`. The +agent config discovers the exporter by its Compose service label, so nothing +else changes. -The exporter is `nvidia_gpu_exporter`, not dcgm-exporter: DCGM's advantage is its -`DCGM_FI_PROF_*` profiling fields, which NVIDIA document as datacentre-only. On a -consumer card they are absent, and dcgm-exporter still requires `SYS_ADMIN`. +The exporter is `nvidia_gpu_exporter`, not dcgm-exporter. DCGM's profiling +fields are datacentre-only, so on a consumer card it offers nothing extra and +still requires `SYS_ADMIN`. -Three GPU rules are worth adding per GPU host. They are not provisioned by -`bootstrap.sh` yet: +`bootstrap.sh` does not provision GPU alert rules. The signals worth alerting +on per GPU host: -- `nvidia_smi_gpu_recovery_action > 0`: the driver is asking for a reset. The single - best GPU health signal. -- a thermal/power throttle flag. -- XID faults, with an explicit code allowlist rather than every code, since most XIDs - are application faults: +- `nvidia_smi_gpu_recovery_action > 0`: the driver is asking for a reset. +- A thermal or power throttle flag. +- XID faults, with an explicit code allowlist, since most XIDs are + application faults: ```promql time() - nvidia_smi_xid_last_timestamp_seconds{xid=~"48|62|64|74|79|95|119|120"} < 300 ``` -XIDs are the GPU analogue of the crash-loop blind spot. A stuck kernel, an -uncorrectable memory fault, and a card that has fallen off the bus are all invisible to -utilisation graphs. Any of them silently kills a twelve-hour training run. +A stuck kernel, an uncorrectable memory fault, or a card that has fallen off +the bus are all invisible to utilisation graphs. XIDs are how they show. -Both dashboards are provisioned centrally: `dashboards/gpu.json` (vendored from -[14574](https://grafana.com/grafana/dashboards/14574); its multi-GPU companion is -25547 if a host ever grows a second card) and `dashboards/host-containers.json` for -per-container resources. Nothing to import on the project host. +Both dashboards are provisioned centrally: `dashboards/gpu.json` (vendored +from [14574](https://grafana.com/grafana/dashboards/14574)) and +`dashboards/host-containers.json`. Nothing to import on the project host. ## Removing a project -Deleting `config/grafana/alerting/project--.yaml` is **not** enough. -Grafana provisioning creates and updates rules from files, but it never deletes a rule -because its file vanished. The API refuses to delete a provisioned rule too (409, even -with `X-Disable-Provenance`). The orphan keeps evaluating and firing. +Deleting `config/grafana/alerting/project--.yaml` is **not** +enough. Grafana provisioning never deletes a rule because its file vanished, +and the API refuses to delete a provisioned rule (409). The orphan keeps +evaluating and firing. -Delete the file, then drop the rule explicitly with a one-off provisioning file: - -```yaml -# config/grafana/alerting/zz-delete.yaml (temporary) -apiVersion: 1 -deleteRules: - - orgId: 1 - uid: proj-silent-- -``` +1. Delete the project file. +2. Add a temporary provisioning file that drops the rule: -The file has to be temporary. Left in place, it would delete the rule again the next -time `bootstrap.sh` renders it for that project. So: + ```yaml + # config/grafana/alerting/zz-delete.yaml (temporary) + apiVersion: 1 + deleteRules: + - orgId: 1 + uid: proj-silent-- + ``` -1. Restart Grafana. -2. Confirm the rule group is gone. -3. Remove `zz-delete.yaml` and restart Grafana again. -4. Re-run `bootstrap.sh` for a project that remains, so `coverage.yaml` stops listing - the removed one as covered. +3. Restart Grafana and confirm the rule group is gone. +4. Remove `zz-delete.yaml` and restart Grafana again. Left in place, it + deletes the rule again the next time `bootstrap.sh` renders it. +5. Re-run `bootstrap.sh` for a project that remains, so `coverage.yaml` stops + listing the removed one as covered. -## Budgets to plan against +## Limits -- **healthchecks.io free tier is exactly 20 checks.** `bootstrap.sh` creates three per - project/environment, so that is the onboarding ceiling: about six environments. -- **Prometheus's OTLP receiver is documented as "not an efficient way of ingesting - samples"**, for "specific low-volume use cases". A few hundred series per host at 30s - is exactly that case. Revisit on volume: a host past a few thousand active series - should move infrastructure metrics to `remote_write` behind an authenticating proxy, - leaving app metrics on OTLP. +The healthchecks.io free tier is 20 checks. `bootstrap.sh` creates three per +project/environment, so about six environments fit. ## Verifying, from the monitoring host @@ -123,6 +113,7 @@ count(container_start_time_seconds{project="",name!=""}) # one per cont count({job=""}) # app's own SDK metrics ``` -If the first is zero after five minutes, `ProjectTelemetrySilent` will tell you anyway. -Expect it ~20 minutes after the last sample: `absent()` needs the series to go stale -before it reports. +If the first is still zero after five minutes, `ProjectTelemetrySilent` +will say so: it fires after 15 minutes without data, or about 20 minutes +after the last sample once a project has sent any, because `absent()` needs +the series to go stale first. diff --git a/templates/alerting/coverage.yaml.tmpl b/templates/alerting/coverage.yaml.tmpl index 0b2e195..27f987e 100644 --- a/templates/alerting/coverage.yaml.tmpl +++ b/templates/alerting/coverage.yaml.tmpl @@ -1,14 +1,8 @@ -# GENERATED by bootstrap.sh from templates/alerting/coverage.yaml.tmpl. Do not edit the -# rendered file; it is regenerated on every run. +# GENERATED by bootstrap.sh from templates/alerting/coverage.yaml.tmpl. Do not edit +# the rendered file; it is regenerated on every run. # -# The gap this closes: ProjectTelemetrySilent only exists for projects bootstrap.sh was -# actually run for. A project that ships telemetry but skipped bootstrap is silently -# uncovered, one level up from the failure the keystone rule closes. This fires on any -# series whose project/env pair has no rendered rule file. -# -# The selector touches every series every project sends (~3k for one project). Cheap at -# this size, evaluated instantly every 5m; revisit if a project's active series reach -# five figures. +# Fires on any series whose project/env pair has no rendered rule file: a project that +# ships telemetry but was never bootstrapped has no ProjectTelemetrySilent rule. # # Covered right now: __COVERED__ diff --git a/templates/alerting/project.yaml.tmpl b/templates/alerting/project.yaml.tmpl index fb23bb2..84e51c2 100644 --- a/templates/alerting/project.yaml.tmpl +++ b/templates/alerting/project.yaml.tmpl @@ -1,14 +1,10 @@ -# Rendered by bootstrap.sh into config/grafana/alerting/, flat: Grafana's alerting -# provisioner does not recurse. Do not edit the rendered files by hand. Re-run -# bootstrap.sh, or edit this template so every project gets the fix. +# Rendered by bootstrap.sh into config/grafana/alerting/. Do not edit the rendered +# files by hand; edit this template so every project gets the fix. # -# This is the keystone: nothing on a project host can detect its own absence. A project -# that ships telemetry but never had bootstrap.sh run is silently uncovered, which is -# what the ProjectsUncovered rule in coverage.yaml catches one level up. +# Nothing on a project host can detect its own absence, so this rule lives here. # -# bootstrap.sh reads the marker below back to regenerate that coverage rule. It carries -# the pair, not the filename, because both halves may contain a dash: `project-a-b-c` -# cannot be split back into the project and env that produced it. +# bootstrap.sh reads the marker below to regenerate coverage.yaml. It carries the pair, +# not the filename, because both halves may contain a dash. # COVERS: __PROJECT__ __ENV__ apiVersion: 1 @@ -19,10 +15,8 @@ groups: folder: Stack alerts interval: 1m rules: - # A label-only selector, not target_info: agents attach the identity labels to - # the series themselves, so target_info carries none of them. absent() - # yields nothing while telemetry flows, so NoData is the HEALTHY state and must - # map to OK or the rule reads inverted. + # absent() yields nothing while telemetry flows, so NoData is the HEALTHY state + # and must map to OK. - uid: __UID__ title: ProjectTelemetrySilent condition: FIRING diff --git a/templates/alloy/config.alloy b/templates/alloy/config.alloy index 94eb62f..f75fa62 100644 --- a/templates/alloy/config.alloy +++ b/templates/alloy/config.alloy @@ -1,62 +1,47 @@ // SHARED AGENT CONFIG, vendored from the CML monitoring repository at a pinned tag. // Do not edit it per project: every deployment-specific value arrives as an environment // variable (PROJECT, ENVIRONMENT, COMPOSE_PROJECT_NAME, OTEL_EXPORTER_OTLP_ENDPOINT, -// OTLP_AUTH_TOKEN, TELEMETRY_EDGE_KEY). If you need to edit this file to onboard a -// project, that is a bug in the template. Report it upstream. +// OTLP_AUTH_TOKEN, TELEMETRY_EDGE_KEY). If onboarding needs an edit here, that is a +// bug in the template. Report it upstream. // -// It ships this host's telemetry to the shared OpenTelemetry collector run by the -// central monitoring stack: container stdout/stderr as logs, and host resource metrics. -// -// An instrumented application sends its own traces and metrics over OTLP directly. -// Everything else (databases, caches, proxies, frontends, batch jobs) only writes to -// stdout, so this agent picks those up and forwards them on the same OTLP endpoint with -// the same token. Nothing new is published at the edge: the monitoring stack exposes -// `otel.` and does not expose Loki, which has no authentication of its own. - -// The host's own name, read from its /etc/hostname through the rootfs mount: one less -// thing to set per host. The container's own hostname is a docker id, useless as an -// identity. +// Ships what the application cannot report about itself: other containers' stdout as +// logs, host resource metrics, and per-container lifecycle from cAdvisor, all over the +// same OTLP endpoint and token the application uses. + +// The host's name from its own /etc/hostname; the container's hostname is a docker id. local.file "hostname" { filename = "/rootfs/etc/hostname" } -// Discovery and log reading both talk to the Docker API through the socket proxy -// (compose.telemetry.yml), never the socket itself, so this container cannot mutate the -// daemon. The API is what makes container NAMES available; reading the on-disk json logs -// instead would label everything by container id. +// The Docker API is reached through the socket proxy (compose.telemetry.yml), never the +// socket itself, so this container cannot mutate the daemon. discovery.docker "containers" { host = "tcp://docker-socket-proxy:2375" - // The default is 1m and a backup one-shot runs for 60-90s, so a whole backup can - // start and exit inside one poll window and never be discovered, losing the logs - // most worth having. Polling the local socket for a dozen containers is cheap; a - // missed short-lived container is not. + // The default 1m lets a short-lived one-shot (a backup) start and exit inside one + // poll window, never discovered. refresh_interval = "5s" } discovery.relabel "containers" { targets = discovery.docker.containers.targets - // Only this stack. The host may run other Compose projects (monitoring's own, for - // one) that ship their own telemetry; picking them up here would double-report them. + // Only this Compose project; other projects on the host ship their own telemetry. rule { source_labels = ["__meta_docker_container_label_com_docker_compose_project"] regex = sys.env("COMPOSE_PROJECT_NAME") action = "keep" } - // Never tail this pipeline's own containers. When the export path is broken, every - // failed batch writes an error line; tailing that line queues it, it fails, and - // writes another: an amplifying self-feed with no exit. The json-file driver note in - // the overlay only prevents the log-DRIVER loop; this prevents the discovery one. + // Never tail this pipeline's own containers: a broken export path would log an + // error per failed batch, queue it, fail again, with no exit. rule { source_labels = ["__meta_docker_container_label_com_docker_compose_service"] regex = "alloy|docker-socket-proxy" action = "drop" } - // The Compose service name is the stable per-deployable identifier the monitoring - // stack keys on: `api`, never `api-prod-2`. + // The Compose service name is the stable identifier dashboards key on. rule { source_labels = ["__meta_docker_container_label_com_docker_compose_service"] target_label = "service_name" @@ -75,30 +60,23 @@ loki.source.docker "containers" { env = sys.env("ENVIRONMENT"), project = sys.env("PROJECT"), - // Without this every host collapses onto one identity: Alloy scrapes its OWN - // in-container exporter, so the address Prometheus derives `instance` from is - // identical on every machine. Once a second host ships, the series interleave - // and every host-level alert is meaningless. The coalesce keeps a blank - // /etc/hostname from reintroducing that: the container hostname is a poor - // identity, but a distinguishable one. + // `instance` is identical on every host (Alloy scrapes its own in-container + // exporter), so this is the host identity. The coalesce handles a blank + // /etc/hostname. host_name = coalesce(string.trim_space(local.file.hostname.content), constants.hostname), } forward_to = [otelcol.receiver.loki.containers.receiver] } -// Loki-shaped entries in, OpenTelemetry logs out. This is the only reason the agent is -// Alloy and not a plain OTel collector: loki.source.docker reads Docker's log stream -// with real container labels attached. +// Loki-shaped entries in, OpenTelemetry logs out. otelcol.receiver.loki "containers" { output { logs = [otelcol.processor.transform.resource_attributes.input] } } -// Promote the identifiers the monitoring stack keys dashboards on from log-record -// attributes to RESOURCE attributes. Keep error_mode = "ignore": if the conversion ever -// names these attributes differently, logs still arrive unpromoted instead of being -// dropped. +// Promote the identity labels from log-record attributes to resource attributes. +// error_mode = "ignore" so a renamed attribute arrives unpromoted rather than dropped. otelcol.processor.transform "resource_attributes" { error_mode = "ignore" @@ -119,18 +97,10 @@ otelcol.processor.transform "resource_attributes" { } // --------------------------------------------------------------------------- -// Host metrics. node_exporter's collectors, scraped in-process and converted to OTLP so -// they ride the same endpoint and token as the logs above: no second pipeline, no second -// hostname, no second credential. -// -// set_collectors is an allowlist, not the default set, which keeps the series count down. -// `hwmon` feeds a dashboard panel, since thermal data on a physical box is cheap and -// occasionally telling, but it is NOT a control: fan RPM is a symptom of load, legitimate -// load spins fans too, and no threshold on it would be trustworthy. A crash loop is -// detected by container restart count, not by temperature. +// Host metrics: node_exporter's collectors, scraped in-process and converted to OTLP. +// set_collectors is an allowlist, which keeps the series count down. `hwmon` feeds a +// dashboard panel only; nothing alerts on temperature. prometheus.exporter.unix "host" { - // The exporter runs inside the container, so point it at the host's mounted /proc, - // /sys and /. procfs_path = "/host/proc" sysfs_path = "/host/sys" rootfs_path = "/rootfs" @@ -148,29 +118,21 @@ prometheus.exporter.unix "host" { ] filesystem { - // Pseudo-filesystems and per-container overlays: neither is disk anyone can free. + // Pseudo-filesystems and per-container overlays. mount_points_exclude = "^/(dev|proc|sys|run)($|/)|^/rootfs/(dev|proc|sys|run)($|/)|^/var/lib/docker/" } } // --------------------------------------------------------------------------- -// Per-container resource metrics. This is what makes a crash loop visible: cAdvisor has -// no restart counter, but container_start_time_seconds moving repeatedly is the same -// signal, and container_oom_events_total says why. The monitoring stack alerts on -// -// changes(container_start_time_seconds{name!=""}[1h]) > 3 -// -// so this side only has to ship the series. Shipping them at all needs `cgroup: host` -// on this container (see the note in compose.telemetry.yml); without it cAdvisor -// reports one root series and the rule above matches nothing. +// Per-container resource metrics. The monitoring stack detects a crash loop as +// container_start_time_seconds changing repeatedly, and container_oom_events_total says +// why. Needs `cgroup: host` on this container (see compose.telemetry.yml); without it +// cAdvisor reports one root series and the rule matches nothing. prometheus.exporter.cadvisor "containers" { docker_host = "tcp://docker-socket-proxy:2375" - // Both defaults are cardinality traps. store_container_labels = true turns every - // container label AND environment variable into a Prometheus label, which on Compose - // smears com_docker_compose_* across every series; docker_only = false additionally - // emits raw systemd-slice cgroups nobody will ever read. With these two set, a - // ten-container host is a few hundred series. + // Both defaults are cardinality traps: every container label and environment + // variable as a Prometheus label, plus raw systemd-slice cgroups. store_container_labels = false docker_only = true @@ -187,17 +149,9 @@ prometheus.scrape "cadvisor" { job_name = "cadvisor" } -// cAdvisor walks the host's cgroup tree, so it sees EVERY container on the box. The -// COMPOSE_PROJECT_NAME filter on discovery.docker above scopes log tailing only; it does -// not reach this exporter. Without this rule, containers from a dev stack, an -// unrelated tool, or the monitoring stack itself arrive stamped with THIS deployment's -// project and env, firing this project's alerts for something it does not own and -// double-reporting anything that ships its own telemetry. So keep only this project's -// containers, using the compose label the allowlist above preserves. -// -// The unlabelled root-cgroup series drops out with them, which is correct: it carries no -// `name`, so no container rule can match it, and node_exporter above already owns -// host-level totals. +// cAdvisor sees every container on the box, and the discovery.docker filter above does +// not reach it. Keep only this project's, or other stacks' containers arrive stamped +// with this project's labels. The unlabelled root-cgroup series drops out too. prometheus.relabel "cadvisor_scope" { forward_to = [prometheus.relabel.host.receiver] @@ -215,8 +169,7 @@ prometheus.scrape "host" { job_name = "host" } -// Same identifiers the logs carry, so host metrics and container logs can be filtered -// by the same env/project in one dashboard. +// The same identity labels the logs carry. prometheus.relabel "host" { forward_to = [otelcol.receiver.prometheus.host.receiver] @@ -232,7 +185,7 @@ prometheus.relabel "host" { action = "replace" } - // See the note on the log labels: `instance` is not a host identity here. + // See the note on the log labels. rule { target_label = "host_name" replacement = coalesce(string.trim_space(local.file.hostname.content), constants.hostname) @@ -241,11 +194,8 @@ prometheus.relabel "host" { } // --------------------------------------------------------------------------- -// Alloy's own health. The exporter below is the only place that counts what THIS agent -// drops (otelcol_exporter_send_failed_*); unshipped, those counters live on a loopback -// port nobody reads and an outage's log loss goes uncounted. Riding the same pipeline is -// fine: when the pipeline is down these metrics go dark too, and that absence is what the -// ProjectTelemetrySilent rule on the monitoring host fires on. +// Alloy's own health: the only place that counts what this agent drops +// (otelcol_exporter_send_failed_*). prometheus.exporter.self "alloy" { } prometheus.scrape "alloy" { @@ -256,13 +206,8 @@ prometheus.scrape "alloy" { } // --------------------------------------------------------------------------- -// GPU metrics, present only on hosts that run compose.telemetry.gpu.yml. With the -// exporter container absent this finds no targets and costs nothing, so the same config -// file is correct on every host: a GPU host is one extra overlay, not a new design. -// -// Alloy has no native GPU component, and OTel Collector contrib has no NVIDIA receiver, -// so the pattern is an exporter container scraped over the Compose network. Still no new -// agent, no new endpoint and no new credential. +// GPU metrics, present only on hosts that run compose.telemetry.gpu.yml. Without the +// exporter container this finds no targets and costs nothing. discovery.relabel "gpu_exporter" { targets = discovery.docker.containers.targets @@ -289,9 +234,6 @@ prometheus.scrape "gpu" { targets = discovery.relabel.gpu_exporter.output forward_to = [prometheus.relabel.host.receiver] - // 30s matches the other scrapes here: thermal throttling and XID faults persist for - // far longer than one interval, and a training run's utilisation curve is readable - // without sub-minute resolution. scrape_interval = "30s" job_name = "gpu" } @@ -302,10 +244,9 @@ otelcol.receiver.prometheus "host" { } } -// Refuse work before the cgroup does it for us. Without this, a long outage of the -// central monitoring stack grows the sending queue until the OOM killer takes the agent -// AND the self-metrics that would have counted the loss. Sized under the overlay's 512m -// mem_limit; keep the two in step. +// Without this a long outage of the central stack grows the sending queue until the +// OOM killer takes the agent. Sized under the overlay's 512m mem_limit; keep the two +// in step. otelcol.processor.memory_limiter "default" { check_interval = "1s" limit = "400MiB" @@ -326,29 +267,23 @@ otelcol.processor.batch "default" { otelcol.exporter.otlphttp "central" { client { - // The tunnel routes HTTPS to the collector's HTTP receiver only; there is no - // public gRPC path, so this is otlphttp and not otlp. The endpoint MUST be - // https://. An http:// value ships the bearer token and every log line in - // cleartext, and nothing here can tell (see .env.example). + // The tunnel routes HTTPS to the collector's HTTP receiver only, hence otlphttp. + // The endpoint MUST be https://: an http:// value ships the token and every log + // line in cleartext, and nothing here can tell. endpoint = sys.env("OTEL_EXPORTER_OTLP_ENDPOINT") headers = { Authorization = "Bearer " + sys.env("OTLP_AUTH_TOKEN"), // Second, weaker credential for projects whose egress crosses a WAF: the - // project's WAF-skip rule matches THIS header, so the bearer token never - // appears in a ruleset expression readable through the provider's API. It - // grants only the managed-security skip. Projects with no WAF in front of - // their egress leave TELEMETRY_EDGE_KEY empty; the header is then sent - // empty and ignored. + // WAF-skip rule matches this header, so the bearer token never appears in a + // ruleset expression. Empty when unused. "X-Telemetry-Key" = sys.env("TELEMETRY_EDGE_KEY"), } } - // Set here rather than left to the collector defaults: retry for 15 minutes, hold up - // to 2000 batches in memory, then drop and COUNT the drop (shipped by - // prometheus.exporter.self above). This queue is why the overlay caps this - // container's memory; see mem_limit there before raising queue_size. + // Retry for 15 minutes, hold up to 2000 batches in memory, then drop and count the + // drop. See the overlay's mem_limit before raising queue_size. sending_queue { queue_size = 2000 } diff --git a/templates/compose.telemetry.gpu.yml b/templates/compose.telemetry.gpu.yml index 5e55a86..e5a3bb0 100644 --- a/templates/compose.telemetry.gpu.yml +++ b/templates/compose.telemetry.gpu.yml @@ -1,33 +1,18 @@ # compose.telemetry.gpu.yml, VENDORED from the CML monitoring repository at a pinned tag. # -# OPTIONAL OVERLAY: GPU metrics on a host with NVIDIA hardware. -# -# Adds a GPU exporter that the Alloy agent discovers and scrapes automatically. Include -# it only on hosts that have a card: +# OPTIONAL OVERLAY: GPU metrics on a host with NVIDIA hardware. The Alloy agent +# discovers and scrapes the exporter automatically. # # docker compose ... -f compose.telemetry.yml -f compose.telemetry.gpu.yml up -d # -# or set GPU_METRICS=1 in the host's root `.env` and let the deploy recipes include it. -# -# Why nvidia_gpu_exporter and not dcgm-exporter: DCGM's advantage is its DCGM_FI_PROF_* -# profiling fields, which NVIDIA document as datacentre-only hardware features. On a -# GeForce card they are absent, which removes the reason to prefer DCGM while keeping its -# SYS_ADMIN requirement and its 2021-vintage dashboard. Consumer cards also have no -# framebuffer ECC, so the ECC fields are empty too. What remains is the NVML field set, -# which is what this exporter reads. -# -# On datacentre silicon (A100/H100/L40S), profiling, ECC, NVLink and MIG all light up and -# dcgm-exporter becomes the better choice. That is a one-container swap plus one -# dashboard; do not pre-build for it. +# nvidia_gpu_exporter, not dcgm-exporter: DCGM's profiling fields are datacentre-only, +# and on a consumer card it keeps only its SYS_ADMIN requirement. services: nvidia-gpu-exporter: image: utkuozdemir/nvidia_gpu_exporter:1.14.0-nvml@sha256:82acc3fc60a5a709846ea9757bbccb170fd141039fa47e5899c55fe9a60f56fe restart: unless-stopped - # The NVML backend is what produces XID error counters, which the nvidia-smi - # backend cannot report. XIDs are the signal that matters most here: a stuck kernel, - # an uncorrectable memory fault or a card that has fallen off the bus are invisible - # to utilisation graphs, and they are what silently kills a long training run. + # The NVML backend reports XID error counters; the nvidia-smi backend cannot. environment: NVIDIA_DRIVER_CAPABILITIES: utility deploy: @@ -39,15 +24,12 @@ services: capabilities: [gpu] mem_limit: 256m pids_limit: 256 - # The exporter only reads NVML and serves HTTP; the driver files arrive as runtime - # mounts, so a read-only root and an empty capability set cost it nothing. read_only: true cap_drop: - ALL security_opt: - no-new-privileges:true - # No published ports and no host network: Alloy reaches it over the Compose network, - # discovers it by its Compose service label, and nothing else can. + # No published ports: Alloy reaches it over the Compose network. networks: - egress logging: diff --git a/templates/compose.telemetry.yml b/templates/compose.telemetry.yml index 73f4306..003d408 100644 --- a/templates/compose.telemetry.yml +++ b/templates/compose.telemetry.yml @@ -1,22 +1,11 @@ # compose.telemetry.yml, VENDORED from the CML monitoring repository at a pinned tag. +# Vendor it as-is; everything that differs between projects arrives as an environment +# variable. See templates/README.md. # -# One Grafana Alloy agent per deploy host. Vendor it as-is; everything that differs -# between projects arrives as an environment variable. See templates/README.md. -# -# An instrumented application sends its own telemetry. This overlay adds a Grafana -# Alloy agent that collects three things the application cannot report on itself and -# forwards them to the same OTLP endpoint with the same token: -# -# - the stdout of every OTHER container in the stack, as logs; -# - host resource metrics: CPU, memory, load, disk, network, hwmon; -# - per-container resources and lifecycle from cAdvisor, which is what makes a crash -# loop visible on the central monitoring host. -# -# The root deploy recipes auto-include this overlay when OTEL_EXPORTER_OTLP_ENDPOINT is -# set in the host's root `.env`, the same switch that turns on the API's own exporter. -# Hosts without it ship nothing and keep `docker logs` as the only log path. +# One Grafana Alloy agent per host, shipping what the application cannot report on +# itself to the same OTLP endpoint with the same token: other containers' stdout as +# logs, host resource metrics, and per-container lifecycle from cAdvisor. -# Local json-file logging with rotation for the agent stack itself. x-logging: &default-logging driver: json-file options: @@ -24,13 +13,9 @@ x-logging: &default-logging max-file: "3" services: - # Least-privilege gate between Alloy and the Docker API. Alloy never sees the socket: - # it talks HTTP to this proxy, which allows only the read-only endpoints discovery, - # log tailing and cAdvisor need (containers, events, info, version, ping) and refuses - # everything else. POST is off, so nothing behind it can start or exec a container. - # The proxy itself holds the socket, so IT is the root-equivalent component; it runs - # nothing but haproxy with a static allowlist, joins only the internal socket network, - # and is the single place to audit. + # Read-only gate between Alloy and the Docker API. The proxy holds the socket, so it + # is the root-equivalent component: haproxy with a static allowlist, on the internal + # socket network only. docker-socket-proxy: image: tecnativa/docker-socket-proxy:v0.5.0@sha256:1f5038b54f06c3e18422902cf00ba21803d1c97805aae032e5e6673d532d3459 restart: unless-stopped @@ -55,75 +40,53 @@ services: alloy: image: grafana/alloy:v1.18.1@sha256:0f4434c92b3e6cdac38bb129b344e1790c246f7b6e2eaffcc16a5fa363240e33 restart: unless-stopped - # Root for the host mounts below (/host/proc, /host/sys, /rootfs): node_exporter's - # collectors read files that are root-only on the host. The Docker API is reached - # through docker-socket-proxy above, so root here does not imply control of the - # Docker daemon. + # Root to read root-only files under the host mounts below. The Docker API is + # behind the proxy, so root here does not imply control of the daemon. user: root command: - run - /etc/alloy/config.alloy - --storage.path=/var/lib/alloy/data - # Bound to loopback: the diagnostics UI has no auth and nothing needs to reach it - # from another container. + # The diagnostics UI has no auth; keep it on loopback. - --server.http.listen-addr=127.0.0.1:12345 environment: ENVIRONMENT: ${ENVIRONMENT:?telemetry overlay requires ENVIRONMENT} - # The project identity every signal is labelled with, and what the - # ProjectTelemetrySilent rule on the monitoring host keys on. It must match the - # value the monitoring host's onboarding script (bootstrap.sh) was run with. + # Must match the value bootstrap.sh was run with on the monitoring host. PROJECT: ${PROJECT:?telemetry overlay requires PROJECT} - # Alloy only reads containers belonging to this Compose project, so it must know - # which one it is. Compose always injects COMPOSE_PROJECT_NAME into interpolation - # (the deploy directory's base name unless -p or .env overrides it), so this passes - # through whatever the label on the sibling containers says. A `:-` default here - # would be dead text. This is the COMPOSE project (docker's -p), not PROJECT above; - # rename the deploy directory and the filter follows it. + # The Compose project (docker's -p), not PROJECT above. Compose always injects + # it into interpolation, so Alloy's container filter follows the sibling + # containers' label. COMPOSE_PROJECT_NAME: ${COMPOSE_PROJECT_NAME} OTEL_EXPORTER_OTLP_ENDPOINT: ${OTEL_EXPORTER_OTLP_ENDPOINT:?required by the telemetry overlay} OTLP_AUTH_TOKEN: ${OTLP_AUTH_TOKEN:?telemetry overlay requires OTLP_AUTH_TOKEN} - # Second, weaker credential: the Cloudflare WAF-skip rule for the telemetry - # hostname matches this header, so the bearer token itself never appears in a - # Cloudflare ruleset expression. It rotates independently of the token. Optional: - # only projects whose egress crosses a WAF need it. Empty is fine. + # Optional second credential for projects whose egress crosses a WAF; the + # WAF-skip rule matches this header instead of the bearer token. TELEMETRY_EDGE_KEY: ${TELEMETRY_EDGE_KEY:-} volumes: - ./deploy/alloy/config.alloy:/etc/alloy/config.alloy:ro # vendored, never edited - # node_exporter's collectors read the host's own /proc, /sys and filesystems. - # All read-only. This does give the container read access to the whole host - # filesystem through /rootfs, the price of host disk-usage metrics from a - # container. It is no more than a host-installed node_exporter running as root - # already has, but know it rather than discover it. + # node_exporter reads the host's /proc, /sys and filesystems. /rootfs gives + # the container read access to the whole host filesystem, the price of host + # disk-usage metrics from a container. - /proc:/host/proc:ro - /sys:/host/sys:ro - /:/rootfs:ro # cAdvisor's filesystem reads; container metadata comes over the socket proxy. - /var/lib/docker:/var/lib/docker:ro - /dev/disk:/dev/disk:ro - # Read positions survive restarts; without this every restart re-ships whatever - # the containers still have on disk. + # Read positions; without this every restart re-ships what is still on disk. - alloy_data:/var/lib/alloy/data - # cAdvisor finds containers by walking the cgroup hierarchy, NOT through the Docker - # API; the socket proxy only supplies names and metadata for cgroups it has already - # found. Under Docker's default private cgroup namespace this container sees just its - # own cgroup, as "/", so every container_* metric collapses to a single root series - # with no `name` label: the ContainerRestarting rule matches zero series and - # ContainerOOMKilled cannot say which container died. Joining the host's namespace is - # the whole fix; mounting /sys/fs/cgroup as well is redundant, verified on cgroup v2. - # It grants read access to the host's cgroup tree, which is less than the /rootfs and - # /var/lib/docker mounts above already give. + # cAdvisor finds containers by walking the cgroup tree, not through the Docker + # API. Under Docker's default private cgroup namespace it sees only its own, as + # "/", so every container_* metric collapses to one root series with no `name` + # label and the container alerts match nothing. No error is logged. cgroup: host - # cAdvisor reads OOM-kill events from the kernel log; without the device AND - # CAP_SYSLOG (kernel.dmesg_restrict gates /dev/kmsg reads) container_oom_events_total - # never increments and the central ContainerOOMKilled rule can never fire ("Could - # not configure a source for OOM detection" in the agent log is this exact gap). + # OOM-kill events come from the kernel log. Without the device AND CAP_SYSLOG, + # container_oom_events_total never increments ("Could not configure a source for + # OOM detection" in the agent log). devices: - /dev/kmsg:/dev/kmsg - # Root, but not root-with-everything: the default capability set is dropped, as on - # the socket proxy above. DAC_READ_SEARCH is what lets the collectors read root-only - # files under /host/proc, /host/sys and /rootfs; unlike the default set's - # DAC_OVERRIDE it grants no write-side permission bypass. The rest (SETUID, MKNOD, - # NET_RAW, ...) is nothing a metrics agent needs. + # DAC_READ_SEARCH reads root-only files under the host mounts without the + # write-side bypass DAC_OVERRIDE would grant. cap_drop: - ALL cap_add: @@ -131,29 +94,24 @@ services: - DAC_READ_SEARCH security_opt: - no-new-privileges:true - # Bounded like every other service on the host: during a collector outage the export - # queue grows in memory, and without a ceiling the OOM killer picks by RSS. The - # plausible victim is then postgres or the API, so the agent takes down the thing it - # observes. Keep in step with the sending_queue size in config.alloy. + # During a collector outage the export queue grows in memory; without a ceiling + # the OOM killer picks by RSS and may take the application instead. Keep in step + # with the memory_limiter in config.alloy. mem_limit: 512m pids_limit: 512 networks: - egress - docker_socket - # Never through itself: Alloy reporting its own failure to ship logs, by shipping - # a log, is a loop with no exit. + # Never through itself: shipping a log about failing to ship logs has no exit. logging: *default-logging volumes: alloy_data: networks: - # Alloy's way out to the central collector, and where the GPU exporter overlay - # attaches. Declared here so the overlay stands on its own; a project that already - # defines `egress` in its compose.yml keeps its definition, Compose merges the two by - # name. + # Alloy's way out, and where the GPU exporter overlay attaches. A project that + # already defines `egress` keeps its definition; Compose merges by name. egress: - # Carries only the proxied Docker API between the proxy and Alloy. Internal: it has - # no gateway, so nothing on it can reach out and nothing outside can reach in. + # Only the proxied Docker API. No gateway. docker_socket: internal: true diff --git a/templates/run_scheduled.sh b/templates/run_scheduled.sh index 6066e4d..3237fa0 100755 --- a/templates/run_scheduled.sh +++ b/templates/run_scheduled.sh @@ -1,26 +1,18 @@ #!/usr/bin/env bash -# VENDORED from the CML monitoring repository at a pinned tag. Generic: nothing here -# names a project or a job list. +# VENDORED from the CML monitoring repository at a pinned tag. # -# Run one scheduled job and report the result to a dead-man's switch (healthchecks.io). +# Run one scheduled job and report the result to a dead man's switch (healthchecks.io). +# This is the one piece of monitoring that does not share fate with the observability +# stack: a push to an external endpoint still arrives when the telemetry pipeline is +# what broke, and its absence is the alarm. # # Usage: run_scheduled.sh -# job: any name; it selects the ping URL variable and the command below. -# env: the deployment environment, passed through to the command. +# job: selects the `just` recipe and the PING_ URL variable. +# env: passed through to the recipe. # -# Called by this host's systemd units. This is the ONE piece of monitoring that does not -# share fate with the observability stack. Everything else (logs, traces, host metrics) -# flows through the Alloy agent to the shared OpenTelemetry collector, so a dead host, a -# dead collector, a broken tunnel and an expired token all look identical from Grafana: -# silence. A push from this host to an external endpoint still arrives when that pipeline -# is the thing that broke, and its absence is itself the alarm. -# -# The ping lives here, not in each systemd unit, so the reporting is written once. -# -# Ping URLs come from the host file the units load, never from the repository: they are -# capability URLs. The monitoring host's onboarding script (bootstrap.sh) prints the -# PING_* block. An unset URL disables the ping for that job without failing it, so a host -# that has not been wired up yet still runs its jobs. +# Ping URLs are capability URLs: they come from the host file the systemd units load, +# never from the repository. bootstrap.sh prints the PING_* block. An unset URL +# disables the ping for that job without failing it. set -uo pipefail job="${1:-}" @@ -30,9 +22,8 @@ if [[ -z "$job" || -z "$env_name" ]]; then echo "usage: $0 " >&2 exit 2 fi -# Job and env become part of an environment-variable name below, so keep them to -# characters that can appear in one. A stray character would look up the wrong variable -# and disable the ping instead of failing. +# Job and env become part of a variable name below; a stray character would look up +# the wrong variable and silently disable the ping. if [[ ! "$job" =~ ^[A-Za-z0-9_-]+$ || ! "$env_name" =~ ^[A-Za-z0-9_-]+$ ]]; then echo "error: job and env must match [A-Za-z0-9_-]+" >&2 exit 2 @@ -41,16 +32,13 @@ fi ROOT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/.." && pwd)" cd "$ROOT_DIR" || exit 1 -# `just` is resolved at unit-render time and passed in; fall back to PATH for manual runs. JUST_BIN="${JUST_BIN:-just}" -# One `just` recipe per job, named the same as the job. That identity is the contract, so -# there is no mapping table here. Projects that do not use `just` can point JUST_BIN at -# any runner with the same shape. +# One recipe per job, named the same. Projects without `just` can point JUST_BIN at any +# runner with the same shape. command=("$JUST_BIN" "$job" "$env_name") -# One variable per job, so each gets its own check. Sharing a URL would let a frequent -# job's pings mask a rare one's silence, the failure the rare job exists to catch. +# One URL per job: a shared one would let a frequent job's pings mask a rare one's silence. url_var="PING_${job//-/_}" url_var="${url_var^^}" ping_url="${!url_var:-}" @@ -58,19 +46,16 @@ ping_url="${!url_var:-}" output_file="$(mktemp)" trap 'rm -f "$output_file"' EXIT -# Send the job's own output as the failure body, so the alert carries the reason. Note -# that this puts job output (hostnames, paths, backup summaries) in a third party's -# hands. The ping therefore carries no credentials, and the URL itself is the only secret. +# The job's output is the failure body, so the alert carries the reason. This puts job +# output in a third party's hands; keep credentials out of it. ping_fail() { curl -fsS -m 10 --retry 3 --data-binary "@${output_file}" "${ping_url}/fail" -o /dev/null \ || echo "WARNING: failure ping to ${url_var} failed" >&2 } -# A killed job must still report. systemd's TimeoutStartSec TERMs the whole cgroup: the -# job dies, and without this trap bash would die too, before the ping block, so a HUNG job -# would send neither success nor failure and its captured output would be lost. The job -# runs in the background so `wait` can be interrupted by the signal; the child has already -# received the same TERM from systemd (KillMode=control-group). +# A killed job must still report: systemd's TimeoutStartSec TERMs the whole cgroup, and +# without this trap a hung job would send neither ping. The job runs in the background +# so `wait` can be interrupted; the child receives the same TERM from systemd. # shellcheck disable=SC2329 # invoked via the TERM/INT traps below on_terminate() { local sig="$1" From 361c158579802647047dd33cbec4d9754145958d Mon Sep 17 00:00:00 2001 From: Simon van Lierde Date: Sun, 6 Sep 2026 19:19:00 +0000 Subject: [PATCH 60/85] fix(security): keep the ingest token out of the container's argv, refuse readable secret files - smoke posts through curl's stdin config; busybox wget put the token in argv - _expose-guards fails when .env or the tofu state is group/other readable - the prek gitleaks hook mounts the repo read-only like just lint --- .pre-commit-config.yaml | 6 +++--- justfile | 4 +++- scripts/smoke.sh | 9 +++++---- 3 files changed, 11 insertions(+), 8 deletions(-) diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index 62405a0..6ce1b91 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -9,7 +9,7 @@ repos: - id: gitleaks name: gitleaks (staged) entry: >- - docker run --rm --network none -v .:/repo + docker run --rm --network none -v .:/repo:ro zricethezav/gitleaks:v8.30.1@sha256:c00b6bd0aeb3071cbcb79009cb16a60dd9e0a7c60e2be9ab65d25e6bc8abbb7f git --pre-commit --staged --redact --no-banner /repo language: system @@ -21,8 +21,8 @@ repos: entry: >- sh -c 'head -n1 "$1" | grep -Eq "^((build|chore|ci|docs|feat|fix|perf|refactor|revert|style|test)(\([a-z0-9._/-]+\))?!?: - [a-z].{0,70}|(Merge|Revert|fixup!|squash!) .*)$" - || { echo "commit message must be type(scope): lower-case summary, at most 72 chars" >&2; exit 1; }' -- + [a-z].{0,81}|(Merge|Revert|fixup!|squash!) .*)$" + || { echo "commit message must be type(scope): lower-case summary, at most 82 chars" >&2; exit 1; }' -- language: system stages: [commit-msg] - id: check diff --git a/justfile b/justfile index 5f53162..af8ef65 100644 --- a/justfile +++ b/justfile @@ -61,7 +61,8 @@ up: (_queue-volume core_project) _guard-if-exposed _guard-if-exposed: @case "${COMPOSE_FILE:-}" in *compose.tunnel.yml*) just _expose-guards;; esac -# Refuses to expose the stack with the documented default credentials. +# Refuses to expose the stack with the documented default credentials or with +# secret files other local users can read. _expose-guards: @[ "${OTLP_AUTH_TOKEN:-}" != "local-dev-token" ] || { echo "error: OTLP_AUTH_TOKEN is still the local default; generate one (openssl rand -hex 32) before exposing ingestion" >&2; exit 1; } @[ "${GRAFANA_ADMIN_PASSWORD:-}" != "change-me" ] || { echo "error: GRAFANA_ADMIN_PASSWORD is still the documented default; change it before exposing Grafana" >&2; exit 1; } @@ -69,6 +70,7 @@ _expose-guards: @[ "${GRAFANA_COOKIE_SECURE:-false}" = "true" ] || { echo "error: GRAFANA_COOKIE_SECURE must be true when Grafana is served over HTTPS; set it in .env" >&2; exit 1; } @[ "${GRAFANA_JWT_AUTH:-false}" != "true" ] || { [ -n "${CF_ACCESS_TEAM_DOMAIN:-}" ] && [ -n "${CF_ACCESS_AUD:-}" ]; } || { echo "error: GRAFANA_JWT_AUTH=true needs CF_ACCESS_TEAM_DOMAIN and CF_ACCESS_AUD in .env (cd infra && tofu output -raw grafana_access_aud)" >&2; exit 1; } @[ -n "${HEARTBEAT_URL:-}" ] || echo "WARNING: HEARTBEAT_URL is empty; the stack goes live without a dead-man's switch" >&2 + @for f in .env infra/terraform.tfvars infra/terraform.tfstate infra/terraform.tfstate.backup; do [ ! -e "$f" ] || case "$(stat -c %a "$f")" in *00) ;; *) echo "error: $f is readable by other users (mode $(stat -c %a "$f")); it holds live secrets, run: chmod 600 $f" >&2; exit 1;; esac; done @[ -n "${ALERT_WEBHOOK_URL:-}" ] || { echo "error: ALERT_WEBHOOK_URL is empty; every alert would fire into an empty webhook URL and be dropped. The heartbeat keeps pinging either way, so this failure looks healthy from the outside. Set it, or comment out this guard" >&2; exit 1; } # Stop the stack; volumes stay. diff --git a/scripts/smoke.sh b/scripts/smoke.sh index 369469a..c3c7c2b 100755 --- a/scripts/smoke.sh +++ b/scripts/smoke.sh @@ -65,14 +65,15 @@ poll all_up || die "not all $n_jobs scrape targets are up: $(promq up | jq -r '. # ------------------------------------------------------------- data paths # One metric and one log through the collector's bearer auth, read back with # the identity labels the alert rules key on. Posted from inside the stack's -# network with Grafana's wget: the sandbox overlay publishes no ingestion ports. +# network with Grafana's curl: the sandbox overlay publishes no ingestion ports. ts="$(date +%s)000000000" res='{"attributes":[{"key":"service.name","value":{"stringValue":"smoke"}},{"key":"project","value":{"stringValue":"smoke"}},{"key":"env","value":{"stringValue":"ci"}}]}' -# Token on stdin: argv is world-readable in ps. +# Token through curl's stdin config: argv is world-readable in ps, on the host +# and inside the container alike. otlp() { docker compose -p "$project" exec -T grafana sh -c \ - 'read -r t; wget -qO- --header="Authorization: Bearer $t" --header="Content-Type: application/json" --post-data="$1" "http://otel-collector:4318/v1/$2"' \ - _ "$2" "$1" <<<"$OTLP_AUTH_TOKEN" >/dev/null + 'curl -sf -K - -o /dev/null -H "Content-Type: application/json" --data-binary "$1" "http://otel-collector:4318/v1/$2"' \ + _ "$2" "$1" <<<"header = \"Authorization: Bearer $OTLP_AUTH_TOKEN\"" } otlp metrics '{"resourceMetrics":[{"resource":'"$res"',"scopeMetrics":[{"metrics":[{"name":"smoke_up","gauge":{"dataPoints":[{"asInt":"1","timeUnixNano":"'"$ts"'"}]}}]}]}]}' \ || die "the collector refused an OTLP metric with the .env token" From 11e45bd8874d5e01bb74d66fa8f999a3a44b8aae Mon Sep 17 00:00:00 2001 From: Simon van Lierde Date: Sun, 6 Sep 2026 19:19:23 +0000 Subject: [PATCH 61/85] fix(alerting): compare with bool so threshold rules can fire TargetDown, HostDiskSpaceLow and HostDiskFilling returned 0 or a negative value on the state they watch; the shared threshold node fires on > 0. --- config/grafana/alerting/rules.yaml | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/config/grafana/alerting/rules.yaml b/config/grafana/alerting/rules.yaml index 6b53433..f9a1162 100644 --- a/config/grafana/alerting/rules.yaml +++ b/config/grafana/alerting/rules.yaml @@ -1,6 +1,7 @@ # Grafana-managed alert rules (ADR 0002). Every rule has the same shape: the -# PromQL carries its own comparison and returns a series only when the alert -# should fire; the threshold node only asks whether anything came back. +# PromQL carries its own comparison and the threshold node fires on value > 0. +# A comparison whose matching value can be 0 or negative (up == 0, a ratio, +# predict_linear) needs `bool`, or the rule can never fire. # # Keep the total around a dozen. The per-project rules beside this file are # rendered by bootstrap.sh. @@ -97,7 +98,7 @@ groups: refId: QUERY instant: true editorMode: code - expr: up == 0 + expr: up == bool 0 - *firing labels: severity: critical @@ -157,7 +158,7 @@ groups: editorMode: code expr: > (node_filesystem_avail_bytes{fstype!~"tmpfs|ramfs|overlay"} - / node_filesystem_size_bytes{fstype!~"tmpfs|ramfs|overlay"}) < 0.2 + / node_filesystem_size_bytes{fstype!~"tmpfs|ramfs|overlay"}) < bool 0.2 - *firing labels: severity: warning @@ -187,7 +188,7 @@ groups: instant: true editorMode: code expr: > - predict_linear(node_filesystem_avail_bytes{fstype!~"tmpfs|ramfs|overlay"}[6h], 3 * 86400) < 0 + predict_linear(node_filesystem_avail_bytes{fstype!~"tmpfs|ramfs|overlay"}[6h], 3 * 86400) < bool 0 and node_filesystem_size_bytes{fstype!~"tmpfs|ramfs|overlay"} > 10e9 - *firing labels: From 5562601c565c6fecf9f484dc1523d2445b047c4c Mon Sep 17 00:00:00 2001 From: Simon van Lierde Date: Sun, 6 Sep 2026 19:20:38 +0000 Subject: [PATCH 62/85] test(smoke): stop a scrape target and wait for TargetDown behind SMOKE_ALERTS - opt-in locally, on in CI: costs the rule's for plus one evaluation - poll() takes POLL_MAX for the longer wait --- .github/workflows/ci.yml | 7 +++++-- scripts/smoke.sh | 18 +++++++++++++++++- 2 files changed, 22 insertions(+), 3 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 607bfec..8c86d8b 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -3,10 +3,10 @@ name: CI # Pull requests only: main moves through PRs, so a push run would repeat the -# check. Prose changes skip the workflow. infra/ has its own (infra.yml). +# check. No path filter: gitleaks must see every PR, and a skipped workflow +# never reports the checks a PR needs. infra/ also has its own (infra.yml). on: pull_request: - paths-ignore: ["**.md", "docs/**", "LICENSE", "infra/**"] workflow_dispatch: permissions: @@ -38,7 +38,10 @@ jobs: - uses: extractions/setup-just@53165ef7e734c5c07cb06b3c8e7b647c5aa16db3 # v4.0.0 - run: cp .env.example .env - run: just validate + # SMOKE_ALERTS adds the TargetDown round trip (about 3 minutes). - run: just smoke + env: + SMOKE_ALERTS: "1" # The smoke failure messages point at these logs. - if: failure() run: just smoke-logs diff --git a/scripts/smoke.sh b/scripts/smoke.sh index c3c7c2b..24e89db 100755 --- a/scripts/smoke.sh +++ b/scripts/smoke.sh @@ -13,7 +13,7 @@ die() { echo "error: $*" >&2; exit 1; } # Admin basic auth through curl's stdin config: argv is world-readable in ps. gf() { printf 'user = "admin:%s"\n' "$GRAFANA_ADMIN_PASSWORD" | curl -sf -K - "$@"; } promq() { gf -G --data-urlencode "query=$1" "$url/api/datasources/proxy/uid/prometheus/api/v1/query"; } -poll() { local n=0; until "$@"; do n=$((n + 1)); [[ $n -lt 60 ]] || return 1; sleep 1; done; } +poll() { local n=0; until "$@"; do n=$((n + 1)); [[ $n -lt ${POLL_MAX:-60} ]] || return 1; sleep 1; done; } # ------------------------------------------------------------------ provisioning # Grafana skips a broken dashboard or a malformed alert group silently, so the @@ -88,4 +88,20 @@ smoke_log() { poll smoke_metric || die "the smoke metric never reached Prometheus with its project/env/department labels; see just smoke-logs" poll smoke_log || die "the smoke log never reached Loki with its department label; see just smoke-logs" +# ------------------------------------------------------------- alert pipeline +# Opt-in (SMOKE_ALERTS=1): stop one scrape target and wait for TargetDown to +# reach firing. Costs the rule's `for` (2m) plus an evaluation, so it is off +# for the local loop and on in CI. Guards the threshold-node contract in +# rules.yaml: a query whose matching value is 0 never fires without `bool`. +if [[ "${SMOKE_ALERTS:-}" == 1 ]]; then + docker compose -p "$project" stop -t 1 node-exporter >/dev/null + target_down_firing() { + gf "$url/api/prometheus/grafana/api/v1/rules" \ + | jq -e '.data.groups[].rules[] | select(.name == "TargetDown") | .state == "firing"' >/dev/null + } + POLL_MAX=300 poll target_down_firing || die "TargetDown did not fire within 5 minutes of stopping node-exporter" + docker compose -p "$project" start node-exporter >/dev/null + echo "TargetDown fired for the stopped node-exporter" +fi + echo "Stack healthy" From 484d38a011809198b4aedd84aad760368778466b Mon Sep 17 00:00:00 2001 From: Simon van Lierde Date: Sun, 6 Sep 2026 19:21:24 +0000 Subject: [PATCH 63/85] fix(backup): fail on tar errors, pin restore-check's compose files - set -o pipefail: tar's status was lost behind gzip and a partial archive passed - restore-check exports COMPOSE_FILE so the host's overlays stay out --- justfile | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/justfile b/justfile index af8ef65..0b81a40 100644 --- a/justfile +++ b/justfile @@ -191,7 +191,7 @@ backup: (_backup core_project "backups") _backup project dir: mkdir -p {{dir}} @-docker compose -p {{project}} unpause {{stateful}} >/dev/null 2>&1 - @rc=0; m=$(for s in {{stateful}}; do printf -- '-v {{project}}_%s_data:/data/%s ' $s $s; done); docker compose -p {{project}} pause {{stateful}} && docker run --rm --network none $m -v {{absolute_path(dir)}}:/backups {{alpine}} sh -c 'umask 077 && tar cf - -C /data . | gzip -1 > /backups/monitoring-$(date +%Y%m%d-%H%M%S).tar.gz' || rc=$?; docker compose -p {{project}} unpause {{stateful}} || { echo "error: unpause failed; the stack is still paused" >&2; rc=1; }; exit $rc + @rc=0; m=$(for s in {{stateful}}; do printf -- '-v {{project}}_%s_data:/data/%s ' $s $s; done); docker compose -p {{project}} pause {{stateful}} && docker run --rm --network none $m -v {{absolute_path(dir)}}:/backups {{alpine}} sh -c 'set -o pipefail; umask 077 && tar cf - -C /data . | gzip -1 > /backups/monitoring-$(date +%Y%m%d-%H%M%S).tar.gz' || rc=$?; docker compose -p {{project}} unpause {{stateful}} || { echo "error: unpause failed; the stack is still paused" >&2; rc=1; }; exit $rc @ls -lh {{dir}}/ | tail -1 # Restore a backup tarball into the volumes (stops the stack; wipes current state). @@ -209,9 +209,10 @@ _restore project file dir: m=$(for s in {{stateful}}; do printf -- '-v {{project}}_%s_data:/data/%s ' $s $s; done); docker run --rm --network none $m -v {{absolute_path(file)}}:/backup.tar.gz:ro {{alpine}} sh -c 'for d in /data/*; do find "$d" -mindepth 1 -delete; done && tar xzf /backup.tar.gz -C /data' # Needs a booted smoke stack (`just smoke`). Run it after touching _backup/_restore. +# COMPOSE_FILE is pinned so the host's overlay list stays out, as for `smoke`. # Round-trip backup and restore on the smoke stack. restore-check: - @d=$(mktemp -d) && just _backup {{smoke_project}} "$d" && f=$(ls "$d"/monitoring-*.tar.gz) && just _restore {{smoke_project}} "$f" "$d" && docker run --rm --network none -v {{smoke_project}}_grafana_data:/g:ro {{alpine}} test -s /g/grafana.db && echo "Backup round-trip ok"; rc=$?; rm -rf "$d"; exit $rc + @export COMPOSE_FILE=compose.yml:compose.sandbox.yml; d=$(mktemp -d) && just _backup {{smoke_project}} "$d" && f=$(ls "$d"/monitoring-*.tar.gz) && just _restore {{smoke_project}} "$f" "$d" && docker run --rm --network none -v {{smoke_project}}_grafana_data:/g:ro {{alpine}} test -s /g/grafana.db && echo "Backup round-trip ok"; rc=$?; rm -rf "$d"; exit $rc # `--wait` blocks on the healthchecks and fails if any container exits, so a # crash-looping service is caught (after the full timeout). scripts/smoke.sh From 2264c8766a628b1b5011d01cdf91d48af294e70b Mon Sep 17 00:00:00 2001 From: Simon van Lierde Date: Sun, 6 Sep 2026 19:21:33 +0000 Subject: [PATCH 64/85] build: digest-pin the lint, demo and curl images --- compose.demo.yml | 2 +- demo/Dockerfile | 4 ++-- justfile | 14 +++++++------- 3 files changed, 10 insertions(+), 10 deletions(-) diff --git a/compose.demo.yml b/compose.demo.yml index 861f976..e3761ed 100644 --- a/compose.demo.yml +++ b/compose.demo.yml @@ -31,7 +31,7 @@ services: logging: *demo-logging demo-load: - image: curlimages/curl:8.21.0 + image: curlimages/curl:8.21.0@sha256:7c12af72ceb38b7432ab85e1a265cff6ae58e06f95539d539b654f2cfa64bb13 restart: on-failure depends_on: [demo-api] command: diff --git a/demo/Dockerfile b/demo/Dockerfile index 103be01..847e9f8 100644 --- a/demo/Dockerfile +++ b/demo/Dockerfile @@ -1,10 +1,10 @@ -FROM python:3.14.7-slim AS builder +FROM python:3.14.7-slim@sha256:cad9a2c871761c413caa6fdd6441c783451e740a48aaeba60ae62a8b53525ef6 AS builder COPY --from=ghcr.io/astral-sh/uv:0.12.1 /uv /bin/ WORKDIR /app COPY pyproject.toml . RUN uv pip install --system --no-cache -r pyproject.toml -FROM python:3.14.7-slim +FROM python:3.14.7-slim@sha256:cad9a2c871761c413caa6fdd6441c783451e740a48aaeba60ae62a8b53525ef6 COPY --from=builder /usr/local/lib/python3.14/site-packages /usr/local/lib/python3.14/site-packages COPY --from=builder /usr/local/bin /usr/local/bin WORKDIR /app diff --git a/justfile b/justfile index 0b81a40..8eb1137 100644 --- a/justfile +++ b/justfile @@ -5,13 +5,13 @@ stateful := "grafana prometheus loki tempo" # Helper and lint images, pinned once. alpine := "alpine:3.24@sha256:28bd5fe8b56d1bd048e5babf5b10710ebe0bae67db86916198a6eec434943f8b" -jq := "ghcr.io/jqlang/jq:1.8.1" -yamllint := "pipelinecomponents/yamllint:0.35.13" -yamlfmt := "ghcr.io/google/yamlfmt:0.17.2" -actionlint := "rhysd/actionlint:1.7.12" -shellcheck := "koalaman/shellcheck:v0.11.0" -ruff := "ghcr.io/astral-sh/ruff:0.14.2" -tofu := "ghcr.io/opentofu/opentofu:1.12.3" +jq := "ghcr.io/jqlang/jq:1.8.1@sha256:4f34c6d23f4b1372ac789752cc955dc67c2ae177eb1b5860b75cdc5091ce6f91" +yamllint := "pipelinecomponents/yamllint:0.35.13@sha256:5ab5eb7da0ed5e606b07c1723fc8b275e925189f70ac259b26b7329cb5f8f44d" +yamlfmt := "ghcr.io/google/yamlfmt:0.17.2@sha256:fa6874890092db69f35ece6a50e574522cae2a59b6148a1f6ac6d510e5bcf3cc" +actionlint := "rhysd/actionlint:1.7.12@sha256:b1934ee5f1c509618f2508e6eb47ee0d3520686341fec936f3b79331f9315667" +shellcheck := "koalaman/shellcheck:v0.11.0@sha256:61862eba1fcf09a484ebcc6feea46f1782532571a34ed51fedf90dd25f925a8d" +ruff := "ghcr.io/astral-sh/ruff:0.14.2@sha256:636e27f3feb43800e44b0ad48c72811b500a2c6309d094b641a9bf2247f4dbff" +tofu := "ghcr.io/opentofu/opentofu:1.12.3@sha256:a0766d12f07b43e66f2ed40d7a8babe97d581d20339c68ad0ab561737af9a5b3" gitleaks := "zricethezav/gitleaks:v8.30.1@sha256:c00b6bd0aeb3071cbcb79009cb16a60dd9e0a7c60e2be9ab65d25e6bc8abbb7f" alloy := "grafana/alloy:v1.18.1@sha256:0f4434c92b3e6cdac38bb129b344e1790c246f7b6e2eaffcc16a5fa363240e33" lint_images := jq + " " + yamllint + " " + yamlfmt + " " + actionlint + " " + shellcheck + " " + ruff + " " + tofu + " " + gitleaks + " " + alloy From c3c54b84ecefb2d8eaefdaef290f9a2c56e38b23 Mon Sep 17 00:00:00 2001 From: Simon van Lierde Date: Sun, 6 Sep 2026 19:21:40 +0000 Subject: [PATCH 65/85] fix(bootstrap): report a missing or rejected admin password The readback loop folded a 401 into 'rule not provisioned'. Fail fast on an empty password, die on 401/403, escape the password for curl's config. --- bootstrap.sh | 26 ++++++++++++++++++++------ 1 file changed, 20 insertions(+), 6 deletions(-) diff --git a/bootstrap.sh b/bootstrap.sh index 8bfa2ee..1ad8595 100755 --- a/bootstrap.sh +++ b/bootstrap.sh @@ -95,15 +95,29 @@ if docker compose ps --status running --services 2>/dev/null | grep -qx grafana; if [[ -z "${GRAFANA_ADMIN_PASSWORD:-}" && -f .env ]]; then GRAFANA_ADMIN_PASSWORD="$(sed -n 's/^GRAFANA_ADMIN_PASSWORD=//p' .env | tail -1)" fi + [[ -n "${GRAFANA_ADMIN_PASSWORD:-}" ]] \ + || { echo "error: GRAFANA_ADMIN_PASSWORD is not set and not readable from .env; cannot verify the rule" >&2; exit 1; } + # curl's config parser unescapes \ and " inside a quoted value, so escape both. + gpw="${GRAFANA_ADMIN_PASSWORD//\\/\\\\}" + gpw="${gpw//\"/\\\"}" uid="proj-silent-${project}-${env_name}" for _ in $(seq 30); do sleep 2 - if printf 'user = "admin:%s"\n' "$GRAFANA_ADMIN_PASSWORD" \ - | curl -sf -K - "http://localhost:3000/api/v1/provisioning/alert-rules/${uid}" >/dev/null; then - echo "verified rule ${uid} is provisioned" - uid="" - break - fi + # Password on stdin, never argv. A wrong password would otherwise poll + # for a minute and then report the rule as missing. + code="$(printf 'user = "admin:%s"\n' "$gpw" \ + | curl -s -o /dev/null -w '%{http_code}' -K - "http://localhost:3000/api/v1/provisioning/alert-rules/${uid}")" || continue + case "$code" in + 200) + echo "verified rule ${uid} is provisioned" + uid="" + break + ;; + 401 | 403) + echo "error: grafana rejected the admin credentials (HTTP ${code}); check GRAFANA_ADMIN_PASSWORD" >&2 + exit 1 + ;; + esac done [[ -z "$uid" ]] || { echo "error: grafana restarted but rule ${uid} is not provisioned; check 'just logs grafana' for the rejected file" >&2; exit 1; } else From edc664ae5fafdc6318b3c8bbf860071b4781d9c8 Mon Sep 17 00:00:00 2001 From: Simon van Lierde Date: Sun, 6 Sep 2026 19:21:45 +0000 Subject: [PATCH 66/85] fix(infra): surface DNS API failures from the call sites, page policies at 100 lookup_dns_record died inside a command substitution, so an API error still read as 'no record'. --- infra/generate-imports.sh | 17 +++++++++++------ 1 file changed, 11 insertions(+), 6 deletions(-) diff --git a/infra/generate-imports.sh b/infra/generate-imports.sh index 501b3ae..e674f5e 100755 --- a/infra/generate-imports.sh +++ b/infra/generate-imports.sh @@ -59,11 +59,11 @@ done lookup_dns_record() { local hostname="$1" id # Type-filtered: an unfiltered .result[0] could bind a TXT record on the same - # name, and the apply would rewrite it into a CNAME. Two statements, not one - # pipeline: this runs inside `if`/`||`, where errexit is off, and a failed API - # call must not be reported as "no record". + # name, and the apply would rewrite it into a CNAME. This runs in `$(...)`, so + # a `die` here would only exit the subshell and read as "no record": an API + # failure gets its own status 2, which every call site distinguishes from 1. local body - body="$(api "zones/$zone/dns_records?name=$hostname&type=CNAME")" || die "DNS lookup for $hostname failed (token lacks DNS:Read?)" + body="$(api "zones/$zone/dns_records?name=$hostname&type=CNAME")" || return 2 id="$(jq -r '.result[0].id // empty' <<<"$body")" [[ -n "$id" ]] || return 1 printf '%s' "$id" @@ -81,13 +81,18 @@ if [[ -z "$tunnel_id" ]]; then exit 1 fi -grafana_record="$(lookup_dns_record "grafana.$domain")" || die "no CNAME found for grafana.$domain" +grafana_record="$(lookup_dns_record "grafana.$domain")" || { + [[ $? -ne 2 ]] || die "DNS lookup for grafana.$domain failed (token lacks DNS:Read?)" + die "no CNAME found for grafana.$domain" +} # otel. after the rename, otlp. before it. New name first, so a re-run is a no-op. for host in "otel.$domain" "otlp.$domain"; do if otel_record="$(lookup_dns_record "$host")"; then ingestion_host="$host" break + elif [[ $? -eq 2 ]]; then + die "DNS lookup for $host failed (token lacks DNS:Read?)" fi done [[ -n "${ingestion_host:-}" ]] || die "no CNAME found for otel.$domain or otlp.$domain" @@ -113,7 +118,7 @@ fi # An app built in the dashboard usually carries an inline policy with no id to # import; the apply then creates the reusable one and reattaches the app. -access_policies="$(api "accounts/$account/access/policies")" +access_policies="$(api "accounts/$account/access/policies?per_page=100")" access_policy_id="$(jq -r '.result[] | select(.name == "monitoring: allowed emails") | .id' <<<"$access_policies" | head -1)" echo "# Generated by generate-imports.sh on $(date -u +%Y-%m-%dT%H:%M:%SZ). Delete after a successful apply." From 0ed7ff9a7f30c043beb465cb07753beb2f5fb75f Mon Sep 17 00:00:00 2001 From: Simon van Lierde Date: Sun, 6 Sep 2026 19:22:06 +0000 Subject: [PATCH 67/85] fix(dashboards): scope stack-health host panels to the hub's node job --- dashboards/stack-health.json | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/dashboards/stack-health.json b/dashboards/stack-health.json index 362bc8f..91f3ed8 100644 --- a/dashboards/stack-health.json +++ b/dashboards/stack-health.json @@ -30,7 +30,7 @@ }, "targets": [ { - "expr": "100 * (1 - node_filesystem_avail_bytes{fstype!~\"tmpfs|ramfs|overlay\"} / node_filesystem_size_bytes{fstype!~\"tmpfs|ramfs|overlay\"})", + "expr": "100 * (1 - node_filesystem_avail_bytes{job=\"node\",fstype!~\"tmpfs|ramfs|overlay\"} / node_filesystem_size_bytes{job=\"node\",fstype!~\"tmpfs|ramfs|overlay\"})", "legendFormat": "{{mountpoint}}", "refId": "A" } @@ -77,7 +77,7 @@ }, "targets": [ { - "expr": "100 * (1 - avg(rate(node_cpu_seconds_total{mode=\"idle\"}[$__rate_interval])))", + "expr": "100 * (1 - avg(rate(node_cpu_seconds_total{job=\"node\",mode=\"idle\"}[$__rate_interval])))", "legendFormat": "used", "refId": "A" } @@ -111,7 +111,7 @@ }, "targets": [ { - "expr": "100 * (1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)", + "expr": "100 * (1 - node_memory_MemAvailable_bytes{job=\"node\"} / node_memory_MemTotal_bytes{job=\"node\"})", "legendFormat": "used", "refId": "A" } From 04d107e893fe2deb11a5d51a45ca31af85b642e5 Mon Sep 17 00:00:00 2001 From: Simon van Lierde Date: Sun, 6 Sep 2026 19:22:10 +0000 Subject: [PATCH 68/85] docs: five exposure guards, the new alerts, the smoke alert round trip --- CHANGELOG.md | 32 +++++++++++++++++++++++--------- README.md | 18 ++++++++++-------- 2 files changed, 33 insertions(+), 17 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index f91803c..94741bb 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -47,15 +47,16 @@ the config syntax. 1.0 waits for a second consumer to confirm the contract. the exact URLs given; Grafana honours the JWT settings and refuses a forged token; every scrape target is up; a metric and a log round-trip through the collector into Prometheus and Loki with the `project`, `env` and - `department` labels. `just restore-check` rehearses backup and restore on - the smoke volumes. + `department` labels. With `SMOKE_ALERTS=1` (on in CI) it also stops a + scrape target and waits for `TargetDown` to fire. `just restore-check` + rehearses backup and restore on the smoke volumes. - **`just check` is `lint` plus `validate`**. `lint` covers compose files, the rendered alert templates, YAML formatting, workflows, shell scripts, the demo's Python, OpenTofu formatting, dashboard JSON and the datasource uids it names, and git history for secrets. `validate` runs each stack config through the exact image the stack uses. `just hooks` installs git hooks via prek: gitleaks at commit, a Conventional Commits check on the - message, `check` at push. + message, `check` at push, `infra-validate` at push when `infra/` changed. - Dependabot watches the spoke images in `templates/` and the Cloudflare provider in `infra/`. Patch bumps arrive grouped. @@ -68,9 +69,9 @@ the config syntax. 1.0 waits for a second consumer to confirm the contract. source of truth; UI saves are off. - **Tempo stores traces and nothing else**: its metrics generator is gone. RED comes from the applications' own OTLP metrics (ADR 0002). -- CI runs on pull requests only, skips prose-only changes, and validates - `infra/` on its own workflow. `lint` and `validate` + `smoke` run on - separate jobs. `just demo-build` is gone. +- CI runs on pull requests only and validates `infra/` on its own workflow. + `lint` and `validate` + `smoke` run on separate jobs. `just demo-build` is + gone, and nothing in CI builds the demo image; `just demo` is the check. - `just smoke` and `just demo` each run under their own compose project and Grafana port (`compose.sandbox.yml`), so neither can touch a running stack. - Image bumps: Grafana 13.1.4, Loki 3.7.6, Tempo 3.0.3, Prometheus 3.13.2, @@ -100,7 +101,18 @@ the config syntax. 1.0 waits for a second consumer to confirm the contract. still paused when the unpause failed; `bootstrap.sh` printed the hash of empty input for a template missing from the tag; `generate-imports.sh` reported a failed DNS API call as "no record". `bootstrap.sh` now reads - its rule back from Grafana after the restart. + its rule back from Grafana after the restart, and says so when the admin + password is missing or rejected instead of blaming the rule. +- `TargetDown` and `HostDiskSpaceLow` could never fire on the state they + watch: `up == 0` and a 0% free ratio both evaluate to 0, and the threshold + node fires on value > 0. `HostDiskFilling` had the same defect from a + negative projection. All three carry `bool` now, and the smoke alert + round trip guards the contract. +- `just backup` lost tar's exit status behind the gzip pipe; a partial + archive reported success. `just restore-check` no longer inherits the + host's `COMPOSE_FILE`. +- The stack-health dashboard's host panels averaged spoke node metrics into + the hub's CPU, memory and disk. They are scoped to the hub's `node` job. - Grafana and the collector wait for Prometheus to report ready instead of racing it on a cold start. The demo load generator's error rate matches its advertised one in ten. @@ -116,10 +128,12 @@ the config syntax. 1.0 waits for a second consumer to confirm the contract. with no authentication of its own. - `GRAFANA_COOKIE_SECURE` marks the session cookie Secure with strict SameSite. With the tunnel overlay, `just up` refuses to start without it, - an `https://` root URL, non-default credentials, and a webhook URL. + an `https://` root URL, non-default credentials, a webhook URL, and with + `.env` or the OpenTofu state readable by other local users. `just lint` runs those guards both ways. - Hub images are digest-pinned, GitHub Actions are pinned to commit SHAs, the - tunnel token reaches cloudflared via environment rather than argv, and + tunnel token reaches cloudflared via environment rather than argv, the + smoke test's ingest token reaches curl through its stdin config, and `bootstrap.sh` prints `sha256sum -c` lines for the files a spoke vendors. ## [0.2.0] - 2026-07-05 diff --git a/README.md b/README.md index f73b4d0..6a4c295 100644 --- a/README.md +++ b/README.md @@ -88,10 +88,10 @@ Grafana: . `COMPOSE_FILE=compose.yml:compose.tunnel.yml`. Every recipe (`up`, `logs`, `ps`, `backup`) then acts on that set. -With the tunnel overlay active, `just up` refuses to start until four settings +With the tunnel overlay active, `just up` refuses to start until five settings are real: a generated `OTLP_AUTH_TOKEN`, a changed `GRAFANA_ADMIN_PASSWORD`, -`GRAFANA_ROOT_URL` pointing at the tunnel hostname, and -`GRAFANA_COOKIE_SECURE=true`. An empty `HEARTBEAT_URL` only warns. +`GRAFANA_ROOT_URL` pointing at the tunnel hostname, `GRAFANA_COOKIE_SECURE=true`, +and a non-empty `ALERT_WEBHOOK_URL`. An empty `HEARTBEAT_URL` only warns. The Cloudflare edge is code too. The tunnel, its hostnames, DNS, and the Access rule that puts an email one-time PIN in front of Grafana live in @@ -117,11 +117,12 @@ production shape with JWT auth on, then asserts that every dashboard and alert rule provisioned, that the contact points carry the URLs from the environment, that Grafana refuses a forged token, that every scrape target is up, and that a metric and a log posted through the collector come back out of -Prometheus and Loki with their labels. The assertions are in -[scripts/smoke.sh](scripts/smoke.sh). +Prometheus and Loki with their labels. `SMOKE_ALERTS=1 just smoke` also stops +a scrape target and waits for `TargetDown` to fire, about three minutes more. +The assertions are in [scripts/smoke.sh](scripts/smoke.sh). -CI runs `lint` on one job and `validate` plus `smoke` on another, on every -pull request. +CI runs `lint` on one job and `validate` plus `smoke` (with the alert round +trip) on another, on every pull request. ## Sending telemetry from a project @@ -139,7 +140,8 @@ agent that ships container logs and host metrics. Grafana evaluates and delivers the rules in `config/grafana/alerting/`: telemetry silent per project, container crash-looping or OOM-killed, scrape target down, OTel export failures, alert delivery failing, error rate above -5%, disk above 80%. There is no Alertmanager. Grafana rules can query Loki as +5%, disk above 80%, disk projected full within 3 days, and Prometheus head +series above 100k. There is no Alertmanager. Grafana rules can query Loki as well as Prometheus, and one engine means one answer to "who gets told". Notifications go to the webhook in `ALERT_WEBHOOK_URL` (ntfy, Slack, and so From d4d704782d386bbefc2d01177799bcef74317922 Mon Sep 17 00:00:00 2001 From: Simon van Lierde Date: Sun, 6 Sep 2026 19:57:42 +0000 Subject: [PATCH 69/85] fix(ci): update .env file permissions to prevent exposure guards rejection --- .github/workflows/ci.yml | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 8c86d8b..6f6ee90 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -27,7 +27,8 @@ jobs: with: fetch-depth: 0 - uses: extractions/setup-just@53165ef7e734c5c07cb06b3c8e7b647c5aa16db3 # v4.0.0 - - run: cp .env.example .env + # 600, or the exposure guards refuse it as world-readable. + - run: install -m 600 .env.example .env - run: just lint smoke: @@ -36,7 +37,8 @@ jobs: steps: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - uses: extractions/setup-just@53165ef7e734c5c07cb06b3c8e7b647c5aa16db3 # v4.0.0 - - run: cp .env.example .env + # 600, or the exposure guards refuse it as world-readable. + - run: install -m 600 .env.example .env - run: just validate # SMOKE_ALERTS adds the TargetDown round trip (about 3 minutes). - run: just smoke From 66b1a30e0a8323026ea9b6a17d8cbc14719c89e0 Mon Sep 17 00:00:00 2001 From: Simon van Lierde Date: Sun, 6 Sep 2026 23:01:34 +0000 Subject: [PATCH 70/85] refactor(stack): deduplicate compose, rules and the justfile - Fold the repeated hardening, logging and healthcheck blocks into `x-hardened` and `x-healthcheck` anchors; services override after the merge. - Give every alert rule the same `rule_defaults` anchor, and move the capacity group off a 1m interval: nothing there fires in under 15m. - Stamp `department` as a literal in the collector config. One value has ever been correct, and as an indexed Loki stream label a second one would be an index dimension, not a filter; the `DEPARTMENT` variable is gone from `.env`. - Pull the spoke's dummy environment into `spoke_env`, read the Alloy image from the template that pins it, and only pull lint images that are missing. - `bootstrap.sh` reads single keys out of `.env` through one helper and prints the vendoring curls from the same list it hashes. - CI copies `.env.example` with mode 600, or the exposure guards reject it. --- .env.example | 3 - .github/workflows/infra.yml | 2 +- .pre-commit-config.yaml | 5 +- README.md | 2 +- bootstrap.sh | 25 ++-- compose.yml | 62 ++++----- config/grafana/alerting/rules.yaml | 205 ++++++++++------------------- config/otel-collector.yaml | 5 +- docs/RUNBOOK.md | 6 +- justfile | 56 +++++--- scripts/smoke.sh | 9 +- templates/README.md | 5 +- 12 files changed, 158 insertions(+), 227 deletions(-) diff --git a/.env.example b/.env.example index fc2a252..078b0d6 100644 --- a/.env.example +++ b/.env.example @@ -17,9 +17,6 @@ GRAFANA_ROOT_URL=http://localhost:3000 # or logins break. GRAFANA_COOKIE_SECURE=false -# Stamped onto every signal as the `department` label. -DEPARTMENT=cml - # Bearer token every telemetry sender must present (Authorization: Bearer ). # The default only suits local use. Generate a real one for production: # openssl rand -hex 32 diff --git a/.github/workflows/infra.yml b/.github/workflows/infra.yml index 871a4c5..8f5c130 100644 --- a/.github/workflows/infra.yml +++ b/.github/workflows/infra.yml @@ -21,5 +21,5 @@ jobs: steps: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - uses: extractions/setup-just@53165ef7e734c5c07cb06b3c8e7b647c5aa16db3 # v4.0.0 - - run: cp .env.example .env + - run: install -m 600 .env.example .env # 600 or the exposure guards reject it - run: just infra-validate diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index 6ce1b91..333638e 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -8,10 +8,7 @@ repos: hooks: - id: gitleaks name: gitleaks (staged) - entry: >- - docker run --rm --network none -v .:/repo:ro - zricethezav/gitleaks:v8.30.1@sha256:c00b6bd0aeb3071cbcb79009cb16a60dd9e0a7c60e2be9ab65d25e6bc8abbb7f - git --pre-commit --staged --redact --no-banner /repo + entry: just _gitleaks-staged language: system pass_filenames: false always_run: true diff --git a/README.md b/README.md index 6a4c295..53f3333 100644 --- a/README.md +++ b/README.md @@ -42,7 +42,7 @@ whole demo stack. The demo runs under its own compose project on its own port, so it never touches a stack already running on the host. The same holds for `just smoke` -on :3001. Override the ports with `DEMO_PORT` and `SMOKE_PORT`. +on :3001. ## How it works diff --git a/bootstrap.sh b/bootstrap.sh index 1ad8595..2fadf73 100755 --- a/bootstrap.sh +++ b/bootstrap.sh @@ -28,10 +28,12 @@ fi root="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)" cd "$root" -# A bare ./bootstrap.sh does not load .env, so read this one key from it. +# A bare ./bootstrap.sh does not load .env, so read single keys out of it. # Sourcing the whole file would drag the stack's secrets into scope. -if [[ -z "${HEALTHCHECKS_API_KEY:-}" && -f .env ]]; then - HEALTHCHECKS_API_KEY="$(sed -n 's/^HEALTHCHECKS_API_KEY=//p' .env | tail -1)" +env_get() { [[ -f .env ]] && sed -n "s/^$1=//p" .env | tail -1; } + +if [[ -z "${HEALTHCHECKS_API_KEY:-}" ]]; then + HEALTHCHECKS_API_KEY="$(env_get HEALTHCHECKS_API_KEY)" export HEALTHCHECKS_API_KEY fi # Flat, not a subdirectory: Grafana's alerting provisioner does not recurse, and @@ -92,8 +94,8 @@ if docker compose ps --status running --services 2>/dev/null | grep -qx grafana; || echo "WARNING: could not restart grafana; run 'docker compose up -d --force-recreate grafana'" >&2 # Grafana skips a malformed alert group with only a log line, so read the # rule back. - if [[ -z "${GRAFANA_ADMIN_PASSWORD:-}" && -f .env ]]; then - GRAFANA_ADMIN_PASSWORD="$(sed -n 's/^GRAFANA_ADMIN_PASSWORD=//p' .env | tail -1)" + if [[ -z "${GRAFANA_ADMIN_PASSWORD:-}" ]]; then + GRAFANA_ADMIN_PASSWORD="$(env_get GRAFANA_ADMIN_PASSWORD)" fi [[ -n "${GRAFANA_ADMIN_PASSWORD:-}" ]] \ || { echo "error: GRAFANA_ADMIN_PASSWORD is not set and not readable from .env; cannot verify the rule" >&2; exit 1; } @@ -125,9 +127,7 @@ else fi # ------------------------------------------------------- 4. healthchecks.io + printout -hc_note="create these by hand at https://healthchecks.io and note their ping URLs" if [[ -n "${HEALTHCHECKS_API_KEY:-}" ]]; then - hc_note="created via API" # Default job set; override per project with HC_JOBS="backup nightly-sync" etc. for job in ${HC_JOBS:-backup watchdog restore-check}; do # Key via curl's stdin config, not -H: argv is readable in `ps`. @@ -138,17 +138,20 @@ if [[ -n "${HEALTHCHECKS_API_KEY:-}" ]]; then | sed -n 's/.*"ping_url": *"\([^"]*\)".*/ PING_'"$(echo "$job" | tr 'a-z-' 'A-Z_')"'=\1/p' done else - echo "note HEALTHCHECKS_API_KEY unset; ${hc_note}" + echo "note HEALTHCHECKS_API_KEY unset; create these by hand at https://healthchecks.io and note their ping URLs" fi # Computed before the heredoc: a substitution inside `cat < OTLP_AUTH_TOKEN= TELEMETRY_EDGE_KEY= # only if the project's egress crosses a WAF -GPU_METRICS= # 1 on hosts with an NVIDIA card ──────────────────────────────────────────────────────────────────────────── Vendor the templates on the project host (pinned at ${tag}) ──────────────────────────────────────────────────────────────────────────── mkdir -p deploy/alloy scripts -curl -fsSL -o deploy/alloy/config.alloy ${repo_raw}/alloy/config.alloy -curl -fsSL -o compose.telemetry.yml ${repo_raw}/compose.telemetry.yml -curl -fsSL -o compose.telemetry.gpu.yml ${repo_raw}/compose.telemetry.gpu.yml -curl -fsSL -o scripts/run_scheduled.sh ${repo_raw}/run_scheduled.sh +${curls} Verify before executing anything (the hashes come from the ${tag} tag): sha256sum -c <<'SUM' diff --git a/compose.yml b/compose.yml index c1c7c68..9a8d6b9 100644 --- a/compose.yml +++ b/compose.yml @@ -16,12 +16,24 @@ x-logging: &default-logging max-size: "10m" max-file: "3" -x-security: &default-security - - no-new-privileges:true - # Every service drops all capabilities and carries a pids limit. Each mem_limit # is a ceiling sized from observed usage, so one runaway component cannot OOM a -# host that also runs production. +# host that also runs production. Services that need a tighter pids_limit +# override it after the merge. +x-hardened: &hardened + cap_drop: [ALL] + security_opt: + - no-new-privileges:true + pids_limit: 1024 + logging: *default-logging + +x-healthcheck: &healthcheck + interval: 15s + timeout: 3s + retries: 5 + start_period: 30s + # Probe often while starting so `up --wait` returns on the first pass. + start_interval: 2s services: otel-collector: @@ -44,14 +56,11 @@ services: command: ["--config=/etc/otelcol/config.yaml"] environment: OTLP_AUTH_TOKEN: ${OTLP_AUTH_TOKEN:?set OTLP_AUTH_TOKEN in .env} - DEPARTMENT: ${DEPARTMENT:-cml} # Keep in step with memory_limiter in config/otel-collector.yaml. mem_limit: 512m + !!merge <<: *hardened pids_limit: 512 - cap_drop: [ALL] - security_opt: *default-security networks: [default, backend] - logging: *default-logging loki: image: grafana/loki:3.7.6@sha256:efd47c67f9bac88ca29bcf8cb997d9ab29d1848bd0aff579282295542a745952 @@ -62,11 +71,8 @@ services: command: ["-config.file=/etc/loki/loki.yaml"] # No healthcheck: the image is distroless, with no shell or wget to probe with. mem_limit: 2g - pids_limit: 1024 - cap_drop: [ALL] - security_opt: *default-security + !!merge <<: *hardened networks: [backend] - logging: *default-logging tempo: image: grafana/tempo:3.0.3@sha256:0296560ac66f8a3600d7fb3014a52c189d4d9c3549ad6ff441bf2409855d68d5 @@ -77,11 +83,8 @@ services: command: ["-config.file=/etc/tempo/tempo.yaml"] # Distroless too, so no healthcheck. mem_limit: 2g - pids_limit: 1024 - cap_drop: [ALL] - security_opt: *default-security + !!merge <<: *hardened networks: [backend] - logging: *default-logging prometheus: image: prom/prometheus:v3.13.2@sha256:508729e0e2d18e11fd742a5a5ca70e557b940a93948c3c95fd0123a6fd538b69 @@ -99,19 +102,11 @@ services: # Whichever limit hits first wins; the disk-space alert is the backstop. - --storage.tsdb.retention.size=15GB healthcheck: + !!merge <<: *healthcheck test: ["CMD", "wget", "-q", "--spider", "http://localhost:9090/-/ready"] - interval: 15s - timeout: 3s - retries: 5 - start_period: 30s - # Probe often while starting so `up --wait` returns on the first pass. - start_interval: 2s mem_limit: 2g - pids_limit: 1024 - cap_drop: [ALL] - security_opt: *default-security + !!merge <<: *hardened networks: [backend] - logging: *default-logging node-exporter: image: prom/node-exporter:v1.12.1@sha256:1b4e4438faca4dd7e001dd445d161a4a2091b0fededa84093b3a8dfeae1f1be0 @@ -119,17 +114,15 @@ services: command: ["--path.rootfs=/host"] pid: host mem_limit: 128m + !!merge <<: *hardened # Holds pid:host plus the whole host filesystem, hence the tightest limits. - cap_drop: [ALL] read_only: true pids_limit: 64 - security_opt: *default-security volumes: # No rslave (unsupported on Docker Desktop) and no network_mode:host, so # node_network_* describes the container veth, not host NICs. - /:/host:ro networks: [backend] - logging: *default-logging grafana: image: grafana/grafana:13.1.4@sha256:9be3a3ccdb06bcbb127f888b0c4c1d151837443e478887897a63a27d7b348043 @@ -174,18 +167,11 @@ services: GF_AUTH_JWT_USERNAME_CLAIM: email GF_AUTH_JWT_AUTO_SIGN_UP: "true" healthcheck: + !!merge <<: *healthcheck test: ["CMD", "wget", "-q", "--spider", "http://localhost:3000/api/health"] - interval: 15s - timeout: 3s - retries: 5 - start_period: 30s - start_interval: 2s mem_limit: 1g - pids_limit: 1024 - cap_drop: [ALL] - security_opt: *default-security + !!merge <<: *hardened networks: [default, backend] - logging: *default-logging # Loki, Prometheus and Tempo have no authentication of their own, so only # Grafana and the collector can reach them. cloudflared stays on `default`: its diff --git a/config/grafana/alerting/rules.yaml b/config/grafana/alerting/rules.yaml index f9a1162..dbdc80f 100644 --- a/config/grafana/alerting/rules.yaml +++ b/config/grafana/alerting/rules.yaml @@ -6,6 +6,38 @@ # Keep the total around a dozen. The per-project rules beside this file are # rendered by bootstrap.sh. apiVersion: 1 + +# Anchors only. Grafana reads apiVersion and groups; every rule below repeats +# these four blocks verbatim apart from `for` and `expr`, so they live here once +# and are merged in. Nothing here defines a rule. +_defaults: + - &rule_defaults + condition: FIRING + noDataState: OK + execErrState: Error + isPaused: false + - &query_node + refId: QUERY + datasourceUid: prometheus + relativeTimeRange: + from: 3600 + to: 0 + - &query_model + refId: QUERY + instant: true + editorMode: code + - &firing + refId: FIRING + datasourceUid: __expr__ + model: + refId: FIRING + type: threshold + expression: QUERY + conditions: + - evaluator: + type: gt + params: [0] + groups: - orgId: 1 name: meta @@ -16,34 +48,14 @@ groups: # heartbeat and the switch alarms. - uid: watchdog-heartbeat title: Watchdog - condition: FIRING + !!merge <<: *rule_defaults for: 0m - noDataState: OK - execErrState: Error - isPaused: false data: - - refId: QUERY - datasourceUid: prometheus - relativeTimeRange: - from: 3600 - to: 0 + - !!merge <<: *query_node model: - refId: QUERY - instant: true - editorMode: code + !!merge <<: *query_model expr: vector(1) - - &firing - # Identical on every rule; aliased as `- *firing` below. - refId: FIRING - datasourceUid: __expr__ - model: - refId: FIRING - type: threshold - expression: QUERY - conditions: - - evaluator: - type: gt - params: [0] + - *firing labels: severity: none annotations: @@ -52,21 +64,12 @@ groups: # Cannot page when the webhook itself is broken; the heartbeat covers that. - uid: alert-delivery-failing title: AlertDeliveryFailing - condition: FIRING + !!merge <<: *rule_defaults for: 10m - noDataState: OK - execErrState: Error - isPaused: false data: - - refId: QUERY - datasourceUid: prometheus - relativeTimeRange: - from: 3600 - to: 0 + - !!merge <<: *query_node model: - refId: QUERY - instant: true - editorMode: code + !!merge <<: *query_model expr: rate(grafana_alerting_notifications_failed_total[10m]) > 0 - *firing labels: @@ -83,21 +86,12 @@ groups: rules: - uid: target-down title: TargetDown - condition: FIRING + !!merge <<: *rule_defaults for: 2m - noDataState: OK - execErrState: Error - isPaused: false data: - - refId: QUERY - datasourceUid: prometheus - relativeTimeRange: - from: 3600 - to: 0 + - !!merge <<: *query_node model: - refId: QUERY - instant: true - editorMode: code + !!merge <<: *query_model expr: up == bool 0 - *firing labels: @@ -107,21 +101,12 @@ groups: description: "Prometheus cannot scrape {{ $labels.instance }} (job {{ $labels.job }}) for 2 minutes." - uid: otel-export-failures title: OtelExportFailures - condition: FIRING + !!merge <<: *rule_defaults for: 5m - noDataState: OK - execErrState: Error - isPaused: false data: - - refId: QUERY - datasourceUid: prometheus - relativeTimeRange: - from: 3600 - to: 0 + - !!merge <<: *query_node model: - refId: QUERY - instant: true - editorMode: code + !!merge <<: *query_model expr: > sum by (exporter) (rate(otelcol_exporter_send_failed_spans[5m])) > 0 or sum by (exporter) (rate(otelcol_exporter_send_failed_metric_points[5m])) > 0 @@ -137,25 +122,19 @@ groups: - orgId: 1 name: capacity folder: Stack alerts - interval: 1m + # Every rule here waits 15m or more before firing, and the disk fit reads 6h + # of samples per filesystem; evaluating that once a minute bought no warning + # time. 5m is 5x less work for the same alert. + interval: 5m rules: - uid: host-disk-space-low title: HostDiskSpaceLow - condition: FIRING + !!merge <<: *rule_defaults for: 15m - noDataState: OK - execErrState: Error - isPaused: false data: - - refId: QUERY - datasourceUid: prometheus - relativeTimeRange: - from: 3600 - to: 0 + - !!merge <<: *query_node model: - refId: QUERY - instant: true - editorMode: code + !!merge <<: *query_model expr: > (node_filesystem_avail_bytes{fstype!~"tmpfs|ramfs|overlay"} / node_filesystem_size_bytes{fstype!~"tmpfs|ramfs|overlay"}) < bool 0.2 @@ -172,21 +151,12 @@ groups: # EFI, snap) are skipped: they cross zero on noise. - uid: host-disk-filling title: HostDiskFilling - condition: FIRING + !!merge <<: *rule_defaults for: 30m - noDataState: OK - execErrState: Error - isPaused: false data: - - refId: QUERY - datasourceUid: prometheus - relativeTimeRange: - from: 3600 - to: 0 + - !!merge <<: *query_node model: - refId: QUERY - instant: true - editorMode: code + !!merge <<: *query_model expr: > predict_linear(node_filesystem_avail_bytes{fstype!~"tmpfs|ramfs|overlay"}[6h], 3 * 86400) < bool 0 and node_filesystem_size_bytes{fstype!~"tmpfs|ramfs|overlay"} > 10e9 @@ -201,25 +171,17 @@ groups: Extrapolated from the last 6 hours. Find what is growing (a spoke shipping more than before, a log loop, a backup that stopped rotating) before HostDiskSpaceLow makes it urgent. - # ~7x the baseline of ~15k series with one spoke; raise it as spokes are - # onboarded. + # ~14x the baseline of ~7k series with one spoke (measured after the scrape + # trimming in config/prometheus.yaml and the agent config); raise it as spokes + # are onboarded. - uid: prometheus-cardinality title: PrometheusCardinalityHigh - condition: FIRING + !!merge <<: *rule_defaults for: 30m - noDataState: OK - execErrState: Error - isPaused: false data: - - refId: QUERY - datasourceUid: prometheus - relativeTimeRange: - from: 3600 - to: 0 + - !!merge <<: *query_node model: - refId: QUERY - instant: true - editorMode: code + !!merge <<: *query_model expr: prometheus_tsdb_head_series > 100000 - *firing labels: @@ -239,21 +201,12 @@ groups: rules: - uid: container-restarting title: ContainerRestarting - condition: FIRING + !!merge <<: *rule_defaults for: 10m - noDataState: OK - execErrState: Error - isPaused: false data: - - refId: QUERY - datasourceUid: prometheus - relativeTimeRange: - from: 3600 - to: 0 + - !!merge <<: *query_node model: - refId: QUERY - instant: true - editorMode: code + !!merge <<: *query_model expr: changes(container_start_time_seconds{name!=""}[1h]) > 3 - *firing labels: @@ -263,21 +216,12 @@ groups: description: "{{ $labels.name }} on {{ $labels.host_name }} has restarted more than 3 times in the last hour." - uid: container-oom-killed title: ContainerOOMKilled - condition: FIRING + !!merge <<: *rule_defaults for: 0m - noDataState: OK - execErrState: Error - isPaused: false data: - - refId: QUERY - datasourceUid: prometheus - relativeTimeRange: - from: 3600 - to: 0 + - !!merge <<: *query_node model: - refId: QUERY - instant: true - editorMode: code + !!merge <<: *query_model expr: increase(container_oom_events_total[5m]) > 0 - *firing labels: @@ -292,21 +236,12 @@ groups: rules: - uid: high-error-rate title: HighErrorRate - condition: FIRING + !!merge <<: *rule_defaults for: 5m - noDataState: OK - execErrState: Error - isPaused: false data: - - refId: QUERY - datasourceUid: prometheus - relativeTimeRange: - from: 3600 - to: 0 + - !!merge <<: *query_node model: - refId: QUERY - instant: true - editorMode: code + !!merge <<: *query_model # Keyed on (job, project, env): on job alone, a healthy prod # service dilutes a broken staging one below the threshold. expr: > diff --git a/config/otel-collector.yaml b/config/otel-collector.yaml index ec6e886..76bc028 100644 --- a/config/otel-collector.yaml +++ b/config/otel-collector.yaml @@ -24,10 +24,13 @@ receivers: processors: # Stamped here with upsert, so a sender cannot claim another department. + # Literal rather than an env knob: one value has ever been correct, and as an + # indexed Loki stream label a second one would be an index dimension, not a + # filter. Make it a variable again when a second department federates. resource/department: attributes: - key: department - value: ${env:DEPARTMENT} + value: cml action: upsert # Sized to the container's mem_limit (512m in compose.yml); keep the two in step. diff --git a/docs/RUNBOOK.md b/docs/RUNBOOK.md index f842ef8..6129c5c 100644 --- a/docs/RUNBOOK.md +++ b/docs/RUNBOOK.md @@ -183,8 +183,10 @@ cd infra && tofu apply - **State lives on this host only, and it is a secret.** It stores the tunnel secret and every API response in plain text. Copy `infra/terraform.tfstate` off-host next to the backups. Losing it orphans - the Cloudflare resources. They keep running, but the next apply creates - duplicates, and recovery is `tofu import` by hand. + the Cloudflare resources: they keep running, but the next apply creates + duplicates. Recovery is the same import path as the first apply above, + `infra/generate-imports.sh > infra/imports.tf`, which reads the live objects + back out of the Cloudflare API and adopts them into a fresh state. ## Upgrading images diff --git a/justfile b/justfile index 8eb1137..2a90596 100644 --- a/justfile +++ b/justfile @@ -13,15 +13,14 @@ shellcheck := "koalaman/shellcheck:v0.11.0@sha256:61862eba1fcf09a484ebcc6feea46f ruff := "ghcr.io/astral-sh/ruff:0.14.2@sha256:636e27f3feb43800e44b0ad48c72811b500a2c6309d094b641a9bf2247f4dbff" tofu := "ghcr.io/opentofu/opentofu:1.12.3@sha256:a0766d12f07b43e66f2ed40d7a8babe97d581d20339c68ad0ab561737af9a5b3" gitleaks := "zricethezav/gitleaks:v8.30.1@sha256:c00b6bd0aeb3071cbcb79009cb16a60dd9e0a7c60e2be9ab65d25e6bc8abbb7f" -alloy := "grafana/alloy:v1.18.1@sha256:0f4434c92b3e6cdac38bb129b344e1790c246f7b6e2eaffcc16a5fa363240e33" -lint_images := jq + " " + yamllint + " " + yamlfmt + " " + actionlint + " " + shellcheck + " " + ruff + " " + tofu + " " + gitleaks + " " + alloy +lint_images := jq + " " + yamllint + " " + yamlfmt + " " + actionlint + " " + shellcheck + " " + ruff + " " + tofu + " " + gitleaks # `demo` and `smoke` each run a throwaway copy of the core stack under their own # compose project and Grafana port (compose.sandbox.yml), so neither can touch a # stack already running on this host. The explicit -f list keeps the host's # COMPOSE_FILE out of both. demo_project := "monitoring-demo" -demo_port := env("DEMO_PORT", "3002") +demo_port := "3002" compose_demo := "SANDBOX_PORT=" + demo_port + " docker compose -p " + demo_project + " -f compose.yml -f compose.demo.yml -f compose.sandbox.yml" # The project name compose will use, so the queue volume and backups target @@ -33,10 +32,14 @@ core_project := env("COMPOSE_PROJECT_NAME", "monitoring") # real Zero Trust team name can, so the JWK URL can never resolve to a team # someone registers. smoke_project := "monitoring-smoke" -smoke_port := env("SMOKE_PORT", "3001") +smoke_port := "3001" smoke_env := "SANDBOX_PORT=" + smoke_port + " GRAFANA_JWT_AUTH=true CF_ACCESS_TEAM_DOMAIN=smoke.invalid CF_ACCESS_AUD=smoke ALERT_WEBHOOK_URL=https://smoke.invalid/alerts HEARTBEAT_URL=https://smoke.invalid/heartbeat" compose_smoke := smoke_env + " docker compose -p " + smoke_project + " -f compose.yml -f compose.sandbox.yml" +# The spoke overlays interpolate these, so both rendering them and reading an +# image ref out of them needs the set. +spoke_env := "ENVIRONMENT=dummy PROJECT=dummy COMPOSE_PROJECT_NAME=dummy OTEL_EXPORTER_OTLP_ENDPOINT=https://dummy OTLP_AUTH_TOKEN=dummy" + # dashboards/*.json as mounted at /dashboards. 2>/dev/null so an empty # dashboards/ doesn't abort every recipe; `lint` refuses the empty list instead. dash_paths := `ls dashboards/*.json 2>/dev/null | sed 's|^dashboards|/dashboards|' | tr '\n' ' '` @@ -115,7 +118,8 @@ check: lint validate # Static checks in tool images (~280 MB cold, seconds warm). lint: - @printf '%s\n' {{lint_images}} | xargs -P 8 -n 1 docker pull -q >/dev/null + # Digest-pinned, so anything already local is current; only fetch what is missing. + @printf '%s\n' {{lint_images}} | xargs -P 8 -I{} sh -c 'docker image inspect {} >/dev/null 2>&1 || docker pull -q {} >/dev/null' # Explicit -f, not the host's COMPOSE_FILE, so lint means the same here as in CI. docker compose -f compose.yml config -q CLOUDFLARE_TUNNEL_TOKEN=dummy docker compose -f compose.yml -f compose.tunnel.yml config -q @@ -123,7 +127,7 @@ lint: # compose_smoke turns the JWT interpolation on. {{compose_smoke}} config -q # The spoke overlays, which otherwise first fail on a project host after vendoring. - ENVIRONMENT=dummy PROJECT=dummy COMPOSE_PROJECT_NAME=dummy OTEL_EXPORTER_OTLP_ENDPOINT=https://dummy OTLP_AUTH_TOKEN=dummy docker compose -f templates/compose.telemetry.yml -f templates/compose.telemetry.gpu.yml config -q + {{spoke_env}} docker compose -f templates/compose.telemetry.yml -f templates/compose.telemetry.gpu.yml config -q # The exposure guards, both ways: a fully set .env passes, and each # documented default is refused on its own. @good="OTLP_AUTH_TOKEN=t GRAFANA_ADMIN_PASSWORD=p GRAFANA_ROOT_URL=https://g.example GRAFANA_COOKIE_SECURE=true GRAFANA_JWT_AUTH=true CF_ACCESS_TEAM_DOMAIN=d CF_ACCESS_AUD=a HEARTBEAT_URL=https://h ALERT_WEBHOOK_URL=https://w"; \ @@ -152,21 +156,24 @@ lint: # Secrets in git history. Scans commits, not the working tree, so the # gitignored .env never trips it. Needs full history (see ci.yml). docker run --rm --network none -v .:/repo:ro {{gitleaks}} git --redact --no-banner /repo - # The agent config every project host vendors. Image ref matches - # templates/compose.telemetry.yml; keep them in step. - docker run --rm --network none -v ./templates/alloy/config.alloy:/etc/alloy/config.alloy:ro -e COMPOSE_PROJECT_NAME=dummy -e ENVIRONMENT=dummy -e PROJECT=dummy -e OTEL_EXPORTER_OTLP_ENDPOINT=https://dummy -e OTLP_AUTH_TOKEN=dummy -e TELEMETRY_EDGE_KEY= {{alloy}} validate /etc/alloy/config.alloy + # The agent config every project host vendors, checked by the Alloy build + # the template actually pins. + docker run --rm --network none -v ./templates/alloy/config.alloy:/etc/alloy/config.alloy:ro -e COMPOSE_PROJECT_NAME=dummy -e ENVIRONMENT=dummy -e PROJECT=dummy -e OTEL_EXPORTER_OTLP_ENDPOINT=https://dummy -e OTLP_AUTH_TOKEN=dummy -e TELEMETRY_EDGE_KEY= $({{spoke_env}} just _image templates/compose.telemetry.yml alloy) validate /etc/alloy/config.alloy -# Image ref of one compose.yml service. (`config --images ` also lists -# the service's dependencies, hence the json route.) -_image service: - @docker compose -f compose.yml config --format json | docker run --rm -i {{jq}} -er '.services["{{service}}"].image // error("no service {{service}} in compose.yml")' +# Image ref of one service in a compose file. (`config --images ` also +# lists the service's dependencies, hence the json route.) +_image file service: + @docker compose -f {{file}} config --format json | docker run --rm -i {{jq}} -er '.services["{{service}}"].image // error("no service {{service}} in {{file}}")' -# Run each config through the binary that will load it. +# One compose render feeds all four; each config goes through the binary that +# will load it. validate: - docker run --rm --network none -v ./config/prometheus.yaml:/etc/prometheus/prometheus.yaml:ro --entrypoint promtool $(just _image prometheus) check config /etc/prometheus/prometheus.yaml - docker run --rm --network none -e OTLP_AUTH_TOKEN=dummy -e DEPARTMENT=dummy -v ./config/otel-collector.yaml:/etc/otelcol/config.yaml:ro $(just _image otel-collector) validate --config=/etc/otelcol/config.yaml - docker run --rm --network none -v ./config/loki.yaml:/etc/loki/loki.yaml:ro $(just _image loki) -config.file=/etc/loki/loki.yaml -verify-config - docker run --rm --network none -v ./config/tempo.yaml:/etc/tempo/tempo.yaml:ro $(just _image tempo) -config.file=/etc/tempo/tempo.yaml -config.verify=true + @set -e; cfg=$(docker compose -f compose.yml config --format json); \ + img() { printf '%s' "$cfg" | docker run --rm -i {{jq}} -er ".services[\"$1\"].image"; }; \ + docker run --rm --network none -v ./config/prometheus.yaml:/etc/prometheus/prometheus.yaml:ro --entrypoint promtool $(img prometheus) check config /etc/prometheus/prometheus.yaml; \ + docker run --rm --network none -e OTLP_AUTH_TOKEN=dummy -v ./config/otel-collector.yaml:/etc/otelcol/config.yaml:ro $(img otel-collector) validate --config=/etc/otelcol/config.yaml; \ + docker run --rm --network none -v ./config/loki.yaml:/etc/loki/loki.yaml:ro $(img loki) -config.file=/etc/loki/loki.yaml -verify-config; \ + docker run --rm --network none -v ./config/tempo.yaml:/etc/tempo/tempo.yaml:ro $(img tempo) -config.file=/etc/tempo/tempo.yaml -config.verify=true # Runs against a copy of the sources: state and tfvars never enter the # container, which has network access to fetch the provider. @@ -174,6 +181,10 @@ validate: infra-validate: @d=$(mktemp -d) && cp infra/main.tf infra/.terraform.lock.hcl "$d"/ && docker run --rm --entrypoint sh -v "$d":/src:ro {{tofu}} -c 'mkdir /work && cp /src/main.tf /src/.terraform.lock.hcl /work && cd /work && tofu init -backend=false -input=false >/dev/null && tofu validate'; rc=$?; rm -rf "$d"; exit $rc +# gitleaks over the staged diff (the pre-commit hook; see .pre-commit-config.yaml). +_gitleaks-staged: + @docker run --rm --network none -v .:/repo:ro {{gitleaks}} git --pre-commit --staged --redact --no-banner /repo + # Format YAML in place (--user so the rewritten files stay yours). fmt: docker run --rm --network none --user "$(id -u):$(id -g)" -v .:/code -w /code {{yamlfmt}} . @@ -188,10 +199,13 @@ backup: (_backup core_project "backups") # gzip -1: the stack is paused for as long as the tar runs, and the chunks are # already compressed. The unpause runs unconditionally (`pause` can fail # halfway), and a failed unpause fails the recipe. +_mounts project: + @for s in {{stateful}}; do printf -- '-v {{project}}_%s_data:/data/%s ' $s $s; done + _backup project dir: mkdir -p {{dir}} @-docker compose -p {{project}} unpause {{stateful}} >/dev/null 2>&1 - @rc=0; m=$(for s in {{stateful}}; do printf -- '-v {{project}}_%s_data:/data/%s ' $s $s; done); docker compose -p {{project}} pause {{stateful}} && docker run --rm --network none $m -v {{absolute_path(dir)}}:/backups {{alpine}} sh -c 'set -o pipefail; umask 077 && tar cf - -C /data . | gzip -1 > /backups/monitoring-$(date +%Y%m%d-%H%M%S).tar.gz' || rc=$?; docker compose -p {{project}} unpause {{stateful}} || { echo "error: unpause failed; the stack is still paused" >&2; rc=1; }; exit $rc + @rc=0; m=$(just _mounts {{project}}); docker compose -p {{project}} pause {{stateful}} && docker run --rm --network none $m -v {{absolute_path(dir)}}:/backups {{alpine}} sh -c 'set -o pipefail; umask 077 && tar cf - -C /data . | gzip -1 > /backups/monitoring-$(date +%Y%m%d-%H%M%S).tar.gz' || rc=$?; docker compose -p {{project}} unpause {{stateful}} || { echo "error: unpause failed; the stack is still paused" >&2; rc=1; }; exit $rc @ls -lh {{dir}}/ | tail -1 # Restore a backup tarball into the volumes (stops the stack; wipes current state). @@ -205,8 +219,8 @@ _restore project file dir: docker run --rm --network none -v {{absolute_path(file)}}:/backup.tar.gz:ro {{alpine}} tar tzf /backup.tar.gz > /dev/null docker compose -p {{project}} down --remove-orphans mkdir -p {{dir}} - m=$(for s in {{stateful}}; do printf -- '-v {{project}}_%s_data:/data/%s ' $s $s; done); docker run --rm --network none $m -v {{absolute_path(dir)}}:/backups {{alpine}} sh -c 'umask 077 && tar czf /backups/pre-restore-$(date +%Y%m%d-%H%M%S).tar.gz -C /data .' - m=$(for s in {{stateful}}; do printf -- '-v {{project}}_%s_data:/data/%s ' $s $s; done); docker run --rm --network none $m -v {{absolute_path(file)}}:/backup.tar.gz:ro {{alpine}} sh -c 'for d in /data/*; do find "$d" -mindepth 1 -delete; done && tar xzf /backup.tar.gz -C /data' + m=$(just _mounts {{project}}); docker run --rm --network none $m -v {{absolute_path(dir)}}:/backups {{alpine}} sh -c 'set -o pipefail; umask 077 && tar cf - -C /data . | gzip -1 > /backups/pre-restore-$(date +%Y%m%d-%H%M%S).tar.gz' + m=$(just _mounts {{project}}); docker run --rm --network none $m -v {{absolute_path(file)}}:/backup.tar.gz:ro {{alpine}} sh -c 'for d in /data/*; do find "$d" -mindepth 1 -delete; done && tar xzf /backup.tar.gz -C /data' # Needs a booted smoke stack (`just smoke`). Run it after touching _backup/_restore. # COMPOSE_FILE is pinned so the host's overlay list stays out, as for `smoke`. diff --git a/scripts/smoke.sh b/scripts/smoke.sh index 24e89db..4e455db 100755 --- a/scripts/smoke.sh +++ b/scripts/smoke.sh @@ -6,14 +6,13 @@ set -euo pipefail url="${SMOKE_URL:?}" project="${SMOKE_PROJECT:?}" -dept="${DEPARTMENT:-cml}" command -v jq >/dev/null || { echo "error: jq is required" >&2; exit 1; } die() { echo "error: $*" >&2; exit 1; } # Admin basic auth through curl's stdin config: argv is world-readable in ps. gf() { printf 'user = "admin:%s"\n' "$GRAFANA_ADMIN_PASSWORD" | curl -sf -K - "$@"; } promq() { gf -G --data-urlencode "query=$1" "$url/api/datasources/proxy/uid/prometheus/api/v1/query"; } -poll() { local n=0; until "$@"; do n=$((n + 1)); [[ $n -lt ${POLL_MAX:-60} ]] || return 1; sleep 1; done; } +poll() { local n=0; until "$@"; do n=$((n + 1)); [[ $n -lt ${POLL_MAX:-60} ]] || return 1; sleep "${POLL_SLEEP:-1}"; done; } # ------------------------------------------------------------------ provisioning # Grafana skips a broken dashboard or a malformed alert group silently, so the @@ -79,9 +78,9 @@ otlp metrics '{"resourceMetrics":[{"resource":'"$res"',"scopeMetrics":[{"metrics || die "the collector refused an OTLP metric with the .env token" otlp logs '{"resourceLogs":[{"resource":'"$res"',"scopeLogs":[{"logRecords":[{"timeUnixNano":"'"$ts"'","body":{"stringValue":"smoke"}}]}]}]}' \ || die "the collector refused an OTLP log with the .env token" -smoke_metric() { promq "smoke_up{job=\"smoke\",project=\"smoke\",env=\"ci\",department=\"$dept\"}" | jq -e '.data.result | length > 0' >/dev/null; } +smoke_metric() { promq "smoke_up{job=\"smoke\",project=\"smoke\",env=\"ci\",department=\"cml\"}" | jq -e '.data.result | length > 0' >/dev/null; } smoke_log() { - gf -G --data-urlencode "query={project=\"smoke\",env=\"ci\",department=\"$dept\"}" \ + gf -G --data-urlencode "query={project=\"smoke\",env=\"ci\",department=\"cml\"}" \ "$url/api/datasources/proxy/uid/loki/loki/api/v1/query_range" \ | jq -e 'any(.data.result[].values[][1]; . == "smoke")' >/dev/null } @@ -99,7 +98,7 @@ if [[ "${SMOKE_ALERTS:-}" == 1 ]]; then gf "$url/api/prometheus/grafana/api/v1/rules" \ | jq -e '.data.groups[].rules[] | select(.name == "TargetDown") | .state == "firing"' >/dev/null } - POLL_MAX=300 poll target_down_firing || die "TargetDown did not fire within 5 minutes of stopping node-exporter" + POLL_MAX=60 POLL_SLEEP=5 poll target_down_firing || die "TargetDown did not fire within 5 minutes of stopping node-exporter" docker compose -p "$project" start node-exporter >/dev/null echo "TargetDown fired for the stopped node-exporter" fi diff --git a/templates/README.md b/templates/README.md index 406f135..cc56dcd 100644 --- a/templates/README.md +++ b/templates/README.md @@ -49,9 +49,8 @@ file. ## GPU hosts -Include `compose.telemetry.gpu.yml` as well and set `GPU_METRICS=1`. The -agent config discovers the exporter by its Compose service label, so nothing -else changes. +Include `compose.telemetry.gpu.yml` as well. The agent config discovers the +exporter by its Compose service label, so nothing else changes. The exporter is `nvidia_gpu_exporter`, not dcgm-exporter. DCGM's profiling fields are datacentre-only, so on a consumer card it offers nothing extra and From 48519d7807240b82c7a32dd55e9aadec019d9083 Mon Sep 17 00:00:00 2001 From: Simon van Lierde Date: Sun, 6 Sep 2026 23:01:49 +0000 Subject: [PATCH 71/85] perf(prometheus): drop unread series from the hub's own scrapes - Drop histogram buckets from the Grafana, Loki and Tempo self-scrapes. Of those three jobs' 8,045 series, one metric is read anywhere in this repo; the buckets were 5,264 of them. `_sum` and `_count` stay, so latency is still there to explore. - Run node-exporter with the same nine-collector allowlist the spokes' Alloy uses. The defaults added ~1,000 series of cpufreq, thermal and cooling data that no dashboard, alert or runbook reads. Measured on the hub with one spoke: 14.1k active series to 7.0k, 438 samples/s to 206. --- compose.yml | 16 +++++++++++++++- config/prometheus.yaml | 9 +++++++++ 2 files changed, 24 insertions(+), 1 deletion(-) diff --git a/compose.yml b/compose.yml index 9a8d6b9..3a1e316 100644 --- a/compose.yml +++ b/compose.yml @@ -111,7 +111,21 @@ services: node-exporter: image: prom/node-exporter:v1.12.1@sha256:1b4e4438faca4dd7e001dd445d161a4a2091b0fededa84093b3a8dfeae1f1be0 restart: unless-stopped - command: ["--path.rootfs=/host"] + # Same collector allowlist as the spokes' Alloy (templates/alloy/config.alloy). + # The defaults add ~1000 series on this host (cpufreq, thermal, cooling, + # per-collector scrape stats) that no dashboard, alert or runbook reads. + command: + - --path.rootfs=/host + - --collector.disable-defaults + - --collector.cpu + - --collector.diskstats + - --collector.filesystem + - --collector.hwmon + - --collector.loadavg + - --collector.meminfo + - --collector.netdev + - --collector.stat + - --collector.uname pid: host mem_limit: 128m !!merge <<: *hardened diff --git a/config/prometheus.yaml b/config/prometheus.yaml index f82687b..92e2fdc 100644 --- a/config/prometheus.yaml +++ b/config/prometheus.yaml @@ -30,14 +30,23 @@ scrape_configs: static_configs: - targets: ["node-exporter:9100"] + # Only `up` and grafana_alerting_notifications_failed_total are read from these + # three; their histogram buckets were 5.2k of 14.1k head series. The _sum and + # _count series stay, so latency is still visible in Explore. - job_name: grafana static_configs: - targets: ["grafana:3000"] + metric_relabel_configs: &drop_buckets + - source_labels: [__name__] + regex: .*_bucket + action: drop - job_name: loki static_configs: - targets: ["loki:3100"] + metric_relabel_configs: *drop_buckets - job_name: tempo static_configs: - targets: ["tempo:3200"] + metric_relabel_configs: *drop_buckets From a87e4a0bc352fbb6498b86ffa9cf2f1ef59c0d2e Mon Sep 17 00:00:00 2001 From: Simon van Lierde Date: Sun, 6 Sep 2026 23:01:50 +0000 Subject: [PATCH 72/85] perf(alloy): ship only the container metrics the stack reads - Keep the four cAdvisor metrics two dashboard panels and the two container-lifecycle alerts use. cAdvisor emits ~59 metric families per container, and per-container-per-host is the fastest-growing block in the series budget. - Drop Alloy's own histogram buckets: 345 of that job's 760 series, none of them read centrally. - Poll Docker every 15s, so a short-lived one-shot is discovered without hitting the socket proxy at the old rate. --- templates/alloy/config.alloy | 29 ++++++++++++++++++++++++++--- 1 file changed, 26 insertions(+), 3 deletions(-) diff --git a/templates/alloy/config.alloy b/templates/alloy/config.alloy index f75fa62..199c347 100644 --- a/templates/alloy/config.alloy +++ b/templates/alloy/config.alloy @@ -19,8 +19,9 @@ discovery.docker "containers" { host = "tcp://docker-socket-proxy:2375" // The default 1m lets a short-lived one-shot (a backup) start and exit inside one - // poll window, never discovered. - refresh_interval = "5s" + // poll window, never discovered. 15s catches those without polling the socket + // proxy 12x more often than the default forever. + refresh_interval = "15s" } discovery.relabel "containers" { @@ -160,6 +161,16 @@ prometheus.relabel "cadvisor_scope" { regex = sys.env("COMPOSE_PROJECT_NAME") action = "keep" } + + // cAdvisor emits ~59 metric families per container (per-cpu, per-disk, per-NIC, + // TCP state); the stack reads four. The rest is the fastest-growing block in the + // series budget, since it scales per container per host. Add a name here when a + // dashboard or rule starts using one. + rule { + source_labels = ["__name__"] + regex = "container_(start_time_seconds|oom_events_total|cpu_usage_seconds_total|memory_working_set_bytes)" + action = "keep" + } } prometheus.scrape "host" { @@ -200,11 +211,23 @@ prometheus.exporter.self "alloy" { } prometheus.scrape "alloy" { targets = prometheus.exporter.self.alloy.targets - forward_to = [prometheus.relabel.host.receiver] + forward_to = [prometheus.relabel.alloy_self.receiver] scrape_interval = "30s" job_name = "alloy" } +// Alloy is the only exporter in this file with histograms, and nothing central +// reads them: 345 of this job's 760 series. The _sum and _count series survive. +prometheus.relabel "alloy_self" { + forward_to = [prometheus.relabel.host.receiver] + + rule { + source_labels = ["__name__"] + regex = ".*_bucket" + action = "drop" + } +} + // --------------------------------------------------------------------------- // GPU metrics, present only on hosts that run compose.telemetry.gpu.yml. Without the // exporter container this finds no targets and costs nothing. From fa3dd2f1a9676e7fa31f0c0c5ea8736987b05f42 Mon Sep 17 00:00:00 2001 From: Simon van Lierde Date: Sun, 6 Sep 2026 23:01:50 +0000 Subject: [PATCH 73/85] feat(alerting): count ingest per project at the gateway The keystone alert and the coverage backstop asked "is this project sending anything?" by selecting every project-labelled series in the TSDB: a scan that grows with the fleet, and one that a logs-only or traces-only project is invisible to, because it reaches no Prometheus series at all. - Count what arrives at the collector, keyed on the project and env resource attributes, and emit telemetry_{datapoints,logs,spans,metrics}_total. - Point ProjectTelemetrySilent and ProjectsUncovered at those counters. - Convert the connector's delta sums to cumulative: the Prometheus OTLP receiver refuses delta ("invalid temporality and type combination"), which would have failed silently at exactly the layer meant to catch silence. - Count a sender that set no project as "unknown", so an unattributed sender raises ProjectsUncovered instead of vanishing. - Assert the counters in `just smoke`, since two alerts now key on the name. - Show ingest per project on Stack Health. --- bootstrap.sh | 5 +- config/grafana/alerting/coverage.yaml | 6 ++- .../alerting/project-relab-staging.yaml | 5 +- config/otel-collector.yaml | 47 ++++++++++++++++++- dashboards/stack-health.json | 45 +++++++++++++++++- docs/adr/0002-hub-and-spoke-observability.md | 6 ++- scripts/smoke.sh | 5 ++ templates/alerting/coverage.yaml.tmpl | 8 +++- templates/alerting/project.yaml.tmpl | 7 ++- 9 files changed, 121 insertions(+), 13 deletions(-) diff --git a/bootstrap.sh b/bootstrap.sh index 2fadf73..956737c 100755 --- a/bootstrap.sh +++ b/bootstrap.sh @@ -68,7 +68,6 @@ while read -r p e; do rendered="${out_dir}/project-${p}-${e}.yaml" sed -e "s/__PROJECT__/${p}/g" \ -e "s/__ENV__/${e}/g" \ - -e "s/__UID__/proj-silent-${p}-${e}/g" \ templates/alerting/project.yaml.tmpl > "$rendered" echo "rendered $rendered" done <<<"$pairs" @@ -76,8 +75,8 @@ done <<<"$pairs" # ------------------------------------------------------------ 2. the coverage backstop covered="$(echo "$pairs" | sed 's| |/|' | paste -sd',' - | sed 's/,/, /g')" covered_expr="$(echo "$pairs" \ - | sed 's|^\([^ ]*\) \([^ ]*\)$|{project="\1",env="\2"}|' \ - | paste -sd'|' - | sed 's/|/ or /g')" + | sed 's@^\([^ ]*\) \([^ ]*\)$@{__name__=~"telemetry_(datapoints|logs|spans)_total", project="\1",env="\2"}@' \ + | paste -sd'@' - | sed 's/@/ or /g')" sed -e "s@__COVERED__@${covered}@" \ -e "s@__COVERED_EXPR__@${covered_expr}@" \ templates/alerting/coverage.yaml.tmpl > "${out_dir}/coverage.yaml" diff --git a/config/grafana/alerting/coverage.yaml b/config/grafana/alerting/coverage.yaml index 865dbfb..0b972c9 100644 --- a/config/grafana/alerting/coverage.yaml +++ b/config/grafana/alerting/coverage.yaml @@ -31,7 +31,11 @@ groups: refId: QUERY instant: true editorMode: code - expr: count by (project, env) ({project!="", env!=""} unless on (project, env) ({project="relab",env="staging"})) + # The collector's count connector (config/otel-collector.yaml) mints these + # counters for every sender, whatever signal it sends. A bare + # {project!=""} would scan every project-labelled series in the TSDB on + # each evaluation, and would still miss a logs-only or traces-only project. + expr: count by (project, env) ({__name__=~"telemetry_(datapoints|logs|spans)_total", project!="", env!=""} unless on (project, env) ({__name__=~"telemetry_(datapoints|logs|spans)_total", project="relab",env="staging"})) - refId: FIRING datasourceUid: __expr__ model: diff --git a/config/grafana/alerting/project-relab-staging.yaml b/config/grafana/alerting/project-relab-staging.yaml index 0e5a4fe..fd27047 100644 --- a/config/grafana/alerting/project-relab-staging.yaml +++ b/config/grafana/alerting/project-relab-staging.yaml @@ -34,7 +34,10 @@ groups: refId: QUERY instant: true editorMode: code - expr: absent({project="relab",env="staging"}) + # The collector's ingest counters, as in coverage.yaml.tmpl. absent() loads + # every series its selector matches, once a minute per project, so it is + # bounded to a few per service rather than everything the project sends. + expr: absent({__name__=~"telemetry_(datapoints|logs|spans)_total", project="relab",env="staging"}) - refId: FIRING datasourceUid: __expr__ model: diff --git a/config/otel-collector.yaml b/config/otel-collector.yaml index 76bc028..74f3c1e 100644 --- a/config/otel-collector.yaml +++ b/config/otel-collector.yaml @@ -22,6 +22,38 @@ receivers: auth: authenticator: bearertokenauth +connectors: + # A per-(project, env) ingest counter for every signal. The coverage and silence + # rules ask "is this project sending anything?" against a handful of series per + # project instead of scanning every project-labelled series in the TSDB, and they + # see a logs-only or traces-only project, which reaches no other Prometheus series. + # Attributes are read from the resource too, which is where project and env live. + # + # Every data type is named, including the two that carry no project: a type left + # undefined emits the connector's own default metric instead. + count: + datapoints: + telemetry.datapoints: &by_project + description: Telemetry received, by project and environment. + # default_value, or a sender that never set project is counted nowhere and + # stays invisible: no series, so nothing for the coverage rule to fire on. + # Counted as "unknown" it shows up as an uncovered project instead. + attributes: + - key: project + default_value: unknown + - key: env + default_value: unknown + logs: + telemetry.logs: *by_project + spans: + telemetry.spans: *by_project + spanevents: + telemetry.spanevents: *by_project + metrics: + # A metric stream has no attributes of its own; its datapoints carry them. + telemetry.metrics: + description: Metric streams received. + processors: # Stamped here with upsert, so a sender cannot claim another department. # Literal rather than an env knob: one value has ever been correct, and as an @@ -33,6 +65,12 @@ processors: value: cml action: upsert + # The count connector emits delta sums and the Prometheus OTLP receiver refuses + # them ("invalid temporality and type combination"). Converting here keeps the + # conversion state to the handful of counter streams, rather than turning on + # Prometheus's experimental global delta handling. + deltatocumulative: {} + # Sized to the container's mem_limit (512m in compose.yml); keep the two in step. memory_limiter: check_interval: 2s @@ -72,14 +110,19 @@ service: logs: receivers: [otlp] processors: [memory_limiter, resource/department, batch] - exporters: [otlp_http/loki] + exporters: [otlp_http/loki, count] traces: receivers: [otlp] processors: [memory_limiter, resource/department, batch] - exporters: [otlp_grpc/tempo] + exporters: [otlp_grpc/tempo, count] metrics: receivers: [otlp] processors: [memory_limiter, resource/department, batch] + exporters: [otlp_http/prometheus, count] + # The counters themselves. Same limiter and batcher as every other pipeline. + metrics/count: + receivers: [count] + processors: [memory_limiter, deltatocumulative, batch] exporters: [otlp_http/prometheus] telemetry: metrics: diff --git a/dashboards/stack-health.json b/dashboards/stack-health.json index 91f3ed8..7700725 100644 --- a/dashboards/stack-health.json +++ b/dashboards/stack-health.json @@ -210,6 +210,49 @@ "overrides": [] } }, + { + "type": "timeseries", + "title": "Ingest by Project", + "description": "Telemetry accepted per project and environment, from the gateway's count connector. A project missing here is sending nothing, whichever signal it uses.", + "gridPos": { + "h": 8, + "w": 24, + "x": 0, + "y": 16 + }, + "id": 7, + "datasource": { + "type": "prometheus", + "uid": "prometheus" + }, + "targets": [ + { + "expr": "sum by (project, env) (rate(telemetry_datapoints_total[$__rate_interval]))", + "legendFormat": "{{project}}/{{env}} metric points/s", + "refId": "A" + }, + { + "expr": "sum by (project, env) (rate(telemetry_logs_total[$__rate_interval]))", + "legendFormat": "{{project}}/{{env}} log records/s", + "refId": "B" + }, + { + "expr": "sum by (project, env) (rate(telemetry_spans_total[$__rate_interval]))", + "legendFormat": "{{project}}/{{env}} spans/s", + "refId": "C" + } + ], + "fieldConfig": { + "defaults": { + "unit": "ops", + "custom": { + "lineWidth": 2, + "fillOpacity": 10 + } + }, + "overrides": [] + } + }, { "type": "alertlist", "title": "Active Alerts", @@ -219,7 +262,7 @@ "h": 6, "w": 24, "x": 0, - "y": 16 + "y": 24 }, "options": { "viewMode": "list", diff --git a/docs/adr/0002-hub-and-spoke-observability.md b/docs/adr/0002-hub-and-spoke-observability.md index 7b0d247..df9e716 100644 --- a/docs/adr/0002-hub-and-spoke-observability.md +++ b/docs/adr/0002-hub-and-spoke-observability.md @@ -50,7 +50,11 @@ Contracts that make it scale: log exporters stay off); native app metrics own RED; cAdvisor owns container lifecycle; healthchecks.io owns "did the job run"; a host-local drift script owns "is the deployed code the code we think"; nothing derives metrics from - logs or from traces. + logs or from traces. The one exception is accounting, not signal: the + gateway's count connector emits `telemetry_{datapoints,logs,spans}_total` per + project and environment, so the keystone alert below can see a project that + sends only logs or only traces, and can ask whether telemetry is arriving + without reading every series a project has ever produced. - **`ProjectTelemetrySilent` per project/env is the keystone alert**: nothing on a spoke can detect its own absence. Templated and provisioned by `bootstrap.sh`, which is also what creates a project's healthchecks and prints diff --git a/scripts/smoke.sh b/scripts/smoke.sh index 4e455db..04461aa 100755 --- a/scripts/smoke.sh +++ b/scripts/smoke.sh @@ -86,6 +86,11 @@ smoke_log() { } poll smoke_metric || die "the smoke metric never reached Prometheus with its project/env/department labels; see just smoke-logs" poll smoke_log || die "the smoke log never reached Loki with its department label; see just smoke-logs" +# The keystone alert (ProjectTelemetrySilent) and the coverage backstop key on the +# gateway's ingest counters, so a renamed metric or a dropped attribute would +# silence both without any other assertion noticing. +counted() { promq "count by (project, env) ({__name__=~\"telemetry_(datapoints|logs|spans)_total\", project=\"smoke\", env=\"ci\"})" | jq -e '.data.result | length > 0' >/dev/null; } +poll counted || die "the count connector never produced telemetry_*_total{project=smoke,env=ci}; the alert rules that key on it would never fire" # ------------------------------------------------------------- alert pipeline # Opt-in (SMOKE_ALERTS=1): stop one scrape target and wait for TargetDown to diff --git a/templates/alerting/coverage.yaml.tmpl b/templates/alerting/coverage.yaml.tmpl index 27f987e..3e73a08 100644 --- a/templates/alerting/coverage.yaml.tmpl +++ b/templates/alerting/coverage.yaml.tmpl @@ -31,7 +31,11 @@ groups: refId: QUERY instant: true editorMode: code - expr: count by (project, env) ({project!="", env!=""} unless on (project, env) (__COVERED_EXPR__)) + # The collector's count connector (config/otel-collector.yaml) mints these + # counters for every sender, whatever signal it sends. A bare + # {project!=""} would scan every project-labelled series in the TSDB on + # each evaluation, and would still miss a logs-only or traces-only project. + expr: count by (project, env) ({__name__=~"telemetry_(datapoints|logs|spans)_total", project!="", env!=""} unless on (project, env) (__COVERED_EXPR__)) - refId: FIRING datasourceUid: __expr__ model: @@ -46,4 +50,4 @@ groups: severity: warning annotations: summary: "{{ $labels.project }}/{{ $labels.env }} is sending telemetry but has no alert rules" - description: "Telemetry is arriving for a project/environment bootstrap.sh was never run for, so nothing would notice if it went silent. Run ./bootstrap.sh {{ $labels.project }} {{ $labels.env }} on the monitoring host." + description: "Telemetry is arriving for a project/environment bootstrap.sh was never run for, so nothing would notice if it went silent. Run ./bootstrap.sh {{ $labels.project }} {{ $labels.env }} on the monitoring host. A literal `unknown` means the sender set no project or env resource attribute at all: fix the sender (docs/ONBOARDING.md), do not bootstrap a project by that name." diff --git a/templates/alerting/project.yaml.tmpl b/templates/alerting/project.yaml.tmpl index 84e51c2..5884d24 100644 --- a/templates/alerting/project.yaml.tmpl +++ b/templates/alerting/project.yaml.tmpl @@ -17,7 +17,7 @@ groups: rules: # absent() yields nothing while telemetry flows, so NoData is the HEALTHY state # and must map to OK. - - uid: __UID__ + - uid: proj-silent-__PROJECT__-__ENV__ title: ProjectTelemetrySilent condition: FIRING for: 15m @@ -34,7 +34,10 @@ groups: refId: QUERY instant: true editorMode: code - expr: absent({project="__PROJECT__",env="__ENV__"}) + # The collector's ingest counters, as in coverage.yaml.tmpl. absent() loads + # every series its selector matches, once a minute per project, so it is + # bounded to a few per service rather than everything the project sends. + expr: absent({__name__=~"telemetry_(datapoints|logs|spans)_total", project="__PROJECT__",env="__ENV__"}) - refId: FIRING datasourceUid: __expr__ model: From 7618ec12910295761e5d257d0bebc4274c8badfb Mon Sep 17 00:00:00 2001 From: Simon van Lierde Date: Sun, 6 Sep 2026 23:01:51 +0000 Subject: [PATCH 74/85] fix(docs): require project in every sender's resource attributes Both onboarding templates and the demo set env and never project, so an app onboarded by following them shipped telemetry no alert or dashboard could attribute. With the gateway's ingest counters such a sender is now counted as "unknown" and raises ProjectsUncovered, but the docs are where it should not happen in the first place. --- compose.demo.yml | 2 +- docs/ONBOARDING.md | 10 +++++++--- 2 files changed, 8 insertions(+), 4 deletions(-) diff --git a/compose.demo.yml b/compose.demo.yml index e3761ed..2ce0fd9 100644 --- a/compose.demo.yml +++ b/compose.demo.yml @@ -18,7 +18,7 @@ services: depends_on: [otel-collector] environment: OTEL_SERVICE_NAME: demo-api - OTEL_RESOURCE_ATTRIBUTES: env=demo + OTEL_RESOURCE_ATTRIBUTES: project=demo,env=demo OTEL_EXPORTER_OTLP_ENDPOINT: http://otel-collector:4318 OTEL_EXPORTER_OTLP_PROTOCOL: http/protobuf OTEL_EXPORTER_OTLP_HEADERS: Authorization=Bearer ${OTLP_AUTH_TOKEN:?set OTLP_AUTH_TOKEN in .env} diff --git a/docs/ONBOARDING.md b/docs/ONBOARDING.md index f4fb320..eebdc7d 100644 --- a/docs/ONBOARDING.md +++ b/docs/ONBOARDING.md @@ -21,7 +21,11 @@ Never expose 4317/4318 directly. - **`service.name`** is required: one stable name per deployable unit (`relab-api`, not `relab-api-prod-2`). Dashboards key on it. -- **`env`** is `prod`, `staging`, or `dev`, set as a resource attribute. +- **`project`** and **`env`** are both required resource attributes: `project` + is the name `bootstrap.sh` was run with, `env` is `prod`, `staging`, or + `dev`. Together they are what every alert and dashboard filters on, and a + sender that omits them is counted as `unknown` and alerts as an uncovered + project. - **Keep labels low-cardinality.** Prometheus turns every distinct label value into a series. User IDs, request IDs, and timestamps belong in the log line or in span attributes, not in resource attributes or metric labels. @@ -42,7 +46,7 @@ pip install opentelemetry-distro opentelemetry-exporter-otlp opentelemetry-instr ```sh export OTEL_SERVICE_NAME=my-service -export OTEL_RESOURCE_ATTRIBUTES=env=prod +export OTEL_RESOURCE_ATTRIBUTES=project=my-project,env=prod export OTEL_EXPORTER_OTLP_ENDPOINT=https://otel.example.org export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer " @@ -64,7 +68,7 @@ Every OpenTelemetry SDK reads the same four environment variables: OTEL_SERVICE_NAME=my-service OTEL_EXPORTER_OTLP_ENDPOINT=https://otel.example.org OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer " -OTEL_RESOURCE_ATTRIBUTES=env=prod +OTEL_RESOURCE_ATTRIBUTES=project=my-project,env=prod ``` ## Container logs and host metrics From d3ee7bdf0185d54bd3ded2bf59c8dfad7f009121 Mon Sep 17 00:00:00 2001 From: Simon van Lierde Date: Sun, 6 Sep 2026 23:01:52 +0000 Subject: [PATCH 75/85] docs: record the series cut and the per-project ingest counters - Changelog entries for the counters, the halved series count and the onboarding fix, and drop the DEPARTMENT upgrade step it no longer has. - Re-base the cardinality prose on the measured ~7k baseline: the 100k ceiling is ~14x it now, not ~7x. - Verify a spoke with the ingest counter, which sees every signal. --- CHANGELOG.md | 29 +++++++++++++++++++++++++---- docs/RUNBOOK.md | 4 ++-- templates/README.md | 10 +++++----- 3 files changed, 32 insertions(+), 11 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 94741bb..94cff44 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -15,17 +15,20 @@ the config syntax. 1.0 waits for a second consumer to confirm the contract. - Point every spoke's `OTEL_EXPORTER_OTLP_ENDPOINT` at `otel.`. The old `otlp.` name is gone. -- Set `DEPARTMENT` in the hub's `.env`. The collector stamps it on every - signal. - Set `COMPOSE_FILE` in `.env` to name the overlays this host runs (`compose.yml:compose.tunnel.yml` in production). `just up-tunnel` is gone; `just up` runs the exposure guards whenever the tunnel overlay is active. - Remove the orphaned `alertmanager_data` volume when convenient. Alerting is Grafana-managed (ADR 0002). - Re-vendor the templates on each spoke at `v0.3.0` (`bootstrap.sh` prints - the commands): the Alloy agent gains a memory limiter, and the overlay - declares the `egress` network it joins. + the commands): the Alloy agent gains a memory limiter, keeps only the four + cAdvisor metrics this stack reads, and drops its own histogram buckets; the + overlay declares the `egress` network it joins. - Loki streams keep their old label set until they age out (30 days). +- Set `project` as well as `env` in every sender's + `OTEL_RESOURCE_ATTRIBUTES`. A sender that omits either is now counted as + `unknown` and raises `ProjectsUncovered` instead of arriving unattributed + and unnoticed. ### Added @@ -39,6 +42,13 @@ the config syntax. 1.0 waits for a second consumer to confirm the contract. and Tempo too. `HostDiskFilling` (full within 3 days at the current rate) and `PrometheusCardinalityHigh` (over 100k active series) warn ahead of the 80% disk backstop. +- **Per-project ingest counters**: the gateway counts what arrives and emits + `telemetry_{datapoints,logs,spans,metrics}_total` by project and + environment. `ProjectTelemetrySilent` and `ProjectsUncovered` key on those + instead of scanning every project-labelled series, so their cost is flat in + fleet size, and a project that sends only logs or only traces is covered at + last: it reached no Prometheus series before. Stack Health gains an Ingest + by Project panel. - **`infra/generate-imports.sh`** emits OpenTofu `import` blocks for the tunnel, DNS records and Access app built by hand, so the first plan does not create duplicates. @@ -65,6 +75,14 @@ the config syntax. 1.0 waits for a second consumer to confirm the contract. - **Loki indexes only the identity labels** (`service.name`, `department`, `project`, `env`, `host.name`). Everything else is structured metadata, so a sender restart no longer mints a new stream. +- **Half the series are gone.** Prometheus drops the histogram buckets from + the Grafana, Loki and Tempo self-scrapes, the hub's node-exporter runs the + same nine-collector allowlist the spokes' Alloy does, and the agent keeps + only the four cAdvisor metrics this stack reads and drops its own buckets. + Measured on the hub with one spoke: 14.1k active series to 7.0k, 438 + samples/s to 206. Nothing that a dashboard, alert or runbook reads was + dropped; `_sum` and `_count` survive, so latency is still there to explore. + `PrometheusCardinalityHigh`'s 100k ceiling is ~14x the baseline now. - **Dashboards are provisioned, not editable**: `dashboards/*.json` is the source of truth; UI saves are off. - **Tempo stores traces and nothing else**: its metrics generator is gone. @@ -85,6 +103,9 @@ the config syntax. 1.0 waits for a second consumer to confirm the contract. so child spans diluted the ratio, and it aggregated by job alone, so a healthy prod service masked a broken staging one. It now reads the HTTP server metrics, keyed on job, project and env. +- Both onboarding templates and the demo set `env` but not `project` in + `OTEL_RESOURCE_ATTRIBUTES`, so an app that followed them shipped telemetry + no alert or dashboard could attribute to a project. - Trace links from the latency panel resolved to nothing: exemplars carry `traceID`, the datasource looked for `trace_id`. - The Infrastructure Logs dashboard queried labels this stack never set. The diff --git a/docs/RUNBOOK.md b/docs/RUNBOOK.md index 6129c5c..bc8af73 100644 --- a/docs/RUNBOOK.md +++ b/docs/RUNBOOK.md @@ -36,8 +36,8 @@ Retention is only partially size-bounded: Loki and Tempo cannot cap their total size, so the disk alert at 80% is the backstop. Two warnings fire earlier. `HostDiskFilling` means a 6-hour linear fit says a filesystem is full within 3 days. `PrometheusCardinalityHigh` -means active series passed 100k, about 7x the baseline. Series count, not -time, is what grows the TSDB. +means active series passed 100k, about 14x the baseline of ~7k with one +spoke. Series count, not time, is what grows the TSDB. When one fires: diff --git a/templates/README.md b/templates/README.md index cc56dcd..e0e8e64 100644 --- a/templates/README.md +++ b/templates/README.md @@ -32,8 +32,8 @@ stack's own alert rules. Skipping `bootstrap.sh` leaves a project unmonitored with no error anywhere. The rule that notices a host's *silence* lives on this stack, not on the host. The backstop is `ProjectsUncovered`, regenerated on every bootstrap -run: it fires on any series with a `project` label that has no rendered rule -file. +run: it fires on any project the gateway counts telemetry for that has no +rendered rule file, whichever signal that project sends. ## Two settings that silently produce nothing @@ -107,9 +107,9 @@ project/environment, so about six environments fit. ## Verifying, from the monitoring host ```promql -count({project="",env=""}) # non-zero within ~2 min -count(container_start_time_seconds{project="",name!=""}) # one per container -count({job=""}) # app's own SDK metrics +count(telemetry_datapoints_total{project="",env=""}) # non-zero within ~2 min +count(container_start_time_seconds{project="",name!=""}) # one per container +count({job=""}) # app's own SDK metrics ``` If the first is still zero after five minutes, `ProjectTelemetrySilent` From df888d2d6367597bcc201f8744d24a1124b6e239 Mon Sep 17 00:00:00 2001 From: Simon van Lierde Date: Sun, 6 Sep 2026 23:02:28 +0000 Subject: [PATCH 76/85] fix(bootstrap): verify a new spoke with the gateway's ingest counter The printed check still selected every series a project sends, which is both the scan the alert rules moved off and blind to a project that ships only logs or traces. templates/README.md already says the counter. --- bootstrap.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/bootstrap.sh b/bootstrap.sh index 956737c..f8b9568 100755 --- a/bootstrap.sh +++ b/bootstrap.sh @@ -179,6 +179,6 @@ Then include the overlay and bring it up: docker compose -f compose.yml -f compose.telemetry.yml up -d Verify from the monitoring host, within ~2 minutes: - count({project="${project}",env="${env_name}"}) -> non-zero + count(telemetry_datapoints_total{project="${project}",env="${env_name}"}) -> non-zero count(container_start_time_seconds{project="${project}",name!=""}) -> one per container SUMMARY From 02b6b26e53ecd0cce21347abafc7295183b61a41 Mon Sep 17 00:00:00 2001 From: Simon van Lierde Date: Sun, 6 Sep 2026 23:32:14 +0000 Subject: [PATCH 77/85] feat(infra): output the Zero Trust team domain CF_ACCESS_TEAM_DOMAIN was the one Access value copied by hand from the dashboard, and Grafana builds its JWK set URL from it: unset, it fetches signing keys from a subdomain anyone could claim, which is why the exposure guards refuse the pair. The team name is account-wide and predates this config, so read it through a data source and trim the suffix Grafana appends. --- .env.example | 3 ++- infra/main.tf | 13 +++++++++++++ justfile | 2 +- 3 files changed, 16 insertions(+), 2 deletions(-) diff --git a/.env.example b/.env.example index 078b0d6..35e7199 100644 --- a/.env.example +++ b/.env.example @@ -42,7 +42,8 @@ CLOUDFLARE_TUNNEL_TOKEN= # Set to true to let Cloudflare Access sign users into Grafana individually. # Needs both values below; `just up` with the tunnel overlay enforces that. GRAFANA_JWT_AUTH=false -# Your Zero Trust team name: the in https://.cloudflareaccess.com. +# Your Zero Trust team name, the in https://.cloudflareaccess.com: +# cd infra && tofu output -raw grafana_access_team_domain CF_ACCESS_TEAM_DOMAIN= # The Grafana Access application's aud tag: cd infra && tofu output -raw grafana_access_aud CF_ACCESS_AUD= diff --git a/infra/main.tf b/infra/main.tf index b083128..1d2ae19 100644 --- a/infra/main.tf +++ b/infra/main.tf @@ -9,6 +9,7 @@ # tofu plan # expect "0 to add"; see the script's header # tofu apply && rm imports.tf # tofu output -raw tunnel_token # → CLOUDFLARE_TUNNEL_TOKEN in ../.env +# tofu output -raw grafana_access_aud grafana_access_team_domain # → the CF_ACCESS_* pair # # State is local (infra/terraform.tfstate, gitignored) and holds the tunnel secret. @@ -130,6 +131,18 @@ data "cloudflare_zero_trust_tunnel_cloudflared_token" "monitoring" { tunnel_id = cloudflare_zero_trust_tunnel_cloudflared.monitoring.id } +# The Zero Trust team name is account-wide and predates this config, so it is read, +# not managed. Grafana builds its JWK set URL from it, and an unset name would fetch +# signing keys from a subdomain anyone could claim, hence the exposure guards. +data "cloudflare_zero_trust_organization" "team" { + account_id = var.account_id +} + +output "grafana_access_team_domain" { + description = "Set as CF_ACCESS_TEAM_DOMAIN in ../.env. Grafana appends .cloudflareaccess.com." + value = trimsuffix(data.cloudflare_zero_trust_organization.team.auth_domain, ".cloudflareaccess.com") +} + output "grafana_access_aud" { description = "Set as CF_ACCESS_AUD in ../.env so Grafana rejects tokens minted for other Access apps." value = cloudflare_zero_trust_access_application.grafana.aud diff --git a/justfile b/justfile index 2a90596..2af5b57 100644 --- a/justfile +++ b/justfile @@ -71,7 +71,7 @@ _expose-guards: @[ "${GRAFANA_ADMIN_PASSWORD:-}" != "change-me" ] || { echo "error: GRAFANA_ADMIN_PASSWORD is still the documented default; change it before exposing Grafana" >&2; exit 1; } @case "${GRAFANA_ROOT_URL:-}" in https://*) ;; *) echo "error: GRAFANA_ROOT_URL must be the https:// tunnel hostname (got '${GRAFANA_ROOT_URL:-}'); every absolute URL Grafana generates comes from it" >&2; exit 1;; esac @[ "${GRAFANA_COOKIE_SECURE:-false}" = "true" ] || { echo "error: GRAFANA_COOKIE_SECURE must be true when Grafana is served over HTTPS; set it in .env" >&2; exit 1; } - @[ "${GRAFANA_JWT_AUTH:-false}" != "true" ] || { [ -n "${CF_ACCESS_TEAM_DOMAIN:-}" ] && [ -n "${CF_ACCESS_AUD:-}" ]; } || { echo "error: GRAFANA_JWT_AUTH=true needs CF_ACCESS_TEAM_DOMAIN and CF_ACCESS_AUD in .env (cd infra && tofu output -raw grafana_access_aud)" >&2; exit 1; } + @[ "${GRAFANA_JWT_AUTH:-false}" != "true" ] || { [ -n "${CF_ACCESS_TEAM_DOMAIN:-}" ] && [ -n "${CF_ACCESS_AUD:-}" ]; } || { echo "error: GRAFANA_JWT_AUTH=true needs CF_ACCESS_TEAM_DOMAIN and CF_ACCESS_AUD in .env (cd infra && tofu output -raw grafana_access_team_domain grafana_access_aud)" >&2; exit 1; } @[ -n "${HEARTBEAT_URL:-}" ] || echo "WARNING: HEARTBEAT_URL is empty; the stack goes live without a dead-man's switch" >&2 @for f in .env infra/terraform.tfvars infra/terraform.tfstate infra/terraform.tfstate.backup; do [ ! -e "$f" ] || case "$(stat -c %a "$f")" in *00) ;; *) echo "error: $f is readable by other users (mode $(stat -c %a "$f")); it holds live secrets, run: chmod 600 $f" >&2; exit 1;; esac; done @[ -n "${ALERT_WEBHOOK_URL:-}" ] || { echo "error: ALERT_WEBHOOK_URL is empty; every alert would fire into an empty webhook URL and be dropped. The heartbeat keeps pinging either way, so this failure looks healthy from the outside. Set it, or comment out this guard" >&2; exit 1; } From 9d137b40a3e5e2b2df64a95390ca3e6d13e25166 Mon Sep 17 00:00:00 2001 From: Simon van Lierde Date: Sun, 6 Sep 2026 23:32:15 +0000 Subject: [PATCH 78/85] docs(collector): name the real trigger for a department variable The old comment argued a second department could not be a stream label, which is what the label already is. What the literal actually buys is that it cannot be misconfigured per deploy, and the trigger for a variable is a second hub, not federation: a second department federating into this hub means the upsert has to go, since it would stamp our department onto their data. --- config/otel-collector.yaml | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/config/otel-collector.yaml b/config/otel-collector.yaml index 74f3c1e..2be1c6d 100644 --- a/config/otel-collector.yaml +++ b/config/otel-collector.yaml @@ -56,9 +56,11 @@ connectors: processors: # Stamped here with upsert, so a sender cannot claim another department. - # Literal rather than an env knob: one value has ever been correct, and as an - # indexed Loki stream label a second one would be an index dimension, not a - # filter. Make it a variable again when a second department federates. + # Literal rather than an env knob: one value has ever been correct, and a knob here + # is a silent-failure surface. A hub brought up with the wrong value mislabels every + # signal and mints a parallel set of Loki streams, with nothing to catch it. Make it + # a variable when a second department runs its own hub; if one ever federates into + # this one, the upsert itself is what has to go. resource/department: attributes: - key: department From ce1563711c60c3328ddc77dd6f6f96394d94ff77 Mon Sep 17 00:00:00 2001 From: Simon van Lierde Date: Sun, 6 Sep 2026 23:32:15 +0000 Subject: [PATCH 79/85] docs(templates): keep GPU_METRICS as the documented spoke-side switch A host without the NVIDIA container runtime cannot include the GPU overlay at all, so tooling that assembles its `-f` list from the host's .env needs a switch, and every spoke inventing its own name makes one runbook per spoke. None of the vendored files read it; say so where the name is given. --- bootstrap.sh | 2 ++ templates/README.md | 6 ++++++ templates/compose.telemetry.gpu.yml | 6 ++++++ 3 files changed, 14 insertions(+) diff --git a/bootstrap.sh b/bootstrap.sh index f8b9568..768d088 100755 --- a/bootstrap.sh +++ b/bootstrap.sh @@ -162,6 +162,8 @@ ENVIRONMENT=${env_name} OTEL_EXPORTER_OTLP_ENDPOINT=https:// OTLP_AUTH_TOKEN= TELEMETRY_EDGE_KEY= # only if the project's egress crosses a WAF +GPU_METRICS= # 1 on a host with an NVIDIA card, read by your deploy + # tooling to include compose.telemetry.gpu.yml ──────────────────────────────────────────────────────────────────────────── Vendor the templates on the project host (pinned at ${tag}) diff --git a/templates/README.md b/templates/README.md index e0e8e64..12ff9c0 100644 --- a/templates/README.md +++ b/templates/README.md @@ -52,6 +52,12 @@ rendered rule file, whichever signal that project sends. Include `compose.telemetry.gpu.yml` as well. The agent config discovers the exporter by its Compose service label, so nothing else changes. +A host without the NVIDIA container runtime cannot include the overlay at all +(`up` aborts with "could not select device driver"), so deploy tooling that +assembles its `-f` list from the host's `.env` needs a switch. Use +`GPU_METRICS=1` for it. None of the vendored files read that name; it is a +convention, so that one runbook covers every spoke. + The exporter is `nvidia_gpu_exporter`, not dcgm-exporter. DCGM's profiling fields are datacentre-only, so on a consumer card it offers nothing extra and still requires `SYS_ADMIN`. diff --git a/templates/compose.telemetry.gpu.yml b/templates/compose.telemetry.gpu.yml index e5a3bb0..e3e8890 100644 --- a/templates/compose.telemetry.gpu.yml +++ b/templates/compose.telemetry.gpu.yml @@ -5,6 +5,12 @@ # # docker compose ... -f compose.telemetry.yml -f compose.telemetry.gpu.yml up -d # +# Deploy tooling that builds its `-f` list from the host's `.env` should key on +# `GPU_METRICS=1`. Nothing in these templates reads it, but one name across every spoke +# keeps the runbooks identical, and the switch has to exist: including this overlay on a +# host without the NVIDIA container runtime makes `up` abort with "could not select +# device driver". +# # nvidia_gpu_exporter, not dcgm-exporter: DCGM's profiling fields are datacentre-only, # and on a consumer card it keeps only its SYS_ADMIN requirement. From 2c5b8ba02d950e808a39f23c631af8fcbf4b8748 Mon Sep 17 00:00:00 2001 From: Simon van Lierde Date: Sun, 6 Sep 2026 23:32:34 +0000 Subject: [PATCH 80/85] docs: changelog the Zero Trust team-domain output --- CHANGELOG.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 94cff44..f9ef6fb 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -49,6 +49,10 @@ the config syntax. 1.0 waits for a second consumer to confirm the contract. fleet size, and a project that sends only logs or only traces is covered at last: it reached no Prometheus series before. Stack Health gains an Ingest by Project panel. +- **`grafana_access_team_domain`** joins the OpenTofu outputs, so neither + Access value is copied out of the dashboard by hand. Grafana builds its JWK + set URL from the team name, and an unset one fetches signing keys from a + claimable subdomain. - **`infra/generate-imports.sh`** emits OpenTofu `import` blocks for the tunnel, DNS records and Access app built by hand, so the first plan does not create duplicates. From 25daa4d637e183b7464b1c0567f18a8f043232de Mon Sep 17 00:00:00 2001 From: Simon van Lierde Date: Sun, 6 Sep 2026 23:46:56 +0000 Subject: [PATCH 81/85] refactor: drop duplicate image lookup and widen the ingest-counter selector - validate reuses _image instead of an inline copy of the jq lookup - telemetry.spanevents counts one series, nothing reads it per project - rules, bootstrap and smoke match telemetry_.+_total in one form --- bootstrap.sh | 2 +- config/grafana/alerting/coverage.yaml | 2 +- config/grafana/alerting/project-relab-staging.yaml | 2 +- config/otel-collector.yaml | 11 ++++++++--- justfile | 13 +++++-------- scripts/smoke.sh | 2 +- templates/alerting/coverage.yaml.tmpl | 2 +- templates/alerting/project.yaml.tmpl | 2 +- 8 files changed, 19 insertions(+), 17 deletions(-) diff --git a/bootstrap.sh b/bootstrap.sh index 768d088..8a49aa7 100755 --- a/bootstrap.sh +++ b/bootstrap.sh @@ -75,7 +75,7 @@ done <<<"$pairs" # ------------------------------------------------------------ 2. the coverage backstop covered="$(echo "$pairs" | sed 's| |/|' | paste -sd',' - | sed 's/,/, /g')" covered_expr="$(echo "$pairs" \ - | sed 's@^\([^ ]*\) \([^ ]*\)$@{__name__=~"telemetry_(datapoints|logs|spans)_total", project="\1",env="\2"}@' \ + | sed 's@^\([^ ]*\) \([^ ]*\)$@{__name__=~"telemetry_.+_total", project="\1",env="\2"}@' \ | paste -sd'@' - | sed 's/@/ or /g')" sed -e "s@__COVERED__@${covered}@" \ -e "s@__COVERED_EXPR__@${covered_expr}@" \ diff --git a/config/grafana/alerting/coverage.yaml b/config/grafana/alerting/coverage.yaml index 0b972c9..8c62d06 100644 --- a/config/grafana/alerting/coverage.yaml +++ b/config/grafana/alerting/coverage.yaml @@ -35,7 +35,7 @@ groups: # counters for every sender, whatever signal it sends. A bare # {project!=""} would scan every project-labelled series in the TSDB on # each evaluation, and would still miss a logs-only or traces-only project. - expr: count by (project, env) ({__name__=~"telemetry_(datapoints|logs|spans)_total", project!="", env!=""} unless on (project, env) ({__name__=~"telemetry_(datapoints|logs|spans)_total", project="relab",env="staging"})) + expr: count by (project, env) ({__name__=~"telemetry_.+_total", project!="", env!=""} unless on (project, env) ({__name__=~"telemetry_.+_total", project="relab",env="staging"})) - refId: FIRING datasourceUid: __expr__ model: diff --git a/config/grafana/alerting/project-relab-staging.yaml b/config/grafana/alerting/project-relab-staging.yaml index fd27047..fda2de9 100644 --- a/config/grafana/alerting/project-relab-staging.yaml +++ b/config/grafana/alerting/project-relab-staging.yaml @@ -37,7 +37,7 @@ groups: # The collector's ingest counters, as in coverage.yaml.tmpl. absent() loads # every series its selector matches, once a minute per project, so it is # bounded to a few per service rather than everything the project sends. - expr: absent({__name__=~"telemetry_(datapoints|logs|spans)_total", project="relab",env="staging"}) + expr: absent({__name__=~"telemetry_.+_total", project="relab",env="staging"}) - refId: FIRING datasourceUid: __expr__ model: diff --git a/config/otel-collector.yaml b/config/otel-collector.yaml index 2be1c6d..523aceb 100644 --- a/config/otel-collector.yaml +++ b/config/otel-collector.yaml @@ -30,7 +30,9 @@ connectors: # Attributes are read from the resource too, which is where project and env live. # # Every data type is named, including the two that carry no project: a type left - # undefined emits the connector's own default metric instead. + # undefined emits the connector's own default metric instead. The rules match + # telemetry_.+_total and key on the project label, so the two without it never + # reach them. count: datapoints: telemetry.datapoints: &by_project @@ -47,10 +49,13 @@ connectors: telemetry.logs: *by_project spans: telemetry.spans: *by_project + # Neither of these is read by a rule. A metric stream has no attributes of its + # own (its datapoints carry them), and span events are only ever counted + # alongside their spans, so one series each is enough. spanevents: - telemetry.spanevents: *by_project + telemetry.spanevents: + description: Span events received. metrics: - # A metric stream has no attributes of its own; its datapoints carry them. telemetry.metrics: description: Metric streams received. diff --git a/justfile b/justfile index 2af5b57..9aa490b 100644 --- a/justfile +++ b/justfile @@ -165,15 +165,12 @@ lint: _image file service: @docker compose -f {{file}} config --format json | docker run --rm -i {{jq}} -er '.services["{{service}}"].image // error("no service {{service}} in {{file}}")' -# One compose render feeds all four; each config goes through the binary that -# will load it. +# Each config goes through the binary that will load it. validate: - @set -e; cfg=$(docker compose -f compose.yml config --format json); \ - img() { printf '%s' "$cfg" | docker run --rm -i {{jq}} -er ".services[\"$1\"].image"; }; \ - docker run --rm --network none -v ./config/prometheus.yaml:/etc/prometheus/prometheus.yaml:ro --entrypoint promtool $(img prometheus) check config /etc/prometheus/prometheus.yaml; \ - docker run --rm --network none -e OTLP_AUTH_TOKEN=dummy -v ./config/otel-collector.yaml:/etc/otelcol/config.yaml:ro $(img otel-collector) validate --config=/etc/otelcol/config.yaml; \ - docker run --rm --network none -v ./config/loki.yaml:/etc/loki/loki.yaml:ro $(img loki) -config.file=/etc/loki/loki.yaml -verify-config; \ - docker run --rm --network none -v ./config/tempo.yaml:/etc/tempo/tempo.yaml:ro $(img tempo) -config.file=/etc/tempo/tempo.yaml -config.verify=true + docker run --rm --network none -v ./config/prometheus.yaml:/etc/prometheus/prometheus.yaml:ro --entrypoint promtool $(just _image compose.yml prometheus) check config /etc/prometheus/prometheus.yaml + docker run --rm --network none -e OTLP_AUTH_TOKEN=dummy -v ./config/otel-collector.yaml:/etc/otelcol/config.yaml:ro $(just _image compose.yml otel-collector) validate --config=/etc/otelcol/config.yaml + docker run --rm --network none -v ./config/loki.yaml:/etc/loki/loki.yaml:ro $(just _image compose.yml loki) -config.file=/etc/loki/loki.yaml -verify-config + docker run --rm --network none -v ./config/tempo.yaml:/etc/tempo/tempo.yaml:ro $(just _image compose.yml tempo) -config.file=/etc/tempo/tempo.yaml -config.verify=true # Runs against a copy of the sources: state and tfvars never enter the # container, which has network access to fetch the provider. diff --git a/scripts/smoke.sh b/scripts/smoke.sh index 04461aa..4b9d02e 100755 --- a/scripts/smoke.sh +++ b/scripts/smoke.sh @@ -89,7 +89,7 @@ poll smoke_log || die "the smoke log never reached Loki with its department labe # The keystone alert (ProjectTelemetrySilent) and the coverage backstop key on the # gateway's ingest counters, so a renamed metric or a dropped attribute would # silence both without any other assertion noticing. -counted() { promq "count by (project, env) ({__name__=~\"telemetry_(datapoints|logs|spans)_total\", project=\"smoke\", env=\"ci\"})" | jq -e '.data.result | length > 0' >/dev/null; } +counted() { promq "count by (project, env) ({__name__=~\"telemetry_.+_total\", project=\"smoke\", env=\"ci\"})" | jq -e '.data.result | length > 0' >/dev/null; } poll counted || die "the count connector never produced telemetry_*_total{project=smoke,env=ci}; the alert rules that key on it would never fire" # ------------------------------------------------------------- alert pipeline diff --git a/templates/alerting/coverage.yaml.tmpl b/templates/alerting/coverage.yaml.tmpl index 3e73a08..ae87b7f 100644 --- a/templates/alerting/coverage.yaml.tmpl +++ b/templates/alerting/coverage.yaml.tmpl @@ -35,7 +35,7 @@ groups: # counters for every sender, whatever signal it sends. A bare # {project!=""} would scan every project-labelled series in the TSDB on # each evaluation, and would still miss a logs-only or traces-only project. - expr: count by (project, env) ({__name__=~"telemetry_(datapoints|logs|spans)_total", project!="", env!=""} unless on (project, env) (__COVERED_EXPR__)) + expr: count by (project, env) ({__name__=~"telemetry_.+_total", project!="", env!=""} unless on (project, env) (__COVERED_EXPR__)) - refId: FIRING datasourceUid: __expr__ model: diff --git a/templates/alerting/project.yaml.tmpl b/templates/alerting/project.yaml.tmpl index 5884d24..3afcedc 100644 --- a/templates/alerting/project.yaml.tmpl +++ b/templates/alerting/project.yaml.tmpl @@ -37,7 +37,7 @@ groups: # The collector's ingest counters, as in coverage.yaml.tmpl. absent() loads # every series its selector matches, once a minute per project, so it is # bounded to a few per service rather than everything the project sends. - expr: absent({__name__=~"telemetry_(datapoints|logs|spans)_total", project="__PROJECT__",env="__ENV__"}) + expr: absent({__name__=~"telemetry_.+_total", project="__PROJECT__",env="__ENV__"}) - refId: FIRING datasourceUid: __expr__ model: From 37c20ce901ef3199fcb070dabf127d5c0daf18b7 Mon Sep 17 00:00:00 2001 From: Simon van Lierde Date: Sun, 6 Sep 2026 23:54:22 +0000 Subject: [PATCH 82/85] docs: changelog the ingest-counter selector --- CHANGELOG.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index f9ef6fb..e71d2e0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -43,9 +43,10 @@ the config syntax. 1.0 waits for a second consumer to confirm the contract. and `PrometheusCardinalityHigh` (over 100k active series) warn ahead of the 80% disk backstop. - **Per-project ingest counters**: the gateway counts what arrives and emits - `telemetry_{datapoints,logs,spans,metrics}_total` by project and - environment. `ProjectTelemetrySilent` and `ProjectsUncovered` key on those - instead of scanning every project-labelled series, so their cost is flat in + `telemetry_{datapoints,logs,spans}_total` by project and environment + (`metrics` and `spanevents` are counted once, without labels). The rules + match `telemetry_.+_total`; `ProjectTelemetrySilent` and `ProjectsUncovered` + key on those instead of scanning every project-labelled series, so their cost is flat in fleet size, and a project that sends only logs or only traces is covered at last: it reached no Prometheus series before. Stack Health gains an Ingest by Project panel. From c66b943897de6a25cab00ad4033d2e201da01c96 Mon Sep 17 00:00:00 2001 From: Simon van Lierde Date: Sun, 6 Sep 2026 23:58:11 +0000 Subject: [PATCH 83/85] docs(infra): name the token permission the team-domain data source needs --- infra/main.tf | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/infra/main.tf b/infra/main.tf index 1d2ae19..6bd548c 100644 --- a/infra/main.tf +++ b/infra/main.tf @@ -3,7 +3,9 @@ # # Bootstrap (owner-run, once): # cp terraform.tfvars.example terraform.tfvars # then fill it in -# export CLOUDFLARE_API_TOKEN=... # needs Tunnel:Edit, DNS:Edit, Access:Edit +# export CLOUDFLARE_API_TOKEN=... # needs Tunnel:Edit, DNS:Edit, Access:Edit, and +# # "Access: Organizations, Identity Providers, and +# # Groups: Read" for the team-domain data source # cd infra && tofu init # ./generate-imports.sh > imports.tf # the edge already exists: adopt it first # tofu plan # expect "0 to add"; see the script's header From 8cd81268edb430a4766b2b778f687e29d5dc7c9a Mon Sep 17 00:00:00 2001 From: Simon van Lierde Date: Mon, 7 Sep 2026 00:09:42 +0000 Subject: [PATCH 84/85] fix(volumes): keep the collector queue volume under compose management - move the queue chown into a one-shot otel-queue-init service, so a plain `docker compose up` works and the volume is created labelled. Pre-creating it with `docker volume create` left it unlabelled, and `down --volumes` then never reclaimed it - refuse to restore when a stateful volume is missing: `docker run -v` would create it empty, making the pre-restore snapshot a tarball of nothing - drop the dead Alertmanager cleanup step and the stale "use just up" warning from the runbook, and note that the alpine pin now lives in two places --- compose.yml | 26 +++++++++++++++++++++++++- docs/RUNBOOK.md | 15 +++++++-------- justfile | 13 ++++--------- 3 files changed, 36 insertions(+), 18 deletions(-) diff --git a/compose.yml b/compose.yml index 3a1e316..f9a1cff 100644 --- a/compose.yml +++ b/compose.yml @@ -36,10 +36,33 @@ x-healthcheck: &healthcheck start_interval: 2s services: + # A fresh named volume is root-owned and the collector image is distroless, so + # it cannot chown its own queue directory at startup. This one-shot service + # does it first. It lives here rather than in the justfile so that a plain + # `docker compose up` works, and so the volume stays Compose-managed: a + # `docker volume create` from outside leaves it unlabelled, and then + # `down --volumes` never removes it. + otel-queue-init: + image: alpine:3.24@sha256:28bd5fe8b56d1bd048e5babf5b10710ebe0bae67db86916198a6eec434943f8b + command: ["chown", "10001:10001", "/q"] + restart: "no" + network_mode: none + volumes: + - otel_queue:/q + cap_drop: [ALL] + cap_add: [CHOWN] + security_opt: + - no-new-privileges:true + pids_limit: 16 + mem_limit: 32m + logging: *default-logging + otel-collector: image: otel/opentelemetry-collector-contrib:0.156.0@sha256:125bdbeb7590cc1952c5b3430ecf14063568980c2c93d5b38676cc0446ed8108 restart: unless-stopped depends_on: + otel-queue-init: + condition: service_completed_successfully loki: condition: service_started tempo: @@ -51,7 +74,8 @@ services: - "127.0.0.1:4318:4318" # OTLP HTTP volumes: - ./config/otel-collector.yaml:/etc/otelcol/config.yaml:ro - # File-backed exporter queue. Must be owned by uid 10001; `just up` chowns it. + # File-backed exporter queue. Must be owned by uid 10001; otel-queue-init + # chowns it before this service starts. - otel_queue:/var/lib/otelcol/queue command: ["--config=/etc/otelcol/config.yaml"] environment: diff --git a/docs/RUNBOOK.md b/docs/RUNBOOK.md index bc8af73..4d4c146 100644 --- a/docs/RUNBOOK.md +++ b/docs/RUNBOOK.md @@ -66,6 +66,8 @@ loss, which every component handles through its write-ahead log. - The tarball is mode 0600 and contains secrets, including the Grafana database. Copy it off-host over a private channel. +- `just restore` refuses to run if the stack's volumes do not exist yet. On a + fresh host, run `just up` once to create them, then restore. - It covers the docker volumes only. The OpenTofu state for the Cloudflare edge is not in it. See "Where every secret lives" below. - During the pause the collector buffers incoming telemetry for five minutes @@ -205,7 +207,8 @@ resources. Nothing watches the tool images pinned in the `justfile` (yamllint, actionlint, shellcheck, ruff, gitleaks, OpenTofu, jq, Alloy, alpine). Bump -those by hand. +those by hand. The alpine pin appears twice: in the `justfile` and on +`otel-queue-init` in `compose.yml`. Bump both together. After merging, on the host: @@ -213,10 +216,6 @@ After merging, on the host: git pull && just pull && just up ``` -Use `just up`, not `docker compose up -d`. The recipe first chowns the -collector's queue volume to uid 10001. A raw compose up leaves a new volume -root-owned and the collector crash-looping. - -Coming from a release that still ran Alertmanager, its volume outlives the -service: run `docker volume rm monitoring_alertmanager_data` once the new -stack is up. +`just up` adds the exposure guards, but a plain `docker compose up -d` is now +safe too: the `otel-queue-init` service chowns the collector's queue volume to +uid 10001 before the collector starts. diff --git a/justfile b/justfile index 9aa490b..5eede5e 100644 --- a/justfile +++ b/justfile @@ -48,16 +48,10 @@ dash_paths := `ls dashboards/*.json 2>/dev/null | sed 's|^dashboards|/dashboards default: @just --list -# A fresh named volume is root-owned and the collector image is distroless, so -# chown its queue volume to uid 10001 here. Idempotent; every up-path runs it. -_queue-volume project: - @docker volume create {{project}}_otel_queue > /dev/null - @docker run --rm --network none -v {{project}}_otel_queue:/q {{alpine}} chown 10001:10001 /q - # COMPOSE_FILE in .env names the overlays. With the tunnel overlay active, this # refuses to start until the exposure guards pass. # Start the stack (Grafana at http://localhost:3000). -up: (_queue-volume core_project) _guard-if-exposed +up: _guard-if-exposed docker compose up -d # The exposure guards, only when the tunnel overlay is in play. @@ -81,7 +75,7 @@ down: docker compose down --remove-orphans # Core stack plus a demo telemetry source, isolated from any running stack (:3002). -demo: (_queue-volume demo_project) +demo: {{compose_demo}} up -d --build # Stop the demo telemetry source; the demo project's core stack keeps running. @@ -213,6 +207,7 @@ restore file: (_restore core_project file "backups") # extract dies halfway, that is the only way back. _restore project file dir: @[ -f "{{file}}" ] || { echo "error: {{file}} not found" >&2; exit 1; } + @for s in {{stateful}}; do docker volume inspect {{project}}_${s}_data > /dev/null 2>&1 || { echo "error: volume {{project}}_${s}_data does not exist. 'docker run -v' would create it empty, so the pre-restore snapshot below would be a tarball of nothing. On a fresh host run 'just up' once first; otherwise check COMPOSE_PROJECT_NAME." >&2; exit 1; }; done docker run --rm --network none -v {{absolute_path(file)}}:/backup.tar.gz:ro {{alpine}} tar tzf /backup.tar.gz > /dev/null docker compose -p {{project}} down --remove-orphans mkdir -p {{dir}} @@ -229,7 +224,7 @@ restore-check: # crash-looping service is caught (after the full timeout). scripts/smoke.sh # asserts what lands after that: provisioning, scrapes, the data paths. # Boot an isolated copy of the core stack and assert it works end to end. -smoke: (_queue-volume smoke_project) +smoke: {{compose_smoke}} up -d --wait --wait-timeout 120 {{smoke_env}} SMOKE_URL=http://localhost:{{smoke_port}} SMOKE_PROJECT={{smoke_project}} scripts/smoke.sh From d7444c3c1fa4493bac898dc9ca56d47448785445 Mon Sep 17 00:00:00 2001 From: Simon van Lierde Date: Mon, 7 Sep 2026 00:29:47 +0000 Subject: [PATCH 85/85] docs: changelog the queue init service --- CHANGELOG.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index e71d2e0..db1d9b3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -37,7 +37,9 @@ the config syntax. 1.0 waits for a second consumer to confirm the contract. enforce the pair, and the smoke test boots with it on. - **Durable export queue**: the collector's send queues live on a file-backed `otel_queue` volume, so telemetry buffered during an outage survives a - collector restart. + collector restart. A one-shot `otel-queue-init` service owns the volume for + the collector, so a plain `docker compose up` works and `down --volumes` + reclaims it. - **Self-monitoring covers every service**: Prometheus scrapes Grafana, Loki and Tempo too. `HostDiskFilling` (full within 3 days at the current rate) and `PrometheusCardinalityHigh` (over 100k active series) warn ahead of the