From 9b136f7776a6e9daec995d5e76450d17aa5da5e4 Mon Sep 17 00:00:00 2001 From: Simon van Lierde Date: Mon, 7 Sep 2026 05:12:25 +0000 Subject: [PATCH 1/2] docs: sync README and runbook with 0.3.0 and 0.3.1 - README alerting list: cardinality ceiling is 30k, add the spike and uncovered-project rules - README hooks row: infra-validate runs at push when infra/ changed - Runbook: document cutting a release, which lived only in a workflow comment --- README.md | 11 ++++++----- docs/RUNBOOK.md | 21 +++++++++++++++++++++ 2 files changed, 27 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index 53f3333..57774fd 100644 --- a/README.md +++ b/README.md @@ -108,7 +108,7 @@ top of [infra/main.tf](infra/main.tf). | `just check` | `lint` plus `validate` | | `just smoke` | Boots an isolated copy of the stack and asserts it works end to end | | `just restore-check` | Rehearses backup and restore on the smoke stack | -| `just hooks` | Installs the git hooks: gitleaks at commit, a Conventional Commits check on the message, `check` at push | +| `just hooks` | Installs the git hooks: gitleaks at commit, a Conventional Commits check on the message, `check` at push, `infra-validate` at push when `infra/` changed | Every check runs in a pinned container. Nothing is installed on the host. @@ -138,10 +138,11 @@ agent that ships container logs and host metrics. ## Alerting Grafana evaluates and delivers the rules in `config/grafana/alerting/`: -telemetry silent per project, container crash-looping or OOM-killed, scrape -target down, OTel export failures, alert delivery failing, error rate above -5%, disk above 80%, disk projected full within 3 days, and Prometheus head -series above 100k. There is no Alertmanager. Grafana rules can query Loki as +telemetry silent per project, a project sending telemetry with no rule file, +container crash-looping or OOM-killed, scrape target down, OTel export +failures, alert delivery failing, error rate above 5%, disk above 80%, disk +projected full within 3 days, Prometheus active series above 30k, and 5,000 +new series in 30 minutes. There is no Alertmanager. Grafana rules can query Loki as well as Prometheus, and one engine means one answer to "who gets told". Notifications go to the webhook in `ALERT_WEBHOOK_URL` (ntfy, Slack, and so diff --git a/docs/RUNBOOK.md b/docs/RUNBOOK.md index c717ea1..4cc5aed 100644 --- a/docs/RUNBOOK.md +++ b/docs/RUNBOOK.md @@ -196,6 +196,27 @@ cd infra && tofu apply `infra/generate-imports.sh > infra/imports.tf`, which reads the live objects back out of the Cloudflare API and adopts them into a fresh state. +## Cutting a release + +A tag is the deployable unit: `bootstrap.sh` pins every spoke's vendored +templates to the latest tag, and refuses to run without one. + +1. Add a `## [x.y.z] - date` section to `CHANGELOG.md` and merge it. Put + anything a spoke must do (re-vendor templates, change a variable) under + an `Upgrade` heading. +2. Tag and push: + + ```sh + git tag vx.y.z && git push origin vx.y.z + ``` + + The release workflow runs `just check` and publishes the GitHub release + with that changelog section as its body. No section, no release: the tag + stays, so fix the changelog on `main` and re-tag. +3. Deploy the hub (below), then run `./bootstrap.sh ` for each + spoke and follow what it prints. The checksums it emits are for the new + tag. + ## Upgrading images Dependabot opens PRs that bump the pinned versions, and CI runs `just validate` From 0ad433975b2c6647efa56f71dde589aac7c487d5 Mon Sep 17 00:00:00 2001 From: Simon van Lierde Date: Mon, 7 Sep 2026 05:17:36 +0000 Subject: [PATCH 2/2] docs(readme): rewrap the alerting paragraph --- README.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 57774fd..1a36466 100644 --- a/README.md +++ b/README.md @@ -142,8 +142,9 @@ telemetry silent per project, a project sending telemetry with no rule file, container crash-looping or OOM-killed, scrape target down, OTel export failures, alert delivery failing, error rate above 5%, disk above 80%, disk projected full within 3 days, Prometheus active series above 30k, and 5,000 -new series in 30 minutes. There is no Alertmanager. Grafana rules can query Loki as -well as Prometheus, and one engine means one answer to "who gets told". +new series in 30 minutes. There is no Alertmanager. Grafana rules can query +Loki as well as Prometheus, and one engine means one answer to "who gets +told". Notifications go to the webhook in `ALERT_WEBHOOK_URL` (ntfy, Slack, and so on). One rule, `Watchdog`, fires permanently and posts to `HEARTBEAT_URL`