Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 8 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -108,7 +108,7 @@ top of [infra/main.tf](infra/main.tf).
| `just check` | `lint` plus `validate` |
| `just smoke` | Boots an isolated copy of the stack and asserts it works end to end |
| `just restore-check` | Rehearses backup and restore on the smoke stack |
| `just hooks` | Installs the git hooks: gitleaks at commit, a Conventional Commits check on the message, `check` at push |
| `just hooks` | Installs the git hooks: gitleaks at commit, a Conventional Commits check on the message, `check` at push, `infra-validate` at push when `infra/` changed |

Every check runs in a pinned container. Nothing is installed on the host.

Expand Down Expand Up @@ -138,11 +138,13 @@ agent that ships container logs and host metrics.
## Alerting

Grafana evaluates and delivers the rules in `config/grafana/alerting/`:
telemetry silent per project, container crash-looping or OOM-killed, scrape
target down, OTel export failures, alert delivery failing, error rate above
5%, disk above 80%, disk projected full within 3 days, and Prometheus head
series above 100k. There is no Alertmanager. Grafana rules can query Loki as
well as Prometheus, and one engine means one answer to "who gets told".
telemetry silent per project, a project sending telemetry with no rule file,
container crash-looping or OOM-killed, scrape target down, OTel export
failures, alert delivery failing, error rate above 5%, disk above 80%, disk
projected full within 3 days, Prometheus active series above 30k, and 5,000
new series in 30 minutes. There is no Alertmanager. Grafana rules can query
Loki as well as Prometheus, and one engine means one answer to "who gets
told".

Notifications go to the webhook in `ALERT_WEBHOOK_URL` (ntfy, Slack, and so
on). One rule, `Watchdog`, fires permanently and posts to `HEARTBEAT_URL`
Expand Down
21 changes: 21 additions & 0 deletions docs/RUNBOOK.md
Original file line number Diff line number Diff line change
Expand Up @@ -196,6 +196,27 @@ cd infra && tofu apply
`infra/generate-imports.sh > infra/imports.tf`, which reads the live objects
back out of the Cloudflare API and adopts them into a fresh state.

## Cutting a release

A tag is the deployable unit: `bootstrap.sh` pins every spoke's vendored
templates to the latest tag, and refuses to run without one.

1. Add a `## [x.y.z] - date` section to `CHANGELOG.md` and merge it. Put
anything a spoke must do (re-vendor templates, change a variable) under
an `Upgrade` heading.
2. Tag and push:

```sh
git tag vx.y.z && git push origin vx.y.z
```

The release workflow runs `just check` and publishes the GitHub release
with that changelog section as its body. No section, no release: the tag
stays, so fix the changelog on `main` and re-tag.
3. Deploy the hub (below), then run `./bootstrap.sh <project> <env>` for each
spoke and follow what it prints. The checksums it emits are for the new
tag.

## Upgrading images

Dependabot opens PRs that bump the pinned versions, and CI runs `just validate`
Expand Down