From 56b73e0e0d85bb27be0bd229f58e400117a4044e Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 19 Jul 2026 16:11:55 +0000 Subject: [PATCH 01/14] docs(adr): propose edge security tier (rate limiting + CrowdSec) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add ADR-0043 proposing an edge security tier authored from one variables.yaml `security:` block: nginx-native rate limiting (named profiles + default, rendered into the edge nginx.conf) and CrowdSec (firewall bouncer at nftables plus the free community blocklist) installed per public-edge host. Status: proposed — design checkpoint for review before implementation. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_011RB2WPLAzoDT1bjXrMfoXi --- ...dge-security-rate-limiting-and-crowdsec.md | 237 ++++++++++++++++++ 1 file changed, 237 insertions(+) create mode 100644 docs/adr/0043-edge-security-rate-limiting-and-crowdsec.md diff --git a/docs/adr/0043-edge-security-rate-limiting-and-crowdsec.md b/docs/adr/0043-edge-security-rate-limiting-and-crowdsec.md new file mode 100644 index 0000000..9e17316 --- /dev/null +++ b/docs/adr/0043-edge-security-rate-limiting-and-crowdsec.md @@ -0,0 +1,237 @@ +--- +status: proposed +date: 2026-07-19 +issue: "TBD" +--- + +# Edge security tier: rate limiting and CrowdSec + +The public edge (ingress + gateway nginx) is the most-probed surface in the fleet: +scanning bots hammer `/.env`, `/wp-admin`, `/.git/config`, phpMyAdmin, and known-CVE +paths continuously. inforge terminates that traffic at a stock nginx.org build it +renders deterministically (`internal/nginx`), fronting custom Go services behind a +mesh. Today nothing throttles or bans abusive clients at that edge. + +This ADR adds an **edge security tier** with two complementary, independently-toggled +capabilities, authored from one `security:` block in `variables.yaml`: + +1. **Rate limiting** — nginx-native `limit_req`/`limit_conn`, rendered into the edge + nginx.conf. Real-time L7 throttling (returns `429`) of single/small-source request + floods. No host agent, no nginx module. +2. **CrowdSec** — a per-host agent (engine + local API) that parses the nginx access + logs, ban decisions enforced by the **firewall bouncer** at nftables, plus the free + crowd-sourced **community blocklist** for pre-emptive known-bad-IP bans. + +The two layers feed each other: nginx throttles a flood in real time and logs the +`429`s; CrowdSec reads those logs and bans the persistent offender at nftables so it +stops reaching nginx at all. + +## Scope boundary: what this does and does not stop + +Being explicit, because "DoS protection" over-promises if left unqualified: + +| Tier | Example | Handled by | +|---|---|---| +| L3/L4 volumetric | SYN/UDP flood, amplification | **Not the origin** — Hetzner's free network-edge DDoS protection (already in place). Saturates the uplink before nginx runs. | +| L7 single/small-source abuse | one IP at 500 req/s; brute-force; scraping | **This ADR** — nginx rate limiting + CrowdSec banning. The 95% case. | +| L7 distributed botnet flood | thousands of IPs, low rate each, real endpoint | **Partial** — the community blocklist catches known-bad IPs; the rest genuinely needs an upstream scrubber (Cloudflare). No self-hosted origin tool fully solves this, and we do not claim to. | + +Choosing CrowdSec over Cloudflare's WAF is a deliberate trade: one less external +lock-in, at the cost of not covering the distributed-botnet tier. That gap is +documented, not hidden. + +## Authoring + +One env-level block in `variables.yaml`, with a per-edge opt-out. Both capabilities are +off unless the block enables them. + +```yaml +security: + crowdsec: + enabled: true # opt-in per env; installs on every public edge host + # version: "1.6.4" # optional pin; else inforge's DefaultVersion + # console: true # optional dashboard enrollment (needs reserved secret) + rate_limit: + default_profile: standard # applied to EVERY public server unless overridden + profiles: + standard: + requests_per_second: 20 + burst: 40 # queued excess before a 429 + max_connections: 40 # concurrent conns per client IP + strict: + requests_per_second: 5 + burst: 10 + max_connections: 10 + api: + requests_per_second: 100 + burst: 200 + max_connections: 200 +``` + +**Profiles, not per-route detail** (mirrors the observability notification-profile +idiom — `Profiles map[string]Profile` + a `default_profile`, referenced by name). Limits +are named once; a route selects one with a single token. `default_profile` applies to +every public server automatically, so "all routes at 20 rps" needs zero per-route +authoring. + +**Per-resource override** references a profile by name (never re-specifies limits): + +```yaml +# service route (resources//regional/service//manifest.yaml) +routes: + - type: tls-termination + listen: 443 + target: 8080 + rate_limit_profile: strict # overrides the default for this route + +# app (app//manifest.yaml) and gateway (gateway//manifest.yaml) +rate_limit_profile: api # one profile per app / per gateway +``` + +`rate_limit_profile: none` is the reserved sentinel meaning **no limiting** on that +resource (parallels a `muted` route in notifications). `none` cannot be defined as a +real profile. + +**Per-edge opt-out** — an ingress/gateway leaves the whole security tier: + +```yaml +# ingress//manifest.yaml or gateway//manifest.yaml +security: false # this edge gets neither CrowdSec nor rate limiting +``` + +**Optional CrowdSec console** enrollment token is a reserved secret (never inlined), +mirroring `observability/otlp_auth`: +`inforge secret set security crowdsec_enroll --reserved`. The community blocklist +needs no secret. + +## Decisions + +### Rate limiting + +- **nginx-native `limit_req` + `limit_conn`, rendered — not a module or a sidecar.** It + is stock-nginx.org (no lua/OpenResty), flows through the existing `internal/nginx` + crossplane render, and preserves the deterministic-bytes property. `limit_req` is the + leaky-bucket request-rate limiter; `limit_conn` caps concurrent connections per IP. +- **Named profiles + a default profile**, keyed by client IP (`$binary_remote_addr`). + Modeled directly on `types.NotificationsSpec` profiles: define once, reference by + name, one env-level default. Each *referenced* profile renders one + `limit_req_zone`/`limit_conn_zone` in `http{}` (deduped by profile name); every server + using it emits `limit_req zone=

burst= nodelay;` + `limit_conn

_conn ;`. + `limit_req_status 429;` / `limit_conn_status 429;` so rejects are `429` (CrowdSec- + parseable), not the nginx default `503`. +- **Applies to the http edge servers** (tls-termination routes, apps, gateway). **Health + and ACME servers are exempt** — liveness probes and cert issuance must never be + throttled. **Forward (L4) routes** get only the profile's `max_connections` via a + `stream{}` `limit_conn` (a stream-context zone); `requests_per_second`/`burst` do not + apply at L4 and are ignored for a forward route (validation notes this rather than + erroring, so a profile can serve both). +- **The private mesh nginx (`internal/meshnginx`) is untouched.** Rate limiting is an + internet-edge concern; east-west peer traffic is already allowlist-gated. + +### CrowdSec + +- **The firewall bouncer (nftables), not the lua nginx bouncer.** The lua/AppSec bouncer + needs OpenResty and would fight the stock nginx build; the firewall bouncer enforces at + nftables from log analysis and touches no nginx config. inforge manages **no host packet + filter today** (the firewall is the Hetzner *cloud* firewall via the provider), so the + bouncer owns its own nft table with zero conflict — confirmed against the codebase. +- **Off-the-shelf `.deb`, checksum-verified download — mirrors ADR-0031 (otelcol).** New + pure package `internal/crowdsec` (stdlib-only, Pulumi-free, like `internal/otelcol` and + `internal/nginx`) renders the idempotent install shell (download the version-pinned + `crowdsec` + `crowdsec-firewall-bouncer-nftables` `.deb`s, verify sha256, `apt-get + install` the local files) and the on-host config. A `DefaultVersion` is pinned and + bumped deliberately. +- **Community blocklist on by default; console optional.** `cscli capi register` (no + secret) enrolls the machine to the Central API and pulls the crowd-sourced blocklist — + the pre-emptive known-bad-IP protection that is CrowdSec's edge over fail2ban. Console + (dashboard) enrollment is a separate opt-in gated on the reserved `crowdsec_enroll` + secret. +- **inforge renders acquisition + bouncer config; hub content is fetched imperatively.** + inforge writes `/etc/crowdsec/acquis.d/nginx.yaml` (pointing at `/var/log/nginx/*.log`) + and the firewall-bouncer YAML deterministically, and leaves the package `config.yaml` + default. Hub collections (`crowdsecurity/nginx`, `base-http-scenarios`, `http-cve`) are + installed via `cscli collections install`, which is a **network-dependent, imperative** + step — a deliberate deviation from the deterministic-render norm, accepted for the same + reason ADR-0031 accepted third-party apt: not re-implementing a solved packaging + problem. `cscli` install is idempotent (re-run is a no-op/upgrade). +- **Bouncer API key: minted like the Postgres monitor role.** The firewall bouncer + authenticates to the local API with a key inforge generates (`random.RandomPassword`, + the ADR-0037 monitor-password precedent), registered on-host and written into the + bouncer config in the same pass. `cscli bouncers add` is **not idempotent** (errors if + the bouncer exists), so the bootstrap does `cscli bouncers delete inforge-fw --force || + true; cscli bouncers add inforge-fw --key ` — safe because the key is inforge-owned + and rewritten atomically alongside. The key routes through `ApplyT` + `safeTrigger` so + it is encrypted in Pulumi state and never lands in the unencrypted `Triggers` array + (enforced by the `dbrole_test.go` pattern). +- **Edges only in v1, gated on env config.** Unlike otelcol (every VM), CrowdSec installs + only on ingress + gateway hosts — where the public HTTP traffic lands. The host set is + the same `ingressHostUnion(routes, apps, health, gateways)` that `realizeIngress` + computes, so it cannot drift from where nginx actually runs. Designed so a later "every + VM" expansion (sshd brute-force coverage) is a config flip, not a rewrite. +- **No new open ports.** The local API is loopback (`127.0.0.1:8080`); the community + blocklist pull is outbound. `types.FirewallPorts` is unchanged. + +## Wiring (implementation shape) + +Grounded in the existing otelcol path (`program.provisionObservability`, +`program.go:714`) and ingress realization (`program.go:1717`): + +- **`internal/crowdsec`** — `paths.go` (constants: package names, service names, + `ConfigPath`, `BouncerConfigPath`, `AcquisPath`, `DefaultVersion`, reserved-secret + namespace/key) + `install.go` (`InstallScript`, `AcquisScript`, `BouncerScript`, + `EnrollScript`, `WriteFileScript`). Pure, Pulumi-free. +- **`types`** — new `SecurityConfig` (with `Crowdsec` + `RateLimit` sub-blocks and + profiles) on `EnvironmentVariables`; a `Security *bool` opt-out on `IngressSpec` and + `GatewaySpec`; a `RateLimitProfile string` on `RouteSpec`, `AppSpec`, `GatewaySpec`. + Accessors defaulting like `DashboardsEnabled()`. +- **`schemas`** — extend `environment.json` (the `security` block + profiles), + `ingress.json`/`gateway.json` (`security`, `rate_limit_profile`), `service.json` (route + `rate_limit_profile`), `app.json` (`rate_limit_profile`). +- **`loader`** — normalize/trim; default `rate_limit.default_profile`; carry profiles. +- **`validate`** — new `checkSecurity`: `default_profile` and every `rate_limit_profile` + reference resolve to a defined profile (or `none`); profile bounds + (`requests_per_second > 0`, `burst >= 0`, `max_connections >= 0`); reserved name `none` + not defined; `crowdsec.enabled` with no edge host in the env is an error; console + enrollment configured ⇒ reserved secret present. +- **`internal/nginx/config.go`** — thread the resolved rate-limit plan into `Render(...)` + and `IngressProvider.Realize(...)`; emit the zones + per-server directives. The + `Realize` signature change ripples to the Hetzner provider and the `types` interface + (a mechanical, compile-checked change). +- **`program.go`** — new `provisionCrowdsec(...)` copied from `provisionObservability` but + iterating the ingress∪gateway host set (respecting per-edge opt-out), sharing the + memoized `gates` cloud-init map, called in the scope loop beside the observability pass. + Rate-limit config is resolved per host and passed into `realizeIngress`. +- **Docs** — public docs updated per rule `update-public-docs-with-resource-changes`; + AGENTS.md gains a "Security (edge)" section. + +## Consequences + +- A persisted bouncer-API-key file on the edge host, like the otelcol/OTLP credential — + `0600`, owned by the bouncer user. Same accepted divergence from the tmpfs-only service + secret posture (host infrastructure that must boot independently). +- Deploy now has a network dependency on the CrowdSec hub for collection install on a host + that has never fetched them; a hub outage fails the CrowdSec pass on a fresh edge (an + already-provisioned host re-runs as a no-op). +- The edge nginx.conf grows `limit_req_zone`/`limit_conn_zone` blocks; render stays + deterministic (zones deduped and sorted by profile name). +- CrowdSec keeps local state (SQLite decisions) on the edge host — surviving reloads, + rebuilt on host replacement (bans re-learned; the community blocklist re-pulls). +- New third-party apt source on edge hosts (CrowdSec), same trade-off class as ADR-0031. + +## Considered and rejected + +- **Cloudflare WAF at the edge** — strongest and lowest-ops, but adds external lock-in and + conflicts with the grey-cloud/origin-ACME design (orange-clouding breaks origin HTTP-01 + and needs origin-IP firewalling). Explicitly declined in favor of self-hosting; may be + revisited for the distributed-botnet tier this ADR does not cover. +- **ModSecurity + OWASP CRS / CrowdSec AppSec (true request-inspection WAF)** — needs a + version-coupled dynamic module (ModSecurity) or lua (AppSec) against an auto-updating + mainline nginx; high maintenance and false-positive tuning for a threat (dumb scanning) + the banning layer already handles. Deferred until a specific app needs signature + inspection. +- **fail2ban instead of CrowdSec** — simpler, no external dependency, but no crowd-sourced + blocklist, which is the main pre-emptive value here. +- **A standalone `waf` asset type** — over-models: enforcement must be co-located with the + edge host, so a separate placement FK invites a meaningless "WAF on a host with no + ingress" case. The env-block + per-edge opt-out gives the same control with far less + surface. From 75eef004168dbdf927484f21fe80b4f9c66cb0ce Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 19 Jul 2026 16:24:06 +0000 Subject: [PATCH 02/14] =?UTF-8?q?docs(adr):=20propose=20edge=20security=20?= =?UTF-8?q?tier=20=E2=80=94=20rate=20limiting=20+=20CrowdSec?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Design record (ADR-0043, proposed) for a public-edge security tier with two independently-toggled capabilities authored from one variables.yaml `security:` block: - Rate limiting: nginx-native limit_req/limit_conn rendered into the edge nginx.conf, with named profiles + a default_profile (mirroring the observability notification-profile idiom). Per-profile keying is `ip` (default) or a custom header with automatic IP fallback; JWT-claim keying is excluded at the edge per the ADR-0032 gateway trust boundary. - CrowdSec: per-host agent (edges only) with the nftables firewall bouncer plus the free community blocklist, installed like the otelcol collector (ADR-0031). Documents the L7-vs-volumetric DoS scope boundary explicitly and the wiring shape (internal/crowdsec package, provisionCrowdsec pass, ingressHostUnion targeting). No implementation yet — design for review. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_011RB2WPLAzoDT1bjXrMfoXi --- ...3-edge-security-rate-limiting-and-crowdsec.md | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/docs/adr/0043-edge-security-rate-limiting-and-crowdsec.md b/docs/adr/0043-edge-security-rate-limiting-and-crowdsec.md index 9e17316..cfebf59 100644 --- a/docs/adr/0043-edge-security-rate-limiting-and-crowdsec.md +++ b/docs/adr/0043-edge-security-rate-limiting-and-crowdsec.md @@ -55,14 +55,17 @@ security: default_profile: standard # applied to EVERY public server unless overridden profiles: standard: + key: ip # rate-limit key: client IP (default) requests_per_second: 20 burst: 40 # queued excess before a 429 max_connections: 40 # concurrent conns per client IP strict: + key: ip requests_per_second: 5 burst: 10 max_connections: 10 api: + key: { header: X-Api-Key } # key on the API-key header, fall back to IP when absent requests_per_second: 100 burst: 200 max_connections: 200 @@ -119,6 +122,19 @@ needs no secret. using it emits `limit_req zone=

burst= nodelay;` + `limit_conn

_conn ;`. `limit_req_status 429;` / `limit_conn_status 429;` so rejects are `429` (CrowdSec- parseable), not the nginx default `503`. +- **Keying is per-profile: `ip` (default) or a custom header, never a JWT claim at the + edge.** `key: ip` uses `$binary_remote_addr` (accurate here — grey-cloud records and the + ssl_preread `set_real_ip_from`/`proxy_protocol` recovery both yield the true client IP). + `key: { header: X-Api-Key }` keys `limit_req` on `$http_x_api_key`, but nginx **skips a + limit whose key is empty** (a bypass for header-less requests), so header keying always + renders through a `map` that falls back to `$binary_remote_addr` when the header is + absent. `limit_conn` stays IP-keyed regardless (connections are not per-request-header). + **JWT-claim keying is deliberately excluded**: stock nginx cannot parse a JWT (Plus/njs + only), and — the load-bearing reason — the gateway does not validate the JWT (ADR-0032: + the service does), so a claim key at the edge is *unverified* and spoofable for quota + evasion (set `sub=` to burn their budget, or rotate `sub` for fresh unlimited + buckets). Per-verified-client quotas belong at the service, which holds the validated + identity; the edge offers IP and header keying only. - **Applies to the http edge servers** (tls-termination routes, apps, gateway). **Health and ACME servers are exempt** — liveness probes and cert issuance must never be throttled. **Forward (L4) routes** get only the profile's `max_connections` via a From 6f940b6937d3f834835ab7103ba678703f564924 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 19 Jul 2026 16:34:07 +0000 Subject: [PATCH 03/14] docs(adr): rate limiting is IP-keyed in v1; note gateway-auth coupling Per review: v1 keys rate limits on client IP only. Identity-based keying (API key / verified client) is deferred until authentication moves into the gateway, at which point the edge holds a verified claim and per-identity keying becomes safe on the same limit_req mechanism. Adds a Future-direction note pointing at a gateway-auth ADR (nginx auth_request + an auth service). Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_011RB2WPLAzoDT1bjXrMfoXi --- ...dge-security-rate-limiting-and-crowdsec.md | 35 ++++++++++--------- 1 file changed, 19 insertions(+), 16 deletions(-) diff --git a/docs/adr/0043-edge-security-rate-limiting-and-crowdsec.md b/docs/adr/0043-edge-security-rate-limiting-and-crowdsec.md index cfebf59..30359a6 100644 --- a/docs/adr/0043-edge-security-rate-limiting-and-crowdsec.md +++ b/docs/adr/0043-edge-security-rate-limiting-and-crowdsec.md @@ -55,17 +55,14 @@ security: default_profile: standard # applied to EVERY public server unless overridden profiles: standard: - key: ip # rate-limit key: client IP (default) requests_per_second: 20 burst: 40 # queued excess before a 429 max_connections: 40 # concurrent conns per client IP strict: - key: ip requests_per_second: 5 burst: 10 max_connections: 10 api: - key: { header: X-Api-Key } # key on the API-key header, fall back to IP when absent requests_per_second: 100 burst: 200 max_connections: 200 @@ -122,19 +119,15 @@ needs no secret. using it emits `limit_req zone=

burst= nodelay;` + `limit_conn

_conn ;`. `limit_req_status 429;` / `limit_conn_status 429;` so rejects are `429` (CrowdSec- parseable), not the nginx default `503`. -- **Keying is per-profile: `ip` (default) or a custom header, never a JWT claim at the - edge.** `key: ip` uses `$binary_remote_addr` (accurate here — grey-cloud records and the - ssl_preread `set_real_ip_from`/`proxy_protocol` recovery both yield the true client IP). - `key: { header: X-Api-Key }` keys `limit_req` on `$http_x_api_key`, but nginx **skips a - limit whose key is empty** (a bypass for header-less requests), so header keying always - renders through a `map` that falls back to `$binary_remote_addr` when the header is - absent. `limit_conn` stays IP-keyed regardless (connections are not per-request-header). - **JWT-claim keying is deliberately excluded**: stock nginx cannot parse a JWT (Plus/njs - only), and — the load-bearing reason — the gateway does not validate the JWT (ADR-0032: - the service does), so a claim key at the edge is *unverified* and spoofable for quota - evasion (set `sub=` to burn their budget, or rotate `sub` for fresh unlimited - buckets). Per-verified-client quotas belong at the service, which holds the validated - identity; the edge offers IP and header keying only. +- **v1 keys on client IP only (`$binary_remote_addr`).** Accurate here — grey-cloud records + and the ssl_preread `set_real_ip_from`/`proxy_protocol` recovery both yield the true + client IP. Identity-based keying (per API key or per verified client) is **deferred**, + and deferred for a *reason that resolves itself*: the gateway does not validate the JWT + today (ADR-0032 — the service does), so any identity claim at the edge is unverified and + spoofable for quota evasion. The moment authentication moves into the gateway (see + "Future" below), the claim becomes *verified at the edge* and per-client-identity keying + becomes safe to add on the same `limit_req_zone` mechanism — at which point a profile + gains an optional `key:` field. Until then, IP is the only trustworthy edge key. - **Applies to the http edge servers** (tls-termination routes, apps, gateway). **Health and ACME servers are exempt** — liveness probes and cert issuance must never be throttled. **Forward (L4) routes** get only the profile's `max_connections` via a @@ -234,6 +227,16 @@ Grounded in the existing otelcol path (`program.provisionObservability`, rebuilt on host replacement (bans re-learned; the community blocklist re-pulls). - New third-party apt source on edge hosts (CrowdSec), same trade-off class as ADR-0031. +## Future direction + +- **Gateway authentication.** Today the gateway routes and forwards the JWT for the + service to validate (ADR-0032). Moving *authentication* (token validation → a verified + identity header) into the gateway while keeping *authorization* at the service is a + natural next step; it also unlocks identity-based rate-limit keying here (the `key:` + field noted above). The mechanism that fits this ADR's stock-nginx constraint is nginx's + built-in `auth_request` delegating to a dedicated auth service — no lua/njs JWT logic in + the router. That belongs in its own ADR; this one only notes the coupling. + ## Considered and rejected - **Cloudflare WAF at the edge** — strongest and lowest-ops, but adds external lock-in and From f58b1d2f63c0d51be13ea7f800eaee1cb9836e67 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 19 Jul 2026 16:56:50 +0000 Subject: [PATCH 04/14] docs(adr): propose gateway authentication tier MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Companion to ADR-0043. Moves authentication into the north-south gateway while keeping authorization at the service (ADR-0032 defense-in-depth preserved): - nginx stays the router; auth is added via the built-in auth_request directive delegating to a co-located auth service reached over loopback/unix socket — no lua/njs JWT logic in nginx. - Custom Rust auth service (jsonwebtoken + cached JWKS) over an off-the-shelf gateway, consistent with the in-house mesh/PKI/agent posture; Ory Oathkeeper kept as the buy fallback. - Verified identity crosses to the service as headers, trusted only via the mesh from the gateway leaf. - Records the preaccess-vs-access phase-ordering trap: a single-pass limit_req on an auth-set header fails open; identity keying needs a second nginx pass or auth-service enforcement. - HA state model: per-node volumetric limits (approximate, CrowdSec-backstopped), shared-store (Redis) per-identity quotas in the auth service, and a central CrowdSec LAPI for fleet-wide bans; quota checks fail open. No implementation — design for review. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_011RB2WPLAzoDT1bjXrMfoXi --- docs/adr/0044-gateway-authentication-tier.md | 164 +++++++++++++++++++ 1 file changed, 164 insertions(+) create mode 100644 docs/adr/0044-gateway-authentication-tier.md diff --git a/docs/adr/0044-gateway-authentication-tier.md b/docs/adr/0044-gateway-authentication-tier.md new file mode 100644 index 0000000..e1f28bc --- /dev/null +++ b/docs/adr/0044-gateway-authentication-tier.md @@ -0,0 +1,164 @@ +--- +status: proposed +date: 2026-07-19 +issue: "TBD" +--- + +# Gateway authentication tier + +Today the north-south daemon gateway (ADR-0032) routes and forwards: it TLS-terminates, +matches the request path against the derived public-path globs (ADR-0034), and hands the +request to the local mesh egress with `X-Mesh-Target`, forwarding the daemon's +`Authorization` header **untouched** for the owning service to validate. Authentication is +entirely a service concern. + +This ADR moves **authentication** to the gateway while keeping **authorization** at the +service. The gateway becomes the first gate that produces a *verified* identity; the +service still enforces what that identity may do. This is a companion to ADR-0043 (edge +security tier) — it is the change that later makes per-identity rate limiting and quotas +safe at the edge. + +## The split: authenticate at the edge, authorize at the service + +- **Authentication** — "is this a valid, unexpired, correctly-signed token from our + issuer?" — **moves to the gateway.** Unauthenticated traffic is rejected at the edge + before it consumes a backend or traverses the mesh; the token is validated once, not + re-validated in every service. +- **Authorization** — "may *this* identity perform *this* action on *this* resource?" — + **stays at the service.** Only the service knows its resource model, and keeping it there + means a gateway compromise does not grant backend access. This preserves the ADR-0032 + defense-in-depth rather than replacing it. + +The gateway is therefore **not** the sole gatekeeper. It is the first gate; the service +remains the last one, and trusts the gateway's verdict only because it arrives over the +mesh (mTLS-proven `X-Service-Identity=gateway`), never from an arbitrary caller. + +## Decisions + +- **nginx stays the router; authentication is added via the built-in `auth_request` + directive — no lua/njs JWT logic in nginx.** nginx is an excellent TLS-terminating, + path-routing edge and is already rendered deterministically (`internal/nginx`, + ADR-0043). `auth_request` issues an internal subrequest to an auth endpoint before the + content phase: a `2xx` admits the request (and nginx forwards headers the auth service + returns), a `401/403` rejects it at the edge, any other status (including auth-service + unavailability) fails the request `500`. Token validation lives in a service behind that + seam, not in a per-request JS/lua blob inside the router. Rejected alternatives (njs JWT, + a full market gateway) are recorded below. + +- **The auth service is co-located on each gateway host and reached over loopback (or a + unix socket), not through the mesh.** `auth_request` fires once per incoming request, so + the auth service's latency and availability gate *every* request. A same-host loopback + call (or a unix-domain socket — no TCP port to collide or firewall) keeps that to + microseconds; routing the auth subrequest through the mesh would add an mTLS handshake + and two proxy hops to the hot path of every request. This mirrors the existing trust + boundary: the gateway already speaks only to `127.0.0.1:` (its local + mesh proxy) and trusts it because they share a host. The auth service may still be a mesh + member for its *own outbound* needs (fetching JWKS from the issuer, calling an identity + service), cached in-memory so it is not a per-request outbound; only the inbound + `auth_request` path is loopback. + +- **Build the auth service in Rust rather than adopting a market gateway.** The validation + surface is small and bounded — verify the token with a vetted library (`jsonwebtoken`, + which handles the crypto that is dangerous to hand-roll: `alg:none`, algorithm + confusion, signature verification), check `iss`/`aud`/`exp`/`nbf` with clock skew, cache + JWKS with rotation, emit the verified-identity headers, return `200`/`401`. This fits the + project's build-and-own posture (inforge, the mesh, the PKI, and the agent are all + in-house), reuses existing Rust (wardnet-cloud) and can share the daemon PKI trust model, + and deploys as just another co-located mesh member inforge already knows how to run. Auth + here is simpler than the mTLS mesh + PKI already operated. Ory Oathkeeper is the + considered off-the-shelf fallback (see below). + +- **Verified identity crosses to the service as headers, set explicitly.** The auth + subrequest's response headers are not auto-forwarded. nginx captures them with + `auth_request_set $var $upstream_http_

` and re-emits them onto the real upstream + request via `proxy_set_header` (e.g. `X-Auth-Subject`, plus any needed claim headers). + The service trusts these **only** because they arrive over the mesh from the gateway leaf + — a caller cannot inject them directly (the mesh admits only allowlisted peers, and the + gateway is a distinct mesh identity). The exact header set is a contract to fix when this + is implemented; `Authorization` continues to be forwarded so the service can re-derive + claims if it wishes. + +- **Rate limiting on a verified claim requires a second pass or auth-service enforcement — + a single-pass `limit_req` on an auth-set header silently does nothing.** nginx evaluates + `limit_req`/`limit_conn` in the **preaccess** phase, which runs *before* `auth_request` + (the **access** phase). On one pass the rate-limit key is read before the auth subrequest + runs, so a key referencing an auth-set header is empty — and nginx skips a limit whose + key is empty (it fails *open*). Two supported ways to actually key on a verified + identity: + 1. **Two-pass nginx** — the front server authenticates and stamps `X-Auth-Subject`, then + `proxy_pass`es to an internal loopback server whose preaccess `limit_req` keys on that + now-present header. One extra internal hop, idiomatic here (the ssl_preread→loopback + terminators and gateway→mesh egress are the same shape). + 2. **Enforce per-identity quotas in the auth service** — it already holds the verified + claims; it checks the counter and returns `429` from the subrequest. This is preferred + for anything richer than a leaky bucket (per-plan quotas, sliding windows) and is the + only option that scales to a global limit across nodes (next decision). + The ADR-0043 IP/volumetric layer stays in nginx preaccess, independent of auth. + +- **HA: separate limits by their state requirement — do not give nginx a global brain.** + nginx `limit_req`/`limit_conn` zones are per-host shared memory (across workers on one + box, not across hosts), so N gateway nodes enforce N independent limits. The design + splits accordingly: + - **Volumetric / IP limits stay per-node and approximate** — for flood protection, exact + global counting does not matter; per-node caps still stop a source from running wild, + and CrowdSec's nftables bans (ADR-0043) are the fleet backstop. + - **Per-identity quotas move to a shared store** — the auth service (which runs on every + gateway node and holds the verified identity) checks a **Redis-backed token bucket / + sliding window**, atomic via a Lua script, keyed by the verified client id. HA changes + only the *backing* of the counter (in-process → Redis), not the `auth_request`→`429` + shape. + - **CrowdSec bans move to a central/shared LAPI** — one node runs the local API, others + run agent+bouncer against it, so ban decisions are fleet-wide; the community blocklist + is already shared. + - **Failure posture**: the shared quota store is on the hot path, so quota checks **fail + open** — a store blip must not `503` the API — and the per-node nginx IP cap still + stands as the volumetric backstop. The two layers back each other up. + +## Request flow (target) + +``` +client ──TLS──▶ gateway nginx ──loopback──▶ auth service (200 + X-Auth-Subject | 401) + │ │ (validates JWT; JWKS cached; + │ │ optional per-identity quota via Redis) + └─(if 200, +X-Auth-Subject)──loopback──▶ mesh egress ──mTLS──▶ service + (trusts identity via + mesh; enforces authZ) +``` + +## Consequences + +- A new co-located daemon on every gateway host (the auth service), deployed and + supervised like the mesh proxy — `Restart=always` per `daemon-units-restart-on-any-exit`, + since a dead auth service fails all gateway traffic closed. +- The gateway nginx render (`gatewayServer`, `internal/nginx`) gains the `auth_request` + location and the `auth_request_set`/`proxy_set_header` identity plumbing; the two-pass + variant adds an internal server when identity-keyed rate limiting is enabled. +- A new hot-path dependency on the issuer's JWKS (cached, rotated) for the auth service, + and — only when global per-identity quotas are enabled — on a shared store (Redis). +- The ADR-0043 note "gateway auth unlocks identity keying" is made precise: the unlock is a + *verified identity existing at the edge*, enforced via a second nginx pass or in the auth + service — not `limit_req` suddenly seeing the claim on one pass. +- Authorization stays distributed across services; this ADR deliberately does not + centralize policy. A future policy-decision layer (if ever wanted) is out of scope. + +## Considered and rejected + +- **JWT validation inside nginx (njs or lua).** njs is available in the nginx.org repo and + can decode/verify a token, but it puts security-critical crypto in a per-request script + inside the router — hard to test, clutters the deterministic render, and owns JWKS + rotation / signature edge cases in the wrong place. `auth_request` to a real service is + cleaner and keeps nginx doing what it is good at. +- **A full market API gateway (Kong / Tyk / APISIX / Traefik).** Replaces the clean, + self-rendered nginx edge with a new runtime and config dialect and a larger operational + surface, to buy capabilities obtainable from `auth_request` + a small service. A step away + from the architecture, not toward it. +- **Ory Oathkeeper (kept as the buy fallback).** A purpose-built identity-&-access decision + proxy that fits the `auth_request` pattern and is config-driven. Reasonable if owning + token validation is undesirable; declined as the default only because the custom Rust + service is a small, well-understood surface consistent with everything else in-house. +- **Making the gateway the sole authenticator/authorizer.** Rejected — it would create a + single bypass point and discard the ADR-0032 service-side checks. The gateway + authenticates; the service still authorizes. +- **Global rate limiting inside nginx.** Not possible without an external store and lua/njs; + the shared-state need is met in the auth service instead, where the verified identity and + a Redis client already naturally live. From 9d2a0a2ef83550749d9c9263f15339b4d8c8cf88 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 19 Jul 2026 17:18:41 +0000 Subject: [PATCH 05/14] feat(nginx): render IP-based rate limiting on the ingress edge (ADR-0043) Rate limiting is a blanket ingress security measure: one uniform IP-keyed limit applied to every public server on an edge, not a per-route knob. Per-route / per-identity limits are a gateway-module concern (ADR-0044), not this layer. - types: RateLimitProfile (resolved limit: RPS/Burst/MaxConn, fixed zone stem) rides on IngressRoute/IngressApp/IngressGateway, so neither the IngressProvider.Realize signature nor the Hetzner provider changes. - internal/nginx: emit one limit_req_zone/limit_conn_zone per context in http{} and stream{}, with per-location limit_req/limit_conn on the service-facing locations. Health and ACME servers, and the gateway's edge health-probe locations, are exempt. Forward (L4) routes get limit_conn only. Throttled requests answer 429 (CrowdSec-parseable). A nil profile renders byte-identically to before (golden test unchanged). Also updates ADR-0043 (drop per-route profiles; uniform security floor) and ADR-0044 (per-route/identity limits owned here; auth_request vs custom nginx dynamic module left as an open enforcement choice). Wiring (types.SecurityConfig, schemas, validate, program resolution) lands in the next commit. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_011RB2WPLAzoDT1bjXrMfoXi --- ...dge-security-rate-limiting-and-crowdsec.md | 149 +++++++-------- docs/adr/0044-gateway-authentication-tier.md | 17 ++ internal/nginx/config.go | 170 +++++++++++++++--- internal/nginx/config_test.go | 62 +++++++ internal/types/types.go | 27 +++ 5 files changed, 309 insertions(+), 116 deletions(-) diff --git a/docs/adr/0043-edge-security-rate-limiting-and-crowdsec.md b/docs/adr/0043-edge-security-rate-limiting-and-crowdsec.md index 30359a6..2eaaa41 100644 --- a/docs/adr/0043-edge-security-rate-limiting-and-crowdsec.md +++ b/docs/adr/0043-edge-security-rate-limiting-and-crowdsec.md @@ -52,45 +52,19 @@ security: # version: "1.6.4" # optional pin; else inforge's DefaultVersion # console: true # optional dashboard enrollment (needs reserved secret) rate_limit: - default_profile: standard # applied to EVERY public server unless overridden - profiles: - standard: - requests_per_second: 20 - burst: 40 # queued excess before a 429 - max_connections: 40 # concurrent conns per client IP - strict: - requests_per_second: 5 - burst: 10 - max_connections: 10 - api: - requests_per_second: 100 - burst: 200 - max_connections: 200 + enabled: true # one blanket IP-based limit on EVERY public edge server + requests_per_second: 20 + burst: 40 # queued excess before a 429 + max_connections: 40 # concurrent conns per client IP ``` -**Profiles, not per-route detail** (mirrors the observability notification-profile -idiom — `Profiles map[string]Profile` + a `default_profile`, referenced by name). Limits -are named once; a route selects one with a single token. `default_profile` applies to -every public server automatically, so "all routes at 20 rps" needs zero per-route -authoring. - -**Per-resource override** references a profile by name (never re-specifies limits): - -```yaml -# service route (resources//regional/service//manifest.yaml) -routes: - - type: tls-termination - listen: 443 - target: 8080 - rate_limit_profile: strict # overrides the default for this route - -# app (app//manifest.yaml) and gateway (gateway//manifest.yaml) -rate_limit_profile: api # one profile per app / per gateway -``` - -`rate_limit_profile: none` is the reserved sentinel meaning **no limiting** on that -resource (parallels a `muted` route in notifications). `none` cannot be defined as a -real profile. +**One uniform limit, no per-route knobs.** IP rate limiting is a blanket *security* +measure — a floor applied identically to every public server on an edge, exactly like +CrowdSec. There is deliberately no per-route/per-app profile: a single +`requests_per_second`/`burst`/`max_connections` covers the whole edge. Per-route and +per-identity limits are a *different* concern (application logic, not a security floor) +and belong to the gateway module once authentication is at the edge — see ADR-0044; they +are enforced there (the auth service or a custom nginx dynamic module), never here. **Per-edge opt-out** — an ingress/gateway leaves the whole security tier: @@ -112,28 +86,27 @@ needs no secret. is stock-nginx.org (no lua/OpenResty), flows through the existing `internal/nginx` crossplane render, and preserves the deterministic-bytes property. `limit_req` is the leaky-bucket request-rate limiter; `limit_conn` caps concurrent connections per IP. -- **Named profiles + a default profile**, keyed by client IP (`$binary_remote_addr`). - Modeled directly on `types.NotificationsSpec` profiles: define once, reference by - name, one env-level default. Each *referenced* profile renders one - `limit_req_zone`/`limit_conn_zone` in `http{}` (deduped by profile name); every server - using it emits `limit_req zone=

burst= nodelay;` + `limit_conn

_conn ;`. - `limit_req_status 429;` / `limit_conn_status 429;` so rejects are `429` (CrowdSec- - parseable), not the nginx default `503`. -- **v1 keys on client IP only (`$binary_remote_addr`).** Accurate here — grey-cloud records - and the ssl_preread `set_real_ip_from`/`proxy_protocol` recovery both yield the true - client IP. Identity-based keying (per API key or per verified client) is **deferred**, - and deferred for a *reason that resolves itself*: the gateway does not validate the JWT - today (ADR-0032 — the service does), so any identity claim at the edge is unverified and - spoofable for quota evasion. The moment authentication moves into the gateway (see - "Future" below), the claim becomes *verified at the edge* and per-client-identity keying - becomes safe to add on the same `limit_req_zone` mechanism — at which point a profile - gains an optional `key:` field. Until then, IP is the only trustworthy edge key. -- **Applies to the http edge servers** (tls-termination routes, apps, gateway). **Health - and ACME servers are exempt** — liveness probes and cert issuance must never be - throttled. **Forward (L4) routes** get only the profile's `max_connections` via a - `stream{}` `limit_conn` (a stream-context zone); `requests_per_second`/`burst` do not - apply at L4 and are ignored for a forward route (validation notes this rather than - erroring, so a profile can serve both). +- **One uniform limit per edge, keyed by client IP (`$binary_remote_addr`) — no per-route + profiles.** Rate limiting is a security floor, so the same `requests_per_second`/`burst`/ + `max_connections` is stamped on every public server of an edge, not tuned per route. The + renderer resolves that single limit onto each `IngressRoute`/`IngressApp`/`IngressGateway` + and emits one `limit_req_zone`/`limit_conn_zone` in `http{}` (deduped — one zone, fixed + stem), with `limit_req zone= burst= nodelay;` + `limit_conn ;` in each + service-facing location. `limit_req_status 429;` / `limit_conn_status 429;` make a + throttled request answer `429` (CrowdSec-parseable), not the nginx default `503`. +- **Client IP is the only key.** Accurate here — grey-cloud records and the ssl_preread + `set_real_ip_from`/`proxy_protocol` recovery both yield the true client IP. Per-identity + keying is out of scope for this layer entirely: it is application logic, it requires a + *verified* identity (which the edge does not have until gateway auth lands — ADR-0032), + and it belongs to the gateway module (ADR-0044), not the ingress security floor. +- **Applies to the service-facing http locations** (tls-termination routes, apps, gateway + mesh routes). **Health and ACME servers are exempt** — liveness probes and cert issuance + must never be throttled; they are rendered by separate functions that carry no limit, and + the gateway's own edge health-probe locations are left unlimited too. **Forward (L4) + routes** get only `max_connections` via a `stream{}` `limit_conn` (its own zone + namespace); `requests_per_second`/`burst` do not apply at L4. A forward sharing a mixed + ssl_preread port is not connection-limited (its socket also carries the terminators' + traffic) — a documented v1 limitation. - **The private mesh nginx (`internal/meshnginx`) is untouched.** Rate limiting is an internet-edge concern; east-west peer traffic is already allowlist-gated. @@ -189,27 +162,28 @@ Grounded in the existing otelcol path (`program.provisionObservability`, `ConfigPath`, `BouncerConfigPath`, `AcquisPath`, `DefaultVersion`, reserved-secret namespace/key) + `install.go` (`InstallScript`, `AcquisScript`, `BouncerScript`, `EnrollScript`, `WriteFileScript`). Pure, Pulumi-free. -- **`types`** — new `SecurityConfig` (with `Crowdsec` + `RateLimit` sub-blocks and - profiles) on `EnvironmentVariables`; a `Security *bool` opt-out on `IngressSpec` and - `GatewaySpec`; a `RateLimitProfile string` on `RouteSpec`, `AppSpec`, `GatewaySpec`. - Accessors defaulting like `DashboardsEnabled()`. -- **`schemas`** — extend `environment.json` (the `security` block + profiles), - `ingress.json`/`gateway.json` (`security`, `rate_limit_profile`), `service.json` (route - `rate_limit_profile`), `app.json` (`rate_limit_profile`). -- **`loader`** — normalize/trim; default `rate_limit.default_profile`; carry profiles. -- **`validate`** — new `checkSecurity`: `default_profile` and every `rate_limit_profile` - reference resolve to a defined profile (or `none`); profile bounds - (`requests_per_second > 0`, `burst >= 0`, `max_connections >= 0`); reserved name `none` - not defined; `crowdsec.enabled` with no edge host in the env is an error; console - enrollment configured ⇒ reserved secret present. -- **`internal/nginx/config.go`** — thread the resolved rate-limit plan into `Render(...)` - and `IngressProvider.Realize(...)`; emit the zones + per-server directives. The - `Realize` signature change ripples to the Hetzner provider and the `types` interface - (a mechanical, compile-checked change). -- **`program.go`** — new `provisionCrowdsec(...)` copied from `provisionObservability` but - iterating the ingress∪gateway host set (respecting per-edge opt-out), sharing the - memoized `gates` cloud-init map, called in the scope loop beside the observability pass. - Rate-limit config is resolved per host and passed into `realizeIngress`. +- **`types`** — new `SecurityConfig` (with `Crowdsec` + a flat `RateLimit` + `{enabled, requests_per_second, burst, max_connections}`) on `EnvironmentVariables`; a + `Security *bool` opt-out on `IngressSpec` and `GatewaySpec`. The resolved + `RateLimitProfile` (a fixed-stem struct) already rides on the ingress-derived server + structs — **no per-route spec fields, and the `IngressProvider.Realize` signature is + unchanged.** Accessors defaulting like `DashboardsEnabled()`. +- **`schemas`** — extend `ingress.json`/`gateway.json` with the boolean `security` opt-out. + The `security` block lives in `variables.yaml`, which is struct-decoded (like + `observability:`), so no JSON schema changes for the block itself. No per-route schema + changes. +- **`loader`** — normalize/trim; default the rate-limit fields. +- **`validate`** — new `checkSecurity`: rate-limit bounds (`requests_per_second > 0`, + `burst >= 0`, `max_connections >= 0`) when enabled; `crowdsec.enabled` with no edge host + in the env is an error; console enrollment configured ⇒ reserved secret present. +- **`internal/nginx/config.go`** — DONE (commit 1): emits the zones + per-location + directives from the `RateLimit` field on the server structs; render is fully unit-tested, + and a nil profile renders byte-identically to before. +- **`program.go`** — resolve the single per-edge `RateLimitProfile` from the env + `SecurityConfig` (unless the edge opted out) and stamp it on every server the ingress + derivations build for that host; then the new `provisionCrowdsec(...)` (copied from + `provisionObservability`) iterating the ingress∪gateway host set, sharing the memoized + `gates` cloud-init map, called in the scope loop beside the observability pass. - **Docs** — public docs updated per rule `update-public-docs-with-resource-changes`; AGENTS.md gains a "Security (edge)" section. @@ -222,20 +196,19 @@ Grounded in the existing otelcol path (`program.provisionObservability`, that has never fetched them; a hub outage fails the CrowdSec pass on a fresh edge (an already-provisioned host re-runs as a no-op). - The edge nginx.conf grows `limit_req_zone`/`limit_conn_zone` blocks; render stays - deterministic (zones deduped and sorted by profile name). + deterministic (one deduped zone per context, fixed stem). - CrowdSec keeps local state (SQLite decisions) on the edge host — surviving reloads, rebuilt on host replacement (bans re-learned; the community blocklist re-pulls). - New third-party apt source on edge hosts (CrowdSec), same trade-off class as ADR-0031. ## Future direction -- **Gateway authentication.** Today the gateway routes and forwards the JWT for the - service to validate (ADR-0032). Moving *authentication* (token validation → a verified - identity header) into the gateway while keeping *authorization* at the service is a - natural next step; it also unlocks identity-based rate-limit keying here (the `key:` - field noted above). The mechanism that fits this ADR's stock-nginx constraint is nginx's - built-in `auth_request` delegating to a dedicated auth service — no lua/njs JWT logic in - the router. That belongs in its own ADR; this one only notes the coupling. +- **Per-route / per-identity limits are a gateway concern, not this layer's.** This ADR is + a blanket IP security floor. Finer limits (per route, per API key, per verified client) + are application logic that requires a *verified* identity, and are enforced at the gateway + module once authentication is at the edge — see ADR-0044 (gateway authentication tier), + which owns both the `auth_request`/dynamic-module mechanism and any identity-keyed + limiting. This ADR only notes the coupling; it intentionally ships no per-route surface. ## Considered and rejected diff --git a/docs/adr/0044-gateway-authentication-tier.md b/docs/adr/0044-gateway-authentication-tier.md index e1f28bc..1fe350f 100644 --- a/docs/adr/0044-gateway-authentication-tier.md +++ b/docs/adr/0044-gateway-authentication-tier.md @@ -35,6 +35,23 @@ mesh (mTLS-proven `X-Service-Identity=gateway`), never from an arbitrary caller. ## Decisions +- **This tier owns per-route and per-identity limits; the ingress security floor + (ADR-0043) does not.** ADR-0043 is a blanket IP rate limit applied uniformly to every + edge server as a security measure. Finer limits — per route, per API key, per verified + client — are application logic keyed on a *verified* identity, which only exists once + authentication is at the gateway. They are therefore defined and enforced here, on top of + the IP floor below. + +- **The enforcement mechanism for auth (and identity-keyed limits) is an open choice + between two shapes, both compatible with the stock-nginx render.** (a) `auth_request` → + a co-located auth service (below), or (b) a **custom nginx dynamic module** (compiled, + e.g. via `ngx-rust`) that validates the token and enforces per-route/identity limits + inline — no per-request subrequest, richest control, but it owns a module built against + each nginx version (the version-coupling cost that ruled out ModSecurity in ADR-0043). + The `auth_request` shape is the lower-risk default and the one the rest of this ADR + details; the dynamic module is the higher-ceiling alternative if the subrequest hop or + the split enforcement point ever bites. This choice is deliberately left open here. + - **nginx stays the router; authentication is added via the built-in `auth_request` directive — no lua/njs JWT logic in nginx.** nginx is an excellent TLS-terminating, path-routing edge and is already rendered deterministically (`internal/nginx`, diff --git a/internal/nginx/config.go b/internal/nginx/config.go index 3910dde..0ca6e9d 100644 --- a/internal/nginx/config.go +++ b/internal/nginx/config.go @@ -298,6 +298,19 @@ func httpBlock(terminate []types.IngressRoute, apps []types.IngressApp, health [ dir("real_ip_header", "proxy_protocol"), ) } + // Rate-limit zones for every profile an http server on this host references, + // declared once in http{} before the servers that consume them (ADR-0043). + var httpProfiles []*types.RateLimitProfile + for _, r := range terminate { + httpProfiles = append(httpProfiles, r.RateLimit) + } + for _, a := range apps { + httpProfiles = append(httpProfiles, a.RateLimit) + } + for _, g := range gateways { + httpProfiles = append(httpProfiles, g.RateLimit) + } + children = append(children, httpRateLimitZones(httpProfiles)...) for _, r := range terminate { children = append(children, terminateServer(r, listenDir(r.Listen))) } @@ -333,6 +346,12 @@ func httpBlock(terminate []types.IngressRoute, apps []types.IngressApp, health [ // accepting the PROXY protocol). Render has already verified r.Backend is non-empty. func terminateServer(r types.IngressRoute, listen *crossplane.Directive) *crossplane.Directive { serverName := append([]string{}, r.FQDNs...) + loc := append(rlHTTPLimitDirs(r.RateLimit), + dir("proxy_pass", fmt.Sprintf("http://%s:%d", r.Backend, r.Target)), + dir("proxy_set_header", "Host", "$host"), + dir("proxy_set_header", "X-Forwarded-For", "$proxy_add_x_forwarded_for"), + dir("proxy_set_header", "X-Forwarded-Proto", "$scheme"), + ) return block("server", nil, listen, dir("server_name", serverName...), @@ -340,12 +359,7 @@ func terminateServer(r types.IngressRoute, listen *crossplane.Directive) *crossp dir("ssl_certificate", "$acme_certificate"), dir("ssl_certificate_key", "$acme_certificate_key"), dir("ssl_certificate_cache", "max=2"), - block("location", []string{"/"}, - dir("proxy_pass", fmt.Sprintf("http://%s:%d", r.Backend, r.Target)), - dir("proxy_set_header", "Host", "$host"), - dir("proxy_set_header", "X-Forwarded-For", "$proxy_add_x_forwarded_for"), - dir("proxy_set_header", "X-Forwarded-Proto", "$scheme"), - ), + block("location", []string{"/"}, loc...), ) } @@ -359,6 +373,7 @@ func appServer(a types.IngressApp, listen *crossplane.Directive) *crossplane.Dir if a.Spa { fallback = []string{"$uri", "$uri/", "/index.html"} } + loc := append(rlHTTPLimitDirs(a.RateLimit), dir("try_files", fallback...)) return block("server", nil, listen, dir("server_name", a.FQDN), @@ -368,9 +383,7 @@ func appServer(a types.IngressApp, listen *crossplane.Directive) *crossplane.Dir dir("ssl_certificate_cache", "max=2"), dir("root", a.Root), dir("index", "index.html"), - block("location", []string{"/"}, - dir("try_files", fallback...), - ), + block("location", []string{"/"}, loc...), ) } @@ -423,23 +436,22 @@ func gatewayServer(g types.IngressGateway, listen *crossplane.Directive, regexOf )) } for _, rt := range routes { - children = append(children, - block("location", []string{"~", regexOf[rt.Pattern]}, - dir("proxy_http_version", "1.1"), - dir("proxy_set_header", "Upgrade", "$http_upgrade"), - dir("proxy_set_header", "Connection", "$connection_upgrade"), - dir("proxy_set_header", "Host", "$host"), - // SET, never append: this is the internet-facing first hop, so a - // daemon-supplied X-Forwarded-For is untrusted input — appending - // ($proxy_add_x_forwarded_for) would forward a forged first entry - // through the mesh and defeat per-IP audit/rate-limit logic. - dir("proxy_set_header", "X-Forwarded-For", "$remote_addr"), - dir("proxy_set_header", "X-Forwarded-Proto", "$scheme"), - dir("proxy_set_header", "X-Mesh-Target", rt.Service), - dir("proxy_read_timeout", "3600s"), - dir("proxy_pass", fmt.Sprintf("http://127.0.0.1:%d", meshpaths.GatewayEgressPort)), - ), + loc := append(rlHTTPLimitDirs(g.RateLimit), + dir("proxy_http_version", "1.1"), + dir("proxy_set_header", "Upgrade", "$http_upgrade"), + dir("proxy_set_header", "Connection", "$connection_upgrade"), + dir("proxy_set_header", "Host", "$host"), + // SET, never append: this is the internet-facing first hop, so a + // daemon-supplied X-Forwarded-For is untrusted input — appending + // ($proxy_add_x_forwarded_for) would forward a forged first entry + // through the mesh and defeat per-IP audit/rate-limit logic. + dir("proxy_set_header", "X-Forwarded-For", "$remote_addr"), + dir("proxy_set_header", "X-Forwarded-Proto", "$scheme"), + dir("proxy_set_header", "X-Mesh-Target", rt.Service), + dir("proxy_read_timeout", "3600s"), + dir("proxy_pass", fmt.Sprintf("http://127.0.0.1:%d", meshpaths.GatewayEgressPort)), ) + children = append(children, block("location", []string{"~", regexOf[rt.Pattern]}, loc...)) } children = append(children, jsonNotFoundLocation()) return block("server", nil, children...) @@ -505,6 +517,20 @@ func healthServer(h types.IngressHealth, healthPort int) *crossplane.Directive { // ports keep the plain raw-L4 server. func streamBlock(forwardOnly []types.IngressRoute, mixed []mixedPort) *crossplane.Directive { var children crossplane.Directives + // Stream limit_conn zones for the forward servers carrying a rate-limit profile, + // declared once at stream{} scope before the servers use them (ADR-0043). Only + // connection limiting applies at L4 — there is no request rate to shape. A forward + // sharing a mixed port is fronted by the ssl_preread server and is not limited here + // (its socket also carries the terminators' traffic); only forward-only ports are. + var streamProfiles []*types.RateLimitProfile + for _, r := range forwardOnly { + streamProfiles = append(streamProfiles, r.RateLimit) + } + for _, p := range dedupeProfiles(streamProfiles) { + if p.MaxConn > 0 { + children = append(children, dir("limit_conn_zone", "$binary_remote_addr", "zone="+rlStreamConnZone(p.Name)+":"+rlZoneSize)) + } + } for _, mp := range mixed { entries := make(crossplane.Directives, 0, len(mp.fqdns)+1) for _, fqdn := range mp.fqdns { @@ -525,15 +551,103 @@ func streamBlock(forwardOnly []types.IngressRoute, mixed []mixedPort) *crossplan )) } for _, r := range forwardOnly { - children = append(children, block("server", nil, - dir("listen", strconv.Itoa(r.Listen)), + srv := crossplane.Directives{dir("listen", strconv.Itoa(r.Listen))} + srv = append(srv, rlStreamLimitDirs(r.RateLimit)...) + srv = append(srv, dir("proxy_pass", fmt.Sprintf("%s:%d", r.Backend, r.Target)), dir("proxy_protocol", "on"), - )) + ) + children = append(children, block("server", nil, srv...)) } return block("stream", nil, children...) } +// Rate limiting (ADR-0043). Each referenced profile becomes one shared-memory zone in +// http{} (and/or stream{}); every server carrying the profile emits the matching +// limit_req/limit_conn directives inside its location. Keying is always the client IP +// ($binary_remote_addr) — accurate here because set_real_ip recovers the true address on +// ssl_preread'd mixed ports in POST_READ, before the preaccess phase where limit_req runs. +const rlZoneSize = "10m" + +// rlReqZone/rlConnZone/rlStreamConnZone name the shared-memory zones for a profile. +// The prefixes keep them clear of other nginx names and keep the http and stream +// limit_conn zones distinct (separate nginx contexts, rendered independently). +func rlReqZone(name string) string { return "rl_" + name } +func rlConnZone(name string) string { return "rlc_" + name } +func rlStreamConnZone(name string) string { return "rls_" + name } + +// rlHTTPLimitDirs are the per-location limit directives for an http server carrying +// profile p (nil -> none). limit_req is emitted only when the profile sets a rate, +// limit_conn only when it caps connections — so a profile may do either or both. +func rlHTTPLimitDirs(p *types.RateLimitProfile) []*crossplane.Directive { + if p == nil { + return nil + } + var d []*crossplane.Directive + if p.RPS > 0 { + d = append(d, dir("limit_req", "zone="+rlReqZone(p.Name), fmt.Sprintf("burst=%d", p.Burst), "nodelay")) + } + if p.MaxConn > 0 { + d = append(d, dir("limit_conn", rlConnZone(p.Name), strconv.Itoa(p.MaxConn))) + } + return d +} + +// rlStreamLimitDirs are the per-server limit directives for a stream (forward) server: +// only connection limiting applies at L4 (there is no request rate to shape). +func rlStreamLimitDirs(p *types.RateLimitProfile) []*crossplane.Directive { + if p == nil || p.MaxConn <= 0 { + return nil + } + return []*crossplane.Directive{dir("limit_conn", rlStreamConnZone(p.Name), strconv.Itoa(p.MaxConn))} +} + +// dedupeProfiles returns the distinct profiles among the pointers, sorted by name, so +// zone emission is deterministic and each zone is declared exactly once per context. +func dedupeProfiles(ps []*types.RateLimitProfile) []*types.RateLimitProfile { + seen := map[string]*types.RateLimitProfile{} + for _, p := range ps { + if p != nil { + seen[p.Name] = p + } + } + names := make([]string, 0, len(seen)) + for n := range seen { + names = append(names, n) + } + sort.Strings(names) + out := make([]*types.RateLimitProfile, 0, len(names)) + for _, n := range names { + out = append(out, seen[n]) + } + return out +} + +// httpRateLimitZones renders the http{} shared-memory zones plus the status overrides +// for every profile an http server references. limit_req_status/limit_conn_status make a +// throttled request answer 429 (CrowdSec-parseable) rather than nginx's default 503. +func httpRateLimitZones(profiles []*types.RateLimitProfile) []*crossplane.Directive { + var out []*crossplane.Directive + anyReq, anyConn := false, false + for _, p := range dedupeProfiles(profiles) { + if p.RPS > 0 { + out = append(out, dir("limit_req_zone", "$binary_remote_addr", "zone="+rlReqZone(p.Name)+":"+rlZoneSize, fmt.Sprintf("rate=%dr/s", p.RPS))) + anyReq = true + } + if p.MaxConn > 0 { + out = append(out, dir("limit_conn_zone", "$binary_remote_addr", "zone="+rlConnZone(p.Name)+":"+rlZoneSize)) + anyConn = true + } + } + if anyReq { + out = append(out, dir("limit_req_status", "429")) + } + if anyConn { + out = append(out, dir("limit_conn_status", "429")) + } + return out +} + // sortedIntKeys returns the keys of an int set as an ascending slice. func sortedIntKeys(set map[int]bool) []int { out := make([]int, 0, len(set)) diff --git a/internal/nginx/config_test.go b/internal/nginx/config_test.go index 6926a43..f06172b 100644 --- a/internal/nginx/config_test.go +++ b/internal/nginx/config_test.go @@ -76,6 +76,68 @@ stream { assert.Equal(t, want, got) } +// TestRenderRateLimit checks the ADR-0043 IP rate-limit rendering: one uniform limit +// stamped on a tls-termination route (http limit_req + limit_conn) and a forward route +// (stream limit_conn), with each shared-memory zone declared exactly once per context +// and throttled requests answered 429 rather than nginx's default 503. +func TestRenderRateLimit(t *testing.T) { + rl := &types.RateLimitProfile{Name: "edge", RPS: 20, Burst: 40, MaxConn: 40} + routes := []types.IngressRoute{ + {Service: "dns", Type: types.IngressTypeForward, Listen: 853, Target: 5353, Backend: "127.0.0.1", RateLimit: rl}, + {Service: "api", Type: types.IngressTypeTLSTermination, Listen: 443, Target: 8080, Backend: "127.0.0.1", + FQDNs: []string{"api.svc.prd.use1.wardnet.network"}, RateLimit: rl}, + } + got, err := Render(routes, nil, nil, 0, nil) + require.NoError(t, err) + + // http{} zones, declared once, with the 429 status overrides. + assert.Contains(t, got, "limit_req_zone $binary_remote_addr zone=rl_edge:10m rate=20r/s;") + assert.Contains(t, got, "limit_conn_zone $binary_remote_addr zone=rlc_edge:10m;") + assert.Contains(t, got, "limit_req_status 429;") + assert.Contains(t, got, "limit_conn_status 429;") + assert.Equal(t, 1, strings.Count(got, "limit_req_zone $binary_remote_addr zone=rl_edge"), "req zone declared once") + + // per-location http limits on the tls-termination server. + assert.Contains(t, got, "limit_req zone=rl_edge burst=40 nodelay;") + assert.Contains(t, got, "limit_conn rlc_edge 40;") + + // stream (forward) connection limit — its own zone namespace. + assert.Contains(t, got, "limit_conn_zone $binary_remote_addr zone=rls_edge:10m;") + assert.Contains(t, got, "limit_conn rls_edge 40;") +} + +// TestRenderRateLimitExemptsHealthAndACME confirms the limit lands only on the +// service-facing locations: the public health servers and the :80 ACME/redirect server +// are never rate-limited (they are rendered by separate functions that carry no profile). +func TestRenderRateLimitExemptsHealthAndACME(t *testing.T) { + rl := &types.RateLimitProfile{Name: "edge", RPS: 20, Burst: 40, MaxConn: 40} + routes := []types.IngressRoute{ + {Service: "api", Type: types.IngressTypeTLSTermination, Listen: 443, Target: 8080, Backend: "127.0.0.1", + FQDNs: []string{"api.svc.prd.use1.wardnet.network"}, RateLimit: rl}, + } + health := []types.IngressHealth{ + {Service: "api", FQDN: "api.svc.prd.use1.wardnet.network", Target: 9000, Backend: "127.0.0.1", Paths: []string{"/healthz"}}, + } + got, err := Render(routes, nil, health, 81, nil) + require.NoError(t, err) + // Exactly one limited location — the tls-termination route. Health + ACME are exempt. + assert.Equal(t, 1, strings.Count(got, "limit_req zone=rl_edge")) + assert.Contains(t, got, "listen 81;") // the health server exists... +} + +// TestRenderNoRateLimit guards the golden's premise: a nil profile emits no limit +// directives at all, so an env without rate limiting renders byte-identically to before. +func TestRenderNoRateLimit(t *testing.T) { + routes := []types.IngressRoute{ + {Service: "api", Type: types.IngressTypeTLSTermination, Listen: 443, Target: 8080, Backend: "127.0.0.1", + FQDNs: []string{"api.svc.prd.use1.wardnet.network"}}, + } + got, err := Render(routes, nil, nil, 0, nil) + require.NoError(t, err) + assert.NotContains(t, got, "limit_req") + assert.NotContains(t, got, "limit_conn") +} + // TestRenderMixedPreread pins the mixed-port shape: two tls-termination services // and one forward (passthrough) sharing listen 443. The public :443 moves into a // stream{} ssl_preread server that maps known SNIs to an internal loopback diff --git a/internal/types/types.go b/internal/types/types.go index 7a0e333..b413886 100644 --- a/internal/types/types.go +++ b/internal/types/types.go @@ -359,6 +359,24 @@ type PKIResourceSpec struct { Validity string `yaml:"validity,omitempty"` // optional CA validity (e.g. "10y") } +// RateLimitProfile is the resolved IP-based rate limit applied to an edge's public +// servers (ADR-0043). Rate limiting is a blanket ingress SECURITY measure, not per-route +// tuning: one limit is derived at deploy from the env's security.rate_limit block and +// stamped UNIFORMLY on every server of an edge (per-route / per-identity limits are a +// gateway-module concern — ADR-0044 — not this layer). It rides on the ingress-derived +// server structs (IngressRoute/IngressApp/IngressGateway) so the nginx renderer emits the +// shared-memory zone and the per-server limit directives without a wider signature; a nil +// pointer means the edge has no rate limiting (disabled, or opted out via `security: +// false`). Name is the nginx zone stem (a fixed constant, since the limit is uniform). +// Keying is always the client IP (ADR-0043); RPS/Burst drive limit_req (http only — +// ignored for an L4 forward), MaxConn drives limit_conn (http and stream). +type RateLimitProfile struct { + Name string // nginx zone stem (a fixed constant — the limit is edge-uniform) + RPS int // requests/second (limit_req rate); 0 disables request-rate limiting + Burst int // queued excess before a 429 (limit_req burst) + MaxConn int // max concurrent connections per client IP (limit_conn); 0 disables +} + // IngressRoute is one typed inbound routing entry the ingress proxy (nginx) // realizes, derived from one route of one service that references the ingress. // The ingress is the sole public entry point: nginx fronts the service on the @@ -385,6 +403,10 @@ type IngressRoute struct { Listen int // public port the ingress accepts traffic on Target int // backend port the service listens on Backend string // backend address nginx proxies to ("127.0.0.1" co-located; private IP cross-host) + // RateLimit is the resolved rate-limit profile for this route (nil = none). A + // tls-termination route uses RPS/Burst (limit_req) and MaxConn (limit_conn); a + // forward route uses only MaxConn (stream limit_conn — L4 has no request rate). + RateLimit *RateLimitProfile } // IngressApp is one static front-end (SPA) the ingress proxy (nginx) serves from @@ -401,6 +423,8 @@ type IngressApp struct { FQDN string // fully-qualified app domain (single SNI / ACME cert) Root string // on-host document root nginx serves (the `current` symlink) Spa bool // true -> try_files fallback to /index.html (SPA deep links) + // RateLimit is the resolved rate-limit profile for this app's server (nil = none). + RateLimit *RateLimitProfile } // IngressHealth is one service health endpoint the ingress proxy (nginx) surfaces, @@ -435,6 +459,9 @@ type IngressGateway struct { FQDN string // fully-qualified gateway domain (single SNI / ACME cert) Routes []IngressGatewayRoute HealthProbePaths []string // exact paths on the 443 server nginx answers 200 "ok" directly (edge liveness) + // RateLimit is the resolved rate-limit profile applied to this gateway's mesh-route + // locations (nil = none). The edge health-probe locations are never limited. + RateLimit *RateLimitProfile } // IngressGatewayRoute is one derived path route on the gateway server: daemon From 6448011d2f8c78b04fd06fa55e5191b0800cca08 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 19 Jul 2026 17:37:58 +0000 Subject: [PATCH 06/14] feat(security): env rate-limit authoring + wiring, validation, docs (ADR-0043) Wires the ADR-0043 blanket IP rate limit end to end (slice 1 complete): - types: SecurityConfig{Crowdsec, RateLimit} on EnvironmentVariables + ResolvedRateLimit(); Security *bool per-edge opt-out on IngressSpec/GatewaySpec. - schemas: `security` boolean added to ingress.json + gateway.json. - validate: checkSecurity enforces rate-limit bounds when enabled. - program: stampRateLimit resolves the single uniform limit and stamps it on every derived edge server whose ingress/gateway has not opted out; threaded through realizeIngress. - tests: checkSecurity bounds + stampRateLimit uniform/opt-out/disabled cases; fixed two existing realizeIngress test callers for the new parameter. - docs: variables.yaml security block + ingress/gateway `security:` opt-out. CrowdSec realization is slice 2 (the CrowdsecConfig fields are carried here so the authoring surface is stable). Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_011RB2WPLAzoDT1bjXrMfoXi --- internal/types/types.go | 59 +++++++++++++++ internal/validate/security.go | 36 ++++++++++ internal/validate/security_test.go | 53 ++++++++++++++ internal/validate/validate.go | 1 + program/gate_test.go | 2 +- program/global_test.go | 2 +- program/program.go | 68 +++++++++++++++++- program/ratelimit_test.go | 75 ++++++++++++++++++++ schemas/gateway.json | 2 +- schemas/ingress.json | 2 +- website/docs/configuration/variables-yaml.md | 30 ++++++++ website/docs/resources/gateway.md | 2 + website/docs/resources/ingress.md | 2 + 13 files changed, 328 insertions(+), 6 deletions(-) create mode 100644 internal/validate/security.go create mode 100644 internal/validate/security_test.go create mode 100644 program/ratelimit_test.go diff --git a/internal/types/types.go b/internal/types/types.go index b413886..d13e128 100644 --- a/internal/types/types.go +++ b/internal/types/types.go @@ -45,6 +45,10 @@ type IngressSpec struct { Container string `yaml:"container"` Host string `yaml:"host"` // FK -> compute resource name (same scope); reuses the host's provisioning/firewall/SSH HealthProbesPort int `yaml:"health_probes_port,omitempty"` // public port nginx exposes service health checks on (defaults to 81 when omitted); opened only when a referencing service declares its own health_probes_port + // Security opts this edge out of the env-level security tier (ADR-0043) when set to + // false: no rate limiting (and, slice 2, no CrowdSec). Nil/absent = the env policy + // applies. A *bool distinguishes "unset" from an explicit opt-out. + Security *bool `yaml:"security,omitempty"` } // AppSpec is one front-end (static SPA) resource — a bundle served from an @@ -84,6 +88,10 @@ type GatewaySpec struct { Services []string `yaml:"services"` // FKs -> services (same scope) exposed at the edge; the routing table is derived from their mesh.public_paths HealthProbesPort int `yaml:"health_probes_port,omitempty"` // public port the gateway host exposes its listed services' health checks on (plain HTTP, Host-demuxed by service FQDN; defaults to 81) HealthProbePaths []string `yaml:"health_probe_paths,omitempty"` // exact paths on the gateway's own 443 server that nginx answers 200 "ok" directly (edge liveness over the real TLS path); optional + // Security opts this gateway edge out of the env-level security tier (ADR-0043) when + // set to false: no rate limiting on its mesh routes (and, slice 2, no CrowdSec on its + // host). Nil/absent = the env policy applies. + Security *bool `yaml:"security,omitempty"` } // EffectiveHealthProbesPort is the gateway twin of the ingress method: the @@ -821,6 +829,57 @@ type EnvironmentVariables struct { BaseDomain string `yaml:"base_domain"` SSH SSHConfig `yaml:"ssh"` Observability ObservabilityConfig `yaml:"observability"` + Security SecurityConfig `yaml:"security"` +} + +// SecurityConfig is the optional env-level edge security tier (ADR-0043): a blanket IP +// rate limit and (slice 2) the CrowdSec agent, applied to every public edge (ingress + +// gateway) host. Both are off unless enabled. Like ObservabilityConfig it lives in +// variables.yaml and is struct-decoded — there is no JSON schema for the block. An edge +// opts the whole tier out with `security: false` on its ingress/gateway spec. +type SecurityConfig struct { + Crowdsec CrowdsecConfig `yaml:"crowdsec"` + RateLimit RateLimitConfig `yaml:"rate_limit"` +} + +// CrowdsecConfig toggles the per-edge-host CrowdSec agent + firewall bouncer (slice 2). +// Version pins the installed release (else the package DefaultVersion); Console opts in +// to dashboard enrollment (gated on a reserved secret). The fields are carried now so +// the authoring surface is stable; the realization lands in the CrowdSec slice. +type CrowdsecConfig struct { + Enabled bool `yaml:"enabled"` + Version string `yaml:"version,omitempty"` + Console bool `yaml:"console,omitempty"` +} + +// RateLimitConfig is the blanket IP rate limit (ADR-0043): one uniform limit applied to +// every public server on an edge — a security floor, not per-route tuning (per-route / +// per-identity limits are a gateway concern, ADR-0044). Keying is always the client IP. +type RateLimitConfig struct { + Enabled bool `yaml:"enabled"` + RequestsPerSecond int `yaml:"requests_per_second"` + Burst int `yaml:"burst"` + MaxConnections int `yaml:"max_connections"` +} + +// RateLimitZoneStem is the fixed nginx zone stem for the edge-uniform limit. It is a +// constant (not per-route) because one limit covers the whole edge (ADR-0043). +const RateLimitZoneStem = "edge" + +// ResolvedRateLimit returns the profile to stamp on an edge's public servers, or nil +// when rate limiting is disabled or has nothing to enforce (both limits zero). The +// program applies it uniformly to every server of every edge that has not opted out. +func (s SecurityConfig) ResolvedRateLimit() *RateLimitProfile { + rl := s.RateLimit + if !rl.Enabled || (rl.RequestsPerSecond <= 0 && rl.MaxConnections <= 0) { + return nil + } + return &RateLimitProfile{ + Name: RateLimitZoneStem, + RPS: rl.RequestsPerSecond, + Burst: rl.Burst, + MaxConn: rl.MaxConnections, + } } // ObservabilityConfig is the optional env-level observability block (ADR-0031, diff --git a/internal/validate/security.go b/internal/validate/security.go new file mode 100644 index 0000000..b400249 --- /dev/null +++ b/internal/validate/security.go @@ -0,0 +1,36 @@ +package validate + +import ( + "fmt" + "path/filepath" + + "github.com/wardnet/inforge/internal/types" +) + +// checkSecurity validates the env-level edge security tier authoring (ADR-0043): the +// blanket rate-limit bounds. It reads only the literal variables.yaml structure — no +// secrets, no environment. A disabled or absent block is a no-op. The per-edge +// `security:` opt-out is a plain boolean enforced by the ingress/gateway JSON schema, so +// it needs nothing here. +func checkSecurity(r *reporter, dir, env string, sec types.SecurityConfig) { + rl := sec.RateLimit + if !rl.Enabled { + return + } + var errs []string + if rl.RequestsPerSecond < 0 { + errs = append(errs, fmt.Sprintf("security.rate_limit.requests_per_second must be >= 0, got %d", rl.RequestsPerSecond)) + } + if rl.Burst < 0 { + errs = append(errs, fmt.Sprintf("security.rate_limit.burst must be >= 0, got %d", rl.Burst)) + } + if rl.MaxConnections < 0 { + errs = append(errs, fmt.Sprintf("security.rate_limit.max_connections must be >= 0, got %d", rl.MaxConnections)) + } + if rl.RequestsPerSecond == 0 && rl.MaxConnections == 0 { + errs = append(errs, "security.rate_limit.enabled is true but both requests_per_second and max_connections are 0 — nothing to limit (set one, or enabled: false)") + } + if len(errs) > 0 { + r.fail(filepath.Join(dir, env, "variables.yaml"), errs...) + } +} diff --git a/internal/validate/security_test.go b/internal/validate/security_test.go new file mode 100644 index 0000000..1341078 --- /dev/null +++ b/internal/validate/security_test.go @@ -0,0 +1,53 @@ +package validate + +import ( + "testing" + + "github.com/stretchr/testify/assert" + + "github.com/wardnet/inforge/internal/types" +) + +func secCfg(rl types.RateLimitConfig) types.SecurityConfig { + return types.SecurityConfig{RateLimit: rl} +} + +func TestCheckSecurityDisabledIsNoop(t *testing.T) { + r := &reporter{} + // Disabled, even with nonsense bounds, is not validated. + checkSecurity(r, t.TempDir(), "prd", secCfg(types.RateLimitConfig{RequestsPerSecond: -5})) + assert.False(t, r.failed) +} + +func TestCheckSecurityValid(t *testing.T) { + r := &reporter{} + checkSecurity(r, t.TempDir(), "prd", secCfg(types.RateLimitConfig{ + Enabled: true, RequestsPerSecond: 20, Burst: 40, MaxConnections: 40, + })) + assert.False(t, r.failed) +} + +func TestCheckSecurityRejectsNegativeBounds(t *testing.T) { + for _, rl := range []types.RateLimitConfig{ + {Enabled: true, RequestsPerSecond: -1}, + {Enabled: true, RequestsPerSecond: 10, Burst: -1}, + {Enabled: true, MaxConnections: -1}, + } { + r := &reporter{} + checkSecurity(r, t.TempDir(), "prd", secCfg(rl)) + assert.True(t, r.failed, "expected failure for %+v", rl) + } +} + +func TestCheckSecurityRejectsEnabledButNothingToLimit(t *testing.T) { + r := &reporter{} + checkSecurity(r, t.TempDir(), "prd", secCfg(types.RateLimitConfig{Enabled: true})) + assert.True(t, r.failed) +} + +// A connection-only limit (rps 0, max_connections > 0) is valid — limit_conn alone. +func TestCheckSecurityConnectionOnlyIsValid(t *testing.T) { + r := &reporter{} + checkSecurity(r, t.TempDir(), "prd", secCfg(types.RateLimitConfig{Enabled: true, MaxConnections: 100})) + assert.False(t, r.failed) +} diff --git a/internal/validate/validate.go b/internal/validate/validate.go index 9aa36d3..899b612 100644 --- a/internal/validate/validate.go +++ b/internal/validate/validate.go @@ -521,6 +521,7 @@ func ValidateResources(env, dir string, defaults types.ProviderDefaults, opts .. globalBase := filepath.Join(base, "global") checkVariables(r, vars, filepath.Join(base, "variables.yaml")) checkObservability(r, dir, env, vars.Observability) + checkSecurity(r, dir, env, vars.Security) checkRegionsFile(r, regionTable, global, filepath.Join(base, "regions.yaml")) // Validate the global slice in a GLOBAL-ONLY context (globalRefs nil): its FK diff --git a/program/gate_test.go b/program/gate_test.go index 10d6f8c..e89c965 100644 --- a/program/gate_test.go +++ b/program/gate_test.go @@ -245,7 +245,7 @@ func TestTLSAndServiceShareOneGate(t *testing.T) { "bridge-01": {PublicIP: pulumi.String("1.2.3.4").ToStringOutput()}, } gates := map[string]pulumi.Resource{} - if err := realizeIngress(ctx, reg, res, computeOut, gates, nil, "priv", "prd", "use1", "example.com", "", types.ProviderDefaults{}); err != nil { + if err := realizeIngress(ctx, reg, res, computeOut, gates, nil, "priv", "prd", "use1", "example.com", "", types.SecurityConfig{}, types.ProviderDefaults{}); err != nil { return err } return provisionServices(ctx, res, computeOut, map[string]serviceMaterial{}, gates, "priv", "prd", "us-east-1", "use1", "example.com", "1.2.3", nil) diff --git a/program/global_test.go b/program/global_test.go index df3388f..7b7c0ef 100644 --- a/program/global_test.go +++ b/program/global_test.go @@ -238,7 +238,7 @@ func TestGlobalServiceRealizesRegionLess(t *testing.T) { } gates := map[string]pulumi.Resource{} // Empty slug throughout — exactly how the scopes loop drives the global slice. - if err := realizeIngress(ctx, reg, res, computeOut, gates, nil, "priv", "prd", "", "wardnet.network", "", types.ProviderDefaults{}); err != nil { + if err := realizeIngress(ctx, reg, res, computeOut, gates, nil, "priv", "prd", "", "wardnet.network", "", types.SecurityConfig{}, types.ProviderDefaults{}); err != nil { return err } return provisionServices(ctx, res, computeOut, map[string]serviceMaterial{}, gates, "priv", "prd", globalScope, "", "wardnet.network", "1.2.3", nil) diff --git a/program/program.go b/program/program.go index 231db53..e95cf01 100644 --- a/program/program.go +++ b/program/program.go @@ -387,7 +387,7 @@ func Run(ctx *pulumi.Context) error { if err != nil { return err } - if err := realizeIngress(ctx, sc.reg, sc.res, computeOutputs[sc.key], gates, appSeeds, vars.SSH.DeployPrivateKey, env, sc.slug, vars.BaseDomain, ephemeralSlug, providerDefaults); err != nil { + if err := realizeIngress(ctx, sc.reg, sc.res, computeOutputs[sc.key], gates, appSeeds, vars.SSH.DeployPrivateKey, env, sc.slug, vars.BaseDomain, ephemeralSlug, vars.Security, providerDefaults); err != nil { return err } // The east-west mesh proxy (ADR-0032) materializes on every host running a @@ -1714,7 +1714,7 @@ func serviceDeprovisionScript(svc types.ServiceSpec) string { // (cross-host, using the backend's PrivateIP). FQDNs are env-scoped here so the // provider stays a pure installer. Hosts are realized in sorted order so the // resource graph is stable across runs. -func realizeIngress(ctx *pulumi.Context, reg registry.ProviderRegistry, res types.Resources, computeOut map[string]types.ComputeOutputs, gates map[string]pulumi.Resource, appSeeds map[string][]pulumi.Resource, deployPrivateKey, env, slug, baseDomain, ephemeralSlug string, defaults types.ProviderDefaults) error { +func realizeIngress(ctx *pulumi.Context, reg registry.ProviderRegistry, res types.Resources, computeOut map[string]types.ComputeOutputs, gates map[string]pulumi.Resource, appSeeds map[string][]pulumi.Resource, deployPrivateKey, env, slug, baseDomain, ephemeralSlug string, sec types.SecurityConfig, defaults types.ProviderDefaults) error { canonical := naming.CanonicalComputeKeys(res.Compute) routesByHostKey, _, err := ingressRoutesByHost(res, canonical, env, slug, baseDomain) if err != nil { @@ -1722,6 +1722,10 @@ func realizeIngress(ctx *pulumi.Context, reg registry.ProviderRegistry, res type } appsByHostKey := ingressAppsByHost(res, canonical, slug, baseDomain, ephemeralSlug) gatewaysByHostKey := gatewaysByHost(res, canonical, slug, baseDomain, ephemeralSlug) + // Stamp the env-uniform IP rate limit (ADR-0043) onto every edge server whose edge + // has not opted out (`security: false`). Rate limiting is a blanket security floor, + // so the same profile lands on every route/app/gateway — there is no per-route knob. + stampRateLimit(sec, res, routesByHostKey, appsByHostKey, gatewaysByHostKey) // Resolve the ingress-tier services once and feed both derivations below — the // health entries and the cross-host backend set — so the service list is walked a // single time per host realization. @@ -1819,6 +1823,66 @@ func realizeIngress(ctx *pulumi.Context, reg registry.ProviderRegistry, res type return nil } +// stampRateLimit resolves the env's single blanket IP rate limit (ADR-0043) and stamps +// it onto every derived edge server whose owning edge has not opted out via +// `security: false`. It is a no-op when rate limiting is disabled. A route's edge is the +// ingress its service references; an app's edge is the ingress it targets; a gateway is +// its own edge. Rate limiting is a uniform security floor, so a stamped server always +// gets the same profile — there is deliberately no per-route variation. +func stampRateLimit(sec types.SecurityConfig, res types.Resources, routes map[string][]types.IngressRoute, apps map[string][]types.IngressApp, gateways map[string][]types.IngressGateway) { + rl := sec.ResolvedRateLimit() + if rl == nil { + return + } + svcIngress := fkByName(res.Service, func(s types.ServiceSpec) (string, string) { return s.Name, s.Ingress }) + appIngress := fkByName(res.App, func(a types.AppSpec) (string, string) { return a.Name, a.Ingress }) + ingOptOut := securityOptOut(res.Ingress, func(i types.IngressSpec) (string, *bool) { return i.Name, i.Security }) + gwOptOut := securityOptOut(res.Gateway, func(g types.GatewaySpec) (string, *bool) { return g.Name, g.Security }) + for hk := range routes { + for i := range routes[hk] { + if !ingOptOut[svcIngress[routes[hk][i].Service]] { + routes[hk][i].RateLimit = rl + } + } + } + for hk := range apps { + for i := range apps[hk] { + if !ingOptOut[appIngress[apps[hk][i].Name]] { + apps[hk][i].RateLimit = rl + } + } + } + for hk := range gateways { + for i := range gateways[hk] { + if !gwOptOut[gateways[hk][i].Name] { + gateways[hk][i].RateLimit = rl + } + } + } +} + +// fkByName builds a name -> foreign-key map from a spec slice, given a projector that +// returns each spec's (name, fk). Used to resolve a derived server back to its edge. +func fkByName[T any](specs []T, project func(T) (string, string)) map[string]string { + m := make(map[string]string, len(specs)) + for _, s := range specs { + name, fk := project(s) + m[name] = fk + } + return m +} + +// securityOptOut builds a name -> opted-out map from a spec slice: a resource is opted +// out only when its `security:` flag is explicitly false (nil/absent = env policy applies). +func securityOptOut[T any](specs []T, project func(T) (string, *bool)) map[string]bool { + m := make(map[string]bool, len(specs)) + for _, s := range specs { + name, flag := project(s) + m[name] = flag != nil && !*flag + } + return m +} + // ingressHostsByName maps each ingress resource name to the canonical specKey of // its host. Validation guarantees the host FK resolves; an unresolved one is // skipped defensively (the caller then finds no routes for it). diff --git a/program/ratelimit_test.go b/program/ratelimit_test.go new file mode 100644 index 0000000..e38b434 --- /dev/null +++ b/program/ratelimit_test.go @@ -0,0 +1,75 @@ +package program + +import ( + "testing" + + "github.com/stretchr/testify/assert" + + "github.com/wardnet/inforge/internal/types" +) + +// stampFixture builds a minimal edge: one service+app behind ingress "main", plus a +// gateway "gw", each surfaced as one derived server on host "h". +func stampFixture() (types.Resources, map[string][]types.IngressRoute, map[string][]types.IngressApp, map[string][]types.IngressGateway) { + res := types.Resources{ + Service: []types.ServiceSpec{{Name: "api", Ingress: "main"}}, + App: []types.AppSpec{{Name: "web", Ingress: "main"}}, + Ingress: []types.IngressSpec{{Name: "main"}}, + Gateway: []types.GatewaySpec{{Name: "gw"}}, + } + routes := map[string][]types.IngressRoute{"h": {{Service: "api"}}} + apps := map[string][]types.IngressApp{"h": {{Name: "web"}}} + gws := map[string][]types.IngressGateway{"h": {{Name: "gw"}}} + return res, routes, apps, gws +} + +func enabledSec() types.SecurityConfig { + return types.SecurityConfig{RateLimit: types.RateLimitConfig{ + Enabled: true, RequestsPerSecond: 20, Burst: 40, MaxConnections: 40, + }} +} + +func TestStampRateLimitAppliesUniformly(t *testing.T) { + res, routes, apps, gws := stampFixture() + stampRateLimit(enabledSec(), res, routes, apps, gws) + assert.NotNil(t, routes["h"][0].RateLimit) + assert.NotNil(t, apps["h"][0].RateLimit) + assert.NotNil(t, gws["h"][0].RateLimit) + // Same fixed-stem profile everywhere. + assert.Equal(t, types.RateLimitZoneStem, routes["h"][0].RateLimit.Name) + assert.Equal(t, 20, routes["h"][0].RateLimit.RPS) + assert.Equal(t, 40, gws["h"][0].RateLimit.MaxConn) +} + +func TestStampRateLimitDisabledStampsNothing(t *testing.T) { + res, routes, apps, gws := stampFixture() + stampRateLimit(types.SecurityConfig{}, res, routes, apps, gws) + assert.Nil(t, routes["h"][0].RateLimit) + assert.Nil(t, apps["h"][0].RateLimit) + assert.Nil(t, gws["h"][0].RateLimit) +} + +func TestStampRateLimitIngressOptOut(t *testing.T) { + res, routes, apps, gws := stampFixture() + res.Ingress[0].Security = ptrBool(false) // this ingress opts out + stampRateLimit(enabledSec(), res, routes, apps, gws) + assert.Nil(t, routes["h"][0].RateLimit, "route on an opted-out ingress is not limited") + assert.Nil(t, apps["h"][0].RateLimit, "app on an opted-out ingress is not limited") + assert.NotNil(t, gws["h"][0].RateLimit, "the gateway is a separate edge and stays limited") +} + +func TestStampRateLimitGatewayOptOut(t *testing.T) { + res, routes, apps, gws := stampFixture() + res.Gateway[0].Security = ptrBool(false) + stampRateLimit(enabledSec(), res, routes, apps, gws) + assert.NotNil(t, routes["h"][0].RateLimit) + assert.Nil(t, gws["h"][0].RateLimit, "an opted-out gateway is not limited") +} + +// security: true is an explicit opt-IN (identical to absent), never an opt-out. +func TestStampRateLimitExplicitTrueStillApplies(t *testing.T) { + res, routes, apps, gws := stampFixture() + res.Ingress[0].Security = ptrBool(true) + stampRateLimit(enabledSec(), res, routes, apps, gws) + assert.NotNil(t, routes["h"][0].RateLimit) +} diff --git a/schemas/gateway.json b/schemas/gateway.json index 229ceb2..1205689 100644 --- a/schemas/gateway.json +++ b/schemas/gateway.json @@ -1 +1 @@ -{"$schema":"https://json-schema.org/draft/2020-12/schema","$id":"gateway.json","title":"Gateway (north-south daemon API gateway) resource","type":"object","additionalProperties":false,"required":["name","container","host","pki","subdomain","services"],"properties":{"name":{"type":"string","pattern":"^[a-z][a-z0-9_-]*$"},"container":{"type":"string","minLength":1,"description":"Logical container/grouping the gateway belongs to (same field as other resources)."},"host":{"type":"string","minLength":1,"description":"Name of the compute resource (in the same scope) this gateway runs on. The gateway reuses the host's provisioning/firewall/SSH and inherits its provider; it declares no provider of its own."},"pki":{"type":"string","minLength":1,"description":"Name of the two-tier (mesh) PKI in pki.enc.yaml the gateway's client leaf (/gateway) mints from. Every listed service must declare the same pki (a callee only trusts callers chaining to its own mesh)."},"subdomain":{"type":"string","minLength":1,"description":"Public subdomain daemons connect to. The fully-qualified domain is composed at realization from the scope and base domain."},"services":{"type":"array","minItems":1,"description":"names of the services (same scope) exposed at the internet edge. The gateway's routing table is DERIVED from each listed service's mesh.public_paths (ADR-0034); a request matching no public glob is answered 404 (JSON) at the edge. Each listed service must permit the gateway (list \"gateway\" in its mesh.allowed_services) and declare at least one public path.","items":{"type":"string","pattern":"^[a-z][a-z0-9_-]*$"}},"health_probes_port":{"type":"integer","minimum":1,"maximum":65535,"description":"public port the gateway host exposes its listed services' health checks on (plain HTTP, Host-demuxed by service FQDN). Defaults to 81. Opened only when a listed service declares its own health_probes_port."},"health_probe_paths":{"type":"array","minItems":1,"description":"exact paths on the gateway's own 443 server that nginx answers 200 \"ok\" directly — edge liveness over the real daemon path (DNS+cert+TLS). Must not overlap any listed service's public_paths.","items":{"type":"string","pattern":"^/"}}}} +{"$schema":"https://json-schema.org/draft/2020-12/schema","$id":"gateway.json","title":"Gateway (north-south daemon API gateway) resource","type":"object","additionalProperties":false,"required":["name","container","host","pki","subdomain","services"],"properties":{"name":{"type":"string","pattern":"^[a-z][a-z0-9_-]*$"},"container":{"type":"string","minLength":1,"description":"Logical container/grouping the gateway belongs to (same field as other resources)."},"host":{"type":"string","minLength":1,"description":"Name of the compute resource (in the same scope) this gateway runs on. The gateway reuses the host's provisioning/firewall/SSH and inherits its provider; it declares no provider of its own."},"pki":{"type":"string","minLength":1,"description":"Name of the two-tier (mesh) PKI in pki.enc.yaml the gateway's client leaf (/gateway) mints from. Every listed service must declare the same pki (a callee only trusts callers chaining to its own mesh)."},"subdomain":{"type":"string","minLength":1,"description":"Public subdomain daemons connect to. The fully-qualified domain is composed at realization from the scope and base domain."},"services":{"type":"array","minItems":1,"description":"names of the services (same scope) exposed at the internet edge. The gateway's routing table is DERIVED from each listed service's mesh.public_paths (ADR-0034); a request matching no public glob is answered 404 (JSON) at the edge. Each listed service must permit the gateway (list \"gateway\" in its mesh.allowed_services) and declare at least one public path.","items":{"type":"string","pattern":"^[a-z][a-z0-9_-]*$"}},"health_probes_port":{"type":"integer","minimum":1,"maximum":65535,"description":"public port the gateway host exposes its listed services' health checks on (plain HTTP, Host-demuxed by service FQDN). Defaults to 81. Opened only when a listed service declares its own health_probes_port."},"health_probe_paths":{"type":"array","minItems":1,"description":"exact paths on the gateway's own 443 server that nginx answers 200 \"ok\" directly — edge liveness over the real daemon path (DNS+cert+TLS). Must not overlap any listed service's public_paths.","items":{"type":"string","pattern":"^/"}},"security":{"type":"boolean","description":"Opt this gateway edge out of the env-level security tier (ADR-0043) when false: no rate limiting (and no CrowdSec). Absent = the env security policy applies."}}} diff --git a/schemas/ingress.json b/schemas/ingress.json index 258c39a..828e953 100644 --- a/schemas/ingress.json +++ b/schemas/ingress.json @@ -1 +1 @@ -{"$schema":"https://json-schema.org/draft/2020-12/schema","$id":"ingress.json","title":"Ingress resource","type":"object","additionalProperties":false,"required":["name","container","host"],"properties":{"name":{"type":"string","pattern":"^[a-z][a-z0-9_-]*$"},"container":{"type":"string","minLength":1,"description":"Logical container/grouping the ingress belongs to (same field as other resources)."},"host":{"type":"string","minLength":1,"description":"Name of the compute resource (in the same scope) this ingress runs on. The ingress reuses the host's provisioning/firewall/SSH and inherits its provider; it declares no provider of its own."},"health_probes_port":{"type":"integer","minimum":1,"maximum":65535,"description":"Public port nginx exposes service health checks on (defaults to 81 when omitted). Opened to 0.0.0.0/0 only when at least one referencing service declares its own health_probes_port. Must not equal a route listen port on this host, nor 80."}}} +{"$schema":"https://json-schema.org/draft/2020-12/schema","$id":"ingress.json","title":"Ingress resource","type":"object","additionalProperties":false,"required":["name","container","host"],"properties":{"name":{"type":"string","pattern":"^[a-z][a-z0-9_-]*$"},"container":{"type":"string","minLength":1,"description":"Logical container/grouping the ingress belongs to (same field as other resources)."},"host":{"type":"string","minLength":1,"description":"Name of the compute resource (in the same scope) this ingress runs on. The ingress reuses the host's provisioning/firewall/SSH and inherits its provider; it declares no provider of its own."},"health_probes_port":{"type":"integer","minimum":1,"maximum":65535,"description":"Public port nginx exposes service health checks on (defaults to 81 when omitted). Opened to 0.0.0.0/0 only when at least one referencing service declares its own health_probes_port. Must not equal a route listen port on this host, nor 80."},"security":{"type":"boolean","description":"Opt this edge out of the env-level security tier (ADR-0043) when false: no rate limiting (and no CrowdSec). Absent = the env security policy applies."}}} diff --git a/website/docs/configuration/variables-yaml.md b/website/docs/configuration/variables-yaml.md index 714256e..4b367a8 100644 --- a/website/docs/configuration/variables-yaml.md +++ b/website/docs/configuration/variables-yaml.md @@ -20,6 +20,13 @@ ssh: # required when using compute observability: # optional — Grafana Cloud integration otlp_endpoint: https://otlp-gateway-....grafana.net/otlp # host/DB metrics collector grafana_url: https://myorg.grafana.net # dashboards + alerts target + +security: # optional — edge security tier (public ingress + gateway hosts) + rate_limit: + enabled: true # one blanket IP-based limit on every public edge server + requests_per_second: 20 + burst: 40 + max_connections: 40 ``` ## Fields @@ -67,6 +74,23 @@ are [reserved secrets](/cli/secret) in `secrets.enc.yaml`, never committed here. (from `observability/notifications.yaml`) that built-in alerts and any alert omitting `profile:` route through. Required once alerts are managed. +### `security` + +Optional edge security tier applied to the public [ingress](/resources/ingress) and +[gateway](/resources/gateway) hosts. Off unless enabled. An individual ingress or gateway +opts the whole tier out with `security: false` on its own spec. + +- `rate_limit` — a single **blanket, IP-based** rate limit applied uniformly to every + public server on every edge. It is a security floor, not per-route tuning: the same + limit covers all routes, apps, and gateway paths (per-route / per-identity limits are a + future gateway concern). Requests over the limit are answered `429`. Fields: + - `enabled` — turn rate limiting on (default off). + - `requests_per_second` — sustained per-client-IP request rate (`limit_req`). + - `burst` — how many excess requests may queue before a `429` is returned. + - `max_connections` — concurrent connections allowed per client IP (`limit_conn`). + + Health-check and ACME (certificate) endpoints are never rate-limited. + ## Example ```yaml title="resources/prd/variables.yaml" @@ -80,4 +104,10 @@ observability: default_profile: prod # notification profile for built-in + un-profiled alerts # built_in_dashboards: false # opt out of the generated dashboards # built_in_alerts: false # opt out of the generated alert rules +security: + rate_limit: + enabled: true + requests_per_second: 20 + burst: 40 + max_connections: 40 ``` diff --git a/website/docs/resources/gateway.md b/website/docs/resources/gateway.md index e67080a..25f733d 100644 --- a/website/docs/resources/gateway.md +++ b/website/docs/resources/gateway.md @@ -27,6 +27,7 @@ services: # the services exposed at the edge — routing is DERI - tunneller health_probe_paths: # optional — edge liveness: nginx answers 200 "ok" on the 443 server - /livez +# security: false # optional — opt this edge out of the env security tier (rate limiting) ``` ## Fields @@ -41,6 +42,7 @@ health_probe_paths: # optional — edge liveness: nginx answers 200 "ok" o | `services` | array | Yes (min 1) | The services (same scope) exposed at the internet edge. The routing table is **derived**: one nginx regex location per (listed service, [`mesh.public_paths`](./service#path-level-exposure) glob), target named in `X-Mesh-Target`. A request matching no public glob is answered at the edge with an `application/json` `404` body `{"error":"not_found"}` — it never traverses to any service. | | `health_probes_port` | int | No | Public **plain-HTTP** port the gateway host exposes its listed services' health checks on (default `81`), demuxed by request `Host` (the service's FQDN) — the gateway twin of the ingress [health port](./ingress#health-probes). Opened to the internet only when a listed service declares its own [`health_probes_port`](./service#health-probes). Must not be `80` or `443`, and must **match** a co-hosted ingress's health port (one public health port per host). | | `health_probe_paths` | array | No | Exact paths on the gateway's **own `443` server** that nginx answers `200 "ok"` directly — edge liveness proving the real daemon path (DNS + cert + TLS) without touching any backend. `inforge validate` rejects a path that any listed service's public glob claims. | +| `security` | bool | No | Set `false` to opt this gateway out of the env-level [security tier](/configuration/variables-yaml#security) — its mesh-route locations get no rate limiting. Absent/`true` = the env policy applies. Health-probe locations are never rate-limited regardless. | **A gateway is a scope singleton** — at most one per scope: it is the scope's one public daemon edge. It may only list services in the **same scope**. diff --git a/website/docs/resources/ingress.md b/website/docs/resources/ingress.md index 5550b90..26735b1 100644 --- a/website/docs/resources/ingress.md +++ b/website/docs/resources/ingress.md @@ -31,6 +31,7 @@ name: edge # required — ingress name (unique within the scope) container: bridge # required — grouping label host: bridge # required — FK -> compute name (same scope); the vm nginx runs on health_probes_port: 81 # optional — public port for service health checks (default 81) +# security: false # optional — opt this edge out of the env security tier (rate limiting) ``` | Field | Type | Required | Description | @@ -39,6 +40,7 @@ health_probes_port: 81 # optional — public port for service health checks | `container` | string | Yes | Grouping label (tags, like other resources). | | `host` | string | Yes | **Name** of the Compute resource (same scope) this ingress runs on. Must be a single-instance `vm`. A `global/` prefix is rejected — a global ingress is declared in the global slice itself. **At most one ingress per scope** — the ingress's DNS name (`ingress[.].`, see below) is scope-singular, which also implies one per host. | | `health_probes_port` | int | No | Public port nginx exposes service [health checks](#health-probes) on. Defaults to `81`. Opened to the internet only when a referencing service declares its own `health_probes_port`. Must not be `80` or a route `listen` port on this host. | +| `security` | bool | No | Set `false` to opt this ingress out of the env-level [security tier](/configuration/variables-yaml#security) — its servers get no rate limiting. Absent/`true` = the env policy applies. | ## What it serves From 62c017644b3d25d2a2046e29d77470284d684a00 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 19 Jul 2026 17:51:35 +0000 Subject: [PATCH 07/14] docs(adr): make CrowdSec observable via the existing otelcol pipeline MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Record the slice-2 observability decision: CrowdSec fails silently (unreadable nginx logs -> parses/bans nothing while cscli still looks healthy), so slice 2 wires its Prometheus metrics (agent 127.0.0.1:6060 + firewall-bouncer loopback endpoint) into the otelcol collector via a prometheus receiver, gated on the same observability.otlp_endpoint and exported through the existing otlphttp -> Grafana Cloud path — mirroring the ADR-0037 postgresql receiver. Acquisition health (cs_reader_hits > 0) is the load-bearing "actually working" signal; the deploy also asserts cscli lapi status + bouncer registration at apply time. Dashboards + two alerts (acquisition stalled, bouncer not pulling) are slice 2b. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_011RB2WPLAzoDT1bjXrMfoXi --- ...43-edge-security-rate-limiting-and-crowdsec.md | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/docs/adr/0043-edge-security-rate-limiting-and-crowdsec.md b/docs/adr/0043-edge-security-rate-limiting-and-crowdsec.md index 2eaaa41..809da67 100644 --- a/docs/adr/0043-edge-security-rate-limiting-and-crowdsec.md +++ b/docs/adr/0043-edge-security-rate-limiting-and-crowdsec.md @@ -152,6 +152,21 @@ needs no secret. VM" expansion (sshd brute-force coverage) is a config flip, not a rewrite. - **No new open ports.** The local API is loopback (`127.0.0.1:8080`); the community blocklist pull is outbound. `types.FirewallPorts` is unchanged. +- **Observable through the existing otelcol pipeline — CrowdSec's silent failure mode + demands it.** CrowdSec fails quietly: if it cannot read the nginx logs (rotation, path, + permissions) it parses nothing and bans nothing while every `cscli` command still looks + healthy. So slice 2 wires its telemetry into the pipeline inforge already runs (ADR-0031/ + 0037): enable the agent's Prometheus endpoint (`127.0.0.1:6060`) and the firewall + bouncer's loopback metrics endpoint, and add a `prometheus` receiver to the edge host's + otelcol collector — gated on the same `observability.otlp_endpoint`, exported through the + same `otlphttp` → Grafana Cloud, tagged with the ADR-0030 resource attributes, exactly as + the `postgresql` receiver is added on a cluster host (ADR-0037). The load-bearing signal + is **acquisition health** (`cs_reader_hits` / parse rate > 0 = CrowdSec is actually seeing + traffic); the bouncer's last-pull timestamp catches enforcement drift. The deploy also + asserts liveness at apply time — `cscli lapi status` and the bouncer appearing in `cscli + bouncers list` — so a CrowdSec that did not come up fails the deploy rather than sitting + dead. Dashboards and the two built-in alerts ("acquisition stalled", "bouncer not + pulling") are a thin slice 2b once metrics flow. ## Wiring (implementation shape) From 8bd190aa89057da2dacaa45fb5e33c2a88559395 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 19 Jul 2026 18:00:22 +0000 Subject: [PATCH 08/14] feat(crowdsec): pure install/config rendering package (ADR-0043, slice 2 pt1) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit New stdlib-only, Pulumi-free internal/crowdsec (like internal/nginx and internal/otelcol): renders the idempotent host shell + config for the CrowdSec agent + nftables firewall bouncer, ready for the program to wire onto edge hosts. - InstallScript: adds the CrowdSec packagecloud apt repo with a pinned signed-by keyring (the internal/nginx nginx.org-repo pattern), installs crowdsec + crowdsec-firewall-bouncer-nftables, enables both units. Optional agent-only version pin (the bouncer versions independently). - ConfigScript: nginx acquisition drop-in + prometheus config overlay, hub collections (nginx/base-http/http-cve), CAPI (community blocklist) registration guarded for idempotency, agent reload. - BouncerScript: idempotent bouncer-key registration (delete-then-add) + the bouncer .local overlay (nftables/DROP + loopback prometheus); key base64- transported into the file (never plaintext in the config write). - AssertScript: deploy-time liveness gate (cscli lapi status + bouncer registered) — CrowdSec otherwise fails silently. - Metrics endpoints (agent 127.0.0.1:6060, bouncer :60601) exposed for the slice-2 otelcol prometheus receiver. Also corrects the ADR install decision: packagecloud apt repo, not a checksummed .deb download, since CrowdSec ships its packages (with deps) through that repo. Program wiring (provisionCrowdsec) and the otelcol receiver are the next commits. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_011RB2WPLAzoDT1bjXrMfoXi --- ...dge-security-rate-limiting-and-crowdsec.md | 16 +- internal/crowdsec/install.go | 148 ++++++++++++++++++ internal/crowdsec/install_test.go | 89 +++++++++++ internal/crowdsec/paths.go | 58 +++++++ internal/crowdsec/shell.go | 40 +++++ 5 files changed, 345 insertions(+), 6 deletions(-) create mode 100644 internal/crowdsec/install.go create mode 100644 internal/crowdsec/install_test.go create mode 100644 internal/crowdsec/paths.go create mode 100644 internal/crowdsec/shell.go diff --git a/docs/adr/0043-edge-security-rate-limiting-and-crowdsec.md b/docs/adr/0043-edge-security-rate-limiting-and-crowdsec.md index 809da67..e139d2a 100644 --- a/docs/adr/0043-edge-security-rate-limiting-and-crowdsec.md +++ b/docs/adr/0043-edge-security-rate-limiting-and-crowdsec.md @@ -117,12 +117,16 @@ needs no secret. nftables from log analysis and touches no nginx config. inforge manages **no host packet filter today** (the firewall is the Hetzner *cloud* firewall via the provider), so the bouncer owns its own nft table with zero conflict — confirmed against the codebase. -- **Off-the-shelf `.deb`, checksum-verified download — mirrors ADR-0031 (otelcol).** New - pure package `internal/crowdsec` (stdlib-only, Pulumi-free, like `internal/otelcol` and - `internal/nginx`) renders the idempotent install shell (download the version-pinned - `crowdsec` + `crowdsec-firewall-bouncer-nftables` `.deb`s, verify sha256, `apt-get - install` the local files) and the on-host config. A `DefaultVersion` is pinned and - bumped deliberately. +- **Signed apt repo (packagecloud) — mirrors the nginx.org repo pattern, not the otelcol + `.deb` download.** CrowdSec ships `crowdsec` + `crowdsec-firewall-bouncer-nftables` + (with dependencies) through the `packagecloud.io/crowdsec/crowdsec` apt repository, not + as standalone checksummed `.deb`s, so the robust install adds that repo with a pinned + `signed-by` keyring exactly as `internal/nginx/install.go` adds the nginx.org repo, then + `apt-get install`s both packages. New pure package `internal/crowdsec` (stdlib-only, + Pulumi-free, like `internal/otelcol`/`internal/nginx`) renders that idempotent install + shell and the on-host config. Version pinning is optional and applies to the agent only — + the bouncer carries its own independent version scheme (0.0.x vs the agent's 1.x), so a + single pin cannot cover both; unpinned installs the repo's current compatible pair. - **Community blocklist on by default; console optional.** `cscli capi register` (no secret) enrolls the machine to the Central API and pulls the crowd-sourced blocklist — the pre-emptive known-bad-IP protection that is CrowdSec's edge over fail2ban. Console diff --git a/internal/crowdsec/install.go b/internal/crowdsec/install.go new file mode 100644 index 0000000..4264acc --- /dev/null +++ b/internal/crowdsec/install.go @@ -0,0 +1,148 @@ +package crowdsec + +import ( + "fmt" + "strings" + + "github.com/wardnet/inforge/internal/aptlock" +) + +// InstallScript adds the CrowdSec packagecloud apt repository (a pinned signed-by +// keyring, the same shape as internal/nginx's nginx.org repo) and installs the agent + +// nftables firewall bouncer. It is idempotent: the keyring is written once, the repo +// list is rewritten in place, and `apt-get install` is a no-op when both packages are +// already current. agentVersion pins the AGENT only (empty = the repo's current pair) — +// the bouncer versions independently (0.0.x vs the agent's 1.x), so one pin cannot cover +// both. The first apt call is an update (through aptlock, retrying a held lists lock) so +// a fresh image never installs against a missing index. +func InstallScript(agentVersion string) string { + agentPkg := AgentPackage + if agentVersion != "" { + agentPkg = fmt.Sprintf("%s=%s", AgentPackage, agentVersion) + } + return fmt.Sprintf(`#!/usr/bin/env bash +set -euo pipefail + +%[1]s +sudo DEBIAN_FRONTEND=noninteractive apt-get -o DPkg::Lock::Timeout=300 install -y curl ca-certificates gnupg + +. /etc/os-release + +# CrowdSec packagecloud signing key (armored .asc used directly via signed-by, no dearmor — +# which would fail on a host with no controlling tty). Download to a temp then install into +# place so a curl failure never leaves a partial keyring the presence guard would trust. +if [ ! -f %[2]s ]; then + asc=$(mktemp) + curl -fsSL %[3]s -o "$asc" + sudo install -m 0644 "$asc" %[2]s + rm -f "$asc" +fi +echo "deb [signed-by=%[2]s] https://packagecloud.io/crowdsec/crowdsec/${ID}/ ${VERSION_CODENAME} main" \ + | sudo tee %[4]s >/dev/null + +%[1]s +sudo DEBIAN_FRONTEND=noninteractive apt-get -o DPkg::Lock::Timeout=300 install -y %[5]s %[6]s + +sudo systemctl enable %[7]s %[8]s +`, aptlock.UpdateCmd(), KeyringPath, RepoKeyURL, RepoListPath, agentPkg, BouncerPackage, AgentService, BouncerService) +} + +// ConfigScript writes the agent overlays (prometheus config.yaml.local + the nginx +// acquisition drop-in), installs the hub collections, registers to the Central API for +// the community blocklist, and reloads the agent. Every step is idempotent: the files are +// rewritten, `cscli collections install` is a no-op/upgrade when present, and CAPI +// registration is skipped when already registered. Collection install and CAPI +// registration reach the CrowdSec hub/central API (a deploy-time network dependency). +func ConfigScript() string { + return strings.Join([]string{ + "set -e", + writeFileScript(AgentConfigLocal, agentPrometheusLocal(), "0644", "root:root"), + writeFileScript(AcquisPath, nginxAcquis(), "0644", "root:root"), + fmt.Sprintf("sudo cscli collections install %s", strings.Join(Collections, " ")), + // Register to the Central API (the community blocklist) unless already registered. + "sudo cscli capi status >/dev/null 2>&1 || sudo cscli capi register", + // Reload to pick up the new acquisition + collections; restart if reload is unsupported. + fmt.Sprintf("sudo systemctl reload %s 2>/dev/null || sudo systemctl restart %s", AgentService, AgentService), + }, "\n") +} + +// BouncerScript registers the inforge-owned bouncer key with the local API and writes the +// bouncer's `.local` overlay (nftables mode, the LAPI URL, the api_key, DROP action, and +// its loopback prometheus endpoint), then restarts the bouncer. apiKey must be +// YAML-safe (alphanumeric) — the program mints it with random.RandomPassword(special=off) +// — since it is written unquoted into the overlay. `cscli bouncers add` errors when the +// name already exists, so a delete-then-add makes re-deploys idempotent; the same key is +// rewritten into the overlay in this pass. The key does appear briefly in argv on +// `cscli bouncers add --key`; it is a host-local capability credential and the host is +// already root-trusted, an accepted minor exposure (cscli offers no stdin key input). +func BouncerScript(apiKey string) string { + return strings.Join([]string{ + "set -e", + fmt.Sprintf("sudo cscli bouncers delete %s >/dev/null 2>&1 || true", shQuote(BouncerName)), + fmt.Sprintf("sudo cscli bouncers add %s --key %s", shQuote(BouncerName), shQuote(apiKey)), + writeFileScript(BouncerConfigLocal, bouncerLocal(apiKey), "0600", "root:root"), + fmt.Sprintf("sudo systemctl restart %s", BouncerService), + }, "\n") +} + +// EnrollScript enrolls the agent to the CrowdSec console with the given token and +// restarts the agent. It is optional (gated on the reserved crowdsec_enroll secret). The +// enroll is a no-op-safe best effort: an already-enrolled instance is left as-is. +func EnrollScript(token string) string { + return strings.Join([]string{ + "set -e", + fmt.Sprintf("sudo cscli console enroll %s || true", shQuote(token)), + fmt.Sprintf("sudo systemctl restart %s", AgentService), + }, "\n") +} + +// AssertScript fails the deploy when CrowdSec did not come up: the agent's local API must +// answer, and the inforge bouncer must be registered with it. It is the deploy-time +// liveness gate (CrowdSec otherwise fails silently — ADR-0043). +func AssertScript() string { + return strings.Join([]string{ + "set -e", + "sudo cscli lapi status", + fmt.Sprintf("sudo cscli bouncers list -o json | grep -q %s", shQuote(BouncerName)), + }, "\n") +} + +// agentPrometheusLocal is the agent config.yaml overlay enabling the loopback prometheus +// endpoint the host collector scrapes. +func agentPrometheusLocal() string { + return fmt.Sprintf(`# Managed by inforge (ADR-0043) — CrowdSec agent prometheus overlay. +prometheus: + enabled: true + level: full + listen_addr: %s + listen_port: %d +`, AgentMetricsAddr, AgentMetricsPort) +} + +// nginxAcquis is the acquisition drop-in pointing CrowdSec at the ingress nginx logs. +func nginxAcquis() string { + return `# Managed by inforge (ADR-0043) — CrowdSec nginx log acquisition. +source: file +filenames: + - /var/log/nginx/access.log + - /var/log/nginx/error.log +labels: + type: nginx +` +} + +// bouncerLocal is the firewall bouncer's config overlay. apiKey is written unquoted, so +// the caller must supply a YAML-safe (alphanumeric) key. +func bouncerLocal(apiKey string) string { + return fmt.Sprintf(`# Managed by inforge (ADR-0043) — CrowdSec firewall bouncer overlay. +mode: nftables +api_url: %s +api_key: %s +deny_action: DROP +disable_ipv6: false +prometheus: + enabled: true + listen_addr: %s + listen_port: %d +`, LAPIURL, apiKey, BouncerMetricsAddr, BouncerMetricsPort) +} diff --git a/internal/crowdsec/install_test.go b/internal/crowdsec/install_test.go new file mode 100644 index 0000000..05bfa30 --- /dev/null +++ b/internal/crowdsec/install_test.go @@ -0,0 +1,89 @@ +package crowdsec + +import ( + "strings" + "testing" + + "github.com/stretchr/testify/assert" +) + +func TestInstallScriptAddsRepoAndInstallsBothPackages(t *testing.T) { + s := InstallScript("") + assert.Contains(t, s, "signed-by="+KeyringPath) + assert.Contains(t, s, "packagecloud.io/crowdsec/crowdsec/${ID}/") + assert.Contains(t, s, "install -y "+AgentPackage+" "+BouncerPackage) + assert.Contains(t, s, "systemctl enable "+AgentService+" "+BouncerService) + // No version pin: the bare agent package, not agent=. + assert.NotContains(t, s, AgentPackage+"=") +} + +func TestInstallScriptPinsAgentVersionOnly(t *testing.T) { + s := InstallScript("1.6.4") + assert.Contains(t, s, "install -y "+AgentPackage+"=1.6.4 "+BouncerPackage) + // The bouncer is never pinned (independent version scheme). + assert.NotContains(t, s, BouncerPackage+"=") +} + +func TestConfigScriptInstallsCollectionsAndRegistersCAPI(t *testing.T) { + s := ConfigScript() + assert.Contains(t, s, "cscli collections install crowdsecurity/nginx crowdsecurity/base-http-scenarios crowdsecurity/http-cve") + // CAPI registration is guarded so a re-deploy of an already-registered host is a no-op. + assert.Contains(t, s, "cscli capi status >/dev/null 2>&1 || sudo cscli capi register") + // The acquisition + prometheus overlays are written (base64-transported, so their + // paths appear in the mv target). + assert.Contains(t, s, AcquisPath) + assert.Contains(t, s, AgentConfigLocal) +} + +func TestBouncerScriptIsIdempotentAndRestarts(t *testing.T) { + s := BouncerScript("KEY123abc") + assert.Contains(t, s, "cscli bouncers delete 'inforge-fw'") + assert.Contains(t, s, "cscli bouncers add 'inforge-fw' --key 'KEY123abc'") + assert.Contains(t, s, BouncerConfigLocal) + assert.Contains(t, s, "systemctl restart "+BouncerService) +} + +func TestAssertScriptChecksLAPIAndBouncer(t *testing.T) { + s := AssertScript() + assert.Contains(t, s, "cscli lapi status") + assert.Contains(t, s, "cscli bouncers list -o json | grep -q 'inforge-fw'") +} + +func TestBouncerLocalRendersDeterministicYAML(t *testing.T) { + const want = `# Managed by inforge (ADR-0043) — CrowdSec firewall bouncer overlay. +mode: nftables +api_url: http://127.0.0.1:8080/ +api_key: KEY123abc +deny_action: DROP +disable_ipv6: false +prometheus: + enabled: true + listen_addr: 127.0.0.1 + listen_port: 60601 +` + assert.Equal(t, want, bouncerLocal("KEY123abc")) +} + +func TestNginxAcquisPointsAtNginxLogs(t *testing.T) { + a := nginxAcquis() + assert.Contains(t, a, "/var/log/nginx/access.log") + assert.Contains(t, a, "/var/log/nginx/error.log") + assert.Contains(t, a, "type: nginx") +} + +func TestAgentPrometheusOverlayBindsLoopback(t *testing.T) { + p := agentPrometheusLocal() + assert.Contains(t, p, "enabled: true") + assert.Contains(t, p, "listen_addr: 127.0.0.1") + assert.Contains(t, p, "listen_port: 6060") +} + +// The bouncer api_key must never appear as plaintext in the config-write step — it is +// base64-transported through the shell (only the cscli argv carries it, a documented +// accepted exposure). +func TestBouncerKeyNotPlaintextInConfigWrite(t *testing.T) { + s := BouncerScript("SUPERSECRETKEY") + // The overlay content is base64'd, so the raw key appears exactly once in the whole + // script: the `cscli bouncers add --key` argv (a documented accepted exposure). + assert.Equal(t, 1, strings.Count(s, "SUPERSECRETKEY")) +} diff --git a/internal/crowdsec/paths.go b/internal/crowdsec/paths.go new file mode 100644 index 0000000..0fa0e60 --- /dev/null +++ b/internal/crowdsec/paths.go @@ -0,0 +1,58 @@ +// Package crowdsec renders the on-host install shell and configuration for the CrowdSec +// agent + nftables firewall bouncer that inforge installs on public edge hosts +// (ADR-0043). It is deploy-side only, stdlib-only, and holds no provider/Pulumi +// dependencies — the program wires it to edge hosts, exactly as internal/nginx and +// internal/otelcol are wired. The rendering is pure (config/params in, shell string out) +// and deterministic, so the same inputs always produce the same bytes. +package crowdsec + +const ( + // AgentPackage / BouncerPackage are the apt packages installed from the CrowdSec + // packagecloud repository. BouncerService/AgentService are their systemd units. + AgentPackage = "crowdsec" + BouncerPackage = "crowdsec-firewall-bouncer-nftables" + AgentService = "crowdsec" + BouncerService = "crowdsec-firewall-bouncer" + + // LAPIURL is the loopback local API the agent serves and the bouncer authenticates to. + LAPIURL = "http://127.0.0.1:8080/" + + // BouncerName is the stable inforge-owned bouncer identity registered with the LAPI. + BouncerName = "inforge-fw" + + // On-host config paths. inforge writes `.local` overlays and an acquis.d drop-in + // rather than editing the packaged files, so a package upgrade never reverts our + // config (CrowdSec merges *.local over the base config, and reads acquis.d/*.yaml). + AgentConfigLocal = "/etc/crowdsec/config.yaml.local" + AcquisPath = "/etc/crowdsec/acquis.d/nginx.yaml" + BouncerConfigLocal = "/etc/crowdsec/bouncers/crowdsec-firewall-bouncer.yaml.local" + + // Prometheus metrics endpoints (loopback only), scraped by the host otelcol collector + // via a prometheus receiver (ADR-0043, mirroring the ADR-0037 postgresql receiver). + AgentMetricsAddr = "127.0.0.1" + AgentMetricsPort = 6060 + BouncerMetricsAddr = "127.0.0.1" + BouncerMetricsPort = 60601 + + // packagecloud apt repo, added with a pinned signed-by keyring exactly as + // internal/nginx adds the nginx.org repo (an armored .asc used directly, no dearmor). + RepoKeyURL = "https://packagecloud.io/crowdsec/crowdsec/gpgkey" + KeyringPath = "/usr/share/keyrings/crowdsec_crowdsec-archive-keyring.asc" + RepoListPath = "/etc/apt/sources.list.d/crowdsec.list" + + // ReservedNamespace/EnrollSecretKey locate the optional console enrollment token in + // the env's secrets.enc.yaml — an inforge reserved secret (ADR-0043), mirroring + // otelcol.AuthSecretNamespace/AuthSecretKey. The community blocklist needs no secret. + ReservedNamespace = "security" + EnrollSecretKey = "crowdsec_enroll" // #nosec G101 -- a lookup key name, not a credential +) + +// Collections are the CrowdSec hub items installed on an nginx edge: nginx access/error +// log parsing + scenarios, the generic HTTP attack scenarios, and known-CVE probing. +// They are installed imperatively by cscli, which fetches them from the CrowdSec hub — +// a deliberate network dependency at deploy time (ADR-0043). +var Collections = []string{ + "crowdsecurity/nginx", + "crowdsecurity/base-http-scenarios", + "crowdsecurity/http-cve", +} diff --git a/internal/crowdsec/shell.go b/internal/crowdsec/shell.go new file mode 100644 index 0000000..0e46d8c --- /dev/null +++ b/internal/crowdsec/shell.go @@ -0,0 +1,40 @@ +package crowdsec + +import ( + "encoding/base64" + "fmt" + "path" + "strings" +) + +// shQuote single-quotes s for safe interpolation into a POSIX shell command, escaping +// any embedded single quote as the standard '\'' sequence. The package renders host +// shell but stays Pulumi-free (pure, like internal/nginx and internal/otelcol), so it +// carries its own minimal quoting rather than importing the remote helpers. +func shQuote(s string) string { + return "'" + strings.ReplaceAll(s, "'", `'\''`) + "'" +} + +// base64Encode returns the standard base64 of s, for transporting arbitrary bytes +// (including a secret) through a shell command without exposing them in argv. +func base64Encode(s string) string { + return base64.StdEncoding.EncodeToString([]byte(s)) +} + +// writeFileScript renders sudo shell that writes content to absPath with the given mode +// and owner, creating the parent dir. content is base64-encoded for transport so +// arbitrary bytes (including a secret) survive intact and never appear in argv as +// plaintext. The write stages to a temp then atomic-mv's into place (safe on coreutils 9 +// where `install /dev/stdin ` fails), mirroring internal/otelcol. +func writeFileScript(absPath, content, mode, owner string) string { + enc := base64Encode(content) + dir := path.Dir(absPath) + return strings.Join([]string{ + fmt.Sprintf("sudo install -d -m 0755 %s", shQuote(dir)), + fmt.Sprintf(`tmp=$(sudo mktemp %s)`, shQuote(dir+"/.inforge-write.XXXXXX")), + fmt.Sprintf(`printf %%s %s | base64 -d | sudo tee "$tmp" >/dev/null`, shQuote(enc)), + fmt.Sprintf(`sudo chmod %s "$tmp"`, shQuote(mode)), + fmt.Sprintf(`sudo chown %s "$tmp"`, shQuote(owner)), + fmt.Sprintf(`sudo mv "$tmp" %s`, shQuote(absPath)), + }, "\n") +} From 8f750a3a046dd42a174766c25d3e846e41b00dcd Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 19 Jul 2026 18:41:55 +0000 Subject: [PATCH 09/14] feat(crowdsec): provisionCrowdsec pass over edge hosts (ADR-0043, slice 2 pt2) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Wire the CrowdSec agent + firewall bouncer onto public-edge hosts. Mirrors provisionObservability: per-host loop gated on the shared cloud-init gates, secret woven through ApplyT + safeTrigger. - provisionCrowdsec: gated on security.crowdsec.enabled; targets the ingress ∪ gateway host set (crowdsecEdgeHosts), honoring the per-edge `security: false` opt-out. Per host: install -> agent config -> bouncer -> optional console enroll -> deploy-time liveness assertion, chained by DependsOn. - Bouncer API key minted per host via random.RandomPassword (alphanumeric, so YAML-safe), built inside ApplyT so it is encrypted in state; Triggers use safeTrigger, never the raw key. - Optional console enrollment token decrypted once per deploy from the reserved security/crowdsec_enroll secret (mirrors the OTLP auth decrypt), marked secret. - Called in the scope loop after provisionObservability so the collector that will scrape CrowdSec's metrics is already present. - Tests: crowdsecEdgeHosts includes only ingress/gateway hosts and honors the opt-out. Remaining: the otelcol prometheus receiver scraping CrowdSec's loopback metrics endpoints (slice 2 pt3). Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_011RB2WPLAzoDT1bjXrMfoXi --- program/crowdsec_test.go | 49 ++++++++++++ program/program.go | 166 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 215 insertions(+) create mode 100644 program/crowdsec_test.go diff --git a/program/crowdsec_test.go b/program/crowdsec_test.go new file mode 100644 index 0000000..9ce37a0 --- /dev/null +++ b/program/crowdsec_test.go @@ -0,0 +1,49 @@ +package program + +import ( + "testing" + + "github.com/stretchr/testify/assert" + + "github.com/wardnet/inforge/internal/naming" + "github.com/wardnet/inforge/internal/types" +) + +func edgeFixture() types.Resources { + return types.Resources{ + Compute: []types.ComputeSpec{ + {Name: "edge", Kind: "vm", InstanceCount: 1}, + {Name: "gwhost", Kind: "vm", InstanceCount: 1}, + {Name: "worker", Kind: "vm", InstanceCount: 1}, + }, + Ingress: []types.IngressSpec{{Name: "main", Host: "edge"}}, + Gateway: []types.GatewaySpec{{Name: "gw", Host: "gwhost"}}, + } +} + +func TestCrowdsecEdgeHostsIncludesIngressAndGatewayOnly(t *testing.T) { + res := edgeFixture() + canonical := naming.CanonicalComputeKeys(res.Compute) + edge := crowdsecEdgeHosts(res, canonical) + assert.True(t, edge[canonical["edge"]], "ingress host is an edge") + assert.True(t, edge[canonical["gwhost"]], "gateway host is an edge") + assert.False(t, edge[canonical["worker"]], "a plain worker gets no CrowdSec") + assert.Len(t, edge, 2) +} + +func TestCrowdsecEdgeHostsHonorsOptOut(t *testing.T) { + res := edgeFixture() + res.Ingress[0].Security = ptrBool(false) // this ingress opts out + canonical := naming.CanonicalComputeKeys(res.Compute) + edge := crowdsecEdgeHosts(res, canonical) + assert.False(t, edge[canonical["edge"]], "opted-out ingress host is excluded") + assert.True(t, edge[canonical["gwhost"]], "the gateway host is a separate edge and stays in") +} + +func TestCrowdsecEdgeHostsExplicitTrueStillIncluded(t *testing.T) { + res := edgeFixture() + res.Gateway[0].Security = ptrBool(true) // explicit opt-in == absent + canonical := naming.CanonicalComputeKeys(res.Compute) + edge := crowdsecEdgeHosts(res, canonical) + assert.True(t, edge[canonical["gwhost"]]) +} diff --git a/program/program.go b/program/program.go index e95cf01..ea5f6e7 100644 --- a/program/program.go +++ b/program/program.go @@ -20,6 +20,7 @@ import ( "github.com/pulumi/pulumi/sdk/v3/go/pulumi/config" "github.com/wardnet/inforge/internal/agent" "github.com/wardnet/inforge/internal/app" + "github.com/wardnet/inforge/internal/crowdsec" "github.com/wardnet/inforge/internal/dbbackup" "github.com/wardnet/inforge/internal/grant" "github.com/wardnet/inforge/internal/hostpaths" @@ -308,6 +309,22 @@ func Run(ctx *pulumi.Context) error { obsAuthB64 = pulumi.ToSecret(pulumi.String(base64.StdEncoding.EncodeToString([]byte(authRaw)))).(pulumi.StringOutput) } + // Optional CrowdSec console enrollment token (ADR-0043): decrypted once per deploy + // only when console enrollment is requested, and marked secret so it is encrypted in + // Pulumi state. Enabled with no token is a hard misconfiguration (fail at up, skipped + // in preview). The community blocklist needs no secret, so this is console-only. + var csEnrollToken pulumi.StringOutput + if vars.Security.Crowdsec.Enabled && vars.Security.Crowdsec.Console { + tokRaw, err := decryptReservedSecret(dir, srcEnv, crowdsec.ReservedNamespace, crowdsec.EnrollSecretKey, ctx.DryRun()) + if err != nil { + return err + } + if tokRaw == "" && !ctx.DryRun() { + return fmt.Errorf("crowdsec: console enrollment is enabled but secrets.enc.yaml is missing or has no %s/%s token — run `inforge secret set %s %s %s --reserved` and commit the store", crowdsec.ReservedNamespace, crowdsec.EnrollSecretKey, srcEnv, crowdsec.ReservedNamespace, crowdsec.EnrollSecretKey) + } + csEnrollToken = pulumi.ToSecret(pulumi.String(tokRaw)).(pulumi.StringOutput) + } + // Env-level Grafana dashboards (ADR-0038): when grafana_url is configured, this env's // built-in dashboards are pushed via the pulumiverse/grafana provider. Org-global, so // realized ONCE here (not per scope) and env-prefixed. A no-op when grafana_url is empty. @@ -427,6 +444,13 @@ func Run(ctx *pulumi.Context) error { if err := provisionObservability(ctx, sc.res, computeOutputs[sc.key], databaseOutputs[sc.key], gates, dbHostTails, vars.Observability, obsAuthB64, vars.SSH.DeployPrivateKey, env, sc.slug); err != nil { return err } + // The edge security tier (ADR-0043): install CrowdSec + the nftables firewall + // bouncer on every public-edge (ingress/gateway) host, gated on + // security.crowdsec.enabled and the per-edge opt-out. Runs after observability so + // the collector (which will scrape CrowdSec's metrics) is already installed. + if err := provisionCrowdsec(ctx, sc.res, computeOutputs[sc.key], gates, vars.Security, csEnrollToken, vars.SSH.DeployPrivateKey, env, sc.slug); err != nil { + return err + } } return nil @@ -877,6 +901,148 @@ func provisionObservability(ctx *pulumi.Context, res types.Resources, computeOut return nil } +// provisionCrowdsec installs the CrowdSec agent + nftables firewall bouncer on every +// public-edge host in this scope (ADR-0043), gated on security.crowdsec.enabled: it is a +// no-op when disabled or when no non-opted-out edge exists. Unlike the observability +// collector (every VM), CrowdSec lands only on ingress/gateway hosts — where public HTTP +// traffic terminates. Each host runs install -> agent config -> bouncer -> (optional +// console enroll) -> liveness assertion, chained by DependsOn and gated on the host's +// cloud-init readiness (shared gates map). The bouncer API key is minted per host +// (random, alphanumeric so it is YAML-safe) and woven through ApplyT so it is encrypted +// in state and never lands in the unencrypted Triggers array — the deliverServiceSecrets +// / monitor-role invariant. enrollToken is used only when console enrollment is on. +func provisionCrowdsec(ctx *pulumi.Context, res types.Resources, computeOut map[string]types.ComputeOutputs, gates map[string]pulumi.Resource, sec types.SecurityConfig, enrollToken pulumi.StringOutput, deployPrivateKey, env, slug string) error { + if !sec.Crowdsec.Enabled { + return nil + } + edge := crowdsecEdgeHosts(res, naming.CanonicalComputeKeys(res.Compute)) + if len(edge) == 0 { + return nil + } + deployUserByCompute := naming.DeployUsersByHost(res.Compute) + for _, hostKey := range sortedKeys(computeOut) { + if !edge[hostKey] { + continue + } + host := computeOut[hostKey] + deployUser := deployUserByCompute[hostKey] + if !ctx.DryRun() { + if deployUser == "" { + return fmt.Errorf("crowdsec: host %q has no deploy_user; inforge needs one to SSH and install CrowdSec", hostKey) + } + if deployPrivateKey == "" { + return fmt.Errorf("crowdsec: no deploy private key configured (set the deploy_private_key stack config or INFORGE_DEPLOY_PRIVATE_KEY)") + } + } + gate, err := cloudInitGate(ctx, gates, hostKey, host, deployPrivateKey, env, slug) + if err != nil { + return err + } + conn := iremote.Connection(host.PublicIP, deployUser, deployPrivateKey) + name := naming.Resource(env, slug, "crowdsec", hostKey) + + // 1. Install the agent + bouncer from the packagecloud repo. First per-host command, + // so it carries the cloud-init gate; later steps chain off it. + install := crowdsec.InstallScript(sec.Crowdsec.Version) + installCmd, err := remote.NewCommand(ctx, name+"-install", &remote.CommandArgs{ + Connection: conn, + Create: pulumi.String(install), + Update: pulumi.String(install), + Triggers: pulumi.Array{pulumi.String(install)}, + }, pulumi.DependsOn([]pulumi.Resource{gate})) + if err != nil { + return fmt.Errorf("crowdsec: host %q: install: %w", hostKey, err) + } + + // 2. Agent config: nginx acquisition + prometheus overlays, hub collections, CAPI + // (community blocklist) registration, reload. + cfg := crowdsec.ConfigScript() + cfgCmd, err := remote.NewCommand(ctx, name+"-config", &remote.CommandArgs{ + Connection: conn, + Create: pulumi.String(cfg), + Update: pulumi.String(cfg), + Triggers: pulumi.Array{pulumi.String(cfg)}, + }, pulumi.DependsOn([]pulumi.Resource{installCmd})) + if err != nil { + return fmt.Errorf("crowdsec: host %q: configure agent: %w", hostKey, err) + } + + // 3. Bouncer: mint an inforge-owned API key (alphanumeric, YAML-safe), register it, + // write the overlay, restart. The key is secret, so the whole script is built + // inside an ApplyT and its Triggers use safeTrigger (never the raw key). + key, err := random.NewRandomPassword(ctx, name+"-bouncer-key", &random.RandomPasswordArgs{ + Length: pulumi.Int(48), + Special: pulumi.Bool(false), + }) + if err != nil { + return fmt.Errorf("crowdsec: host %q: bouncer key: %w", hostKey, err) + } + bouncerScript := key.Result.ApplyT(func(k string) string { return crowdsec.BouncerScript(k) }).(pulumi.StringOutput) + bouncerCmd, err := remote.NewCommand(ctx, name+"-bouncer", &remote.CommandArgs{ + Connection: conn, + Create: bouncerScript, + Update: bouncerScript, + Triggers: pulumi.Array{safeTrigger(bouncerScript)}, + }, pulumi.DependsOn([]pulumi.Resource{cfgCmd})) + if err != nil { + return fmt.Errorf("crowdsec: host %q: configure bouncer: %w", hostKey, err) + } + lastDep := pulumi.Resource(bouncerCmd) + + // 4. Optional console enrollment (gated on the reserved crowdsec_enroll secret). + if sec.Crowdsec.Console { + enrollScript := enrollToken.ApplyT(func(tok string) string { return crowdsec.EnrollScript(tok) }).(pulumi.StringOutput) + enrollCmd, err := remote.NewCommand(ctx, name+"-enroll", &remote.CommandArgs{ + Connection: conn, + Create: enrollScript, + Update: enrollScript, + Triggers: pulumi.Array{safeTrigger(enrollScript)}, + }, pulumi.DependsOn([]pulumi.Resource{bouncerCmd})) + if err != nil { + return fmt.Errorf("crowdsec: host %q: console enroll: %w", hostKey, err) + } + lastDep = enrollCmd + } + + // 5. Deploy-time liveness gate: fail the deploy if the agent's local API is not up + // or the bouncer did not register (CrowdSec otherwise fails silently — ADR-0043). + assert := crowdsec.AssertScript() + if _, err := remote.NewCommand(ctx, name+"-assert", &remote.CommandArgs{ + Connection: conn, + Create: pulumi.String(assert), + Update: pulumi.String(assert), + Triggers: pulumi.Array{safeTrigger(bouncerScript)}, + }, pulumi.DependsOn([]pulumi.Resource{lastDep})); err != nil { + return fmt.Errorf("crowdsec: host %q: liveness assertion: %w", hostKey, err) + } + } + return nil +} + +// crowdsecEdgeHosts returns the set of canonical host keys that run a public edge (an +// ingress or a gateway) and have not opted out of the security tier via `security: false` +// (ADR-0043). A host is included if at least one non-opted-out edge resource lands on it. +func crowdsecEdgeHosts(res types.Resources, canonical map[string]string) map[string]bool { + edge := map[string]bool{} + for _, ing := range res.Ingress { + if ing.Security != nil && !*ing.Security { + continue + } + if hk, ok := canonical[ing.Host]; ok { + edge[hk] = true + } + } + for _, gw := range res.Gateway { + if gw.Security != nil && !*gw.Security { + continue + } + if hk, ok := canonical[gw.Host]; ok { + edge[hk] = true + } + } + return edge +} + // pgMaxIdentifierLen is Postgres's NAMEDATALEN-1: the longest identifier the server // stores. A longer name is SILENTLY TRUNCATED to this many bytes at CREATE ROLE (even // double-quoted) — see checkDBRoleNames for why that must be rejected rather than From 1ffc2c455785cd8136a1c4cd9f7faab4a4615aa5 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 19 Jul 2026 18:46:22 +0000 Subject: [PATCH 10/14] feat(crowdsec): scrape CrowdSec metrics via the otelcol collector (ADR-0043, slice 2 pt3) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Make CrowdSec observable through the existing pipeline (ADR-0031/0037): the edge host's collector scrapes CrowdSec's loopback Prometheus endpoints and exports them to Grafana Cloud on the same otlphttp path, correlating on host.id. - otelcol.Render: optional variadic crowdsecScrape []string adds a prometheus receiver + its own job/pipeline scraping the given loopback endpoints, with a resource processor stamping service.name=CrowdsecServiceName + per-node HostID + the shared ADR-0030 identity (mirrors the postgresql receiver). Empty leaves the config byte-identical, so the 10 host-metrics-only callers are untouched. - provisionObservability: gains the security config, computes the CrowdSec edge set, and passes the agent (127.0.0.1:6060) + bouncer (:60601) endpoints for edge hosts. Gated on the same otlp_endpoint — CrowdSec metrics ride the observability pipeline or not at all. - Tests: receiver present with endpoints + valid YAML; absent by default. This completes slice 2 (CrowdSec): install/config package, provisionCrowdsec edge pass, and metrics. Dashboards + the two alerts (acquisition stalled, bouncer not pulling) remain as slice 2b. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_011RB2WPLAzoDT1bjXrMfoXi --- internal/otelcol/config.go | 40 ++++++++++++++++++++++++++++++++- internal/otelcol/config_test.go | 22 ++++++++++++++++++ program/dbrole_test.go | 4 ++-- program/program.go | 20 ++++++++++++++--- 4 files changed, 80 insertions(+), 6 deletions(-) diff --git a/internal/otelcol/config.go b/internal/otelcol/config.go index 15f3efd..aadca59 100644 --- a/internal/otelcol/config.go +++ b/internal/otelcol/config.go @@ -23,6 +23,13 @@ const ServiceName = "wardnet-host-metrics" // last of which becomes meaningful once a cluster spans multiple hosts. const DBServiceName = "wardnet-db-metrics" +// CrowdsecServiceName is the OTel service.name (Prometheus job) stamped on CrowdSec +// metrics scraped on edge hosts (ADR-0043) — a distinct job from host and DB metrics so +// CrowdSec dashboards/alerts scope cleanly, still correlating with them on host.id. The +// load-bearing signal is acquisition health (parse rate > 0 = CrowdSec is actually seeing +// nginx traffic), which catches its silent-failure mode. +const CrowdsecServiceName = "wardnet-crowdsec-metrics" + // CollectionInterval is how often the hostmetrics receiver scrapes. const CollectionInterval = "60s" @@ -104,7 +111,11 @@ func identityAttrs(attrs Attributes) []map[string]any { // to a host-metrics-only collector. Each pg target gets its own receiver + metrics // pipeline whose resource processor stamps service.name=DBServiceName, // service.instance.id=HostID, db.cluster.name=, plus the shared identity. -func Render(endpoint string, attrs Attributes, pg []PostgresTarget) (string, error) { +// crowdsecScrape, when non-empty, adds a prometheus receiver scraping those loopback +// "host:port" endpoints (the CrowdSec agent + firewall bouncer, ADR-0043) on their own +// job/pipeline; empty leaves the config byte-identical. It is variadic so the many +// host-metrics-only callers stay unchanged. +func Render(endpoint string, attrs Attributes, pg []PostgresTarget, crowdsecScrape ...string) (string, error) { if endpoint == "" { return "", fmt.Errorf("otelcol: empty OTLP endpoint") } @@ -212,6 +223,33 @@ func Render(endpoint string, attrs Attributes, pg []PostgresTarget) (string, err } } + // CrowdSec metrics (ADR-0043): one prometheus receiver scraping the agent + firewall + // bouncer loopback endpoints on an edge host, on its own job + pipeline. The resource + // processor overwrites the scrape-derived job/instance with a fixed CrowdSec job name + // and the per-node HostID (like the pg targets) plus the shared identity, so CrowdSec + // health correlates with host + app telemetry on host.id. Empty targets = no receiver. + if len(crowdsecScrape) > 0 { + receivers["prometheus/crowdsec"] = map[string]any{ + "config": map[string]any{ + "scrape_configs": []map[string]any{{ + "job_name": "crowdsec", + "scrape_interval": CollectionInterval, + "static_configs": []map[string]any{{"targets": crowdsecScrape}}, + }}, + }, + } + csResourceAttrs := append([]map[string]any{ + upsert("service.name", CrowdsecServiceName), + upsert("service.instance.id", attrs.HostID), + }, identityAttrs(attrs)...) + processors["resource/crowdsec"] = map[string]any{"attributes": csResourceAttrs} + pipelines["metrics/crowdsec"] = map[string]any{ + "receivers": []string{"prometheus/crowdsec"}, + "processors": []string{"resource/crowdsec", "batch"}, + "exporters": []string{"otlphttp"}, + } + } + cfg := map[string]any{ "receivers": receivers, "processors": processors, diff --git a/internal/otelcol/config_test.go b/internal/otelcol/config_test.go index 8b059ef..d7cf843 100644 --- a/internal/otelcol/config_test.go +++ b/internal/otelcol/config_test.go @@ -251,3 +251,25 @@ func TestRenderNoPostgresIsHostOnly(t *testing.T) { assert.NotContains(t, a, "transform/db-labels") assert.NotContains(t, a, "postgresql") } + +func TestRenderCrowdsecReceiver(t *testing.T) { + out, err := Render("https://otlp.example/v1", fullAttrs(), nil, "127.0.0.1:6060", "127.0.0.1:60601") + require.NoError(t, err) + // A dedicated prometheus receiver + job scraping both loopback endpoints, its own + // pipeline, and the CrowdSec job name overwriting the scrape-derived one. + assert.Contains(t, out, "prometheus/crowdsec") + assert.Contains(t, out, "job_name: crowdsec") + assert.Contains(t, out, "127.0.0.1:6060") + assert.Contains(t, out, "127.0.0.1:60601") + assert.Contains(t, out, "metrics/crowdsec") + assert.Contains(t, out, CrowdsecServiceName) + // The rendered config must still be valid YAML. + var parsed map[string]any + require.NoError(t, yaml.Unmarshal([]byte(out), &parsed)) +} + +func TestRenderNoCrowdsecByDefault(t *testing.T) { + out, err := Render("https://otlp.example/v1", fullAttrs(), nil) + require.NoError(t, err) + assert.NotContains(t, out, "crowdsec") +} diff --git a/program/dbrole_test.go b/program/dbrole_test.go index 4514577..45ff087 100644 --- a/program/dbrole_test.go +++ b/program/dbrole_test.go @@ -56,7 +56,7 @@ func TestObservabilityTriggersAreNotSecret(t *testing.T) { err := pulumi.RunErr(func(ctx *pulumi.Context) error { authB64 := pulumi.ToSecret(pulumi.String("dXNlcjpwdw==")).(pulumi.StringOutput) obs := types.ObservabilityConfig{OTLPEndpoint: "https://otlp.example.com/v1/metrics"} - return provisionObservability(ctx, res, computeOut, nil, map[string]pulumi.Resource{}, nil, obs, authB64, "priv", "prd", "use1") + return provisionObservability(ctx, res, computeOut, nil, map[string]pulumi.Resource{}, nil, obs, types.SecurityConfig{}, authB64, "priv", "prd", "use1") }, pulumi.WithMocks("project", "stack", mocks)) require.NoError(t, err) @@ -109,7 +109,7 @@ func TestMonitorMintWaitsOnServiceRoleMints(t *testing.T) { dbOut := map[string]types.DatabaseOutputs{"app": {RoleProvisioner: p}} authB64 := pulumi.ToSecret(pulumi.String("dXNlcjpwdw==")).(pulumi.StringOutput) obs := types.ObservabilityConfig{OTLPEndpoint: "https://otlp.example.com/v1/metrics"} - return provisionObservability(ctx, res, computeOut, dbOut, map[string]pulumi.Resource{}, nil, obs, authB64, "priv", "prd", "use1") + return provisionObservability(ctx, res, computeOut, dbOut, map[string]pulumi.Resource{}, nil, obs, types.SecurityConfig{}, authB64, "priv", "prd", "use1") }, pulumi.WithMocks("project", "stack", mocks)) require.NoError(t, err) diff --git a/program/program.go b/program/program.go index ea5f6e7..b56c9b1 100644 --- a/program/program.go +++ b/program/program.go @@ -441,7 +441,7 @@ func Run(ctx *pulumi.Context) error { if err := provisionServices(ctx, sc.res, computeOutputs[sc.key], serviceSecrets, gates, vars.SSH.DeployPrivateKey, env, sc.key, sc.slug, vars.BaseDomain, inforgeVersion, meshReloads); err != nil { return err } - if err := provisionObservability(ctx, sc.res, computeOutputs[sc.key], databaseOutputs[sc.key], gates, dbHostTails, vars.Observability, obsAuthB64, vars.SSH.DeployPrivateKey, env, sc.slug); err != nil { + if err := provisionObservability(ctx, sc.res, computeOutputs[sc.key], databaseOutputs[sc.key], gates, dbHostTails, vars.Observability, vars.Security, obsAuthB64, vars.SSH.DeployPrivateKey, env, sc.slug); err != nil { return err } // The edge security tier (ADR-0043): install CrowdSec + the nftables firewall @@ -746,11 +746,18 @@ func provisionService(ctx *pulumi.Context, svc types.ServiceSpec, host types.Com // authB64 is the base64 OTLP Basic-auth value, already marked secret by the caller; // the credential write is built inside an ApplyT over it so the secret is encrypted // in Pulumi state (never written as plaintext), mirroring deliverServiceSecrets. -func provisionObservability(ctx *pulumi.Context, res types.Resources, computeOut map[string]types.ComputeOutputs, dbOut map[string]types.DatabaseOutputs, gates map[string]pulumi.Resource, dbHostTails map[string]pulumi.Resource, obs types.ObservabilityConfig, authB64 pulumi.StringOutput, deployPrivateKey, env, slug string) error { +func provisionObservability(ctx *pulumi.Context, res types.Resources, computeOut map[string]types.ComputeOutputs, dbOut map[string]types.DatabaseOutputs, gates map[string]pulumi.Resource, dbHostTails map[string]pulumi.Resource, obs types.ObservabilityConfig, sec types.SecurityConfig, authB64 pulumi.StringOutput, deployPrivateKey, env, slug string) error { if obs.OTLPEndpoint == "" { return nil } deployUserByCompute := naming.DeployUsersByHost(res.Compute) + // CrowdSec edge hosts (ADR-0043): on these the collector also scrapes CrowdSec's + // loopback Prometheus endpoints. Empty when CrowdSec is off, so a non-edge host (or a + // CrowdSec-disabled env) renders a host-metrics-only collector, byte-identical. + csEdge := map[string]bool{} + if sec.Crowdsec.Enabled { + csEdge = crowdsecEdgeHosts(res, naming.CanonicalComputeKeys(res.Compute)) + } // Postgres cluster ports per host, single-sourced with the realization + firewall // (ADR-0037); used to add a postgresql receiver per co-located cluster. ports := clusterPortsByHost(res, naming.CanonicalComputeKeys(res.Compute)) @@ -860,6 +867,13 @@ func provisionObservability(ctx *pulumi.Context, res types.Resources, computeOut pgPasswords = append(pgPasswords, pw.Result) } + var csScrape []string + if csEdge[hostKey] { + csScrape = []string{ + fmt.Sprintf("%s:%d", crowdsec.AgentMetricsAddr, crowdsec.AgentMetricsPort), + fmt.Sprintf("%s:%d", crowdsec.BouncerMetricsAddr, crowdsec.BouncerMetricsPort), + } + } config, err := otelcol.Render(obs.OTLPEndpoint, otelcol.Attributes{ HostID: naming.Resource(env, slug, "vm", hostKey), CloudProvider: host.CloudProvider, @@ -868,7 +882,7 @@ func provisionObservability(ctx *pulumi.Context, res types.Resources, computeOut MachineType: host.MachineType, Environment: env, RegionSlug: slug, - }, pgTargets) + }, pgTargets, csScrape...) if err != nil { return fmt.Errorf("observability: host %q: render config: %w", hostKey, err) } From a706a5e9c0d100a1919e464bb9e5a32b4fd275d6 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 19 Jul 2026 18:47:14 +0000 Subject: [PATCH 11/14] docs: document the security.crowdsec variables block (ADR-0043) Add CrowdSec to the variables.yaml reference: the crowdsec block (enabled, version, console), the reserved security/crowdsec_enroll secret, and the note that its metrics ride the observability pipeline when otlp_endpoint is set. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_011RB2WPLAzoDT1bjXrMfoXi --- website/docs/configuration/variables-yaml.md | 23 ++++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/website/docs/configuration/variables-yaml.md b/website/docs/configuration/variables-yaml.md index 4b367a8..aba80d1 100644 --- a/website/docs/configuration/variables-yaml.md +++ b/website/docs/configuration/variables-yaml.md @@ -22,6 +22,8 @@ observability: # optional — Grafana Cloud integration grafana_url: https://myorg.grafana.net # dashboards + alerts target security: # optional — edge security tier (public ingress + gateway hosts) + crowdsec: + enabled: true # IP banning (nftables) + free community blocklist on edges rate_limit: enabled: true # one blanket IP-based limit on every public edge server requests_per_second: 20 @@ -91,6 +93,24 @@ opts the whole tier out with `security: false` on its own spec. Health-check and ACME (certificate) endpoints are never rate-limited. +- `crowdsec` — installs the [CrowdSec](https://www.crowdsec.net) agent + nftables firewall + bouncer on every edge host. It parses the ingress nginx logs, bans abusive and + known-bad IPs at the kernel (nftables), and pulls the free crowd-sourced community + blocklist for pre-emptive protection. Fields: + - `enabled` — turn CrowdSec on (default off). + - `version` — optional pin for the `crowdsec` agent package. The firewall bouncer + versions independently, so it is never pinned; omit to install the repo's current pair. + - `console` — enroll the host in the CrowdSec console dashboard. Requires the reserved + secret `security/crowdsec_enroll` + (`inforge secret set security crowdsec_enroll --reserved`). The community + blocklist itself needs no secret. + + When `observability.otlp_endpoint` is set, CrowdSec's own metrics (log-acquisition rate, + active decisions, bouncer pulls) are scraped by the host collector and shipped to Grafana + Cloud alongside host and database metrics — so you can see it working, and catch its + silent-failure mode (unreadable logs ⇒ nothing parsed). A deploy fails if CrowdSec does + not come up on an edge host. + ## Example ```yaml title="resources/prd/variables.yaml" @@ -105,6 +125,9 @@ observability: # built_in_dashboards: false # opt out of the generated dashboards # built_in_alerts: false # opt out of the generated alert rules security: + crowdsec: + enabled: true + # console: true # dashboard enrollment (needs security/crowdsec_enroll) rate_limit: enabled: true requests_per_second: 20 From 8bc886de88eee41c641fc58cc44fad3359ee693f Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 19 Jul 2026 19:18:48 +0000 Subject: [PATCH 12/14] feat(observability): CrowdSec built-in dashboard + two alerts (ADR-0043, slice 2b) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Close out CrowdSec observability with a built-in Grafana dashboard and two alerts, gated on security.crowdsec.enabled exactly as the Postgres built-ins are gated on a cluster existing (so a non-CrowdSec env gets neither). - grafanadash.Crowdsec: env-scoped dashboard over cs_* metrics — acquisition rate (the health signal), active decisions, decisions by reason, scenario hits, and the firewall bouncer's drops + last-pull freshness. - grafanaalert.CrowdsecBuiltIns: "Acquisition Stalled" (parse rate ~0 → blind) and "Bouncer Not Pulling" (last-pull stale → enforcement drift). Both NoDataOK, so an unverified metric name or not-yet-scraped agent never false-fires — they alert only on a present-but-bad signal. - program/grafana.go: thread hasCrowdsec (= vars.Security.Crowdsec.Enabled) through realizeGrafana/realizeGrafanaAlerts; gate the CrowdSec dashboard + alerts on it. - Tests + docs (built-in dashboard/alert tables). Metric names (cs_parser_hits_total, cs_bouncer_last_pull, …) are the standard CrowdSec series and should be confirmed on a real edge; NoDataOK keeps a mismatch from firing meanwhile. This completes ADR-0043 (edge security tier): rate limiting + CrowdSec, wired, observable, and documented. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_011RB2WPLAzoDT1bjXrMfoXi --- internal/grafanaalert/crowdsec.go | 34 +++++++++++ internal/grafanaalert/crowdsec_test.go | 35 +++++++++++ internal/grafanadash/crowdsec.go | 64 +++++++++++++++++++++ internal/grafanadash/crowdsec_test.go | 41 +++++++++++++ program/grafana.go | 22 ++++++- program/program.go | 5 +- website/docs/configuration/observability.md | 12 +++- 7 files changed, 207 insertions(+), 6 deletions(-) create mode 100644 internal/grafanaalert/crowdsec.go create mode 100644 internal/grafanaalert/crowdsec_test.go create mode 100644 internal/grafanadash/crowdsec.go create mode 100644 internal/grafanadash/crowdsec_test.go diff --git a/internal/grafanaalert/crowdsec.go b/internal/grafanaalert/crowdsec.go new file mode 100644 index 0000000..10ac364 --- /dev/null +++ b/internal/grafanaalert/crowdsec.go @@ -0,0 +1,34 @@ +package grafanaalert + +import "fmt" + +// CrowdsecBuiltIns returns the built-in CrowdSec alert rules for one env (ADR-0043), +// generated from the CrowdSec agent + firewall-bouncer metrics scraped on edge hosts. +// Like the Postgres built-ins, the caller gates these on CrowdSec actually being enabled +// (security.crowdsec.enabled) so they cannot evaluate on an env that runs no CrowdSec. +// +// Both use NoDataOK, not NoDataAlerting: the exact cs_* metric names are only confirmed +// against a running agent, so ABSENCE (a name mismatch, or CrowdSec not yet scraped) must +// not fire — these alert only on a present-but-bad signal. Once the metric names are +// verified on a real edge, an absence variant can be tightened to NoDataAlerting. +func CrowdsecBuiltIns(env string) []Alert { + e := envMatcher(env) + return []Alert{ + // Acquisition stalled: CrowdSec is present but parsing ~no log lines, so it is + // blind to attacks. A genuinely idle edge can also read zero — the threshold is + // fixed (like every built-in); opt out and re-author if that is expected. + b("CrowdSec Acquisition Stalled", + fmt.Sprintf(`sum by (instance) (rate(cs_parser_hits_total{%s}[10m]))`, e), + "< 0.001", "15m", SeverityWarning, + "CrowdSec on {{ $labels.instance }} is parsing no log lines — acquisition has stalled (it is blind to attacks).", + NoDataOK), + + // Bouncer not pulling: the firewall bouncer has not fetched decisions recently, so + // new bans are not being enforced at the edge. cs_bouncer_last_pull is a unix ts. + b("CrowdSec Bouncer Not Pulling", + fmt.Sprintf(`time() - max by (instance) (cs_bouncer_last_pull{%s})`, e), + "> 600", "10m", SeverityWarning, + "CrowdSec firewall bouncer on {{ $labels.instance }} has not pulled decisions in over 10m — enforcement is drifting.", + NoDataOK), + } +} diff --git a/internal/grafanaalert/crowdsec_test.go b/internal/grafanaalert/crowdsec_test.go new file mode 100644 index 0000000..d6cebf1 --- /dev/null +++ b/internal/grafanaalert/crowdsec_test.go @@ -0,0 +1,35 @@ +package grafanaalert + +import ( + "strings" + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" +) + +func TestCrowdsecBuiltIns(t *testing.T) { + as := CrowdsecBuiltIns("prd") + require.Len(t, as, 2) + + by := map[string]Alert{} + for _, a := range as { + by[a.Rule.Name] = a + } + require.Contains(t, by, "CrowdSec Acquisition Stalled") + require.Contains(t, by, "CrowdSec Bouncer Not Pulling") + + for _, a := range as { + // Warnings, env-scoped, and NoDataOK — absence (an unverified metric name, or + // CrowdSec not yet scraped) must not fire; they alert only on a present-but-bad signal. + assert.Equal(t, SeverityWarning, a.Severity, a.Rule.Name) + assert.Equal(t, NoDataOK, a.Rule.NoDataState, a.Rule.Name) + require.NotEmpty(t, a.Rule.Data) + assert.Contains(t, a.Rule.Data[0].Model, `deployment_environment_name=\"prd\"`, a.Rule.Name) + } + + // The acquisition alert fires on a low parse rate; the bouncer alert on a stale pull. + assert.Contains(t, by["CrowdSec Acquisition Stalled"].Rule.Data[0].Model, "cs_parser_hits_total") + assert.Contains(t, by["CrowdSec Bouncer Not Pulling"].Rule.Data[0].Model, "cs_bouncer_last_pull") + assert.True(t, strings.HasPrefix(by["CrowdSec Acquisition Stalled"].Rule.Data[0].Model, "{"), "model is JSON") +} diff --git a/internal/grafanadash/crowdsec.go b/internal/grafanadash/crowdsec.go new file mode 100644 index 0000000..090c8cc --- /dev/null +++ b/internal/grafanadash/crowdsec.go @@ -0,0 +1,64 @@ +package grafanadash + +import "fmt" + +// Crowdsec renders the built-in CrowdSec dashboard for one env (ADR-0043) from the agent +// + firewall-bouncer metrics scraped on edge hosts (service.name wardnet-crowdsec-metrics): +// an overview (active bans, acquisition health, edges reporting), acquisition/detection +// detail, decisions, and the firewall bouncer's drops + pull freshness. uid is the +// env-prefixed dashboard UID (grafana.DashboardUID). The CrowdSec metric names are the +// standard cs_* series; a name that turns out different on the running agent simply shows +// "No data" in its panel (harmless), unlike an alert. +func Crowdsec(env, uid string) (string, error) { + e := envMatcher(env) + inst := `instance=~"$instance"` + ri := "$__rate_interval" + + vars := []map[string]any{queryVar("instance", "Edge", "cs_parser_hits_total", e)} + + panels := []map[string]any{ + row("Overview", 0), + stat("Active Decisions", gp(0, 1, 5, 4), + []map[string]any{target(fmt.Sprintf("sum(cs_active_decisions{%s})", e), "__auto")}, + "none", "Currently active IP ban decisions across all edges.", nil), + stat("Bans Added (range)", gp(5, 1, 5, 4), + []map[string]any{target(fmt.Sprintf("sum(increase(cs_lapi_decisions_total{%s}[$__range]))", e), "__auto")}, + "none", "New decisions recorded in the selected time range.", nil), + stat("Acquisition Rate", gp(10, 1, 7, 4), + []map[string]any{target(fmt.Sprintf("sum(rate(cs_parser_hits_total{%s}[%s]))", e, ri), "__auto")}, + "cps", "Log lines parsed per second — the health signal. Zero on a busy edge means CrowdSec is blind.", nil), + stat("Edges Reporting", gp(17, 1, 7, 4), + []map[string]any{target(fmt.Sprintf("count(count by (instance) (cs_parser_hits_total{%s}))", e), "__auto")}, + "none", "Edge hosts reporting CrowdSec metrics.", nil), + + row("Acquisition & Detection", 5), + ts("Log acquisition rate by source", gp(0, 6, 12, 8), + targets([2]string{fmt.Sprintf("sum by (source) (rate(cs_reader_hits_total{%s}[%s]))", e, ri), "{{source}}"}), + "cps"), + ts("Scenario hits", gp(12, 6, 12, 8), + targets([2]string{fmt.Sprintf("sum by (name) (rate(cs_scenario_hits_total{%s}[%s]))", e, ri), "{{name}}"}), + "cps"), + + row("Decisions", 14), + ts("Decisions added by reason", gp(0, 15, 12, 8), + targets([2]string{fmt.Sprintf("sum by (reason) (rate(cs_lapi_decisions_total{%s}[%s]))", e, ri), "{{reason}}"}), + "cps"), + ts("Active decisions by origin", gp(12, 15, 12, 8), + targets([2]string{fmt.Sprintf("sum by (origin) (cs_active_decisions{%s})", e), "{{origin}}"}), + "none"), + + row("Firewall Bouncer", 23), + ts("Bouncer dropped packets/s by edge", gp(0, 24, 12, 8), + targets([2]string{fmt.Sprintf("sum by (instance) (rate(cs_bouncer_dropped_packets_total{%s, %s}[%s]))", e, inst, ri), "{{instance}}"}), + "cps"), + ts("Bouncer dropped bytes/s by edge", gp(12, 24, 12, 8), + targets([2]string{fmt.Sprintf("sum by (instance) (rate(cs_bouncer_dropped_bytes_total{%s, %s}[%s]))", e, inst, ri), "{{instance}}"}), + "Bps"), + stat("Bouncer last-pull age by edge", gp(0, 32, 24, 6), + []map[string]any{target(fmt.Sprintf("time() - max by (instance) (cs_bouncer_last_pull{%s, %s})", e, inst), "{{instance}}")}, + "s", "Seconds since each edge's firewall bouncer last pulled decisions. A growing value means enforcement is drifting.", + thresholds([2]any{"green", nil}, [2]any{"orange", 300}, [2]any{"red", 900})), + } + + return dashboard("CrowdSec Monitoring", uid, []string{"inforge", "otel", "crowdsec"}, vars, panels) +} diff --git a/internal/grafanadash/crowdsec_test.go b/internal/grafanadash/crowdsec_test.go new file mode 100644 index 0000000..28c4d7b --- /dev/null +++ b/internal/grafanadash/crowdsec_test.go @@ -0,0 +1,41 @@ +package grafanadash + +import ( + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" +) + +func TestCrowdsecRenders(t *testing.T) { + out, err := Crowdsec("prd", "inforge-prd-crowdsec") + require.NoError(t, err) + m := parse(t, out) + + assert.Equal(t, "inforge-prd-crowdsec", m["uid"]) + assert.Equal(t, "CrowdSec Monitoring", m["title"]) + // Every query is env-scoped, and the CrowdSec signals + sections are present. + assert.Contains(t, out, `deployment_environment_name=\"prd\"`) + for _, want := range []string{ + "Overview", "Acquisition \\u0026 Detection", "Decisions", "Firewall Bouncer", + "cs_active_decisions", + "cs_parser_hits_total", + "cs_reader_hits_total", + "cs_lapi_decisions_total", + "cs_scenario_hits_total", + "cs_bouncer_dropped_packets_total", + "cs_bouncer_dropped_bytes_total", + "cs_bouncer_last_pull", + `"label":"Edge"`, + } { + assert.Contains(t, out, want) + } +} + +func TestCrowdsecDeterministic(t *testing.T) { + a, err := Crowdsec("prd", "u") + require.NoError(t, err) + b, err := Crowdsec("prd", "u") + require.NoError(t, err) + assert.Equal(t, a, b, "render must be byte-stable") +} diff --git a/program/grafana.go b/program/grafana.go index 5caf552..5d63304 100644 --- a/program/grafana.go +++ b/program/grafana.go @@ -30,7 +30,7 @@ import ( // inforge envs share one Grafana org cleanly. Built-in dashboards and built-in alerts // are each opt-out per env (obs.DashboardsEnabled / obs.AlertsEnabled); routing uses // per-rule notification settings, so no org-singleton notification policy is touched. -func realizeGrafana(ctx *pulumi.Context, dir, srcEnv, env string, obs types.ObservabilityConfig, hasDatabase, dryRun bool, services []grafanaalert.ServiceScope) error { +func realizeGrafana(ctx *pulumi.Context, dir, srcEnv, env string, obs types.ObservabilityConfig, hasDatabase, hasCrowdsec, dryRun bool, services []grafanaalert.ServiceScope) error { if obs.GrafanaURL == "" { return nil } @@ -89,6 +89,17 @@ func realizeGrafana(ctx *pulumi.Context, dir, srcEnv, env string, obs types.Obse return err } } + // The CrowdSec dashboard exists only when CrowdSec is enabled (ADR-0043), the same + // resource-gating the Postgres dashboard/alerts use — a non-CrowdSec env gets none. + if hasCrowdsec { + body, err := grafanadash.Crowdsec(env, grafana.DashboardUID(env, "crowdsec")) + if err != nil { + return err + } + if err := newDashboard("crowdsec", body); err != nil { + return err + } + } } // Custom dashboards: Grafana-exported files committed under this env's @@ -109,7 +120,7 @@ func realizeGrafana(ctx *pulumi.Context, dir, srcEnv, env string, obs types.Obse } // Alert rules + contact points (opt-out per env for the built-ins). - if err := realizeGrafanaAlerts(ctx, prov, folder, dir, srcEnv, env, obs, hasDatabase, dryRun, services); err != nil { + if err := realizeGrafanaAlerts(ctx, prov, folder, dir, srcEnv, env, obs, hasDatabase, hasCrowdsec, dryRun, services); err != nil { return err } return nil @@ -122,7 +133,7 @@ func realizeGrafana(ctx *pulumi.Context, dir, srcEnv, env string, obs types.Obse // custom alerts. Each rule routes via its own NotificationSettings.ContactPoint — // resolved from the alert's profile + severity — so the org notification policy is // never touched. A route marked `muted: true` drops the rule. -func realizeGrafanaAlerts(ctx *pulumi.Context, prov *grafanasdk.Provider, folder *oss.Folder, dir, srcEnv, env string, obs types.ObservabilityConfig, hasDatabase, dryRun bool, services []grafanaalert.ServiceScope) error { +func realizeGrafanaAlerts(ctx *pulumi.Context, prov *grafanasdk.Provider, folder *oss.Folder, dir, srcEnv, env string, obs types.ObservabilityConfig, hasDatabase, hasCrowdsec, dryRun bool, services []grafanaalert.ServiceScope) error { notif, err := loader.LoadNotifications(srcEnv, dir) if err != nil { return err @@ -163,6 +174,11 @@ func realizeGrafanaAlerts(ctx *pulumi.Context, prov *grafanasdk.Provider, folder if obs.AlertsEnabled() { alerts = append(alerts, grafanaalert.BuiltIns(env, hasDatabase)...) alerts = append(alerts, grafanaalert.ServiceBuiltIns(env, services)...) + // CrowdSec alerts only when CrowdSec is enabled (ADR-0043) — otherwise the + // acquisition/bouncer rules would evaluate against series that never exist. + if hasCrowdsec { + alerts = append(alerts, grafanaalert.CrowdsecBuiltIns(env)...) + } } customAlerts, err := grafanaalert.Custom(alertsSpec.Alerts) if err != nil { diff --git a/program/program.go b/program/program.go index b56c9b1..db03a54 100644 --- a/program/program.go +++ b/program/program.go @@ -329,7 +329,10 @@ func Run(ctx *pulumi.Context) error { // built-in dashboards are pushed via the pulumiverse/grafana provider. Org-global, so // realized ONCE here (not per scope) and env-prefixed. A no-op when grafana_url is empty. hasDatabase := len(res.DatabaseCluster) > 0 || len(globalRes.DatabaseCluster) > 0 - if err := realizeGrafana(ctx, dir, srcEnv, env, vars.Observability, hasDatabase, ctx.DryRun(), serviceScopes(regionTable, res, globalRes)); err != nil { + // CrowdSec dashboard/alerts are gated on it being enabled for this env (ADR-0043), + // mirroring how the Postgres built-ins are gated on a cluster existing. + hasCrowdsec := vars.Security.Crowdsec.Enabled + if err := realizeGrafana(ctx, dir, srcEnv, env, vars.Observability, hasDatabase, hasCrowdsec, ctx.DryRun(), serviceScopes(regionTable, res, globalRes)); err != nil { return err } diff --git a/website/docs/configuration/observability.md b/website/docs/configuration/observability.md index 3e88892..7c8bd93 100644 --- a/website/docs/configuration/observability.md +++ b/website/docs/configuration/observability.md @@ -24,6 +24,7 @@ per-node / per-cluster / per-database / per-service drill-down. | **Infrastructure** | host metrics (ADR-0031, `system_*`) | fleet CPU / memory / disk / network, per-node uptime and restarts | | **Database** | Postgres receiver metrics (ADR-0037, `postgresql_*`) | connections, transaction rate, DB size, checkpoints — fleet, per cluster, and per database | | **Service** | wardnet-cloud RED + domain metrics | request rate, 5xx error rate, and latency percentiles per service and route, plus the domain counters (DDNS networks provisioned, active tunnels, tenant tombstone sweeps) | +| **CrowdSec** | CrowdSec agent + firewall-bouncer metrics (ADR-0043, `cs_*`) | log-acquisition rate (the "is it working" signal), active ban decisions, decisions by reason, scenario hits, and the firewall bouncer's dropped packets/bytes + last-pull freshness. Present only when [`security.crowdsec.enabled`](/configuration/variables-yaml#security). | The **Service** dashboard reads the RED histogram `http_server_request_duration_seconds_*` that every wardnet-cloud service emits, grouped by the `service_name` label Grafana Cloud promotes from the OTLP @@ -71,8 +72,9 @@ to opt out of the generated ones; author your own in `resources//observabil ### Built-in alerts When `built_in_alerts` is on (the default), inforge generates these from the metrics it owns. All are -env-scoped and fire per node / cluster; the Postgres ones only exist when the env has a database -cluster. Thresholds are fixed — to tune, opt out and re-author as custom alerts. +env-scoped and fire per node / cluster / edge; the Postgres ones only exist when the env has a +database cluster, and the CrowdSec ones only when [`security.crowdsec.enabled`](/configuration/variables-yaml#security). +Thresholds are fixed — to tune, opt out and re-author as custom alerts. | Alert | Condition | For | Severity | |---|---|---|---| @@ -85,6 +87,12 @@ cluster. Thresholds are fixed — to tune, opt out and re-author as custom alert | Postgres Connections High | backends > 80% of max per cluster | 10m | warning | | Postgres High Rollback Ratio | rollbacks > 25% of transactions per cluster | 15m | warning | | Postgres Metrics Missing | no cluster reporting (NoData → alerting) | 5m | critical | +| CrowdSec Acquisition Stalled | parse rate ≈ 0 per edge (CrowdSec is blind) | 15m | warning | +| CrowdSec Bouncer Not Pulling | firewall bouncer last-pull > 10m stale per edge | 10m | warning | + +The CrowdSec alerts use `NoData → OK` (not `alerting`): they fire only on a present-but-bad signal, +so a metric-name mismatch or a not-yet-scraped agent never false-fires. Confirm CrowdSec is exporting +metrics (the built-in CrowdSec dashboard shows a non-zero acquisition rate) after the first deploy. ### Custom alerts From eb262b56d9d99b7a8b0d358b874c4a7ce45adc06 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 19 Jul 2026 19:26:52 +0000 Subject: [PATCH 13/14] fix(crowdsec): address code-review findings in the install shell Four robustness fixes from the branch code review: 1. Restart policy for the CrowdSec units. A crashed crowdsec-firewall-bouncer silently stops enforcing bans; add a Restart=always systemd drop-in for both units (the internal/nginx precedent / daemon-units-restart-on-any-exit rule) so a security daemon self-heals. 2. `cscli hub update` before `cscli collections install`. A fresh host has no hub index, so the install otherwise fails ("can't find collection") and fails the deploy under set -e. 3. Retry `cscli lapi status` in the liveness assertion. The LAPI needs a moment to rebind :8080 after the restart; poll briefly before the final unsuppressed check so a healthy CrowdSec is not failed on a race. 4. Surface `cscli console enroll` failures instead of `|| true`. A bad/expired console token was silently swallowed (green deploy, no enrollment); now it is logged to stderr while staying non-fatal. Tests assert each fix (restart drop-in, hub-update ordering, lapi retry, enroll error surfaced). Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_011RB2WPLAzoDT1bjXrMfoXi --- internal/crowdsec/install.go | 32 +++++++++++++++++++++++++++---- internal/crowdsec/install_test.go | 28 +++++++++++++++++++++++++++ 2 files changed, 56 insertions(+), 4 deletions(-) diff --git a/internal/crowdsec/install.go b/internal/crowdsec/install.go index 4264acc..f95f1db 100644 --- a/internal/crowdsec/install.go +++ b/internal/crowdsec/install.go @@ -43,6 +43,22 @@ echo "deb [signed-by=%[2]s] https://packagecloud.io/crowdsec/crowdsec/${ID}/ ${V %[1]s sudo DEBIAN_FRONTEND=noninteractive apt-get -o DPkg::Lock::Timeout=300 install -y %[5]s %[6]s +# A dead security daemon silently stops enforcing (a crashed bouncer = no bans applied). +# The packaged units may carry no restart policy, so guarantee one via a drop-in — the same +# policy inforge writes for every unit it owns (.agents/rules/daemon-units-restart-on-any-exit.md). +for unit in %[7]s %[8]s; do + sudo install -d -m 0755 "/etc/systemd/system/${unit}.service.d" + sudo tee "/etc/systemd/system/${unit}.service.d/10-restart.conf" >/dev/null <<'UNIT' +# Managed by inforge — a daemon has no correct exit. +[Unit] +StartLimitIntervalSec=0 + +[Service] +Restart=always +RestartSec=5 +UNIT +done +sudo systemctl daemon-reload sudo systemctl enable %[7]s %[8]s `, aptlock.UpdateCmd(), KeyringPath, RepoKeyURL, RepoListPath, agentPkg, BouncerPackage, AgentService, BouncerService) } @@ -58,6 +74,9 @@ func ConfigScript() string { "set -e", writeFileScript(AgentConfigLocal, agentPrometheusLocal(), "0644", "root:root"), writeFileScript(AcquisPath, nginxAcquis(), "0644", "root:root"), + // Refresh the hub index first — a fresh host has none, and `collections install` + // fails ("can't find collection") without it, failing the deploy under set -e. + "sudo cscli hub update", fmt.Sprintf("sudo cscli collections install %s", strings.Join(Collections, " ")), // Register to the Central API (the community blocklist) unless already registered. "sudo cscli capi status >/dev/null 2>&1 || sudo cscli capi register", @@ -86,12 +105,14 @@ func BouncerScript(apiKey string) string { } // EnrollScript enrolls the agent to the CrowdSec console with the given token and -// restarts the agent. It is optional (gated on the reserved crowdsec_enroll secret). The -// enroll is a no-op-safe best effort: an already-enrolled instance is left as-is. +// restarts the agent. It is optional (gated on the reserved crowdsec_enroll secret). +// Console enrollment is non-fatal (an already-enrolled host re-enrolls with an error we +// tolerate), but a failure is SURFACED to the deploy log rather than silently swallowed — +// so a bad or expired token is visible instead of a green deploy that never enrolled. func EnrollScript(token string) string { return strings.Join([]string{ "set -e", - fmt.Sprintf("sudo cscli console enroll %s || true", shQuote(token)), + fmt.Sprintf(`sudo cscli console enroll %s || echo "inforge: cscli console enroll failed (already enrolled, or a bad/expired token) — verify with 'cscli console status'" >&2`, shQuote(token)), fmt.Sprintf("sudo systemctl restart %s", AgentService), }, "\n") } @@ -102,7 +123,10 @@ func EnrollScript(token string) string { func AssertScript() string { return strings.Join([]string{ "set -e", - "sudo cscli lapi status", + // The local API needs a moment to rebind :8080 after the restart above, so poll it + // briefly before asserting — otherwise a healthy CrowdSec fails the deploy on a race. + "for i in 1 2 3 4 5; do sudo cscli lapi status >/dev/null 2>&1 && break; sleep 2; done", + "sudo cscli lapi status", // final, unsuppressed — fails the deploy if still down fmt.Sprintf("sudo cscli bouncers list -o json | grep -q %s", shQuote(BouncerName)), }, "\n") } diff --git a/internal/crowdsec/install_test.go b/internal/crowdsec/install_test.go index 05bfa30..4bdf936 100644 --- a/internal/crowdsec/install_test.go +++ b/internal/crowdsec/install_test.go @@ -24,6 +24,14 @@ func TestInstallScriptPinsAgentVersionOnly(t *testing.T) { assert.NotContains(t, s, BouncerPackage+"=") } +func TestInstallScriptGivesUnitsARestartPolicy(t *testing.T) { + s := InstallScript("") + // A crashed security daemon must self-heal, not silently stop enforcing. + assert.Contains(t, s, "Restart=always") + assert.Contains(t, s, "10-restart.conf") + assert.Contains(t, s, "systemctl daemon-reload") +} + func TestConfigScriptInstallsCollectionsAndRegistersCAPI(t *testing.T) { s := ConfigScript() assert.Contains(t, s, "cscli collections install crowdsecurity/nginx crowdsecurity/base-http-scenarios crowdsecurity/http-cve") @@ -33,6 +41,26 @@ func TestConfigScriptInstallsCollectionsAndRegistersCAPI(t *testing.T) { // paths appear in the mv target). assert.Contains(t, s, AcquisPath) assert.Contains(t, s, AgentConfigLocal) + // The hub index is refreshed BEFORE collections install, or a fresh host can't find them. + assert.Contains(t, s, "cscli hub update") + assert.Less(t, strings.Index(s, "cscli hub update"), strings.Index(s, "cscli collections install"), + "hub update must precede collections install") +} + +func TestAssertScriptRetriesLAPIStatus(t *testing.T) { + s := AssertScript() + // Poll before the final unsuppressed check, so a just-restarted LAPI does not race. + assert.Contains(t, s, "for i in 1 2 3 4 5") + assert.Contains(t, s, "cscli lapi status") + assert.Contains(t, s, "cscli bouncers list -o json | grep -q 'inforge-fw'") +} + +func TestEnrollScriptSurfacesFailure(t *testing.T) { + s := EnrollScript("tok") + assert.Contains(t, s, "cscli console enroll 'tok'") + // Failure is logged, not silently swallowed by `|| true`. + assert.NotContains(t, s, "|| true") + assert.Contains(t, s, ">&2") } func TestBouncerScriptIsIdempotentAndRestarts(t *testing.T) { From 48327bf8246683a6faa0135e4cc0b5d058e055fb Mon Sep 17 00:00:00 2001 From: Pedro Gomes Date: Mon, 20 Jul 2026 08:59:22 +0100 Subject: [PATCH 14/14] fix(crowdsec): correct metric names confirmed against a real host MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The CrowdSec install shell and the metric names used by the built-in dashboard and alerts (ADR-0043, slice 2b) were authored from docs and had never been run. Verified end-to-end on a throwaway Hetzner VM (Ubuntu 24.04 noble, amd64) running the exact scripts internal/crowdsec renders, against crowdsec 1.7.8 + crowdsec-firewall-bouncer-nftables 0.0.34. The install path was correct as written and is unchanged: the armored .asc works via signed-by with no dearmor, the packagecloud source line resolves, the acquis.d drop-in and both .local overlays are read, every cscli invocation matches, nftables enforcement lands, and the systemd Restart=always drop-in survives a kill -9. The metric names were substantially wrong. The firewall bouncer does not use the cs_ prefix at all: cs_reader_hits_total -> cs_filesource_hits_total cs_scenario_hits_total -> cs_bucket_poured_total cs_lapi_decisions_total -> no such counter (gauge only) cs_bouncer_dropped_packets_total -> fw_bouncer_dropped_packets cs_bouncer_dropped_bytes_total -> fw_bouncer_dropped_bytes cs_bouncer_last_pull -> does not exist cs_bouncer_last_pull never existed, so the "Bouncer Not Pulling" alert could not fire under any condition — the exact silent failure it was meant to catch. It now alerts on a stalled lapi_requests_total rate, the bouncer's real pull-freshness signal. Since CrowdSec publishes no decisions-added counter, no panel uses increase() over decisions; the "bans added" stat now reports fw_bouncer_banned_ips. cs_parser_hits_total and cs_active_decisions were correct as written. Tests assert the corrected names and now also assert the absent ones stay absent, so a silently-dead query cannot regress back in. ADR-0043 records the verification and drops the unverified caveats, retaining one open question: whether the otelcol prometheus -> otlphttp hop rewrites counter suffixes, which needs a live Grafana Cloud scrape to settle. --- ...dge-security-rate-limiting-and-crowdsec.md | 50 +++++++++++++++++- internal/grafanaalert/crowdsec.go | 25 +++++---- internal/grafanaalert/crowdsec_test.go | 13 +++-- internal/grafanadash/crowdsec.go | 51 ++++++++++++------- internal/grafanadash/crowdsec_test.go | 26 +++++++--- website/docs/configuration/observability.md | 4 +- 6 files changed, 128 insertions(+), 41 deletions(-) diff --git a/docs/adr/0043-edge-security-rate-limiting-and-crowdsec.md b/docs/adr/0043-edge-security-rate-limiting-and-crowdsec.md index e139d2a..cba0a5f 100644 --- a/docs/adr/0043-edge-security-rate-limiting-and-crowdsec.md +++ b/docs/adr/0043-edge-security-rate-limiting-and-crowdsec.md @@ -165,8 +165,9 @@ needs no secret. otelcol collector — gated on the same `observability.otlp_endpoint`, exported through the same `otlphttp` → Grafana Cloud, tagged with the ADR-0030 resource attributes, exactly as the `postgresql` receiver is added on a cluster host (ADR-0037). The load-bearing signal - is **acquisition health** (`cs_reader_hits` / parse rate > 0 = CrowdSec is actually seeing - traffic); the bouncer's last-pull timestamp catches enforcement drift. The deploy also + is **acquisition health** (`cs_filesource_hits_total` / `cs_parser_hits_total` rate > 0 = + CrowdSec is actually seeing traffic); the bouncer's LAPI request rate catches enforcement + drift. The deploy also asserts liveness at apply time — `cscli lapi status` and the bouncer appearing in `cscli bouncers list` — so a CrowdSec that did not come up fails the deploy rather than sitting dead. Dashboards and the two built-in alerts ("acquisition stalled", "bouncer not @@ -246,3 +247,48 @@ Grounded in the existing otelcol path (`program.provisionObservability`, edge host, so a separate placement FK invites a meaningless "WAF on a host with no ingress" case. The env-block + per-edge opt-out gives the same control with far less surface. + +## Host verification (2026-07-20) + +The install shell and the metric names in slice 2b were originally authored from docs. They +have since been run end-to-end on a throwaway Hetzner VM (Ubuntu 24.04 `noble`, amd64, +`crowdsec` 1.7.8 + `crowdsec-firewall-bouncer-nftables` 0.0.34), rendering the exact scripts +`internal/crowdsec` produces. Confirmed as designed: + +- **packagecloud repo** — the armored `.asc` works directly via `signed-by` with no + `gpg --dearmor`, and `.../crowdsec/crowdsec/ubuntu/ noble main` resolves; both packages install. +- **`acquis.d` drop-in** — `/etc/crowdsec/acquis.d/nginx.yaml` is read (`acquisition_dir` is a + package default) and `crowdsecurity/nginx-logs` parses the access log. Note CrowdSec tails + from end-of-file, so a source only appears in `cscli metrics` once new lines arrive. +- **`.local` overlays** — both merge; the bouncer logs `with additional values from + '…/crowdsec-firewall-bouncer.yaml.local'`. +- **cscli syntax** — `hub update`, `collections install`, `capi status`/`register`, + `bouncers add --key`/`delete`, `lapi status`, `bouncers list -o json`, and the positional + `console enroll ` all match. (The Debian package auto-registers CAPI, so the + `capi status || capi register` guard correctly no-ops.) +- **nftables enforcement** — `table ip crowdsec` / `table ip6 crowdsec6` with real `drop` + rules; a `cscli decisions add` lands in a `crowdsec-blacklists-cscli` set within seconds. +- **systemd `Restart=always`** — the drop-in applies to both units; `kill -9` respawns. +- **Metrics ports** — agent `127.0.0.1:6060` and bouncer `127.0.0.1:60601` are both correct. + +**Metric names were substantially wrong** and are corrected in `internal/grafanadash` and +`internal/grafanaalert`. The bouncer does **not** use the `cs_` prefix: + +| Assumed | Actual | +|---|---| +| `cs_reader_hits_total` | `cs_filesource_hits_total` | +| `cs_scenario_hits_total` | `cs_bucket_poured_total` | +| `cs_lapi_decisions_total` | *(no such counter — decisions are the `cs_active_decisions` gauge only)* | +| `cs_bouncer_dropped_packets_total` | `fw_bouncer_dropped_packets` | +| `cs_bouncer_dropped_bytes_total` | `fw_bouncer_dropped_bytes` | +| `cs_bouncer_last_pull` | *(does not exist — use the bouncer's `lapi_requests_total` rate)* | + +`cs_parser_hits_total` and `cs_active_decisions` were correct as written. Because there is no +decisions-added counter, no panel uses `increase()` over decisions; the "bans added" stat now +reports `fw_bouncer_banned_ips` (what the firewall actually holds). `fw_bouncer_dropped_*` are +declared `gauge` despite being cumulative — `rate()` remains correct over them. + +**Still unverified:** how the OTLP hop renames these series. The collector's `prometheus` +receiver → `otlphttp` → Grafana Cloud path may add or strip a `_total` suffix on counters; that +needs one live scrape against the real endpoint to pin down, and the queries above assume the +names pass through unchanged. diff --git a/internal/grafanaalert/crowdsec.go b/internal/grafanaalert/crowdsec.go index 10ac364..5a91f48 100644 --- a/internal/grafanaalert/crowdsec.go +++ b/internal/grafanaalert/crowdsec.go @@ -7,10 +7,15 @@ import "fmt" // Like the Postgres built-ins, the caller gates these on CrowdSec actually being enabled // (security.crowdsec.enabled) so they cannot evaluate on an env that runs no CrowdSec. // -// Both use NoDataOK, not NoDataAlerting: the exact cs_* metric names are only confirmed -// against a running agent, so ABSENCE (a name mismatch, or CrowdSec not yet scraped) must -// not fire — these alert only on a present-but-bad signal. Once the metric names are -// verified on a real edge, an absence variant can be tightened to NoDataAlerting. +// Both metric names are confirmed against a real host (crowdsec 1.7.8 + +// crowdsec-firewall-bouncer-nftables 0.0.34): the agent exposes cs_parser_hits_total on +// :6060 and the bouncer exposes lapi_requests_total on :60601. Note the bouncer's series +// are prefixed fw_bouncer_*, NOT cs_bouncer_*, and it publishes no last-pull timestamp — +// its LAPI call counter is the pull-freshness signal instead. +// +// Both still use NoDataOK, not NoDataAlerting: an edge that is scraped but not yet +// reporting (or an env between deploys) must not page. These alert only on a +// present-but-bad signal; absence is covered by the collector's own up/health signal. func CrowdsecBuiltIns(env string) []Alert { e := envMatcher(env) return []Alert{ @@ -23,12 +28,14 @@ func CrowdsecBuiltIns(env string) []Alert { "CrowdSec on {{ $labels.instance }} is parsing no log lines — acquisition has stalled (it is blind to attacks).", NoDataOK), - // Bouncer not pulling: the firewall bouncer has not fetched decisions recently, so - // new bans are not being enforced at the edge. cs_bouncer_last_pull is a unix ts. + // Bouncer not pulling: the firewall bouncer has stopped calling the local API, so new + // bans are not reaching nftables and enforcement is drifting. The bouncer polls + // /v1/decisions/stream continuously, so a flat-zero request rate is the stall signal + // (it publishes no last-pull timestamp to compare against time()). b("CrowdSec Bouncer Not Pulling", - fmt.Sprintf(`time() - max by (instance) (cs_bouncer_last_pull{%s})`, e), - "> 600", "10m", SeverityWarning, - "CrowdSec firewall bouncer on {{ $labels.instance }} has not pulled decisions in over 10m — enforcement is drifting.", + fmt.Sprintf(`sum by (instance) (rate(lapi_requests_total{%s}[10m]))`, e), + "< 0.001", "10m", SeverityWarning, + "CrowdSec firewall bouncer on {{ $labels.instance }} has not called the local API in over 10m — new bans are not being enforced.", NoDataOK), } } diff --git a/internal/grafanaalert/crowdsec_test.go b/internal/grafanaalert/crowdsec_test.go index d6cebf1..a6b2630 100644 --- a/internal/grafanaalert/crowdsec_test.go +++ b/internal/grafanaalert/crowdsec_test.go @@ -20,16 +20,21 @@ func TestCrowdsecBuiltIns(t *testing.T) { require.Contains(t, by, "CrowdSec Bouncer Not Pulling") for _, a := range as { - // Warnings, env-scoped, and NoDataOK — absence (an unverified metric name, or - // CrowdSec not yet scraped) must not fire; they alert only on a present-but-bad signal. + // Warnings, env-scoped, and NoDataOK — an edge that is scraped but not yet + // reporting must not page; they alert only on a present-but-bad signal. assert.Equal(t, SeverityWarning, a.Severity, a.Rule.Name) assert.Equal(t, NoDataOK, a.Rule.NoDataState, a.Rule.Name) require.NotEmpty(t, a.Rule.Data) assert.Contains(t, a.Rule.Data[0].Model, `deployment_environment_name=\"prd\"`, a.Rule.Name) } - // The acquisition alert fires on a low parse rate; the bouncer alert on a stale pull. + // The acquisition alert fires on a low parse rate; the bouncer alert on a stalled LAPI + // request rate. Both names are host-verified (ADR-0043 "Host verification"). assert.Contains(t, by["CrowdSec Acquisition Stalled"].Rule.Data[0].Model, "cs_parser_hits_total") - assert.Contains(t, by["CrowdSec Bouncer Not Pulling"].Rule.Data[0].Model, "cs_bouncer_last_pull") + assert.Contains(t, by["CrowdSec Bouncer Not Pulling"].Rule.Data[0].Model, "lapi_requests_total") assert.True(t, strings.HasPrefix(by["CrowdSec Acquisition Stalled"].Rule.Data[0].Model, "{"), "model is JSON") + + // cs_bouncer_last_pull does not exist on a real bouncer — an alert on it could never + // fire, which is exactly the silent failure this rule is meant to catch. + assert.NotContains(t, by["CrowdSec Bouncer Not Pulling"].Rule.Data[0].Model, "cs_bouncer_last_pull") } diff --git a/internal/grafanadash/crowdsec.go b/internal/grafanadash/crowdsec.go index 090c8cc..43564cd 100644 --- a/internal/grafanadash/crowdsec.go +++ b/internal/grafanadash/crowdsec.go @@ -5,10 +5,21 @@ import "fmt" // Crowdsec renders the built-in CrowdSec dashboard for one env (ADR-0043) from the agent // + firewall-bouncer metrics scraped on edge hosts (service.name wardnet-crowdsec-metrics): // an overview (active bans, acquisition health, edges reporting), acquisition/detection -// detail, decisions, and the firewall bouncer's drops + pull freshness. uid is the -// env-prefixed dashboard UID (grafana.DashboardUID). The CrowdSec metric names are the -// standard cs_* series; a name that turns out different on the running agent simply shows -// "No data" in its panel (harmless), unlike an alert. +// detail, decisions, and the firewall bouncer's drops + LAPI pull activity. uid is the +// env-prefixed dashboard UID (grafana.DashboardUID). +// +// Every metric name below was confirmed against a real host (crowdsec 1.7.8 + +// crowdsec-firewall-bouncer-nftables 0.0.34 on Ubuntu noble). Two naming facts are easy to +// get wrong and are load-bearing here: +// - The AGENT exposes cs_* on :6060, but the BOUNCER exposes fw_bouncer_* (NOT cs_bouncer_*) +// with no _total suffix, plus a bare lapi_requests_total, on :60601. +// - There is no cs_scenario_hits_total / cs_reader_hits_total / cs_lapi_decisions_total — +// the real series are cs_bucket_poured_total, cs_filesource_hits_total and the +// cs_active_decisions gauge. Decisions are a GAUGE only; no add-counter exists, so no +// panel here uses increase() over decisions. +// +// fw_bouncer_dropped_* are declared `gauge` though they are cumulative totals; rate() is +// still correct over them (it treats a decrease as a reset), which is what the drop panels use. func Crowdsec(env, uid string) (string, error) { e := envMatcher(env) inst := `instance=~"$instance"` @@ -21,9 +32,11 @@ func Crowdsec(env, uid string) (string, error) { stat("Active Decisions", gp(0, 1, 5, 4), []map[string]any{target(fmt.Sprintf("sum(cs_active_decisions{%s})", e), "__auto")}, "none", "Currently active IP ban decisions across all edges.", nil), - stat("Bans Added (range)", gp(5, 1, 5, 4), - []map[string]any{target(fmt.Sprintf("sum(increase(cs_lapi_decisions_total{%s}[$__range]))", e), "__auto")}, - "none", "New decisions recorded in the selected time range.", nil), + // CrowdSec publishes no decisions-added counter, so this reports what the firewall + // is actually enforcing (bouncer-side) next to what LAPI knows (cs_active_decisions). + stat("IPs Banned at Firewall", gp(5, 1, 5, 4), + []map[string]any{target(fmt.Sprintf("sum(fw_bouncer_banned_ips{%s})", e), "__auto")}, + "none", "IP addresses currently loaded into the edge firewall's nftables ban sets.", nil), stat("Acquisition Rate", gp(10, 1, 7, 4), []map[string]any{target(fmt.Sprintf("sum(rate(cs_parser_hits_total{%s}[%s]))", e, ri), "__auto")}, "cps", "Log lines parsed per second — the health signal. Zero on a busy edge means CrowdSec is blind.", nil), @@ -33,31 +46,33 @@ func Crowdsec(env, uid string) (string, error) { row("Acquisition & Detection", 5), ts("Log acquisition rate by source", gp(0, 6, 12, 8), - targets([2]string{fmt.Sprintf("sum by (source) (rate(cs_reader_hits_total{%s}[%s]))", e, ri), "{{source}}"}), + targets([2]string{fmt.Sprintf("sum by (source) (rate(cs_filesource_hits_total{%s}[%s]))", e, ri), "{{source}}"}), "cps"), ts("Scenario hits", gp(12, 6, 12, 8), - targets([2]string{fmt.Sprintf("sum by (name) (rate(cs_scenario_hits_total{%s}[%s]))", e, ri), "{{name}}"}), + targets([2]string{fmt.Sprintf("sum by (name) (rate(cs_bucket_poured_total{%s}[%s]))", e, ri), "{{name}}"}), "cps"), row("Decisions", 14), - ts("Decisions added by reason", gp(0, 15, 12, 8), - targets([2]string{fmt.Sprintf("sum by (reason) (rate(cs_lapi_decisions_total{%s}[%s]))", e, ri), "{{reason}}"}), - "cps"), + ts("Active decisions by reason", gp(0, 15, 12, 8), + targets([2]string{fmt.Sprintf("sum by (reason) (cs_active_decisions{%s})", e), "{{reason}}"}), + "none"), ts("Active decisions by origin", gp(12, 15, 12, 8), targets([2]string{fmt.Sprintf("sum by (origin) (cs_active_decisions{%s})", e), "{{origin}}"}), "none"), row("Firewall Bouncer", 23), ts("Bouncer dropped packets/s by edge", gp(0, 24, 12, 8), - targets([2]string{fmt.Sprintf("sum by (instance) (rate(cs_bouncer_dropped_packets_total{%s, %s}[%s]))", e, inst, ri), "{{instance}}"}), + targets([2]string{fmt.Sprintf("sum by (instance) (rate(fw_bouncer_dropped_packets{%s, %s}[%s]))", e, inst, ri), "{{instance}}"}), "cps"), ts("Bouncer dropped bytes/s by edge", gp(12, 24, 12, 8), - targets([2]string{fmt.Sprintf("sum by (instance) (rate(cs_bouncer_dropped_bytes_total{%s, %s}[%s]))", e, inst, ri), "{{instance}}"}), + targets([2]string{fmt.Sprintf("sum by (instance) (rate(fw_bouncer_dropped_bytes{%s, %s}[%s]))", e, inst, ri), "{{instance}}"}), "Bps"), - stat("Bouncer last-pull age by edge", gp(0, 32, 24, 6), - []map[string]any{target(fmt.Sprintf("time() - max by (instance) (cs_bouncer_last_pull{%s, %s})", e, inst), "{{instance}}")}, - "s", "Seconds since each edge's firewall bouncer last pulled decisions. A growing value means enforcement is drifting.", - thresholds([2]any{"green", nil}, [2]any{"orange", 300}, [2]any{"red", 900})), + // The bouncer exposes no last-pull timestamp; its LAPI call counter is the freshness + // signal instead — a healthy bouncer polls /v1/decisions/stream continuously, so a + // flat-zero rate means it has stopped pulling and enforcement is drifting. + ts("Bouncer LAPI pull rate by edge", gp(0, 32, 24, 6), + targets([2]string{fmt.Sprintf("sum by (instance) (rate(lapi_requests_total{%s, %s}[%s]))", e, inst, ri), "{{instance}}"}), + "reqps"), } return dashboard("CrowdSec Monitoring", uid, []string{"inforge", "otel", "crowdsec"}, vars, panels) diff --git a/internal/grafanadash/crowdsec_test.go b/internal/grafanadash/crowdsec_test.go index 28c4d7b..fa18458 100644 --- a/internal/grafanadash/crowdsec_test.go +++ b/internal/grafanadash/crowdsec_test.go @@ -16,20 +16,34 @@ func TestCrowdsecRenders(t *testing.T) { assert.Equal(t, "CrowdSec Monitoring", m["title"]) // Every query is env-scoped, and the CrowdSec signals + sections are present. assert.Contains(t, out, `deployment_environment_name=\"prd\"`) + // These names are verified against a real host (crowdsec 1.7.8 + firewall-bouncer + // 0.0.34) — see the "Host verification" section of ADR-0043. Note the bouncer's + // series are fw_bouncer_* with no _total suffix, NOT cs_bouncer_*. for _, want := range []string{ "Overview", "Acquisition \\u0026 Detection", "Decisions", "Firewall Bouncer", "cs_active_decisions", "cs_parser_hits_total", - "cs_reader_hits_total", - "cs_lapi_decisions_total", - "cs_scenario_hits_total", - "cs_bouncer_dropped_packets_total", - "cs_bouncer_dropped_bytes_total", - "cs_bouncer_last_pull", + "cs_filesource_hits_total", + "cs_bucket_poured_total", + "fw_bouncer_banned_ips", + "fw_bouncer_dropped_packets", + "fw_bouncer_dropped_bytes", + "lapi_requests_total", `"label":"Edge"`, } { assert.Contains(t, out, want) } + + // Guard the exact mistakes host verification caught: these series do not exist, and a + // panel querying one silently shows "No data" forever. + for _, absent := range []string{ + "cs_reader_hits_total", + "cs_scenario_hits_total", + "cs_lapi_decisions_total", + "cs_bouncer_", + } { + assert.NotContains(t, out, absent) + } } func TestCrowdsecDeterministic(t *testing.T) { diff --git a/website/docs/configuration/observability.md b/website/docs/configuration/observability.md index 7c8bd93..6c80049 100644 --- a/website/docs/configuration/observability.md +++ b/website/docs/configuration/observability.md @@ -24,7 +24,7 @@ per-node / per-cluster / per-database / per-service drill-down. | **Infrastructure** | host metrics (ADR-0031, `system_*`) | fleet CPU / memory / disk / network, per-node uptime and restarts | | **Database** | Postgres receiver metrics (ADR-0037, `postgresql_*`) | connections, transaction rate, DB size, checkpoints — fleet, per cluster, and per database | | **Service** | wardnet-cloud RED + domain metrics | request rate, 5xx error rate, and latency percentiles per service and route, plus the domain counters (DDNS networks provisioned, active tunnels, tenant tombstone sweeps) | -| **CrowdSec** | CrowdSec agent + firewall-bouncer metrics (ADR-0043, `cs_*`) | log-acquisition rate (the "is it working" signal), active ban decisions, decisions by reason, scenario hits, and the firewall bouncer's dropped packets/bytes + last-pull freshness. Present only when [`security.crowdsec.enabled`](/configuration/variables-yaml#security). | +| **CrowdSec** | CrowdSec agent (`cs_*`) + firewall-bouncer (`fw_bouncer_*`) metrics (ADR-0043) | log-acquisition rate (the "is it working" signal), active ban decisions by reason, scenario hits, IPs banned at the firewall, and the bouncer's dropped packets/bytes + LAPI pull rate. Present only when [`security.crowdsec.enabled`](/configuration/variables-yaml#security). | The **Service** dashboard reads the RED histogram `http_server_request_duration_seconds_*` that every wardnet-cloud service emits, grouped by the `service_name` label Grafana Cloud promotes from the OTLP @@ -88,7 +88,7 @@ Thresholds are fixed — to tune, opt out and re-author as custom alerts. | Postgres High Rollback Ratio | rollbacks > 25% of transactions per cluster | 15m | warning | | Postgres Metrics Missing | no cluster reporting (NoData → alerting) | 5m | critical | | CrowdSec Acquisition Stalled | parse rate ≈ 0 per edge (CrowdSec is blind) | 15m | warning | -| CrowdSec Bouncer Not Pulling | firewall bouncer last-pull > 10m stale per edge | 10m | warning | +| CrowdSec Bouncer Not Pulling | firewall bouncer's LAPI request rate ≈ 0 per edge | 10m | warning | The CrowdSec alerts use `NoData → OK` (not `alerting`): they fire only on a present-but-bad signal, so a metric-name mismatch or a not-yet-scraped agent never false-fires. Confirm CrowdSec is exporting