diff --git a/docs/adr/0043-edge-security-rate-limiting-and-crowdsec.md b/docs/adr/0043-edge-security-rate-limiting-and-crowdsec.md new file mode 100644 index 0000000..cba0a5f --- /dev/null +++ b/docs/adr/0043-edge-security-rate-limiting-and-crowdsec.md @@ -0,0 +1,294 @@ +--- +status: proposed +date: 2026-07-19 +issue: "TBD" +--- + +# Edge security tier: rate limiting and CrowdSec + +The public edge (ingress + gateway nginx) is the most-probed surface in the fleet: +scanning bots hammer `/.env`, `/wp-admin`, `/.git/config`, phpMyAdmin, and known-CVE +paths continuously. inforge terminates that traffic at a stock nginx.org build it +renders deterministically (`internal/nginx`), fronting custom Go services behind a +mesh. Today nothing throttles or bans abusive clients at that edge. + +This ADR adds an **edge security tier** with two complementary, independently-toggled +capabilities, authored from one `security:` block in `variables.yaml`: + +1. **Rate limiting** — nginx-native `limit_req`/`limit_conn`, rendered into the edge + nginx.conf. Real-time L7 throttling (returns `429`) of single/small-source request + floods. No host agent, no nginx module. +2. **CrowdSec** — a per-host agent (engine + local API) that parses the nginx access + logs, ban decisions enforced by the **firewall bouncer** at nftables, plus the free + crowd-sourced **community blocklist** for pre-emptive known-bad-IP bans. + +The two layers feed each other: nginx throttles a flood in real time and logs the +`429`s; CrowdSec reads those logs and bans the persistent offender at nftables so it +stops reaching nginx at all. + +## Scope boundary: what this does and does not stop + +Being explicit, because "DoS protection" over-promises if left unqualified: + +| Tier | Example | Handled by | +|---|---|---| +| L3/L4 volumetric | SYN/UDP flood, amplification | **Not the origin** — Hetzner's free network-edge DDoS protection (already in place). Saturates the uplink before nginx runs. | +| L7 single/small-source abuse | one IP at 500 req/s; brute-force; scraping | **This ADR** — nginx rate limiting + CrowdSec banning. The 95% case. | +| L7 distributed botnet flood | thousands of IPs, low rate each, real endpoint | **Partial** — the community blocklist catches known-bad IPs; the rest genuinely needs an upstream scrubber (Cloudflare). No self-hosted origin tool fully solves this, and we do not claim to. | + +Choosing CrowdSec over Cloudflare's WAF is a deliberate trade: one less external +lock-in, at the cost of not covering the distributed-botnet tier. That gap is +documented, not hidden. + +## Authoring + +One env-level block in `variables.yaml`, with a per-edge opt-out. Both capabilities are +off unless the block enables them. + +```yaml +security: + crowdsec: + enabled: true # opt-in per env; installs on every public edge host + # version: "1.6.4" # optional pin; else inforge's DefaultVersion + # console: true # optional dashboard enrollment (needs reserved secret) + rate_limit: + enabled: true # one blanket IP-based limit on EVERY public edge server + requests_per_second: 20 + burst: 40 # queued excess before a 429 + max_connections: 40 # concurrent conns per client IP +``` + +**One uniform limit, no per-route knobs.** IP rate limiting is a blanket *security* +measure — a floor applied identically to every public server on an edge, exactly like +CrowdSec. There is deliberately no per-route/per-app profile: a single +`requests_per_second`/`burst`/`max_connections` covers the whole edge. Per-route and +per-identity limits are a *different* concern (application logic, not a security floor) +and belong to the gateway module once authentication is at the edge — see ADR-0044; they +are enforced there (the auth service or a custom nginx dynamic module), never here. + +**Per-edge opt-out** — an ingress/gateway leaves the whole security tier: + +```yaml +# ingress//manifest.yaml or gateway//manifest.yaml +security: false # this edge gets neither CrowdSec nor rate limiting +``` + +**Optional CrowdSec console** enrollment token is a reserved secret (never inlined), +mirroring `observability/otlp_auth`: +`inforge secret set security crowdsec_enroll --reserved`. The community blocklist +needs no secret. + +## Decisions + +### Rate limiting + +- **nginx-native `limit_req` + `limit_conn`, rendered — not a module or a sidecar.** It + is stock-nginx.org (no lua/OpenResty), flows through the existing `internal/nginx` + crossplane render, and preserves the deterministic-bytes property. `limit_req` is the + leaky-bucket request-rate limiter; `limit_conn` caps concurrent connections per IP. +- **One uniform limit per edge, keyed by client IP (`$binary_remote_addr`) — no per-route + profiles.** Rate limiting is a security floor, so the same `requests_per_second`/`burst`/ + `max_connections` is stamped on every public server of an edge, not tuned per route. The + renderer resolves that single limit onto each `IngressRoute`/`IngressApp`/`IngressGateway` + and emits one `limit_req_zone`/`limit_conn_zone` in `http{}` (deduped — one zone, fixed + stem), with `limit_req zone= burst= nodelay;` + `limit_conn ;` in each + service-facing location. `limit_req_status 429;` / `limit_conn_status 429;` make a + throttled request answer `429` (CrowdSec-parseable), not the nginx default `503`. +- **Client IP is the only key.** Accurate here — grey-cloud records and the ssl_preread + `set_real_ip_from`/`proxy_protocol` recovery both yield the true client IP. Per-identity + keying is out of scope for this layer entirely: it is application logic, it requires a + *verified* identity (which the edge does not have until gateway auth lands — ADR-0032), + and it belongs to the gateway module (ADR-0044), not the ingress security floor. +- **Applies to the service-facing http locations** (tls-termination routes, apps, gateway + mesh routes). **Health and ACME servers are exempt** — liveness probes and cert issuance + must never be throttled; they are rendered by separate functions that carry no limit, and + the gateway's own edge health-probe locations are left unlimited too. **Forward (L4) + routes** get only `max_connections` via a `stream{}` `limit_conn` (its own zone + namespace); `requests_per_second`/`burst` do not apply at L4. A forward sharing a mixed + ssl_preread port is not connection-limited (its socket also carries the terminators' + traffic) — a documented v1 limitation. +- **The private mesh nginx (`internal/meshnginx`) is untouched.** Rate limiting is an + internet-edge concern; east-west peer traffic is already allowlist-gated. + +### CrowdSec + +- **The firewall bouncer (nftables), not the lua nginx bouncer.** The lua/AppSec bouncer + needs OpenResty and would fight the stock nginx build; the firewall bouncer enforces at + nftables from log analysis and touches no nginx config. inforge manages **no host packet + filter today** (the firewall is the Hetzner *cloud* firewall via the provider), so the + bouncer owns its own nft table with zero conflict — confirmed against the codebase. +- **Signed apt repo (packagecloud) — mirrors the nginx.org repo pattern, not the otelcol + `.deb` download.** CrowdSec ships `crowdsec` + `crowdsec-firewall-bouncer-nftables` + (with dependencies) through the `packagecloud.io/crowdsec/crowdsec` apt repository, not + as standalone checksummed `.deb`s, so the robust install adds that repo with a pinned + `signed-by` keyring exactly as `internal/nginx/install.go` adds the nginx.org repo, then + `apt-get install`s both packages. New pure package `internal/crowdsec` (stdlib-only, + Pulumi-free, like `internal/otelcol`/`internal/nginx`) renders that idempotent install + shell and the on-host config. Version pinning is optional and applies to the agent only — + the bouncer carries its own independent version scheme (0.0.x vs the agent's 1.x), so a + single pin cannot cover both; unpinned installs the repo's current compatible pair. +- **Community blocklist on by default; console optional.** `cscli capi register` (no + secret) enrolls the machine to the Central API and pulls the crowd-sourced blocklist — + the pre-emptive known-bad-IP protection that is CrowdSec's edge over fail2ban. Console + (dashboard) enrollment is a separate opt-in gated on the reserved `crowdsec_enroll` + secret. +- **inforge renders acquisition + bouncer config; hub content is fetched imperatively.** + inforge writes `/etc/crowdsec/acquis.d/nginx.yaml` (pointing at `/var/log/nginx/*.log`) + and the firewall-bouncer YAML deterministically, and leaves the package `config.yaml` + default. Hub collections (`crowdsecurity/nginx`, `base-http-scenarios`, `http-cve`) are + installed via `cscli collections install`, which is a **network-dependent, imperative** + step — a deliberate deviation from the deterministic-render norm, accepted for the same + reason ADR-0031 accepted third-party apt: not re-implementing a solved packaging + problem. `cscli` install is idempotent (re-run is a no-op/upgrade). +- **Bouncer API key: minted like the Postgres monitor role.** The firewall bouncer + authenticates to the local API with a key inforge generates (`random.RandomPassword`, + the ADR-0037 monitor-password precedent), registered on-host and written into the + bouncer config in the same pass. `cscli bouncers add` is **not idempotent** (errors if + the bouncer exists), so the bootstrap does `cscli bouncers delete inforge-fw --force || + true; cscli bouncers add inforge-fw --key ` — safe because the key is inforge-owned + and rewritten atomically alongside. The key routes through `ApplyT` + `safeTrigger` so + it is encrypted in Pulumi state and never lands in the unencrypted `Triggers` array + (enforced by the `dbrole_test.go` pattern). +- **Edges only in v1, gated on env config.** Unlike otelcol (every VM), CrowdSec installs + only on ingress + gateway hosts — where the public HTTP traffic lands. The host set is + the same `ingressHostUnion(routes, apps, health, gateways)` that `realizeIngress` + computes, so it cannot drift from where nginx actually runs. Designed so a later "every + VM" expansion (sshd brute-force coverage) is a config flip, not a rewrite. +- **No new open ports.** The local API is loopback (`127.0.0.1:8080`); the community + blocklist pull is outbound. `types.FirewallPorts` is unchanged. +- **Observable through the existing otelcol pipeline — CrowdSec's silent failure mode + demands it.** CrowdSec fails quietly: if it cannot read the nginx logs (rotation, path, + permissions) it parses nothing and bans nothing while every `cscli` command still looks + healthy. So slice 2 wires its telemetry into the pipeline inforge already runs (ADR-0031/ + 0037): enable the agent's Prometheus endpoint (`127.0.0.1:6060`) and the firewall + bouncer's loopback metrics endpoint, and add a `prometheus` receiver to the edge host's + otelcol collector — gated on the same `observability.otlp_endpoint`, exported through the + same `otlphttp` → Grafana Cloud, tagged with the ADR-0030 resource attributes, exactly as + the `postgresql` receiver is added on a cluster host (ADR-0037). The load-bearing signal + is **acquisition health** (`cs_filesource_hits_total` / `cs_parser_hits_total` rate > 0 = + CrowdSec is actually seeing traffic); the bouncer's LAPI request rate catches enforcement + drift. The deploy also + asserts liveness at apply time — `cscli lapi status` and the bouncer appearing in `cscli + bouncers list` — so a CrowdSec that did not come up fails the deploy rather than sitting + dead. Dashboards and the two built-in alerts ("acquisition stalled", "bouncer not + pulling") are a thin slice 2b once metrics flow. + +## Wiring (implementation shape) + +Grounded in the existing otelcol path (`program.provisionObservability`, +`program.go:714`) and ingress realization (`program.go:1717`): + +- **`internal/crowdsec`** — `paths.go` (constants: package names, service names, + `ConfigPath`, `BouncerConfigPath`, `AcquisPath`, `DefaultVersion`, reserved-secret + namespace/key) + `install.go` (`InstallScript`, `AcquisScript`, `BouncerScript`, + `EnrollScript`, `WriteFileScript`). Pure, Pulumi-free. +- **`types`** — new `SecurityConfig` (with `Crowdsec` + a flat `RateLimit` + `{enabled, requests_per_second, burst, max_connections}`) on `EnvironmentVariables`; a + `Security *bool` opt-out on `IngressSpec` and `GatewaySpec`. The resolved + `RateLimitProfile` (a fixed-stem struct) already rides on the ingress-derived server + structs — **no per-route spec fields, and the `IngressProvider.Realize` signature is + unchanged.** Accessors defaulting like `DashboardsEnabled()`. +- **`schemas`** — extend `ingress.json`/`gateway.json` with the boolean `security` opt-out. + The `security` block lives in `variables.yaml`, which is struct-decoded (like + `observability:`), so no JSON schema changes for the block itself. No per-route schema + changes. +- **`loader`** — normalize/trim; default the rate-limit fields. +- **`validate`** — new `checkSecurity`: rate-limit bounds (`requests_per_second > 0`, + `burst >= 0`, `max_connections >= 0`) when enabled; `crowdsec.enabled` with no edge host + in the env is an error; console enrollment configured ⇒ reserved secret present. +- **`internal/nginx/config.go`** — DONE (commit 1): emits the zones + per-location + directives from the `RateLimit` field on the server structs; render is fully unit-tested, + and a nil profile renders byte-identically to before. +- **`program.go`** — resolve the single per-edge `RateLimitProfile` from the env + `SecurityConfig` (unless the edge opted out) and stamp it on every server the ingress + derivations build for that host; then the new `provisionCrowdsec(...)` (copied from + `provisionObservability`) iterating the ingress∪gateway host set, sharing the memoized + `gates` cloud-init map, called in the scope loop beside the observability pass. +- **Docs** — public docs updated per rule `update-public-docs-with-resource-changes`; + AGENTS.md gains a "Security (edge)" section. + +## Consequences + +- A persisted bouncer-API-key file on the edge host, like the otelcol/OTLP credential — + `0600`, owned by the bouncer user. Same accepted divergence from the tmpfs-only service + secret posture (host infrastructure that must boot independently). +- Deploy now has a network dependency on the CrowdSec hub for collection install on a host + that has never fetched them; a hub outage fails the CrowdSec pass on a fresh edge (an + already-provisioned host re-runs as a no-op). +- The edge nginx.conf grows `limit_req_zone`/`limit_conn_zone` blocks; render stays + deterministic (one deduped zone per context, fixed stem). +- CrowdSec keeps local state (SQLite decisions) on the edge host — surviving reloads, + rebuilt on host replacement (bans re-learned; the community blocklist re-pulls). +- New third-party apt source on edge hosts (CrowdSec), same trade-off class as ADR-0031. + +## Future direction + +- **Per-route / per-identity limits are a gateway concern, not this layer's.** This ADR is + a blanket IP security floor. Finer limits (per route, per API key, per verified client) + are application logic that requires a *verified* identity, and are enforced at the gateway + module once authentication is at the edge — see ADR-0044 (gateway authentication tier), + which owns both the `auth_request`/dynamic-module mechanism and any identity-keyed + limiting. This ADR only notes the coupling; it intentionally ships no per-route surface. + +## Considered and rejected + +- **Cloudflare WAF at the edge** — strongest and lowest-ops, but adds external lock-in and + conflicts with the grey-cloud/origin-ACME design (orange-clouding breaks origin HTTP-01 + and needs origin-IP firewalling). Explicitly declined in favor of self-hosting; may be + revisited for the distributed-botnet tier this ADR does not cover. +- **ModSecurity + OWASP CRS / CrowdSec AppSec (true request-inspection WAF)** — needs a + version-coupled dynamic module (ModSecurity) or lua (AppSec) against an auto-updating + mainline nginx; high maintenance and false-positive tuning for a threat (dumb scanning) + the banning layer already handles. Deferred until a specific app needs signature + inspection. +- **fail2ban instead of CrowdSec** — simpler, no external dependency, but no crowd-sourced + blocklist, which is the main pre-emptive value here. +- **A standalone `waf` asset type** — over-models: enforcement must be co-located with the + edge host, so a separate placement FK invites a meaningless "WAF on a host with no + ingress" case. The env-block + per-edge opt-out gives the same control with far less + surface. + +## Host verification (2026-07-20) + +The install shell and the metric names in slice 2b were originally authored from docs. They +have since been run end-to-end on a throwaway Hetzner VM (Ubuntu 24.04 `noble`, amd64, +`crowdsec` 1.7.8 + `crowdsec-firewall-bouncer-nftables` 0.0.34), rendering the exact scripts +`internal/crowdsec` produces. Confirmed as designed: + +- **packagecloud repo** — the armored `.asc` works directly via `signed-by` with no + `gpg --dearmor`, and `.../crowdsec/crowdsec/ubuntu/ noble main` resolves; both packages install. +- **`acquis.d` drop-in** — `/etc/crowdsec/acquis.d/nginx.yaml` is read (`acquisition_dir` is a + package default) and `crowdsecurity/nginx-logs` parses the access log. Note CrowdSec tails + from end-of-file, so a source only appears in `cscli metrics` once new lines arrive. +- **`.local` overlays** — both merge; the bouncer logs `with additional values from + '…/crowdsec-firewall-bouncer.yaml.local'`. +- **cscli syntax** — `hub update`, `collections install`, `capi status`/`register`, + `bouncers add --key`/`delete`, `lapi status`, `bouncers list -o json`, and the positional + `console enroll ` all match. (The Debian package auto-registers CAPI, so the + `capi status || capi register` guard correctly no-ops.) +- **nftables enforcement** — `table ip crowdsec` / `table ip6 crowdsec6` with real `drop` + rules; a `cscli decisions add` lands in a `crowdsec-blacklists-cscli` set within seconds. +- **systemd `Restart=always`** — the drop-in applies to both units; `kill -9` respawns. +- **Metrics ports** — agent `127.0.0.1:6060` and bouncer `127.0.0.1:60601` are both correct. + +**Metric names were substantially wrong** and are corrected in `internal/grafanadash` and +`internal/grafanaalert`. The bouncer does **not** use the `cs_` prefix: + +| Assumed | Actual | +|---|---| +| `cs_reader_hits_total` | `cs_filesource_hits_total` | +| `cs_scenario_hits_total` | `cs_bucket_poured_total` | +| `cs_lapi_decisions_total` | *(no such counter — decisions are the `cs_active_decisions` gauge only)* | +| `cs_bouncer_dropped_packets_total` | `fw_bouncer_dropped_packets` | +| `cs_bouncer_dropped_bytes_total` | `fw_bouncer_dropped_bytes` | +| `cs_bouncer_last_pull` | *(does not exist — use the bouncer's `lapi_requests_total` rate)* | + +`cs_parser_hits_total` and `cs_active_decisions` were correct as written. Because there is no +decisions-added counter, no panel uses `increase()` over decisions; the "bans added" stat now +reports `fw_bouncer_banned_ips` (what the firewall actually holds). `fw_bouncer_dropped_*` are +declared `gauge` despite being cumulative — `rate()` remains correct over them. + +**Still unverified:** how the OTLP hop renames these series. The collector's `prometheus` +receiver → `otlphttp` → Grafana Cloud path may add or strip a `_total` suffix on counters; that +needs one live scrape against the real endpoint to pin down, and the queries above assume the +names pass through unchanged. diff --git a/docs/adr/0044-gateway-authentication-tier.md b/docs/adr/0044-gateway-authentication-tier.md new file mode 100644 index 0000000..1fe350f --- /dev/null +++ b/docs/adr/0044-gateway-authentication-tier.md @@ -0,0 +1,181 @@ +--- +status: proposed +date: 2026-07-19 +issue: "TBD" +--- + +# Gateway authentication tier + +Today the north-south daemon gateway (ADR-0032) routes and forwards: it TLS-terminates, +matches the request path against the derived public-path globs (ADR-0034), and hands the +request to the local mesh egress with `X-Mesh-Target`, forwarding the daemon's +`Authorization` header **untouched** for the owning service to validate. Authentication is +entirely a service concern. + +This ADR moves **authentication** to the gateway while keeping **authorization** at the +service. The gateway becomes the first gate that produces a *verified* identity; the +service still enforces what that identity may do. This is a companion to ADR-0043 (edge +security tier) — it is the change that later makes per-identity rate limiting and quotas +safe at the edge. + +## The split: authenticate at the edge, authorize at the service + +- **Authentication** — "is this a valid, unexpired, correctly-signed token from our + issuer?" — **moves to the gateway.** Unauthenticated traffic is rejected at the edge + before it consumes a backend or traverses the mesh; the token is validated once, not + re-validated in every service. +- **Authorization** — "may *this* identity perform *this* action on *this* resource?" — + **stays at the service.** Only the service knows its resource model, and keeping it there + means a gateway compromise does not grant backend access. This preserves the ADR-0032 + defense-in-depth rather than replacing it. + +The gateway is therefore **not** the sole gatekeeper. It is the first gate; the service +remains the last one, and trusts the gateway's verdict only because it arrives over the +mesh (mTLS-proven `X-Service-Identity=gateway`), never from an arbitrary caller. + +## Decisions + +- **This tier owns per-route and per-identity limits; the ingress security floor + (ADR-0043) does not.** ADR-0043 is a blanket IP rate limit applied uniformly to every + edge server as a security measure. Finer limits — per route, per API key, per verified + client — are application logic keyed on a *verified* identity, which only exists once + authentication is at the gateway. They are therefore defined and enforced here, on top of + the IP floor below. + +- **The enforcement mechanism for auth (and identity-keyed limits) is an open choice + between two shapes, both compatible with the stock-nginx render.** (a) `auth_request` → + a co-located auth service (below), or (b) a **custom nginx dynamic module** (compiled, + e.g. via `ngx-rust`) that validates the token and enforces per-route/identity limits + inline — no per-request subrequest, richest control, but it owns a module built against + each nginx version (the version-coupling cost that ruled out ModSecurity in ADR-0043). + The `auth_request` shape is the lower-risk default and the one the rest of this ADR + details; the dynamic module is the higher-ceiling alternative if the subrequest hop or + the split enforcement point ever bites. This choice is deliberately left open here. + +- **nginx stays the router; authentication is added via the built-in `auth_request` + directive — no lua/njs JWT logic in nginx.** nginx is an excellent TLS-terminating, + path-routing edge and is already rendered deterministically (`internal/nginx`, + ADR-0043). `auth_request` issues an internal subrequest to an auth endpoint before the + content phase: a `2xx` admits the request (and nginx forwards headers the auth service + returns), a `401/403` rejects it at the edge, any other status (including auth-service + unavailability) fails the request `500`. Token validation lives in a service behind that + seam, not in a per-request JS/lua blob inside the router. Rejected alternatives (njs JWT, + a full market gateway) are recorded below. + +- **The auth service is co-located on each gateway host and reached over loopback (or a + unix socket), not through the mesh.** `auth_request` fires once per incoming request, so + the auth service's latency and availability gate *every* request. A same-host loopback + call (or a unix-domain socket — no TCP port to collide or firewall) keeps that to + microseconds; routing the auth subrequest through the mesh would add an mTLS handshake + and two proxy hops to the hot path of every request. This mirrors the existing trust + boundary: the gateway already speaks only to `127.0.0.1:` (its local + mesh proxy) and trusts it because they share a host. The auth service may still be a mesh + member for its *own outbound* needs (fetching JWKS from the issuer, calling an identity + service), cached in-memory so it is not a per-request outbound; only the inbound + `auth_request` path is loopback. + +- **Build the auth service in Rust rather than adopting a market gateway.** The validation + surface is small and bounded — verify the token with a vetted library (`jsonwebtoken`, + which handles the crypto that is dangerous to hand-roll: `alg:none`, algorithm + confusion, signature verification), check `iss`/`aud`/`exp`/`nbf` with clock skew, cache + JWKS with rotation, emit the verified-identity headers, return `200`/`401`. This fits the + project's build-and-own posture (inforge, the mesh, the PKI, and the agent are all + in-house), reuses existing Rust (wardnet-cloud) and can share the daemon PKI trust model, + and deploys as just another co-located mesh member inforge already knows how to run. Auth + here is simpler than the mTLS mesh + PKI already operated. Ory Oathkeeper is the + considered off-the-shelf fallback (see below). + +- **Verified identity crosses to the service as headers, set explicitly.** The auth + subrequest's response headers are not auto-forwarded. nginx captures them with + `auth_request_set $var $upstream_http_
` and re-emits them onto the real upstream + request via `proxy_set_header` (e.g. `X-Auth-Subject`, plus any needed claim headers). + The service trusts these **only** because they arrive over the mesh from the gateway leaf + — a caller cannot inject them directly (the mesh admits only allowlisted peers, and the + gateway is a distinct mesh identity). The exact header set is a contract to fix when this + is implemented; `Authorization` continues to be forwarded so the service can re-derive + claims if it wishes. + +- **Rate limiting on a verified claim requires a second pass or auth-service enforcement — + a single-pass `limit_req` on an auth-set header silently does nothing.** nginx evaluates + `limit_req`/`limit_conn` in the **preaccess** phase, which runs *before* `auth_request` + (the **access** phase). On one pass the rate-limit key is read before the auth subrequest + runs, so a key referencing an auth-set header is empty — and nginx skips a limit whose + key is empty (it fails *open*). Two supported ways to actually key on a verified + identity: + 1. **Two-pass nginx** — the front server authenticates and stamps `X-Auth-Subject`, then + `proxy_pass`es to an internal loopback server whose preaccess `limit_req` keys on that + now-present header. One extra internal hop, idiomatic here (the ssl_preread→loopback + terminators and gateway→mesh egress are the same shape). + 2. **Enforce per-identity quotas in the auth service** — it already holds the verified + claims; it checks the counter and returns `429` from the subrequest. This is preferred + for anything richer than a leaky bucket (per-plan quotas, sliding windows) and is the + only option that scales to a global limit across nodes (next decision). + The ADR-0043 IP/volumetric layer stays in nginx preaccess, independent of auth. + +- **HA: separate limits by their state requirement — do not give nginx a global brain.** + nginx `limit_req`/`limit_conn` zones are per-host shared memory (across workers on one + box, not across hosts), so N gateway nodes enforce N independent limits. The design + splits accordingly: + - **Volumetric / IP limits stay per-node and approximate** — for flood protection, exact + global counting does not matter; per-node caps still stop a source from running wild, + and CrowdSec's nftables bans (ADR-0043) are the fleet backstop. + - **Per-identity quotas move to a shared store** — the auth service (which runs on every + gateway node and holds the verified identity) checks a **Redis-backed token bucket / + sliding window**, atomic via a Lua script, keyed by the verified client id. HA changes + only the *backing* of the counter (in-process → Redis), not the `auth_request`→`429` + shape. + - **CrowdSec bans move to a central/shared LAPI** — one node runs the local API, others + run agent+bouncer against it, so ban decisions are fleet-wide; the community blocklist + is already shared. + - **Failure posture**: the shared quota store is on the hot path, so quota checks **fail + open** — a store blip must not `503` the API — and the per-node nginx IP cap still + stands as the volumetric backstop. The two layers back each other up. + +## Request flow (target) + +``` +client ──TLS──▶ gateway nginx ──loopback──▶ auth service (200 + X-Auth-Subject | 401) + │ │ (validates JWT; JWKS cached; + │ │ optional per-identity quota via Redis) + └─(if 200, +X-Auth-Subject)──loopback──▶ mesh egress ──mTLS──▶ service + (trusts identity via + mesh; enforces authZ) +``` + +## Consequences + +- A new co-located daemon on every gateway host (the auth service), deployed and + supervised like the mesh proxy — `Restart=always` per `daemon-units-restart-on-any-exit`, + since a dead auth service fails all gateway traffic closed. +- The gateway nginx render (`gatewayServer`, `internal/nginx`) gains the `auth_request` + location and the `auth_request_set`/`proxy_set_header` identity plumbing; the two-pass + variant adds an internal server when identity-keyed rate limiting is enabled. +- A new hot-path dependency on the issuer's JWKS (cached, rotated) for the auth service, + and — only when global per-identity quotas are enabled — on a shared store (Redis). +- The ADR-0043 note "gateway auth unlocks identity keying" is made precise: the unlock is a + *verified identity existing at the edge*, enforced via a second nginx pass or in the auth + service — not `limit_req` suddenly seeing the claim on one pass. +- Authorization stays distributed across services; this ADR deliberately does not + centralize policy. A future policy-decision layer (if ever wanted) is out of scope. + +## Considered and rejected + +- **JWT validation inside nginx (njs or lua).** njs is available in the nginx.org repo and + can decode/verify a token, but it puts security-critical crypto in a per-request script + inside the router — hard to test, clutters the deterministic render, and owns JWKS + rotation / signature edge cases in the wrong place. `auth_request` to a real service is + cleaner and keeps nginx doing what it is good at. +- **A full market API gateway (Kong / Tyk / APISIX / Traefik).** Replaces the clean, + self-rendered nginx edge with a new runtime and config dialect and a larger operational + surface, to buy capabilities obtainable from `auth_request` + a small service. A step away + from the architecture, not toward it. +- **Ory Oathkeeper (kept as the buy fallback).** A purpose-built identity-&-access decision + proxy that fits the `auth_request` pattern and is config-driven. Reasonable if owning + token validation is undesirable; declined as the default only because the custom Rust + service is a small, well-understood surface consistent with everything else in-house. +- **Making the gateway the sole authenticator/authorizer.** Rejected — it would create a + single bypass point and discard the ADR-0032 service-side checks. The gateway + authenticates; the service still authorizes. +- **Global rate limiting inside nginx.** Not possible without an external store and lua/njs; + the shared-state need is met in the auth service instead, where the verified identity and + a Redis client already naturally live. diff --git a/internal/crowdsec/install.go b/internal/crowdsec/install.go new file mode 100644 index 0000000..f95f1db --- /dev/null +++ b/internal/crowdsec/install.go @@ -0,0 +1,172 @@ +package crowdsec + +import ( + "fmt" + "strings" + + "github.com/wardnet/inforge/internal/aptlock" +) + +// InstallScript adds the CrowdSec packagecloud apt repository (a pinned signed-by +// keyring, the same shape as internal/nginx's nginx.org repo) and installs the agent + +// nftables firewall bouncer. It is idempotent: the keyring is written once, the repo +// list is rewritten in place, and `apt-get install` is a no-op when both packages are +// already current. agentVersion pins the AGENT only (empty = the repo's current pair) — +// the bouncer versions independently (0.0.x vs the agent's 1.x), so one pin cannot cover +// both. The first apt call is an update (through aptlock, retrying a held lists lock) so +// a fresh image never installs against a missing index. +func InstallScript(agentVersion string) string { + agentPkg := AgentPackage + if agentVersion != "" { + agentPkg = fmt.Sprintf("%s=%s", AgentPackage, agentVersion) + } + return fmt.Sprintf(`#!/usr/bin/env bash +set -euo pipefail + +%[1]s +sudo DEBIAN_FRONTEND=noninteractive apt-get -o DPkg::Lock::Timeout=300 install -y curl ca-certificates gnupg + +. /etc/os-release + +# CrowdSec packagecloud signing key (armored .asc used directly via signed-by, no dearmor — +# which would fail on a host with no controlling tty). Download to a temp then install into +# place so a curl failure never leaves a partial keyring the presence guard would trust. +if [ ! -f %[2]s ]; then + asc=$(mktemp) + curl -fsSL %[3]s -o "$asc" + sudo install -m 0644 "$asc" %[2]s + rm -f "$asc" +fi +echo "deb [signed-by=%[2]s] https://packagecloud.io/crowdsec/crowdsec/${ID}/ ${VERSION_CODENAME} main" \ + | sudo tee %[4]s >/dev/null + +%[1]s +sudo DEBIAN_FRONTEND=noninteractive apt-get -o DPkg::Lock::Timeout=300 install -y %[5]s %[6]s + +# A dead security daemon silently stops enforcing (a crashed bouncer = no bans applied). +# The packaged units may carry no restart policy, so guarantee one via a drop-in — the same +# policy inforge writes for every unit it owns (.agents/rules/daemon-units-restart-on-any-exit.md). +for unit in %[7]s %[8]s; do + sudo install -d -m 0755 "/etc/systemd/system/${unit}.service.d" + sudo tee "/etc/systemd/system/${unit}.service.d/10-restart.conf" >/dev/null <<'UNIT' +# Managed by inforge — a daemon has no correct exit. +[Unit] +StartLimitIntervalSec=0 + +[Service] +Restart=always +RestartSec=5 +UNIT +done +sudo systemctl daemon-reload +sudo systemctl enable %[7]s %[8]s +`, aptlock.UpdateCmd(), KeyringPath, RepoKeyURL, RepoListPath, agentPkg, BouncerPackage, AgentService, BouncerService) +} + +// ConfigScript writes the agent overlays (prometheus config.yaml.local + the nginx +// acquisition drop-in), installs the hub collections, registers to the Central API for +// the community blocklist, and reloads the agent. Every step is idempotent: the files are +// rewritten, `cscli collections install` is a no-op/upgrade when present, and CAPI +// registration is skipped when already registered. Collection install and CAPI +// registration reach the CrowdSec hub/central API (a deploy-time network dependency). +func ConfigScript() string { + return strings.Join([]string{ + "set -e", + writeFileScript(AgentConfigLocal, agentPrometheusLocal(), "0644", "root:root"), + writeFileScript(AcquisPath, nginxAcquis(), "0644", "root:root"), + // Refresh the hub index first — a fresh host has none, and `collections install` + // fails ("can't find collection") without it, failing the deploy under set -e. + "sudo cscli hub update", + fmt.Sprintf("sudo cscli collections install %s", strings.Join(Collections, " ")), + // Register to the Central API (the community blocklist) unless already registered. + "sudo cscli capi status >/dev/null 2>&1 || sudo cscli capi register", + // Reload to pick up the new acquisition + collections; restart if reload is unsupported. + fmt.Sprintf("sudo systemctl reload %s 2>/dev/null || sudo systemctl restart %s", AgentService, AgentService), + }, "\n") +} + +// BouncerScript registers the inforge-owned bouncer key with the local API and writes the +// bouncer's `.local` overlay (nftables mode, the LAPI URL, the api_key, DROP action, and +// its loopback prometheus endpoint), then restarts the bouncer. apiKey must be +// YAML-safe (alphanumeric) — the program mints it with random.RandomPassword(special=off) +// — since it is written unquoted into the overlay. `cscli bouncers add` errors when the +// name already exists, so a delete-then-add makes re-deploys idempotent; the same key is +// rewritten into the overlay in this pass. The key does appear briefly in argv on +// `cscli bouncers add --key`; it is a host-local capability credential and the host is +// already root-trusted, an accepted minor exposure (cscli offers no stdin key input). +func BouncerScript(apiKey string) string { + return strings.Join([]string{ + "set -e", + fmt.Sprintf("sudo cscli bouncers delete %s >/dev/null 2>&1 || true", shQuote(BouncerName)), + fmt.Sprintf("sudo cscli bouncers add %s --key %s", shQuote(BouncerName), shQuote(apiKey)), + writeFileScript(BouncerConfigLocal, bouncerLocal(apiKey), "0600", "root:root"), + fmt.Sprintf("sudo systemctl restart %s", BouncerService), + }, "\n") +} + +// EnrollScript enrolls the agent to the CrowdSec console with the given token and +// restarts the agent. It is optional (gated on the reserved crowdsec_enroll secret). +// Console enrollment is non-fatal (an already-enrolled host re-enrolls with an error we +// tolerate), but a failure is SURFACED to the deploy log rather than silently swallowed — +// so a bad or expired token is visible instead of a green deploy that never enrolled. +func EnrollScript(token string) string { + return strings.Join([]string{ + "set -e", + fmt.Sprintf(`sudo cscli console enroll %s || echo "inforge: cscli console enroll failed (already enrolled, or a bad/expired token) — verify with 'cscli console status'" >&2`, shQuote(token)), + fmt.Sprintf("sudo systemctl restart %s", AgentService), + }, "\n") +} + +// AssertScript fails the deploy when CrowdSec did not come up: the agent's local API must +// answer, and the inforge bouncer must be registered with it. It is the deploy-time +// liveness gate (CrowdSec otherwise fails silently — ADR-0043). +func AssertScript() string { + return strings.Join([]string{ + "set -e", + // The local API needs a moment to rebind :8080 after the restart above, so poll it + // briefly before asserting — otherwise a healthy CrowdSec fails the deploy on a race. + "for i in 1 2 3 4 5; do sudo cscli lapi status >/dev/null 2>&1 && break; sleep 2; done", + "sudo cscli lapi status", // final, unsuppressed — fails the deploy if still down + fmt.Sprintf("sudo cscli bouncers list -o json | grep -q %s", shQuote(BouncerName)), + }, "\n") +} + +// agentPrometheusLocal is the agent config.yaml overlay enabling the loopback prometheus +// endpoint the host collector scrapes. +func agentPrometheusLocal() string { + return fmt.Sprintf(`# Managed by inforge (ADR-0043) — CrowdSec agent prometheus overlay. +prometheus: + enabled: true + level: full + listen_addr: %s + listen_port: %d +`, AgentMetricsAddr, AgentMetricsPort) +} + +// nginxAcquis is the acquisition drop-in pointing CrowdSec at the ingress nginx logs. +func nginxAcquis() string { + return `# Managed by inforge (ADR-0043) — CrowdSec nginx log acquisition. +source: file +filenames: + - /var/log/nginx/access.log + - /var/log/nginx/error.log +labels: + type: nginx +` +} + +// bouncerLocal is the firewall bouncer's config overlay. apiKey is written unquoted, so +// the caller must supply a YAML-safe (alphanumeric) key. +func bouncerLocal(apiKey string) string { + return fmt.Sprintf(`# Managed by inforge (ADR-0043) — CrowdSec firewall bouncer overlay. +mode: nftables +api_url: %s +api_key: %s +deny_action: DROP +disable_ipv6: false +prometheus: + enabled: true + listen_addr: %s + listen_port: %d +`, LAPIURL, apiKey, BouncerMetricsAddr, BouncerMetricsPort) +} diff --git a/internal/crowdsec/install_test.go b/internal/crowdsec/install_test.go new file mode 100644 index 0000000..4bdf936 --- /dev/null +++ b/internal/crowdsec/install_test.go @@ -0,0 +1,117 @@ +package crowdsec + +import ( + "strings" + "testing" + + "github.com/stretchr/testify/assert" +) + +func TestInstallScriptAddsRepoAndInstallsBothPackages(t *testing.T) { + s := InstallScript("") + assert.Contains(t, s, "signed-by="+KeyringPath) + assert.Contains(t, s, "packagecloud.io/crowdsec/crowdsec/${ID}/") + assert.Contains(t, s, "install -y "+AgentPackage+" "+BouncerPackage) + assert.Contains(t, s, "systemctl enable "+AgentService+" "+BouncerService) + // No version pin: the bare agent package, not agent=. + assert.NotContains(t, s, AgentPackage+"=") +} + +func TestInstallScriptPinsAgentVersionOnly(t *testing.T) { + s := InstallScript("1.6.4") + assert.Contains(t, s, "install -y "+AgentPackage+"=1.6.4 "+BouncerPackage) + // The bouncer is never pinned (independent version scheme). + assert.NotContains(t, s, BouncerPackage+"=") +} + +func TestInstallScriptGivesUnitsARestartPolicy(t *testing.T) { + s := InstallScript("") + // A crashed security daemon must self-heal, not silently stop enforcing. + assert.Contains(t, s, "Restart=always") + assert.Contains(t, s, "10-restart.conf") + assert.Contains(t, s, "systemctl daemon-reload") +} + +func TestConfigScriptInstallsCollectionsAndRegistersCAPI(t *testing.T) { + s := ConfigScript() + assert.Contains(t, s, "cscli collections install crowdsecurity/nginx crowdsecurity/base-http-scenarios crowdsecurity/http-cve") + // CAPI registration is guarded so a re-deploy of an already-registered host is a no-op. + assert.Contains(t, s, "cscli capi status >/dev/null 2>&1 || sudo cscli capi register") + // The acquisition + prometheus overlays are written (base64-transported, so their + // paths appear in the mv target). + assert.Contains(t, s, AcquisPath) + assert.Contains(t, s, AgentConfigLocal) + // The hub index is refreshed BEFORE collections install, or a fresh host can't find them. + assert.Contains(t, s, "cscli hub update") + assert.Less(t, strings.Index(s, "cscli hub update"), strings.Index(s, "cscli collections install"), + "hub update must precede collections install") +} + +func TestAssertScriptRetriesLAPIStatus(t *testing.T) { + s := AssertScript() + // Poll before the final unsuppressed check, so a just-restarted LAPI does not race. + assert.Contains(t, s, "for i in 1 2 3 4 5") + assert.Contains(t, s, "cscli lapi status") + assert.Contains(t, s, "cscli bouncers list -o json | grep -q 'inforge-fw'") +} + +func TestEnrollScriptSurfacesFailure(t *testing.T) { + s := EnrollScript("tok") + assert.Contains(t, s, "cscli console enroll 'tok'") + // Failure is logged, not silently swallowed by `|| true`. + assert.NotContains(t, s, "|| true") + assert.Contains(t, s, ">&2") +} + +func TestBouncerScriptIsIdempotentAndRestarts(t *testing.T) { + s := BouncerScript("KEY123abc") + assert.Contains(t, s, "cscli bouncers delete 'inforge-fw'") + assert.Contains(t, s, "cscli bouncers add 'inforge-fw' --key 'KEY123abc'") + assert.Contains(t, s, BouncerConfigLocal) + assert.Contains(t, s, "systemctl restart "+BouncerService) +} + +func TestAssertScriptChecksLAPIAndBouncer(t *testing.T) { + s := AssertScript() + assert.Contains(t, s, "cscli lapi status") + assert.Contains(t, s, "cscli bouncers list -o json | grep -q 'inforge-fw'") +} + +func TestBouncerLocalRendersDeterministicYAML(t *testing.T) { + const want = `# Managed by inforge (ADR-0043) — CrowdSec firewall bouncer overlay. +mode: nftables +api_url: http://127.0.0.1:8080/ +api_key: KEY123abc +deny_action: DROP +disable_ipv6: false +prometheus: + enabled: true + listen_addr: 127.0.0.1 + listen_port: 60601 +` + assert.Equal(t, want, bouncerLocal("KEY123abc")) +} + +func TestNginxAcquisPointsAtNginxLogs(t *testing.T) { + a := nginxAcquis() + assert.Contains(t, a, "/var/log/nginx/access.log") + assert.Contains(t, a, "/var/log/nginx/error.log") + assert.Contains(t, a, "type: nginx") +} + +func TestAgentPrometheusOverlayBindsLoopback(t *testing.T) { + p := agentPrometheusLocal() + assert.Contains(t, p, "enabled: true") + assert.Contains(t, p, "listen_addr: 127.0.0.1") + assert.Contains(t, p, "listen_port: 6060") +} + +// The bouncer api_key must never appear as plaintext in the config-write step — it is +// base64-transported through the shell (only the cscli argv carries it, a documented +// accepted exposure). +func TestBouncerKeyNotPlaintextInConfigWrite(t *testing.T) { + s := BouncerScript("SUPERSECRETKEY") + // The overlay content is base64'd, so the raw key appears exactly once in the whole + // script: the `cscli bouncers add --key` argv (a documented accepted exposure). + assert.Equal(t, 1, strings.Count(s, "SUPERSECRETKEY")) +} diff --git a/internal/crowdsec/paths.go b/internal/crowdsec/paths.go new file mode 100644 index 0000000..0fa0e60 --- /dev/null +++ b/internal/crowdsec/paths.go @@ -0,0 +1,58 @@ +// Package crowdsec renders the on-host install shell and configuration for the CrowdSec +// agent + nftables firewall bouncer that inforge installs on public edge hosts +// (ADR-0043). It is deploy-side only, stdlib-only, and holds no provider/Pulumi +// dependencies — the program wires it to edge hosts, exactly as internal/nginx and +// internal/otelcol are wired. The rendering is pure (config/params in, shell string out) +// and deterministic, so the same inputs always produce the same bytes. +package crowdsec + +const ( + // AgentPackage / BouncerPackage are the apt packages installed from the CrowdSec + // packagecloud repository. BouncerService/AgentService are their systemd units. + AgentPackage = "crowdsec" + BouncerPackage = "crowdsec-firewall-bouncer-nftables" + AgentService = "crowdsec" + BouncerService = "crowdsec-firewall-bouncer" + + // LAPIURL is the loopback local API the agent serves and the bouncer authenticates to. + LAPIURL = "http://127.0.0.1:8080/" + + // BouncerName is the stable inforge-owned bouncer identity registered with the LAPI. + BouncerName = "inforge-fw" + + // On-host config paths. inforge writes `.local` overlays and an acquis.d drop-in + // rather than editing the packaged files, so a package upgrade never reverts our + // config (CrowdSec merges *.local over the base config, and reads acquis.d/*.yaml). + AgentConfigLocal = "/etc/crowdsec/config.yaml.local" + AcquisPath = "/etc/crowdsec/acquis.d/nginx.yaml" + BouncerConfigLocal = "/etc/crowdsec/bouncers/crowdsec-firewall-bouncer.yaml.local" + + // Prometheus metrics endpoints (loopback only), scraped by the host otelcol collector + // via a prometheus receiver (ADR-0043, mirroring the ADR-0037 postgresql receiver). + AgentMetricsAddr = "127.0.0.1" + AgentMetricsPort = 6060 + BouncerMetricsAddr = "127.0.0.1" + BouncerMetricsPort = 60601 + + // packagecloud apt repo, added with a pinned signed-by keyring exactly as + // internal/nginx adds the nginx.org repo (an armored .asc used directly, no dearmor). + RepoKeyURL = "https://packagecloud.io/crowdsec/crowdsec/gpgkey" + KeyringPath = "/usr/share/keyrings/crowdsec_crowdsec-archive-keyring.asc" + RepoListPath = "/etc/apt/sources.list.d/crowdsec.list" + + // ReservedNamespace/EnrollSecretKey locate the optional console enrollment token in + // the env's secrets.enc.yaml — an inforge reserved secret (ADR-0043), mirroring + // otelcol.AuthSecretNamespace/AuthSecretKey. The community blocklist needs no secret. + ReservedNamespace = "security" + EnrollSecretKey = "crowdsec_enroll" // #nosec G101 -- a lookup key name, not a credential +) + +// Collections are the CrowdSec hub items installed on an nginx edge: nginx access/error +// log parsing + scenarios, the generic HTTP attack scenarios, and known-CVE probing. +// They are installed imperatively by cscli, which fetches them from the CrowdSec hub — +// a deliberate network dependency at deploy time (ADR-0043). +var Collections = []string{ + "crowdsecurity/nginx", + "crowdsecurity/base-http-scenarios", + "crowdsecurity/http-cve", +} diff --git a/internal/crowdsec/shell.go b/internal/crowdsec/shell.go new file mode 100644 index 0000000..0e46d8c --- /dev/null +++ b/internal/crowdsec/shell.go @@ -0,0 +1,40 @@ +package crowdsec + +import ( + "encoding/base64" + "fmt" + "path" + "strings" +) + +// shQuote single-quotes s for safe interpolation into a POSIX shell command, escaping +// any embedded single quote as the standard '\'' sequence. The package renders host +// shell but stays Pulumi-free (pure, like internal/nginx and internal/otelcol), so it +// carries its own minimal quoting rather than importing the remote helpers. +func shQuote(s string) string { + return "'" + strings.ReplaceAll(s, "'", `'\''`) + "'" +} + +// base64Encode returns the standard base64 of s, for transporting arbitrary bytes +// (including a secret) through a shell command without exposing them in argv. +func base64Encode(s string) string { + return base64.StdEncoding.EncodeToString([]byte(s)) +} + +// writeFileScript renders sudo shell that writes content to absPath with the given mode +// and owner, creating the parent dir. content is base64-encoded for transport so +// arbitrary bytes (including a secret) survive intact and never appear in argv as +// plaintext. The write stages to a temp then atomic-mv's into place (safe on coreutils 9 +// where `install /dev/stdin ` fails), mirroring internal/otelcol. +func writeFileScript(absPath, content, mode, owner string) string { + enc := base64Encode(content) + dir := path.Dir(absPath) + return strings.Join([]string{ + fmt.Sprintf("sudo install -d -m 0755 %s", shQuote(dir)), + fmt.Sprintf(`tmp=$(sudo mktemp %s)`, shQuote(dir+"/.inforge-write.XXXXXX")), + fmt.Sprintf(`printf %%s %s | base64 -d | sudo tee "$tmp" >/dev/null`, shQuote(enc)), + fmt.Sprintf(`sudo chmod %s "$tmp"`, shQuote(mode)), + fmt.Sprintf(`sudo chown %s "$tmp"`, shQuote(owner)), + fmt.Sprintf(`sudo mv "$tmp" %s`, shQuote(absPath)), + }, "\n") +} diff --git a/internal/grafanaalert/crowdsec.go b/internal/grafanaalert/crowdsec.go new file mode 100644 index 0000000..5a91f48 --- /dev/null +++ b/internal/grafanaalert/crowdsec.go @@ -0,0 +1,41 @@ +package grafanaalert + +import "fmt" + +// CrowdsecBuiltIns returns the built-in CrowdSec alert rules for one env (ADR-0043), +// generated from the CrowdSec agent + firewall-bouncer metrics scraped on edge hosts. +// Like the Postgres built-ins, the caller gates these on CrowdSec actually being enabled +// (security.crowdsec.enabled) so they cannot evaluate on an env that runs no CrowdSec. +// +// Both metric names are confirmed against a real host (crowdsec 1.7.8 + +// crowdsec-firewall-bouncer-nftables 0.0.34): the agent exposes cs_parser_hits_total on +// :6060 and the bouncer exposes lapi_requests_total on :60601. Note the bouncer's series +// are prefixed fw_bouncer_*, NOT cs_bouncer_*, and it publishes no last-pull timestamp — +// its LAPI call counter is the pull-freshness signal instead. +// +// Both still use NoDataOK, not NoDataAlerting: an edge that is scraped but not yet +// reporting (or an env between deploys) must not page. These alert only on a +// present-but-bad signal; absence is covered by the collector's own up/health signal. +func CrowdsecBuiltIns(env string) []Alert { + e := envMatcher(env) + return []Alert{ + // Acquisition stalled: CrowdSec is present but parsing ~no log lines, so it is + // blind to attacks. A genuinely idle edge can also read zero — the threshold is + // fixed (like every built-in); opt out and re-author if that is expected. + b("CrowdSec Acquisition Stalled", + fmt.Sprintf(`sum by (instance) (rate(cs_parser_hits_total{%s}[10m]))`, e), + "< 0.001", "15m", SeverityWarning, + "CrowdSec on {{ $labels.instance }} is parsing no log lines — acquisition has stalled (it is blind to attacks).", + NoDataOK), + + // Bouncer not pulling: the firewall bouncer has stopped calling the local API, so new + // bans are not reaching nftables and enforcement is drifting. The bouncer polls + // /v1/decisions/stream continuously, so a flat-zero request rate is the stall signal + // (it publishes no last-pull timestamp to compare against time()). + b("CrowdSec Bouncer Not Pulling", + fmt.Sprintf(`sum by (instance) (rate(lapi_requests_total{%s}[10m]))`, e), + "< 0.001", "10m", SeverityWarning, + "CrowdSec firewall bouncer on {{ $labels.instance }} has not called the local API in over 10m — new bans are not being enforced.", + NoDataOK), + } +} diff --git a/internal/grafanaalert/crowdsec_test.go b/internal/grafanaalert/crowdsec_test.go new file mode 100644 index 0000000..a6b2630 --- /dev/null +++ b/internal/grafanaalert/crowdsec_test.go @@ -0,0 +1,40 @@ +package grafanaalert + +import ( + "strings" + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" +) + +func TestCrowdsecBuiltIns(t *testing.T) { + as := CrowdsecBuiltIns("prd") + require.Len(t, as, 2) + + by := map[string]Alert{} + for _, a := range as { + by[a.Rule.Name] = a + } + require.Contains(t, by, "CrowdSec Acquisition Stalled") + require.Contains(t, by, "CrowdSec Bouncer Not Pulling") + + for _, a := range as { + // Warnings, env-scoped, and NoDataOK — an edge that is scraped but not yet + // reporting must not page; they alert only on a present-but-bad signal. + assert.Equal(t, SeverityWarning, a.Severity, a.Rule.Name) + assert.Equal(t, NoDataOK, a.Rule.NoDataState, a.Rule.Name) + require.NotEmpty(t, a.Rule.Data) + assert.Contains(t, a.Rule.Data[0].Model, `deployment_environment_name=\"prd\"`, a.Rule.Name) + } + + // The acquisition alert fires on a low parse rate; the bouncer alert on a stalled LAPI + // request rate. Both names are host-verified (ADR-0043 "Host verification"). + assert.Contains(t, by["CrowdSec Acquisition Stalled"].Rule.Data[0].Model, "cs_parser_hits_total") + assert.Contains(t, by["CrowdSec Bouncer Not Pulling"].Rule.Data[0].Model, "lapi_requests_total") + assert.True(t, strings.HasPrefix(by["CrowdSec Acquisition Stalled"].Rule.Data[0].Model, "{"), "model is JSON") + + // cs_bouncer_last_pull does not exist on a real bouncer — an alert on it could never + // fire, which is exactly the silent failure this rule is meant to catch. + assert.NotContains(t, by["CrowdSec Bouncer Not Pulling"].Rule.Data[0].Model, "cs_bouncer_last_pull") +} diff --git a/internal/grafanadash/crowdsec.go b/internal/grafanadash/crowdsec.go new file mode 100644 index 0000000..43564cd --- /dev/null +++ b/internal/grafanadash/crowdsec.go @@ -0,0 +1,79 @@ +package grafanadash + +import "fmt" + +// Crowdsec renders the built-in CrowdSec dashboard for one env (ADR-0043) from the agent +// + firewall-bouncer metrics scraped on edge hosts (service.name wardnet-crowdsec-metrics): +// an overview (active bans, acquisition health, edges reporting), acquisition/detection +// detail, decisions, and the firewall bouncer's drops + LAPI pull activity. uid is the +// env-prefixed dashboard UID (grafana.DashboardUID). +// +// Every metric name below was confirmed against a real host (crowdsec 1.7.8 + +// crowdsec-firewall-bouncer-nftables 0.0.34 on Ubuntu noble). Two naming facts are easy to +// get wrong and are load-bearing here: +// - The AGENT exposes cs_* on :6060, but the BOUNCER exposes fw_bouncer_* (NOT cs_bouncer_*) +// with no _total suffix, plus a bare lapi_requests_total, on :60601. +// - There is no cs_scenario_hits_total / cs_reader_hits_total / cs_lapi_decisions_total — +// the real series are cs_bucket_poured_total, cs_filesource_hits_total and the +// cs_active_decisions gauge. Decisions are a GAUGE only; no add-counter exists, so no +// panel here uses increase() over decisions. +// +// fw_bouncer_dropped_* are declared `gauge` though they are cumulative totals; rate() is +// still correct over them (it treats a decrease as a reset), which is what the drop panels use. +func Crowdsec(env, uid string) (string, error) { + e := envMatcher(env) + inst := `instance=~"$instance"` + ri := "$__rate_interval" + + vars := []map[string]any{queryVar("instance", "Edge", "cs_parser_hits_total", e)} + + panels := []map[string]any{ + row("Overview", 0), + stat("Active Decisions", gp(0, 1, 5, 4), + []map[string]any{target(fmt.Sprintf("sum(cs_active_decisions{%s})", e), "__auto")}, + "none", "Currently active IP ban decisions across all edges.", nil), + // CrowdSec publishes no decisions-added counter, so this reports what the firewall + // is actually enforcing (bouncer-side) next to what LAPI knows (cs_active_decisions). + stat("IPs Banned at Firewall", gp(5, 1, 5, 4), + []map[string]any{target(fmt.Sprintf("sum(fw_bouncer_banned_ips{%s})", e), "__auto")}, + "none", "IP addresses currently loaded into the edge firewall's nftables ban sets.", nil), + stat("Acquisition Rate", gp(10, 1, 7, 4), + []map[string]any{target(fmt.Sprintf("sum(rate(cs_parser_hits_total{%s}[%s]))", e, ri), "__auto")}, + "cps", "Log lines parsed per second — the health signal. Zero on a busy edge means CrowdSec is blind.", nil), + stat("Edges Reporting", gp(17, 1, 7, 4), + []map[string]any{target(fmt.Sprintf("count(count by (instance) (cs_parser_hits_total{%s}))", e), "__auto")}, + "none", "Edge hosts reporting CrowdSec metrics.", nil), + + row("Acquisition & Detection", 5), + ts("Log acquisition rate by source", gp(0, 6, 12, 8), + targets([2]string{fmt.Sprintf("sum by (source) (rate(cs_filesource_hits_total{%s}[%s]))", e, ri), "{{source}}"}), + "cps"), + ts("Scenario hits", gp(12, 6, 12, 8), + targets([2]string{fmt.Sprintf("sum by (name) (rate(cs_bucket_poured_total{%s}[%s]))", e, ri), "{{name}}"}), + "cps"), + + row("Decisions", 14), + ts("Active decisions by reason", gp(0, 15, 12, 8), + targets([2]string{fmt.Sprintf("sum by (reason) (cs_active_decisions{%s})", e), "{{reason}}"}), + "none"), + ts("Active decisions by origin", gp(12, 15, 12, 8), + targets([2]string{fmt.Sprintf("sum by (origin) (cs_active_decisions{%s})", e), "{{origin}}"}), + "none"), + + row("Firewall Bouncer", 23), + ts("Bouncer dropped packets/s by edge", gp(0, 24, 12, 8), + targets([2]string{fmt.Sprintf("sum by (instance) (rate(fw_bouncer_dropped_packets{%s, %s}[%s]))", e, inst, ri), "{{instance}}"}), + "cps"), + ts("Bouncer dropped bytes/s by edge", gp(12, 24, 12, 8), + targets([2]string{fmt.Sprintf("sum by (instance) (rate(fw_bouncer_dropped_bytes{%s, %s}[%s]))", e, inst, ri), "{{instance}}"}), + "Bps"), + // The bouncer exposes no last-pull timestamp; its LAPI call counter is the freshness + // signal instead — a healthy bouncer polls /v1/decisions/stream continuously, so a + // flat-zero rate means it has stopped pulling and enforcement is drifting. + ts("Bouncer LAPI pull rate by edge", gp(0, 32, 24, 6), + targets([2]string{fmt.Sprintf("sum by (instance) (rate(lapi_requests_total{%s, %s}[%s]))", e, inst, ri), "{{instance}}"}), + "reqps"), + } + + return dashboard("CrowdSec Monitoring", uid, []string{"inforge", "otel", "crowdsec"}, vars, panels) +} diff --git a/internal/grafanadash/crowdsec_test.go b/internal/grafanadash/crowdsec_test.go new file mode 100644 index 0000000..fa18458 --- /dev/null +++ b/internal/grafanadash/crowdsec_test.go @@ -0,0 +1,55 @@ +package grafanadash + +import ( + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" +) + +func TestCrowdsecRenders(t *testing.T) { + out, err := Crowdsec("prd", "inforge-prd-crowdsec") + require.NoError(t, err) + m := parse(t, out) + + assert.Equal(t, "inforge-prd-crowdsec", m["uid"]) + assert.Equal(t, "CrowdSec Monitoring", m["title"]) + // Every query is env-scoped, and the CrowdSec signals + sections are present. + assert.Contains(t, out, `deployment_environment_name=\"prd\"`) + // These names are verified against a real host (crowdsec 1.7.8 + firewall-bouncer + // 0.0.34) — see the "Host verification" section of ADR-0043. Note the bouncer's + // series are fw_bouncer_* with no _total suffix, NOT cs_bouncer_*. + for _, want := range []string{ + "Overview", "Acquisition \\u0026 Detection", "Decisions", "Firewall Bouncer", + "cs_active_decisions", + "cs_parser_hits_total", + "cs_filesource_hits_total", + "cs_bucket_poured_total", + "fw_bouncer_banned_ips", + "fw_bouncer_dropped_packets", + "fw_bouncer_dropped_bytes", + "lapi_requests_total", + `"label":"Edge"`, + } { + assert.Contains(t, out, want) + } + + // Guard the exact mistakes host verification caught: these series do not exist, and a + // panel querying one silently shows "No data" forever. + for _, absent := range []string{ + "cs_reader_hits_total", + "cs_scenario_hits_total", + "cs_lapi_decisions_total", + "cs_bouncer_", + } { + assert.NotContains(t, out, absent) + } +} + +func TestCrowdsecDeterministic(t *testing.T) { + a, err := Crowdsec("prd", "u") + require.NoError(t, err) + b, err := Crowdsec("prd", "u") + require.NoError(t, err) + assert.Equal(t, a, b, "render must be byte-stable") +} diff --git a/internal/nginx/config.go b/internal/nginx/config.go index 3910dde..0ca6e9d 100644 --- a/internal/nginx/config.go +++ b/internal/nginx/config.go @@ -298,6 +298,19 @@ func httpBlock(terminate []types.IngressRoute, apps []types.IngressApp, health [ dir("real_ip_header", "proxy_protocol"), ) } + // Rate-limit zones for every profile an http server on this host references, + // declared once in http{} before the servers that consume them (ADR-0043). + var httpProfiles []*types.RateLimitProfile + for _, r := range terminate { + httpProfiles = append(httpProfiles, r.RateLimit) + } + for _, a := range apps { + httpProfiles = append(httpProfiles, a.RateLimit) + } + for _, g := range gateways { + httpProfiles = append(httpProfiles, g.RateLimit) + } + children = append(children, httpRateLimitZones(httpProfiles)...) for _, r := range terminate { children = append(children, terminateServer(r, listenDir(r.Listen))) } @@ -333,6 +346,12 @@ func httpBlock(terminate []types.IngressRoute, apps []types.IngressApp, health [ // accepting the PROXY protocol). Render has already verified r.Backend is non-empty. func terminateServer(r types.IngressRoute, listen *crossplane.Directive) *crossplane.Directive { serverName := append([]string{}, r.FQDNs...) + loc := append(rlHTTPLimitDirs(r.RateLimit), + dir("proxy_pass", fmt.Sprintf("http://%s:%d", r.Backend, r.Target)), + dir("proxy_set_header", "Host", "$host"), + dir("proxy_set_header", "X-Forwarded-For", "$proxy_add_x_forwarded_for"), + dir("proxy_set_header", "X-Forwarded-Proto", "$scheme"), + ) return block("server", nil, listen, dir("server_name", serverName...), @@ -340,12 +359,7 @@ func terminateServer(r types.IngressRoute, listen *crossplane.Directive) *crossp dir("ssl_certificate", "$acme_certificate"), dir("ssl_certificate_key", "$acme_certificate_key"), dir("ssl_certificate_cache", "max=2"), - block("location", []string{"/"}, - dir("proxy_pass", fmt.Sprintf("http://%s:%d", r.Backend, r.Target)), - dir("proxy_set_header", "Host", "$host"), - dir("proxy_set_header", "X-Forwarded-For", "$proxy_add_x_forwarded_for"), - dir("proxy_set_header", "X-Forwarded-Proto", "$scheme"), - ), + block("location", []string{"/"}, loc...), ) } @@ -359,6 +373,7 @@ func appServer(a types.IngressApp, listen *crossplane.Directive) *crossplane.Dir if a.Spa { fallback = []string{"$uri", "$uri/", "/index.html"} } + loc := append(rlHTTPLimitDirs(a.RateLimit), dir("try_files", fallback...)) return block("server", nil, listen, dir("server_name", a.FQDN), @@ -368,9 +383,7 @@ func appServer(a types.IngressApp, listen *crossplane.Directive) *crossplane.Dir dir("ssl_certificate_cache", "max=2"), dir("root", a.Root), dir("index", "index.html"), - block("location", []string{"/"}, - dir("try_files", fallback...), - ), + block("location", []string{"/"}, loc...), ) } @@ -423,23 +436,22 @@ func gatewayServer(g types.IngressGateway, listen *crossplane.Directive, regexOf )) } for _, rt := range routes { - children = append(children, - block("location", []string{"~", regexOf[rt.Pattern]}, - dir("proxy_http_version", "1.1"), - dir("proxy_set_header", "Upgrade", "$http_upgrade"), - dir("proxy_set_header", "Connection", "$connection_upgrade"), - dir("proxy_set_header", "Host", "$host"), - // SET, never append: this is the internet-facing first hop, so a - // daemon-supplied X-Forwarded-For is untrusted input — appending - // ($proxy_add_x_forwarded_for) would forward a forged first entry - // through the mesh and defeat per-IP audit/rate-limit logic. - dir("proxy_set_header", "X-Forwarded-For", "$remote_addr"), - dir("proxy_set_header", "X-Forwarded-Proto", "$scheme"), - dir("proxy_set_header", "X-Mesh-Target", rt.Service), - dir("proxy_read_timeout", "3600s"), - dir("proxy_pass", fmt.Sprintf("http://127.0.0.1:%d", meshpaths.GatewayEgressPort)), - ), + loc := append(rlHTTPLimitDirs(g.RateLimit), + dir("proxy_http_version", "1.1"), + dir("proxy_set_header", "Upgrade", "$http_upgrade"), + dir("proxy_set_header", "Connection", "$connection_upgrade"), + dir("proxy_set_header", "Host", "$host"), + // SET, never append: this is the internet-facing first hop, so a + // daemon-supplied X-Forwarded-For is untrusted input — appending + // ($proxy_add_x_forwarded_for) would forward a forged first entry + // through the mesh and defeat per-IP audit/rate-limit logic. + dir("proxy_set_header", "X-Forwarded-For", "$remote_addr"), + dir("proxy_set_header", "X-Forwarded-Proto", "$scheme"), + dir("proxy_set_header", "X-Mesh-Target", rt.Service), + dir("proxy_read_timeout", "3600s"), + dir("proxy_pass", fmt.Sprintf("http://127.0.0.1:%d", meshpaths.GatewayEgressPort)), ) + children = append(children, block("location", []string{"~", regexOf[rt.Pattern]}, loc...)) } children = append(children, jsonNotFoundLocation()) return block("server", nil, children...) @@ -505,6 +517,20 @@ func healthServer(h types.IngressHealth, healthPort int) *crossplane.Directive { // ports keep the plain raw-L4 server. func streamBlock(forwardOnly []types.IngressRoute, mixed []mixedPort) *crossplane.Directive { var children crossplane.Directives + // Stream limit_conn zones for the forward servers carrying a rate-limit profile, + // declared once at stream{} scope before the servers use them (ADR-0043). Only + // connection limiting applies at L4 — there is no request rate to shape. A forward + // sharing a mixed port is fronted by the ssl_preread server and is not limited here + // (its socket also carries the terminators' traffic); only forward-only ports are. + var streamProfiles []*types.RateLimitProfile + for _, r := range forwardOnly { + streamProfiles = append(streamProfiles, r.RateLimit) + } + for _, p := range dedupeProfiles(streamProfiles) { + if p.MaxConn > 0 { + children = append(children, dir("limit_conn_zone", "$binary_remote_addr", "zone="+rlStreamConnZone(p.Name)+":"+rlZoneSize)) + } + } for _, mp := range mixed { entries := make(crossplane.Directives, 0, len(mp.fqdns)+1) for _, fqdn := range mp.fqdns { @@ -525,15 +551,103 @@ func streamBlock(forwardOnly []types.IngressRoute, mixed []mixedPort) *crossplan )) } for _, r := range forwardOnly { - children = append(children, block("server", nil, - dir("listen", strconv.Itoa(r.Listen)), + srv := crossplane.Directives{dir("listen", strconv.Itoa(r.Listen))} + srv = append(srv, rlStreamLimitDirs(r.RateLimit)...) + srv = append(srv, dir("proxy_pass", fmt.Sprintf("%s:%d", r.Backend, r.Target)), dir("proxy_protocol", "on"), - )) + ) + children = append(children, block("server", nil, srv...)) } return block("stream", nil, children...) } +// Rate limiting (ADR-0043). Each referenced profile becomes one shared-memory zone in +// http{} (and/or stream{}); every server carrying the profile emits the matching +// limit_req/limit_conn directives inside its location. Keying is always the client IP +// ($binary_remote_addr) — accurate here because set_real_ip recovers the true address on +// ssl_preread'd mixed ports in POST_READ, before the preaccess phase where limit_req runs. +const rlZoneSize = "10m" + +// rlReqZone/rlConnZone/rlStreamConnZone name the shared-memory zones for a profile. +// The prefixes keep them clear of other nginx names and keep the http and stream +// limit_conn zones distinct (separate nginx contexts, rendered independently). +func rlReqZone(name string) string { return "rl_" + name } +func rlConnZone(name string) string { return "rlc_" + name } +func rlStreamConnZone(name string) string { return "rls_" + name } + +// rlHTTPLimitDirs are the per-location limit directives for an http server carrying +// profile p (nil -> none). limit_req is emitted only when the profile sets a rate, +// limit_conn only when it caps connections — so a profile may do either or both. +func rlHTTPLimitDirs(p *types.RateLimitProfile) []*crossplane.Directive { + if p == nil { + return nil + } + var d []*crossplane.Directive + if p.RPS > 0 { + d = append(d, dir("limit_req", "zone="+rlReqZone(p.Name), fmt.Sprintf("burst=%d", p.Burst), "nodelay")) + } + if p.MaxConn > 0 { + d = append(d, dir("limit_conn", rlConnZone(p.Name), strconv.Itoa(p.MaxConn))) + } + return d +} + +// rlStreamLimitDirs are the per-server limit directives for a stream (forward) server: +// only connection limiting applies at L4 (there is no request rate to shape). +func rlStreamLimitDirs(p *types.RateLimitProfile) []*crossplane.Directive { + if p == nil || p.MaxConn <= 0 { + return nil + } + return []*crossplane.Directive{dir("limit_conn", rlStreamConnZone(p.Name), strconv.Itoa(p.MaxConn))} +} + +// dedupeProfiles returns the distinct profiles among the pointers, sorted by name, so +// zone emission is deterministic and each zone is declared exactly once per context. +func dedupeProfiles(ps []*types.RateLimitProfile) []*types.RateLimitProfile { + seen := map[string]*types.RateLimitProfile{} + for _, p := range ps { + if p != nil { + seen[p.Name] = p + } + } + names := make([]string, 0, len(seen)) + for n := range seen { + names = append(names, n) + } + sort.Strings(names) + out := make([]*types.RateLimitProfile, 0, len(names)) + for _, n := range names { + out = append(out, seen[n]) + } + return out +} + +// httpRateLimitZones renders the http{} shared-memory zones plus the status overrides +// for every profile an http server references. limit_req_status/limit_conn_status make a +// throttled request answer 429 (CrowdSec-parseable) rather than nginx's default 503. +func httpRateLimitZones(profiles []*types.RateLimitProfile) []*crossplane.Directive { + var out []*crossplane.Directive + anyReq, anyConn := false, false + for _, p := range dedupeProfiles(profiles) { + if p.RPS > 0 { + out = append(out, dir("limit_req_zone", "$binary_remote_addr", "zone="+rlReqZone(p.Name)+":"+rlZoneSize, fmt.Sprintf("rate=%dr/s", p.RPS))) + anyReq = true + } + if p.MaxConn > 0 { + out = append(out, dir("limit_conn_zone", "$binary_remote_addr", "zone="+rlConnZone(p.Name)+":"+rlZoneSize)) + anyConn = true + } + } + if anyReq { + out = append(out, dir("limit_req_status", "429")) + } + if anyConn { + out = append(out, dir("limit_conn_status", "429")) + } + return out +} + // sortedIntKeys returns the keys of an int set as an ascending slice. func sortedIntKeys(set map[int]bool) []int { out := make([]int, 0, len(set)) diff --git a/internal/nginx/config_test.go b/internal/nginx/config_test.go index 6926a43..f06172b 100644 --- a/internal/nginx/config_test.go +++ b/internal/nginx/config_test.go @@ -76,6 +76,68 @@ stream { assert.Equal(t, want, got) } +// TestRenderRateLimit checks the ADR-0043 IP rate-limit rendering: one uniform limit +// stamped on a tls-termination route (http limit_req + limit_conn) and a forward route +// (stream limit_conn), with each shared-memory zone declared exactly once per context +// and throttled requests answered 429 rather than nginx's default 503. +func TestRenderRateLimit(t *testing.T) { + rl := &types.RateLimitProfile{Name: "edge", RPS: 20, Burst: 40, MaxConn: 40} + routes := []types.IngressRoute{ + {Service: "dns", Type: types.IngressTypeForward, Listen: 853, Target: 5353, Backend: "127.0.0.1", RateLimit: rl}, + {Service: "api", Type: types.IngressTypeTLSTermination, Listen: 443, Target: 8080, Backend: "127.0.0.1", + FQDNs: []string{"api.svc.prd.use1.wardnet.network"}, RateLimit: rl}, + } + got, err := Render(routes, nil, nil, 0, nil) + require.NoError(t, err) + + // http{} zones, declared once, with the 429 status overrides. + assert.Contains(t, got, "limit_req_zone $binary_remote_addr zone=rl_edge:10m rate=20r/s;") + assert.Contains(t, got, "limit_conn_zone $binary_remote_addr zone=rlc_edge:10m;") + assert.Contains(t, got, "limit_req_status 429;") + assert.Contains(t, got, "limit_conn_status 429;") + assert.Equal(t, 1, strings.Count(got, "limit_req_zone $binary_remote_addr zone=rl_edge"), "req zone declared once") + + // per-location http limits on the tls-termination server. + assert.Contains(t, got, "limit_req zone=rl_edge burst=40 nodelay;") + assert.Contains(t, got, "limit_conn rlc_edge 40;") + + // stream (forward) connection limit — its own zone namespace. + assert.Contains(t, got, "limit_conn_zone $binary_remote_addr zone=rls_edge:10m;") + assert.Contains(t, got, "limit_conn rls_edge 40;") +} + +// TestRenderRateLimitExemptsHealthAndACME confirms the limit lands only on the +// service-facing locations: the public health servers and the :80 ACME/redirect server +// are never rate-limited (they are rendered by separate functions that carry no profile). +func TestRenderRateLimitExemptsHealthAndACME(t *testing.T) { + rl := &types.RateLimitProfile{Name: "edge", RPS: 20, Burst: 40, MaxConn: 40} + routes := []types.IngressRoute{ + {Service: "api", Type: types.IngressTypeTLSTermination, Listen: 443, Target: 8080, Backend: "127.0.0.1", + FQDNs: []string{"api.svc.prd.use1.wardnet.network"}, RateLimit: rl}, + } + health := []types.IngressHealth{ + {Service: "api", FQDN: "api.svc.prd.use1.wardnet.network", Target: 9000, Backend: "127.0.0.1", Paths: []string{"/healthz"}}, + } + got, err := Render(routes, nil, health, 81, nil) + require.NoError(t, err) + // Exactly one limited location — the tls-termination route. Health + ACME are exempt. + assert.Equal(t, 1, strings.Count(got, "limit_req zone=rl_edge")) + assert.Contains(t, got, "listen 81;") // the health server exists... +} + +// TestRenderNoRateLimit guards the golden's premise: a nil profile emits no limit +// directives at all, so an env without rate limiting renders byte-identically to before. +func TestRenderNoRateLimit(t *testing.T) { + routes := []types.IngressRoute{ + {Service: "api", Type: types.IngressTypeTLSTermination, Listen: 443, Target: 8080, Backend: "127.0.0.1", + FQDNs: []string{"api.svc.prd.use1.wardnet.network"}}, + } + got, err := Render(routes, nil, nil, 0, nil) + require.NoError(t, err) + assert.NotContains(t, got, "limit_req") + assert.NotContains(t, got, "limit_conn") +} + // TestRenderMixedPreread pins the mixed-port shape: two tls-termination services // and one forward (passthrough) sharing listen 443. The public :443 moves into a // stream{} ssl_preread server that maps known SNIs to an internal loopback diff --git a/internal/otelcol/config.go b/internal/otelcol/config.go index 15f3efd..aadca59 100644 --- a/internal/otelcol/config.go +++ b/internal/otelcol/config.go @@ -23,6 +23,13 @@ const ServiceName = "wardnet-host-metrics" // last of which becomes meaningful once a cluster spans multiple hosts. const DBServiceName = "wardnet-db-metrics" +// CrowdsecServiceName is the OTel service.name (Prometheus job) stamped on CrowdSec +// metrics scraped on edge hosts (ADR-0043) — a distinct job from host and DB metrics so +// CrowdSec dashboards/alerts scope cleanly, still correlating with them on host.id. The +// load-bearing signal is acquisition health (parse rate > 0 = CrowdSec is actually seeing +// nginx traffic), which catches its silent-failure mode. +const CrowdsecServiceName = "wardnet-crowdsec-metrics" + // CollectionInterval is how often the hostmetrics receiver scrapes. const CollectionInterval = "60s" @@ -104,7 +111,11 @@ func identityAttrs(attrs Attributes) []map[string]any { // to a host-metrics-only collector. Each pg target gets its own receiver + metrics // pipeline whose resource processor stamps service.name=DBServiceName, // service.instance.id=HostID, db.cluster.name=, plus the shared identity. -func Render(endpoint string, attrs Attributes, pg []PostgresTarget) (string, error) { +// crowdsecScrape, when non-empty, adds a prometheus receiver scraping those loopback +// "host:port" endpoints (the CrowdSec agent + firewall bouncer, ADR-0043) on their own +// job/pipeline; empty leaves the config byte-identical. It is variadic so the many +// host-metrics-only callers stay unchanged. +func Render(endpoint string, attrs Attributes, pg []PostgresTarget, crowdsecScrape ...string) (string, error) { if endpoint == "" { return "", fmt.Errorf("otelcol: empty OTLP endpoint") } @@ -212,6 +223,33 @@ func Render(endpoint string, attrs Attributes, pg []PostgresTarget) (string, err } } + // CrowdSec metrics (ADR-0043): one prometheus receiver scraping the agent + firewall + // bouncer loopback endpoints on an edge host, on its own job + pipeline. The resource + // processor overwrites the scrape-derived job/instance with a fixed CrowdSec job name + // and the per-node HostID (like the pg targets) plus the shared identity, so CrowdSec + // health correlates with host + app telemetry on host.id. Empty targets = no receiver. + if len(crowdsecScrape) > 0 { + receivers["prometheus/crowdsec"] = map[string]any{ + "config": map[string]any{ + "scrape_configs": []map[string]any{{ + "job_name": "crowdsec", + "scrape_interval": CollectionInterval, + "static_configs": []map[string]any{{"targets": crowdsecScrape}}, + }}, + }, + } + csResourceAttrs := append([]map[string]any{ + upsert("service.name", CrowdsecServiceName), + upsert("service.instance.id", attrs.HostID), + }, identityAttrs(attrs)...) + processors["resource/crowdsec"] = map[string]any{"attributes": csResourceAttrs} + pipelines["metrics/crowdsec"] = map[string]any{ + "receivers": []string{"prometheus/crowdsec"}, + "processors": []string{"resource/crowdsec", "batch"}, + "exporters": []string{"otlphttp"}, + } + } + cfg := map[string]any{ "receivers": receivers, "processors": processors, diff --git a/internal/otelcol/config_test.go b/internal/otelcol/config_test.go index 8b059ef..d7cf843 100644 --- a/internal/otelcol/config_test.go +++ b/internal/otelcol/config_test.go @@ -251,3 +251,25 @@ func TestRenderNoPostgresIsHostOnly(t *testing.T) { assert.NotContains(t, a, "transform/db-labels") assert.NotContains(t, a, "postgresql") } + +func TestRenderCrowdsecReceiver(t *testing.T) { + out, err := Render("https://otlp.example/v1", fullAttrs(), nil, "127.0.0.1:6060", "127.0.0.1:60601") + require.NoError(t, err) + // A dedicated prometheus receiver + job scraping both loopback endpoints, its own + // pipeline, and the CrowdSec job name overwriting the scrape-derived one. + assert.Contains(t, out, "prometheus/crowdsec") + assert.Contains(t, out, "job_name: crowdsec") + assert.Contains(t, out, "127.0.0.1:6060") + assert.Contains(t, out, "127.0.0.1:60601") + assert.Contains(t, out, "metrics/crowdsec") + assert.Contains(t, out, CrowdsecServiceName) + // The rendered config must still be valid YAML. + var parsed map[string]any + require.NoError(t, yaml.Unmarshal([]byte(out), &parsed)) +} + +func TestRenderNoCrowdsecByDefault(t *testing.T) { + out, err := Render("https://otlp.example/v1", fullAttrs(), nil) + require.NoError(t, err) + assert.NotContains(t, out, "crowdsec") +} diff --git a/internal/types/types.go b/internal/types/types.go index 7a0e333..d13e128 100644 --- a/internal/types/types.go +++ b/internal/types/types.go @@ -45,6 +45,10 @@ type IngressSpec struct { Container string `yaml:"container"` Host string `yaml:"host"` // FK -> compute resource name (same scope); reuses the host's provisioning/firewall/SSH HealthProbesPort int `yaml:"health_probes_port,omitempty"` // public port nginx exposes service health checks on (defaults to 81 when omitted); opened only when a referencing service declares its own health_probes_port + // Security opts this edge out of the env-level security tier (ADR-0043) when set to + // false: no rate limiting (and, slice 2, no CrowdSec). Nil/absent = the env policy + // applies. A *bool distinguishes "unset" from an explicit opt-out. + Security *bool `yaml:"security,omitempty"` } // AppSpec is one front-end (static SPA) resource — a bundle served from an @@ -84,6 +88,10 @@ type GatewaySpec struct { Services []string `yaml:"services"` // FKs -> services (same scope) exposed at the edge; the routing table is derived from their mesh.public_paths HealthProbesPort int `yaml:"health_probes_port,omitempty"` // public port the gateway host exposes its listed services' health checks on (plain HTTP, Host-demuxed by service FQDN; defaults to 81) HealthProbePaths []string `yaml:"health_probe_paths,omitempty"` // exact paths on the gateway's own 443 server that nginx answers 200 "ok" directly (edge liveness over the real TLS path); optional + // Security opts this gateway edge out of the env-level security tier (ADR-0043) when + // set to false: no rate limiting on its mesh routes (and, slice 2, no CrowdSec on its + // host). Nil/absent = the env policy applies. + Security *bool `yaml:"security,omitempty"` } // EffectiveHealthProbesPort is the gateway twin of the ingress method: the @@ -359,6 +367,24 @@ type PKIResourceSpec struct { Validity string `yaml:"validity,omitempty"` // optional CA validity (e.g. "10y") } +// RateLimitProfile is the resolved IP-based rate limit applied to an edge's public +// servers (ADR-0043). Rate limiting is a blanket ingress SECURITY measure, not per-route +// tuning: one limit is derived at deploy from the env's security.rate_limit block and +// stamped UNIFORMLY on every server of an edge (per-route / per-identity limits are a +// gateway-module concern — ADR-0044 — not this layer). It rides on the ingress-derived +// server structs (IngressRoute/IngressApp/IngressGateway) so the nginx renderer emits the +// shared-memory zone and the per-server limit directives without a wider signature; a nil +// pointer means the edge has no rate limiting (disabled, or opted out via `security: +// false`). Name is the nginx zone stem (a fixed constant, since the limit is uniform). +// Keying is always the client IP (ADR-0043); RPS/Burst drive limit_req (http only — +// ignored for an L4 forward), MaxConn drives limit_conn (http and stream). +type RateLimitProfile struct { + Name string // nginx zone stem (a fixed constant — the limit is edge-uniform) + RPS int // requests/second (limit_req rate); 0 disables request-rate limiting + Burst int // queued excess before a 429 (limit_req burst) + MaxConn int // max concurrent connections per client IP (limit_conn); 0 disables +} + // IngressRoute is one typed inbound routing entry the ingress proxy (nginx) // realizes, derived from one route of one service that references the ingress. // The ingress is the sole public entry point: nginx fronts the service on the @@ -385,6 +411,10 @@ type IngressRoute struct { Listen int // public port the ingress accepts traffic on Target int // backend port the service listens on Backend string // backend address nginx proxies to ("127.0.0.1" co-located; private IP cross-host) + // RateLimit is the resolved rate-limit profile for this route (nil = none). A + // tls-termination route uses RPS/Burst (limit_req) and MaxConn (limit_conn); a + // forward route uses only MaxConn (stream limit_conn — L4 has no request rate). + RateLimit *RateLimitProfile } // IngressApp is one static front-end (SPA) the ingress proxy (nginx) serves from @@ -401,6 +431,8 @@ type IngressApp struct { FQDN string // fully-qualified app domain (single SNI / ACME cert) Root string // on-host document root nginx serves (the `current` symlink) Spa bool // true -> try_files fallback to /index.html (SPA deep links) + // RateLimit is the resolved rate-limit profile for this app's server (nil = none). + RateLimit *RateLimitProfile } // IngressHealth is one service health endpoint the ingress proxy (nginx) surfaces, @@ -435,6 +467,9 @@ type IngressGateway struct { FQDN string // fully-qualified gateway domain (single SNI / ACME cert) Routes []IngressGatewayRoute HealthProbePaths []string // exact paths on the 443 server nginx answers 200 "ok" directly (edge liveness) + // RateLimit is the resolved rate-limit profile applied to this gateway's mesh-route + // locations (nil = none). The edge health-probe locations are never limited. + RateLimit *RateLimitProfile } // IngressGatewayRoute is one derived path route on the gateway server: daemon @@ -794,6 +829,57 @@ type EnvironmentVariables struct { BaseDomain string `yaml:"base_domain"` SSH SSHConfig `yaml:"ssh"` Observability ObservabilityConfig `yaml:"observability"` + Security SecurityConfig `yaml:"security"` +} + +// SecurityConfig is the optional env-level edge security tier (ADR-0043): a blanket IP +// rate limit and (slice 2) the CrowdSec agent, applied to every public edge (ingress + +// gateway) host. Both are off unless enabled. Like ObservabilityConfig it lives in +// variables.yaml and is struct-decoded — there is no JSON schema for the block. An edge +// opts the whole tier out with `security: false` on its ingress/gateway spec. +type SecurityConfig struct { + Crowdsec CrowdsecConfig `yaml:"crowdsec"` + RateLimit RateLimitConfig `yaml:"rate_limit"` +} + +// CrowdsecConfig toggles the per-edge-host CrowdSec agent + firewall bouncer (slice 2). +// Version pins the installed release (else the package DefaultVersion); Console opts in +// to dashboard enrollment (gated on a reserved secret). The fields are carried now so +// the authoring surface is stable; the realization lands in the CrowdSec slice. +type CrowdsecConfig struct { + Enabled bool `yaml:"enabled"` + Version string `yaml:"version,omitempty"` + Console bool `yaml:"console,omitempty"` +} + +// RateLimitConfig is the blanket IP rate limit (ADR-0043): one uniform limit applied to +// every public server on an edge — a security floor, not per-route tuning (per-route / +// per-identity limits are a gateway concern, ADR-0044). Keying is always the client IP. +type RateLimitConfig struct { + Enabled bool `yaml:"enabled"` + RequestsPerSecond int `yaml:"requests_per_second"` + Burst int `yaml:"burst"` + MaxConnections int `yaml:"max_connections"` +} + +// RateLimitZoneStem is the fixed nginx zone stem for the edge-uniform limit. It is a +// constant (not per-route) because one limit covers the whole edge (ADR-0043). +const RateLimitZoneStem = "edge" + +// ResolvedRateLimit returns the profile to stamp on an edge's public servers, or nil +// when rate limiting is disabled or has nothing to enforce (both limits zero). The +// program applies it uniformly to every server of every edge that has not opted out. +func (s SecurityConfig) ResolvedRateLimit() *RateLimitProfile { + rl := s.RateLimit + if !rl.Enabled || (rl.RequestsPerSecond <= 0 && rl.MaxConnections <= 0) { + return nil + } + return &RateLimitProfile{ + Name: RateLimitZoneStem, + RPS: rl.RequestsPerSecond, + Burst: rl.Burst, + MaxConn: rl.MaxConnections, + } } // ObservabilityConfig is the optional env-level observability block (ADR-0031, diff --git a/internal/validate/security.go b/internal/validate/security.go new file mode 100644 index 0000000..b400249 --- /dev/null +++ b/internal/validate/security.go @@ -0,0 +1,36 @@ +package validate + +import ( + "fmt" + "path/filepath" + + "github.com/wardnet/inforge/internal/types" +) + +// checkSecurity validates the env-level edge security tier authoring (ADR-0043): the +// blanket rate-limit bounds. It reads only the literal variables.yaml structure — no +// secrets, no environment. A disabled or absent block is a no-op. The per-edge +// `security:` opt-out is a plain boolean enforced by the ingress/gateway JSON schema, so +// it needs nothing here. +func checkSecurity(r *reporter, dir, env string, sec types.SecurityConfig) { + rl := sec.RateLimit + if !rl.Enabled { + return + } + var errs []string + if rl.RequestsPerSecond < 0 { + errs = append(errs, fmt.Sprintf("security.rate_limit.requests_per_second must be >= 0, got %d", rl.RequestsPerSecond)) + } + if rl.Burst < 0 { + errs = append(errs, fmt.Sprintf("security.rate_limit.burst must be >= 0, got %d", rl.Burst)) + } + if rl.MaxConnections < 0 { + errs = append(errs, fmt.Sprintf("security.rate_limit.max_connections must be >= 0, got %d", rl.MaxConnections)) + } + if rl.RequestsPerSecond == 0 && rl.MaxConnections == 0 { + errs = append(errs, "security.rate_limit.enabled is true but both requests_per_second and max_connections are 0 — nothing to limit (set one, or enabled: false)") + } + if len(errs) > 0 { + r.fail(filepath.Join(dir, env, "variables.yaml"), errs...) + } +} diff --git a/internal/validate/security_test.go b/internal/validate/security_test.go new file mode 100644 index 0000000..1341078 --- /dev/null +++ b/internal/validate/security_test.go @@ -0,0 +1,53 @@ +package validate + +import ( + "testing" + + "github.com/stretchr/testify/assert" + + "github.com/wardnet/inforge/internal/types" +) + +func secCfg(rl types.RateLimitConfig) types.SecurityConfig { + return types.SecurityConfig{RateLimit: rl} +} + +func TestCheckSecurityDisabledIsNoop(t *testing.T) { + r := &reporter{} + // Disabled, even with nonsense bounds, is not validated. + checkSecurity(r, t.TempDir(), "prd", secCfg(types.RateLimitConfig{RequestsPerSecond: -5})) + assert.False(t, r.failed) +} + +func TestCheckSecurityValid(t *testing.T) { + r := &reporter{} + checkSecurity(r, t.TempDir(), "prd", secCfg(types.RateLimitConfig{ + Enabled: true, RequestsPerSecond: 20, Burst: 40, MaxConnections: 40, + })) + assert.False(t, r.failed) +} + +func TestCheckSecurityRejectsNegativeBounds(t *testing.T) { + for _, rl := range []types.RateLimitConfig{ + {Enabled: true, RequestsPerSecond: -1}, + {Enabled: true, RequestsPerSecond: 10, Burst: -1}, + {Enabled: true, MaxConnections: -1}, + } { + r := &reporter{} + checkSecurity(r, t.TempDir(), "prd", secCfg(rl)) + assert.True(t, r.failed, "expected failure for %+v", rl) + } +} + +func TestCheckSecurityRejectsEnabledButNothingToLimit(t *testing.T) { + r := &reporter{} + checkSecurity(r, t.TempDir(), "prd", secCfg(types.RateLimitConfig{Enabled: true})) + assert.True(t, r.failed) +} + +// A connection-only limit (rps 0, max_connections > 0) is valid — limit_conn alone. +func TestCheckSecurityConnectionOnlyIsValid(t *testing.T) { + r := &reporter{} + checkSecurity(r, t.TempDir(), "prd", secCfg(types.RateLimitConfig{Enabled: true, MaxConnections: 100})) + assert.False(t, r.failed) +} diff --git a/internal/validate/validate.go b/internal/validate/validate.go index 9aa36d3..899b612 100644 --- a/internal/validate/validate.go +++ b/internal/validate/validate.go @@ -521,6 +521,7 @@ func ValidateResources(env, dir string, defaults types.ProviderDefaults, opts .. globalBase := filepath.Join(base, "global") checkVariables(r, vars, filepath.Join(base, "variables.yaml")) checkObservability(r, dir, env, vars.Observability) + checkSecurity(r, dir, env, vars.Security) checkRegionsFile(r, regionTable, global, filepath.Join(base, "regions.yaml")) // Validate the global slice in a GLOBAL-ONLY context (globalRefs nil): its FK diff --git a/program/crowdsec_test.go b/program/crowdsec_test.go new file mode 100644 index 0000000..9ce37a0 --- /dev/null +++ b/program/crowdsec_test.go @@ -0,0 +1,49 @@ +package program + +import ( + "testing" + + "github.com/stretchr/testify/assert" + + "github.com/wardnet/inforge/internal/naming" + "github.com/wardnet/inforge/internal/types" +) + +func edgeFixture() types.Resources { + return types.Resources{ + Compute: []types.ComputeSpec{ + {Name: "edge", Kind: "vm", InstanceCount: 1}, + {Name: "gwhost", Kind: "vm", InstanceCount: 1}, + {Name: "worker", Kind: "vm", InstanceCount: 1}, + }, + Ingress: []types.IngressSpec{{Name: "main", Host: "edge"}}, + Gateway: []types.GatewaySpec{{Name: "gw", Host: "gwhost"}}, + } +} + +func TestCrowdsecEdgeHostsIncludesIngressAndGatewayOnly(t *testing.T) { + res := edgeFixture() + canonical := naming.CanonicalComputeKeys(res.Compute) + edge := crowdsecEdgeHosts(res, canonical) + assert.True(t, edge[canonical["edge"]], "ingress host is an edge") + assert.True(t, edge[canonical["gwhost"]], "gateway host is an edge") + assert.False(t, edge[canonical["worker"]], "a plain worker gets no CrowdSec") + assert.Len(t, edge, 2) +} + +func TestCrowdsecEdgeHostsHonorsOptOut(t *testing.T) { + res := edgeFixture() + res.Ingress[0].Security = ptrBool(false) // this ingress opts out + canonical := naming.CanonicalComputeKeys(res.Compute) + edge := crowdsecEdgeHosts(res, canonical) + assert.False(t, edge[canonical["edge"]], "opted-out ingress host is excluded") + assert.True(t, edge[canonical["gwhost"]], "the gateway host is a separate edge and stays in") +} + +func TestCrowdsecEdgeHostsExplicitTrueStillIncluded(t *testing.T) { + res := edgeFixture() + res.Gateway[0].Security = ptrBool(true) // explicit opt-in == absent + canonical := naming.CanonicalComputeKeys(res.Compute) + edge := crowdsecEdgeHosts(res, canonical) + assert.True(t, edge[canonical["gwhost"]]) +} diff --git a/program/dbrole_test.go b/program/dbrole_test.go index 4514577..45ff087 100644 --- a/program/dbrole_test.go +++ b/program/dbrole_test.go @@ -56,7 +56,7 @@ func TestObservabilityTriggersAreNotSecret(t *testing.T) { err := pulumi.RunErr(func(ctx *pulumi.Context) error { authB64 := pulumi.ToSecret(pulumi.String("dXNlcjpwdw==")).(pulumi.StringOutput) obs := types.ObservabilityConfig{OTLPEndpoint: "https://otlp.example.com/v1/metrics"} - return provisionObservability(ctx, res, computeOut, nil, map[string]pulumi.Resource{}, nil, obs, authB64, "priv", "prd", "use1") + return provisionObservability(ctx, res, computeOut, nil, map[string]pulumi.Resource{}, nil, obs, types.SecurityConfig{}, authB64, "priv", "prd", "use1") }, pulumi.WithMocks("project", "stack", mocks)) require.NoError(t, err) @@ -109,7 +109,7 @@ func TestMonitorMintWaitsOnServiceRoleMints(t *testing.T) { dbOut := map[string]types.DatabaseOutputs{"app": {RoleProvisioner: p}} authB64 := pulumi.ToSecret(pulumi.String("dXNlcjpwdw==")).(pulumi.StringOutput) obs := types.ObservabilityConfig{OTLPEndpoint: "https://otlp.example.com/v1/metrics"} - return provisionObservability(ctx, res, computeOut, dbOut, map[string]pulumi.Resource{}, nil, obs, authB64, "priv", "prd", "use1") + return provisionObservability(ctx, res, computeOut, dbOut, map[string]pulumi.Resource{}, nil, obs, types.SecurityConfig{}, authB64, "priv", "prd", "use1") }, pulumi.WithMocks("project", "stack", mocks)) require.NoError(t, err) diff --git a/program/gate_test.go b/program/gate_test.go index 10d6f8c..e89c965 100644 --- a/program/gate_test.go +++ b/program/gate_test.go @@ -245,7 +245,7 @@ func TestTLSAndServiceShareOneGate(t *testing.T) { "bridge-01": {PublicIP: pulumi.String("1.2.3.4").ToStringOutput()}, } gates := map[string]pulumi.Resource{} - if err := realizeIngress(ctx, reg, res, computeOut, gates, nil, "priv", "prd", "use1", "example.com", "", types.ProviderDefaults{}); err != nil { + if err := realizeIngress(ctx, reg, res, computeOut, gates, nil, "priv", "prd", "use1", "example.com", "", types.SecurityConfig{}, types.ProviderDefaults{}); err != nil { return err } return provisionServices(ctx, res, computeOut, map[string]serviceMaterial{}, gates, "priv", "prd", "us-east-1", "use1", "example.com", "1.2.3", nil) diff --git a/program/global_test.go b/program/global_test.go index df3388f..7b7c0ef 100644 --- a/program/global_test.go +++ b/program/global_test.go @@ -238,7 +238,7 @@ func TestGlobalServiceRealizesRegionLess(t *testing.T) { } gates := map[string]pulumi.Resource{} // Empty slug throughout — exactly how the scopes loop drives the global slice. - if err := realizeIngress(ctx, reg, res, computeOut, gates, nil, "priv", "prd", "", "wardnet.network", "", types.ProviderDefaults{}); err != nil { + if err := realizeIngress(ctx, reg, res, computeOut, gates, nil, "priv", "prd", "", "wardnet.network", "", types.SecurityConfig{}, types.ProviderDefaults{}); err != nil { return err } return provisionServices(ctx, res, computeOut, map[string]serviceMaterial{}, gates, "priv", "prd", globalScope, "", "wardnet.network", "1.2.3", nil) diff --git a/program/grafana.go b/program/grafana.go index 5caf552..5d63304 100644 --- a/program/grafana.go +++ b/program/grafana.go @@ -30,7 +30,7 @@ import ( // inforge envs share one Grafana org cleanly. Built-in dashboards and built-in alerts // are each opt-out per env (obs.DashboardsEnabled / obs.AlertsEnabled); routing uses // per-rule notification settings, so no org-singleton notification policy is touched. -func realizeGrafana(ctx *pulumi.Context, dir, srcEnv, env string, obs types.ObservabilityConfig, hasDatabase, dryRun bool, services []grafanaalert.ServiceScope) error { +func realizeGrafana(ctx *pulumi.Context, dir, srcEnv, env string, obs types.ObservabilityConfig, hasDatabase, hasCrowdsec, dryRun bool, services []grafanaalert.ServiceScope) error { if obs.GrafanaURL == "" { return nil } @@ -89,6 +89,17 @@ func realizeGrafana(ctx *pulumi.Context, dir, srcEnv, env string, obs types.Obse return err } } + // The CrowdSec dashboard exists only when CrowdSec is enabled (ADR-0043), the same + // resource-gating the Postgres dashboard/alerts use — a non-CrowdSec env gets none. + if hasCrowdsec { + body, err := grafanadash.Crowdsec(env, grafana.DashboardUID(env, "crowdsec")) + if err != nil { + return err + } + if err := newDashboard("crowdsec", body); err != nil { + return err + } + } } // Custom dashboards: Grafana-exported files committed under this env's @@ -109,7 +120,7 @@ func realizeGrafana(ctx *pulumi.Context, dir, srcEnv, env string, obs types.Obse } // Alert rules + contact points (opt-out per env for the built-ins). - if err := realizeGrafanaAlerts(ctx, prov, folder, dir, srcEnv, env, obs, hasDatabase, dryRun, services); err != nil { + if err := realizeGrafanaAlerts(ctx, prov, folder, dir, srcEnv, env, obs, hasDatabase, hasCrowdsec, dryRun, services); err != nil { return err } return nil @@ -122,7 +133,7 @@ func realizeGrafana(ctx *pulumi.Context, dir, srcEnv, env string, obs types.Obse // custom alerts. Each rule routes via its own NotificationSettings.ContactPoint — // resolved from the alert's profile + severity — so the org notification policy is // never touched. A route marked `muted: true` drops the rule. -func realizeGrafanaAlerts(ctx *pulumi.Context, prov *grafanasdk.Provider, folder *oss.Folder, dir, srcEnv, env string, obs types.ObservabilityConfig, hasDatabase, dryRun bool, services []grafanaalert.ServiceScope) error { +func realizeGrafanaAlerts(ctx *pulumi.Context, prov *grafanasdk.Provider, folder *oss.Folder, dir, srcEnv, env string, obs types.ObservabilityConfig, hasDatabase, hasCrowdsec, dryRun bool, services []grafanaalert.ServiceScope) error { notif, err := loader.LoadNotifications(srcEnv, dir) if err != nil { return err @@ -163,6 +174,11 @@ func realizeGrafanaAlerts(ctx *pulumi.Context, prov *grafanasdk.Provider, folder if obs.AlertsEnabled() { alerts = append(alerts, grafanaalert.BuiltIns(env, hasDatabase)...) alerts = append(alerts, grafanaalert.ServiceBuiltIns(env, services)...) + // CrowdSec alerts only when CrowdSec is enabled (ADR-0043) — otherwise the + // acquisition/bouncer rules would evaluate against series that never exist. + if hasCrowdsec { + alerts = append(alerts, grafanaalert.CrowdsecBuiltIns(env)...) + } } customAlerts, err := grafanaalert.Custom(alertsSpec.Alerts) if err != nil { diff --git a/program/program.go b/program/program.go index 231db53..db03a54 100644 --- a/program/program.go +++ b/program/program.go @@ -20,6 +20,7 @@ import ( "github.com/pulumi/pulumi/sdk/v3/go/pulumi/config" "github.com/wardnet/inforge/internal/agent" "github.com/wardnet/inforge/internal/app" + "github.com/wardnet/inforge/internal/crowdsec" "github.com/wardnet/inforge/internal/dbbackup" "github.com/wardnet/inforge/internal/grant" "github.com/wardnet/inforge/internal/hostpaths" @@ -308,11 +309,30 @@ func Run(ctx *pulumi.Context) error { obsAuthB64 = pulumi.ToSecret(pulumi.String(base64.StdEncoding.EncodeToString([]byte(authRaw)))).(pulumi.StringOutput) } + // Optional CrowdSec console enrollment token (ADR-0043): decrypted once per deploy + // only when console enrollment is requested, and marked secret so it is encrypted in + // Pulumi state. Enabled with no token is a hard misconfiguration (fail at up, skipped + // in preview). The community blocklist needs no secret, so this is console-only. + var csEnrollToken pulumi.StringOutput + if vars.Security.Crowdsec.Enabled && vars.Security.Crowdsec.Console { + tokRaw, err := decryptReservedSecret(dir, srcEnv, crowdsec.ReservedNamespace, crowdsec.EnrollSecretKey, ctx.DryRun()) + if err != nil { + return err + } + if tokRaw == "" && !ctx.DryRun() { + return fmt.Errorf("crowdsec: console enrollment is enabled but secrets.enc.yaml is missing or has no %s/%s token — run `inforge secret set %s %s %s --reserved` and commit the store", crowdsec.ReservedNamespace, crowdsec.EnrollSecretKey, srcEnv, crowdsec.ReservedNamespace, crowdsec.EnrollSecretKey) + } + csEnrollToken = pulumi.ToSecret(pulumi.String(tokRaw)).(pulumi.StringOutput) + } + // Env-level Grafana dashboards (ADR-0038): when grafana_url is configured, this env's // built-in dashboards are pushed via the pulumiverse/grafana provider. Org-global, so // realized ONCE here (not per scope) and env-prefixed. A no-op when grafana_url is empty. hasDatabase := len(res.DatabaseCluster) > 0 || len(globalRes.DatabaseCluster) > 0 - if err := realizeGrafana(ctx, dir, srcEnv, env, vars.Observability, hasDatabase, ctx.DryRun(), serviceScopes(regionTable, res, globalRes)); err != nil { + // CrowdSec dashboard/alerts are gated on it being enabled for this env (ADR-0043), + // mirroring how the Postgres built-ins are gated on a cluster existing. + hasCrowdsec := vars.Security.Crowdsec.Enabled + if err := realizeGrafana(ctx, dir, srcEnv, env, vars.Observability, hasDatabase, hasCrowdsec, ctx.DryRun(), serviceScopes(regionTable, res, globalRes)); err != nil { return err } @@ -387,7 +407,7 @@ func Run(ctx *pulumi.Context) error { if err != nil { return err } - if err := realizeIngress(ctx, sc.reg, sc.res, computeOutputs[sc.key], gates, appSeeds, vars.SSH.DeployPrivateKey, env, sc.slug, vars.BaseDomain, ephemeralSlug, providerDefaults); err != nil { + if err := realizeIngress(ctx, sc.reg, sc.res, computeOutputs[sc.key], gates, appSeeds, vars.SSH.DeployPrivateKey, env, sc.slug, vars.BaseDomain, ephemeralSlug, vars.Security, providerDefaults); err != nil { return err } // The east-west mesh proxy (ADR-0032) materializes on every host running a @@ -424,7 +444,14 @@ func Run(ctx *pulumi.Context) error { if err := provisionServices(ctx, sc.res, computeOutputs[sc.key], serviceSecrets, gates, vars.SSH.DeployPrivateKey, env, sc.key, sc.slug, vars.BaseDomain, inforgeVersion, meshReloads); err != nil { return err } - if err := provisionObservability(ctx, sc.res, computeOutputs[sc.key], databaseOutputs[sc.key], gates, dbHostTails, vars.Observability, obsAuthB64, vars.SSH.DeployPrivateKey, env, sc.slug); err != nil { + if err := provisionObservability(ctx, sc.res, computeOutputs[sc.key], databaseOutputs[sc.key], gates, dbHostTails, vars.Observability, vars.Security, obsAuthB64, vars.SSH.DeployPrivateKey, env, sc.slug); err != nil { + return err + } + // The edge security tier (ADR-0043): install CrowdSec + the nftables firewall + // bouncer on every public-edge (ingress/gateway) host, gated on + // security.crowdsec.enabled and the per-edge opt-out. Runs after observability so + // the collector (which will scrape CrowdSec's metrics) is already installed. + if err := provisionCrowdsec(ctx, sc.res, computeOutputs[sc.key], gates, vars.Security, csEnrollToken, vars.SSH.DeployPrivateKey, env, sc.slug); err != nil { return err } } @@ -722,11 +749,18 @@ func provisionService(ctx *pulumi.Context, svc types.ServiceSpec, host types.Com // authB64 is the base64 OTLP Basic-auth value, already marked secret by the caller; // the credential write is built inside an ApplyT over it so the secret is encrypted // in Pulumi state (never written as plaintext), mirroring deliverServiceSecrets. -func provisionObservability(ctx *pulumi.Context, res types.Resources, computeOut map[string]types.ComputeOutputs, dbOut map[string]types.DatabaseOutputs, gates map[string]pulumi.Resource, dbHostTails map[string]pulumi.Resource, obs types.ObservabilityConfig, authB64 pulumi.StringOutput, deployPrivateKey, env, slug string) error { +func provisionObservability(ctx *pulumi.Context, res types.Resources, computeOut map[string]types.ComputeOutputs, dbOut map[string]types.DatabaseOutputs, gates map[string]pulumi.Resource, dbHostTails map[string]pulumi.Resource, obs types.ObservabilityConfig, sec types.SecurityConfig, authB64 pulumi.StringOutput, deployPrivateKey, env, slug string) error { if obs.OTLPEndpoint == "" { return nil } deployUserByCompute := naming.DeployUsersByHost(res.Compute) + // CrowdSec edge hosts (ADR-0043): on these the collector also scrapes CrowdSec's + // loopback Prometheus endpoints. Empty when CrowdSec is off, so a non-edge host (or a + // CrowdSec-disabled env) renders a host-metrics-only collector, byte-identical. + csEdge := map[string]bool{} + if sec.Crowdsec.Enabled { + csEdge = crowdsecEdgeHosts(res, naming.CanonicalComputeKeys(res.Compute)) + } // Postgres cluster ports per host, single-sourced with the realization + firewall // (ADR-0037); used to add a postgresql receiver per co-located cluster. ports := clusterPortsByHost(res, naming.CanonicalComputeKeys(res.Compute)) @@ -836,6 +870,13 @@ func provisionObservability(ctx *pulumi.Context, res types.Resources, computeOut pgPasswords = append(pgPasswords, pw.Result) } + var csScrape []string + if csEdge[hostKey] { + csScrape = []string{ + fmt.Sprintf("%s:%d", crowdsec.AgentMetricsAddr, crowdsec.AgentMetricsPort), + fmt.Sprintf("%s:%d", crowdsec.BouncerMetricsAddr, crowdsec.BouncerMetricsPort), + } + } config, err := otelcol.Render(obs.OTLPEndpoint, otelcol.Attributes{ HostID: naming.Resource(env, slug, "vm", hostKey), CloudProvider: host.CloudProvider, @@ -844,7 +885,7 @@ func provisionObservability(ctx *pulumi.Context, res types.Resources, computeOut MachineType: host.MachineType, Environment: env, RegionSlug: slug, - }, pgTargets) + }, pgTargets, csScrape...) if err != nil { return fmt.Errorf("observability: host %q: render config: %w", hostKey, err) } @@ -877,6 +918,148 @@ func provisionObservability(ctx *pulumi.Context, res types.Resources, computeOut return nil } +// provisionCrowdsec installs the CrowdSec agent + nftables firewall bouncer on every +// public-edge host in this scope (ADR-0043), gated on security.crowdsec.enabled: it is a +// no-op when disabled or when no non-opted-out edge exists. Unlike the observability +// collector (every VM), CrowdSec lands only on ingress/gateway hosts — where public HTTP +// traffic terminates. Each host runs install -> agent config -> bouncer -> (optional +// console enroll) -> liveness assertion, chained by DependsOn and gated on the host's +// cloud-init readiness (shared gates map). The bouncer API key is minted per host +// (random, alphanumeric so it is YAML-safe) and woven through ApplyT so it is encrypted +// in state and never lands in the unencrypted Triggers array — the deliverServiceSecrets +// / monitor-role invariant. enrollToken is used only when console enrollment is on. +func provisionCrowdsec(ctx *pulumi.Context, res types.Resources, computeOut map[string]types.ComputeOutputs, gates map[string]pulumi.Resource, sec types.SecurityConfig, enrollToken pulumi.StringOutput, deployPrivateKey, env, slug string) error { + if !sec.Crowdsec.Enabled { + return nil + } + edge := crowdsecEdgeHosts(res, naming.CanonicalComputeKeys(res.Compute)) + if len(edge) == 0 { + return nil + } + deployUserByCompute := naming.DeployUsersByHost(res.Compute) + for _, hostKey := range sortedKeys(computeOut) { + if !edge[hostKey] { + continue + } + host := computeOut[hostKey] + deployUser := deployUserByCompute[hostKey] + if !ctx.DryRun() { + if deployUser == "" { + return fmt.Errorf("crowdsec: host %q has no deploy_user; inforge needs one to SSH and install CrowdSec", hostKey) + } + if deployPrivateKey == "" { + return fmt.Errorf("crowdsec: no deploy private key configured (set the deploy_private_key stack config or INFORGE_DEPLOY_PRIVATE_KEY)") + } + } + gate, err := cloudInitGate(ctx, gates, hostKey, host, deployPrivateKey, env, slug) + if err != nil { + return err + } + conn := iremote.Connection(host.PublicIP, deployUser, deployPrivateKey) + name := naming.Resource(env, slug, "crowdsec", hostKey) + + // 1. Install the agent + bouncer from the packagecloud repo. First per-host command, + // so it carries the cloud-init gate; later steps chain off it. + install := crowdsec.InstallScript(sec.Crowdsec.Version) + installCmd, err := remote.NewCommand(ctx, name+"-install", &remote.CommandArgs{ + Connection: conn, + Create: pulumi.String(install), + Update: pulumi.String(install), + Triggers: pulumi.Array{pulumi.String(install)}, + }, pulumi.DependsOn([]pulumi.Resource{gate})) + if err != nil { + return fmt.Errorf("crowdsec: host %q: install: %w", hostKey, err) + } + + // 2. Agent config: nginx acquisition + prometheus overlays, hub collections, CAPI + // (community blocklist) registration, reload. + cfg := crowdsec.ConfigScript() + cfgCmd, err := remote.NewCommand(ctx, name+"-config", &remote.CommandArgs{ + Connection: conn, + Create: pulumi.String(cfg), + Update: pulumi.String(cfg), + Triggers: pulumi.Array{pulumi.String(cfg)}, + }, pulumi.DependsOn([]pulumi.Resource{installCmd})) + if err != nil { + return fmt.Errorf("crowdsec: host %q: configure agent: %w", hostKey, err) + } + + // 3. Bouncer: mint an inforge-owned API key (alphanumeric, YAML-safe), register it, + // write the overlay, restart. The key is secret, so the whole script is built + // inside an ApplyT and its Triggers use safeTrigger (never the raw key). + key, err := random.NewRandomPassword(ctx, name+"-bouncer-key", &random.RandomPasswordArgs{ + Length: pulumi.Int(48), + Special: pulumi.Bool(false), + }) + if err != nil { + return fmt.Errorf("crowdsec: host %q: bouncer key: %w", hostKey, err) + } + bouncerScript := key.Result.ApplyT(func(k string) string { return crowdsec.BouncerScript(k) }).(pulumi.StringOutput) + bouncerCmd, err := remote.NewCommand(ctx, name+"-bouncer", &remote.CommandArgs{ + Connection: conn, + Create: bouncerScript, + Update: bouncerScript, + Triggers: pulumi.Array{safeTrigger(bouncerScript)}, + }, pulumi.DependsOn([]pulumi.Resource{cfgCmd})) + if err != nil { + return fmt.Errorf("crowdsec: host %q: configure bouncer: %w", hostKey, err) + } + lastDep := pulumi.Resource(bouncerCmd) + + // 4. Optional console enrollment (gated on the reserved crowdsec_enroll secret). + if sec.Crowdsec.Console { + enrollScript := enrollToken.ApplyT(func(tok string) string { return crowdsec.EnrollScript(tok) }).(pulumi.StringOutput) + enrollCmd, err := remote.NewCommand(ctx, name+"-enroll", &remote.CommandArgs{ + Connection: conn, + Create: enrollScript, + Update: enrollScript, + Triggers: pulumi.Array{safeTrigger(enrollScript)}, + }, pulumi.DependsOn([]pulumi.Resource{bouncerCmd})) + if err != nil { + return fmt.Errorf("crowdsec: host %q: console enroll: %w", hostKey, err) + } + lastDep = enrollCmd + } + + // 5. Deploy-time liveness gate: fail the deploy if the agent's local API is not up + // or the bouncer did not register (CrowdSec otherwise fails silently — ADR-0043). + assert := crowdsec.AssertScript() + if _, err := remote.NewCommand(ctx, name+"-assert", &remote.CommandArgs{ + Connection: conn, + Create: pulumi.String(assert), + Update: pulumi.String(assert), + Triggers: pulumi.Array{safeTrigger(bouncerScript)}, + }, pulumi.DependsOn([]pulumi.Resource{lastDep})); err != nil { + return fmt.Errorf("crowdsec: host %q: liveness assertion: %w", hostKey, err) + } + } + return nil +} + +// crowdsecEdgeHosts returns the set of canonical host keys that run a public edge (an +// ingress or a gateway) and have not opted out of the security tier via `security: false` +// (ADR-0043). A host is included if at least one non-opted-out edge resource lands on it. +func crowdsecEdgeHosts(res types.Resources, canonical map[string]string) map[string]bool { + edge := map[string]bool{} + for _, ing := range res.Ingress { + if ing.Security != nil && !*ing.Security { + continue + } + if hk, ok := canonical[ing.Host]; ok { + edge[hk] = true + } + } + for _, gw := range res.Gateway { + if gw.Security != nil && !*gw.Security { + continue + } + if hk, ok := canonical[gw.Host]; ok { + edge[hk] = true + } + } + return edge +} + // pgMaxIdentifierLen is Postgres's NAMEDATALEN-1: the longest identifier the server // stores. A longer name is SILENTLY TRUNCATED to this many bytes at CREATE ROLE (even // double-quoted) — see checkDBRoleNames for why that must be rejected rather than @@ -1714,7 +1897,7 @@ func serviceDeprovisionScript(svc types.ServiceSpec) string { // (cross-host, using the backend's PrivateIP). FQDNs are env-scoped here so the // provider stays a pure installer. Hosts are realized in sorted order so the // resource graph is stable across runs. -func realizeIngress(ctx *pulumi.Context, reg registry.ProviderRegistry, res types.Resources, computeOut map[string]types.ComputeOutputs, gates map[string]pulumi.Resource, appSeeds map[string][]pulumi.Resource, deployPrivateKey, env, slug, baseDomain, ephemeralSlug string, defaults types.ProviderDefaults) error { +func realizeIngress(ctx *pulumi.Context, reg registry.ProviderRegistry, res types.Resources, computeOut map[string]types.ComputeOutputs, gates map[string]pulumi.Resource, appSeeds map[string][]pulumi.Resource, deployPrivateKey, env, slug, baseDomain, ephemeralSlug string, sec types.SecurityConfig, defaults types.ProviderDefaults) error { canonical := naming.CanonicalComputeKeys(res.Compute) routesByHostKey, _, err := ingressRoutesByHost(res, canonical, env, slug, baseDomain) if err != nil { @@ -1722,6 +1905,10 @@ func realizeIngress(ctx *pulumi.Context, reg registry.ProviderRegistry, res type } appsByHostKey := ingressAppsByHost(res, canonical, slug, baseDomain, ephemeralSlug) gatewaysByHostKey := gatewaysByHost(res, canonical, slug, baseDomain, ephemeralSlug) + // Stamp the env-uniform IP rate limit (ADR-0043) onto every edge server whose edge + // has not opted out (`security: false`). Rate limiting is a blanket security floor, + // so the same profile lands on every route/app/gateway — there is no per-route knob. + stampRateLimit(sec, res, routesByHostKey, appsByHostKey, gatewaysByHostKey) // Resolve the ingress-tier services once and feed both derivations below — the // health entries and the cross-host backend set — so the service list is walked a // single time per host realization. @@ -1819,6 +2006,66 @@ func realizeIngress(ctx *pulumi.Context, reg registry.ProviderRegistry, res type return nil } +// stampRateLimit resolves the env's single blanket IP rate limit (ADR-0043) and stamps +// it onto every derived edge server whose owning edge has not opted out via +// `security: false`. It is a no-op when rate limiting is disabled. A route's edge is the +// ingress its service references; an app's edge is the ingress it targets; a gateway is +// its own edge. Rate limiting is a uniform security floor, so a stamped server always +// gets the same profile — there is deliberately no per-route variation. +func stampRateLimit(sec types.SecurityConfig, res types.Resources, routes map[string][]types.IngressRoute, apps map[string][]types.IngressApp, gateways map[string][]types.IngressGateway) { + rl := sec.ResolvedRateLimit() + if rl == nil { + return + } + svcIngress := fkByName(res.Service, func(s types.ServiceSpec) (string, string) { return s.Name, s.Ingress }) + appIngress := fkByName(res.App, func(a types.AppSpec) (string, string) { return a.Name, a.Ingress }) + ingOptOut := securityOptOut(res.Ingress, func(i types.IngressSpec) (string, *bool) { return i.Name, i.Security }) + gwOptOut := securityOptOut(res.Gateway, func(g types.GatewaySpec) (string, *bool) { return g.Name, g.Security }) + for hk := range routes { + for i := range routes[hk] { + if !ingOptOut[svcIngress[routes[hk][i].Service]] { + routes[hk][i].RateLimit = rl + } + } + } + for hk := range apps { + for i := range apps[hk] { + if !ingOptOut[appIngress[apps[hk][i].Name]] { + apps[hk][i].RateLimit = rl + } + } + } + for hk := range gateways { + for i := range gateways[hk] { + if !gwOptOut[gateways[hk][i].Name] { + gateways[hk][i].RateLimit = rl + } + } + } +} + +// fkByName builds a name -> foreign-key map from a spec slice, given a projector that +// returns each spec's (name, fk). Used to resolve a derived server back to its edge. +func fkByName[T any](specs []T, project func(T) (string, string)) map[string]string { + m := make(map[string]string, len(specs)) + for _, s := range specs { + name, fk := project(s) + m[name] = fk + } + return m +} + +// securityOptOut builds a name -> opted-out map from a spec slice: a resource is opted +// out only when its `security:` flag is explicitly false (nil/absent = env policy applies). +func securityOptOut[T any](specs []T, project func(T) (string, *bool)) map[string]bool { + m := make(map[string]bool, len(specs)) + for _, s := range specs { + name, flag := project(s) + m[name] = flag != nil && !*flag + } + return m +} + // ingressHostsByName maps each ingress resource name to the canonical specKey of // its host. Validation guarantees the host FK resolves; an unresolved one is // skipped defensively (the caller then finds no routes for it). diff --git a/program/ratelimit_test.go b/program/ratelimit_test.go new file mode 100644 index 0000000..e38b434 --- /dev/null +++ b/program/ratelimit_test.go @@ -0,0 +1,75 @@ +package program + +import ( + "testing" + + "github.com/stretchr/testify/assert" + + "github.com/wardnet/inforge/internal/types" +) + +// stampFixture builds a minimal edge: one service+app behind ingress "main", plus a +// gateway "gw", each surfaced as one derived server on host "h". +func stampFixture() (types.Resources, map[string][]types.IngressRoute, map[string][]types.IngressApp, map[string][]types.IngressGateway) { + res := types.Resources{ + Service: []types.ServiceSpec{{Name: "api", Ingress: "main"}}, + App: []types.AppSpec{{Name: "web", Ingress: "main"}}, + Ingress: []types.IngressSpec{{Name: "main"}}, + Gateway: []types.GatewaySpec{{Name: "gw"}}, + } + routes := map[string][]types.IngressRoute{"h": {{Service: "api"}}} + apps := map[string][]types.IngressApp{"h": {{Name: "web"}}} + gws := map[string][]types.IngressGateway{"h": {{Name: "gw"}}} + return res, routes, apps, gws +} + +func enabledSec() types.SecurityConfig { + return types.SecurityConfig{RateLimit: types.RateLimitConfig{ + Enabled: true, RequestsPerSecond: 20, Burst: 40, MaxConnections: 40, + }} +} + +func TestStampRateLimitAppliesUniformly(t *testing.T) { + res, routes, apps, gws := stampFixture() + stampRateLimit(enabledSec(), res, routes, apps, gws) + assert.NotNil(t, routes["h"][0].RateLimit) + assert.NotNil(t, apps["h"][0].RateLimit) + assert.NotNil(t, gws["h"][0].RateLimit) + // Same fixed-stem profile everywhere. + assert.Equal(t, types.RateLimitZoneStem, routes["h"][0].RateLimit.Name) + assert.Equal(t, 20, routes["h"][0].RateLimit.RPS) + assert.Equal(t, 40, gws["h"][0].RateLimit.MaxConn) +} + +func TestStampRateLimitDisabledStampsNothing(t *testing.T) { + res, routes, apps, gws := stampFixture() + stampRateLimit(types.SecurityConfig{}, res, routes, apps, gws) + assert.Nil(t, routes["h"][0].RateLimit) + assert.Nil(t, apps["h"][0].RateLimit) + assert.Nil(t, gws["h"][0].RateLimit) +} + +func TestStampRateLimitIngressOptOut(t *testing.T) { + res, routes, apps, gws := stampFixture() + res.Ingress[0].Security = ptrBool(false) // this ingress opts out + stampRateLimit(enabledSec(), res, routes, apps, gws) + assert.Nil(t, routes["h"][0].RateLimit, "route on an opted-out ingress is not limited") + assert.Nil(t, apps["h"][0].RateLimit, "app on an opted-out ingress is not limited") + assert.NotNil(t, gws["h"][0].RateLimit, "the gateway is a separate edge and stays limited") +} + +func TestStampRateLimitGatewayOptOut(t *testing.T) { + res, routes, apps, gws := stampFixture() + res.Gateway[0].Security = ptrBool(false) + stampRateLimit(enabledSec(), res, routes, apps, gws) + assert.NotNil(t, routes["h"][0].RateLimit) + assert.Nil(t, gws["h"][0].RateLimit, "an opted-out gateway is not limited") +} + +// security: true is an explicit opt-IN (identical to absent), never an opt-out. +func TestStampRateLimitExplicitTrueStillApplies(t *testing.T) { + res, routes, apps, gws := stampFixture() + res.Ingress[0].Security = ptrBool(true) + stampRateLimit(enabledSec(), res, routes, apps, gws) + assert.NotNil(t, routes["h"][0].RateLimit) +} diff --git a/schemas/gateway.json b/schemas/gateway.json index 229ceb2..1205689 100644 --- a/schemas/gateway.json +++ b/schemas/gateway.json @@ -1 +1 @@ -{"$schema":"https://json-schema.org/draft/2020-12/schema","$id":"gateway.json","title":"Gateway (north-south daemon API gateway) resource","type":"object","additionalProperties":false,"required":["name","container","host","pki","subdomain","services"],"properties":{"name":{"type":"string","pattern":"^[a-z][a-z0-9_-]*$"},"container":{"type":"string","minLength":1,"description":"Logical container/grouping the gateway belongs to (same field as other resources)."},"host":{"type":"string","minLength":1,"description":"Name of the compute resource (in the same scope) this gateway runs on. The gateway reuses the host's provisioning/firewall/SSH and inherits its provider; it declares no provider of its own."},"pki":{"type":"string","minLength":1,"description":"Name of the two-tier (mesh) PKI in pki.enc.yaml the gateway's client leaf (/gateway) mints from. Every listed service must declare the same pki (a callee only trusts callers chaining to its own mesh)."},"subdomain":{"type":"string","minLength":1,"description":"Public subdomain daemons connect to. The fully-qualified domain is composed at realization from the scope and base domain."},"services":{"type":"array","minItems":1,"description":"names of the services (same scope) exposed at the internet edge. The gateway's routing table is DERIVED from each listed service's mesh.public_paths (ADR-0034); a request matching no public glob is answered 404 (JSON) at the edge. Each listed service must permit the gateway (list \"gateway\" in its mesh.allowed_services) and declare at least one public path.","items":{"type":"string","pattern":"^[a-z][a-z0-9_-]*$"}},"health_probes_port":{"type":"integer","minimum":1,"maximum":65535,"description":"public port the gateway host exposes its listed services' health checks on (plain HTTP, Host-demuxed by service FQDN). Defaults to 81. Opened only when a listed service declares its own health_probes_port."},"health_probe_paths":{"type":"array","minItems":1,"description":"exact paths on the gateway's own 443 server that nginx answers 200 \"ok\" directly — edge liveness over the real daemon path (DNS+cert+TLS). Must not overlap any listed service's public_paths.","items":{"type":"string","pattern":"^/"}}}} +{"$schema":"https://json-schema.org/draft/2020-12/schema","$id":"gateway.json","title":"Gateway (north-south daemon API gateway) resource","type":"object","additionalProperties":false,"required":["name","container","host","pki","subdomain","services"],"properties":{"name":{"type":"string","pattern":"^[a-z][a-z0-9_-]*$"},"container":{"type":"string","minLength":1,"description":"Logical container/grouping the gateway belongs to (same field as other resources)."},"host":{"type":"string","minLength":1,"description":"Name of the compute resource (in the same scope) this gateway runs on. The gateway reuses the host's provisioning/firewall/SSH and inherits its provider; it declares no provider of its own."},"pki":{"type":"string","minLength":1,"description":"Name of the two-tier (mesh) PKI in pki.enc.yaml the gateway's client leaf (/gateway) mints from. Every listed service must declare the same pki (a callee only trusts callers chaining to its own mesh)."},"subdomain":{"type":"string","minLength":1,"description":"Public subdomain daemons connect to. The fully-qualified domain is composed at realization from the scope and base domain."},"services":{"type":"array","minItems":1,"description":"names of the services (same scope) exposed at the internet edge. The gateway's routing table is DERIVED from each listed service's mesh.public_paths (ADR-0034); a request matching no public glob is answered 404 (JSON) at the edge. Each listed service must permit the gateway (list \"gateway\" in its mesh.allowed_services) and declare at least one public path.","items":{"type":"string","pattern":"^[a-z][a-z0-9_-]*$"}},"health_probes_port":{"type":"integer","minimum":1,"maximum":65535,"description":"public port the gateway host exposes its listed services' health checks on (plain HTTP, Host-demuxed by service FQDN). Defaults to 81. Opened only when a listed service declares its own health_probes_port."},"health_probe_paths":{"type":"array","minItems":1,"description":"exact paths on the gateway's own 443 server that nginx answers 200 \"ok\" directly — edge liveness over the real daemon path (DNS+cert+TLS). Must not overlap any listed service's public_paths.","items":{"type":"string","pattern":"^/"}},"security":{"type":"boolean","description":"Opt this gateway edge out of the env-level security tier (ADR-0043) when false: no rate limiting (and no CrowdSec). Absent = the env security policy applies."}}} diff --git a/schemas/ingress.json b/schemas/ingress.json index 258c39a..828e953 100644 --- a/schemas/ingress.json +++ b/schemas/ingress.json @@ -1 +1 @@ -{"$schema":"https://json-schema.org/draft/2020-12/schema","$id":"ingress.json","title":"Ingress resource","type":"object","additionalProperties":false,"required":["name","container","host"],"properties":{"name":{"type":"string","pattern":"^[a-z][a-z0-9_-]*$"},"container":{"type":"string","minLength":1,"description":"Logical container/grouping the ingress belongs to (same field as other resources)."},"host":{"type":"string","minLength":1,"description":"Name of the compute resource (in the same scope) this ingress runs on. The ingress reuses the host's provisioning/firewall/SSH and inherits its provider; it declares no provider of its own."},"health_probes_port":{"type":"integer","minimum":1,"maximum":65535,"description":"Public port nginx exposes service health checks on (defaults to 81 when omitted). Opened to 0.0.0.0/0 only when at least one referencing service declares its own health_probes_port. Must not equal a route listen port on this host, nor 80."}}} +{"$schema":"https://json-schema.org/draft/2020-12/schema","$id":"ingress.json","title":"Ingress resource","type":"object","additionalProperties":false,"required":["name","container","host"],"properties":{"name":{"type":"string","pattern":"^[a-z][a-z0-9_-]*$"},"container":{"type":"string","minLength":1,"description":"Logical container/grouping the ingress belongs to (same field as other resources)."},"host":{"type":"string","minLength":1,"description":"Name of the compute resource (in the same scope) this ingress runs on. The ingress reuses the host's provisioning/firewall/SSH and inherits its provider; it declares no provider of its own."},"health_probes_port":{"type":"integer","minimum":1,"maximum":65535,"description":"Public port nginx exposes service health checks on (defaults to 81 when omitted). Opened to 0.0.0.0/0 only when at least one referencing service declares its own health_probes_port. Must not equal a route listen port on this host, nor 80."},"security":{"type":"boolean","description":"Opt this edge out of the env-level security tier (ADR-0043) when false: no rate limiting (and no CrowdSec). Absent = the env security policy applies."}}} diff --git a/website/docs/configuration/observability.md b/website/docs/configuration/observability.md index 3e88892..6c80049 100644 --- a/website/docs/configuration/observability.md +++ b/website/docs/configuration/observability.md @@ -24,6 +24,7 @@ per-node / per-cluster / per-database / per-service drill-down. | **Infrastructure** | host metrics (ADR-0031, `system_*`) | fleet CPU / memory / disk / network, per-node uptime and restarts | | **Database** | Postgres receiver metrics (ADR-0037, `postgresql_*`) | connections, transaction rate, DB size, checkpoints — fleet, per cluster, and per database | | **Service** | wardnet-cloud RED + domain metrics | request rate, 5xx error rate, and latency percentiles per service and route, plus the domain counters (DDNS networks provisioned, active tunnels, tenant tombstone sweeps) | +| **CrowdSec** | CrowdSec agent (`cs_*`) + firewall-bouncer (`fw_bouncer_*`) metrics (ADR-0043) | log-acquisition rate (the "is it working" signal), active ban decisions by reason, scenario hits, IPs banned at the firewall, and the bouncer's dropped packets/bytes + LAPI pull rate. Present only when [`security.crowdsec.enabled`](/configuration/variables-yaml#security). | The **Service** dashboard reads the RED histogram `http_server_request_duration_seconds_*` that every wardnet-cloud service emits, grouped by the `service_name` label Grafana Cloud promotes from the OTLP @@ -71,8 +72,9 @@ to opt out of the generated ones; author your own in `resources//observabil ### Built-in alerts When `built_in_alerts` is on (the default), inforge generates these from the metrics it owns. All are -env-scoped and fire per node / cluster; the Postgres ones only exist when the env has a database -cluster. Thresholds are fixed — to tune, opt out and re-author as custom alerts. +env-scoped and fire per node / cluster / edge; the Postgres ones only exist when the env has a +database cluster, and the CrowdSec ones only when [`security.crowdsec.enabled`](/configuration/variables-yaml#security). +Thresholds are fixed — to tune, opt out and re-author as custom alerts. | Alert | Condition | For | Severity | |---|---|---|---| @@ -85,6 +87,12 @@ cluster. Thresholds are fixed — to tune, opt out and re-author as custom alert | Postgres Connections High | backends > 80% of max per cluster | 10m | warning | | Postgres High Rollback Ratio | rollbacks > 25% of transactions per cluster | 15m | warning | | Postgres Metrics Missing | no cluster reporting (NoData → alerting) | 5m | critical | +| CrowdSec Acquisition Stalled | parse rate ≈ 0 per edge (CrowdSec is blind) | 15m | warning | +| CrowdSec Bouncer Not Pulling | firewall bouncer's LAPI request rate ≈ 0 per edge | 10m | warning | + +The CrowdSec alerts use `NoData → OK` (not `alerting`): they fire only on a present-but-bad signal, +so a metric-name mismatch or a not-yet-scraped agent never false-fires. Confirm CrowdSec is exporting +metrics (the built-in CrowdSec dashboard shows a non-zero acquisition rate) after the first deploy. ### Custom alerts diff --git a/website/docs/configuration/variables-yaml.md b/website/docs/configuration/variables-yaml.md index 714256e..aba80d1 100644 --- a/website/docs/configuration/variables-yaml.md +++ b/website/docs/configuration/variables-yaml.md @@ -20,6 +20,15 @@ ssh: # required when using compute observability: # optional — Grafana Cloud integration otlp_endpoint: https://otlp-gateway-....grafana.net/otlp # host/DB metrics collector grafana_url: https://myorg.grafana.net # dashboards + alerts target + +security: # optional — edge security tier (public ingress + gateway hosts) + crowdsec: + enabled: true # IP banning (nftables) + free community blocklist on edges + rate_limit: + enabled: true # one blanket IP-based limit on every public edge server + requests_per_second: 20 + burst: 40 + max_connections: 40 ``` ## Fields @@ -67,6 +76,41 @@ are [reserved secrets](/cli/secret) in `secrets.enc.yaml`, never committed here. (from `observability/notifications.yaml`) that built-in alerts and any alert omitting `profile:` route through. Required once alerts are managed. +### `security` + +Optional edge security tier applied to the public [ingress](/resources/ingress) and +[gateway](/resources/gateway) hosts. Off unless enabled. An individual ingress or gateway +opts the whole tier out with `security: false` on its own spec. + +- `rate_limit` — a single **blanket, IP-based** rate limit applied uniformly to every + public server on every edge. It is a security floor, not per-route tuning: the same + limit covers all routes, apps, and gateway paths (per-route / per-identity limits are a + future gateway concern). Requests over the limit are answered `429`. Fields: + - `enabled` — turn rate limiting on (default off). + - `requests_per_second` — sustained per-client-IP request rate (`limit_req`). + - `burst` — how many excess requests may queue before a `429` is returned. + - `max_connections` — concurrent connections allowed per client IP (`limit_conn`). + + Health-check and ACME (certificate) endpoints are never rate-limited. + +- `crowdsec` — installs the [CrowdSec](https://www.crowdsec.net) agent + nftables firewall + bouncer on every edge host. It parses the ingress nginx logs, bans abusive and + known-bad IPs at the kernel (nftables), and pulls the free crowd-sourced community + blocklist for pre-emptive protection. Fields: + - `enabled` — turn CrowdSec on (default off). + - `version` — optional pin for the `crowdsec` agent package. The firewall bouncer + versions independently, so it is never pinned; omit to install the repo's current pair. + - `console` — enroll the host in the CrowdSec console dashboard. Requires the reserved + secret `security/crowdsec_enroll` + (`inforge secret set security crowdsec_enroll --reserved`). The community + blocklist itself needs no secret. + + When `observability.otlp_endpoint` is set, CrowdSec's own metrics (log-acquisition rate, + active decisions, bouncer pulls) are scraped by the host collector and shipped to Grafana + Cloud alongside host and database metrics — so you can see it working, and catch its + silent-failure mode (unreadable logs ⇒ nothing parsed). A deploy fails if CrowdSec does + not come up on an edge host. + ## Example ```yaml title="resources/prd/variables.yaml" @@ -80,4 +124,13 @@ observability: default_profile: prod # notification profile for built-in + un-profiled alerts # built_in_dashboards: false # opt out of the generated dashboards # built_in_alerts: false # opt out of the generated alert rules +security: + crowdsec: + enabled: true + # console: true # dashboard enrollment (needs security/crowdsec_enroll) + rate_limit: + enabled: true + requests_per_second: 20 + burst: 40 + max_connections: 40 ``` diff --git a/website/docs/resources/gateway.md b/website/docs/resources/gateway.md index e67080a..25f733d 100644 --- a/website/docs/resources/gateway.md +++ b/website/docs/resources/gateway.md @@ -27,6 +27,7 @@ services: # the services exposed at the edge — routing is DERI - tunneller health_probe_paths: # optional — edge liveness: nginx answers 200 "ok" on the 443 server - /livez +# security: false # optional — opt this edge out of the env security tier (rate limiting) ``` ## Fields @@ -41,6 +42,7 @@ health_probe_paths: # optional — edge liveness: nginx answers 200 "ok" o | `services` | array | Yes (min 1) | The services (same scope) exposed at the internet edge. The routing table is **derived**: one nginx regex location per (listed service, [`mesh.public_paths`](./service#path-level-exposure) glob), target named in `X-Mesh-Target`. A request matching no public glob is answered at the edge with an `application/json` `404` body `{"error":"not_found"}` — it never traverses to any service. | | `health_probes_port` | int | No | Public **plain-HTTP** port the gateway host exposes its listed services' health checks on (default `81`), demuxed by request `Host` (the service's FQDN) — the gateway twin of the ingress [health port](./ingress#health-probes). Opened to the internet only when a listed service declares its own [`health_probes_port`](./service#health-probes). Must not be `80` or `443`, and must **match** a co-hosted ingress's health port (one public health port per host). | | `health_probe_paths` | array | No | Exact paths on the gateway's **own `443` server** that nginx answers `200 "ok"` directly — edge liveness proving the real daemon path (DNS + cert + TLS) without touching any backend. `inforge validate` rejects a path that any listed service's public glob claims. | +| `security` | bool | No | Set `false` to opt this gateway out of the env-level [security tier](/configuration/variables-yaml#security) — its mesh-route locations get no rate limiting. Absent/`true` = the env policy applies. Health-probe locations are never rate-limited regardless. | **A gateway is a scope singleton** — at most one per scope: it is the scope's one public daemon edge. It may only list services in the **same scope**. diff --git a/website/docs/resources/ingress.md b/website/docs/resources/ingress.md index 5550b90..26735b1 100644 --- a/website/docs/resources/ingress.md +++ b/website/docs/resources/ingress.md @@ -31,6 +31,7 @@ name: edge # required — ingress name (unique within the scope) container: bridge # required — grouping label host: bridge # required — FK -> compute name (same scope); the vm nginx runs on health_probes_port: 81 # optional — public port for service health checks (default 81) +# security: false # optional — opt this edge out of the env security tier (rate limiting) ``` | Field | Type | Required | Description | @@ -39,6 +40,7 @@ health_probes_port: 81 # optional — public port for service health checks | `container` | string | Yes | Grouping label (tags, like other resources). | | `host` | string | Yes | **Name** of the Compute resource (same scope) this ingress runs on. Must be a single-instance `vm`. A `global/` prefix is rejected — a global ingress is declared in the global slice itself. **At most one ingress per scope** — the ingress's DNS name (`ingress[.].`, see below) is scope-singular, which also implies one per host. | | `health_probes_port` | int | No | Public port nginx exposes service [health checks](#health-probes) on. Defaults to `81`. Opened to the internet only when a referencing service declares its own `health_probes_port`. Must not be `80` or a route `listen` port on this host. | +| `security` | bool | No | Set `false` to opt this ingress out of the env-level [security tier](/configuration/variables-yaml#security) — its servers get no rate limiting. Absent/`true` = the env policy applies. | ## What it serves