fix(crowdsec): correct metric names confirmed against a real host#245
Merged
Conversation
Add ADR-0043 proposing an edge security tier authored from one variables.yaml `security:` block: nginx-native rate limiting (named profiles + default, rendered into the edge nginx.conf) and CrowdSec (firewall bouncer at nftables plus the free community blocklist) installed per public-edge host. Status: proposed — design checkpoint for review before implementation. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011RB2WPLAzoDT1bjXrMfoXi
Design record (ADR-0043, proposed) for a public-edge security tier with two independently-toggled capabilities authored from one variables.yaml `security:` block: - Rate limiting: nginx-native limit_req/limit_conn rendered into the edge nginx.conf, with named profiles + a default_profile (mirroring the observability notification-profile idiom). Per-profile keying is `ip` (default) or a custom header with automatic IP fallback; JWT-claim keying is excluded at the edge per the ADR-0032 gateway trust boundary. - CrowdSec: per-host agent (edges only) with the nftables firewall bouncer plus the free community blocklist, installed like the otelcol collector (ADR-0031). Documents the L7-vs-volumetric DoS scope boundary explicitly and the wiring shape (internal/crowdsec package, provisionCrowdsec pass, ingressHostUnion targeting). No implementation yet — design for review. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011RB2WPLAzoDT1bjXrMfoXi
Per review: v1 keys rate limits on client IP only. Identity-based keying (API key / verified client) is deferred until authentication moves into the gateway, at which point the edge holds a verified claim and per-identity keying becomes safe on the same limit_req mechanism. Adds a Future-direction note pointing at a gateway-auth ADR (nginx auth_request + an auth service). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011RB2WPLAzoDT1bjXrMfoXi
Companion to ADR-0043. Moves authentication into the north-south gateway while keeping authorization at the service (ADR-0032 defense-in-depth preserved): - nginx stays the router; auth is added via the built-in auth_request directive delegating to a co-located auth service reached over loopback/unix socket — no lua/njs JWT logic in nginx. - Custom Rust auth service (jsonwebtoken + cached JWKS) over an off-the-shelf gateway, consistent with the in-house mesh/PKI/agent posture; Ory Oathkeeper kept as the buy fallback. - Verified identity crosses to the service as headers, trusted only via the mesh from the gateway leaf. - Records the preaccess-vs-access phase-ordering trap: a single-pass limit_req on an auth-set header fails open; identity keying needs a second nginx pass or auth-service enforcement. - HA state model: per-node volumetric limits (approximate, CrowdSec-backstopped), shared-store (Redis) per-identity quotas in the auth service, and a central CrowdSec LAPI for fleet-wide bans; quota checks fail open. No implementation — design for review. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011RB2WPLAzoDT1bjXrMfoXi
…043)
Rate limiting is a blanket ingress security measure: one uniform IP-keyed limit
applied to every public server on an edge, not a per-route knob. Per-route /
per-identity limits are a gateway-module concern (ADR-0044), not this layer.
- types: RateLimitProfile (resolved limit: RPS/Burst/MaxConn, fixed zone stem)
rides on IngressRoute/IngressApp/IngressGateway, so neither the
IngressProvider.Realize signature nor the Hetzner provider changes.
- internal/nginx: emit one limit_req_zone/limit_conn_zone per context in http{}
and stream{}, with per-location limit_req/limit_conn on the service-facing
locations. Health and ACME servers, and the gateway's edge health-probe
locations, are exempt. Forward (L4) routes get limit_conn only. Throttled
requests answer 429 (CrowdSec-parseable). A nil profile renders
byte-identically to before (golden test unchanged).
Also updates ADR-0043 (drop per-route profiles; uniform security floor) and
ADR-0044 (per-route/identity limits owned here; auth_request vs custom nginx
dynamic module left as an open enforcement choice).
Wiring (types.SecurityConfig, schemas, validate, program resolution) lands in
the next commit.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011RB2WPLAzoDT1bjXrMfoXi
…ADR-0043)
Wires the ADR-0043 blanket IP rate limit end to end (slice 1 complete):
- types: SecurityConfig{Crowdsec, RateLimit} on EnvironmentVariables +
ResolvedRateLimit(); Security *bool per-edge opt-out on IngressSpec/GatewaySpec.
- schemas: `security` boolean added to ingress.json + gateway.json.
- validate: checkSecurity enforces rate-limit bounds when enabled.
- program: stampRateLimit resolves the single uniform limit and stamps it on
every derived edge server whose ingress/gateway has not opted out; threaded
through realizeIngress.
- tests: checkSecurity bounds + stampRateLimit uniform/opt-out/disabled cases;
fixed two existing realizeIngress test callers for the new parameter.
- docs: variables.yaml security block + ingress/gateway `security:` opt-out.
CrowdSec realization is slice 2 (the CrowdsecConfig fields are carried here so
the authoring surface is stable).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011RB2WPLAzoDT1bjXrMfoXi
Record the slice-2 observability decision: CrowdSec fails silently (unreadable nginx logs -> parses/bans nothing while cscli still looks healthy), so slice 2 wires its Prometheus metrics (agent 127.0.0.1:6060 + firewall-bouncer loopback endpoint) into the otelcol collector via a prometheus receiver, gated on the same observability.otlp_endpoint and exported through the existing otlphttp -> Grafana Cloud path — mirroring the ADR-0037 postgresql receiver. Acquisition health (cs_reader_hits > 0) is the load-bearing "actually working" signal; the deploy also asserts cscli lapi status + bouncer registration at apply time. Dashboards + two alerts (acquisition stalled, bouncer not pulling) are slice 2b. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011RB2WPLAzoDT1bjXrMfoXi
…e 2 pt1) New stdlib-only, Pulumi-free internal/crowdsec (like internal/nginx and internal/otelcol): renders the idempotent host shell + config for the CrowdSec agent + nftables firewall bouncer, ready for the program to wire onto edge hosts. - InstallScript: adds the CrowdSec packagecloud apt repo with a pinned signed-by keyring (the internal/nginx nginx.org-repo pattern), installs crowdsec + crowdsec-firewall-bouncer-nftables, enables both units. Optional agent-only version pin (the bouncer versions independently). - ConfigScript: nginx acquisition drop-in + prometheus config overlay, hub collections (nginx/base-http/http-cve), CAPI (community blocklist) registration guarded for idempotency, agent reload. - BouncerScript: idempotent bouncer-key registration (delete-then-add) + the bouncer .local overlay (nftables/DROP + loopback prometheus); key base64- transported into the file (never plaintext in the config write). - AssertScript: deploy-time liveness gate (cscli lapi status + bouncer registered) — CrowdSec otherwise fails silently. - Metrics endpoints (agent 127.0.0.1:6060, bouncer :60601) exposed for the slice-2 otelcol prometheus receiver. Also corrects the ADR install decision: packagecloud apt repo, not a checksummed .deb download, since CrowdSec ships its packages (with deps) through that repo. Program wiring (provisionCrowdsec) and the otelcol receiver are the next commits. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011RB2WPLAzoDT1bjXrMfoXi
…ce 2 pt2) Wire the CrowdSec agent + firewall bouncer onto public-edge hosts. Mirrors provisionObservability: per-host loop gated on the shared cloud-init gates, secret woven through ApplyT + safeTrigger. - provisionCrowdsec: gated on security.crowdsec.enabled; targets the ingress ∪ gateway host set (crowdsecEdgeHosts), honoring the per-edge `security: false` opt-out. Per host: install -> agent config -> bouncer -> optional console enroll -> deploy-time liveness assertion, chained by DependsOn. - Bouncer API key minted per host via random.RandomPassword (alphanumeric, so YAML-safe), built inside ApplyT so it is encrypted in state; Triggers use safeTrigger, never the raw key. - Optional console enrollment token decrypted once per deploy from the reserved security/crowdsec_enroll secret (mirrors the OTLP auth decrypt), marked secret. - Called in the scope loop after provisionObservability so the collector that will scrape CrowdSec's metrics is already present. - Tests: crowdsecEdgeHosts includes only ingress/gateway hosts and honors the opt-out. Remaining: the otelcol prometheus receiver scraping CrowdSec's loopback metrics endpoints (slice 2 pt3). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011RB2WPLAzoDT1bjXrMfoXi
…R-0043, slice 2 pt3) Make CrowdSec observable through the existing pipeline (ADR-0031/0037): the edge host's collector scrapes CrowdSec's loopback Prometheus endpoints and exports them to Grafana Cloud on the same otlphttp path, correlating on host.id. - otelcol.Render: optional variadic crowdsecScrape []string adds a prometheus receiver + its own job/pipeline scraping the given loopback endpoints, with a resource processor stamping service.name=CrowdsecServiceName + per-node HostID + the shared ADR-0030 identity (mirrors the postgresql receiver). Empty leaves the config byte-identical, so the 10 host-metrics-only callers are untouched. - provisionObservability: gains the security config, computes the CrowdSec edge set, and passes the agent (127.0.0.1:6060) + bouncer (:60601) endpoints for edge hosts. Gated on the same otlp_endpoint — CrowdSec metrics ride the observability pipeline or not at all. - Tests: receiver present with endpoints + valid YAML; absent by default. This completes slice 2 (CrowdSec): install/config package, provisionCrowdsec edge pass, and metrics. Dashboards + the two alerts (acquisition stalled, bouncer not pulling) remain as slice 2b. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011RB2WPLAzoDT1bjXrMfoXi
Add CrowdSec to the variables.yaml reference: the crowdsec block (enabled, version, console), the reserved security/crowdsec_enroll secret, and the note that its metrics ride the observability pipeline when otlp_endpoint is set. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011RB2WPLAzoDT1bjXrMfoXi
…43, slice 2b) Close out CrowdSec observability with a built-in Grafana dashboard and two alerts, gated on security.crowdsec.enabled exactly as the Postgres built-ins are gated on a cluster existing (so a non-CrowdSec env gets neither). - grafanadash.Crowdsec: env-scoped dashboard over cs_* metrics — acquisition rate (the health signal), active decisions, decisions by reason, scenario hits, and the firewall bouncer's drops + last-pull freshness. - grafanaalert.CrowdsecBuiltIns: "Acquisition Stalled" (parse rate ~0 → blind) and "Bouncer Not Pulling" (last-pull stale → enforcement drift). Both NoDataOK, so an unverified metric name or not-yet-scraped agent never false-fires — they alert only on a present-but-bad signal. - program/grafana.go: thread hasCrowdsec (= vars.Security.Crowdsec.Enabled) through realizeGrafana/realizeGrafanaAlerts; gate the CrowdSec dashboard + alerts on it. - Tests + docs (built-in dashboard/alert tables). Metric names (cs_parser_hits_total, cs_bouncer_last_pull, …) are the standard CrowdSec series and should be confirmed on a real edge; NoDataOK keeps a mismatch from firing meanwhile. This completes ADR-0043 (edge security tier): rate limiting + CrowdSec, wired, observable, and documented. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011RB2WPLAzoDT1bjXrMfoXi
Four robustness fixes from the branch code review:
1. Restart policy for the CrowdSec units. A crashed crowdsec-firewall-bouncer
silently stops enforcing bans; add a Restart=always systemd drop-in for both
units (the internal/nginx precedent / daemon-units-restart-on-any-exit rule)
so a security daemon self-heals.
2. `cscli hub update` before `cscli collections install`. A fresh host has no
hub index, so the install otherwise fails ("can't find collection") and fails
the deploy under set -e.
3. Retry `cscli lapi status` in the liveness assertion. The LAPI needs a moment
to rebind :8080 after the restart; poll briefly before the final unsuppressed
check so a healthy CrowdSec is not failed on a race.
4. Surface `cscli console enroll` failures instead of `|| true`. A bad/expired
console token was silently swallowed (green deploy, no enrollment); now it is
logged to stderr while staying non-fatal.
Tests assert each fix (restart drop-in, hub-update ordering, lapi retry, enroll
error surfaced).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011RB2WPLAzoDT1bjXrMfoXi
The CrowdSec install shell and the metric names used by the built-in dashboard and alerts (ADR-0043, slice 2b) were authored from docs and had never been run. Verified end-to-end on a throwaway Hetzner VM (Ubuntu 24.04 noble, amd64) running the exact scripts internal/crowdsec renders, against crowdsec 1.7.8 + crowdsec-firewall-bouncer-nftables 0.0.34. The install path was correct as written and is unchanged: the armored .asc works via signed-by with no dearmor, the packagecloud source line resolves, the acquis.d drop-in and both .local overlays are read, every cscli invocation matches, nftables enforcement lands, and the systemd Restart=always drop-in survives a kill -9. The metric names were substantially wrong. The firewall bouncer does not use the cs_ prefix at all: cs_reader_hits_total -> cs_filesource_hits_total cs_scenario_hits_total -> cs_bucket_poured_total cs_lapi_decisions_total -> no such counter (gauge only) cs_bouncer_dropped_packets_total -> fw_bouncer_dropped_packets cs_bouncer_dropped_bytes_total -> fw_bouncer_dropped_bytes cs_bouncer_last_pull -> does not exist cs_bouncer_last_pull never existed, so the "Bouncer Not Pulling" alert could not fire under any condition — the exact silent failure it was meant to catch. It now alerts on a stalled lapi_requests_total rate, the bouncer's real pull-freshness signal. Since CrowdSec publishes no decisions-added counter, no panel uses increase() over decisions; the "bans added" stat now reports fw_bouncer_banned_ips. cs_parser_hits_total and cs_active_decisions were correct as written. Tests assert the corrected names and now also assert the absent ones stay absent, so a silently-dead query cannot regress back in. ADR-0043 records the verification and drops the unverified caveats, retaining one open question: whether the otelcol prometheus -> otlphttp hop rewrites counter suffixes, which needs a live Grafana Cloud scrape to settle.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The CrowdSec edge-security tier (ADR-0043, slice 2b) had its install shell and observability metric names authored from docs and never run on a real host. This PR verifies the whole integration end-to-end on a throwaway Hetzner VM (Ubuntu 24.04
noble, amd64) — running the exact scriptsinternal/crowdsecrenders — againstcrowdsec1.7.8 +crowdsec-firewall-bouncer-nftables0.0.34, and fixes what was wrong.What passed unchanged
internal/crowdsecneeded no code changes — the install path was correct as written:.ascworks viasigned-bywith nogpg --dearmor;.../crowdsec/ubuntu/ noble mainresolves; both packages install./etc/crowdsec/acquis.d/nginx.yamlis read;crowdsecurity/nginx-logsparsed the access log..localoverlays — both merge (the bouncer logs it explicitly).hub update,collections install,capi status/register,bouncers add --key/delete,lapi status,bouncers list -o json, positionalconsole enroll <key>all match.table ip crowdsec/crowdsec6with realdroprules; acscli decisions addlands within seconds.127.0.0.1:6060and bouncer127.0.0.1:60601both correct.Restart=always— the drop-in applies to both units;kill -9respawns.What was wrong — the metric names
The firewall bouncer does not use the
cs_prefix at all:cs_reader_hits_totalcs_filesource_hits_totalcs_scenario_hits_totalcs_bucket_poured_totalcs_lapi_decisions_totalcs_active_decisionsgauge)cs_bouncer_dropped_packets_totalfw_bouncer_dropped_packetscs_bouncer_dropped_bytes_totalfw_bouncer_dropped_bytescs_bouncer_last_pullMost consequential:
cs_bouncer_last_pullnever existed, so the "Bouncer Not Pulling" alert could not fire under any condition — the exact silent enforcement failure it was meant to catch. It now alerts on a stalledlapi_requests_totalrate. Since CrowdSec publishes no decisions-added counter, the "bans added" stat now reportsfw_bouncer_banned_ips.cs_parser_hits_totalandcs_active_decisionswere correct as written.Changes
internal/grafanadash/crowdsec.go— corrected dashboard queries.internal/grafanaalert/crowdsec.go— corrected the bouncer alert.docs/adr/0043-...md— records the host verification and drops the "unverified" caveats.website/docs/configuration/observability.md— updated signal descriptions.Remaining caveat (not closed)
Whether the otelcol
prometheus→otlphttp→ Grafana Cloud hop rewrites counter_totalsuffixes is not verified — it needs one live scrape against the real endpoint. Recorded as the ADR's open question rather than quietly closed.Test plan
go build ./...— cleango test ./internal/crowdsec/... ./internal/grafanadash/... ./internal/grafanaalert/... ./internal/otelcol/... ./program/...— all passgolangci-lint runon the changed packages — 0 issuesMerge Commit Message
fix(crowdsec): correct dashboard/alert metric names confirmed against a real host (ADR-0043)