Stop the KnowBe4 fetchers reporting unresolved config as a failing control

CHANGELOG.md

@@ -68,6 +68,40 @@ schemas and the `paramify` CLI — not the internal code.
 
 ### Fixed
 
+- **KnowBe4**: the three group- and campaign-scoped fetchers no longer report an
+  unresolved config as a failing control. The group and campaign titles they match
+  on were hardcoded to one tenant, so pointed anywhere else they emitted
+  `completion_rate: 0` and exited 0 — byte-identical to a tenant where the campaign
+  resolved and genuinely nobody had trained. Two very different states, one output,
+  and no assertion could tell them apart. The names now come from `config_schema`
+  (`high_risk_groups`, `role_specific_campaigns`, `developer_groups`,
+  `developer_campaigns`, `security_awareness_campaigns`, plus
+  `retraining_interval_days`), and a name that matches nothing in the tenant is
+  **not** a fetcher failure — one typo must not turn a whole nightly run red. The
+  fetcher exits 0 and reports every metric it could not measure as `null`, never
+  `0`, alongside a `results.config_resolution` block naming what was requested,
+  what matched, and what the tenant actually has. `null` means "not measured"; `0`
+  still means "measured, and it is zero", so a genuine 0% remains a real finding.
+  A config key that is never wired at all is caught pre-flight by
+  `paramify validate`, since these are `required`.
+- **KnowBe4**: names are matched exactly rather than as substrings. A group
+  configured as `IT` previously also swept in `AUDIT` and `Legal-IT`, inflating the
+  high-risk population.
+- **KnowBe4**: config values reach `jq` as data (`--args` / `$ARGS.positional`)
+  instead of being spliced into the filter text. A campaign title containing a
+  quote or backslash produced a jq compile error before; making the titles
+  customer-supplied would have turned that into a routine failure.
+- **KnowBe4**: all four fetchers assemble their evidence in one `jq` pass. Each
+  record was previously appended by re-running `jq` over the growing output file,
+  which was quadratic — 1500 enrollments took 69s and 3000 took over 120s, so a
+  mid-size tenant blew the runner's 600s cap. 3000 enrollments now completes in
+  about 3s. `training_module_summary` for an empty tenant is `{}` rather than
+  `null`.
+- **KnowBe4**: a response that is not a JSON array (an error body returned with
+  HTTP 200) is recorded as a failure instead of being treated as a page. Pagination
+  previously looped forever on such a body, bounded only by the runner's timeout.
+  Pagination also stops at a 1000-page cap, and `printf '%s'` replaces `echo` on
+  every API response so a backslash in a title survives a non-bash shell.
 - **TUI**: pressing the number of the tab you are already on no longer clears
   focus. Assigning `TabbedContent.active` the value it already holds fires no
   `TabActivated`, so nothing re-homed focus after it was cleared — and because a

examples/knowbe4_run.yaml

@@ -1,8 +1,9 @@
 # End-to-end test manifest: KnowBe4 module-based training summary.
 #
-# Single fetcher, single source. module_based_summary is chosen because it has
-# NO hardcoded customer group/campaign names — it works against any KnowBe4
-# tenant (the other 3 knowbe4 fetchers match specific group/campaign titles).
+# module_based_summary is the runnable default because it takes NO config — it
+# reports on whatever the tenant has, so it works against any KnowBe4 tenant
+# as-is. The other three fetchers measure specific groups and campaigns and so
+# must be told which ones; the commented block below shows that form.
 #
 # Set before running:
 #   export KNOWBE4_API_KEY=<your KnowBe4 Reporting API key>
@@ -15,3 +16,41 @@ run:
       secrets:
         api_key: ${env:KNOWBE4_API_KEY}
         region: ${env:KNOWBE4_REGION}
+
+    # The configurable fetchers. Replace the names with the exact group and
+    # campaign titles from YOUR tenant — they are matched exactly, not as
+    # substrings. List them with:
+    #
+    #   curl -s -H "Authorization: Bearer $KNOWBE4_API_KEY" \
+    #     "https://${KNOWBE4_REGION}.api.knowbe4.com/v1/groups?page=1" | jq -r '.[].name'
+    #   curl -s -H "Authorization: Bearer $KNOWBE4_API_KEY" \
+    #     "https://${KNOWBE4_REGION}.api.knowbe4.com/v1/training/campaigns?page=1" | jq -r '.[].name'
+    #
+    # A name that matches nothing does not fail the run: the fetcher exits 0 and
+    # reports every metric it could not measure as null (never 0), with
+    # results.config_resolution naming what did not resolve. See the category
+    # README for how to read that block.
+    #
+    # - use: knowbe4_security_awareness_training
+    #   secrets:
+    #     api_key: ${env:KNOWBE4_API_KEY}
+    #     region: ${env:KNOWBE4_REGION}
+    #   config:
+    #     security_awareness_campaigns: "2026 Annual Security Awareness Training"
+    #     retraining_interval_days: 365
+    #
+    # - use: knowbe4_high_risk_training
+    #   secrets:
+    #     api_key: ${env:KNOWBE4_API_KEY}
+    #     region: ${env:KNOWBE4_REGION}
+    #   config:
+    #     high_risk_groups: "Cloud Ops,IT,DevOps"
+    #     role_specific_campaigns: "Privileged Users Training (Before CloudOps Access)"
+    #
+    # - use: knowbe4_developer_specific_training
+    #   secrets:
+    #     api_key: ${env:KNOWBE4_API_KEY}
+    #     region: ${env:KNOWBE4_REGION}
+    #   config:
+    #     developer_groups: "Engineering,Developers"
+    #     developer_campaigns: "Developers Training"

fetchers/knowbe4/README.md

@@ -7,10 +7,68 @@ KnowBe4 fetchers pull security awareness and role-based training completion data
 | Variable | Required | Description |
 |---|---|---|
 | `KNOWBE4_API_KEY` | Yes | KnowBe4 Reporting API key |
-| `KNOWBE4_REGION` | Yes | KnowBe4 region identifier: `US`, `EU`, `CA`, `UK`, or `DE` |
+| `KNOWBE4_REGION` | Yes | KnowBe4 region subdomain: `us`, `eu`, `ca`, `uk`, or `de` |
 
 `KNOWBE4_REGION` sets the API hostname (`https://{region}.api.knowbe4.com`). Find your region from your KnowBe4 tenant URL or admin console.
 
+## Which group and campaign names to measure
+
+Three of the four fetchers measure *specific* groups and campaigns, so they need to be told which ones. There are no defaults — the names are yours, and a name is matched **exactly** against the titles in your tenant (not as a substring, so `IT` will not pull in `AUDIT`). Each value is a comma-separated list; whitespace around each name is trimmed.
+
+| Fetcher | Config key | Env var |
+|---|---|---|
+| `knowbe4_security_awareness_training` | `security_awareness_campaigns` | `KNOWBE4_SECURITY_AWARENESS_CAMPAIGNS` |
+| | `retraining_interval_days` (optional, default 365) | `KNOWBE4_RETRAINING_INTERVAL_DAYS` |
+| `knowbe4_high_risk_training` | `high_risk_groups` | `KNOWBE4_HIGH_RISK_GROUPS` |
+| | `role_specific_campaigns` | `KNOWBE4_ROLE_SPECIFIC_CAMPAIGNS` |
+| `knowbe4_developer_specific_training` | `developer_groups` | `KNOWBE4_DEVELOPER_GROUPS` |
+| | `developer_campaigns` | `KNOWBE4_DEVELOPER_CAMPAIGNS` |
+| `knowbe4_module_based_summary` | *none* | — |
+
+`knowbe4_module_based_summary` reports on whatever the tenant has, so it needs no config and works against any tenant as-is.
+
+To find the exact titles to use:
+
+```bash
+curl -s -H "Authorization: Bearer $KNOWBE4_API_KEY" \
+  "https://${KNOWBE4_REGION}.api.knowbe4.com/v1/groups?page=1" | jq -r '.[].name'
+curl -s -H "Authorization: Bearer $KNOWBE4_API_KEY" \
+  "https://${KNOWBE4_REGION}.api.knowbe4.com/v1/training/campaigns?page=1" | jq -r '.[].name'
+```
+
+### What happens when a name does not match
+
+A name that matches nothing in your tenant does **not** fail the fetcher — one typo should not turn a whole nightly run red. Instead the run exits 0 and the evidence says it could not measure:
+
+```json
+"config_resolution": {
+  "status": "unresolved",
+  "measurable": false,
+  "groups": {
+    "requested": ["Cloud Opps"],
+    "matched": [],
+    "unmatched": ["Cloud Opps"]
+  },
+  "groups_present_in_tenant": ["Cloud Ops", "IT Helpdesk", "Platform Team"]
+},
+"summary": {
+  "total_high_risk_users": 0,
+  "completed_training": null,
+  "completion_rate": null
+}
+```
+
+Read it like this:
+
+- **`null` means "not measured."** **`0` means "measured, and it is zero."** A metric the fetcher could not compute is never reported as 0, because 0 reads as a genuine failing control. If `completion_rate` is `0`, nobody completed the training and that is a real finding.
+- **`status`** is `resolved` (every name matched), `partial` (some matched), or `unresolved` (a dimension matched nothing, so nothing is measurable).
+- **`*_present_in_tenant`** appears only when something failed to match, and lists what your tenant actually has — usually enough to spot the typo without opening a shell.
+- Counts of what was *discovered* (`total_groups`, `total_campaigns`, `total_*_users`) stay real numbers even when unresolved.
+
+A `WARN` line naming the unmatched values is also written to stderr, so it shows up in the run output.
+
+A config key that is never wired at all is a different case: it is `required`, so `paramify validate` catches it before any run.
+
 ## Creating an API key
 
 The KnowBe4 Reporting API is typically available to Platinum and Diamond customers. Contact KnowBe4 support if access is not enabled on your account.
@@ -27,15 +85,37 @@ The KnowBe4 Reporting API is typically available to Platinum and Diamond custome
 
 ## Wiring into a manifest
 
-All KnowBe4 fetchers share the same two secrets:
+Every KnowBe4 fetcher takes the same two secrets; three of them also take config.
 
 ```bash
+# No config needed — runs against any tenant.
+paramify manifest add knowbe4_module_based_summary
+paramify manifest set-secret knowbe4_module_based_summary api_key KNOWBE4_API_KEY
+paramify manifest set-secret knowbe4_module_based_summary region KNOWBE4_REGION
+
+# Needs the campaign name(s) to measure.
 paramify manifest add knowbe4_security_awareness_training
 paramify manifest set-secret knowbe4_security_awareness_training api_key KNOWBE4_API_KEY
 paramify manifest set-secret knowbe4_security_awareness_training region KNOWBE4_REGION
+paramify manifest set-config knowbe4_security_awareness_training \
+  security_awareness_campaigns "2026 Annual Security Awareness Training"
+
+# Needs both the groups and the campaign(s) they must complete.
+paramify manifest add knowbe4_high_risk_training
+paramify manifest set-secret knowbe4_high_risk_training api_key KNOWBE4_API_KEY
+paramify manifest set-secret knowbe4_high_risk_training region KNOWBE4_REGION
+paramify manifest set-config knowbe4_high_risk_training high_risk_groups "Cloud Ops,IT,DevOps"
+paramify manifest set-config knowbe4_high_risk_training role_specific_campaigns \
+  "Privileged Users Training (Before CloudOps Access)"
+```
+
+Then confirm every required value is set:
+
+```bash
+paramify validate
 ```
 
-Repeat `add` + `set-secret` for each additional KnowBe4 fetcher. Use `paramify catalog` to see all available fetchers.
+Use `paramify catalog` to see all available fetchers, and `paramify describe <fetcher>` for one fetcher's config and secrets. See `examples/knowbe4_run.yaml` for the manifest form.
 
 ## Smoke test
 
@@ -53,5 +133,7 @@ curl -s -H "Authorization: Bearer $KNOWBE4_API_KEY" \
 
 ## Notes
 
-- Use uppercase for the region value (`US`, not `us`).
-- Fetchers paginate using `page=N` until an empty page is returned.
+- Use the lowercase region subdomain (`us`). Hostnames are case-insensitive, so `US` also resolves, but the lowercase form matches KnowBe4's own documentation and `examples/knowbe4_run.yaml`.
+- A comma separates names, and there is no escape for a name that itself contains a comma. Such a name will split into fragments that match nothing — which surfaces as `status: unresolved` naming the fragments, not as silently wrong numbers.
+- Fetchers paginate using `page=N` until an empty page is returned, and stop at a 1000-page cap. A response that is not a JSON array (an error body returned with HTTP 200) is recorded as a failure rather than treated as data, and fails the fetcher.
+- A failed API call still fails the fetcher (nonzero exit). Only config that does not resolve is reported as evidence instead.