Stop the KnowBe4 fetchers reporting unresolved config as a failing control

[21tmccauley]

The bug

The group and campaign titles three of these fetchers matched on were hardcoded to one tenant. Pointed anywhere else they emitted completion_rate: 0 and exited 0 — byte-identical to a tenant where the campaign resolved and genuinely nobody had trained:

typo'd config, users DID pass  -> {"completed_training": 0, "completion_rate": 0}
config OK, nobody passed       -> {"completed_training": 0, "completion_rate": 0}

Two very different states, one output, status: success on both. An auditor reads the first as a failing control. No test could tell them apart, which is why this went unnoticed.

The fix

Names come from config_schema now (high_risk_groups, role_specific_campaigns, developer_groups, developer_campaigns, security_awareness_campaigns, retraining_interval_days).

A name that matches nothing is not a fetcher failure — one typo must not turn a whole nightly run red. The fetcher exits 0 and reports every metric it could not measure as null, never 0, with a results.config_resolution block naming what was requested, what matched, and what the tenant actually has:

"config_resolution": {
  "status": "unresolved", "measurable": false,
  "groups": {"requested": ["Cloud Opps"], "matched": [], "unmatched": ["Cloud Opps"]},
  "groups_present_in_tenant": ["Cloud Ops", "IT Helpdesk", "Platform Team"]
},
"summary": {"total_high_risk_users": 0, "completed_training": null, "completion_rate": null}

null means "not measured". 0 still means "measured, and it is zero" — so a genuine 0% survives as the real finding it is. A key that is never wired at all is caught pre-flight by paramify validate, since these are required.

Also fixed — each a way to get wrong numbers from valid input

• Exact matching, not substring. A group configured as IT was also sweeping in AUDIT and Legal-IT.
• Config reaches jq as data (--args / $ARGS.positional) instead of being spliced into the filter text, where a title containing a quote or backslash produced a compile error. Making titles customer-supplied would have made that routine.
• One jq pass per fetcher. Each record was appended by re-running jq over the growing output file — quadratic. 3000 enrollments went from >120s (over the runner's 600s cap for a mid-size tenant) to ~3s. training_module_summary on an empty tenant is now {} rather than null.
• A non-array response (an error body returned with HTTP 200) is recorded as a failure instead of treated as a page; pagination looped forever on one before. Plus a 1000-page cap and printf '%s' over echo so a backslash in a title survives a non-bash shell.

Verification

Against the real Paramify KnowBe4 tenant (463 enrollments, 8 groups, 5 campaigns), all four fetchers exit 0 and resolve cleanly. Computed completion rates match KnowBe4's own completion_percentage exactly:

campaign	ours	KnowBe4
2026 Annual Security Awareness Training	94	94
Developers Training	100	100
Privileged Users Training (Before CloudOps Access)	100	100

Old vs new on that same real data: 62s → 7s, byte-identical module numbers.

Tests: tests/test_knowbe4_config_resolution.py (28 tests) drives the scripts the way the runner does — stub curl on PATH, config as env, no network. test_c_unmeasured_and_genuine_zero_are_distinguishable is the pair that pins the null-vs-0 distinction; it fails on the old code, which is the point — that bug was previously inexpressible. Suite: 325 passed (297 baseline + 28 new), contract gate 142 passed.

Notes for review

• completion_rate can now be null where it was always an integer. Nothing in this repo constrains it (payload: {} in the envelope schema, comparators/ is a template), but Paramify-side ingestion or any regex validator needs to tolerate null. This is the only change altering an existing field's type.
• Real group titles have suffixes (Cloud Ops Manual Add (FedRAMP Boundary Access)), so the old hardcoded "Cloud Ops" only ever worked because of substring matching. Configs must name the full title — examples/knowbe4_run.yaml and the category README show how to list them.
• Known gap, deliberately out of scope: the fetcher emits a WARN naming unmatched values, but the runner drops stderr on a zero exit (framework/api.py records stderr_tail only when exit_code != 0; executor.py does not forward stderr to on_line). So the evidence artifact is the only channel that reaches an operator today. One-line fix in api.py, left alone to avoid colliding with in-flight work on that file.
• Also out of scope: rate-limit retry/backoff. framework/runner/retry.py is still empty; a 429 fails the fetcher cleanly rather than producing partial evidence.

🤖 Generated with Claude Code

[21tmccauley]

Follow-up: 2e8dd3c — CI caught a Linux-only bug that would have shipped

The first push failed test (py 3.10–3.13) on one test. Worth reading, because it was a production bug, not a test problem.

The single-jq-pass refactor handed whole response arrays to jq as --argjson strings. Linux caps a single argument at MAX_ARG_STRLEN (131072 bytes): over that, execve fails E2BIG, jq never runs, and the fetcher writes an empty evidence file. macOS has no per-argument cap, so every local run passed.

deploy/ runs on Linux, and the real tenant sits just under the cliff:

  463 enrollments (real tenant) =    99,079 bytes   <- 25% under, passed by luck
 3000 enrollments               =   645,090 bytes   <- fails on Linux
 5000 enrollments               = 1,075,890 bytes   <- also clears macOS ARG_MAX

Fix: every large input goes to a temp file read with --slurpfile. Measured on identical mock data the change is free and behaviour-neutral — 9.67s → 9.69s, byte-identical output.

test_f now builds 5000 enrollments instead of 3000 so it also exceeds macOS's 1048576-byte total ARG_MAX. An argv-shaped regression now fails on a dev machine rather than waiting for CI.

This is the argument for keeping the scale fixture at 5000 even though it costs a few seconds: it is the only thing between an argv mistake and empty evidence in production.