diff --git a/CHANGELOG.md b/CHANGELOG.md index 73eed73..22c3b51 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -68,6 +68,40 @@ schemas and the `paramify` CLI — not the internal code. ### Fixed +- **KnowBe4**: the three group- and campaign-scoped fetchers no longer report an + unresolved config as a failing control. The group and campaign titles they match + on were hardcoded to one tenant, so pointed anywhere else they emitted + `completion_rate: 0` and exited 0 — byte-identical to a tenant where the campaign + resolved and genuinely nobody had trained. Two very different states, one output, + and no assertion could tell them apart. The names now come from `config_schema` + (`high_risk_groups`, `role_specific_campaigns`, `developer_groups`, + `developer_campaigns`, `security_awareness_campaigns`, plus + `retraining_interval_days`), and a name that matches nothing in the tenant is + **not** a fetcher failure — one typo must not turn a whole nightly run red. The + fetcher exits 0 and reports every metric it could not measure as `null`, never + `0`, alongside a `results.config_resolution` block naming what was requested, + what matched, and what the tenant actually has. `null` means "not measured"; `0` + still means "measured, and it is zero", so a genuine 0% remains a real finding. + A config key that is never wired at all is caught pre-flight by + `paramify validate`, since these are `required`. +- **KnowBe4**: names are matched exactly rather than as substrings. A group + configured as `IT` previously also swept in `AUDIT` and `Legal-IT`, inflating the + high-risk population. +- **KnowBe4**: config values reach `jq` as data (`--args` / `$ARGS.positional`) + instead of being spliced into the filter text. A campaign title containing a + quote or backslash produced a jq compile error before; making the titles + customer-supplied would have turned that into a routine failure. +- **KnowBe4**: all four fetchers assemble their evidence in one `jq` pass. Each + record was previously appended by re-running `jq` over the growing output file, + which was quadratic — 1500 enrollments took 69s and 3000 took over 120s, so a + mid-size tenant blew the runner's 600s cap. 3000 enrollments now completes in + about 3s. `training_module_summary` for an empty tenant is `{}` rather than + `null`. +- **KnowBe4**: a response that is not a JSON array (an error body returned with + HTTP 200) is recorded as a failure instead of being treated as a page. Pagination + previously looped forever on such a body, bounded only by the runner's timeout. + Pagination also stops at a 1000-page cap, and `printf '%s'` replaces `echo` on + every API response so a backslash in a title survives a non-bash shell. - **TUI**: pressing the number of the tab you are already on no longer clears focus. Assigning `TabbedContent.active` the value it already holds fires no `TabActivated`, so nothing re-homed focus after it was cleared — and because a diff --git a/examples/knowbe4_run.yaml b/examples/knowbe4_run.yaml index 7ec9cc5..5abee24 100644 --- a/examples/knowbe4_run.yaml +++ b/examples/knowbe4_run.yaml @@ -1,8 +1,9 @@ # End-to-end test manifest: KnowBe4 module-based training summary. # -# Single fetcher, single source. module_based_summary is chosen because it has -# NO hardcoded customer group/campaign names — it works against any KnowBe4 -# tenant (the other 3 knowbe4 fetchers match specific group/campaign titles). +# module_based_summary is the runnable default because it takes NO config — it +# reports on whatever the tenant has, so it works against any KnowBe4 tenant +# as-is. The other three fetchers measure specific groups and campaigns and so +# must be told which ones; the commented block below shows that form. # # Set before running: # export KNOWBE4_API_KEY= @@ -15,3 +16,41 @@ run: secrets: api_key: ${env:KNOWBE4_API_KEY} region: ${env:KNOWBE4_REGION} + + # The configurable fetchers. Replace the names with the exact group and + # campaign titles from YOUR tenant — they are matched exactly, not as + # substrings. List them with: + # + # curl -s -H "Authorization: Bearer $KNOWBE4_API_KEY" \ + # "https://${KNOWBE4_REGION}.api.knowbe4.com/v1/groups?page=1" | jq -r '.[].name' + # curl -s -H "Authorization: Bearer $KNOWBE4_API_KEY" \ + # "https://${KNOWBE4_REGION}.api.knowbe4.com/v1/training/campaigns?page=1" | jq -r '.[].name' + # + # A name that matches nothing does not fail the run: the fetcher exits 0 and + # reports every metric it could not measure as null (never 0), with + # results.config_resolution naming what did not resolve. See the category + # README for how to read that block. + # + # - use: knowbe4_security_awareness_training + # secrets: + # api_key: ${env:KNOWBE4_API_KEY} + # region: ${env:KNOWBE4_REGION} + # config: + # security_awareness_campaigns: "2026 Annual Security Awareness Training" + # retraining_interval_days: 365 + # + # - use: knowbe4_high_risk_training + # secrets: + # api_key: ${env:KNOWBE4_API_KEY} + # region: ${env:KNOWBE4_REGION} + # config: + # high_risk_groups: "Cloud Ops,IT,DevOps" + # role_specific_campaigns: "Privileged Users Training (Before CloudOps Access)" + # + # - use: knowbe4_developer_specific_training + # secrets: + # api_key: ${env:KNOWBE4_API_KEY} + # region: ${env:KNOWBE4_REGION} + # config: + # developer_groups: "Engineering,Developers" + # developer_campaigns: "Developers Training" diff --git a/fetchers/knowbe4/README.md b/fetchers/knowbe4/README.md index 32b3cf4..4d080df 100644 --- a/fetchers/knowbe4/README.md +++ b/fetchers/knowbe4/README.md @@ -7,10 +7,68 @@ KnowBe4 fetchers pull security awareness and role-based training completion data | Variable | Required | Description | |---|---|---| | `KNOWBE4_API_KEY` | Yes | KnowBe4 Reporting API key | -| `KNOWBE4_REGION` | Yes | KnowBe4 region identifier: `US`, `EU`, `CA`, `UK`, or `DE` | +| `KNOWBE4_REGION` | Yes | KnowBe4 region subdomain: `us`, `eu`, `ca`, `uk`, or `de` | `KNOWBE4_REGION` sets the API hostname (`https://{region}.api.knowbe4.com`). Find your region from your KnowBe4 tenant URL or admin console. +## Which group and campaign names to measure + +Three of the four fetchers measure *specific* groups and campaigns, so they need to be told which ones. There are no defaults — the names are yours, and a name is matched **exactly** against the titles in your tenant (not as a substring, so `IT` will not pull in `AUDIT`). Each value is a comma-separated list; whitespace around each name is trimmed. + +| Fetcher | Config key | Env var | +|---|---|---| +| `knowbe4_security_awareness_training` | `security_awareness_campaigns` | `KNOWBE4_SECURITY_AWARENESS_CAMPAIGNS` | +| | `retraining_interval_days` (optional, default 365) | `KNOWBE4_RETRAINING_INTERVAL_DAYS` | +| `knowbe4_high_risk_training` | `high_risk_groups` | `KNOWBE4_HIGH_RISK_GROUPS` | +| | `role_specific_campaigns` | `KNOWBE4_ROLE_SPECIFIC_CAMPAIGNS` | +| `knowbe4_developer_specific_training` | `developer_groups` | `KNOWBE4_DEVELOPER_GROUPS` | +| | `developer_campaigns` | `KNOWBE4_DEVELOPER_CAMPAIGNS` | +| `knowbe4_module_based_summary` | *none* | — | + +`knowbe4_module_based_summary` reports on whatever the tenant has, so it needs no config and works against any tenant as-is. + +To find the exact titles to use: + +```bash +curl -s -H "Authorization: Bearer $KNOWBE4_API_KEY" \ + "https://${KNOWBE4_REGION}.api.knowbe4.com/v1/groups?page=1" | jq -r '.[].name' +curl -s -H "Authorization: Bearer $KNOWBE4_API_KEY" \ + "https://${KNOWBE4_REGION}.api.knowbe4.com/v1/training/campaigns?page=1" | jq -r '.[].name' +``` + +### What happens when a name does not match + +A name that matches nothing in your tenant does **not** fail the fetcher — one typo should not turn a whole nightly run red. Instead the run exits 0 and the evidence says it could not measure: + +```json +"config_resolution": { + "status": "unresolved", + "measurable": false, + "groups": { + "requested": ["Cloud Opps"], + "matched": [], + "unmatched": ["Cloud Opps"] + }, + "groups_present_in_tenant": ["Cloud Ops", "IT Helpdesk", "Platform Team"] +}, +"summary": { + "total_high_risk_users": 0, + "completed_training": null, + "completion_rate": null +} +``` + +Read it like this: + +- **`null` means "not measured."** **`0` means "measured, and it is zero."** A metric the fetcher could not compute is never reported as 0, because 0 reads as a genuine failing control. If `completion_rate` is `0`, nobody completed the training and that is a real finding. +- **`status`** is `resolved` (every name matched), `partial` (some matched), or `unresolved` (a dimension matched nothing, so nothing is measurable). +- **`*_present_in_tenant`** appears only when something failed to match, and lists what your tenant actually has — usually enough to spot the typo without opening a shell. +- Counts of what was *discovered* (`total_groups`, `total_campaigns`, `total_*_users`) stay real numbers even when unresolved. + +A `WARN` line naming the unmatched values is also written to stderr, so it shows up in the run output. + +A config key that is never wired at all is a different case: it is `required`, so `paramify validate` catches it before any run. + ## Creating an API key The KnowBe4 Reporting API is typically available to Platinum and Diamond customers. Contact KnowBe4 support if access is not enabled on your account. @@ -27,15 +85,37 @@ The KnowBe4 Reporting API is typically available to Platinum and Diamond custome ## Wiring into a manifest -All KnowBe4 fetchers share the same two secrets: +Every KnowBe4 fetcher takes the same two secrets; three of them also take config. ```bash +# No config needed — runs against any tenant. +paramify manifest add knowbe4_module_based_summary +paramify manifest set-secret knowbe4_module_based_summary api_key KNOWBE4_API_KEY +paramify manifest set-secret knowbe4_module_based_summary region KNOWBE4_REGION + +# Needs the campaign name(s) to measure. paramify manifest add knowbe4_security_awareness_training paramify manifest set-secret knowbe4_security_awareness_training api_key KNOWBE4_API_KEY paramify manifest set-secret knowbe4_security_awareness_training region KNOWBE4_REGION +paramify manifest set-config knowbe4_security_awareness_training \ + security_awareness_campaigns "2026 Annual Security Awareness Training" + +# Needs both the groups and the campaign(s) they must complete. +paramify manifest add knowbe4_high_risk_training +paramify manifest set-secret knowbe4_high_risk_training api_key KNOWBE4_API_KEY +paramify manifest set-secret knowbe4_high_risk_training region KNOWBE4_REGION +paramify manifest set-config knowbe4_high_risk_training high_risk_groups "Cloud Ops,IT,DevOps" +paramify manifest set-config knowbe4_high_risk_training role_specific_campaigns \ + "Privileged Users Training (Before CloudOps Access)" +``` + +Then confirm every required value is set: + +```bash +paramify validate ``` -Repeat `add` + `set-secret` for each additional KnowBe4 fetcher. Use `paramify catalog` to see all available fetchers. +Use `paramify catalog` to see all available fetchers, and `paramify describe ` for one fetcher's config and secrets. See `examples/knowbe4_run.yaml` for the manifest form. ## Smoke test @@ -53,5 +133,7 @@ curl -s -H "Authorization: Bearer $KNOWBE4_API_KEY" \ ## Notes -- Use uppercase for the region value (`US`, not `us`). -- Fetchers paginate using `page=N` until an empty page is returned. +- Use the lowercase region subdomain (`us`). Hostnames are case-insensitive, so `US` also resolves, but the lowercase form matches KnowBe4's own documentation and `examples/knowbe4_run.yaml`. +- A comma separates names, and there is no escape for a name that itself contains a comma. Such a name will split into fragments that match nothing — which surfaces as `status: unresolved` naming the fragments, not as silently wrong numbers. +- Fetchers paginate using `page=N` until an empty page is returned, and stop at a 1000-page cap. A response that is not a JSON array (an error body returned with HTTP 200) is recorded as a failure rather than treated as data, and fails the fetcher. +- A failed API call still fails the fetcher (nonzero exit). Only config that does not resolve is reported as evidence instead. diff --git a/fetchers/knowbe4/developer_specific_training/fetcher.sh b/fetchers/knowbe4/developer_specific_training/fetcher.sh index b09fc44..cc80448 100755 --- a/fetchers/knowbe4/developer_specific_training/fetcher.sh +++ b/fetchers/knowbe4/developer_specific_training/fetcher.sh @@ -2,212 +2,296 @@ # # KnowBe4 — Developer-Specific Training Validation # -# Identifies users in developer-related groups (Engineering, Developer, Dev), -# checks enrollment/completion status for the "Developers Training" campaign, -# and reports per-user status with summary metrics. +# Checks completion of the campaign(s) named in KNOWBE4_DEVELOPER_CAMPAIGNS for the +# members of the groups named in KNOWBE4_DEVELOPER_GROUPS. +# +# Group and campaign names come from config, never from this file: hardcoding them +# made the fetcher report a confident 0% on any tenant that named things +# differently. Names are matched EXACTLY — the substring matching this replaced let +# a group named "Dev" pull in anything containing "Dev". +# +# A name that matches nothing in the tenant is NOT a fetcher failure — one typo must +# not turn a whole run red — so it exits 0 and reports every metric it could not +# measure as null, with results.config_resolution naming what did not resolve. +# null means "not measured"; 0 means "measured, and it is zero". # # Output: $EVIDENCE_DIR/knowbe4_developer_specific_training.json -# Required env: KNOWBE4_API_KEY, KNOWBE4_REGION +# Required env: KNOWBE4_API_KEY, KNOWBE4_REGION, +# KNOWBE4_DEVELOPER_GROUPS, KNOWBE4_DEVELOPER_CAMPAIGNS set -o pipefail # Interim v0.x: load .env if present. Runner + secret resolver replaces this. [ -f .env ] && { set -a; . .env; set +a; } +FETCHER=knowbe4_developer_specific_training + OUTPUT_DIR="${EVIDENCE_DIR:-./evidence}" mkdir -p "$OUTPUT_DIR" +OUTPUT_JSON="$OUTPUT_DIR/${FETCHER}.json" + +_FAILURE_LOG="$(mktemp -t ${FETCHER}_fail.XXXXXX)" +# Large jq inputs go in by FILE, never as an argv string: Linux caps a single +# argument at MAX_ARG_STRLEN (128KB), so --argjson with a full enrollments or +# members array fails execve with E2BIG and jq never runs — leaving an empty +# evidence file. macOS has no per-argument cap, which is exactly how this hid +# during local dev. +_TMP_ENROLLMENTS="$(mktemp -t ${FETCHER}_enroll.XXXXXX)" +_TMP_MEMBERS="$(mktemp -t ${FETCHER}_members.XXXXXX)" +_TMP_MATCHED_GROUPS="$(mktemp -t ${FETCHER}_mgroups.XXXXXX)" +_TMP_MATCHED_CAMPAIGNS="$(mktemp -t ${FETCHER}_mcamps.XXXXXX)" +_TMP_GROUPS_PRESENT="$(mktemp -t ${FETCHER}_gpresent.XXXXXX)" +_TMP_CAMPAIGNS_PRESENT="$(mktemp -t ${FETCHER}_cpresent.XXXXXX)" +_TMP_GROUP_RES="$(mktemp -t ${FETCHER}_gres.XXXXXX)" +_TMP_CAMPAIGN_RES="$(mktemp -t ${FETCHER}_cres.XXXXXX)" +trap 'rm -f "$_FAILURE_LOG" "$_TMP_ENROLLMENTS" "$_TMP_MEMBERS" \ + "$_TMP_MATCHED_GROUPS" "$_TMP_MATCHED_CAMPAIGNS" "$_TMP_GROUPS_PRESENT" \ + "$_TMP_CAMPAIGNS_PRESENT" "$_TMP_GROUP_RES" "$_TMP_CAMPAIGN_RES"' EXIT + +log_info() { printf '%s INFO %s %s\n' "$(date -u +'%Y-%m-%d %H:%M:%S')" "$FETCHER" "$*" >&2; } +log_warn() { printf '%s WARN %s %s\n' "$(date -u +'%Y-%m-%d %H:%M:%S')" "$FETCHER" "$*" >&2; } +log_error() { printf '%s ERROR %s %s\n' "$(date -u +'%Y-%m-%d %H:%M:%S')" "$FETCHER" "$*" >&2; } if [ -z "${KNOWBE4_API_KEY:-}" ]; then - echo "ERROR knowbe4_developer_specific_training: KNOWBE4_API_KEY is not set" >&2 + log_error "KNOWBE4_API_KEY is not set" exit 1 fi if [ -z "${KNOWBE4_REGION:-}" ]; then - echo "ERROR knowbe4_developer_specific_training: KNOWBE4_REGION is not set" >&2 + log_error "KNOWBE4_REGION is not set" exit 1 fi -OUTPUT_JSON="$OUTPUT_DIR/knowbe4_developer_specific_training.json" -_FETCHER_TMP_JSON="$(mktemp -t knowbe4_developer_specific_training.XXXXXX.json)" -_FAILURE_LOG="$(mktemp -t knowbe4_developer_specific_training_fail.XXXXXX)" -trap 'rm -f "$_FETCHER_TMP_JSON" "$_FAILURE_LOG"' EXIT - -log_info() { - printf '%s INFO knowbe4_developer_specific_training %s\n' "$(date -u +'%Y-%m-%d %H:%M:%S')" "$*" >&2 -} - -log_error() { - printf '%s ERROR knowbe4_developer_specific_training %s\n' "$(date -u +'%Y-%m-%d %H:%M:%S')" "$*" >&2 +# Split a comma-separated config value into _SPLIT[], trimming surrounding +# whitespace and dropping empty elements. `read -ra`, not an unquoted expansion, +# so a name containing a glob character is never pathname-expanded. +split_config() { + local raw="$1" item + local -a parts=() + _SPLIT=() + IFS=',' read -ra parts <<< "$raw" + for item in "${parts[@]}"; do + item="${item#"${item%%[![:space:]]*}"}" + item="${item%"${item##*[![:space:]]}"}" + [ -n "$item" ] && _SPLIT+=("$item") + done } -DEVELOPER_CAMPAIGNS=("Developers Training") - +# printf '%s', never echo: bash's echo leaves backslashes alone but sh's and zsh's +# do not, and KnowBe4 group and campaign titles are free text. make_api_call() { local endpoint=$1 local url="https://${KNOWBE4_REGION}.api.knowbe4.com/v1/${endpoint}" local response - if ! response=$(curl -sf -H "Authorization: Bearer ${KNOWBE4_API_KEY}" -H "Content-Type: application/json" "${url}"); then - echo "GET ${endpoint}" >> "$_FAILURE_LOG" - echo "{}" + if ! response=$(curl -sf -H "Authorization: Bearer ${KNOWBE4_API_KEY}" \ + -H "Content-Type: application/json" "${url}"); then + printf 'GET %s\n' "$endpoint" >> "$_FAILURE_LOG" + printf '%s' '[]' return 1 fi - if ! echo "$response" | jq . >/dev/null 2>&1; then - echo "GET ${endpoint} (invalid JSON)" >> "$_FAILURE_LOG" - echo "{}" + # Anything that is not a JSON array is an error body, not a page. Recording it + # rather than treating it as data is what stops pagination looping forever on a + # 200-with-error-payload. + if ! printf '%s' "$response" | jq -e 'type == "array"' >/dev/null 2>&1; then + printf 'GET %s (response was not a JSON array)\n' "$endpoint" >> "$_FAILURE_LOG" + printf '%s' '[]' return 1 fi - echo "$response" - return 0 + printf '%s' "$response" } +_MAX_PAGES=1000 + make_paginated_api_call() { local endpoint="$1" - local page=1 - local all_results="[]" - local separator + local page=1 all_results="[]" response count separator if [[ "$endpoint" == *\?* ]]; then separator="&"; else separator="?"; fi - while true; do - response=$(make_api_call "${endpoint}${separator}page=${page}") - count=$(echo "$response" | jq 'length' 2>/dev/null || echo 0) - if [ "$count" -eq 0 ]; then break; fi - all_results=$(jq -s '.[0] + .[1]' <(echo "$all_results") <(echo "$response")) + while [ "$page" -le "$_MAX_PAGES" ]; do + if ! response=$(make_api_call "${endpoint}${separator}page=${page}"); then + break + fi + count=$(printf '%s' "$response" | jq 'length') + [ "$count" -eq 0 ] && break + all_results=$(jq -s '.[0] + .[1]' \ + <(printf '%s' "$all_results") <(printf '%s' "$response")) page=$((page + 1)) done - echo "$all_results" + if [ "$page" -gt "$_MAX_PAGES" ]; then + printf 'GET %s (stopped at the %s-page cap)\n' "$endpoint" "$_MAX_PAGES" >> "$_FAILURE_LOG" + fi + printf '%s' "$all_results" +} + +# Requested vs present, computed by jq from data. The names are never spliced into +# a jq program, so quotes, backslashes and $ in a title are just text. +resolve_names() { + local present_json="$1"; shift + jq -n -c --argjson present "$present_json" --args ' + ($ARGS.positional) as $req + | { + requested: $req, + matched: [ $req[] | select(. as $n | $present | any(. == $n)) ], + unmatched: [ $req[] | select(. as $n | $present | any(. == $n) | not) ] + } + ' -- "$@" } -# Initialize output structure. -echo '{ - "results": { - "developer_users": [], - "developer_campaigns": [], - "enrollments": [], - "user_training_status": {}, - "developer_groups": [], - "summary": { - "total_developer_users": 0, - "completed_training": 0, - "in_progress": 0, - "past_due": 0, - "not_started": 0, - "completion_rate": 0, - "total_campaigns": 0, - "total_groups": 0 - } - } -}' > "$OUTPUT_JSON" - -users_response=$(make_paginated_api_call "users") +split_config "${KNOWBE4_DEVELOPER_GROUPS:-}" +requested_groups=("${_SPLIT[@]}") +split_config "${KNOWBE4_DEVELOPER_CAMPAIGNS:-}" +requested_campaigns=("${_SPLIT[@]}") + +[ "${#requested_groups[@]}" -eq 0 ] && \ + log_warn "KNOWBE4_DEVELOPER_GROUPS is empty — there is no population to measure" +[ "${#requested_campaigns[@]}" -eq 0 ] && \ + log_warn "KNOWBE4_DEVELOPER_CAMPAIGNS is empty — there is no campaign to measure" + campaigns_response=$(make_paginated_api_call "training/campaigns") -enrollments_response=$(make_paginated_api_call "training/enrollments?exclude_archived_users=true&include_campaign_id=true") groups_response=$(make_paginated_api_call "groups") +enrollments_response=$(make_paginated_api_call \ + "training/enrollments?exclude_archived_users=true&include_campaign_id=true") -# Capture developer campaigns -for campaign_name in "${DEVELOPER_CAMPAIGNS[@]}"; do - echo "$campaigns_response" | jq -c --arg name "$campaign_name" \ - '.[] | select(.name == $name)' | while read -r campaign; do - jq --argjson campaign "$campaign" \ - '.results.developer_campaigns += [$campaign]' \ - "$OUTPUT_JSON" > "$_FETCHER_TMP_JSON" && mv "$_FETCHER_TMP_JSON" "$OUTPUT_JSON" - done -done - -developer_groups=("Engineering" "Developer" "Developers" "Dev") -developer_users=() - -while read -r group_json; do - group_name=$(echo "$group_json" | jq -r '.name') - group_id=$(echo "$group_json" | jq -r '.id') - for group in "${developer_groups[@]}"; do - if [[ "$group_name" == *"$group"* ]]; then - group_members=$(make_paginated_api_call "groups/$group_id/members") - jq --arg name "$group_name" --arg id "$group_id" \ - '.results.developer_groups += [{"name": $name, "id": $id}]' \ - "$OUTPUT_JSON" > "$_FETCHER_TMP_JSON" && mv "$_FETCHER_TMP_JSON" "$OUTPUT_JSON" - while read -r user_json; do - user_email=$(echo "$user_json" | jq -r '.email') - if [[ ! " ${developer_users[@]} " =~ " ${user_email} " ]]; then - developer_users+=("$user_email") - minimal_user=$(echo "$user_json" | jq '{id: .id, email: .email, status: .status}') - jq --argjson user "$minimal_user" \ - '.results.developer_users += [$user]' \ - "$OUTPUT_JSON" > "$_FETCHER_TMP_JSON" && mv "$_FETCHER_TMP_JSON" "$OUTPUT_JSON" - fi - done < <(echo "$group_members" | jq -c '.[] | select(.status == "active")') - break - fi - done -done < <(echo "$groups_response" | jq -c '.[]') - -for email in "${developer_users[@]}"; do - user=$(echo "$users_response" | jq -c --arg email "$email" '.[] | select(.email == $email)') - user_id=$(echo "$user" | jq -r '.id') - - campaign_filter=$(printf ' or .campaign_name == "%s"' "${DEVELOPER_CAMPAIGNS[@]}") - campaign_filter=${campaign_filter# or } - - user_enrollments=$(echo "$enrollments_response" | jq -c \ - --arg user_id "$user_id" \ - ".[] | select(.user.id == (\$user_id|tonumber) and ( $campaign_filter ))" | jq -s '.') - - user_status="not_started" - if [ "$user_enrollments" != "[]" ]; then - echo "$user_enrollments" | jq -c '.[] | del(.policy_acknowledged)' | while read -r e; do - jq --argjson enrollment "$e" \ - '.results.enrollments += [$enrollment]' \ - "$OUTPUT_JSON" > "$_FETCHER_TMP_JSON" && mv "$_FETCHER_TMP_JSON" "$OUTPUT_JSON" - done - - if echo "$user_enrollments" | jq -e 'all(.status == "Passed")' >/dev/null; then - user_status="completed" - elif echo "$user_enrollments" | jq -e 'any(.status == "Past Due")' >/dev/null; then - user_status="past_due" - elif echo "$user_enrollments" | jq -e 'any(.status == "In Progress")' >/dev/null; then - user_status="in_progress" - fi - fi +campaigns_present=$(printf '%s' "$campaigns_response" | jq -c '[.[].name] | unique') +groups_present=$(printf '%s' "$groups_response" | jq -c '[.[].name] | unique') +campaign_res=$(resolve_names "$campaigns_present" "${requested_campaigns[@]}") +group_res=$(resolve_names "$groups_present" "${requested_groups[@]}") - jq --arg email "$email" --arg status "$user_status" \ - '.results.user_training_status[$email] = $status' \ - "$OUTPUT_JSON" > "$_FETCHER_TMP_JSON" && mv "$_FETCHER_TMP_JSON" "$OUTPUT_JSON" -done - -total_developer_users=$(jq '.results.developer_users | length' "$OUTPUT_JSON") -completed_training=$(jq '.results.user_training_status | to_entries | map(select(.value=="completed")) | length' "$OUTPUT_JSON") -in_progress=$(jq '.results.user_training_status | to_entries | map(select(.value=="in_progress")) | length' "$OUTPUT_JSON") -past_due=$(jq '.results.user_training_status | to_entries | map(select(.value=="past_due")) | length' "$OUTPUT_JSON") -not_started=$(jq '.results.user_training_status | to_entries | map(select(.value=="not_started")) | length' "$OUTPUT_JSON") -campaigns=$(jq '.results.developer_campaigns | length' "$OUTPUT_JSON") -groups=$(jq '.results.developer_groups | length' "$OUTPUT_JSON") -completion_rate=0 -if [ "$total_developer_users" -gt 0 ]; then - completion_rate=$((completed_training * 100 / total_developer_users)) -fi +matched_campaign_objs=$(jq -c --argjson res "$campaign_res" \ + '[ .[] | select(.name as $n | $res.matched | any(. == $n)) ]' <<< "$campaigns_response") +matched_groups=$(jq -c --argjson res "$group_res" \ + '[ .[] | select(.name as $n | $res.matched | any(. == $n)) | {id: (.id|tostring), name: .name} ]' \ + <<< "$groups_response") + +# Members of the matched groups only — an unmatched name costs no API call. +# `while read` over a process substitution, so the array survives (a pipeline +# would run the loop in a subshell and silently discard it). +members="[]" +while read -r group_id; do + [ -z "$group_id" ] && continue + page=$(make_paginated_api_call "groups/${group_id}/members") + members=$(jq -s '.[0] + .[1]' <(printf '%s' "$members") <(printf '%s' "$page")) +done < <(printf '%s' "$matched_groups" | jq -r '.[].id') + +# One jq pass builds the whole document. The previous version re-ran jq over the +# growing output file once per record, which was quadratic. +printf '%s' "$enrollments_response" > "$_TMP_ENROLLMENTS" +printf '%s' "$members" > "$_TMP_MEMBERS" +printf '%s' "$matched_groups" > "$_TMP_MATCHED_GROUPS" +printf '%s' "$matched_campaign_objs" > "$_TMP_MATCHED_CAMPAIGNS" +printf '%s' "$campaigns_present" > "$_TMP_CAMPAIGNS_PRESENT" +printf '%s' "$groups_present" > "$_TMP_GROUPS_PRESENT" +printf '%s' "$campaign_res" > "$_TMP_CAMPAIGN_RES" +printf '%s' "$group_res" > "$_TMP_GROUP_RES" + +jq -n \ + --slurpfile enrollments_in "$_TMP_ENROLLMENTS" \ + --slurpfile members_in "$_TMP_MEMBERS" \ + --slurpfile matched_groups_in "$_TMP_MATCHED_GROUPS" \ + --slurpfile matched_campaigns_in "$_TMP_MATCHED_CAMPAIGNS" \ + --slurpfile campaign_res_in "$_TMP_CAMPAIGN_RES" \ + --slurpfile group_res_in "$_TMP_GROUP_RES" \ + --slurpfile campaigns_present_in "$_TMP_CAMPAIGNS_PRESENT" \ + --slurpfile groups_present_in "$_TMP_GROUPS_PRESENT" ' + # --slurpfile wraps the contents of each file in an array; [0] unwraps it. + ($enrollments_in[0]) as $enrollments + | ($members_in[0]) as $members + | ($matched_groups_in[0]) as $matched_groups + | ($matched_campaigns_in[0]) as $matched_campaigns + | ($campaign_res_in[0]) as $campaign_res + | ($group_res_in[0]) as $group_res + | ($campaigns_present_in[0]) as $campaigns_present + | ($groups_present_in[0]) as $groups_present + | ($campaign_res.matched) as $mc + | ($group_res.matched) as $mg + # Measurable needs both dimensions: a population to measure and a campaign to + # measure it against. Nothing downstream of an unmatched name may be a number. + | ((($mc | length) > 0) and (($mg | length) > 0)) as $measurable + | (if $measurable | not then "unresolved" + elif (($campaign_res.unmatched | length) == 0 + and ($group_res.unmatched | length) == 0) then "resolved" + else "partial" end) as $status + | ( [ $members[] | select(.status == "active") ] | unique_by(.id) ) as $users + | [ $enrollments[] | select(.campaign_name as $c | $mc | any(. == $c)) ] as $scoped + | ( $users | map( + . as $u + | ( [ $scoped[] | select(.user.id == $u.id) ] ) as $ue + | { + email: $u.email, + status: ( + if ($ue | length) == 0 then "not_started" + elif ($ue | all(.status == "Passed")) then "completed" + elif ($ue | any(.status == "Past Due")) then "past_due" + elif ($ue | any(.status == "In Progress")) then "in_progress" + else "not_started" end) + } ) ) as $rows + | ($rows | map(select(.status == "completed")) | length) as $completed + | ($rows | map(select(.status == "in_progress")) | length) as $in_progress + | ($rows | map(select(.status == "past_due")) | length) as $past_due + | ($rows | map(select(.status == "not_started")) | length) as $not_started + | ($users | length) as $total_users + | { + results: { + config_resolution: ( + { + status: $status, + measurable: $measurable, + groups: $group_res, + campaigns: $campaign_res + } + # Only when something failed to match, so a healthy run stays terse. + + (if ($group_res.unmatched | length) > 0 + then { groups_present_in_tenant: $groups_present } else {} end) + + (if ($campaign_res.unmatched | length) > 0 + then { campaigns_present_in_tenant: $campaigns_present } else {} end) + ), + developer_users: [ $users[] | {id, email, status} ], + developer_groups: $matched_groups, + developer_campaigns: $matched_campaigns, + enrollments: [ $scoped[] | del(.policy_acknowledged) ], + user_training_status: ( + if $measurable then ($rows | map({key: .email, value: .status}) | from_entries) + else {} end), + summary: { + # Discovered counts stay real: 0 groups matched is an accurate 0. + total_developer_users: $total_users, + total_groups: ($mg | length), + total_campaigns: ($mc | length), + # Compliance metrics are null unless they were actually measurable. + completed_training: (if $measurable then $completed else null end), + in_progress: (if $measurable then $in_progress else null end), + past_due: (if $measurable then $past_due else null end), + not_started: (if $measurable then $not_started else null end), + completion_rate: ( + if $measurable | not then null + elif $total_users == 0 then 0 + else (($completed * 100 / $total_users) | floor) end) + } + } + } +' > "$OUTPUT_JSON" -jq --arg total "$total_developer_users" \ - --arg completed "$completed_training" \ - --arg in_progress "$in_progress" \ - --arg past_due "$past_due" \ - --arg not_started "$not_started" \ - --arg rate "$completion_rate" \ - --arg campaigns "$campaigns" \ - --arg groups "$groups" \ - '.results.summary = { - "total_developer_users": ($total|tonumber), - "completed_training": ($completed|tonumber), - "in_progress": ($in_progress|tonumber), - "past_due": ($past_due|tonumber), - "not_started": ($not_started|tonumber), - "completion_rate": ($rate|tonumber), - "total_campaigns": ($campaigns|tonumber), - "total_groups": ($groups|tonumber) - }' "$OUTPUT_JSON" > "$_FETCHER_TMP_JSON" && mv "$_FETCHER_TMP_JSON" "$OUTPUT_JSON" +if [ ! -s "$OUTPUT_JSON" ]; then + log_error "failed to assemble the evidence document" + exit 1 +fi +# A failed API call is still a failure. Only *config* that does not resolve is +# reported as evidence instead of raised as an error. failure_count=$(wc -l < "$_FAILURE_LOG" 2>/dev/null | tr -d ' ') failure_count=${failure_count:-0} if [ "$failure_count" -gt 0 ]; then - log_error "Encountered $failure_count API failures during collection" + log_error "encountered $failure_count API failure(s) during collection" exit 1 fi +status=$(jq -r '.results.config_resolution.status' "$OUTPUT_JSON") +if [ "$status" != "resolved" ]; then + log_warn "config_resolution.status=${status} — unmeasurable metrics are reported as null, not 0" + log_warn "group name(s) not found in tenant: $(jq -r '.results.config_resolution.groups.unmatched | join(", ")' "$OUTPUT_JSON")" + log_warn "campaign name(s) not found in tenant: $(jq -r '.results.config_resolution.campaigns.unmatched | join(", ")' "$OUTPUT_JSON")" + log_warn "groups present in tenant: $(jq -r '.results.config_resolution.groups_present_in_tenant // [] | join(", ")' "$OUTPUT_JSON")" + log_warn "campaigns present in tenant: $(jq -r '.results.config_resolution.campaigns_present_in_tenant // [] | join(", ")' "$OUTPUT_JSON")" +fi + log_info "Evidence saved to $OUTPUT_JSON" +exit 0 diff --git a/fetchers/knowbe4/developer_specific_training/fetcher.yaml b/fetchers/knowbe4/developer_specific_training/fetcher.yaml index ae92bab..8931bca 100644 --- a/fetchers/knowbe4/developer_specific_training/fetcher.yaml +++ b/fetchers/knowbe4/developer_specific_training/fetcher.yaml @@ -1,10 +1,22 @@ name: knowbe4_developer_specific_training version: 0.1.0 -description: Validates completion of developer-specific security training campaigns in KnowBe4. Identifies users in developer-related groups, checks their enrollment and completion status, and reports per-user training status with summary metrics. +description: Validates completion of the developer-specific training campaign(s) named in config for members of the developer groups named in config. Reports per-user training status with summary metrics. Names are matched exactly; when one matches nothing in the tenant, every compliance metric is reported as null (not 0) and results.config_resolution names what did not resolve. category: knowbe4 supports_targets: false +config_schema: + developer_groups: + type: string + required: true + env: KNOWBE4_DEVELOPER_GROUPS + description: Comma-separated KnowBe4 group names whose members are treated as developers (e.g. "Engineering,Developers"). Matched exactly against the group titles in your tenant, not as substrings. Whitespace around each name is trimmed. + developer_campaigns: + type: string + required: true + env: KNOWBE4_DEVELOPER_CAMPAIGNS + description: Comma-separated training campaign names those users are required to complete, matched exactly against the campaign titles in your tenant. + runtime: type: bash entry: fetcher.sh diff --git a/fetchers/knowbe4/high_risk_training/fetcher.sh b/fetchers/knowbe4/high_risk_training/fetcher.sh index bf92054..c9c081e 100755 --- a/fetchers/knowbe4/high_risk_training/fetcher.sh +++ b/fetchers/knowbe4/high_risk_training/fetcher.sh @@ -2,209 +2,296 @@ # # KnowBe4 — High-Risk (Role-Specific) Training Validation # -# Identifies users in high-risk groups (Cloud Ops, IT, DevOps), checks -# completion of role-specific campaigns ("Privileged Users Training (Before -# CloudOps Access)"), reports per-user status with summary metrics. +# Checks completion of the campaign(s) named in KNOWBE4_ROLE_SPECIFIC_CAMPAIGNS for +# the members of the groups named in KNOWBE4_HIGH_RISK_GROUPS. +# +# Group and campaign names come from config, never from this file: hardcoding them +# made the fetcher report a confident 0% on any tenant that named things +# differently. Names are matched EXACTLY — the substring matching this replaced let +# a group named "IT" pull in "AUDIT" and "Legal-IT". +# +# A name that matches nothing in the tenant is NOT a fetcher failure — one typo must +# not turn a whole run red — so it exits 0 and reports every metric it could not +# measure as null, with results.config_resolution naming what did not resolve. +# null means "not measured"; 0 means "measured, and it is zero". # # Output: $EVIDENCE_DIR/knowbe4_high_risk_training.json -# Required env: KNOWBE4_API_KEY, KNOWBE4_REGION +# Required env: KNOWBE4_API_KEY, KNOWBE4_REGION, +# KNOWBE4_HIGH_RISK_GROUPS, KNOWBE4_ROLE_SPECIFIC_CAMPAIGNS set -o pipefail +# Interim v0.x: load .env if present. Runner + secret resolver replaces this. [ -f .env ] && { set -a; . .env; set +a; } +FETCHER=knowbe4_high_risk_training + OUTPUT_DIR="${EVIDENCE_DIR:-./evidence}" mkdir -p "$OUTPUT_DIR" +OUTPUT_JSON="$OUTPUT_DIR/${FETCHER}.json" + +_FAILURE_LOG="$(mktemp -t ${FETCHER}_fail.XXXXXX)" +# Large jq inputs go in by FILE, never as an argv string: Linux caps a single +# argument at MAX_ARG_STRLEN (128KB), so --argjson with a full enrollments or +# members array fails execve with E2BIG and jq never runs — leaving an empty +# evidence file. macOS has no per-argument cap, which is exactly how this hid +# during local dev. +_TMP_ENROLLMENTS="$(mktemp -t ${FETCHER}_enroll.XXXXXX)" +_TMP_MEMBERS="$(mktemp -t ${FETCHER}_members.XXXXXX)" +_TMP_MATCHED_GROUPS="$(mktemp -t ${FETCHER}_mgroups.XXXXXX)" +_TMP_MATCHED_CAMPAIGNS="$(mktemp -t ${FETCHER}_mcamps.XXXXXX)" +_TMP_GROUPS_PRESENT="$(mktemp -t ${FETCHER}_gpresent.XXXXXX)" +_TMP_CAMPAIGNS_PRESENT="$(mktemp -t ${FETCHER}_cpresent.XXXXXX)" +_TMP_GROUP_RES="$(mktemp -t ${FETCHER}_gres.XXXXXX)" +_TMP_CAMPAIGN_RES="$(mktemp -t ${FETCHER}_cres.XXXXXX)" +trap 'rm -f "$_FAILURE_LOG" "$_TMP_ENROLLMENTS" "$_TMP_MEMBERS" \ + "$_TMP_MATCHED_GROUPS" "$_TMP_MATCHED_CAMPAIGNS" "$_TMP_GROUPS_PRESENT" \ + "$_TMP_CAMPAIGNS_PRESENT" "$_TMP_GROUP_RES" "$_TMP_CAMPAIGN_RES"' EXIT + +log_info() { printf '%s INFO %s %s\n' "$(date -u +'%Y-%m-%d %H:%M:%S')" "$FETCHER" "$*" >&2; } +log_warn() { printf '%s WARN %s %s\n' "$(date -u +'%Y-%m-%d %H:%M:%S')" "$FETCHER" "$*" >&2; } +log_error() { printf '%s ERROR %s %s\n' "$(date -u +'%Y-%m-%d %H:%M:%S')" "$FETCHER" "$*" >&2; } if [ -z "${KNOWBE4_API_KEY:-}" ]; then - echo "ERROR knowbe4_high_risk_training: KNOWBE4_API_KEY is not set" >&2 + log_error "KNOWBE4_API_KEY is not set" exit 1 fi if [ -z "${KNOWBE4_REGION:-}" ]; then - echo "ERROR knowbe4_high_risk_training: KNOWBE4_REGION is not set" >&2 + log_error "KNOWBE4_REGION is not set" exit 1 fi -OUTPUT_JSON="$OUTPUT_DIR/knowbe4_high_risk_training.json" -_FETCHER_TMP_JSON="$(mktemp -t knowbe4_high_risk_training.XXXXXX.json)" -_FAILURE_LOG="$(mktemp -t knowbe4_high_risk_training_fail.XXXXXX)" -trap 'rm -f "$_FETCHER_TMP_JSON" "$_FAILURE_LOG"' EXIT - -log_info() { - printf '%s INFO knowbe4_high_risk_training %s\n' "$(date -u +'%Y-%m-%d %H:%M:%S')" "$*" >&2 -} - -log_error() { - printf '%s ERROR knowbe4_high_risk_training %s\n' "$(date -u +'%Y-%m-%d %H:%M:%S')" "$*" >&2 +# Split a comma-separated config value into _SPLIT[], trimming surrounding +# whitespace and dropping empty elements. `read -ra`, not an unquoted expansion, +# so a name containing a glob character is never pathname-expanded. +split_config() { + local raw="$1" item + local -a parts=() + _SPLIT=() + IFS=',' read -ra parts <<< "$raw" + for item in "${parts[@]}"; do + item="${item#"${item%%[![:space:]]*}"}" + item="${item%"${item##*[![:space:]]}"}" + [ -n "$item" ] && _SPLIT+=("$item") + done } -ROLE_SPECIFIC_CAMPAIGNS=("Privileged Users Training (Before CloudOps Access)") - +# printf '%s', never echo: bash's echo leaves backslashes alone but sh's and zsh's +# do not, and KnowBe4 group and campaign titles are free text. make_api_call() { local endpoint=$1 local url="https://${KNOWBE4_REGION}.api.knowbe4.com/v1/${endpoint}" local response - if ! response=$(curl -sf -H "Authorization: Bearer ${KNOWBE4_API_KEY}" -H "Content-Type: application/json" "${url}"); then - echo "GET ${endpoint}" >> "$_FAILURE_LOG" - echo "{}" + if ! response=$(curl -sf -H "Authorization: Bearer ${KNOWBE4_API_KEY}" \ + -H "Content-Type: application/json" "${url}"); then + printf 'GET %s\n' "$endpoint" >> "$_FAILURE_LOG" + printf '%s' '[]' return 1 fi - if ! echo "$response" | jq . >/dev/null 2>&1; then - echo "GET ${endpoint} (invalid JSON)" >> "$_FAILURE_LOG" - echo "{}" + # Anything that is not a JSON array is an error body, not a page. Recording it + # rather than treating it as data is what stops pagination looping forever on a + # 200-with-error-payload. + if ! printf '%s' "$response" | jq -e 'type == "array"' >/dev/null 2>&1; then + printf 'GET %s (response was not a JSON array)\n' "$endpoint" >> "$_FAILURE_LOG" + printf '%s' '[]' return 1 fi - echo "$response" - return 0 + printf '%s' "$response" } +_MAX_PAGES=1000 + make_paginated_api_call() { local endpoint="$1" - local page=1 - local all_results="[]" - local separator + local page=1 all_results="[]" response count separator if [[ "$endpoint" == *\?* ]]; then separator="&"; else separator="?"; fi - while true; do - response=$(make_api_call "${endpoint}${separator}page=${page}") - count=$(echo "$response" | jq 'length' 2>/dev/null || echo 0) - if [ "$count" -eq 0 ]; then break; fi - all_results=$(jq -s '.[0] + .[1]' <(echo "$all_results") <(echo "$response")) + while [ "$page" -le "$_MAX_PAGES" ]; do + if ! response=$(make_api_call "${endpoint}${separator}page=${page}"); then + break + fi + count=$(printf '%s' "$response" | jq 'length') + [ "$count" -eq 0 ] && break + all_results=$(jq -s '.[0] + .[1]' \ + <(printf '%s' "$all_results") <(printf '%s' "$response")) page=$((page + 1)) done - echo "$all_results" + if [ "$page" -gt "$_MAX_PAGES" ]; then + printf 'GET %s (stopped at the %s-page cap)\n' "$endpoint" "$_MAX_PAGES" >> "$_FAILURE_LOG" + fi + printf '%s' "$all_results" +} + +# Requested vs present, computed by jq from data. The names are never spliced into +# a jq program, so quotes, backslashes and $ in a title are just text. +resolve_names() { + local present_json="$1"; shift + jq -n -c --argjson present "$present_json" --args ' + ($ARGS.positional) as $req + | { + requested: $req, + matched: [ $req[] | select(. as $n | $present | any(. == $n)) ], + unmatched: [ $req[] | select(. as $n | $present | any(. == $n) | not) ] + } + ' -- "$@" } -echo '{ - "results": { - "high_risk_users": [], - "role_specific_campaigns": [], - "enrollments": [], - "user_training_status": {}, - "high_risk_groups": [], - "summary": { - "total_high_risk_users": 0, - "completed_training": 0, - "in_progress": 0, - "past_due": 0, - "not_started": 0, - "completion_rate": 0, - "total_campaigns": 0, - "total_groups": 0 - } - } -}' > "$OUTPUT_JSON" - -users_response=$(make_paginated_api_call "users") +split_config "${KNOWBE4_HIGH_RISK_GROUPS:-}" +requested_groups=("${_SPLIT[@]}") +split_config "${KNOWBE4_ROLE_SPECIFIC_CAMPAIGNS:-}" +requested_campaigns=("${_SPLIT[@]}") + +[ "${#requested_groups[@]}" -eq 0 ] && \ + log_warn "KNOWBE4_HIGH_RISK_GROUPS is empty — there is no population to measure" +[ "${#requested_campaigns[@]}" -eq 0 ] && \ + log_warn "KNOWBE4_ROLE_SPECIFIC_CAMPAIGNS is empty — there is no campaign to measure" + campaigns_response=$(make_paginated_api_call "training/campaigns") -enrollments_response=$(make_paginated_api_call "training/enrollments?exclude_archived_users=true&include_campaign_id=true") groups_response=$(make_paginated_api_call "groups") +enrollments_response=$(make_paginated_api_call \ + "training/enrollments?exclude_archived_users=true&include_campaign_id=true") -for campaign_name in "${ROLE_SPECIFIC_CAMPAIGNS[@]}"; do - echo "$campaigns_response" | jq -c --arg name "$campaign_name" \ - '.[] | select(.name == $name)' | while read -r campaign; do - jq --argjson campaign "$campaign" \ - '.results.role_specific_campaigns += [$campaign]' \ - "$OUTPUT_JSON" > "$_FETCHER_TMP_JSON" && mv "$_FETCHER_TMP_JSON" "$OUTPUT_JSON" - done -done - -high_risk_groups=("Cloud Ops" "IT" "DevOps") -high_risk_users=() -while read -r group_json; do - group_name=$(echo "$group_json" | jq -r '.name') - group_id=$(echo "$group_json" | jq -r '.id') - for risk_group in "${high_risk_groups[@]}"; do - if [[ "$group_name" == *"$risk_group"* ]]; then - group_members=$(make_api_call "groups/$group_id/members") - jq --arg name "$group_name" --arg id "$group_id" \ - '.results.high_risk_groups += [{"name": $name, "id": $id}]' \ - "$OUTPUT_JSON" > "$_FETCHER_TMP_JSON" && mv "$_FETCHER_TMP_JSON" "$OUTPUT_JSON" - while read -r user_json; do - user_email=$(echo "$user_json" | jq -r '.email') - if [[ ! " ${high_risk_users[@]} " =~ " ${user_email} " ]]; then - high_risk_users+=("$user_email") - minimal_user=$(echo "$user_json" | jq '{id: .id, email: .email, status: .status}') - jq --argjson user "$minimal_user" \ - '.results.high_risk_users += [$user]' \ - "$OUTPUT_JSON" > "$_FETCHER_TMP_JSON" && mv "$_FETCHER_TMP_JSON" "$OUTPUT_JSON" - fi - done < <(echo "$group_members" | jq -c '.[] | select(.status == "active")') - break - fi - done -done < <(echo "$groups_response" | jq -c '.[]') - -for user_email in "${high_risk_users[@]}"; do - user=$(echo "$users_response" | jq -c --arg email "$user_email" '.[] | select(.email == $email)') - user_id=$(echo "$user" | jq -r '.id') - - campaign_filter=$(printf ' or .campaign_name == "%s"' "${ROLE_SPECIFIC_CAMPAIGNS[@]}") - campaign_filter=${campaign_filter# or } - - user_enrollments=$(echo "$enrollments_response" | jq -c \ - --arg user_id "$user_id" \ - ".[] | select(.user.id == (\$user_id|tonumber) and ( $campaign_filter ))" | jq -s '.') - - user_status="not_started" - if [ "$user_enrollments" != "[]" ]; then - echo "$user_enrollments" | jq -c '.[]' | while read -r enrollment; do - clean_enrollment=$(echo "$enrollment" | jq 'del(.policy_acknowledged)') - jq --argjson enrollment "$clean_enrollment" \ - '.results.enrollments += [$enrollment]' \ - "$OUTPUT_JSON" > "$_FETCHER_TMP_JSON" && mv "$_FETCHER_TMP_JSON" "$OUTPUT_JSON" - done - - if echo "$user_enrollments" | jq -e 'all(.status == "Passed")' >/dev/null 2>&1; then - user_status="completed" - elif echo "$user_enrollments" | jq -e 'any(.status == "Past Due")' >/dev/null 2>&1; then - user_status="past_due" - elif echo "$user_enrollments" | jq -e 'any(.status == "In Progress")' >/dev/null 2>&1; then - user_status="in_progress" - fi - fi +campaigns_present=$(printf '%s' "$campaigns_response" | jq -c '[.[].name] | unique') +groups_present=$(printf '%s' "$groups_response" | jq -c '[.[].name] | unique') +campaign_res=$(resolve_names "$campaigns_present" "${requested_campaigns[@]}") +group_res=$(resolve_names "$groups_present" "${requested_groups[@]}") - jq --arg email "$user_email" --arg status "$user_status" \ - '.results.user_training_status[$email] = $status' \ - "$OUTPUT_JSON" > "$_FETCHER_TMP_JSON" && mv "$_FETCHER_TMP_JSON" "$OUTPUT_JSON" -done - -total_high_risk_users=$(jq '.results.high_risk_users | length' "$OUTPUT_JSON") -completed_training=$(jq '.results.user_training_status | to_entries | map(select(.value == "completed")) | length' "$OUTPUT_JSON") -in_progress=$(jq '.results.user_training_status | to_entries | map(select(.value == "in_progress")) | length' "$OUTPUT_JSON") -past_due=$(jq '.results.user_training_status | to_entries | map(select(.value == "past_due")) | length' "$OUTPUT_JSON") -not_started=$(jq '.results.user_training_status | to_entries | map(select(.value == "not_started")) | length' "$OUTPUT_JSON") -total_campaigns=$(jq '.results.role_specific_campaigns | length' "$OUTPUT_JSON") -total_groups=$(jq '.results.high_risk_groups | length' "$OUTPUT_JSON") -completion_rate=0 -if [ "$total_high_risk_users" -gt 0 ]; then - completion_rate=$((completed_training * 100 / total_high_risk_users)) -fi +matched_campaign_objs=$(jq -c --argjson res "$campaign_res" \ + '[ .[] | select(.name as $n | $res.matched | any(. == $n)) ]' <<< "$campaigns_response") +matched_groups=$(jq -c --argjson res "$group_res" \ + '[ .[] | select(.name as $n | $res.matched | any(. == $n)) | {id: (.id|tostring), name: .name} ]' \ + <<< "$groups_response") + +# Members of the matched groups only — an unmatched name costs no API call. +# `while read` over a process substitution, so the array survives (a pipeline +# would run the loop in a subshell and silently discard it). +members="[]" +while read -r group_id; do + [ -z "$group_id" ] && continue + page=$(make_paginated_api_call "groups/${group_id}/members") + members=$(jq -s '.[0] + .[1]' <(printf '%s' "$members") <(printf '%s' "$page")) +done < <(printf '%s' "$matched_groups" | jq -r '.[].id') + +# One jq pass builds the whole document. The previous version re-ran jq over the +# growing output file once per record, which was quadratic. +printf '%s' "$enrollments_response" > "$_TMP_ENROLLMENTS" +printf '%s' "$members" > "$_TMP_MEMBERS" +printf '%s' "$matched_groups" > "$_TMP_MATCHED_GROUPS" +printf '%s' "$matched_campaign_objs" > "$_TMP_MATCHED_CAMPAIGNS" +printf '%s' "$campaigns_present" > "$_TMP_CAMPAIGNS_PRESENT" +printf '%s' "$groups_present" > "$_TMP_GROUPS_PRESENT" +printf '%s' "$campaign_res" > "$_TMP_CAMPAIGN_RES" +printf '%s' "$group_res" > "$_TMP_GROUP_RES" + +jq -n \ + --slurpfile enrollments_in "$_TMP_ENROLLMENTS" \ + --slurpfile members_in "$_TMP_MEMBERS" \ + --slurpfile matched_groups_in "$_TMP_MATCHED_GROUPS" \ + --slurpfile matched_campaigns_in "$_TMP_MATCHED_CAMPAIGNS" \ + --slurpfile campaign_res_in "$_TMP_CAMPAIGN_RES" \ + --slurpfile group_res_in "$_TMP_GROUP_RES" \ + --slurpfile campaigns_present_in "$_TMP_CAMPAIGNS_PRESENT" \ + --slurpfile groups_present_in "$_TMP_GROUPS_PRESENT" ' + # --slurpfile wraps the contents of each file in an array; [0] unwraps it. + ($enrollments_in[0]) as $enrollments + | ($members_in[0]) as $members + | ($matched_groups_in[0]) as $matched_groups + | ($matched_campaigns_in[0]) as $matched_campaigns + | ($campaign_res_in[0]) as $campaign_res + | ($group_res_in[0]) as $group_res + | ($campaigns_present_in[0]) as $campaigns_present + | ($groups_present_in[0]) as $groups_present + | ($campaign_res.matched) as $mc + | ($group_res.matched) as $mg + # Measurable needs both dimensions: a population to measure and a campaign to + # measure it against. Nothing downstream of an unmatched name may be a number. + | ((($mc | length) > 0) and (($mg | length) > 0)) as $measurable + | (if $measurable | not then "unresolved" + elif (($campaign_res.unmatched | length) == 0 + and ($group_res.unmatched | length) == 0) then "resolved" + else "partial" end) as $status + | ( [ $members[] | select(.status == "active") ] | unique_by(.id) ) as $users + | [ $enrollments[] | select(.campaign_name as $c | $mc | any(. == $c)) ] as $scoped + | ( $users | map( + . as $u + | ( [ $scoped[] | select(.user.id == $u.id) ] ) as $ue + | { + email: $u.email, + status: ( + if ($ue | length) == 0 then "not_started" + elif ($ue | all(.status == "Passed")) then "completed" + elif ($ue | any(.status == "Past Due")) then "past_due" + elif ($ue | any(.status == "In Progress")) then "in_progress" + else "not_started" end) + } ) ) as $rows + | ($rows | map(select(.status == "completed")) | length) as $completed + | ($rows | map(select(.status == "in_progress")) | length) as $in_progress + | ($rows | map(select(.status == "past_due")) | length) as $past_due + | ($rows | map(select(.status == "not_started")) | length) as $not_started + | ($users | length) as $total_users + | { + results: { + config_resolution: ( + { + status: $status, + measurable: $measurable, + groups: $group_res, + campaigns: $campaign_res + } + # Only when something failed to match, so a healthy run stays terse. + + (if ($group_res.unmatched | length) > 0 + then { groups_present_in_tenant: $groups_present } else {} end) + + (if ($campaign_res.unmatched | length) > 0 + then { campaigns_present_in_tenant: $campaigns_present } else {} end) + ), + high_risk_users: [ $users[] | {id, email, status} ], + high_risk_groups: $matched_groups, + role_specific_campaigns: $matched_campaigns, + enrollments: [ $scoped[] | del(.policy_acknowledged) ], + user_training_status: ( + if $measurable then ($rows | map({key: .email, value: .status}) | from_entries) + else {} end), + summary: { + # Discovered counts stay real: 0 groups matched is an accurate 0. + total_high_risk_users: $total_users, + total_groups: ($mg | length), + total_campaigns: ($mc | length), + # Compliance metrics are null unless they were actually measurable. + completed_training: (if $measurable then $completed else null end), + in_progress: (if $measurable then $in_progress else null end), + past_due: (if $measurable then $past_due else null end), + not_started: (if $measurable then $not_started else null end), + completion_rate: ( + if $measurable | not then null + elif $total_users == 0 then 0 + else (($completed * 100 / $total_users) | floor) end) + } + } + } +' > "$OUTPUT_JSON" -jq --arg total "$total_high_risk_users" \ - --arg completed "$completed_training" \ - --arg in_progress "$in_progress" \ - --arg past_due "$past_due" \ - --arg not_started "$not_started" \ - --arg rate "$completion_rate" \ - --arg campaigns "$total_campaigns" \ - --arg groups "$total_groups" \ - '.results.summary = { - "total_high_risk_users": ($total|tonumber), - "completed_training": ($completed|tonumber), - "in_progress": ($in_progress|tonumber), - "past_due": ($past_due|tonumber), - "not_started": ($not_started|tonumber), - "completion_rate": ($rate|tonumber), - "total_campaigns": ($campaigns|tonumber), - "total_groups": ($groups|tonumber) - }' "$OUTPUT_JSON" > "$_FETCHER_TMP_JSON" && mv "$_FETCHER_TMP_JSON" "$OUTPUT_JSON" +if [ ! -s "$OUTPUT_JSON" ]; then + log_error "failed to assemble the evidence document" + exit 1 +fi +# A failed API call is still a failure. Only *config* that does not resolve is +# reported as evidence instead of raised as an error. failure_count=$(wc -l < "$_FAILURE_LOG" 2>/dev/null | tr -d ' ') failure_count=${failure_count:-0} if [ "$failure_count" -gt 0 ]; then - log_error "Encountered $failure_count API failures during collection" + log_error "encountered $failure_count API failure(s) during collection" exit 1 fi +status=$(jq -r '.results.config_resolution.status' "$OUTPUT_JSON") +if [ "$status" != "resolved" ]; then + log_warn "config_resolution.status=${status} — unmeasurable metrics are reported as null, not 0" + log_warn "group name(s) not found in tenant: $(jq -r '.results.config_resolution.groups.unmatched | join(", ")' "$OUTPUT_JSON")" + log_warn "campaign name(s) not found in tenant: $(jq -r '.results.config_resolution.campaigns.unmatched | join(", ")' "$OUTPUT_JSON")" + log_warn "groups present in tenant: $(jq -r '.results.config_resolution.groups_present_in_tenant // [] | join(", ")' "$OUTPUT_JSON")" + log_warn "campaigns present in tenant: $(jq -r '.results.config_resolution.campaigns_present_in_tenant // [] | join(", ")' "$OUTPUT_JSON")" +fi + log_info "Evidence saved to $OUTPUT_JSON" +exit 0 diff --git a/fetchers/knowbe4/high_risk_training/fetcher.yaml b/fetchers/knowbe4/high_risk_training/fetcher.yaml index 85ed344..02046b1 100644 --- a/fetchers/knowbe4/high_risk_training/fetcher.yaml +++ b/fetchers/knowbe4/high_risk_training/fetcher.yaml @@ -1,10 +1,22 @@ name: knowbe4_high_risk_training version: 0.1.0 -description: Validates completion of role-specific training (e.g. "Privileged Users Training") for users in high-risk groups (Cloud Ops, IT, DevOps). Reports per-user status with summary metrics. +description: Validates completion of the role-specific training campaign(s) named in config for members of the high-risk groups named in config. Reports per-user status with summary metrics. Names are matched exactly; when one matches nothing in the tenant, every compliance metric is reported as null (not 0) and results.config_resolution names what did not resolve. category: knowbe4 supports_targets: false +config_schema: + high_risk_groups: + type: string + required: true + env: KNOWBE4_HIGH_RISK_GROUPS + description: Comma-separated KnowBe4 group names whose members are treated as high-risk (e.g. "Cloud Ops,IT,DevOps"). Matched exactly against the group titles in your tenant, not as substrings, so "IT" will not pull in "AUDIT". Whitespace around each name is trimmed. + role_specific_campaigns: + type: string + required: true + env: KNOWBE4_ROLE_SPECIFIC_CAMPAIGNS + description: Comma-separated training campaign names those users are required to complete, matched exactly against the campaign titles in your tenant. + runtime: type: bash entry: fetcher.sh diff --git a/fetchers/knowbe4/module_based_summary/fetcher.sh b/fetchers/knowbe4/module_based_summary/fetcher.sh index 4f55892..7db2e16 100755 --- a/fetchers/knowbe4/module_based_summary/fetcher.sh +++ b/fetchers/knowbe4/module_based_summary/fetcher.sh @@ -3,133 +3,141 @@ # KnowBe4 — Module-Based Training Summary # # Aggregates all active training enrollments by module name with per-module -# assignment and completion metrics. Used for compliance evidence where -# completion does not map cleanly to a single campaign. +# assignment and completion metrics. Used for compliance evidence where completion +# does not map cleanly to a single campaign. +# +# This fetcher takes no group or campaign config: it reports on whatever the tenant +# has, so it works against any tenant as-is. # # Output: $EVIDENCE_DIR/knowbe4_module_based_summary.json # Required env: KNOWBE4_API_KEY, KNOWBE4_REGION set -o pipefail +# Interim v0.x: load .env if present. Runner + secret resolver replaces this. [ -f .env ] && { set -a; . .env; set +a; } +FETCHER=knowbe4_module_based_summary + OUTPUT_DIR="${EVIDENCE_DIR:-./evidence}" mkdir -p "$OUTPUT_DIR" +OUTPUT_JSON="$OUTPUT_DIR/${FETCHER}.json" + +_FAILURE_LOG="$(mktemp -t ${FETCHER}_fail.XXXXXX)" +# Large jq inputs go in by FILE, never as an argv string: Linux caps a single +# argument at MAX_ARG_STRLEN (128KB), so --argjson with a full enrollments array +# fails execve with E2BIG and jq never runs — leaving an empty evidence file. +# macOS has no per-argument cap, which is exactly how this hid during local dev. +_TMP_ENROLLMENTS="$(mktemp -t ${FETCHER}_enroll.XXXXXX)" +trap 'rm -f "$_FAILURE_LOG" "$_TMP_ENROLLMENTS"' EXIT + +log_info() { printf '%s INFO %s %s\n' "$(date -u +'%Y-%m-%d %H:%M:%S')" "$FETCHER" "$*" >&2; } +log_error() { printf '%s ERROR %s %s\n' "$(date -u +'%Y-%m-%d %H:%M:%S')" "$FETCHER" "$*" >&2; } if [ -z "${KNOWBE4_API_KEY:-}" ]; then - echo "ERROR knowbe4_module_based_summary: KNOWBE4_API_KEY is not set" >&2 + log_error "KNOWBE4_API_KEY is not set" exit 1 fi if [ -z "${KNOWBE4_REGION:-}" ]; then - echo "ERROR knowbe4_module_based_summary: KNOWBE4_REGION is not set" >&2 + log_error "KNOWBE4_REGION is not set" exit 1 fi -OUTPUT_JSON="$OUTPUT_DIR/knowbe4_module_based_summary.json" -_FETCHER_TMP_JSON="$(mktemp -t knowbe4_module_based_summary.XXXXXX.json)" -_FAILURE_LOG="$(mktemp -t knowbe4_module_based_summary_fail.XXXXXX)" -trap 'rm -f "$_FETCHER_TMP_JSON" "$_FAILURE_LOG"' EXIT - -log_info() { - printf '%s INFO knowbe4_module_based_summary %s\n' "$(date -u +'%Y-%m-%d %H:%M:%S')" "$*" >&2 -} - -log_error() { - printf '%s ERROR knowbe4_module_based_summary %s\n' "$(date -u +'%Y-%m-%d %H:%M:%S')" "$*" >&2 -} - +# printf '%s', never echo: bash's echo leaves backslashes alone but sh's and zsh's +# do not, and KnowBe4 module titles are free text. make_api_call() { local endpoint=$1 local url="https://${KNOWBE4_REGION}.api.knowbe4.com/v1/${endpoint}" local response - if ! response=$(curl -sf -H "Authorization: Bearer ${KNOWBE4_API_KEY}" -H "Content-Type: application/json" "${url}"); then - echo "GET ${endpoint}" >> "$_FAILURE_LOG" - echo "{}" + if ! response=$(curl -sf -H "Authorization: Bearer ${KNOWBE4_API_KEY}" \ + -H "Content-Type: application/json" "${url}"); then + printf 'GET %s\n' "$endpoint" >> "$_FAILURE_LOG" + printf '%s' '[]' return 1 fi - if ! echo "$response" | jq . >/dev/null 2>&1; then - echo "GET ${endpoint} (invalid JSON)" >> "$_FAILURE_LOG" - echo "{}" + # Anything that is not a JSON array is an error body, not a page. Recording it + # rather than treating it as data is what stops pagination looping forever on a + # 200-with-error-payload. + if ! printf '%s' "$response" | jq -e 'type == "array"' >/dev/null 2>&1; then + printf 'GET %s (response was not a JSON array)\n' "$endpoint" >> "$_FAILURE_LOG" + printf '%s' '[]' return 1 fi - echo "$response" - return 0 + printf '%s' "$response" } +_MAX_PAGES=1000 + make_paginated_api_call() { local endpoint="$1" - local page=1 - local all_results="[]" - local separator + local page=1 all_results="[]" response count separator if [[ "$endpoint" == *\?* ]]; then separator="&"; else separator="?"; fi - while true; do - response=$(make_api_call "${endpoint}${separator}page=${page}") - count=$(echo "$response" | jq 'length' 2>/dev/null || echo 0) - if [ "$count" -eq 0 ]; then break; fi - all_results=$(jq -s '.[0] + .[1]' <(echo "$all_results") <(echo "$response")) + while [ "$page" -le "$_MAX_PAGES" ]; do + if ! response=$(make_api_call "${endpoint}${separator}page=${page}"); then + break + fi + count=$(printf '%s' "$response" | jq 'length') + [ "$count" -eq 0 ] && break + all_results=$(jq -s '.[0] + .[1]' \ + <(printf '%s' "$all_results") <(printf '%s' "$response")) page=$((page + 1)) done - echo "$all_results" + if [ "$page" -gt "$_MAX_PAGES" ]; then + printf 'GET %s (stopped at the %s-page cap)\n' "$endpoint" "$_MAX_PAGES" >> "$_FAILURE_LOG" + fi + printf '%s' "$all_results" } -echo '{ - "results": { - "enrollments": [], - "summary": { - "training_module_summary": {} - } - } -}' > "$OUTPUT_JSON" - -enrollments_response=$(make_paginated_api_call "training/enrollments?exclude_archived_users=true&include_campaign_id=true") - -echo "$enrollments_response" | jq -c '.[] | del(.policy_acknowledged)' | while read -r enrollment; do - jq --argjson e "$enrollment" \ - '.results.enrollments += [$e]' \ - "$OUTPUT_JSON" > "$_FETCHER_TMP_JSON" && mv "$_FETCHER_TMP_JSON" "$OUTPUT_JSON" -done - -module_summary=$(jq ' - .results.enrollments - | sort_by(.module_name) - | group_by(.module_name) - | map( - . as $group - | { - module: $group[0].module_name, - assigned: ($group | length), - passed: ($group | map(select(.status == "Passed")) | length), - completion_rate: - (if ($group | length) > 0 - then ( - (($group | map(select(.status == "Passed")) | length) * 100.0 - / ($group | length) - ) | floor - ) - else 0 - end) +enrollments_response=$(make_paginated_api_call \ + "training/enrollments?exclude_archived_users=true&include_campaign_id=true") + +# One jq pass builds the whole document. The previous version appended each record +# by re-running jq over the growing output file, which was quadratic: 1500 +# enrollments took 69s and 3000 took over 120s, so a mid-size tenant blew the +# runner's 600s cap. +printf '%s' "$enrollments_response" > "$_TMP_ENROLLMENTS" + +jq -n --slurpfile enrollments "$_TMP_ENROLLMENTS" ' + [ $enrollments[0][] | del(.policy_acknowledged) ] as $rows + | { + results: { + enrollments: $rows, + summary: { + training_module_summary: ( + $rows + | group_by(.module_name) + | map({ + key: (.[0].module_name // "(unnamed module)"), + value: { + assigned: length, + passed: (map(select(.status == "Passed")) | length), + completion_rate: ( + if length > 0 + then (((map(select(.status == "Passed")) | length) * 100 / length) | floor) + else 0 end) + } + }) + # from_entries on [] gives {} — an empty tenant is an empty map, not + # the null the previous `add` produced. + | from_entries + ) + } } - ) - | map({ - (.module): { - assigned: .assigned, - passed: .passed, - completion_rate: .completion_rate - } - }) - | add -' "$OUTPUT_JSON") + } +' > "$OUTPUT_JSON" -jq --argjson module_summary "$module_summary" \ - '.results.summary.training_module_summary = $module_summary' \ - "$OUTPUT_JSON" > "$_FETCHER_TMP_JSON" && mv "$_FETCHER_TMP_JSON" "$OUTPUT_JSON" +if [ ! -s "$OUTPUT_JSON" ]; then + log_error "failed to assemble the evidence document" + exit 1 +fi failure_count=$(wc -l < "$_FAILURE_LOG" 2>/dev/null | tr -d ' ') failure_count=${failure_count:-0} if [ "$failure_count" -gt 0 ]; then - log_error "Encountered $failure_count API failures during collection" + log_error "encountered $failure_count API failure(s) during collection" exit 1 fi log_info "Evidence saved to $OUTPUT_JSON" +exit 0 diff --git a/fetchers/knowbe4/security_awareness_training/fetcher.sh b/fetchers/knowbe4/security_awareness_training/fetcher.sh index 0f21b6b..a0e45f9 100755 --- a/fetchers/knowbe4/security_awareness_training/fetcher.sh +++ b/fetchers/knowbe4/security_awareness_training/fetcher.sh @@ -1,190 +1,271 @@ #!/bin/bash # -# KnowBe4 — Annual Security Awareness Training Validation +# KnowBe4 — Security Awareness Training Validation # -# Tracks completion of the annual SAT campaign for all active users. -# Flags users needing retraining (last completion > 1 year ago). +# Tracks completion of the campaign(s) named in KNOWBE4_SECURITY_AWARENESS_CAMPAIGNS +# for all active users, and flags users whose last completion is older than the +# retraining interval. +# +# Campaign names come from config, never from this file: hardcoding them made the +# fetcher report a confident 0% on any tenant that named its campaigns differently. +# A name that matches nothing in the tenant is NOT a fetcher failure — one typo must +# not turn a whole run red — so it exits 0 and reports every metric it could not +# measure as null, with results.config_resolution naming what did not resolve. +# null means "not measured"; 0 means "measured, and it is zero". # # Output: $EVIDENCE_DIR/knowbe4_security_awareness_training.json -# Required env: KNOWBE4_API_KEY, KNOWBE4_REGION +# Required env: KNOWBE4_API_KEY, KNOWBE4_REGION, KNOWBE4_SECURITY_AWARENESS_CAMPAIGNS +# Optional env: KNOWBE4_RETRAINING_INTERVAL_DAYS (default 365) set -o pipefail +# Interim v0.x: load .env if present. Runner + secret resolver replaces this. [ -f .env ] && { set -a; . .env; set +a; } +FETCHER=knowbe4_security_awareness_training + OUTPUT_DIR="${EVIDENCE_DIR:-./evidence}" mkdir -p "$OUTPUT_DIR" +OUTPUT_JSON="$OUTPUT_DIR/${FETCHER}.json" + +_FAILURE_LOG="$(mktemp -t ${FETCHER}_fail.XXXXXX)" +# Large jq inputs go in by FILE, never as an argv string: Linux caps a single +# argument at MAX_ARG_STRLEN (128KB), so --argjson with a full users or +# enrollments array fails execve with E2BIG and jq never runs — leaving an empty +# evidence file. macOS has no per-argument cap, which is exactly how this hid +# during local dev. +_TMP_USERS="$(mktemp -t ${FETCHER}_users.XXXXXX)" +_TMP_ENROLLMENTS="$(mktemp -t ${FETCHER}_enroll.XXXXXX)" +_TMP_CAMPAIGNS_PRESENT="$(mktemp -t ${FETCHER}_cpresent.XXXXXX)" +_TMP_CAMPAIGN_RES="$(mktemp -t ${FETCHER}_cres.XXXXXX)" +trap 'rm -f "$_FAILURE_LOG" "$_TMP_USERS" "$_TMP_ENROLLMENTS" \ + "$_TMP_CAMPAIGNS_PRESENT" "$_TMP_CAMPAIGN_RES"' EXIT + +log_info() { printf '%s INFO %s %s\n' "$(date -u +'%Y-%m-%d %H:%M:%S')" "$FETCHER" "$*" >&2; } +log_warn() { printf '%s WARN %s %s\n' "$(date -u +'%Y-%m-%d %H:%M:%S')" "$FETCHER" "$*" >&2; } +log_error() { printf '%s ERROR %s %s\n' "$(date -u +'%Y-%m-%d %H:%M:%S')" "$FETCHER" "$*" >&2; } if [ -z "${KNOWBE4_API_KEY:-}" ]; then - echo "ERROR knowbe4_security_awareness_training: KNOWBE4_API_KEY is not set" >&2 + log_error "KNOWBE4_API_KEY is not set" exit 1 fi if [ -z "${KNOWBE4_REGION:-}" ]; then - echo "ERROR knowbe4_security_awareness_training: KNOWBE4_REGION is not set" >&2 + log_error "KNOWBE4_REGION is not set" exit 1 fi -OUTPUT_JSON="$OUTPUT_DIR/knowbe4_security_awareness_training.json" -_FETCHER_TMP_JSON="$(mktemp -t knowbe4_security_awareness_training.XXXXXX.json)" -_FAILURE_LOG="$(mktemp -t knowbe4_security_awareness_training_fail.XXXXXX)" -trap 'rm -f "$_FETCHER_TMP_JSON" "$_FAILURE_LOG"' EXIT - -log_info() { - printf '%s INFO knowbe4_security_awareness_training %s\n' "$(date -u +'%Y-%m-%d %H:%M:%S')" "$*" >&2 -} +RETRAINING_INTERVAL_DAYS="${KNOWBE4_RETRAINING_INTERVAL_DAYS:-365}" +case "$RETRAINING_INTERVAL_DAYS" in + ''|*[!0-9]*) + log_error "KNOWBE4_RETRAINING_INTERVAL_DAYS must be a whole number of days, got '$RETRAINING_INTERVAL_DAYS'" + exit 1 ;; +esac +RETRAIN_CUTOFF_EPOCH=$(( $(date -u +%s) - RETRAINING_INTERVAL_DAYS * 86400 )) -log_error() { - printf '%s ERROR knowbe4_security_awareness_training %s\n' "$(date -u +'%Y-%m-%d %H:%M:%S')" "$*" >&2 +# Split a comma-separated config value into _SPLIT[], trimming surrounding +# whitespace and dropping empty elements. `read -ra`, not an unquoted expansion, +# so a name containing a glob character is never pathname-expanded. +split_config() { + local raw="$1" item + local -a parts=() + _SPLIT=() + IFS=',' read -ra parts <<< "$raw" + for item in "${parts[@]}"; do + item="${item#"${item%%[![:space:]]*}"}" + item="${item%"${item##*[![:space:]]}"}" + [ -n "$item" ] && _SPLIT+=("$item") + done } -SECURITY_AWARENESS_CAMPAIGN="2026 Annual Security Awareness Training" -# Portable 1-year-ago timestamp (handle macOS vs GNU date) -ONE_YEAR_AGO=$(date -u -v-1y +%s 2>/dev/null || date -u -d "1 year ago" +%s) - +# printf '%s', never echo: bash's echo leaves backslashes alone but sh's and zsh's +# do not, and KnowBe4 group and campaign titles are free text. make_api_call() { local endpoint=$1 local url="https://${KNOWBE4_REGION}.api.knowbe4.com/v1/${endpoint}" local response - if ! response=$(curl -sf -H "Authorization: Bearer ${KNOWBE4_API_KEY}" -H "Content-Type: application/json" "${url}"); then - echo "GET ${endpoint}" >> "$_FAILURE_LOG" - echo "{}" + if ! response=$(curl -sf -H "Authorization: Bearer ${KNOWBE4_API_KEY}" \ + -H "Content-Type: application/json" "${url}"); then + printf 'GET %s\n' "$endpoint" >> "$_FAILURE_LOG" + printf '%s' '[]' return 1 fi - if ! echo "$response" | jq . >/dev/null 2>&1; then - echo "GET ${endpoint} (invalid JSON)" >> "$_FAILURE_LOG" - echo "{}" + # Anything that is not a JSON array is an error body, not a page. Recording it + # rather than treating it as data is what stops pagination looping forever on a + # 200-with-error-payload. + if ! printf '%s' "$response" | jq -e 'type == "array"' >/dev/null 2>&1; then + printf 'GET %s (response was not a JSON array)\n' "$endpoint" >> "$_FAILURE_LOG" + printf '%s' '[]' return 1 fi - echo "$response" - return 0 + printf '%s' "$response" } +_MAX_PAGES=1000 + make_paginated_api_call() { local endpoint="$1" - local page=1 - local all_results="[]" - local separator + local page=1 all_results="[]" response count separator if [[ "$endpoint" == *\?* ]]; then separator="&"; else separator="?"; fi - while true; do - response=$(make_api_call "${endpoint}${separator}page=${page}") - count=$(echo "$response" | jq 'length' 2>/dev/null || echo 0) - if [ "$count" -eq 0 ]; then break; fi - all_results=$(jq -s '.[0] + .[1]' <(echo "$all_results") <(echo "$response")) + while [ "$page" -le "$_MAX_PAGES" ]; do + if ! response=$(make_api_call "${endpoint}${separator}page=${page}"); then + break + fi + count=$(printf '%s' "$response" | jq 'length') + [ "$count" -eq 0 ] && break + all_results=$(jq -s '.[0] + .[1]' \ + <(printf '%s' "$all_results") <(printf '%s' "$response")) page=$((page + 1)) done - echo "$all_results" + if [ "$page" -gt "$_MAX_PAGES" ]; then + printf 'GET %s (stopped at the %s-page cap)\n' "$endpoint" "$_MAX_PAGES" >> "$_FAILURE_LOG" + fi + printf '%s' "$all_results" } -echo '{ - "results": { - "users": [], - "enrollments": [], - "user_training_status": {}, - "user_retraining_required": {}, - "summary": { - "total_users": 0, - "completed_training": 0, - "in_progress": 0, - "past_due": 0, - "not_started": 0, - "needs_retraining": 0, - "completion_rate": 0 - } - } -}' > "$OUTPUT_JSON" +# Requested vs present, computed by jq from data. The names are never spliced into +# a jq program, so quotes, backslashes and $ in a title are just text. +resolve_names() { + local present_json="$1"; shift + jq -n -c --argjson present "$present_json" --args ' + ($ARGS.positional) as $req + | { + requested: $req, + matched: [ $req[] | select(. as $n | $present | any(. == $n)) ], + unmatched: [ $req[] | select(. as $n | $present | any(. == $n) | not) ] + } + ' -- "$@" +} + +split_config "${KNOWBE4_SECURITY_AWARENESS_CAMPAIGNS:-}" +requested_campaigns=("${_SPLIT[@]}") +if [ "${#requested_campaigns[@]}" -eq 0 ]; then + log_warn "KNOWBE4_SECURITY_AWARENESS_CAMPAIGNS is empty — there is no campaign to measure" +fi users_response=$(make_paginated_api_call "users") -enrollments_response=$(make_paginated_api_call "training/enrollments?exclude_archived_users=true&include_campaign_id=true") - -security_awareness_enrollments=$(echo "$enrollments_response" | jq -c \ - --arg campaign "$SECURITY_AWARENESS_CAMPAIGN" \ - '[.[] | select(.campaign_name == $campaign)]') - -echo "$users_response" | jq -c '.[] | select(.status == "active")' | while read -r user; do - user_id=$(echo "$user" | jq -r '.id') - user_email=$(echo "$user" | jq -r '.email') - - minimal_user=$(echo "$user" | jq '{id: .id, email: .email, status: .status}') - jq --argjson user "$minimal_user" \ - '.results.users += [$user]' \ - "$OUTPUT_JSON" > "$_FETCHER_TMP_JSON" && mv "$_FETCHER_TMP_JSON" "$OUTPUT_JSON" - - user_enrollments=$(echo "$security_awareness_enrollments" | jq -c \ - --arg user_id "$user_id" \ - '[.[] | select(.user.id == ($user_id|tonumber))]') - - user_status="not_started" - needs_retraining=false - - if echo "$user_enrollments" | jq -e 'type=="array" and length > 0' >/dev/null; then - while read -r enrollment; do - jq --argjson enrollment "$enrollment" \ - '.results.enrollments += [$enrollment]' \ - "$OUTPUT_JSON" > "$_FETCHER_TMP_JSON" && mv "$_FETCHER_TMP_JSON" "$OUTPUT_JSON" - done < <(echo "$user_enrollments" | jq -c '.[]') - - total_modules=$(echo "$user_enrollments" | jq 'length') - passed_modules=$(echo "$user_enrollments" | jq '[.[] | select(.status == "Passed")] | length') - - if [ "$passed_modules" -eq "$total_modules" ] && [ "$total_modules" -gt 0 ]; then - user_status="completed" - latest_passed_date=$(echo "$user_enrollments" | jq -r '[.[] | .completion_date] | max') - # Portable date parse: try GNU date first, fall back to BSD/macOS - completed_epoch=$(date -u -d "$latest_passed_date" +%s 2>/dev/null \ - || date -j -u -f "%Y-%m-%dT%H:%M:%S" "${latest_passed_date%.*}" +%s 2>/dev/null \ - || echo "") - if [ -n "$completed_epoch" ] && [ "$completed_epoch" -lt "$ONE_YEAR_AGO" ]; then - needs_retraining=true - fi - elif echo "$user_enrollments" | jq -e 'any(.status == "Past Due")' >/dev/null; then - user_status="past_due" - elif echo "$user_enrollments" | jq -e 'any(.status == "In Progress" or .status == "Passed")' >/dev/null; then - user_status="in_progress" - fi - fi +campaigns_response=$(make_paginated_api_call "training/campaigns") +enrollments_response=$(make_paginated_api_call \ + "training/enrollments?exclude_archived_users=true&include_campaign_id=true") - jq --arg email "$user_email" \ - --arg status "$user_status" \ - --argjson retrain "$needs_retraining" \ - '.results.user_training_status[$email] = $status - | .results.user_retraining_required[$email] = $retrain' \ - "$OUTPUT_JSON" > "$_FETCHER_TMP_JSON" && mv "$_FETCHER_TMP_JSON" "$OUTPUT_JSON" -done - -total_users=$(jq '.results.users | length' "$OUTPUT_JSON") -completed_training=$(jq '.results.user_training_status | to_entries | map(select(.value == "completed")) | length' "$OUTPUT_JSON") -in_progress=$(jq '.results.user_training_status | to_entries | map(select(.value == "in_progress")) | length' "$OUTPUT_JSON") -past_due=$(jq '.results.user_training_status | to_entries | map(select(.value == "past_due")) | length' "$OUTPUT_JSON") -not_started=$(jq '.results.user_training_status | to_entries | map(select(.value == "not_started")) | length' "$OUTPUT_JSON") -needs_retraining=$(jq '.results.user_retraining_required | to_entries | map(select(.value == true)) | length' "$OUTPUT_JSON") -completion_rate=0 -if [ "$total_users" -gt 0 ]; then - completion_rate=$((completed_training * 100 / total_users)) -fi +campaigns_present=$(printf '%s' "$campaigns_response" | jq -c '[.[].name] | unique') +campaign_res=$(resolve_names "$campaigns_present" "${requested_campaigns[@]}") + +# One jq pass builds the whole document. The previous version re-ran jq over the +# growing output file once per record, which was quadratic — 3k enrollments blew +# the runner's 600s cap. +printf '%s' "$users_response" > "$_TMP_USERS" +printf '%s' "$enrollments_response" > "$_TMP_ENROLLMENTS" +printf '%s' "$campaigns_present" > "$_TMP_CAMPAIGNS_PRESENT" +printf '%s' "$campaign_res" > "$_TMP_CAMPAIGN_RES" -jq --arg total "$total_users" \ - --arg completed "$completed_training" \ - --arg in_progress "$in_progress" \ - --arg past_due "$past_due" \ - --arg not_started "$not_started" \ - --arg needs_retraining "$needs_retraining" \ - --arg rate "$completion_rate" \ - '.results.summary = { - "total_users": ($total|tonumber), - "completed_training": ($completed|tonumber), - "in_progress": ($in_progress|tonumber), - "past_due": ($past_due|tonumber), - "not_started": ($not_started|tonumber), - "needs_retraining": ($needs_retraining|tonumber), - "completion_rate": ($rate|tonumber) - }' "$OUTPUT_JSON" > "$_FETCHER_TMP_JSON" && mv "$_FETCHER_TMP_JSON" "$OUTPUT_JSON" +jq -n \ + --slurpfile users_in "$_TMP_USERS" \ + --slurpfile enrollments_in "$_TMP_ENROLLMENTS" \ + --slurpfile campaign_res_in "$_TMP_CAMPAIGN_RES" \ + --slurpfile campaigns_present_in "$_TMP_CAMPAIGNS_PRESENT" \ + --argjson cutoff "$RETRAIN_CUTOFF_EPOCH" \ + --argjson interval_days "$RETRAINING_INTERVAL_DAYS" ' + # --slurpfile wraps the contents of each file in an array; [0] unwraps it. + ($users_in[0]) as $users + | ($enrollments_in[0]) as $enrollments + | ($campaign_res_in[0]) as $campaign_res + | ($campaigns_present_in[0]) as $campaigns_present + | ($campaign_res.matched) as $matched + # Measurable only if at least one requested campaign exists in the tenant. + # Nothing downstream of an unmatched name may be reported as a number. + | (($matched | length) > 0) as $measurable + | (if $measurable | not then "unresolved" + elif ($campaign_res.unmatched | length) == 0 then "resolved" + else "partial" end) as $status + | [ $users[] | select(.status == "active") ] as $active + | [ $enrollments[] | select(.campaign_name as $c | $matched | any(. == $c)) ] as $scoped + | ( $active | map( + . as $u + | ( [ $scoped[] | select(.user.id == $u.id) ] ) as $ue + | ( [ $ue[] | .completion_date | select(. != null) + | (try (sub("\\.[0-9]+";"") | fromdateiso8601) catch null) + | select(. != null) ] | max ) as $latest + | { + email: $u.email, + status: ( + if ($ue | length) == 0 then "not_started" + elif ($ue | all(.status == "Passed")) then "completed" + elif ($ue | any(.status == "Past Due")) then "past_due" + elif ($ue | any(.status == "In Progress" or .status == "Passed")) + then "in_progress" + else "not_started" end), + retrain: ( + if ($ue | length) > 0 and ($ue | all(.status == "Passed")) and $latest != null + then ($latest < $cutoff) else false end) + } ) ) as $rows + | ($rows | map(select(.status == "completed")) | length) as $completed + | ($rows | map(select(.status == "in_progress")) | length) as $in_progress + | ($rows | map(select(.status == "past_due")) | length) as $past_due + | ($rows | map(select(.status == "not_started")) | length) as $not_started + | ($rows | map(select(.retrain)) | length) as $needs_retraining + | ($active | length) as $total_users + | { + results: { + config_resolution: ( + { + status: $status, + measurable: $measurable, + campaigns: $campaign_res, + retraining_interval_days: $interval_days + } + # Only when something failed to match, so a healthy run stays terse. + + (if ($campaign_res.unmatched | length) > 0 + then { campaigns_present_in_tenant: $campaigns_present } + else {} end) + ), + users: [ $active[] | {id, email, status} ], + enrollments: [ $scoped[] | del(.policy_acknowledged) ], + user_training_status: ( + if $measurable then ($rows | map({key: .email, value: .status}) | from_entries) + else {} end), + user_retraining_required: ( + if $measurable then ($rows | map({key: .email, value: .retrain}) | from_entries) + else {} end), + summary: { + # Discovered counts stay real: 0 users found is an accurate 0. + total_users: $total_users, + total_campaigns: ($matched | length), + # Compliance metrics are null unless they were actually measurable. + completed_training: (if $measurable then $completed else null end), + in_progress: (if $measurable then $in_progress else null end), + past_due: (if $measurable then $past_due else null end), + not_started: (if $measurable then $not_started else null end), + needs_retraining: (if $measurable then $needs_retraining else null end), + completion_rate: ( + if $measurable | not then null + elif $total_users == 0 then 0 + else (($completed * 100 / $total_users) | floor) end) + } + } + } +' > "$OUTPUT_JSON" +if [ ! -s "$OUTPUT_JSON" ]; then + log_error "failed to assemble the evidence document" + exit 1 +fi + +# A failed API call is still a failure. Only *config* that does not resolve is +# reported as evidence instead of raised as an error. failure_count=$(wc -l < "$_FAILURE_LOG" 2>/dev/null | tr -d ' ') failure_count=${failure_count:-0} if [ "$failure_count" -gt 0 ]; then - log_error "Encountered $failure_count API failures during collection" + log_error "encountered $failure_count API failure(s) during collection" exit 1 fi +status=$(jq -r '.results.config_resolution.status' "$OUTPUT_JSON") +if [ "$status" != "resolved" ]; then + log_warn "config_resolution.status=${status} — unmeasurable metrics are reported as null, not 0" + log_warn "campaign name(s) not found in tenant: $(jq -r '.results.config_resolution.campaigns.unmatched | join(", ")' "$OUTPUT_JSON")" + log_warn "campaign names present in tenant: $(jq -r '.results.config_resolution.campaigns_present_in_tenant // [] | join(", ")' "$OUTPUT_JSON")" +fi + log_info "Evidence saved to $OUTPUT_JSON" +exit 0 diff --git a/fetchers/knowbe4/security_awareness_training/fetcher.yaml b/fetchers/knowbe4/security_awareness_training/fetcher.yaml index b1a4ed6..4e293ae 100644 --- a/fetchers/knowbe4/security_awareness_training/fetcher.yaml +++ b/fetchers/knowbe4/security_awareness_training/fetcher.yaml @@ -1,10 +1,23 @@ name: knowbe4_security_awareness_training version: 0.1.0 -description: Validates completion of the annual Security Awareness Training campaign for all active users. Tracks per-user status (completed / in_progress / past_due / not_started) and flags users needing retraining (last completion > 1 year ago). +description: Validates completion of the Security Awareness Training campaign(s) named in config, for all active users. Tracks per-user status (completed / in_progress / past_due / not_started) and flags users whose last completion is older than the retraining interval. When a configured campaign name matches nothing in the tenant, every compliance metric is reported as null (not 0) and results.config_resolution names what did not resolve. category: knowbe4 supports_targets: false +config_schema: + security_awareness_campaigns: + type: string + required: true + env: KNOWBE4_SECURITY_AWARENESS_CAMPAIGNS + description: Comma-separated KnowBe4 training campaign names to measure, matched exactly against the campaign titles in your tenant (e.g. "2026 Annual Security Awareness Training"). Whitespace around each name is trimmed. + retraining_interval_days: + type: integer + required: false + default: 365 + env: KNOWBE4_RETRAINING_INTERVAL_DAYS + description: A user whose most recent completion is older than this many days is flagged as needing retraining. + runtime: type: bash entry: fetcher.sh diff --git a/tests/test_knowbe4_config_resolution.py b/tests/test_knowbe4_config_resolution.py new file mode 100644 index 0000000..e5e8836 --- /dev/null +++ b/tests/test_knowbe4_config_resolution.py @@ -0,0 +1,498 @@ +"""KnowBe4 fetchers must never report an unresolved config as a failing control. + +The bug this suite pins: the group and campaign names these fetchers match on +were hardcoded to one customer's tenant. Pointed at any other tenant they +emitted ``completion_rate: 0`` and exited 0 — byte-identical to a tenant where +the campaign resolved and genuinely nobody had trained. Two states, one output: + + typo'd config, users DID pass -> {"completed_training": 0, "completion_rate": 0} + config OK, nobody passed -> {"completed_training": 0, "completion_rate": 0} + +No assertion could separate them, so the contract now does. A compliance metric +the fetcher could not measure is ``null``; ``0`` means measured and zero. That +one distinction is what makes the failure expressible, and ``test_b_*`` vs +``test_c_*`` below are the pair that would have caught it. + +Deliberately NOT a failure: an unresolvable name exits 0 and records itself in +``results.config_resolution``. A typo in one group name must not turn a whole +nightly run red (``paramify run`` exits 1 if any fetcher fails), and evidence +that says "I could not measure this" is more useful than an absent artifact. +A config key that was never wired at all is a different case — ``required: true`` +means ``paramify validate`` catches it pre-flight. + +The fetchers are bash scripts the runner exec's, so these tests drive them the +way the runner does: a stub ``curl`` on PATH serving canned KnowBe4 pages, and +the config passed as the env vars each fetcher.yaml declares. No network, no +tenant, no jq/curl mocking inside the script itself. + +Run: ``pytest tests/test_knowbe4_config_resolution.py`` +""" + +from __future__ import annotations + +import json +import os +import shutil +import subprocess +import time +from pathlib import Path + +import pytest + +REPO_ROOT = Path(__file__).resolve().parent.parent +KNOWBE4_ROOT = REPO_ROOT / "fetchers" / "knowbe4" + +# Stub curl: resolves https://.api.knowbe4.com/v1/?...page=N to +# $MOCK_DATA/_p.json, and serves [] for anything absent +# so pagination terminates the way the real API's empty last page does. Honors +# $MOCK_FAIL to emulate curl -f's nonzero exit on a 4xx/5xx. +_STUB_CURL = r"""#!/bin/bash +[ -n "$MOCK_FAIL" ] && exit 22 +url="" +for a in "$@"; do case "$a" in https://*) url="$a";; esac; done +page=1 +[[ "$url" =~ page=([0-9]+) ]] && page="${BASH_REMATCH[1]}" +ep="${url#*/v1/}"; ep="${ep%%\?*}" +f="$MOCK_DATA/$(echo "$ep" | tr '/' '_')_p${page}.json" +[ -f "$f" ] && { cat "$f"; exit 0; } +printf '%s' '[]' +""" + +# ---------------------------------------------------------------------------- # +# Tenant fixtures. Each is the full set of API pages a run will see. +# ---------------------------------------------------------------------------- # + +_USERS = [ + {"id": 1, "email": "a@ex.com", "status": "active"}, + {"id": 2, "email": "b@ex.com", "status": "active"}, + {"id": 3, "email": "c@ex.com", "status": "archived"}, +] + +# Tenant that matches the config the tests pass: campaigns and groups line up. +TENANT_MATCHING = { + "users": _USERS, + "groups": [ + {"id": 10, "name": "Engineering"}, + {"id": 11, "name": "Cloud Ops"}, + {"id": 12, "name": "Recruiting"}, + ], + "groups_10_members": [{"id": 1, "email": "a@ex.com", "status": "active"}], + "groups_11_members": [{"id": 2, "email": "b@ex.com", "status": "active"}], + "training_campaigns": [ + {"campaign_id": 100, "name": "2026 Annual Security Awareness Training"}, + {"campaign_id": 101, "name": "Developers Training"}, + {"campaign_id": 102, "name": "Privileged Users Training (Before CloudOps Access)"}, + ], + "training_enrollments": [ + {"enrollment_id": 1, "campaign_name": "2026 Annual Security Awareness Training", + "module_name": "SAT", "user": {"id": 1, "email": "a@ex.com"}, + "status": "Passed", "completion_date": "2026-03-01T10:00:00.000Z", + "policy_acknowledged": False}, + {"enrollment_id": 2, "campaign_name": "2026 Annual Security Awareness Training", + "module_name": "SAT", "user": {"id": 2, "email": "b@ex.com"}, + "status": "In Progress", "completion_date": None, + "policy_acknowledged": False}, + {"enrollment_id": 3, "campaign_name": "Developers Training", + "module_name": "Secure Coding", "user": {"id": 1, "email": "a@ex.com"}, + "status": "Passed", "completion_date": "2026-02-01T10:00:00.000Z", + "policy_acknowledged": False}, + {"enrollment_id": 4, "campaign_name": "Privileged Users Training (Before CloudOps Access)", + "module_name": "Privileged Access", "user": {"id": 2, "email": "b@ex.com"}, + "status": "Past Due", "completion_date": None, + "policy_acknowledged": False}, + ], +} + +# Same tenant shape, but every user PASSED the security-awareness campaign. Used +# with a typo'd config: the truth is 100% trained, so a 0% report is provably wrong. +TENANT_ALL_PASSED = { + **TENANT_MATCHING, + "training_enrollments": [ + {"enrollment_id": 1, "campaign_name": "2026 Annual Security Awareness Training", + "module_name": "SAT", "user": {"id": 1, "email": "a@ex.com"}, + "status": "Passed", "completion_date": "2026-03-01T10:00:00.000Z"}, + {"enrollment_id": 2, "campaign_name": "2026 Annual Security Awareness Training", + "module_name": "SAT", "user": {"id": 2, "email": "b@ex.com"}, + "status": "Passed", "completion_date": "2026-03-02T10:00:00.000Z"}, + ], +} + +# Config resolves, and nobody has passed. A genuine failing control. +TENANT_NOBODY_PASSED = { + **TENANT_MATCHING, + "training_enrollments": [ + {"enrollment_id": 1, "campaign_name": "2026 Annual Security Awareness Training", + "module_name": "SAT", "user": {"id": 1, "email": "a@ex.com"}, + "status": "Not Started", "completion_date": None}, + {"enrollment_id": 2, "campaign_name": "2026 Annual Security Awareness Training", + "module_name": "SAT", "user": {"id": 2, "email": "b@ex.com"}, + "status": "Not Started", "completion_date": None}, + ], +} + +# Names that break a jq filter built by string interpolation. The fetchers must +# pass config to jq as data (--args/$ARGS.positional), never as program text. +HOSTILE_NAMES = [ + 'Privileged "Admin" Training', + "Dev\\Ops Training", + "Cost $ Training", + "Développeur's Training", + "Training (Before CloudOps Access)", +] + +TENANT_HOSTILE = { + "users": _USERS, + "groups": [{"id": 10, "name": n} for n in HOSTILE_NAMES], + "training_campaigns": [{"campaign_id": 200 + i, "name": n} for i, n in enumerate(HOSTILE_NAMES)], + "training_enrollments": [ + {"enrollment_id": i, "campaign_name": n, "module_name": "M", + "user": {"id": 1, "email": "a@ex.com"}, "status": "Passed", + "completion_date": "2026-03-01T10:00:00.000Z"} + for i, n in enumerate(HOSTILE_NAMES) + ], + "groups_10_members": [{"id": 1, "email": "a@ex.com", "status": "active"}], +} + +# Config each fetcher needs to resolve cleanly against TENANT_MATCHING. +GOOD_CONFIG = { + "knowbe4_security_awareness_training": { + "KNOWBE4_SECURITY_AWARENESS_CAMPAIGNS": "2026 Annual Security Awareness Training", + }, + "knowbe4_developer_specific_training": { + "KNOWBE4_DEVELOPER_GROUPS": "Engineering", + "KNOWBE4_DEVELOPER_CAMPAIGNS": "Developers Training", + }, + "knowbe4_high_risk_training": { + "KNOWBE4_HIGH_RISK_GROUPS": "Cloud Ops", + "KNOWBE4_ROLE_SPECIFIC_CAMPAIGNS": "Privileged Users Training (Before CloudOps Access)", + }, +} + +# Same keys, every value typo'd. Nothing will match. +TYPO_CONFIG = { + "knowbe4_security_awareness_training": { + "KNOWBE4_SECURITY_AWARENESS_CAMPAIGNS": "2026 Anual Security Awareness Training", + }, + "knowbe4_developer_specific_training": { + "KNOWBE4_DEVELOPER_GROUPS": "Enginering", + "KNOWBE4_DEVELOPER_CAMPAIGNS": "Developer Training", + }, + "knowbe4_high_risk_training": { + "KNOWBE4_HIGH_RISK_GROUPS": "Cloud Opps", + "KNOWBE4_ROLE_SPECIFIC_CAMPAIGNS": "Priviledged Users Training", + }, +} + +CONFIGURABLE = sorted(GOOD_CONFIG) + +# Every compliance metric that must go null when the fetcher could not measure. +# Counts of what was *discovered* (matched groups/campaigns, users found) stay +# real numbers — 0 discovered is an accurate 0, not an unmeasured value. +COMPLIANCE_METRICS = ( + "completed_training", "in_progress", "past_due", "not_started", + "completion_rate", "needs_retraining", +) + + +# ---------------------------------------------------------------------------- # +# Harness +# ---------------------------------------------------------------------------- # + +def _write_tenant(data_dir: Path, tenant: dict) -> None: + data_dir.mkdir(parents=True, exist_ok=True) + for endpoint, rows in tenant.items(): + (data_dir / f"{endpoint}_p1.json").write_text(json.dumps(rows)) + + +def run_fetcher(fetcher: str, tenant: dict, config: dict, tmp_path: Path, + fail_api: bool = False, timeout: int = 180): + """Run one KnowBe4 fetcher against a canned tenant. Returns (exit, evidence, stderr). + + `evidence` is the parsed payload the fetcher wrote, or None if it wrote nothing. + """ + short = fetcher.removeprefix("knowbe4_") + fetcher_dir = KNOWBE4_ROOT / short + assert fetcher_dir.is_dir(), f"no such fetcher dir: {fetcher_dir}" + + bin_dir = tmp_path / "bin" + bin_dir.mkdir(parents=True, exist_ok=True) + curl = bin_dir / "curl" + curl.write_text(_STUB_CURL) + curl.chmod(0o755) + + data_dir = tmp_path / "data" + _write_tenant(data_dir, tenant) + evidence_dir = tmp_path / "evidence" + + env = { + "PATH": f"{bin_dir}:{os.environ['PATH']}", + "HOME": str(tmp_path), + "MOCK_DATA": str(data_dir), + "EVIDENCE_DIR": str(evidence_dir), + "KNOWBE4_API_KEY": "test-key", + "KNOWBE4_REGION": "us", + **config, + } + if fail_api: + env["MOCK_FAIL"] = "1" + + proc = subprocess.run( + ["bash", "fetcher.sh"], + cwd=fetcher_dir, env=env, capture_output=True, text=True, timeout=timeout, + ) + out = evidence_dir / f"{fetcher}.json" + evidence = json.loads(out.read_text()) if out.exists() else None + return proc.returncode, evidence, proc.stderr + + +@pytest.fixture(autouse=True) +def _require_tools(): + for tool in ("bash", "jq"): + if shutil.which(tool) is None: + pytest.skip(f"{tool} not on PATH") + + +# ---------------------------------------------------------------------------- # +# A — config resolves against the tenant: real numbers, resolved +# ---------------------------------------------------------------------------- # + +@pytest.mark.parametrize("fetcher", CONFIGURABLE) +def test_a_resolved_config_reports_real_numbers(fetcher, tmp_path): + code, ev, _ = run_fetcher(fetcher, TENANT_MATCHING, GOOD_CONFIG[fetcher], tmp_path) + assert code == 0 + res = ev["results"]["config_resolution"] + assert res["status"] == "resolved", res + assert res["measurable"] is True + assert res["campaigns"]["unmatched"] == [] + # Measured, so no compliance metric is null. + summary = ev["results"]["summary"] + for metric in COMPLIANCE_METRICS: + if metric in summary: + assert summary[metric] is not None, f"{metric} should be measured" + + +def test_a_security_awareness_counts_are_right(tmp_path): + """One user Passed, one In Progress -> 50%. Pins the arithmetic, not just nullness.""" + code, ev, _ = run_fetcher( + "knowbe4_security_awareness_training", TENANT_MATCHING, + GOOD_CONFIG["knowbe4_security_awareness_training"], tmp_path, + ) + assert code == 0 + s = ev["results"]["summary"] + assert s["total_users"] == 2 # the archived user is excluded + assert s["completed_training"] == 1 + assert s["in_progress"] == 1 + assert s["completion_rate"] == 50 + + +# ---------------------------------------------------------------------------- # +# B — config does NOT resolve: null metrics, exit 0, self-diagnosing +# ---------------------------------------------------------------------------- # + +@pytest.mark.parametrize("fetcher", CONFIGURABLE) +def test_b_unresolved_config_does_not_fail_the_fetcher(fetcher, tmp_path): + """A typo must not turn the run red — it exits 0 and says so in the evidence.""" + code, ev, stderr = run_fetcher(fetcher, TENANT_ALL_PASSED, TYPO_CONFIG[fetcher], tmp_path) + assert code == 0, f"a config typo must not fail the fetcher; stderr:\n{stderr}" + assert ev is not None, "unresolved config must still produce an evidence artifact" + assert ev["results"]["config_resolution"]["status"] == "unresolved" + # The fetcher emits WARN, but note the runner currently drops stderr on a + # zero exit (framework/api.py records stderr_tail only when exit_code != 0, + # and executor.py does not forward stderr to on_line). So the evidence + # artifact — not this line — is what actually reaches an operator today. + assert "WARN" in stderr + + +@pytest.mark.parametrize("fetcher", CONFIGURABLE) +def test_b_unresolved_config_nulls_every_compliance_metric(fetcher, tmp_path): + """The core contract. Nothing it could not measure may be reported as 0.""" + _, ev, _ = run_fetcher(fetcher, TENANT_ALL_PASSED, TYPO_CONFIG[fetcher], tmp_path) + summary = ev["results"]["summary"] + assert ev["results"]["config_resolution"]["measurable"] is False + for metric in COMPLIANCE_METRICS: + if metric in summary: + assert summary[metric] is None, ( + f"{metric} is {summary[metric]!r}; an unmeasured metric must be null, " + "never 0 — 0 reads as a genuine failing control" + ) + + +@pytest.mark.parametrize("fetcher", CONFIGURABLE) +def test_b_unresolved_config_names_what_it_looked_for_and_what_exists(fetcher, tmp_path): + """Self-diagnosing: the fix is visible in the artifact without a shell.""" + _, ev, _ = run_fetcher(fetcher, TENANT_ALL_PASSED, TYPO_CONFIG[fetcher], tmp_path) + res = ev["results"]["config_resolution"] + typo_values = set() + for raw in TYPO_CONFIG[fetcher].values(): + typo_values.update(v.strip() for v in raw.split(",")) + + requested = set(res["campaigns"]["requested"]) | set(res.get("groups", {}).get("requested", [])) + assert typo_values <= requested, "must echo back what it was asked to match" + assert res["campaigns"]["matched"] == [] + assert set(res["campaigns"]["unmatched"]) == set(res["campaigns"]["requested"]) + # And what the tenant actually has, so the typo is obvious. + present = res["campaigns_present_in_tenant"] + assert "2026 Annual Security Awareness Training" in present + + +# ---------------------------------------------------------------------------- # +# C — the pair that pins the distinction B alone cannot +# ---------------------------------------------------------------------------- # + +def test_c_genuine_zero_percent_is_still_reported_as_zero(tmp_path): + """Config resolves, nobody trained. That is a real finding and must survive.""" + code, ev, _ = run_fetcher( + "knowbe4_security_awareness_training", TENANT_NOBODY_PASSED, + GOOD_CONFIG["knowbe4_security_awareness_training"], tmp_path, + ) + assert code == 0 + res = ev["results"]["config_resolution"] + assert res["status"] == "resolved" + assert res["measurable"] is True + s = ev["results"]["summary"] + assert s["completed_training"] == 0, "measured zero must stay 0, not become null" + assert s["completion_rate"] == 0 + assert s["total_users"] == 2 + + +def test_c_unmeasured_and_genuine_zero_are_distinguishable(tmp_path): + """The regression this suite exists for. + + Before the fix these two runs produced byte-identical summaries and both + exited 0. If this test ever passes trivially again, the bug is back. + """ + fetcher = "knowbe4_security_awareness_training" + _, unmeasured, _ = run_fetcher( + fetcher, TENANT_ALL_PASSED, TYPO_CONFIG[fetcher], tmp_path / "unmeasured") + _, genuine, _ = run_fetcher( + fetcher, TENANT_NOBODY_PASSED, GOOD_CONFIG[fetcher], tmp_path / "genuine") + + assert unmeasured["results"]["summary"] != genuine["results"]["summary"] + assert unmeasured["results"]["summary"]["completion_rate"] is None + assert genuine["results"]["summary"]["completion_rate"] == 0 + + +# ---------------------------------------------------------------------------- # +# D — config is data, not jq program text +# ---------------------------------------------------------------------------- # + +@pytest.mark.parametrize("name", HOSTILE_NAMES) +def test_d_names_that_would_break_an_interpolated_jq_filter(name, tmp_path): + """Quotes, backslashes and $ in a campaign name must match, not crash jq.""" + code, ev, stderr = run_fetcher( + "knowbe4_security_awareness_training", TENANT_HOSTILE, + {"KNOWBE4_SECURITY_AWARENESS_CAMPAIGNS": name}, tmp_path, + ) + assert code == 0, f"jq error on {name!r}?\n{stderr}" + assert "jq: error" not in stderr and "compile error" not in stderr + res = ev["results"]["config_resolution"] + assert res["campaigns"]["matched"] == [name], f"{name!r} did not match; {res}" + assert res["measurable"] is True + + +def test_d_config_parsing_trims_and_drops_empties(tmp_path): + """' A , B ,,' is three tokens of whitespace and two real names.""" + code, ev, _ = run_fetcher( + "knowbe4_high_risk_training", TENANT_MATCHING, + { + "KNOWBE4_HIGH_RISK_GROUPS": " Cloud Ops ,, Engineering ", + "KNOWBE4_ROLE_SPECIFIC_CAMPAIGNS": "Privileged Users Training (Before CloudOps Access) ,", + }, + tmp_path, + ) + assert code == 0 + res = ev["results"]["config_resolution"] + assert sorted(res["groups"]["requested"]) == ["Cloud Ops", "Engineering"] + assert res["campaigns"]["requested"] == [ + "Privileged Users Training (Before CloudOps Access)"] + assert res["groups"]["unmatched"] == [] + + +def test_d_group_matching_is_exact_not_substring(tmp_path): + """'IT' must not sweep in 'AUDIT' or 'Legal-IT' the way substring matching did.""" + tenant = { + **TENANT_MATCHING, + "groups": [ + {"id": 30, "name": "AUDIT"}, + {"id": 31, "name": "Legal-IT"}, + {"id": 32, "name": "IT"}, + ], + "groups_32_members": [{"id": 1, "email": "a@ex.com", "status": "active"}], + } + _, ev, _ = run_fetcher( + "knowbe4_high_risk_training", tenant, + { + "KNOWBE4_HIGH_RISK_GROUPS": "IT", + "KNOWBE4_ROLE_SPECIFIC_CAMPAIGNS": "Privileged Users Training (Before CloudOps Access)", + }, + tmp_path, + ) + matched = ev["results"]["config_resolution"]["groups"]["matched"] + assert matched == ["IT"], f"substring bleed: {matched}" + + +# ---------------------------------------------------------------------------- # +# E — hard API failure is still a failure +# ---------------------------------------------------------------------------- # + +@pytest.mark.parametrize("fetcher", CONFIGURABLE + ["knowbe4_module_based_summary"]) +def test_e_api_failure_exits_nonzero(fetcher, tmp_path): + """Unresolved config is exit 0; an unreachable API is not. Keep them distinct.""" + config = GOOD_CONFIG.get(fetcher, {}) + code, _, stderr = run_fetcher(fetcher, TENANT_MATCHING, config, tmp_path, fail_api=True) + assert code != 0, "a failed API call must still fail the fetcher" + assert "ERROR" in stderr + + +# ---------------------------------------------------------------------------- # +# F — the quadratic append is gone +# ---------------------------------------------------------------------------- # + +def test_f_scales_to_a_realistic_enrollment_count(tmp_path): + """5k enrollments must finish fast, and must not be passed to jq via argv. + + Two regressions are pinned here, and the fixture size is load-bearing for both. + + 1. The per-record ``jq``-rewrite this replaced was quadratic: 1500 enrollments + took 69s and 3000 took over 120s, so a mid-size tenant blew the runner's + 600s cap (framework/runner/executor.py). The time bound catches that. + 2. Passing the array as one ``--argjson`` string dies at Linux's + MAX_ARG_STRLEN (131072 bytes for a single argument): execve fails E2BIG, + jq never runs, and the fetcher writes an EMPTY evidence file. macOS has no + per-argument cap, so 3000 enrollments (645KB) passed locally and only + failed in CI. 5000 enrollments is ~1.08MB, which also clears macOS's + 1048576-byte total ARG_MAX — so an argv regression now fails on both. + + For scale, the real Paramify tenant is 463 enrollments = 99KB, only 25% under + the Linux per-argument cliff. + """ + n = 5000 + tenant = { + "users": _USERS, + "training_enrollments": [ + {"enrollment_id": i, "campaign_name": "FY26", + "module_name": f"Mod {i % 20}", + "user": {"id": i % 200, "email": f"u{i % 200}@ex.com"}, + "status": "Passed" if i % 3 else "In Progress", + "completion_date": "2026-03-01T10:00:00.000Z", + "policy_acknowledged": False} + for i in range(n) + ], + } + start = time.monotonic() + code, ev, stderr = run_fetcher("knowbe4_module_based_summary", tenant, {}, tmp_path) + elapsed = time.monotonic() - start + + assert code == 0, stderr + assert len(ev["results"]["enrollments"]) == n + assert ev["results"]["summary"]["training_module_summary"]["Mod 0"]["assigned"] == n // 20 + assert elapsed < 30, f"{n} enrollments took {elapsed:.0f}s — per-record rewrite is back?" + + +def test_f_empty_tenant_summary_is_an_object_not_null(tmp_path): + """No enrollments is an empty map. `null` made downstream readers special-case it.""" + code, ev, _ = run_fetcher( + "knowbe4_module_based_summary", {"training_enrollments": []}, {}, tmp_path) + assert code == 0 + assert ev["results"]["summary"]["training_module_summary"] == {} + assert ev["results"]["enrollments"] == []