Verification Gate is green having executed zero commands: run_verification.py counts 'skipped' as pass

[avrabe]

Found during a multi-persona review of REQ-GUARD-GATE-EVIDENCE-002. Verified at 618b29df.

This is the requirement's own defect shape — an operation that can produce "nothing happened" must render differently from "it worked" — inside the required context that certifies it.

Executed

$ tools/run_verification.py --filter '(and (= type "feature") (has-field "test-name"))'
...
passed:  0
failed:  0
skipped: 64
EXIT=0

64 artifacts matched. Zero commands ran. Exit 0. In .github/workflows/verification-gate.yml:100-101 that makes steps.verify.outcome == 'success', the fail-step never runs, and the required context Verification Gate (rivet-driven) reports green having executed nothing.

Mechanism

tools/run_verification.py:127-130:

        if not steps:
            print(f"[SKIP] {artifact_id} (no fields.steps[].run)")
            result.skipped.append(artifact_id)
            continue

:164:

    return 0 if result.failed_count == 0 else 1

Line 110 already handles the empty-match case (if not ids: return 1). So "the filter matched nothing" was closed and "the filter matched, and nothing ran" was left open. That contrast is the finding — the author saw the hazard and stopped one step short of it.

Scale

93 of 237 type: feature artifacts have zero runnable steps; 85 of those are status: implemented. 66 sit in safety/stpa/validation.yaml, whose test-name: values include TBD, tests::*, and golden model and validation tests.

The filter is author-controlled from the PR body via the Verify-Filter: line (verification-gate.yml:52-72), so the selected set is not fixed.

The recent regression

All 8 newly-added TEST-GUARD-* artifacts in artifacts/verification.yaml have fields.steps: 0 and links: null:

TEST-GUARD-LEAN-SORRY-FLOOR      steps=0  links=NONE  implemented
TEST-GUARD-CLIPPY-SCOPE          steps=0  links=NONE  implemented
TEST-GUARD-FUZZ-TARGETS          steps=0  links=NONE  implemented
TEST-GUARD-VERIFICATION-FILTERS  steps=0  links=NONE  implemented
TEST-GUARD-CHANGED-PATHS         steps=0  links=NONE  implemented
TEST-GUARD-MUTANTS-TRUNCATION    steps=0  links=NONE  implemented
TEST-GUARD-RELEASE-PLANE         steps=0  links=NONE  implemented
TEST-GUARD-STATUS-VOCAB          steps=0  links=NONE  proposed

The three older -001-era ones each carry 2 real steps and satisfies/verifies links:

TEST-GUARD-HUMAN-SCOPED          steps=2  links=yes   implemented
TEST-GUARD-GATE-EVIDENCE         steps=2  links=yes   implemented
TEST-GUARD-FMT-WORKSPACES        steps=2  links=yes   implemented

So this is a regression in convention, not the original design — and repo-wide, 162 of 170 type: feature artifacts in verification.yaml do carry links:. The artifacts asserting that (c)–(h) are verified are themselves unexecutable prose, and grep 'target: REQ-GUARD-GATE-EVIDENCE-002' artifacts/ returns nothing: the V is closed in prose only.

And the (c) guardrail cannot see it

tools/check_verification_filters.py classifies only - run: steps, so a step-less artifact contributes zero to every counter. Executed on fixtures:

input	exit	output
a step-less implemented artifact	0	steps read: 0 … All 0 evidence-claiming filtered steps select at least one test.
one real step	0	steps read: 1 … All 1 …
both together	0	steps read: 1 … All 1 … — byte-identical to the previous row

Distinct inputs, identical outputs. (c) made "a filter that selects nothing" fail; "no filter at all" still yields the ideal reading.

Suggested fix

1. run_verification.py: a skipped artifact that claims a completed status must not pass. At minimum a --max-skipped floor in the same shape as check_fmt_workspaces.py's --min-workspaces; better, exit non-zero when any implemented/verified artifact in the selected set has no runnable step.
2. check_verification_filters.py: count step-less evidence-claiming artifacts and fail on them, so "no filter" and "vacuous filter" are the same finding.
3. Backfill fields.steps and satisfies/verifies links on the 8 TEST-GUARD-* artifacts.
4. The 66 STPA entries with TBD-class test-name values need triage — likely to proposed.

Candidate for a REQ-GUARD-GATE-EVIDENCE-003 successor: the -002 obligations each fixed one gate; this is the layer that runs the gates.

🤖 Generated with Claude Code