"mutation-tested" is an unrepeatable prose claim, and its universal form in REQ-GUARD-GATE-EVIDENCE-002 is false

[avrabe]

Found during a multi-persona review of REQ-GUARD-GATE-EVIDENCE-002. Verified at 618b29df. Self-reported: the false claim is mine.

The claim

artifacts/requirements.yaml:4989, inside REQ-GUARD-GATE-EVIDENCE-002 itself:

"Each ships a self-test that runs BEFORE the gate it guards is allowed to judge anything, and each was mutation-tested to show its cases are load-bearing rather than merely green."

That is a universal over six tools. It is backed by two dated testimonies about two tools (artifacts/verification.yaml:4329 TEST-GUARD-CHANGED-PATHS, :4393 TEST-GUARD-MUTANTS-TRUNCATION) plus one comment about a third (ci.yml:524, check_release_plane.py) — and no mechanism anywhere. git ls-tree -r HEAD --name-only | grep -i mutat returns only mutants-weekly.yml and check_mutants_report.py; no harness, no CI step, no recorded output.

The universal is false, demonstrably

Two mutants applied in a scratch copy, each asserted to have changed the file before drawing a conclusion:

Mutant A — tools/check_verification_filters.py:212:

  if not any(filt in name for name in inventory)
→ if not any(filt.split('::')[-1] in name for name in inventory)

→ 13 passed, 0 failed — SURVIVES.

Not cosmetic: it changes vacuity semantics for every module-path filter, and 16 of the 60 distinct filters in artifacts/verification.yaml contain :: (ingest::tests, engine::tests::gptp_, assertion::tests::eval_count, curves::piecewise, …). No self-test case has a :: in its filter, so split('::')[-1] is the identity on every fixture and the suite is structurally blind to it.

Mutant B — tools/check_lean_sorries.py, delete the entire below-floor ratchet-notice branch → 8 passed, 0 failed — SURVIVES.

So the requirement asserts a property of six tools that fails for at least two. Inside the requirement whose subject is "a guard whose ERROR path yields its IDEAL reading is unfalsifiable by construction, and no amount of watching it pass distinguishes clean from blind."

The uncomfortable correlation

The two tools that claim mutation testing (#384, #389) have the two strongest suites in the set. The two with real live holes (#385 pattern width, #388 cross-package scope) claim nothing. The claim is anti-correlated with need — which is what you would expect, since it was reached for only where confidence was already high. Qualifying the prose does not fix that; mechanising it does.

Proposed fix — tools/check_self_test_potency.py

The hard part already exists: all seven guard tools expose --self-test with a stable contract (exit 0 clean, non-zero on any failed case). The harness is then ~40 lines of stdlib:

1. read the tool source,
2. apply a declared textual substitution,
3. assert the file actually changed (a mutation that silently fails to apply reports as "not caught" and is worthless — this repo has already been bitten by exactly that),
4. run --self-test, assert it now exits non-zero,
5. restore; report per-mutant CAUGHT/SURVIVED.

Mutants declared in a table next to the tools they target, run as one more step inside the already-required Verification Gate (rivet-driven). It would have caught both mutants above.

Caveat worth writing into the requirement afterwards: mutation testing confirms the cases present are load-bearing. It cannot show the case set covers the hazard. Both live holes prove this from the other side — check_lean_sorries.py:178 contains a case that affirms the narrow pattern as intended, so there is no guard there to mutate; check_verification_filters.py has no case for the cross-package hole at all. A green potency run on either would be true and still miss the defect.

Interim, regardless of whether the harness lands

• Delete or narrow the universal at requirements.yaml:4989 — it is unsupported even before the mutants falsified it.
• The two artifact entries may keep their testimony, but qualified: name the commit, not the date (a date does not identify a file); state the mutations by content; and carry the expiry clause explicitly — "MANUAL, NOT RE-RUN: this result does not survive an edit to that file, and nothing detects when it stops holding." That converts a claim into a recorded known gap, which is the one form that is honest and stays honest.

Related

.github/workflows/ci.yml:388 still reads "Nothing here yet fails a truncated run: that detector, and the runner disk-pressure fault behind it, are #389." That became false when #401 merged, two commits ago — the same "code drifted out from under its comment" species that #385's own fix narrative identifies as its root cause. One-line deletion.

🤖 Generated with Claude Code