Mutants Weekly now reports honestly — and reports a 1.7% kill rate (23 caught / 1326 viable)

[avrabe]

What changed

#387 fixed the report-path misread in mutants-weekly.yml (the same mutants-out/ vs mutants-out/mutants.out/ bug as #381). Before, this workflow rendered 0 missed / 0 caught / 0 unviable — not merely wrong but impossible, and still a plausible-looking table.

First honest run: 30996683710, on main @ 9903d1a, shard 0/8:

1421 mutants tested in 2h: 1303 missed, 23 caught, 95 unviable
Summarised 1303 missed / 23 caught / 95 unviable / 0 timeout.

The reporting fix works. This issue is about what it revealed.

The number

23 caught out of 1326 viable — a 1.7% kill rate.

For comparison, the gating job (Mutation Testing, -p spar-analysis, same -- --lib) on the same commit: 708 caught / 292 missed ≈ 71%.

A 1.7% vs 71% gap between two runs of the same tool with the same test-selection flag is too large to be a coverage difference alone.

What is measured, and what is not

Measured:

• Both invocations pass -- --lib, so both restrict cargo test to lib unit tests.
  • weekly: cargo mutants <pkgs> --shard 0/8 --timeout 180 --jobs 8 --output mutants-out -- --lib
  • gating: cargo mutants -p spar-analysis --timeout 120 --jobs 4 --output mutants-out -- --lib
• --lib being the culprit is ruled out for the biggest contributor: spar-cli has 181 #[test] in src/ (plus 113 in tests/). Its lib tests exist and are numerous.
• Every MISSED line in the log reports Ns build + **0s test**. Example:

  MISSED crates/spar-cli/src/lsp.rs:3153:21: replace == with != in
         completion_context_from_cst in 8s build + 0s test
  

• 23 mutants were caught, so the suite is not universally inert.
• Wall clock: 1421 mutants in 2h at --jobs 8. At ~8s build each that is ~24 min of build work across 8 workers, so ~2h is unexplained by build time alone.

Not established — candidate hypotheses only, none verified:

1. The mutated copy's cargo test --lib selects a package set that builds the lib but runs (almost) no test binary.
2. --jobs 8 on the lean-mem runner causes test processes to be starved or killed in a way that reads as "not caught" rather than as a timeout (0 timeout argues against this, but the bucket only counts cargo-mutants' own timeout).
3. The kill rate is genuine and spar-cli's lib tests simply do not exercise the mutated regions (lsp.rs completion logic dominates the sample).

Hypothesis 3 would be a real and useful finding. 1 and 2 would mean the workflow is still measuring the adjacent question — a mutation score for a test suite that did not run.

Why this is not urgent, and not ignorable

Mutants Weekly is advisory; it gates nothing and blocks no merge. Nothing here should hold a release.

But an advisory number that is wrong is worse than no number, because it will be read. Until the 0s test is explained, do not quote the 1.7% as a coverage figure and do not set any ratchet from this run — that is exactly how #381's threshold came to be derived from a truncated report (see #389).

Suggested first step

Cheapest discriminator, and it does not require a full 2h run: run one shard with a single known-covered mutant and capture the mutated copy's actual cargo test invocation and output (cargo-mutants writes per-mutant logs under mutants.out/log/). If the log shows running 0 tests, it is hypothesis 1 and the fix is in test selection, not in the test suite.

The mutants-report artifact from 30996683710 contains those logs and is already uploaded.

Refs #381, #382, #389