Gremlins mutation testing tracker

[cplieger]

Gremlins mutation testing tracker

Auto-updated weekly by cplieger/ci/.github/workflows/weekly-gremlins.yaml.
Last update: 2026-08-31 07:56

This week: 95.1% efficacy (±0.0% across 3 runs), 93.5% mutant coverage. 7 confirmed live mutants.
Trend: ↗ +1.7% from 12-week mean (93.4%).

Rolling 12-week history

Run (UTC)	Mean efficacy	Stddev	Mutant coverage	Live mutants	Δ efficacy
2026-08-31 07:56	95.1%	±0.0%	93.5%	7	+0.0%
2026-08-24 05:00	95.1%	±0.0%	93.5%	7	+4.3%
2026-08-21 15:35	90.8%	±0.0%	92.2%	13	-0.4%
2026-08-17 04:29	91.2%	±0.0%	92.5%	13	+0.0%
2026-08-10 04:03	91.2%	±0.0%	92.5%	13	+0.0%
2026-07-27 04:17	91.2%	±0.0%	92.5%	13	-4.6%
2026-07-20 04:11	95.8%	±0.0%	94.4%	5	-4.2%
2026-07-13 02:16	100.0%	±0.0%	94.9%	0	+0.0%
2026-07-07 23:28	100.0%	±0.0%	88.1%	0	+1.8%
2026-06-29 19:29	98.2%	±0.0%	87.9%	2	+0.0%
2026-06-29 00:22	98.2%	±0.0%	87.9%	2	-0.8%
2026-06-22	99.0%	±0.0%	91.4%	1	+28.5%

Current live mutants (7, this week)

Solid gap — LIVED in all 3 runs (known coverage hole) — 7

internal/collect/collect.go

• L496 — CONDITIONALS_BOUNDARY
• L499 — ARITHMETIC_BASE

internal/github/condcache.go

• L85 — CONDITIONALS_BOUNDARY
• L195 — CONDITIONALS_BOUNDARY
• L198 — ARITHMETIC_BASE
• L201 — CONDITIONALS_BOUNDARY
• L209 — CONDITIONALS_BOUNDARY

How to read

• Mean efficacy: % of runnable mutants killed (or timed-out, treated as caught), averaged across the N runs
• Stddev: variance across runs — high stddev (>3%) signals flaky tests
• Mutant coverage: % of mutants reached by the test suite (test depth)

The "Current live mutants" section is bucketed by how many of the N runs the
mutant LIVED in (N = attempts that week, normally 3):

• Rare flake (1/N): a mutant your tests usually KILL but occasionally let
  through. Most actionable — almost always means a flaky test that's not
  reliable. Open the section to see the file:line; expect to fix the test, not
  the production code.
• Weakly flaky (k/N for 1<k<N): tests catch this mutant inconsistently.
  Less ideal than rare-flake but still worth investigating.
• Solid gap (N/N): every run lets this mutant through. Either there's no
  test exercising the path, or the test asserts the wrong thing. Action: add a
  test or strengthen an existing assertion.

The mutation-regression label is added when this week's mean efficacy drops

5% below the rolling 12-week mean.

A ⚠️ line under "This week" means the number describes the measurement, not the
suite: either the attempts disagreed about which mutants survive (one verdict per
disagreement is false), or the score jumped to a flawless 100% from a week with
live mutants. Equivalent mutants are a permanent noise floor, so a reproducible
100% is not something a test suite can reach.

Free-form notes

Add anything below — won't be touched by the auto-updater.