ci: per-PR Mutation Testing occupies the scarce lean-mem pool for up to 4h with a ~4% success rate

[avrabe]

Summary

The mutants (Mutation Testing) job in ci.yml runs on every code-touching PR/push, on the scarce 4-runner lean-mem pool, with timeout-minutes: 240. By the workflow's own measurements it succeeds ~1 run in 25. The result is that a 4-runner pool shared by the whole org is occupied for hours by a job that is usually cancelled before it finishes — starving the gating jobs (Miri, Verification Gate, Coverage) that queue behind it.

Credit where due: the exhaustive full-workspace run was already correctly split out to mutants-weekly.yml. This issue is only about the per-PR gating job that remained on lean-mem.

Evidence

From the workflow's own comment (ci.yml, mutants job):

"The only run of this job to reach success in the last 25 CI runs took 182.3 minutes; the other 16 were cancelled by a later push."

So ~1 success in 25 runs, at ~3 hours each, with 16 cancelled mid-flight — each of those still held a lean-mem runner until the cancellation landed. That's mostly wasted occupancy of the org's scarcest CI resource.

From a 14-day org-wide runner audit (sampled Actions jobs, grouped by runner class):

runner class	jobs	median wait	p95 wait	>5 min	non-success
lean-mem (4 runners)	284	64 min	~23 h	56%	42%
rust-cpu (7 runners)	1129	0.3 min	52 min	31%	14%
light (1)	489	0.1 min	4.6 min	4%	8%

lean-mem is the fleet's only contended class — rust-cpu was ~86% idle in the same window. spar's share of the lean-mem load in that sample:

repo · job	instances	trigger	med wait	fail%
spar · Miri	47	PR/push	7 min	21%
spar · Mutation Testing	46	PR/push	12 min	28%

Live observation (2026-08-07 ~00:30Z): spar: Mutation Testing was occupying 2 of the 4 lean-mem runners simultaneously.

Why it matters

lean-mem is 4 runners for the entire org (rivet Miri/Verus, witness, sigil fuzz, spar Miri/mutants all target it). A per-PR job with a 4-hour ceiling and a ~4% success rate is the single easiest thing to move off it. Meanwhile rust-cpu has 7 runners and sat ~86% idle over the audit window — there is spare capacity right next door.

Suggested fixes (cheapest first)

1. Move the gating mutants job to rust-cpu. It's already narrowed to spar-analysis (not the full workspace), so the RAM argument for lean-mem is much weaker than it was. One-line runs-on change; frees the scarce pool immediately.
2. Cut timeout-minutes: 240. Four hours on a shared 4-runner pool is a long time to hold a slot for a job that is usually superseded. Something closer to the measured 182 min — or lower with (3) — bounds the damage.
3. Scope to changed code — cargo mutants --in-diff so PR-time mutation shrinks with the diff instead of re-running the whole crate. This likely also fixes the cancellation rate, since faster runs finish before the next push.
4. Optional: keep it on lean-mem but make it non-blocking/nightly, leaving mutants-weekly.yml as the thorough pass.

(1) alone resolves the contention; (1)+(3) would also make the job actually useful per-PR.

Minor: stale comment

Both the miri and mutants job comments describe lean-mem as "24 G MemoryHigh". It is currently MemoryHigh=32G, MemoryMax=48G (a hard cap was added 2026-06-25 after a runaway test OOM-killed the host). Worth updating so the sizing rationale reflects reality.

Acceptance

• Per-PR mutation testing no longer occupies lean-mem, or is bounded so it can't hold a slot for hours.
• lean-mem median job wait drops back to minutes, so Miri / Verification Gate / Coverage stop queueing behind it.

Related: pulseengine/rivet#523 (same pattern in rivet — per-PR mutants on lean-mem), pulseengine/sigil#146 (nightly fuzz fan-out on lean-mem), rivet#509 (self-hosted pool fragility).

---

Filed from a 14-day runner-utilization audit of the pulseengine self-hosted fleet plus live observation. Suggested labels: ci, infrastructure.

[avrabe]

Measured the last 25 CI workflow runs on spar (2026-08-07), pulling per-job
started_at/completed_at for Mutation Testing against each run's
created_at. The contention finding holds. One input number does not.

Corroborated: the queueing is real

	median	max
queue wait (run created → job started)	20 min	96 min
run time	49 min	152 min

And the 2-of-4 observation reproduces exactly: at 01:26:01Z and 01:27:04Z,
two Mutation Testing jobs started 63 seconds apart — one on PR #396, one on
main after #395 merged. PR #396 waited 57 minutes for a slot and #396's
predecessor waited 58; one run on feat/taprio-cmd-filter-390 waited 96.

Correction: the ~4% success rate is stale

The "~1 success in 25 runs" figure traces to the comment in ci.yml:331-332:

The only run of this job to reach success in the last 25 CI runs took
182.3 minutes; the other 16 were cancelled by a later push.

That was true when written. Over the current 25-run window it is not:

outcome	count
success	15
cancelled	8
skipped	1

15 of 24 non-skipped = 62.5% success, not ~4%.

The mechanism is worth naming, because it changes the fix. All eight
cancellations fall on a single branch (fix/fixture-vm-flake-lock) — one
push-heavy debugging session where each new push cancelled the previous run via
cancel-in-progress. Cancellation here is episodic, correlated with rapid
iteration on one branch, not a steady-state property of the job. The 16/25
figure was itself sampled during such a burst.

This does not weaken the case for moving off lean-mem — a 49-minute median job
waiting 20 minutes for a slot on a 4-runner pool is still the right thing to
move. It does mean the job is usually finishing and usually useful, so
suggestion (4) (make it non-blocking/nightly) is a bigger loss than the ~4%
figure implied.

Bearing on suggestion (2)

Cutting timeout-minutes: 240 "closer to the measured 182 min" is safe, but 182
is not the right anchor either — observed run time is median 49 min, max 152
min. A 180-minute cap costs nothing. A 120-minute cap would have killed the
legitimate 152-minute run on batch/ci-guardrails.

Also worth updating ci.yml:331-332 itself as part of this — it is the source
of the stale figure, and it will keep being quoted while it sits there.

Agreed on (1) and (3)

Nothing contradicts those. (1) is a one-line runs-on change and frees the
scarce pool immediately; (3) --in-diff would cut the 49-minute median directly
and shrink the cancellation window that produces the bursts above.

Related

#398 (filed a few hours ago) is the complement of suggestion (2) from the other
direction: Mutation Testing is one of only 3 of the 18 required contexts
that declares a timeout-minutes at all. The other 15 inherit the 360-minute
default. Between them: this job's ceiling is too high, and fifteen others have
no ceiling.

Method note: measured via the Actions jobs API rather than from the workflow
comment, which is exactly how the discrepancy surfaced.