Seen on #540's clippy + test (ubuntu-latest), attempt 2 (run 33939306963), during a window when GitHub's runners were visibly overloaded (apt timeouts across several workflows):
thread 'the_instrument_survives_saturated_counters_and_a_hostile_host' panicked at tests/health_reports_degradation.rs:305:5:
this case must hand the registry back clean, or it poisons whichever case libtest happens to run next:
[Warning { code: "resource.io_stall", message: "io stalled: EVERY task in this container was blocked on io 18.4% of the last 10s (some 19.3%) — work is waiting for DISK. …" }]
Passed on the ubuntu lanes of e2f9557 (run 33931911725) and ce36e0d (run 33913080025).
Mechanism. The test saturates the counters, then runs the REAL probes (probe_memory, probe_contention) "on whatever this host is", asserts the health payload survives, and then clears only the two counters it set (report_backpressure_drops(0), a healthy report_network_rounds) before asserting !degraded_mode(). On a host that is actually stalled, probe_contention reads PSI and raises a true resource.io_stall alarm, which nothing in the cleanup clears — so the instrument did exactly what it should (reported a real stall) and the test called that a failure. The assertion's own message names the right concern (a process-global registry leaking into the next case), but the way it checks it — "nothing is degraded" — is only true on a healthy host.
Ask: make the cleanup total rather than partial — clear every code the case could have raised (its own two AND anything the probes can raise, or a test-only degradation::reset()), and assert on the snapshot being empty of this case's codes rather than on degraded_mode() globally. A hostile host is the one place this test claims to be correct; it should not be the one place it fails.
Context: found while clearing main's lanes after the 0.5.197 merge (#529, #538, #539, #540, #542).
🤖 Generated with Claude Code
https://claude.ai/code/session_01QVZ3v83uX8noLxXr7d7pu9
Seen on #540's
clippy + test (ubuntu-latest), attempt 2 (run 33939306963), during a window when GitHub's runners were visibly overloaded (apt timeouts across several workflows):Passed on the ubuntu lanes of e2f9557 (run 33931911725) and ce36e0d (run 33913080025).
Mechanism. The test saturates the counters, then runs the REAL probes (
probe_memory,probe_contention) "on whatever this host is", asserts the health payload survives, and then clears only the two counters it set (report_backpressure_drops(0), a healthyreport_network_rounds) before asserting!degraded_mode(). On a host that is actually stalled,probe_contentionreads PSI and raises a trueresource.io_stallalarm, which nothing in the cleanup clears — so the instrument did exactly what it should (reported a real stall) and the test called that a failure. The assertion's own message names the right concern (a process-global registry leaking into the next case), but the way it checks it — "nothing is degraded" — is only true on a healthy host.Ask: make the cleanup total rather than partial — clear every code the case could have raised (its own two AND anything the probes can raise, or a test-only
degradation::reset()), and assert on the snapshot being empty of this case's codes rather than ondegraded_mode()globally. A hostile host is the one place this test claims to be correct; it should not be the one place it fails.Context: found while clearing main's lanes after the 0.5.197 merge (#529, #538, #539, #540, #542).
🤖 Generated with Claude Code
https://claude.ai/code/session_01QVZ3v83uX8noLxXr7d7pu9