feat(coverage): add noise tolerance to the coverage gates#23
Merged
Conversation
A strict < comparison failed PRs on dips smaller than the displayed
precision ("86.1% vs baseline 86.1%, regressed 0.0%") even when the PR
touched no code in that language. Both gates now tolerate a configurable
dip, compared at the report's tenth-of-a-point display precision so what
is shown and what is gated always agree:
- coverage.tolerance (default 0.1pp) for the aggregate gate
- coverage.patch.tolerance (default 0.1pp) for the patch gate,
deliberately independent so loosening the noisy aggregate knob never
weakens the untested-new-code check
To keep tolerated dips from compounding into an unbounded downward
ratchet, the baseline writers restore any within-tolerance dip to the
prior high-water value when recording; only a beyond-tolerance drop
(FAIL-visible on the PR that introduced it) resets the baseline.
Load() rejects negative/NaN/Inf tolerances, which would otherwise
silently invert or disable the gates.
Claude-Session: https://claude.ai/code/session_01QMTWVM4pUB4xy6F6P9wWYA
pedromvgomes
force-pushed
the
feature/vulture-inselberg
branch
from
July 15, 2026 03:19
06994e0 to
2cc292c
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The coverage gate failed PRs on sub-rounding-error regressions — a strict
cur < basecomparison printed[FAIL] rust: 86.1% (baseline 86.1%, regressed 0.0%)on PRs containing no Rust changes at all.What
coverage.tolerance(default 0.1pp): the aggregate gate now tolerates dips up to this many percentage points below baseline. The comparison happens at the report's tenth-of-a-point display precision (regressedBeyond), so what's shown and what's gated always agree — raw float subtraction decided an exactly-at-tolerance dip by representation noise (86.2−86.1 > 0.1 but 86.1−86.0 ≤ 0.1).coverage.patch.tolerance(default 0.1pp): the patch gate gets its own knob, deliberately independent, so loosening the aggregate tolerance for a noisy suite never silently weakens the untested-new-code check.Load()rejects negative/NaN/±Inf tolerances, which would otherwise invert the gate (failing improved coverage) or silently disable both gates.Tests
New:
TestDiffReportPassesWithinTolerance,TestDiffReportFailsBeyondTolerance,TestDiffReportToleranceBoundaryIsRepresentationIndependent,TestRegressedBeyond,TestWithToleratedDipsRestored,TestDefaultCoverageTolerance,TestLoadRejectsInvalidTolerance,TestLoadCoverageToleranceExplicitZero. Full suite,go vet,gofmtclean.Merge Commit Message
feat(coverage): tolerate sub-noise coverage dips (0.1pp default, anti-ratchet recording, validated config knobs for aggregate and patch gates)
https://claude.ai/code/session_01QMTWVM4pUB4xy6F6P9wWYA