Skip to content

fix(coverage): never let a partial run shrink the baseline; UNMEASURED vs DROPPED#20

Merged
pedromvgomes merged 1 commit into
mainfrom
feature/vulture-inselberg
Jul 14, 2026
Merged

fix(coverage): never let a partial run shrink the baseline; UNMEASURED vs DROPPED#20
pedromvgomes merged 1 commit into
mainfrom
feature/vulture-inselberg

Conversation

@pedromvgomes

Copy link
Copy Markdown
Contributor

Problem

wardnet/wardnet#892 (a Rust-only PR) showed typescript: no longer measured (baseline was 93.8%) even though no TypeScript code was touched — wardnet's CI path-filters its coverage jobs, so the frontend coverage job was skipped and bulwark saw no TS report. Two defects behind that symptom:

  1. Misleading wording: a language that is still detected in the tree but produced no measurement this run isn't "no longer measured" — its coverage step just didn't run.
  2. Silent baseline shrinkage (the dangerous one): the same partial report, recorded on a push to main or cached on a PR cache miss, overwrote the baseline and silently deleted the unmeasured language's entry. Every later PR then saw it as [NEW], gated against nothing, permanently. The "never cache an empty baseline" guard doesn't catch this — the report isn't empty, just partial.

Fix

  • Both baseline writers merge instead of overwrite. carryForwardBaseline copies each detected-but-unmeasured language's entry from the nearest prior baseline via the new gitstate.PriorBaselines, which starts at the recorded SHA itself (a re-run or concurrent per-language job never clobbers a fresher same-commit entry; works even on depth-1 checkouts) before walking first-parent ancestors, skipping poisoned {} entries and stopping once every needed language is filled.
  • Nothing vanishes silently. An unmeasured language no prior had is named in a stderr warning (shallow history is the usual culprit). The recorded coverage baseline line names anything carried.
  • Deletion still deletes. An undetected language — source removed, or enabled: false in .bulwark.yml, now actually honored via enabledEcosystems — is never carried; its entry dies on the next record.
  • [UNMEASURED] vs [DROPPED]. diffReport reports a still-detected-but-unmeasured language as [UNMEASURED] <lang>: not measured this run (baseline X%), reserving [DROPPED] for source that actually left the tree. Neither fails the gate on its own.
  • PR comment regex made tag-generic in action.yml — the enumerated PASS|FAIL|NEW|DROPPED list would have silently filtered the new tag out of the sticky comment.

Testing

Test-first throughout: TestPriorBaselinesNearestCommitWinsPerLanguage (real git fixture: same-SHA entry wins, per-language nearest fill, empty-entry skip, maxDepth, missing-branch tolerance), TestUnmeasuredLanguages, TestMergeCarried, TestEnabledEcosystemsDropsDisabledLanguages, TestDiffReportUnmeasuredWhenLanguageStillDetected (reproduces #892's report shape). Full go test -race ./..., go vet, gofmt, golangci-lint clean. The cov_detail pipeline was simulated against real gate output to confirm [UNMEASURED] reaches the PR comment.

Merge Commit Message

fix(coverage): carry detected-but-unmeasured languages forward in baseline writes (never shrink on partial runs) and report them as [UNMEASURED] instead of [DROPPED]

https://claude.ai/code/session_01QMTWVM4pUB4xy6F6P9wWYA

…tected-but-unmeasured languages as UNMEASURED

A path-filtered CI run (wardnet/wardnet#892: Rust-only PR skips the
frontend coverage job) left typescript unmeasured, so the gate printed
the misleading "[DROPPED] typescript: no longer measured" — and worse,
the same partial report recorded on main (or cached on a PR cache miss)
would silently delete the language's baseline entry, un-gating it
permanently. The empty-baseline guard doesn't catch this: the report
isn't empty, just partial.

Both baseline writers (record-on-main and computeBaselineAt's
cache-miss path) now carry detected-but-unmeasured languages forward
from the nearest prior baseline via gitstate.PriorBaselines, which
starts at the recorded SHA itself — so re-runs and concurrent
per-language jobs never clobber a fresher same-commit entry, and a
depth-1 checkout still sees it — before walking first-parent ancestors,
skipping poisoned {} entries and stopping once every needed language is
filled. Anything no prior had is named in a stderr warning instead of
vanishing. A language that is genuinely gone (source deleted, or
enabled: false — now honored via enabledEcosystems) stays undetected
and its entry dies with the next record.

diffReport splits the old DROPPED case: still-detected languages get
"[UNMEASURED] <lang>: not measured this run (baseline X%)", reserving
DROPPED for source that actually left the tree. action.yml's cov_detail
grep becomes tag-generic so the new tag (and any future one) reaches
the PR comment instead of being silently filtered.

Claude-Session: https://claude.ai/code/session_01QMTWVM4pUB4xy6F6P9wWYA
@pedromvgomes
pedromvgomes merged commit c20b279 into main Jul 14, 2026
4 checks passed
@pedromvgomes
pedromvgomes deleted the feature/vulture-inselberg branch July 14, 2026 11:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant