feat(coverage): record the baseline when running on main#18
Merged
Conversation
A repo whose coverage comes from a multi-job CI pipeline can never compute a
baseline. computeBaselineAt's throwaway worktree is a bare checkout: no
cargo-llvm-cov, no yarn/Node, none of the reports the pipeline stages. It
measures nothing, so the baseline is empty and the gate compares against
nothing — which is exactly how wardnet ran for months.
The numbers it kept failing to reconstruct were numbers it had already measured.
`bulwark coverage` runs on main too, reads that pipeline's reports (--tests=skip),
prints rust: 85.7%, typescript: 93.9% — and threw them away. The baseline it
needed was in its hands.
So: when HeadSHA == BaseSHA (running ON the merge-base — a push to main — rather
than ahead of it), there is nothing to gate against, because the current commit
IS the baseline. Record what was just measured to bulwark-state and stop. No test
re-run, no extra tooling, no worktree. computeBaselineAt stays as the fallback for
a main commit bulwark never ran on.
Verified end-to-end against a throwaway repo with its own origin: a run on main
records `{"go": 50}`, and a PR branch that adds untested code then fails against
it — [FAIL] go: 20.0% (baseline 50.0%) and [FAIL] go patch: 0.0% (0/5 new lines) —
with no worktree recompute at all. That is the gate firing for the first time on
a repo shaped like wardnet.
Consumers must run `bulwark coverage` on pushes to main, not only on PRs.
Claude-Session: https://claude.ai/code/session_01GbTu5D5FuSGj9Hqo1YE4ci
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The baseline was in bulwark's hands, and it threw it away
#17 made wardnet's failure legible. This fixes it.
wardnet's coverage comes from a multi-job pipeline (daemon / site / frontend jobs emit LCOV; bulwark consumes it with
--tests=skip). Such a repo can never compute a baseline:computeBaselineAt's throwaway worktree is a bare checkout with nocargo-llvm-cov, no yarn/Node, and none of the staged reports the pipeline provides. It measures nothing. That's why every baseline on wardnet'sbulwark-statebranch was{}and its gate compared against nothing for months.But
bulwark coveragealso runs on main (theCoverageworkflow is called fromci.yml, not justpr.yml). There it reads the pipeline's reports and printsrust: 85.7%, typescript: 93.9%— the exact numbers it later fails to reconstruct in a worktree — and discards them.The fix
When
HeadSHA == BaseSHA— running on the merge-base (a push to main) rather than ahead of it (a PR) — there is nothing to gate against, because the current commit is the baseline. So record what was just measured tobulwark-stateand stop.No test re-run. No
cargo-llvm-cov. No yarn. No worktree. The numbers already exist; bulwark just had to keep them.computeBaselineAtstays as the fallback for a main commit bulwark never ran on, so nothing regresses for repos (like inforge) where recompute already works.Verification — the gate firing, end to end
Against a throwaway repo with its own bare origin:
No
no cached baseline … computing one nowline in run 2 — it read the recorded baseline directly. Aggregate and patch both caught the regression. That is the gate working for the first time on a repo shaped like wardnet.go build,go test -race,golangci-lintall clean.Consumer requirement (worth calling out)
Consumers must run
bulwark coverageon pushes to main, not only on PRs — otherwise there's nothing to record and they fall back to the (often impossible) recompute path. wardnet already does; AGENTS.md now says so explicitly.Merge Commit Message
feat(coverage): record the baseline when running on main