Skip to content

feat(coverage): record the baseline when running on main#18

Merged
pedromvgomes merged 1 commit into
mainfrom
feature/baseline-on-main
Jul 12, 2026
Merged

feat(coverage): record the baseline when running on main#18
pedromvgomes merged 1 commit into
mainfrom
feature/baseline-on-main

Conversation

@pedromvgomes

Copy link
Copy Markdown
Contributor

The baseline was in bulwark's hands, and it threw it away

#17 made wardnet's failure legible. This fixes it.

wardnet's coverage comes from a multi-job pipeline (daemon / site / frontend jobs emit LCOV; bulwark consumes it with --tests=skip). Such a repo can never compute a baseline: computeBaselineAt's throwaway worktree is a bare checkout with no cargo-llvm-cov, no yarn/Node, and none of the staged reports the pipeline provides. It measures nothing. That's why every baseline on wardnet's bulwark-state branch was {} and its gate compared against nothing for months.

But bulwark coverage also runs on main (the Coverage workflow is called from ci.yml, not just pr.yml). There it reads the pipeline's reports and prints rust: 85.7%, typescript: 93.9% — the exact numbers it later fails to reconstruct in a worktree — and discards them.

The fix

When HeadSHA == BaseSHA — running on the merge-base (a push to main) rather than ahead of it (a PR) — there is nothing to gate against, because the current commit is the baseline. So record what was just measured to bulwark-state and stop.

No test re-run. No cargo-llvm-cov. No yarn. No worktree. The numbers already exist; bulwark just had to keep them.

computeBaselineAt stays as the fallback for a main commit bulwark never ran on, so nothing regresses for repos (like inforge) where recompute already works.

Verification — the gate firing, end to end

Against a throwaway repo with its own bare origin:

RUN 1 — on main (HEAD == merge-base):
  recorded coverage baseline for 41b74f8: go: 50.0%
  bulwark-state:41b74f8.json  =>  {"go": 50}

RUN 2 — PR branch adding untested code:
  [FAIL]    go: 20.0% (baseline 50.0%, regressed 30.0%)
  [FAIL]    go patch: 0.0% (0/5 new lines; baseline 50.0%)

No no cached baseline … computing one now line in run 2 — it read the recorded baseline directly. Aggregate and patch both caught the regression. That is the gate working for the first time on a repo shaped like wardnet.

go build, go test -race, golangci-lint all clean.

Consumer requirement (worth calling out)

Consumers must run bulwark coverage on pushes to main, not only on PRs — otherwise there's nothing to record and they fall back to the (often impossible) recompute path. wardnet already does; AGENTS.md now says so explicitly.

Merge Commit Message

feat(coverage): record the baseline when running on main

A repo whose coverage comes from a multi-job CI pipeline can never compute a
baseline. computeBaselineAt's throwaway worktree is a bare checkout: no
cargo-llvm-cov, no yarn/Node, none of the reports the pipeline stages. It
measures nothing, so the baseline is empty and the gate compares against
nothing — which is exactly how wardnet ran for months.

The numbers it kept failing to reconstruct were numbers it had already measured.
`bulwark coverage` runs on main too, reads that pipeline's reports (--tests=skip),
prints rust: 85.7%, typescript: 93.9% — and threw them away. The baseline it
needed was in its hands.

So: when HeadSHA == BaseSHA (running ON the merge-base — a push to main — rather
than ahead of it), there is nothing to gate against, because the current commit
IS the baseline. Record what was just measured to bulwark-state and stop. No test
re-run, no extra tooling, no worktree. computeBaselineAt stays as the fallback for
a main commit bulwark never ran on.

Verified end-to-end against a throwaway repo with its own origin: a run on main
records `{"go": 50}`, and a PR branch that adds untested code then fails against
it — [FAIL] go: 20.0% (baseline 50.0%) and [FAIL] go patch: 0.0% (0/5 new lines) —
with no worktree recompute at all. That is the gate firing for the first time on
a repo shaped like wardnet.

Consumers must run `bulwark coverage` on pushes to main, not only on PRs.

Claude-Session: https://claude.ai/code/session_01GbTu5D5FuSGj9Hqo1YE4ci
@pedromvgomes
pedromvgomes merged commit d9e8db9 into main Jul 12, 2026
4 checks passed
@pedromvgomes
pedromvgomes deleted the feature/baseline-on-main branch July 12, 2026 15:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant