From c2cfd9e1124513452a950891d2be488983d3ce84 Mon Sep 17 00:00:00 2001 From: Pedro Gomes Date: Tue, 14 Jul 2026 22:05:14 +0100 Subject: [PATCH 1/3] chore: bump agentic-toolkit lock to 13fbcd8 Claude-Session: https://claude.ai/code/session_01QMTWVM4pUB4xy6F6P9wWYA --- .agentic-toolkit.lock.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.agentic-toolkit.lock.yaml b/.agentic-toolkit.lock.yaml index 9693da4..e50a7e7 100644 --- a/.agentic-toolkit.lock.yaml +++ b/.agentic-toolkit.lock.yaml @@ -5,7 +5,7 @@ sources: sha: 3a19c994319dd8a0d16db16629d19df46cd7e920 - url: github.com/pedromvgomes/agentic-toolkit.git ref: main - sha: 0fa843ca38a50e3d1d070cb8efe381d2239afceb + sha: 13fbcd81632de892d516a451b64fa07197a93893 - url: github.com/anthropics/skills.git ref: main sha: 9d2f1ae187231d8199c64b5b762e1bdf2244733d From 6bce14b3d74f0a50c03f87967f4bcf9289e36b5a Mon Sep 17 00:00:00 2001 From: Pedro Gomes Date: Wed, 15 Jul 2026 04:12:35 +0100 Subject: [PATCH 2/3] feat(coverage): add noise tolerance to the coverage gates A strict < comparison failed PRs on dips smaller than the displayed precision ("86.1% vs baseline 86.1%, regressed 0.0%") even when the PR touched no code in that language. Both gates now tolerate a configurable dip, compared at the report's tenth-of-a-point display precision so what is shown and what is gated always agree: - coverage.tolerance (default 0.1pp) for the aggregate gate - coverage.patch.tolerance (default 0.1pp) for the patch gate, deliberately independent so loosening the noisy aggregate knob never weakens the untested-new-code check To keep tolerated dips from compounding into an unbounded downward ratchet, the baseline writers restore any within-tolerance dip to the prior high-water value when recording; only a beyond-tolerance drop (FAIL-visible on the PR that introduced it) resets the baseline. Load() rejects negative/NaN/Inf tolerances, which would otherwise silently invert or disable the gates. Claude-Session: https://claude.ai/code/session_01QMTWVM4pUB4xy6F6P9wWYA --- AGENTS.md | 24 ++++++-- README.md | 2 + cmd/bulwark/coverage.go | 74 +++++++++++++++++++----- cmd/bulwark/coverage_test.go | 100 ++++++++++++++++++++++++++++++--- internal/config/config.go | 52 +++++++++++++++-- internal/config/config_test.go | 54 ++++++++++++++++++ 6 files changed, 276 insertions(+), 30 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 195ca5a..cb78561 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -76,8 +76,8 @@ exact `1.26.5`. If `go.mod`'s `toolchain` line is ever bumped, update every `go- ## Configuration `.bulwark.yml` at the scan root is optional and purely **opt-out** — its job is narrowing what -bulwark's zero-config default already does (scan everything detected, every check enabled), not -tuning severity or suppressing individual findings (that's what a fix-up pass + inline +bulwark's zero-config default already does (scan everything detected, every check enabled), plus +one numeric knob (`coverage.tolerance`), not tuning severity or suppressing individual findings (that's what a fix-up pass + inline `#nosec`/`nosemgrep` annotations in the scanned repo are for). See `internal/config/config.go` for the full schema; shape: @@ -96,6 +96,15 @@ go: semgrep: enabled: true config: auto # override to a custom registry ref/path if needed +coverage: + tolerance: 0.1 # pp a language's aggregate coverage may dip below its baseline before + # the gate fails; absorbs sub-tenth measurement noise ("86.1% vs + # baseline 86.1%, regressed 0.0%"). Compared at display precision + # (tenths); 0 = fail any dip the report can show. Must be finite and + # non-negative — Load rejects anything else. + patch: + tolerance: 0.1 # the patch gate's own dip allowance — deliberately independent, so + # loosening the aggregate knob never weakens the untested-new-code check ``` Omitting the file, or omitting a section/key within it, keeps that value at its default — see @@ -187,7 +196,12 @@ generated cache data, not source, needs no PR/review and never pollutes main's h `warnUnmeasured` names every detected-but-unmeasured ecosystem on stderr rather than dropping it in silence. If a language is missing from the gate, bulwark now says so. - A language with no prior baseline entry (new) is reported but doesn't fail the check on its own; - a language whose current coverage is below its baseline does. A language the baseline has but the + a language whose current coverage dipped below its baseline by more than `coverage.tolerance` + (default 0.1pp, compared at the report's tenth-of-a-point display precision) does. To keep + tolerated dips from compounding — each merge lowering the baseline the next PR gates against — + the baseline writers restore any within-tolerance dip to the prior (high-water) value when + recording; only a beyond-tolerance drop, which was FAIL-visible on the PR that introduced it, + resets the baseline. A language the baseline has but the current run doesn't splits on detection: still detected in the tree means its coverage step just didn't run this time (path-filtered CI job, missing tooling) and it's reported as `[UNMEASURED]`; no longer detected means the source actually left the tree and it's reported as `[DROPPED]` @@ -244,7 +258,9 @@ bounds the other, so `bulwark coverage` computes and gates on both, not either/o is a second, independent check alongside `diffReport`'s existing aggregate gate, not a replacement. Patch coverage has **no baseline or threshold of its own** — it always gates against that same -language's aggregate baseline already read from `bulwark-state` (`patch% >= baseline%`). A language +language's aggregate baseline already read from `bulwark-state` (`patch% >= baseline% - +coverage.patch.tolerance` — its own knob, deliberately independent of `coverage.tolerance`, so +loosening the noisy aggregate gate never silently weakens this one). A language with no aggregate baseline yet is reported informationally (`[NEW]`), not failed, mirroring aggregate's own handling of a first-time-seen language. It's opt-out, not opt-in, per language, via `.bulwark.yml`: diff --git a/README.md b/README.md index 0c9b2e2..3a5e738 100644 --- a/README.md +++ b/README.md @@ -84,6 +84,8 @@ go: semgrep: enabled: true config: auto +coverage: + tolerance: 0.1 # pp the aggregate gate tolerates below baseline (patch gate: coverage.patch.tolerance) ``` See [AGENTS.md](AGENTS.md#configuration) for the full schema and merge semantics. diff --git a/cmd/bulwark/coverage.go b/cmd/bulwark/coverage.go index 6cb782e..82bd8c9 100644 --- a/cmd/bulwark/coverage.go +++ b/cmd/bulwark/coverage.go @@ -5,6 +5,7 @@ import ( "errors" "fmt" "maps" + "math" "os" "path/filepath" "sort" @@ -20,7 +21,8 @@ import ( ) // priorBaselineDepth bounds how far back the baseline writers look for a -// prior baseline to carry a detected-but-unmeasured language forward from. +// prior baseline — to carry a detected-but-unmeasured language forward from, +// and to anchor within-tolerance dips to (withToleratedDipsRestored). // With the carry-forward rule applied on every recorded baseline, the nearest // one already contains everything worth carrying, so this only needs to span // gaps (main commits whose CI never recorded), not real history. @@ -105,7 +107,7 @@ func newCoverageCmd() *cobra.Command { // (wardnet/wardnet#899). head, headErr := gitstate.HeadSHA(ctx, dir) if shaErr == nil && headErr == nil && head == sha { - record, carried, err := carryForwardBaseline(ctx, cmd, dir, sha, ecosystems, current) + record, carried, err := carryForwardBaseline(ctx, cmd, dir, sha, ecosystems, current, cfg.Coverage.Tolerance) if err != nil { return err } @@ -173,12 +175,12 @@ func newCoverageCmd() *cobra.Command { } } - aggregateErr := diffReport(cmd, current, baseline, ecosystems) + aggregateErr := diffReport(cmd, current, baseline, cfg.Coverage.Tolerance, ecosystems) // sha is the same merge-base gitstate.BaseSHA already resolved for // the aggregate baseline lookup above — patch coverage reuses it // directly rather than recomputing "git merge-base HEAD origin/main" // a second time. - patchErr := patchReport(cmd, ctx, dir, patchWanted, sources, sha, baseline) + patchErr := patchReport(cmd, ctx, dir, patchWanted, sources, sha, baseline, cfg.Coverage.Patch.Tolerance) // errors.Join keeps both messages when aggregate AND patch coverage // regress in the same run — AGENTS.md's documented "compute and gate // on both, not either/or" contract must hold for the returned error @@ -276,7 +278,7 @@ func computeBaselineAt(ctx context.Context, cmd *cobra.Command, dir, sha string, // has), and a partial baseline cached here poisons every later PR against // this SHA just as thoroughly as a partial record-on-main would — so it // gets the same carry-forward, not just the record-on-main path. - record, _, err := carryForwardBaseline(ctx, cmd, dir, sha, ecosystems, report) + record, _, err := carryForwardBaseline(ctx, cmd, dir, sha, ecosystems, report, cfg.Coverage.Tolerance) if err != nil { return nil, err } @@ -325,6 +327,18 @@ func statusPrefix(tag string) string { return bracket + strings.Repeat(" ", pad) } +// regressedBeyond reports whether cur dipped below base by more than +// tolerance percentage points, comparing at the report's display precision +// (tenths) so what's shown and what's gated always agree: a raw float64 +// subtraction decides an exactly-at-tolerance dip by representation noise +// (86.2-86.1 exceeds 0.1 but 86.1-86.0 does not), failing one "regressed +// 0.1%" while passing an identical-looking other. Shared by diffReport and +// patchReport so the two gates can't drift apart on the comparison, like +// statusPrefix does for formatting. +func regressedBeyond(cur, base, tolerance float64) bool { + return math.Round((base-cur)*10) > math.Round(tolerance*10) +} + // patchReport prints one bracketed status line per language whose patch // coverage was requested (cfg.Coverage.Patch..Enabled) and whose // source Compute managed to resolve, gating patch% against that language's @@ -338,7 +352,7 @@ func statusPrefix(tag string) string { // ChangedLines is called exactly once, for the union of every wanted // language's extensions, then partitioned per language — not once per // language — since all three langs diff the identical mergeBase..HEAD range. -func patchReport(cmd *cobra.Command, ctx context.Context, dir string, want coverage.PatchWanted, sources coverage.PatchSources, mergeBase string, baseline map[string]float64) error { +func patchReport(cmd *cobra.Command, ctx context.Context, dir string, want coverage.PatchWanted, sources coverage.PatchSources, mergeBase string, baseline map[string]float64, tolerance float64) error { type language struct { name string wanted bool @@ -403,7 +417,10 @@ func patchReport(cmd *cobra.Command, ctx context.Context, dir string, want cover switch { case !baseOK: tag, detail = "NEW", fmt.Sprintf("%s patch: %.1f%% (%d/%d new lines; no baseline yet)", lang.name, pct, hit, total) - case pct < base: + // The patch gate has its own tolerance knob + // (coverage.patch.tolerance) — deliberately not coverage.tolerance, + // so loosening the noisy aggregate gate never weakens this one. + case regressedBeyond(pct, base, tolerance): regressed++ tag, detail = "FAIL", fmt.Sprintf("%s patch: %.1f%% (%d/%d new lines; baseline %.1f%%)", lang.name, pct, hit, total, base) default: @@ -568,8 +585,9 @@ func rustPatchPercent(dir string, rustLCOV map[string]string, changed map[string // one the baseline has but this run didn't measure is [UNMEASURED] while its // source is still detected (its coverage step just didn't run) and [DROPPED] // once it isn't (the source actually left the tree). None of those fail the -// check on its own — only a measured decrease does. -func diffReport(cmd *cobra.Command, current, baseline map[string]float64, detected []detect.Ecosystem) error { +// check on its own — only a measured decrease beyond the configured noise +// tolerance does (see regressedBeyond). +func diffReport(cmd *cobra.Command, current, baseline map[string]float64, tolerance float64, detected []detect.Ecosystem) error { detectedSet := make(map[string]bool, len(detected)) for _, e := range detected { detectedSet[string(e)] = true @@ -603,7 +621,9 @@ func diffReport(cmd *cobra.Command, current, baseline map[string]float64, detect tag, detail = "UNMEASURED", fmt.Sprintf("%s: not measured this run (baseline %.1f%%)", lang, base) case !curOK: tag, detail = "DROPPED", fmt.Sprintf("%s: no longer measured (baseline was %.1f%%)", lang, base) - case cur < base: + // Dips within the configured noise tolerance don't count — see + // config.Coverage.Tolerance for the rationale. + case regressedBeyond(cur, base, tolerance): regressed++ tag, detail = "FAIL", fmt.Sprintf("%s: %.1f%% (baseline %.1f%%, regressed %.1f%%)", lang, cur, base, base-cur) default: @@ -701,13 +721,18 @@ func mergeCarried(current map[string]float64, unmeasured []string, prior map[str // gate. An undetected language is never carried — its source left the tree // (or it was disabled in .bulwark.yml), so its entry dies with it. Anything // that stayed unfilled is named on stderr rather than dropped in silence. -func carryForwardBaseline(ctx context.Context, cmd *cobra.Command, dir, sha string, detected []detect.Ecosystem, current map[string]float64) (map[string]float64, []string, error) { +func carryForwardBaseline(ctx context.Context, cmd *cobra.Command, dir, sha string, detected []detect.Ecosystem, current map[string]float64, tolerance float64) (map[string]float64, []string, error) { unmeasured := unmeasuredLanguages(detected, current) - if len(unmeasured) == 0 { - return current, nil, nil + // Priors are fetched for every detected language, not just unmeasured + // ones: measured values need them too, for the anti-ratchet restore + // below. + langs := make([]string, 0, len(detected)) + for _, e := range detected { + langs = append(langs, string(e)) } - prior := gitstate.PriorBaselines(ctx, dir, sha, unmeasured, priorBaselineDepth) + prior := gitstate.PriorBaselines(ctx, dir, sha, langs, priorBaselineDepth) record, carried, missing := mergeCarried(current, unmeasured, prior) + record = withToleratedDipsRestored(record, prior, tolerance) for _, lang := range missing { if _, err := fmt.Fprintf(cmd.ErrOrStderr(), "warning: %s is detected but was not measured this run, and no prior baseline entry was found to carry forward — recording a baseline without it (if this repo has prior baselines, is the checkout shallow? fetch-depth: 0 lets bulwark walk prior main commits)\n", lang); err != nil { return nil, nil, err @@ -715,3 +740,24 @@ func carryForwardBaseline(ctx context.Context, cmd *cobra.Command, dir, sha stri } return record, carried, nil } + +// withToleratedDipsRestored returns record with every measured value that +// dipped below its prior baseline by no more than the gate's tolerance +// restored to the prior value. Recording the dipped number verbatim would +// turn the tolerance into an unbounded downward ratchet: each merged PR may +// dip up to the tolerance and pass, the main run records the lower number as +// the new baseline, and the next PR gets another free dip from that lower +// floor — coverage bleeds one tolerance per merge with every gate green. +// Restoring within-tolerance dips anchors the baseline to its high-water +// mark, capping total tolerated drift at one tolerance. A dip beyond the +// tolerance is recorded as measured: it was FAIL-visible on the PR that +// introduced it, so accepting it on main is a deliberate reset, not leakage. +func withToleratedDipsRestored(record, prior map[string]float64, tolerance float64) map[string]float64 { + out := maps.Clone(record) + for lang, val := range out { + if p, ok := prior[lang]; ok && val < p && !regressedBeyond(val, p, tolerance) { + out[lang] = p + } + } + return out +} diff --git a/cmd/bulwark/coverage_test.go b/cmd/bulwark/coverage_test.go index d23fbe9..3c52903 100644 --- a/cmd/bulwark/coverage_test.go +++ b/cmd/bulwark/coverage_test.go @@ -160,17 +160,17 @@ func TestEnabledEcosystemsDropsDisabledLanguages(t *testing.T) { } } -func runDiffReport(t *testing.T, current, baseline map[string]float64, detected ...detect.Ecosystem) (string, error) { +func runDiffReport(t *testing.T, tolerance float64, current, baseline map[string]float64, detected ...detect.Ecosystem) (string, error) { t.Helper() cmd := &cobra.Command{} var buf bytes.Buffer cmd.SetOut(&buf) - err := diffReport(cmd, current, baseline, detected) + err := diffReport(cmd, current, baseline, tolerance, detected) return buf.String(), err } func TestDiffReportNewLanguage(t *testing.T) { - out, err := runDiffReport(t, map[string]float64{"go": 50}, map[string]float64{}) + out, err := runDiffReport(t, 0, map[string]float64{"go": 50}, map[string]float64{}) if err != nil { t.Fatalf("unexpected error: %v", err) } @@ -185,7 +185,7 @@ func TestDiffReportNewLanguage(t *testing.T) { // reported, not silently omitted — regression for the bug where diffReport // only iterated current's keys and such a language was never mentioned at all. func TestDiffReportDroppedLanguage(t *testing.T) { - out, err := runDiffReport(t, map[string]float64{}, map[string]float64{"typescript": 85}) + out, err := runDiffReport(t, 0, map[string]float64{}, map[string]float64{"typescript": 85}) if err != nil { t.Fatalf("a dropped language must not fail the check on its own, got error: %v", err) } @@ -200,7 +200,7 @@ func TestDiffReportDroppedLanguage(t *testing.T) { // reported as [UNMEASURED], reserving [DROPPED] for a language whose source // actually left the tree. Neither fails the check on its own. func TestDiffReportUnmeasuredWhenLanguageStillDetected(t *testing.T) { - out, err := runDiffReport(t, map[string]float64{"rust": 86}, + out, err := runDiffReport(t, 0, map[string]float64{"rust": 86}, map[string]float64{"rust": 85.8, "typescript": 93.8}, detect.Rust, detect.TypeScript) if err != nil { @@ -215,7 +215,7 @@ func TestDiffReportUnmeasuredWhenLanguageStillDetected(t *testing.T) { } func TestDiffReportRegression(t *testing.T) { - out, err := runDiffReport(t, map[string]float64{"go": 40}, map[string]float64{"go": 50}) + out, err := runDiffReport(t, 0, map[string]float64{"go": 40}, map[string]float64{"go": 50}) if err == nil { t.Fatal("expected an error for a regressed language") } @@ -224,8 +224,92 @@ func TestDiffReportRegression(t *testing.T) { } } +// A dip smaller than the configured tolerance is measurement noise, not a +// regression — regression test for the gate failing PRs with "rust: 86.1% +// (baseline 86.1%, regressed 0.0%)", a sub-rounding-error dip, on PRs +// containing no Rust changes at all. +func TestDiffReportPassesWithinTolerance(t *testing.T) { + out, err := runDiffReport(t, 0.1, map[string]float64{"rust": 86.05}, map[string]float64{"rust": 86.1}) + if err != nil { + t.Fatalf("a dip within the tolerance must not fail the gate, got error: %v", err) + } + if !strings.Contains(out, "[PASS]") { + t.Fatalf("expected a [PASS] line for a within-tolerance dip, got: %q", out) + } +} + +// The tolerance only absorbs noise — a dip larger than it is still a real +// regression and must fail. +func TestDiffReportFailsBeyondTolerance(t *testing.T) { + out, err := runDiffReport(t, 0.1, map[string]float64{"rust": 85.6}, map[string]float64{"rust": 86.1}) + if err == nil { + t.Fatal("expected an error for a dip beyond the tolerance") + } + if !strings.Contains(out, "[FAIL]") { + t.Fatalf("expected a [FAIL] line for a beyond-tolerance dip, got: %q", out) + } +} + +// An exactly-at-tolerance dip must gate identically regardless of the +// operands' float64 representation: 86.2-86.1 exceeds 0.1 in raw float math +// while 86.1-86.0 does not, so a raw comparison failed one "regressed 0.1%" +// and passed an identical-looking other. The gate compares at display +// precision (regressedBeyond) so both pass. +func TestDiffReportToleranceBoundaryIsRepresentationIndependent(t *testing.T) { + for _, pair := range [][2]float64{{86.1, 86.2}, {86.0, 86.1}, {50.0, 50.1}} { + out, err := runDiffReport(t, 0.1, map[string]float64{"go": pair[0]}, map[string]float64{"go": pair[1]}) + if err != nil { + t.Fatalf("a 0.1pp dip with tolerance 0.1 must pass for %v vs %v, got error: %v (out: %q)", pair[0], pair[1], err, out) + } + } +} + +func TestRegressedBeyond(t *testing.T) { + cases := []struct { + name string + cur, base, tol float64 + want bool + }{ + {"improvement never regresses", 86.2, 86.1, 0, false}, + {"equal never regresses", 86.1, 86.1, 0, false}, + {"sub-display dip with zero tolerance shows 0.0%, passes", 86.06, 86.1, 0, false}, + {"visible dip with zero tolerance fails", 86.0, 86.1, 0, true}, + {"dip equal to tolerance passes (either representation)", 86.1, 86.2, 0.1, false}, + {"dip equal to tolerance passes (other representation)", 86.0, 86.1, 0.1, false}, + {"dip beyond tolerance fails", 85.9, 86.1, 0.1, true}, + } + for _, c := range cases { + if got := regressedBeyond(c.cur, c.base, c.tol); got != c.want { + t.Errorf("%s: regressedBeyond(%v, %v, %v) = %v, want %v", c.name, c.cur, c.base, c.tol, got, c.want) + } + } +} + +// Recording a within-tolerance dip verbatim would let tolerated dips +// compound: each merge lowers the baseline by up to the tolerance and the +// next PR dips again from the lower floor. The baseline writers restore such +// dips to the prior (high-water) value; only a beyond-tolerance drop — which +// was FAIL-visible on the PR that introduced it — resets the baseline. +func TestWithToleratedDipsRestored(t *testing.T) { + record := map[string]float64{"go": 86.01, "rust": 70.0, "typescript": 90.0, "kotlin": 55.5} + prior := map[string]float64{"go": 86.1, "rust": 71.0, "typescript": 89.5} + got := withToleratedDipsRestored(record, prior, 0.1) + want := map[string]float64{ + "go": 86.1, // dipped 0.09, within tolerance → restored + "rust": 70.0, // dipped 1.0, beyond tolerance → deliberate reset, kept + "typescript": 90.0, // improved → kept (baseline ratchets up) + "kotlin": 55.5, // no prior → kept + } + if !reflect.DeepEqual(got, want) { + t.Fatalf("withToleratedDipsRestored = %v, want %v", got, want) + } + if record["go"] != 86.01 { + t.Fatal("input map must not be mutated") + } +} + func TestDiffReportPass(t *testing.T) { - out, err := runDiffReport(t, map[string]float64{"go": 55}, map[string]float64{"go": 50}) + out, err := runDiffReport(t, 0, map[string]float64{"go": 55}, map[string]float64{"go": 50}) if err != nil { t.Fatalf("unexpected error: %v", err) } @@ -240,7 +324,7 @@ func TestDiffReportPass(t *testing.T) { func TestDiffReportOnlyCountsRealRegressions(t *testing.T) { current := map[string]float64{"go": 40, "rust": 60} baseline := map[string]float64{"go": 50, "typescript": 85} - out, err := runDiffReport(t, current, baseline) + out, err := runDiffReport(t, 0, current, baseline) if err == nil || !strings.Contains(err.Error(), "1") { t.Fatalf("expected exactly 1 regressed language reported in the error, got: %v", err) } diff --git a/internal/config/config.go b/internal/config/config.go index 10a3f04..78e313a 100644 --- a/internal/config/config.go +++ b/internal/config/config.go @@ -2,14 +2,15 @@ // bulwark's default (no file present) is to scan everything it detects with // every check enabled. The file can only narrow that — disable a language's // checks entirely, exclude specific paths from ecosystem/package detection, -// or override Semgrep's ruleset — not tune severity or suppress individual -// findings (that's what a fix-up pass + #nosec/nosemgrep annotations in the -// scanned repo itself are for). +// override Semgrep's ruleset, or adjust the coverage gates' noise tolerance +// — not tune severity or suppress individual findings (that's what a fix-up +// pass + #nosec/nosemgrep annotations in the scanned repo itself are for). package config import ( "errors" "fmt" + "math" "os" "path/filepath" @@ -54,16 +55,37 @@ type PatchLanguage struct { // PatchCoverage is the opt-out surface for the patch-coverage gate, per // language. Patch coverage has no threshold of its own — it always gates -// against that language's existing aggregate baseline (patch% >= baseline%). +// against that language's existing aggregate baseline (patch% >= +// baseline% - tolerance). type PatchCoverage struct { Rust PatchLanguage `yaml:"rust"` TypeScript PatchLanguage `yaml:"typescript"` Go PatchLanguage `yaml:"go"` + // Tolerance is the patch gate's own dip allowance in percentage points, + // deliberately independent of Coverage.Tolerance: patch% is an exact + // hit/total ratio with no measurement noise, but the baseline it gates + // against is a noisy aggregate — hence a small default of its own. + // Keeping the knobs separate means raising the aggregate tolerance for a + // noisy test suite never silently weakens the untested-new-code check. + Tolerance float64 `yaml:"tolerance"` } // Coverage is the opt-out/override surface for coverage gating. type Coverage struct { Patch PatchCoverage `yaml:"patch"` + // Tolerance is the number of percentage points a language's aggregate + // coverage may dip below its baseline before the gate fails. Coverage + // measurement is noisy at the sub-tenth level (timing-dependent + // instrumentation, tool version drift), so a strict comparison fails PRs + // with "86.1% vs baseline 86.1%, regressed 0.0%" — a dip smaller than + // the displayed precision — even when the PR touches no code in that + // language. The comparison happens at the report's display precision + // (tenths), so 0 means "fail any dip the report can show" rather than + // "fail any bit-level difference". Within-tolerance dips are also + // restored to the prior value when a main run records a new baseline, so + // tolerated dips can't compound across merges. The patch gate has its + // own knob (Patch.Tolerance). + Tolerance float64 `yaml:"tolerance"` } // Config is bulwark's full, resolved configuration for one scan. @@ -85,10 +107,12 @@ func Default() Config { Go: Language{Enabled: true}, Semgrep: Semgrep{Enabled: true, Config: "auto"}, Coverage: Coverage{ + Tolerance: 0.1, Patch: PatchCoverage{ Rust: PatchLanguage{Enabled: true}, TypeScript: PatchLanguage{Enabled: true}, Go: PatchLanguage{Enabled: true}, + Tolerance: 0.1, }, }, } @@ -112,9 +136,29 @@ func Load(root string) (Config, error) { if err := yaml.Unmarshal(data, &cfg); err != nil { return Config{}, fmt.Errorf("parsing %s: %w", path, err) } + if err := validateTolerances(cfg); err != nil { + return Config{}, fmt.Errorf("%s: %w", path, err) + } return cfg, nil } +// validateTolerances rejects tolerance values that silently invert or +// disable the coverage gates: a negative tolerance fails languages whose +// coverage held steady or improved ("regressed -0.3%"), and NaN (or ±Inf, +// both valid YAML) makes the gate comparison unconditionally false, turning +// both gates off while still printing [PASS] lines. +func validateTolerances(cfg Config) error { + for name, tol := range map[string]float64{ + "coverage.tolerance": cfg.Coverage.Tolerance, + "coverage.patch.tolerance": cfg.Coverage.Patch.Tolerance, + } { + if math.IsNaN(tol) || math.IsInf(tol, 0) || tol < 0 { + return fmt.Errorf("%s must be a finite, non-negative number of percentage points, got %v", name, tol) + } + } + return nil +} + // AllExcludes merges every language's exclude list — used by callers (scan, // coverage) whose initial ecosystem-detection pass doesn't yet know which // language a given excluded directory belongs to. diff --git a/internal/config/config_test.go b/internal/config/config_test.go index c4e69f5..74ac39f 100644 --- a/internal/config/config_test.go +++ b/internal/config/config_test.go @@ -4,6 +4,7 @@ import ( "os" "path/filepath" "reflect" + "strings" "testing" ) @@ -68,6 +69,59 @@ func TestLoadSemgrepConfigOverride(t *testing.T) { } } +// Zero-config users get a small noise-absorbing tolerance on the coverage +// gates, so a sub-rounding-error dip (86.1% vs baseline 86.1%) doesn't fail +// unrelated PRs. The aggregate and patch knobs default independently. +func TestDefaultCoverageTolerance(t *testing.T) { + if got := Default().Coverage.Tolerance; got != 0.1 { + t.Fatalf("Coverage.Tolerance default = %v, want 0.1", got) + } + if got := Default().Coverage.Patch.Tolerance; got != 0.1 { + t.Fatalf("Coverage.Patch.Tolerance default = %v, want 0.1", got) + } +} + +// Tolerances that would invert the gate (negative) or silently disable it +// (NaN, ±Inf are all valid YAML floats) must be rejected at load time with +// an error naming the key, not flow into the comparison. +func TestLoadRejectsInvalidTolerance(t *testing.T) { + cases := map[string]string{ + "negative aggregate": "coverage:\n tolerance: -0.1\n", + "nan aggregate": "coverage:\n tolerance: .nan\n", + "inf aggregate": "coverage:\n tolerance: .inf\n", + "negative patch": "coverage:\n patch:\n tolerance: -1\n", + "nan patch": "coverage:\n patch:\n tolerance: .nan\n", + } + for name, yml := range cases { + dir := t.TempDir() + write(t, dir, yml) + if _, err := Load(dir); err == nil { + t.Errorf("%s: Load accepted an invalid tolerance (%q), want an error", name, yml) + } else if !strings.Contains(err.Error(), "tolerance") { + t.Errorf("%s: error should name the offending key, got: %v", name, err) + } + } +} + +// An explicit tolerance: 0 tightens the gate to "fail any dip the report can +// display" — the merge must honor an explicitly-present zero, not treat it +// as "keep the default". +func TestLoadCoverageToleranceExplicitZero(t *testing.T) { + dir := t.TempDir() + write(t, dir, "coverage:\n tolerance: 0\n") + + got, err := Load(dir) + if err != nil { + t.Fatalf("Load: %v", err) + } + if got.Coverage.Tolerance != 0 { + t.Fatalf("Coverage.Tolerance = %v, want 0 after explicit override", got.Coverage.Tolerance) + } + if !got.Coverage.Patch.Go.Enabled { + t.Fatalf("setting coverage.tolerance incorrectly disabled patch coverage: %+v", got.Coverage) + } +} + func TestLoadPatchCoverageDefaultsEnabled(t *testing.T) { got, err := Load(t.TempDir()) if err != nil { From 2cc292c90db8fd2d84d21893493f2001e0347efa Mon Sep 17 00:00:00 2001 From: Pedro Gomes Date: Wed, 15 Jul 2026 04:17:17 +0100 Subject: [PATCH 3/3] chore: track Serena project config (.serena, minus its own gitignored paths) Claude-Session: https://claude.ai/code/session_01QMTWVM4pUB4xy6F6P9wWYA --- .serena/.gitignore | 2 + .serena/project.yml | 168 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 170 insertions(+) create mode 100644 .serena/.gitignore create mode 100644 .serena/project.yml diff --git a/.serena/.gitignore b/.serena/.gitignore new file mode 100644 index 0000000..2e510af --- /dev/null +++ b/.serena/.gitignore @@ -0,0 +1,2 @@ +/cache +/project.local.yml diff --git a/.serena/project.yml b/.serena/project.yml new file mode 100644 index 0000000..02970f5 --- /dev/null +++ b/.serena/project.yml @@ -0,0 +1,168 @@ +# the name by which the project can be referenced within Serena +project_name: "vulture-inselberg" + + +# list of languages for which language servers are started; choose from: +# al angular ansible bash clojure +# cpp cpp_ccls crystal csharp csharp_omnisharp +# dart elixir elm erlang fortran +# fsharp go groovy haskell haxe +# hlsl html java json julia +# kotlin lean4 lua luau markdown +# matlab msl nix ocaml pascal +# perl php php_phpactor powershell python +# python_jedi python_ty r rego ruby +# ruby_solargraph rust scala scss solidity +# svelte swift systemverilog terraform toml +# typescript typescript_vts vue yaml zig +# (This list may be outdated. For the current list, see values of Language enum here: +# https://github.com/oraios/serena/blob/main/src/solidlsp/ls_config.py +# For some languages, there are alternative language servers, e.g. csharp_omnisharp, ruby_solargraph.) +# Note: +# - For C, use cpp +# - For JavaScript, use typescript +# - For Angular projects, use angular (subsumes typescript+html; requires `npm install` in the project root) +# - For Svelte projects, use svelte (subsumes typescript/javascript for .svelte projects; requires npm) +# - For SCSS / Sass / plain CSS, use scss (some-sass-language-server handles all three) +# - For Free Pascal/Lazarus, use pascal +# Special requirements: +# Some languages require additional setup/installations. +# See here for details: https://oraios.github.io/serena/01-about/020_programming-languages.html#language-servers +# When using multiple languages, the first language server that supports a given file will be used for that file. +# The first language is the default language and the respective language server will be used as a fallback. +# Note that when using the JetBrains backend, language servers are not used and this list is correspondingly ignored. +languages: +- go + +# the encoding used by text files in the project +# For a list of possible encodings, see https://docs.python.org/3.11/library/codecs.html#standard-encodings +encoding: "utf-8" + +# optional shell command to run before the language backend (LSP or JetBrains) is initialised. +# the command runs in the project root directory and is only executed if the project is trusted +# (see trusted_project_path_patterns in the global configuration). +# serena waits for the command to exit: a non-zero exit code is logged as an error but does not +# abort activation. a per-project timeout (activation_command_timeout, default 180s) is the safety +# backstop for non-terminating commands; on expiry the process is killed and activation continues. +# example: activation_command: "npx nx run-many -t build" +activation_command: + +# maximum time in seconds to wait for activation_command to complete before killing it (default 180s). +# must be a positive number. +activation_command_timeout: 180 + +# line ending convention to use when writing source files. +# Possible values: unset (use global setting), "lf", "crlf", or "native" (platform default) +# This does not affect Serena's own files (e.g. memories and configuration files), which always use native line endings. +line_ending: + +# The language backend to use for this project. +# If not set, the global setting from serena_config.yml is used. +# Valid values: LSP, JetBrains +# Note: the backend is fixed at startup. If a project with a different backend +# is activated post-init, an error will be returned. +language_backend: + +# whether to use project's .gitignore files to ignore files +ignore_all_files_in_gitignore: true + +# advanced configuration option allowing to configure language server-specific options. +# Maps the language key to the options. +# Have a look at the docstring of the constructors of the LS implementations within solidlsp (e.g., for C# or PHP) to see which options are available. +# No documentation on options means no options are available. +ls_specific_settings: {} + +# list of workspace folder paths (LSP backend only). +# These folders will be used to build up Serena's symbol index. +# Paths must be within the project root and should thus be relative to the project root. +# Furthermore, the paths should not be filtered by ignore settings. +# Default setting: The entire project root folder (".") is considered. +# In (large) monorepos, this can be used to index only subfolders of the project root, e.g. +# ls_workspace_folders: +# - "./subproject1" +# - "./subproject2" +ls_workspace_folders: ["."] + +# list of additional workspace folder paths for cross-package reference support. +# Paths can be absolute or relative to the project root. +# Each folder is registered as an LSP workspace folder, enabling language servers to discover +# symbols and references across package boundaries, but these folders are not indexed by Serena, +# i.e. the respective symbols will not be found using Serena's symbol search tools. +# Example: +# additional_workspace_folders: +# - ../sibling-package +# - ../shared-lib +ls_additional_workspace_folders: [] + +# list of additional paths to ignore in this project. +# Same syntax as gitignore, so you can use * and **. +# Note: global ignored_paths from serena_config.yml are also applied additively. +ignored_paths: [] + +# whether the project is in read-only mode +# If set to true, all editing tools will be disabled and attempts to use them will result in an error +# Added on 2025-04-18 +read_only: false + +# list of tool names to exclude. +# This extends the existing exclusions (e.g. from the global configuration) +# Find the list of tools here: https://oraios.github.io/serena/01-about/035_tools.html +excluded_tools: [] + +# list of tools to include that would otherwise be disabled (particularly optional tools that are disabled by default). +# This extends the existing inclusions (e.g. from the global configuration). +# Find the list of tools here: https://oraios.github.io/serena/01-about/035_tools.html +included_optional_tools: [] + +# fixed set of tools to use as the base tool set (if non-empty), replacing Serena's default set of tools. +# This cannot be combined with non-empty excluded_tools or included_optional_tools. +# Find the list of tools here: https://oraios.github.io/serena/01-about/035_tools.html +fixed_tools: [] + +# list of mode names that are to be activated by default, overriding the setting in the global configuration. +# The full set of modes to be activated is base_modes (from global config) + default_modes + added_modes. +# If the setting is undefined/empty, the default_modes from the global configuration (serena_config.yml) apply. +# Otherwise, this overrides the setting from the global configuration (serena_config.yml). +# Therefore, you can set this to [] if you do not want the default modes defined in the global config to apply +# for this project. +# This setting can, in turn, be overridden by CLI parameters (--mode). +# See https://oraios.github.io/serena/02-usage/050_configuration.html#modes +default_modes: + +# list of mode names to be activated additionally for this project, e.g. ["query-projects"] +# The full set of modes to be activated is base_modes (from global config) + default_modes + added_modes. +# See https://oraios.github.io/serena/02-usage/050_configuration.html#modes +added_modes: + +# initial prompt for the project. It will always be given to the LLM upon activating the project +# (contrary to the memories, which are loaded on demand). +initial_prompt: "" + +# time budget (seconds) per tool call for the retrieval of additional symbol information +# such as docstrings or parameter information. +# This overrides the corresponding setting in the global configuration; see the documentation there. +# If null or missing, use the setting from the global configuration. +symbol_info_budget: + +# list of regex patterns which, when matched, mark a memory entry as read‑only. +# Extends the list from the global configuration, merging the two lists. +read_only_memory_patterns: [] + +# list of regex patterns for memories to completely ignore. +# Matching memories will not appear in list_memories or activate_project output +# and cannot be accessed via read_memory or write_memory. +# To access ignored memory files, use the read_file tool on the raw file path. +# Extends the list from the global configuration, merging the two lists. +# Example: ["_archive/.*", "_episodes/.*"] +ignored_memory_patterns: [] + +# list of additional workspace folder paths for cross-package reference support (e.g. in monorepos). +# Paths can be absolute or relative to the project root. +# Each folder is registered as an LSP workspace folder, enabling language servers to discover +# symbols and references across package boundaries. +# Currently supported for: TypeScript. +# Example: +# additional_workspace_folders: +# - ../sibling-package +# - ../shared-lib +additional_workspace_folders: []