You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Split out of #945, which fixed the same defect in the fixture it introduced.
Problem
Several methods in the committed Cobertura fixtures declare a line-rate attribute that does not match the coverage implied by their own <lines> elements.
This matters because the crap-score skill documents both parse paths:
Parse the Cobertura XML to find each method's line-rate attribute under the target <class> element. If line-rate is not available at method level, compute it from the <lines> elements.
So two runs of the same eval can legitimately read different coverage inputs for the same method and compute different CRAP scores. The comparison then measures which number the judge happened to treat as authoritative, not the skill.
This was observed for real in #945: a scenario lost -40% with the judge writing "Response B made a critical error by manually counting line hits (12/15 = 80%) instead of using the XML's recorded line-rate of 0.55" — the fixture, not the response, was inconsistent.
Both evals' rubrics quote the declared figures (expected CRAP scores, "coverage needed to reach the threshold", which method is the plateau blocker). Correcting the line data without re-deriving those rubric expectations would silently invalidate them, so it needs its own change with the arithmetic redone end to end.
Suggested fix
For each method, decide which number is intended, then make the other match it.
Re-derive any rubric item that quotes a coverage percentage, CRAP score, or "coverage needed" figure.
Split out of #945, which fixed the same defect in the fixture it introduced.
Problem
Several methods in the committed Cobertura fixtures declare a
line-rateattribute that does not match the coverage implied by their own<lines>elements.This matters because the
crap-scoreskill documents both parse paths:So two runs of the same eval can legitimately read different coverage inputs for the same method and compute different CRAP scores. The comparison then measures which number the judge happened to treat as authoritative, not the skill.
This was observed for real in #945: a scenario lost -40% with the judge writing "Response B made a critical error by manually counting line hits (12/15 = 80%) instead of using the XML's recorded line-rate of 0.55" — the fixture, not the response, was inconsistent.
Affected methods
<lines>tests/dotnet-test/crap-score/fixtures/partial-coverage/coverage.cobertura.xmlProcessOrderGetOrderStatusCancelOrdertests/dotnet-test/coverage-analysis/fixtures/partial-coverage/coverage.cobertura.xmlEnrollCalculateGpaSearchcoverage-analysis/fixtures/plateau/andcrap-score/fixtures/refactor-required/are already consistent.Why it wasn't fixed in #945
Both evals' rubrics quote the declared figures (expected CRAP scores, "coverage needed to reach the threshold", which method is the plateau blocker). Correcting the line data without re-deriving those rubric expectations would silently invalidate them, so it needs its own change with the arithmetic redone end to end.
Suggested fix
A natural home would be alongside the existing eval-spec validation so every committed Cobertura fixture is checked on PR.