Skip to content

Two Cobertura eval fixtures declare a line-rate their own line data contradicts #950

Description

@Evangelink

Split out of #945, which fixed the same defect in the fixture it introduced.

Problem

Several methods in the committed Cobertura fixtures declare a line-rate attribute that does not match the coverage implied by their own <lines> elements.

This matters because the crap-score skill documents both parse paths:

Parse the Cobertura XML to find each method's line-rate attribute under the target <class> element. If line-rate is not available at method level, compute it from the <lines> elements.

So two runs of the same eval can legitimately read different coverage inputs for the same method and compute different CRAP scores. The comparison then measures which number the judge happened to treat as authoritative, not the skill.

This was observed for real in #945: a scenario lost -40% with the judge writing "Response B made a critical error by manually counting line hits (12/15 = 80%) instead of using the XML's recorded line-rate of 0.55" — the fixture, not the response, was inconsistent.

Affected methods

Fixture Method Declared Implied by <lines>
tests/dotnet-test/crap-score/fixtures/partial-coverage/coverage.cobertura.xml ProcessOrder 0.45 0.70 (14/20)
GetOrderStatus 0.60 0.67 (6/9)
CancelOrder 0.80 1.00 (5/5)
tests/dotnet-test/coverage-analysis/fixtures/partial-coverage/coverage.cobertura.xml Enroll 0.75 0.88 (14/16)
CalculateGpa 0.25 0.12 (3/24)
Search 0.50 0.33 (2/6)

coverage-analysis/fixtures/plateau/ and crap-score/fixtures/refactor-required/ are already consistent.

Why it wasn't fixed in #945

Both evals' rubrics quote the declared figures (expected CRAP scores, "coverage needed to reach the threshold", which method is the plateau blocker). Correcting the line data without re-deriving those rubric expectations would silently invalidate them, so it needs its own change with the arithmetic redone end to end.

Suggested fix

  1. For each method, decide which number is intended, then make the other match it.
  2. Re-derive any rubric item that quotes a coverage percentage, CRAP score, or "coverage needed" figure.
  3. Add the invariant check to CI so it cannot regress. Fix dotnet-test findings from the refreshed cross-family eval (#899) #945 used this, which reports the table above:
covered = sum(1 for ln in method.iter("line") if int(ln.get("hits", "0")) > 0)
assert abs(covered / total - float(method.get("line-rate"))) < 0.02

A natural home would be alongside the existing eval-spec validation so every committed Cobertura fixture is checked on PR.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions