Skip to content

Eval harness: stop hiding hard extraction failures in accuracy - #3

Merged
asaptf merged 1 commit into
mainfrom
eval/report-hard-failures
Aug 10, 2026
Merged

Eval harness: stop hiding hard extraction failures in accuracy#3
asaptf merged 1 commit into
mainfrom
eval/report-hard-failures

Conversation

@asaptf

@asaptf asaptf commented Aug 10, 2026

Copy link
Copy Markdown
Owner

When extraction throws, the record gets fieldScores: [], and AccuracySummary.reduce only walks fieldScores — so a failed file contributes nothing to the denominator. It is not counted as a miss; it disappears.

Observed twice during real measurement work in this repo:

  • A run where every file failed printed Overall field accuracy | 0.0% (0/0) — which reads as "very low accuracy" but actually means "nothing was scored at all".
  • A run where half the files failed silently reported the accuracy of the surviving half.

Both mislead in the same direction: failure looks like success at a low score.

What changes

  1. Hard failures are counted and surfaced — a paired file (has ground truth, passed the pairing guard) whose extraction threw. Count and share in the summary, a dedicated markdown section listing the files and their errors, and hardFailure per record in the JSONL.
  2. Failure-inclusive accuracy added — every ground-truth-bearing field on a hard-failed file scores incorrect, since a caller who got an exception got no value.
  3. The published metrics are not redefined. overallAccuracy / presentAccuracy keep the successful-extractions-only definition used for the numbers in CHANGELOG.md for 0.3.0 and 0.4.0. Silently changing what those names mean would make the project's own history incomparable. Every row in the report is now explicitly labelled with which definition it uses.
  4. Compare reports show per-arm hard-failure counts and a failure-inclusive delta, so an A/B where one arm fails more often shows that rather than just an accuracy delta.

Behaviour change worth noting in review

ABCompareRunner's per-field delta previously mapped (aCorrect: true, bCorrect: nil) to 0 — a field the baseline got right and the other arm never produced was invisible in the per-file deltas. nil is now treated as incorrect, so that case is a visible −1. This changes existing compare output; it is the point of the PR rather than a side effect.

Headline case

Before — every file hard-fails:

| Overall field accuracy | 0.0% (0/0) |

After:

| Paired (passed pairing guard) | 3 |
| Successfully extracted (scored) | 0 |
| **Hard failures** (paired, extraction threw) | **3** (100.0% of paired) |
| Overall field accuracy (successful extractions only) | 0.0% (0/0) |
| **Failure-inclusive field accuracy** | **0.0%** (0/9) |

Verification

  • swift build, swift test (root) — 177 tests, no regressions
  • swift format lint --strict — silent
  • Tools/EvalHarness builds; new EvalHarnessTests target — 4 tests covering AccuracySummary.reduce for a mixed run and for a total wipeout
  • Mock accuracy run on fixtures exits 0 and reports hard failures explicitly

Harness-only; Sources/Extract is untouched.

Paired files whose extraction throws no longer vanish from the denominator.
Report hard-failure count/share in AccuracySummary, markdown, and JSONL, and
add a failure-inclusive accuracy that scores every GT field on those files as
incorrect. Keep overallAccuracy/presentAccuracy as successful-extractions-only
so CHANGELOG history stays comparable. A/B compare shows hard-failure counts
and treats a missing field score as incorrect (Δ −1, not silent 0).
@asaptf
asaptf merged commit 9ff35a9 into main Aug 10, 2026
2 of 3 checks passed
@asaptf
asaptf deleted the eval/report-hard-failures branch August 10, 2026 11:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant