Eval harness: arithmetic-invariant switch and per-file progress - #1
Merged
Conversation
EvalInvoice's sum(lineTotals)+tax≈grandTotal check rejects nearly every extraction on weak models, leaving accuracy at 0.0% (0/0) with no scored fields. Add invariant=true|false (default true) on RunConfig / compare specs and --invariant for accuracy so A/B can measure the extractor rather than the gate. Mode is always visible in report labels. Also emit per-file progress to stderr in AccuracyRunner (index, total, relative path, elapsed seconds) so long model runs are not mistaken for hangs. Reports stay metrics-only.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two eval-harness changes, both extracted from an experiment branch and re-authored clean for
main. Harness-only — nothing underSources/Extractchanges, so library behaviour and the published API are untouched.1. Run-level switch for the arithmetic invariant
EvalInvoice.validateInvariants()enforcessum(lineTotals) + tax ≈ grandTotalunconditionally. On a small local model that check rejects nearly every extraction — and a rejected extraction contributes no scored fields at all, so the report reads0.0% (0/0)rather than a low accuracy. Hard failures vanish from the denominator instead of counting as misses.That makes an A/B measure the gate rather than the extractor. Measured on 6 Factur-X documents with Qwen2.5-1.5B: 6/6 files ended in
validationFailed, and the accuracy report scored nothing at all.invariant=true|false(aliasesarithmeticInvariant,arithmetic-invariant)--invariant true|falseon the accuracy pathtrue— existing reports keep their present meaninginvariant=false note=arithmetic-invariant-off). A report whose numbers silently change meaning is worse than no report.2. Per-file progress on stderr
AccuracyRunnernow emits[arm i/n] relative/path (elapsed s)per file.Model-backed runs take minutes per file and print nothing until the end, which makes a slow run indistinguishable from a hung one. A corpus run in this repo was left going for seven hours before it became clear it was making progress but nowhere near finishing — with no way to tell which file it was on.
Progress goes to stderr only; the written reports stay metrics-only, since real corpora are private invoices.
Verification
swift build,swift test— 177 tests, no regressionsswift format lint --strict— silentTools/EvalHarnessbuilds--invariant trueand--invariant false: both exit 0, each report states its modeinvariant=: exits 0, both arm labels show the mode