Let callers keep an extraction whose invariants failed - #5
Merged
Conversation
Default ExtractionOptions.invariantPolicy is .strict — repair, then throw validationFailed, same as today. .reportViolations keeps the same retry budget and returns the last decoded value with remaining InvariantIssues on ExtractionResult. Extract.from still throws under either policy: it returns a bare T with nowhere to attach the issues, so the documented guarantee stays true. Extract.stream matches detailed. Harness invariant= grows a third value (true|false|report) so an A/B can compare the gate against a scored extract.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
validateInvariants()couples the fields it names — that is the point of the check — so the least reliable field decides the fate of all of them. Today, when retries cannot reconcile them, the whole extraction is thrown away.Measured on 30 Factur-X documents, Qwen2.5-7B-4bit,
temperature = 0:.strict(today).reportViolationsThe nine discarded documents were not junk: on the survivors, fields scored 87.8%. They failed one arithmetic check and took everything else down with them.
What this adds
ExtractionOptions.invariantPolicy, default.strict— repair, then throwExtractionError.validationFailed, byte-identical to 0.5.0. Opt into.reportViolationsanddetailed/streamreturn the last decoded value with the remaining issues onExtractionResult.invariantViolations(the existing field-addressableInvariantIssue, not a parallel representation). The repair loop still runs the same number of attempts; only exhaustion changes.Keeping the guarantee honest
Extractabledocuments a headline promise, repeated in the README:Extract.fromstill throws under either policy. It returns a bareTwith nowhere to attach the issues, so handing one back would make that sentence a lie. The sentence stays literally true for every valuefromcan produce and fordetailed/streamunder the default; the opt-in is spelled out next to it, in the doc comment, the README, andDECISIONS.md.A finding worth acting on separately
With the invariant on, the repair retries changed nothing: all 35 files produced byte-identical field scores to the invariant-off run, while costing 47% more wall clock (116s → 171s). At
temperature = 0, a model told that line items do not sum to the total re-emits the same numbers — it does not consider them wrong.That argues for stopping the loop early when an attempt reproduces the previous output. Left out of this PR deliberately so the policy change can be measured on its own.
Harness
The existing
invariant=key grows a third value rather than a second switch:true(check on,.strict),false(check off),report(check on,.reportViolations). Accuracy path:--invariant report.Verification
swift build,swift test— 205 tests; the 190 from 0.5.0 pass unchangedswift format lint --strict— silentstream's.finaldoes under each policyTools/EvalHarnessbuilds; its tests pass