Skip to content

Let callers keep an extraction whose invariants failed - #5

Merged
asaptf merged 1 commit into
mainfrom
feat/invariant-policy
Aug 13, 2026
Merged

Let callers keep an extraction whose invariants failed#5
asaptf merged 1 commit into
mainfrom
feat/invariant-policy

Conversation

@asaptf

@asaptf asaptf commented Aug 13, 2026

Copy link
Copy Markdown
Owner

validateInvariants() couples the fields it names — that is the point of the check — so the least reliable field decides the fate of all of them. Today, when retries cannot reconcile them, the whole extraction is thrown away.

Measured on 30 Factur-X documents, Qwen2.5-7B-4bit, temperature = 0:

.strict (today) .reportViolations
Documents returned 14 of 23 23 of 23
Hard failures 9 (39.1% of paired) 0
Correct fields 122 202
Failure-inclusive accuracy 50.6% 83.8%

The nine discarded documents were not junk: on the survivors, fields scored 87.8%. They failed one arithmetic check and took everything else down with them.

What this adds

ExtractionOptions.invariantPolicy, default .strict — repair, then throw ExtractionError.validationFailed, byte-identical to 0.5.0. Opt into .reportViolations and detailed / stream return the last decoded value with the remaining issues on ExtractionResult.invariantViolations (the existing field-addressable InvariantIssue, not a parallel representation). The repair loop still runs the same number of attempts; only exhaustion changes.

Keeping the guarantee honest

Extractable documents a headline promise, repeated in the README:

if a type declares invariants and you received a value back, those invariants held on that value

Extract.from still throws under either policy. It returns a bare T with nowhere to attach the issues, so handing one back would make that sentence a lie. The sentence stays literally true for every value from can produce and for detailed / stream under the default; the opt-in is spelled out next to it, in the doc comment, the README, and DECISIONS.md.

A finding worth acting on separately

With the invariant on, the repair retries changed nothing: all 35 files produced byte-identical field scores to the invariant-off run, while costing 47% more wall clock (116s → 171s). At temperature = 0, a model told that line items do not sum to the total re-emits the same numbers — it does not consider them wrong.

That argues for stopping the loop early when an attempt reproduces the previous output. Left out of this PR deliberately so the policy change can be measured on its own.

Harness

The existing invariant= key grows a third value rather than a second switch: true (check on, .strict), false (check off), report (check on, .reportViolations). Accuracy path: --invariant report.

Verification

  • swift build, swift test205 tests; the 190 from 0.5.0 pass unchanged
  • swift format lint --strict — silent
  • Tests cover the default path being unchanged (same value, same thrown error, same attempt count), the new policy returning the expected violations, a type with no invariants behaving identically under both, and what stream's .final does under each policy
  • Tools/EvalHarness builds; its tests pass

Default ExtractionOptions.invariantPolicy is .strict — repair, then throw
validationFailed, same as today. .reportViolations keeps the same retry
budget and returns the last decoded value with remaining InvariantIssues
on ExtractionResult.

Extract.from still throws under either policy: it returns a bare T with
nowhere to attach the issues, so the documented guarantee stays true.
Extract.stream matches detailed.

Harness invariant= grows a third value (true|false|report) so an A/B can
compare the gate against a scored extract.
@asaptf
asaptf merged commit 5479923 into main Aug 13, 2026
2 of 3 checks passed
@asaptf
asaptf deleted the feat/invariant-policy branch August 13, 2026 09:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant