Record the repair fixed-point stop as a measured negative - #6
Merged
Conversation
…tive Implemented and measured: the rule never fires. Identical field scores are not identical raw output — the model varies formatting and unscored fields between attempts while the scored values land the same, so a byte-identical fixed point never occurs. Zero files got faster on the same 30 documents. Names the inference error, and why the decoded-value variant is a heuristic we decline rather than a proof.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implemented the optimisation proposed after the
InvariantPolicymeasurement, measured it, and it does not work. Recording it so it is not retried blind.What was tried
Repair retries driven by the arithmetic invariant cost 47% wall clock (116s → 171s on 30 Factur-X documents, Qwen2.5-7B-4bit,
temperature = 0) while producing identical field scores to the invariant-off run. So: if a repair attempt reproduces the previous attempt's raw output verbatim at temperature 0, the next prompt would be identical, a deterministic model cannot differ, and the loop can stop. Provable, not heuristic.Result
It never fires. Zero files got faster on the same 30 documents — 169s → 167s, which is noise.
The mistake worth naming
The inference came from identical field scores, which are not identical raw output. The model varies formatting, ordering, and fields no metric scores, while the scored values land the same. A byte-identical fixed point does not occur, so a rule predicated on one is dead code.
Why it was not shipped anyway
The costs were real where the benefit was not: a source-breaking fourth associated value on
ExtractionError.validationFailed(payload-binding switches stop compiling), a changed repair prompt, and eleven tests to maintain. Paying an API break for no measured gain is a bad trade.Comparing decoded values instead of raw text would fire — but that is a heuristic rather than a proof, since the prompt genuinely differs between those attempts. This project spends its effort removing arbitrary decision rules, not adding them.
What transfers
The repair loop cannot fix arithmetic. The model re-derives the same numbers from the same document because it does not consider them wrong. The lever is not stopping more cleverly — it is not asking the model to do arithmetic at all. That is what
InvariantPolicy.reportViolationsprovides, and why line items reconstructed from detected table geometry are the more promising direction.Docs only.