Record guided generation as a measured negative about the engine - #2
Merged
Merged
Conversation
Four A/B readings on 30 Factur-X documents (-41.4, -44.3, -83.8, -67.2 pp) were all artifacts of distinct defects in AnyLanguageModel's constrained JSON generator, not properties of constrained decoding. Records the five defects, the fingerprints that distinguished artifact from result, and the design point that optional-in-Swift is not optional-in-JSON for constrained decoding. The technique itself remains unmeasured; saying otherwise would close a promising direction on a false basis.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Records the guided-generation experiment in
DECISIONS.mdunder measured negatives, and corrects the now-outdated optimism in Future bridge notes.Why the wording matters
The finding is that AnyLanguageModel 0.8.0's constrained JSON generator is unusable for extraction — not that constrained decoding hurts extraction. We never succeeded in measuring the technique. Recording the second sentence instead of the first would close a promising direction on a false basis, which is exactly the kind of mistake this file exists to prevent.
What it records
Four A/B readings on 30 Factur-X documents, each of which looked like a result and each of which was an artifact:
{}is schema-valid when nothing is required; 7B took it every time473.00→4.73e+31);null-vs-string branch biasIt also records the fingerprints that distinguished artifact from result, since those transfer to future measurements: a field at exactly −100 pp across all documents is never being emitted; a guided arm running faster than the unconstrained one is terminating early; numbers working while strings collapse points at branch selection rather than capability.
Two process notes are kept deliberately. The 7B run existed only as a guard against a small-model artifact (precedent: table injection, −11.4 pp on 1.5B vs +0.8 pp on 7B) and is what exposed the largest defect — one model alone would have produced a coherent and entirely false story. And one fix we specified ourselves is what made the empty object legal; the previous code was wrong, and its wrongness had been hiding the hole.
Carried over for any repeat attempt: optional-in-Swift is not optional-in-JSON for constrained decoding — every key required to appear, absence expressed as
null.Docs only; no code changes.