fix(cli, api): round trip the overall extraction instruction in ixp packages - #455
Open
anachili wants to merge 1 commit into
Open
fix(cli, api): round trip the overall extraction instruction in ixp packages#455anachili wants to merge 1 commit into
anachili wants to merge 1 commit into
Conversation
…ackages An IXP project's overall extraction instruction is fed to the model as part of its prompt, so a project uploaded from a package was running a different prompt from the one the package was downloaded from, and scored differently in Measure as a result — with no indication anything had been dropped. The instruction lives on the `default` label group rather than on the dataset, and neither `LabelGroup` nor `UpdateDataset` modelled it, so it was lost when the package was written and never sent when it was read back. The label defs the group is created with do not cover it. Add it to both, and send it from `re package upload` alongside the model config it already restores. Only send it when the packaged project has one, so uploading an older package leaves the new project's own default in place rather than blanking it. Field group, field and field type instructions were already round tripped; this was the only level that was not.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
An IXP project's overall extraction instruction is fed to the model as part of its prompt, so a project uploaded from a package was running a different prompt from the one the package was downloaded from — and scored differently in Measure as a result, with nothing to indicate anything had been dropped.
This surfaced on a customer promoting projects from a Test to a Production tenant with
re package download/re package upload. The annotated data transferred exactly (same 10 documents, 120 annotations, 101 annotated values, field for field), but the project score fell from 91 to 79, concentrated on the identifier and name fields that most depend on the instruction to disambiguate them (InvoiceNumber 100 → 20, SupplierName 60 → 40, PolicyNumber 50 → 33, TaxAmount 100 → 67).Why it was missed: the instruction lives on the
defaultlabel group, not on the dataset, so it is not covered by the label defs the group is created with. NeitherLabelGroupnorUpdateDatasetmodelled it, so it was dropped when the package was written and never sent when it was read back. Field group, field and field type instructions were already round tripped — this was the only level that was not.It is also the last outstanding item of RE-7680, which named model config, the overall extraction instruction, and custom defined field types. Model config landed in #392 and field types in #394; this half never did.
What changed
LabelGroupgainsinstructions, sore package downloadcaptures it into the package'sdataset.json.UpdateDatasetgainsdefault_label_group_instructions, serialized as_default_label_group_instructions.re package uploadsends it for IXP projects alongside the model config it already restores.Only sent when the packaged project actually has an instruction, so uploading a package downloaded before this change leaves the new project's own default in place rather than blanking it. It is applied in the existing
update_datasetcall rather than a new request — every dataset always has adefaultgroup from creation, so it does not need to be sent after the group's label defs.Breaking (
reinfer-clientAPI):LabelGroupandUpdateDataseteach gain a field, so struct literals of them need an extra initializer.Test plan
UpdateDatasetserializes it under the right key while omitting it entirely when unset.cargo test -p reinfer-client— 41 passed.cargo clippy --workspace --all-targetsandcargo fmt --all --checkclean.label_groupsliteral in thetest_create_update_dataset_customintegration test.Not covered: the round trip against a live tenant. Worth a manual check that downloading a project with an instruction and uploading it into another tenant now reproduces the source's score.
Follow-ups, not in scope here
re update dataset— it is only set via package upload.rule_setonMoonFormFieldDef) are still dropped by packages.instructions=""when serializing field choices, so nothing the CLI sends would take effect.RE-7680