RQ-56-CITE (#911): an artifact cannot claim a test that does not exist - #927
Merged
Conversation
v0.55.0 SHIPPED two rivet artifacts citing `cargo test -- <filter>` for filters matching NOTHING. Such a filter exits 0, so both claimed verification by evidence that never ran, and no gate noticed — a post-release hygiene sweep did. `oracle_wiring_check.py` has enforced exactly this shape for scripts/repro/* since #890. The ARTIFACTS surface never got the equivalent, so the same class stayed expressible one layer over. This is that equivalent — the factory, not another instance. RED-FIRST: the check found the 2 dangling citations independently, 11 of 13 resolving over 4681 test names, so it is not over-flagging. STATUS-AWARE, and the data forced this: E2E-VER-010 is `proposed` and describes tests to be written — a forward-looking citation there is a PLAN, not a lie. Only claiming statuses (implemented/verified/accepted) fail. A gate that flagged both would be noisy, and a noisy gate gets ignored — which is precisely how codecov/patch stopped being read (#923). E2E-VER-009 fixed by correcting what was actually false: its STATUS. It claimed `implemented` for ElfBuilder unit tests (import_count, ImportEntry packing, string-table offsets, FNV hash) that do not exist. NOT re-pointed at `linker_script.rs`, which merely asserts the generated script CONTAINS the string ".meld_import_table" — citing that would restate the same lie more carefully. Test names come from a SOURCE SCAN, not `cargo test --list`: Claim Check does not build, and a gate needing a 20-minute build to answer a 200ms question does not survive. The approximation errs toward FALSE FAILURE (loud, fixable), never false pass. Wired into Claim Check with a non-vacuity assert on a non-empty population. Local: citation gate 0, claim 43/43, rivet ours-errors 0. Refs #911, RQ-56-CITE
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
First v0.56 lane. Closes #911. Theme: the factory, not the instances.
What shipped broken
v0.55.0 shipped two rivet artifacts citing
cargo test -- <filter>for filters matching nothing. Such a filter prints0 passed … 56 filtered outand exits 0 — so both claimed verification by evidence that never ran. No gate noticed; a post-release hygiene sweep did.oracle_wiring_check.pyhas enforced this exact shape forscripts/repro/*since #890. The artifacts surface never got the equivalent, so the class stayed expressible one layer over. This adds it.Red-first
The check found the two dangling citations independently — 11 of 13 resolve over 4,681 test names, so it isn't over-flagging.
Status-aware, because the data forced it
E2E-VER-010isstatus: proposedand describes tests to be written. A forward-looking citation there is a plan, not a lie. Only claiming statuses (implemented/verified/accepted) fail; the rest emit a note that becomes a failure if the status advances first.A gate flagging both would be noisy, and a noisy gate gets ignored — which is exactly how
codecov/patchstopped being read (#923). Scoping it keeps every failure real.The fix corrects what was actually false
E2E-VER-009claimedimplementedfor ElfBuilder unit tests (import_count, ImportEntry packing, string-table offsets, FNV hash) that do not exist. Its status moved toproposed.It was deliberately not re-pointed at
linker_script.rs, which merely asserts the generated script contains the string.meld_import_table. That is not the emission this artifact describes — citing it would restate the same lie more carefully.Design note
Test names come from a source scan, not
cargo test --list:Claim Checkdoesn't build the workspace, and a gate needing a 20-minute build to answer a 200 ms question wouldn't survive. The approximation errs toward false failure (loud, fixable) and cannot produce a false pass — which is the whole point.Wired into
Claim Checkwith a non-vacuity assert on a non-empty population.Local: citation gate 0 · claim 43/43 · rivet ours-errors 0.