Skip to content

RQ-56-CITE (#911): an artifact cannot claim a test that does not exist - #927

Merged
avrabe merged 1 commit into
mainfrom
feat/v056-artifact-citation-gate-911
Aug 8, 2026
Merged

RQ-56-CITE (#911): an artifact cannot claim a test that does not exist#927
avrabe merged 1 commit into
mainfrom
feat/v056-artifact-citation-gate-911

Conversation

@avrabe

@avrabe avrabe commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

First v0.56 lane. Closes #911. Theme: the factory, not the instances.

What shipped broken

v0.55.0 shipped two rivet artifacts citing cargo test -- <filter> for filters matching nothing. Such a filter prints 0 passed … 56 filtered out and exits 0 — so both claimed verification by evidence that never ran. No gate noticed; a post-release hygiene sweep did.

oracle_wiring_check.py has enforced this exact shape for scripts/repro/* since #890. The artifacts surface never got the equivalent, so the class stayed expressible one layer over. This adds it.

Red-first

The check found the two dangling citations independently — 11 of 13 resolve over 4,681 test names, so it isn't over-flagging.

Status-aware, because the data forced it

E2E-VER-010 is status: proposed and describes tests to be written. A forward-looking citation there is a plan, not a lie. Only claiming statuses (implemented/verified/accepted) fail; the rest emit a note that becomes a failure if the status advances first.

A gate flagging both would be noisy, and a noisy gate gets ignored — which is exactly how codecov/patch stopped being read (#923). Scoping it keeps every failure real.

The fix corrects what was actually false

E2E-VER-009 claimed implemented for ElfBuilder unit tests (import_count, ImportEntry packing, string-table offsets, FNV hash) that do not exist. Its status moved to proposed.

It was deliberately not re-pointed at linker_script.rs, which merely asserts the generated script contains the string .meld_import_table. That is not the emission this artifact describes — citing it would restate the same lie more carefully.

Design note

Test names come from a source scan, not cargo test --list: Claim Check doesn't build the workspace, and a gate needing a 20-minute build to answer a 200 ms question wouldn't survive. The approximation errs toward false failure (loud, fixable) and cannot produce a false pass — which is the whole point.

Wired into Claim Check with a non-vacuity assert on a non-empty population.

Local: citation gate 0 · claim 43/43 · rivet ours-errors 0.

v0.55.0 SHIPPED two rivet artifacts citing `cargo test -- <filter>` for filters
matching NOTHING. Such a filter exits 0, so both claimed verification by
evidence that never ran, and no gate noticed — a post-release hygiene sweep did.

`oracle_wiring_check.py` has enforced exactly this shape for scripts/repro/*
since #890. The ARTIFACTS surface never got the equivalent, so the same class
stayed expressible one layer over. This is that equivalent — the factory, not
another instance.

RED-FIRST: the check found the 2 dangling citations independently, 11 of 13
resolving over 4681 test names, so it is not over-flagging.

STATUS-AWARE, and the data forced this: E2E-VER-010 is `proposed` and describes
tests to be written — a forward-looking citation there is a PLAN, not a lie.
Only claiming statuses (implemented/verified/accepted) fail. A gate that flagged
both would be noisy, and a noisy gate gets ignored — which is precisely how
codecov/patch stopped being read (#923).

E2E-VER-009 fixed by correcting what was actually false: its STATUS. It claimed
`implemented` for ElfBuilder unit tests (import_count, ImportEntry packing,
string-table offsets, FNV hash) that do not exist. NOT re-pointed at
`linker_script.rs`, which merely asserts the generated script CONTAINS the
string ".meld_import_table" — citing that would restate the same lie more
carefully.

Test names come from a SOURCE SCAN, not `cargo test --list`: Claim Check does
not build, and a gate needing a 20-minute build to answer a 200ms question does
not survive. The approximation errs toward FALSE FAILURE (loud, fixable), never
false pass.

Wired into Claim Check with a non-vacuity assert on a non-empty population.

Local: citation gate 0, claim 43/43, rivet ours-errors 0.

Refs #911, RQ-56-CITE
@codecov

codecov Bot commented Aug 7, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@avrabe
avrabe enabled auto-merge (squash) August 8, 2026 15:54
@avrabe
avrabe merged commit 624ca2c into main Aug 8, 2026
52 checks passed
@avrabe
avrabe deleted the feat/v056-artifact-citation-gate-911 branch August 8, 2026 15:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Ported from spar's self-audit: 2 verification artifacts cite tests that never run (exit 0), and MSRV is ungated

1 participant