Lane A (A.17-A.18): the deterministic comparator, and a corpus that is provably what it claims - #13
Merged
Merged
Conversation
…s now provably what it claims A.17 implements the version comparator the whole lane exists to feed, and A.18 reviewed it. A.18 returned FAIL with three blockers and six majors; two more rounds followed. This is the result. What the comparator is, and what it refuses S1 corrected the original spec to exactly this: a VERSION COMPARATOR answers "is this installed package inside a vulnerable range", deterministically and with no model. internal/match implements three schemes -- dpkg, rpmvercmp and apk -- and REFUSES everything else by name: npm, PyPI, Go, Maven, NuGet, Cargo, and semver AS A SCHEME. There is no fallback anywhere, and a test asserts the refusal also returns 0 so a caller that ignores the error gets nothing that looks like an answer. That matters more than the coverage does. A silently wrong CVE match either tells someone they are safe when they are not, or floods them until they stop reading any of it. Refusing is the only honest third option. The scheme implementations were compared statement-by-statement against dpkg verrevcmp/order and rpm rpmvercmp and found faithful. The defects were all in the layers ABOVE the comparison. Three blockers, and one fix that over-corrected - An empty advisory cache reported a CLEAN HOST. AssertNotSilentlyClean never read PackagesWithNoAdvisoryData -- the field whose own doc says it exists for that. The silent-clean failure mode, inside the guard named for preventing it. - An epoch spelled on one side only silently cleared a live vulnerability, on RHEL's routine shape. Now a typed refusal, in the direction that would push the installed version out of range, with ordering left untouched and a test pinning that the two rules cannot later be collapsed into one. - The identity check accepted a name spelling it then failed to look up. The fix for that OVER-CORRECTED into a second silent clean -- the purl name won unconditionally, so a purl naming a different package was accepted. The rule is now the one already applied to ecosystem and version: identical after ASCII fold is a spelling and canonicalises; a different name is a conflict. And the surviving name is the CANONICAL form rather than either producer's. The test could not have caught it: it varied only the REPORTED name across spellings of one package, so it exercised the fold and never the disagreement. Its doc now states what it does not cover, so it cannot be counted as covering it. The new test varies the PURL name against a fixed reported name -- the axis that never moved. A refused range also still decided BY ABSENCE, re-arming the exact backport false positive this lane exists to defeat, while the doc asserted the opposite invariant. A refusal now blocks its whole group. The corpus is now true by construction, not by wording Twice the corpus header overstated what had been transcribed -- the second time in the same section -- and the first instance was the sharpest form of it: the rpm corpus stopped exactly where the implementation would fail, so the provenance claim made a truncated set read as exhaustive. That is the third time on this project a provenance or compliance claim has been corrected and stayed false, and patching the wording a third time was not the fix. So provenance is now structural. Every vector is TRANSCRIBED (carrying its file and line) or AUTHORED (carrying the rule it encodes); the two sets have disjoint required fields and there is no third state. Completeness claims are DATA carrying their count, and a test fails in EITHER direction, fails on duplicate loci so a short corpus cannot be padded, and fails if a transcribed vector's source carries no counted claim. The corpus IS the replay now, not a summary of it: all 91 active RPMVERCMP lines from tests/rpmvercmp.at (86 pass, 5 refused, 0 wrong), all 43 rows of scripts/t/Dpkg_Version.t, and -- beyond what was asked -- all 738 ordering rows of apk-tools test/unit/version.data plus its 31-row validity block. Three vectors falsely cited to Dpkg_Version.t are re-tagged AUTHORED with the deb-version(7) rule they encode. rpm_compare.go's header claimed four separator-only vectors; there are five, and it no longer makes a completeness claim at all -- it defers to the counted one. apk R7a justified a refusal by asserting a tokeniser mechanism that is FALSE against the file it cited. src/version.c actually falls back to string sort on a leading zero, and the published fixture line 735 has 8.2.0015 < 8.2.002 -- which numeric comparison orders the other way. Justified from the real text now; R8's rationale, which asserted a token kind the file does not declare, takes the honest form instead: the ordering is unmodelled, so it is refused. AssertNotSilentlyClean could not be made the sufficient statistic its doc promised, so the doc was NARROWED and the over-promising sentence on CoverageReport.Complete was deleted rather than softened -- and every sentence of the new contract, the negative ones included, is asserted by a test. Evidence: gofmt, vet, build clean; go test -count=1 ./... green across 21 packages; no t.Skip added anywhere in internal/match. Determinism proven across SIX separate OS processes over a 53-package corpus -- and it earned that: one mutation was caught by child 2 and not child 1, which is why a single-process repeat would not have done. Every fix verified RED against the pre-fix behaviour before being trusted.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A.17implements the version comparator the whole lane exists to feed;A.18reviewed it and returned FAIL — 3 blockers, 6 majors. Two more rounds followed. This is the result.What it refuses matters more than what it covers
internal/matchimplements dpkg, rpmvercmp and apk, and refuses everything else by name: npm, PyPI, Go, Maven, NuGet, Cargo — and semver as a scheme. No fallback anywhere, and a test asserts the refusal also returns0, so a caller that ignores the error gets nothing that looks like an answer.That's the important property. A silently wrong CVE match either tells someone they're safe when they aren't, or floods them until they stop reading any of it. Refusing is the only honest third option.
The three scheme implementations were compared statement-by-statement against dpkg's
verrevcmp/orderand rpm'srpmvercmpand found faithful — every defect was in the layers above the comparison.Three blockers, and one fix that over-corrected
AssertNotSilentlyCleannever readPackagesWithNoAdvisoryData— the field whose own doc says it exists for that. The silent-clean failure mode, inside the guard named for preventing it.The test couldn't have caught that: it varied only the reported name across spellings of one package, so it exercised the fold and never the disagreement. Its doc now says what it doesn't cover, and the new test varies the purl name — the axis that never moved.
A refused range also still decided by absence, re-arming the exact backport false positive the lane exists to defeat, while the doc asserted the opposite invariant.
The corpus is now true by construction, not by wording
Twice the header overstated what had been transcribed — the second time in the same section. The first instance was the sharpest form of it: the rpm corpus stopped exactly where the implementation would fail, so the provenance claim made a truncated set read as exhaustive.
That's the third time on this project a provenance or compliance claim has been corrected and stayed false. Patching the wording a third time wasn't the fix.
So provenance is now structural: every vector is
TRANSCRIBED(carrying file and line) orAUTHORED(carrying the rule it encodes), with disjoint required fields and no third state. Completeness claims are data carrying their count, and a test fails in either direction, fails on duplicate loci so a short corpus can't be padded, and fails if a transcribed vector's source carries no counted claim.The corpus is the replay now, not a summary of it:
tests/rpmvercmp.atscripts/t/Dpkg_Version.t__DATA__rowstest/unit/version.dataThree vectors falsely cited to
Dpkg_Version.tare re-taggedAUTHORED.rpm_compare.goclaimed four separator-only vectors; there are five, and it no longer makes a completeness claim at all.apk R7a justified a refusal by asserting a tokeniser mechanism that is false against the file it cited.
src/version.cactually falls back to string sort on a leading zero, and the published fixture line 735 has8.2.0015 < 8.2.002— which numeric comparison orders the other way. Justified from the real text now.AssertNotSilentlyCleancouldn't be made the sufficient statistic its doc promised, so the doc was narrowed and the over-promising sentence deleted rather than softened — and every sentence of the new contract, the negative ones included, is asserted by a test.Testing
Determinism proven across six separate OS processes — and it earned that: one mutation was caught by child 2 and not child 1, which is precisely why a single-process repeat wouldn't have done. Every fix verified RED against the pre-fix behaviour before being trusted.