Lane A (A.1-A.6): feed table, cache, sanitizer, and a licence gate that refuses everything

[tom-snyder]

Summary

The ingestion foundation for Lane A — the deterministic, zero-inference lane that owns dependency and host findings by version comparator. A.1 feed table, A.2 cache, A.3 sanitizer, A.4 licence tier gate, A.5/A.6 critics.

Both critics returned FAIL, then three adversarial rounds followed. This PR lands the result, and the most important thing in it is what it does not claim.

The licence gate: trustworthy on refusal, NOT trustworthy on admission

S8 requires reading licence file bodies rather than metadata, because the body is the publisher's evidence. The first implementation read bodies Anvil wrote, committed alongside the claim they validated — circular, and worse than reading metadata because it looks rigorous.

Ruling: don't fetch the texts, fail closed on their absence. mirror/ carries a pinned manifest (canonical URL, expected sha256, claimed SPDX id) and an operator-run acquisition script. Every sha256 is empty, honestly, because no download was performed or authorised — a digest written from memory would be a fabrication. Consequence, stated in three places: a fresh clone admits no feed at all.

Three rounds then attacked the admission path and won every time:

Round	Approach	Defeated by
1	42-substring marker table	OSL-3.0's wording, and formatting — hard line wrapping (the normal shape of a licence file) and NBSP (the normal shape of HTML-sourced text; three pinned URLs are HTML pages)
2	Normalise, then invert the default so unknown = quarantine	Positive identification proved the document contains a permissive licence, not that it is one — MIT+CDDL published at tier 0
3	Identity via a veto index	Abbreviations ("the GPL"), British spelling ("Licence"), dual-licence wording, and a commentary titled "Apache License, Version 2.0"

I stopped there. The remaining vectors need a real licence identifier over a real corpus (askalono, licensee) rather than more substrings — and that's a dependency decision that's yours. The shipped state is safe regardless: every vector sits downstream of an acquisition step nobody has run.

So the package carries a KNOWN LIMITS section naming all eight vectors with working examples, what a green run does not prove, and an explicit statement that it is not a census — three rounds each found new vectors after the previous was called closed.

Four revisions of an overstated compliance document

The tier-2 LICENSE files claimed enforcement they didn't have and were corrected three times while remaining false. This revision applies a different rule — a claim that cannot be demonstrated is deleted, not qualified — and ten specific claims are deleted rather than reworded, each listed in the file. NOT ENFORCED is now the longer section and opens with "THE ADMISSION PATH IS NOT TRUSTWORTHY".

A test that excluded its own counterexamples

TestBothConsumersAgree claimed to sweep all of Unicode proving the sanitizer and licence normaliser share one definition of "renders as nothing". It contained two continue statements skipping exactly the ranges where they disagreed — 962,787 code points once removed.

That's the third variant of this shape here, after a guard that whitelisted return types and a marker table that validated itself against its own entries.

Split into three honestly-named tests: one green over the whole class with no exclusion, one whose restriction is in its name rather than buried in a continue, and the original kept unexcluded and skipped — its message names the exact count, the composition, and says "Delete this line to see the failure." Verified the skip isn't hiding a green test by deleting it and re-measuring: 959,049 exactly.

3,738 of those were a real hole and are closed — the shared class conditioned its Other_Default_Ignorable arm on IsGraphic, so U+2065, U+FFF0-U+FFF8 and the U+E0080-U+E0FFF reserved ranges fell through. The residual is a deliberate policy difference between the two consumers, recorded rather than papered over.

Also here

One shared internal/ingest/invisible package, so the sanitizer and normaliser stop maintaining two hand lists that drift — the defect class §6 closed ten instances of. Its supplement for blank glyphs Unicode doesn't categorise is explicitly labelled, and the test that can't verify it independently is named TestSupplementMembershipIsNotIndependentlyVerified.

The sanitizer's blocker: on hitting the comment-removal pass limit it returned the input intact — so the one input built to exceed the limit was the one that passed through untouched. Now fails closed, with all five HTML5 invisible-content productions handled and a residue oracle derived from unicode tables rather than the hand list that let U+3164 through.

Testing

gofmt / vet / build   clean
go test -count=1      green across all packages, exactly ONE skip (the honest one above)

-race can't run on this Windows host; CI runs it on Linux.

Stated, not hidden: neither internal/ingest/license nor internal/ingest/sanitize has a production importer yet — A.7, the poller, is what wires them — so every control in both is currently exercised only by its own tests.