From 9eda814b9754e87117c71caac28b264c100d4945 Mon Sep 17 00:00:00 2001 From: "Tom D. Snyder" Date: Sat, 8 Aug 2026 17:29:52 -0400 Subject: [PATCH] R.2-R.5: fingerprint, its specification, and the collapsed store R.2 (fingerprint + corpus), R.3 (critic), R.4 (store schema), R.5 (migrations and the FTS5 guard), plus the fixes for the two blockers R.3 raised. The critic returned FAIL and it was right to R.3 did not accept R.2. Both blockers were proved rather than asserted, and both were fixed before this landed. BLOCKER 1 -- the algorithm existed only in Go. The critic re-implemented the plan's four-clause spec text in Python and got 55e27b07... where the committed golden is 13c60ccf... . NormalizeMatch applies nine-plus operations documented nowhere: a 193-entry reserved-word list, identifier-preservation rules, comment syntaxes, string-delimiter and escape rules, a number grammar. That breaks S6 one level up -- a second producer implementing from the written spec emits different digests, silently, forever, which is precisely what "one fingerprint algorithm, defined once" exists to prevent. It also left R.16 with no honest way to go green, since its oracle must be built "from scratch, not by importing fingerprint.go". Ruling: amend the specification, do not weaken the algorithm. The extra rules earn their keep -- abstracting a namespace qualifier collapses Ns::Helper(v) and Other::Helper(v) onto one digest. internal/record/FINGERPRINT-SPEC.md is now the authoritative definition of anvil-fp/v1. It is IN-TREE, not in plan/, because plan/ is gitignored and a second producer working from a clone has to be able to read it. fingerprint_spec_test.go asserts the document's reserved-word list and every algorithm constant match the code, so the spec cannot drift silently -- verified by perturbing the document and watching the guard fire. BLOCKER 2 -- DAST route templating was unimplemented. The spec defines route_template as derived, with numeric/UUID/hash segments replaced by a placeholder; CanonicalRouteTemplate did none of it, and all three DAST fixtures arrived pre-templated so the corpus could not see the gap. Two producers seeing one defect at /api/users/12345/orders would emit three digests. This is worse than the SAST case because the DAST tier is what earns "verified fixed" under S7, and a reproduction that cannot be matched to its prior finding cannot prove a fix. Ruling: area 40 owns the fingerprint, so area 40 canonicalizes. Area D emits whatever route it observed. Pushing it to D would leave two areas each believing the other did it. CanonicalRouteTemplate now templates all-digit, UUID, long-hex and long-alphanumeric segments onto one frozen token, and normalises already-templated segments ({id}, :id) onto the same token so the producer's choice of syntax cannot fork the digest. Thresholds are documented and justified against the stated asymmetry: over-templating merges distinct routes and loses a finding, under-templating is recoverable. Blocker 1 is closed by evidence, not by assertion An independent agent, forbidden from opening fingerprint.go, wrote a Python oracle from FINGERPRINT-SPEC.md alone and reproduced all 8 committed digests and 42/42 mutations on the first run, with no iteration. That is exactly the test R.16 will have to pass. Its honest caveat is recorded as Appendix Z: MATCH means the spec is sufficient for what the corpus exercises, and the corpus exercises a narrow slice. Six under-determined points are listed, of which Z4 matters most -- the ordinal grouping key is not exercised at all, because every SAST fixture supplies a pre-computed ordinal. An independent implementation could get that key wrong and still pass every current fixture. Recording it costs nothing now and would be expensive to rediscover as a digest divergence. Two judgment calls R.2 made, and which side it changed - Ns::Helper: changed the IMPLEMENTATION. The left operand of :: is a namespace or type in every language that has the operator, never a local, so abstracting it contradicted the spec's own "replace LOCAL identifiers". Deliberately not extended to . or ->, whose left operand is usually a receiver bound to a local; a lexer cannot tell a package qualifier from a receiver, and guessing wrong there forks the digest of unchanged code, which is the worse failure. - Ordinals: changed the TEST, whose `want` contradicted its own comment eight lines below and would have put two candidates on ordinal 0 in one group. Digest changes, stated explicitly: only dast-01 and dast-03 moved, both solely because route templating normalised {id}/{path} onto . The other five goldens are byte-identical to their pre-fix values. R.4/R.5: one handoff table carrying all thirteen dispositions and area O's consumption_class, per rulings G9/G10. Forward-only numbered migrations with a checksummed ledger. The FTS5 startup guard creates a real virtual table rather than trusting a version number -- kept even though FTS5 is now positively verified on v1.56.0, because a future bump could drop it silently. Evidence: gofmt, go vet, go build all clean; go test -count=1 ./... green across cmd/anvil, internal/record, internal/store. go test -race cannot run on this Windows host (cgo.exe exit 2, no C toolchain) -- pre-existing and host-wide; CI runs it on Linux. --- internal/record/CRITIQUE-01.md | 585 +++++++ internal/record/FINGERPRINT-SPEC.md | 720 ++++++++ internal/record/fingerprint.go | 1429 ++++++++++++++++ internal/record/fingerprint_spec_test.go | 349 ++++ internal/record/fingerprint_test.go | 1464 +++++++++++++++++ internal/store/ddl.go | 165 ++ internal/store/ddl_test.go | 608 +++++++ internal/store/guards.go | 457 +++++ internal/store/guards_test.go | 317 ++++ internal/store/migrate.go | 477 ++++++ internal/store/migrate_test.go | 556 +++++++ internal/store/migrations/0001_init.sql | 33 + internal/store/schema.sql | 459 ++++++ .../dast-01-sqli-error-based-body.json | 148 ++ .../dast-02-xss-reflected-query.json | 78 + .../dast-03-path-traversal-no-param-name.json | 105 ++ ...or-concrete-numeric-and-uuid-segments.json | 200 +++ .../host-01-openssl-debian.json | 88 + .../sast-01-go-sql-string-concat.json | 103 ++ .../sast-02-python-shell-command.json | 89 + .../sca-01-log4shell-maven.json | 102 ++ 21 files changed, 8532 insertions(+) create mode 100644 internal/record/CRITIQUE-01.md create mode 100644 internal/record/FINGERPRINT-SPEC.md create mode 100644 internal/record/fingerprint.go create mode 100644 internal/record/fingerprint_spec_test.go create mode 100644 internal/record/fingerprint_test.go create mode 100644 internal/store/ddl.go create mode 100644 internal/store/ddl_test.go create mode 100644 internal/store/guards.go create mode 100644 internal/store/guards_test.go create mode 100644 internal/store/migrate.go create mode 100644 internal/store/migrate_test.go create mode 100644 internal/store/migrations/0001_init.sql create mode 100644 internal/store/schema.sql create mode 100644 testdata/fingerprint_corpus/dast-01-sqli-error-based-body.json create mode 100644 testdata/fingerprint_corpus/dast-02-xss-reflected-query.json create mode 100644 testdata/fingerprint_corpus/dast-03-path-traversal-no-param-name.json create mode 100644 testdata/fingerprint_corpus/dast-04-idor-concrete-numeric-and-uuid-segments.json create mode 100644 testdata/fingerprint_corpus/host-01-openssl-debian.json create mode 100644 testdata/fingerprint_corpus/sast-01-go-sql-string-concat.json create mode 100644 testdata/fingerprint_corpus/sast-02-python-shell-command.json create mode 100644 testdata/fingerprint_corpus/sca-01-log4shell-maven.json diff --git a/internal/record/CRITIQUE-01.md b/internal/record/CRITIQUE-01.md new file mode 100644 index 0000000..9be719b --- /dev/null +++ b/internal/record/CRITIQUE-01.md @@ -0,0 +1,585 @@ +# CRITIQUE-01 — R.3, critique of R.1 (Record Field Contract) and R.2 (Fingerprint Specification) + +**Step:** R.3 · **Reviewed:** `internal/record/contract.go`, `internal/record/CONTRACT.md`, +`schemas/anvil-record-v1.schema.json`, `internal/record/fingerprint.go`, +`internal/record/fingerprint_test.go`, `testdata/fingerprint_corpus/*.json`, against +`plan/40-record-and-storage.md` (Record Field Contract + Fingerprint Specification), +`plan/00-SPINE.md` S1/S6/S7/S12, and `plan/IMPLEMENTATION-PLAN.md` §6. + +**Date:** 2026-08-08 · **Branch:** `feat/phase1-record-store` · **No code was modified by this pass.** + +--- + +## 0. WHICH GUARANTEE THIS DOCUMENT ACTUALLY PROVIDES — READ THIS FIRST + +**This was a SAME-FAMILY critic.** The critic and both implementers (R.1, R.2) are Anthropic models. +`plan/00-ROUTING.md` originally required a **different model family** here precisely so that a shared +blind spot could not survive review; the owner withdrew external routes on 2026-08-07 because running +them means copying `plan/` — deliberately private — to a third-party provider. See the OWNER DECISION +block at the top of `00-ROUTING.md`. + +**A later reader must not record this as a cross-family critique.** The cross-family guarantee +`00-ROUTING.md` asked for on data-integrity work has *not* been obtained, and nothing in this document +supplies it. What compensation was applied: + +- Findings were sought by refutation, not assessment: the implementations were assumed wrong. +- Every claim below was checked against the **files**, not against the implementers' prose. The + implementers' reasoning in doc comments is extensive and persuasive; it was treated as a claim to be + falsified, and in three places (findings 2, 3, 8) the doc comment states something the code does not do. +- Determinism was proved **empirically in two separate OS processes**, not by rereading the code and not + by trusting reported output. +- The one property that most needed an outside check — *can an independent re-implementation of the + written specification reproduce the committed digests?* — was answered by **writing that + re-implementation** (in Python, from the plan text, without reading `fingerprint.go`'s lexer) and + comparing digests. It cannot. That is finding 1, and it is the finding a same-family reviewer was most + likely to miss by nodding along with the Go code. + +**A residual same-family risk remains and cannot be closed here:** anything both the implementer and this +critic consider self-evidently correct about hashing, canonicalisation or SARIF was not challenged by an +outside vocabulary. + +--- + +## 1. VERDICT + +| Question R.3 must answer | Verdict | +|---|---| +| Every `00-SPINE.md` S6 field present in the contract | **PASS** (§3, field by field) | +| Fingerprint determinism | **PASS** — proved cross-process (§4) | +| Fingerprint excludes volatile fields | **PASS with one reservation** — package version, line, column, host, port, scheme, payload, timestamp all excluded and proved; the *ruleset* version is hashed and its churn is unbounded (finding 6) | +| Six frozen enums (§6) match, no second copy | **PASS** — literal by literal, three sources agree (§5) | +| Goldens are committed and compared, not regenerated | **PASS** (§6) | +| `CONTRACT.md` ⇄ `contract.go` ⇄ schema agree | **PASS** on all six enums and all S6 fields; one open decision unrecorded (finding 11) | +| **Is `anvil-fp/v1` "one algorithm, defined once" as S6 requires?** | **FAIL** — findings 1 and 2 | + +**Overall: FAIL.** Two blockers. Per the packet's stop condition, R.4 must not start until they are ruled +on. Note carefully that **neither blocker is a bug in the Go code**: the code is clean, well-tested and +internally consistent. Both are failures of the *written specification* to be the thing S6 says it must +be — the single, reproducible definition. Fixing them is an orchestrator ruling on +`plan/40-record-and-storage.md`, not a rewrite of `fingerprint.go`. + +--- + +## 2. GATE OUTPUT — real, re-run by this critic, `-count=1` throughout + +``` +$ gofmt -l . +(no output) + +$ go vet ./... +(no output) + +$ go build ./... +(no output) + +$ go test -count=1 ./... +ok github.com/Susquehanna-Syntax/Anvil/cmd/anvil 0.348s +? github.com/Susquehanna-Syntax/Anvil/cmd/anvil-dast [no test files] +? github.com/Susquehanna-Syntax/Anvil/internal/buildpin [no test files] +ok github.com/Susquehanna-Syntax/Anvil/internal/record 0.677s + +$ go test -count=1 -v ./internal/record/ | grep -c "^--- PASS" +129 +``` + +R.1's `contract_test.go` still passes alongside R.2's additions. Go 1.26.5. + +--- + +## 3. `00-SPINE.md` S6 — one verdict per required field + +S6's required set, in the order S6 lists it. "Where" cites the Go symbol; every row was additionally +checked to exist in `schemas/anvil-record-v1.schema.json` and to be described identically in +`CONTRACT.md`. + +| # | S6 field | Verdict | One-line reason | +|---|---|---|---| +| 1 | `anvil/state` | **PASS** | `State` + `PropAuditState`; six literals exactly as §6 froze them. | +| 2 | `anvil/version` | **PASS** | `PropAuditVersion = "anvil/version"`, present in schema and CONTRACT.md; correctly **not** hashed by any fingerprint tier. | +| 3 | per-half `status` | **PASS** | `HalfStatus` + `PropRunStatus`; `running\|sealed\|failed\|timed_out\|skipped`, matching §6's G5 ruling including the added `timed_out`. | +| 4 | per-half `sealedAt` | **PASS** | `PropRunSealedAt`; required in schema on a sealed half. | +| 5 | `anvil/trust` on every string originating outside Anvil | **PASS** | `Trust` with the three S6 literals, carried as an *object* not a bare enum (CONTRACT.md deviation 3) so one result can hold several provenances; `LegalForExternalString()` enforces the rule at the type level. The richer container is a strict improvement over the plan's table and does not change the literals. | +| 6 | `dast_status` | **PASS** | `DastStatus` + `PropAuditDastStatus`; all nine §6 literals, `skipped_no_manifest` kept distinct from `not_run` as G3+G6 requires. | +| 7 | `dast_coverage` | **PASS** | `PropRunDastCoverage`. | +| 8 | `target_provenance` | **PASS** | `TargetProvenance`, five literals; the G4+G7 split is honoured — the boot/reachability meaning is kept here and D's provisioning-path enum is the separate `TargetProvisioning` (`ephemeral_manifest\|live_url_authorized`). Both fields exist; neither was merged. | +| 9 | `remediable_by_agent` (host findings `false`) | **PASS** | `PropResultRemediableByAgent`; `Validate()` rejects `EvidenceClassHost` with `RemediableByAgent == true` (contract.go ~line 2317). Enforced in code, per S7's instruction. | +| 10 | `INSUFFICIENT_CONTEXT` as a verdict, not a confidence float | **PASS** | `VerdictInsufficientContext = "insufficient_context"`; a separate `anvil/confidence` exists but the verdict is a first-class enum value. | +| 11 | `as_of` | **PASS** | `Advisory.AsOf`; `Validate()` rejects a zero value when an advisory is linked. | +| 12 | `staleness_seconds` | **PASS** | `Advisory.StalenessSeconds`; non-negative enforced. | +| 13 | `parse_degraded` | **PASS** | `parseDegraded` in contract.go, schema and CONTRACT.md. | +| 14 | `endpoint_coverage` | **PASS** | `endpointCoverage` in contract.go and schema. | +| 15 | `inventory_provenance` | **PASS** | `InventoryProvenance` (`runtime_spec\|repo_spec\|static_extraction\|crawl`), in all three sources. | +| 16 | sanitizer state on any reproducer | **PASS** | `Repro.Env.Sanitizers []string`, `json:"sanitizers"`, schema-required; `Validate()` rejects `nil` and demands an empty array for a stock build, which is the right call — `null` and "no sanitizers" are different claims. | +| 17 | ASLR state on any reproducer | **PASS** | `Repro.Env.AslrEnabled bool`, `json:"aslrEnabled"`, schema-required. Key spelling identical in all three sources. | +| 18 | **One fingerprint algorithm, defined once, in the record** | **FAIL** | See findings 1 and 2. The algorithm is defined once *in Go*; the written specification is not sufficient to reproduce it, and one specified step of it is not implemented at all. | +| 19 | Ship a conformance test asserting identical digests on a fixed corpus | **PASS for R.2, at risk for R.16** | The corpus exists (7 fixtures, exceeding the 2/2/1/1 minimum) and is asserted. R.16's *independent* conformance oracle cannot currently be written to pass — finding 1. | + +S6's closing "Ordering" paragraph (re-cut the queue on every version bump; reserve a configurable +fraction of budget for late DAST arrivals) is a scheduler requirement, not a record field, and is not +in R.1's or R.2's scope. **It is not checked here and must not be assumed covered.** + +--- + +## 4. DETERMINISM — hunted first, proved empirically + +### 4.1 Static hunt + +`fingerprint.go` was read in full for the usual leaks. Grep, then reading: + +``` +$ grep -n "time\.\|rand\.\|filepath\.\|os\.\|runtime\.\|unsafe\.\|sync\.\|GOOS\|PathSeparator" internal/record/fingerprint.go +819: // fingerprinted is rejected here rather than at hash time. <- a comment, not a call + +$ grep -n "range " internal/record/fingerprint.go +262: for i, f := range fields <- slice +817: for i, c := range cands <- slice +851: for i, k := range order <- slice +``` + +- **Map iteration:** the only two maps are `metavars` (NormalizeMatch) and `fingerprintReservedWords`. + Both are read by key lookup only; neither is ranged over. Metavariable numbering comes from + `nextMetavar`, a counter advanced in source order, not from map order. **Clean.** +- **Time / randomness / pointers / addresses:** none reachable. `Digest` is a pure function of its + arguments. **Clean.** +- **Sorting:** `AssignSastOrdinals` uses `sort.SliceStable` with a total order (group key, then Line, + then Column, then original index), so ties cannot reorder. **Clean.** +- **Locale-dependent case folding:** `strings.ToUpper`/`ToLower` are used (HTTP method, purl type, host + package manager). Go's non-`Special` casers are Unicode-default and locale-independent — there is no + Turkish-I hazard. **Clean.** (Unicode *normalization* is a different matter — finding 8.) + +### 4.2 Empirical proof, in two separate OS processes + +Not two calls in one process — the failure mode that matters (per-process map seed, address-space state) +is invisible that way. The test binary was compiled once and executed **twice as independent +processes**, each printing the digest of every corpus fixture: + +``` +$ go test -count=1 -c -o $SCRATCH/record.test.exe ./internal/record +$ cd internal/record +$ ANVIL_FINGERPRINT_CROSS_PROCESS_CHILD=1 $SCRATCH/record.test.exe \ + -test.run='^TestCorpusDigestsAreStableAcrossProcesses$' -test.count=1 | grep ANVIL-FP > procA.txt +$ ANVIL_FINGERPRINT_CROSS_PROCESS_CHILD=1 $SCRATCH/record.test.exe \ + -test.run='^TestCorpusDigestsAreStableAcrossProcesses$' -test.count=1 | grep ANVIL-FP > procB.txt + +ANVIL-FP-DIGEST dast-01-sqli-error-based-body ca801b8d64fdabf43aad112e3b62c53211cf369a66683c12352081357eb9d125 +ANVIL-FP-DIGEST dast-02-xss-reflected-query bbe1a200328dfd7415d4a209384515a7ae01cb0494a896e69ba2bf565299c489 +ANVIL-FP-DIGEST dast-03-path-traversal-no-param-name 84fe311debbc87f852e1f4e17a3242d071169448c5852a00b1b32316fb30b036 +ANVIL-FP-DIGEST host-01-openssl-debian c6d2c1a505849bf987d9a03a5a3ce8ab6053a9ae5e0d154d332b464bb69e5953 +ANVIL-FP-DIGEST sast-01-go-sql-string-concat 13c60ccf8ec84530e075db4190005b03bc87050210065398fd48088683bb1aa6 +ANVIL-FP-DIGEST sast-02-python-shell-command d2239ee40c3f9da1cc9db3417962e3794b0c5baca0409015ff331a0aea095242 +ANVIL-FP-DIGEST sca-01-log4shell-maven c3208f00ab0f426a1c2cbb2ccf0033881bb4bf6cb506763dea0840f03dd24cd8 + +$ diff procA.txt procB.txt +IDENTICAL +``` + +Every digest also equals the `expected_digest` committed in the corresponding fixture. + +**Verdict: determinism PASS.** R.2's own `TestCorpusDigestsAreStableAcrossProcesses` is a genuine +cross-process check and not theatre; this critic re-ran the same property independently of it. + +### 4.3 Windows vs POSIX path separators + +Anvil targets Linux and is being developed on Windows, so a separator leak would be invisible on one +platform. Checked specifically: + +- `CanonicalRepoRelPath` and `CanonicalRouteTemplate` are **pure string manipulation**. Neither imports + `path/filepath`, `os.PathSeparator`, nor anything platform-conditional. The whole file imports only + `crypto/sha256`, `encoding/hex`, `fmt`, `sort`, `strconv`, `strings`, `unicode`. +- `sast-01`'s `repo_rel_path_in_windows_form` mutation (`.\internal\api\store.go`) and `sca-01`'s + `manifest_path_in_windows_form` mutation (`services\api\pom.xml`) both reproduce the base digest, and + they passed here on Windows. +- The one hazard the conversion introduces is the reverse direction — see finding 8(b). + +**No separator leak. PASS.** + +--- + +## 5. THE SIX FROZEN ENUMS — compared literal by literal against §6 + +`plan/IMPLEMENTATION-PLAN.md` §6's enum block was compared character by character against +`internal/record/contract.go`, `schemas/anvil-record-v1.schema.json` and `internal/record/CONTRACT.md`. + +| Enum | §6 literals | contract.go | schema `$defs` | CONTRACT.md | +|---|---|---|---|---| +| `anvil/state` | 6 | ✅ identical | ✅ identical | ✅ identical | +| `anvil/status` (per half) | 5 | ✅ | ✅ | ✅ | +| `anvil/dastStatus` | 9 | ✅ | ✅ | ✅ | +| `anvil/target.provenance` | 5 | ✅ | ✅ | ✅ | +| `anvil/target.provisioning` | 2 | ✅ | ✅ | ✅ | +| `anvil/verdict` | 3 | ✅ | ✅ | ✅ | +| `handoff.state` | 13 | ✅ | ✅ | ✅ | + +No drift, no omission, no extra literal, no case difference, no ordering difference in any of the three +sources. §6's "lowercase snake_case is the record's convention" holds for all six. + +**Did R.2 introduce a second, drifting copy? No.** `fingerprint.go` declares **no enum type**. It +consumes `DetectorKindSCA`, `DetectorKindHost`, `InjectionPoint` and `EvidenceSignal` from +`contract.go`, and validates the latter two through `ValidateInjectionPoint`/`ValidateEvidenceSignal` +rather than re-listing their literals. `TestTierValidationRejectsIncompleteInput` proves a +SCREAMING_CASE injection point and an off-spec `db_error_string` are both rejected. The one blemish is +two private duplicated string literals — finding 10, minor, and **not** a §6 violation because they are +not an enum declaration. + +`InjectionPoint` and `EvidenceSignal` are camelCase (`dbErrorString`), which looks like a violation of +§6's snake_case sentence. It is not: neither is one of the six frozen enums, both come from the +Fingerprint Specification's own camelCase text, and `CONTRACT.md` line 443–446 records the reconciliation +explicitly. Consistent across all three sources. **No finding.** + +--- + +## 6. GOLDENS — do they regenerate themselves? + +**No. PASS.** + +- `testdata/fingerprint_corpus/*.json` are committed data files carrying `hashed_fields` and + `expected_digest`. `TestCorpusFixturesProduceTheirDocumentedDigest` compares the implementation's field + list **and** digest against them; nothing writes back. +- There is **no `-update` flag, no `os.WriteFile`, no golden-regeneration path** anywhere in + `fingerprint_test.go`. The file contains an explicit, correct block (lines 429–444) forbidding one. +- `strictUnmarshal` uses `DisallowUnknownFields`, so a misspelled fixture key fails loudly instead of + silently defaulting a hashed field to `""` and locking in a wrong digest. That is the right paranoia. +- Independent check by this critic: each fixture's `expected_digest` was recomputed **outside Go**, in + Python, as `sha256(U+001F.join(hashed_fields))`. All seven matched, and all seven matched the Go + output in §4.2. So the fixtures are internally consistent and the Go code agrees with them. + +**The reservation:** that check proves `expected_digest = H(hashed_fields)`. It does **not** prove +`hashed_fields` is what the *written specification* produces from `input`. That is exactly what R.16 +exists to prove, and finding 1 shows it currently cannot. + +--- + +## 7. NUMBERED GAPS + +### BLOCKER 1 — `normalized_match` is defined only in Go, so R.16's mandated independent oracle cannot reproduce the SAST goldens + +`plan/40-record-and-storage.md` defines the SAST tier's hardest field in four clauses: + +``` +normalized_match := strip comments; collapse whitespace runs to a single space; + replace string/numeric literals with /; + replace local identifiers with positional $1..$N in first-occurrence order. +``` + +`fingerprint.go`'s `NormalizeMatch` implements those four **plus at least nine rules the specification +never states**: a ~200-entry cross-language reserved-word list held verbatim (lines 1120–1172); +identifiers after `.`, `->` or `::` held verbatim; identifiers before `::` held verbatim; identifiers +before `(` held verbatim (lines 1051–1068); CRLF folding; which comment syntaxes exist (`//`, `#`, +`/*…*/`); which string delimiters exist (`"`, `'`, `` ` ``) and where backslash escapes apply; the +number-token grammar (`0xFF`, `1_000`, `3.14f`, `1e-9` are each one token); and a final trim. + +This critic re-implemented the **specification text only** in Python — no reserved-word list, no +selector/callee/scope exceptions, everything else deliberately matched to the Go choices so the +divergence is attributable to the identifier rule alone — and rehashed the committed fixtures: + +``` +sast-01-go-sql-string-concat + spec-text normalized_match : '$1, $2 := $3.$4( + $5 + )' + committed normalized_match : '$1, $2 := $3.Query( + $4 + )' + spec-text digest : 55e27b07806178f7afd55b9178523ac45dd92b5e9aefcb00b2011f6d4cea2eed + committed expected_digest : 13c60ccf8ec84530e075db4190005b03bc87050210065398fd48088683bb1aa6 + *** DIVERGES *** + +sast-02-python-shell-command + spec-text normalized_match : '$1.$2( + $3)' + committed normalized_match : '$1.system( + $2)' + spec-text digest : 3ef752ff8836e5edacf8a8f8e38b895d41d816183147604d2445c884db7d09c0 + committed expected_digest : d2239ee40c3f9da1cc9db3417962e3794b0c5baca0409015ff331a0aea095242 + *** DIVERGES *** +``` + +**Why this is a blocker and not a nit.** R.16's packet requires an oracle that "re-implements the +algorithm text above from scratch, **not** by importing `internal/record/fingerprint.go`, so the oracle +is independent". Any such oracle produces the digests on the left. R.16 then has exactly two ways to go +green, and the plan forbids both: read `fingerprint.go` (destroys independence) or copy +`expected_digest` (the test file's own header at lines 31–33 forbids it, correctly). The spine-mandated +conformance gate is therefore unsatisfiable as written. + +It is also the S6 failure itself, one level up. S6's rule is "**One fingerprint algorithm, defined +once, in the record.**" Today the authoritative definition of `normalized_match` is Go source. A second +producer — the plan's whole premise is that there will be more than one — implementing from +`plan/40-record-and-storage.md` will emit different digests and nothing will surface it. The header +comment of `fingerprint_test.go` asserts the `hashed_fields` "were derived BY HAND from the algorithm +text"; for the two SAST fixtures that derivation must have used knowledge not present in the algorithm +text, and this critic could not reproduce it. **Claim not supported by the artifact.** + +**Proposed fix (orchestrator ruling, not a code change).** Amend the Fingerprint Specification to carry +the full normalization: the reserved-word list verbatim, the selector/callee/scope-resolution rules, the +comment syntaxes, the string delimiters and escape handling, the number-token grammar, CRLF folding and +the trim — and state that any edit to that list is an `anvil-fp/v2` event, which `fingerprint.go` line +1118 already says. Alternatively rule the other way and re-route R.2 to the literal four-clause text, +accepting the loss of discriminating power the implementer argues against at lines 900–907. Either +ruling is defensible; leaving it undecided is not, because R.16 will hit it and the cheapest thing R.16 +can do is quietly weaken its own oracle. + +--- + +### BLOCKER 2 — the DAST tier's only specified normalization is not implemented, and no fixture covers it + +The specification defines the DAST tier's route field as a **derived** value: + +``` +route_template := numeric/UUID/hash path segments replaced with a placeholder token. +``` + +`CanonicalRouteTemplate` (fingerprint.go:702–720) does not do this. It strips a query string or +fragment, collapses slash runs, adds a leading slash and trims a trailing one. It performs **zero** +segment templating. `DastInput`'s doc comment (line 575) reassigns the work to the caller — +"RouteTemplate is the path with volatile segments templated" — which the plan does not say. + +Consequence, and it is precisely S6's named failure: two producers observing the same defect at +`/api/users/12345/orders` will emit `/api/users/12345/orders`, `/api/users/{id}/orders` and +`/api/users/:id/orders` respectively. Three digests, one defect, no error, regression matching silently +dead. Worse than the SAST case, because the DAST tier is the one that earns "verified fixed" under S7 — +a DAST reproduction that cannot be matched to its prior finding cannot prove a fix. + +All three DAST fixtures (`dast-01`, `dast-02`, `dast-03`) arrive **already templated** +(`/api/v1/users/{id}/orders`, `/search`, `/files/{path}`), so the corpus cannot detect this. There is no +test anywhere that feeds a concrete numeric or UUID segment. + +**Proposed fix:** implement the templating in `CanonicalRouteTemplate` — replace segments matching +all-digits, a UUID, and a long hex/base32/base64 run with a single frozen placeholder token, with the +patterns written into the specification — and add a DAST fixture whose input carries +`/api/v1/users/12345/orders` and a UUID segment, with a mutation proving a different id yields the same +digest. If the orchestrator instead rules that templating belongs to the DAST producer (area D), then +`plan/40-record-and-storage.md` must say so, D's packet must own it with a named test, and the +specification's `route_template :=` line must be struck — otherwise two areas each believe the other +does it. + +--- + +### MAJOR 3 — `primaryLocationLineHash` is required by the contract, assigned to the fingerprint engine by the plan, and implemented by nobody + +- `plan/40-record-and-storage.md` line 653 lists `result.partialFingerprints["primaryLocationLineHash"]` + as **"required when a physical location exists"**, producer column **"fingerprint engine"**, consumer + column "GitHub upload path only (R.14)". +- `contract.go:2328` makes it a hard `Validate()` failure: + `partialFingerprints["primaryLocationLineHash"] is required when a physical code location exists`. +- `fingerprint.go:77–83` declines to implement it and reassigns production to R.14: *"It is not an + anvil-fp/v1 tier … it lives under `PartialFingerprintPrimaryLocationLineHash` and is owned by the + GitHub projection (R.14)."* + +The technical argument for keeping a line-dependent hash out of this file is good. But the plan's +producer column says fingerprint engine, and R.2 unilaterally moved it. Net effect **today**: no code in +the tree can construct a SARIF result with a physical code location that passes `Validate()`. R.4 (the +store) and every SAST producer will hit this. + +**Proposed fix:** an explicit ruling on the owner, plus whichever of these follows — R.2 ships a +separate `PrimaryLocationLineHash(...)` helper in its own file (identity untouched, no line number one +import from `Digest`), or R.14's packet is amended to own it and `plan/40`'s producer column is corrected +to say so. Do not leave `Validate()` enforcing a field with no producer. + +--- + +### MAJOR 4 — the ordinal group key omits `enclosing_symbol_path`, so an unrelated edit elsewhere in the file churns a finding's identity + +`AssignSastOrdinals` (fingerprint.go:824–833) groups on +`TargetID | RuleIDVersioned | CanonicalRepoRelPath | NormalizeMatch(Snippet)` — faithfully following the +specification's "index among all matches of the same `rule_id` in the same `repo_relpath` whose +`normalized_match` is IDENTICAL". `enclosing_symbol_path` is **hashed** but is **not** in the group key. + +Failing scenario: + +1. `store.go` contains `func A() { … exec(x) … }` at line 10 and `func B() { … exec(y) … }` at line 50. + Both normalise to `exec($1)`. +2. They are already distinguishable — `enclosing_symbol_path` differs — so they cannot collide. + Nevertheless they land in one ordinal group: A gets 0, B gets 1. +3. Someone deletes `func A` for unrelated reasons. +4. B's ordinal drops to 1 → 0. **B's digest changes.** B is reported resolved and re-opened as new. + `first_seen_at` resets, age-based ranking resets, any suppression keyed on B's fingerprint silently + stops applying, and a `handoff` row keyed on the old digest is orphaned. + +Adding `enclosing_symbol_path` to the group key removes the churn entirely and loses nothing: within one +symbol, ordinal still breaks the duplicate-call-site collision that motivated it (`fingerprint.go:37–41`). +`TestAssignSastOrdinals` sets `EnclosingSymbolPath: "sym"` on every candidate, so the case is untested. +This requires a one-line specification amendment as well as the code change, since the group key is +spec text. + +--- + +### MAJOR 5 — normalization is aggressive enough that two different sinks share a `normalized_match`, and identity then rests on the least stable field + +The implementer's own test documents it (`fingerprint_test.go:1063–1076`): `exec(cmd)` and `exec(other)` +both normalise to `exec($1)`, because `exec` is a preserved callee and the argument is abstracted. + +That is intended for renames. The consequence is not: two *semantically different* call sites — one +executing `cmd`, one executing `other` — are distinguished **only by `ordinal`**, which is derived from +line order. Swap the two lines and the two findings **swap identities**. Not churn — misattribution. +A triage verdict, a suppression, a `handoff` row and a "verified fixed" claim all transfer to the wrong +finding, and every field of both records still looks internally consistent. + +`fingerprint.go:784–790` records the ordinal's instability as a known limitation and points at +research/07's matching cascade to recover. That cascade does not help here: both findings are live, both +digests exist, and the cascade has nothing to disambiguate on. No test asserts against the swap. + +**Proposed fix:** either narrow abstraction so an argument identifier's *role* survives (e.g. abstract +only identifiers appearing more than once, or keep the first argument of a preserved callee), or accept +the risk explicitly in the specification and require the store to alert when two findings in one +(rule, path, normalized) group exchange ordinals between scans. Silent acceptance is the one option that +should not survive. + +--- + +### MAJOR 6 — a ruleset version bump re-mints every SAST and DAST fingerprint, with no bound and no migration + +`rule_id_versioned` is hashed in both the SAST (fingerprint.go:387–395) and DAST (638–647) tiers, per the +specification. `fingerprint_test.go:529` pins `"different rule version"` as *required* to change the +digest, and `fingerprint.go:319–321` argues the case: "a rule that changed what it matches is a different +rule." + +The argument is sound for a rule whose semantics changed. It is applied, however, to a token that moves +on the **ruleset's** release cadence — the fixtures use `opengrep…@2026.07.1` and +`nuclei:…@2026.07.1`. opengrep and nuclei-templates ship routinely; S7 already requires nuclei-templates +to be "pinned by commit SHA and diffed before promotion", i.e. bumped deliberately and often. Every such +bump resolves and re-opens **the entire SAST and DAST finding population at once**, resetting +`first_seen_at`, resetting age-based ranking, dropping every fingerprint-keyed suppression, and orphaning +every `handoff` row — with no error anywhere. The plan's only migration protocol (dual-write `v1` and +`v2` for one retention cycle) covers **algorithm** version changes, not rule-version changes, so nothing +absorbs this. + +R.3's remit is explicitly "exclusion of volatile fields". A token that turns over on a weekly-to-monthly +cadence, driven by an upstream project rather than by the scanned code, is volatile. + +**Proposed fix (one of):** hash the rule id **without** its version and carry the version as an unhashed +result attribute — the rule's *identity* is stable even when its ruleset ships; or extend the dual-write +migration protocol to cover ruleset bumps, with the store matching `old_rule_version OR new_rule_version` +for one cycle; or record an explicit accepted-risk ruling in `plan/40-record-and-storage.md` stating that +ruleset bumps mass-reset finding identity and naming what compensates. The current position — hash it, +pin the behaviour in a test, say nothing about the consequence — is the one that fails silently. + +--- + +### MINOR 7 — a C0 control byte in a snippet makes a finding unfingerprintable instead of being normalized away + +`NormalizeMatch` collapses whitespace and drops comments, but any other character falls through to +`out = append(out, c)` verbatim (fingerprint.go:1070–1073). A C0 control that is not +`unicode.IsSpace` — `NUL`, `BEL`, `ESC`, `\x1f` itself — therefore survives normalization, and `Digest` +then rejects the whole field (fingerprint.go:262–271). `Sast` returns an error and the finding **cannot +be hashed at all**. + +`Digest`'s rejection is right — a field carrying `U+001F` would move a field boundary. The gap is that +`NormalizeMatch` does not guarantee its output is hashable, so the failure lands on the producer as a +hard error. If any caller drops errored findings (the natural thing to do in a scan loop), a real +vulnerability on a line containing a stray control byte becomes invisible. S7's threat model makes this +worth closing: an attacker who can land a byte in a source file, a generated file or a vendored blob can +make the finding on that line unreportable. + +**Proposed fix:** have `NormalizeMatch` strip C0 and DEL the way it strips comments, and add a test that +`Sast` succeeds on a snippet containing `\x00` and `\x1f`. Keep `Digest`'s rejection as the backstop. + +*Confidence note: confirmed by reading the code path, **not executed** — this critic's packet forbids +adding a test file. `unicode.IsSpace` covers `\t \n \v \f \r`, space, U+0085 and U+00A0, and not `\x00` +or `\x07`, so those reach the `default` branch.* + +--- + +### MINOR 8 — `CanonicalRepoRelPath`'s stated goal is not achieved, and its backslash rewrite can merge two distinct POSIX files + +The doc comment (fingerprint.go:669–671) claims backslash conversion exists "so a Windows producer and a +Linux producer scanning the same repository agree". Three ways they still do not: + +- **(a) Case.** The function deliberately does not case-fold (lines 674–677), with a correct reason. + But Windows filesystems are case-insensitive, so a Windows producer may legitimately report + `Internal/API/Store.go` where a Linux producer reports `internal/api/store.go`. Different digests. + `TestCanonicalRepoRelPath` pins case preservation as intended, which pins the divergence too. +- **(b) Backslash is a legal POSIX filename character.** A real Linux file named `a\b.go` canonicalises + onto `a/b.go` and collides with the genuinely different file at that path. Rare, but it is a + *collision* — two findings, one identity, one lost on upsert — which the whole ordinal mechanism + exists to prevent. +- **(c) Unicode form.** No NFC/NFD normalization anywhere. A path or symbol containing a precomposed + vs decomposed accented character hashes differently, which is the macOS-checkout case. + +**Proposed fix:** narrow the doc claim to what is true ("separator form only"), or add NFC normalization +and decide the case question explicitly. (b) is cheap to fix — reject a path containing a backslash on +POSIX rather than rewriting it — but needs a ruling, since the Windows-producer fixture depends on the +rewrite. + +--- + +### MINOR 9 — five undocumented canonicalizations sit between fixture input and hashed field + +Each is defensible; none appears in `plan/40-record-and-storage.md`'s algorithm text, and each is another +way R.16's oracle will diverge (same root cause as finding 1, listed separately because writing them +down is nearly free): + +| Canonicalization | Where | In the spec text? | +|---|---|---| +| HTTP method upper-cased and trimmed | fingerprint.go:617–623 | no | +| Route query string / fragment stripped, slashes collapsed, leading `/` added, trailing `/` trimmed | 702–720 | no | +| purl scheme and type lower-cased | 736–739, 764–769 | no | +| purl subpath / qualifiers / version stripped | 750–758 | version yes, `?`/`#` no | +| host package manager lower-cased and trimmed; `:` forbidden in it | 472–474, 529–535 | no | +| repo path: backslashes, slash runs, leading `./` and `/`, trailing `/` | 678–689 | no | + +The corpus *tests* all of them via mutations, which is good, and pins them against future drift. It just +does not make them reproducible from the specification. + +--- + +### MINOR 10 — two of four tiers take their hashed discriminator from a private literal, two from the contract enum + +`tierTokenSast = "sast"` and `tierTokenDast = "dast"` (fingerprint.go:195–201) duplicate the values of +`DetectorKindSast` and `DetectorKindDast`, while the SCA and host tiers hash `string(kind)` taken from +the contract enum (line 501). The stated reason is to avoid confusing the tier token with the *evidence +class* — but `DetectorKind` is not the evidence class; `EvidenceClass` is, and it is a separate type with +`sast_reachable`/`sast_static_only`. `DetectorKind` already means exactly what the tier token means. + +Not a §6 violation (no enum is declared, no shared vocabulary is redefined) and not a bug today — +`TestTiersUseTheirSpecifiedDiscriminator` asserts both spellings. It is a second copy of a hashed +literal, in the one file whose entire premise is that a second copy of a hashed value is how +`anvil-fp/v1` got two meanings in the first place. + +**Proposed fix:** use `string(DetectorKindSast)` / `string(DetectorKindDast)` and keep the comment +explaining why the tier token is not the evidence class. + +--- + +### MINOR 11 — an R.2 decision that `CONTRACT.md` explicitly assigns to R.2 was never recorded + +`CONTRACT.md` deviation 2 (lines 429–432): *"`partialFingerprints["regionSha256"]` reserved … **R.2 +decides whether to populate it**, and R.2 may strike it if the algorithm has no use for it."* + +`fingerprint.go` never mentions `regionSha256`. The decision is neither made nor struck; it has silently +become nobody's. `research/24` names `fingerprint.region_sha256` non-negotiable for the coding-agent +handoff, so R.9/R.10 will meet it again with no ruling to consult. + +**Proposed fix:** R.2 (or the orchestrator) records one sentence in `CONTRACT.md` — populated by X, or +struck — before R.4. + +--- + +## 8. WHAT THIS PASS DID *NOT* CHECK + +Stated so the gap is visible rather than assumed covered: + +- **No cross-family review was obtained.** See §0. +- `contract.go` is ~104 KB. Its **six frozen enums, every S6 field, and the validation rules touching + them** were read closely. Its SARIF projection, JSON round-trip and the remainder of `Validate()` were + not audited line by line; `contract_test.go`'s 129 passing assertions were re-run, not re-derived. +- The **schema** was compared for enum literals and S6 field presence. It was not validated against a + JSON Schema meta-schema, and no instance document was validated against it by this pass. +- S6's **ordering / budget-reservation** requirement is out of R.1's and R.2's scope and was not checked + anywhere. +- Finding 7 is reasoned from the code path, not executed — this packet forbids adding a test file. +- The claim that the corpus `hashed_fields` were "derived by hand from the algorithm text" is + **unverifiable from the artifact** and, for the two SAST fixtures, is contradicted by finding 1's + experiment. Treat the goldens as pinning `fingerprint.go`'s behaviour, which is still worth having. + +--- + +## 9. WHAT R.4 MAY AND MAY NOT ASSUME + +**May assume, now proved:** the six frozen enums are correct and singly-sourced across Go, schema and +prose; every S6 field exists; digests are 64 lowercase hex, never truncated, deterministic across +processes and platforms; the package version, line, column, host, port, scheme, payload and timestamps +are all excluded by construction and by test; `UNIQUE (target_id, fingerprint)` is safe to build on for +the SCA and host tiers. + +**May not assume:** that a second producer implementing from `plan/40-record-and-storage.md` will emit +the same SAST or DAST digest (findings 1, 2); that `route_template` is normalized by anything +(finding 2); that a SAST finding's identity survives an unrelated deletion in the same file (finding 4) +or a ruleset bump (finding 6); that `primaryLocationLineHash` has a producer (finding 3). + +Findings 1 and 2 are blockers. Per the packet's stop condition, R.1 or R.2 — or, more likely here, the +Fingerprint Specification section of `plan/40-record-and-storage.md` — must be rerouted and re-reviewed +to all-PASS before R.4 starts. diff --git a/internal/record/FINGERPRINT-SPEC.md b/internal/record/FINGERPRINT-SPEC.md new file mode 100644 index 0000000..df5c43b --- /dev/null +++ b/internal/record/FINGERPRINT-SPEC.md @@ -0,0 +1,720 @@ +# FINGERPRINT-SPEC.md — the authoritative definition of `anvil-fp/v1` + +**Status:** normative. **Version:** `anvil-fp/v1`. **Last amended:** 2026-08-08. + +This document is the single definition of Anvil's finding-identity algorithm. It is deliberately +**in-tree**: `plan/` is gitignored, so a second producer working from a clone of this repository must +be able to read the complete algorithm here and nowhere else, and emit byte-identical digests. + +`plan/40-record-and-storage.md`'s "Fingerprint Specification" section remains the record of *why* the +research-branch conflict was resolved the way it was. It is a **summary**, not the definition. +`internal/record/CRITIQUE-01.md` (finding 1) proved the summary insufficient by re-implementing its +four-clause `normalized_match` text in Python and obtaining `55e27b07…` where the committed golden for +`sast-01-go-sql-string-concat` is `13c60ccf…`. The orchestrator ruled on 2026-08-08 that the +implementation was right and the specification incomplete, and that the fix was to write the +specification down completely. This file is that ruling discharged. **Where this document and +`plan/40-record-and-storage.md` disagree, this document governs.** + +`internal/record/fingerprint.go` implements it. `internal/record/fingerprint_spec_test.go` asserts that +the machine-checkable parts of this document — the reserved-word list and the algorithm constants — +are exactly the ones in the code, so the two cannot drift apart silently. + +--- + +## 0. The versioning rule — read before editing anything below + +**Every rule in this document is load-bearing on stored identity.** Changing *any* of it — +adding one reserved word, moving one threshold by one, reordering two normalization steps, altering a +token's spelling — changes digests that are already stored, and a changed digest means a finding is +reported resolved and re-opened as new: `first_seen_at` resets, age-based ranking resets, every +fingerprint-keyed suppression silently stops applying, and every `handoff` row keyed on the old digest +is orphaned. Nothing logs an error when this happens. That is the exact failure `plan/00-SPINE.md` S6 +exists to prevent: *"two producers emitting different hashes means regression matching silently fails +forever."* + +> **Any change to this document's algorithm is an `anvil-fp/v2` event, never a `v1` edit.** + +`fingerprint.go`'s `fingerprintReservedWords` comment says the same thing, and the two must keep saying +it. Shipping `v2` means: bump the algorithm name, dual-write both `v1` and `v2` values into +`finding_fingerprint` for one full retention cycle, match on `v1 OR v2` during that cycle, then retire +`v1`. Do not edit a golden digest to make a test pass. + +The **prose, rationale and examples** in this document may be improved freely. Only the rules may not. + +--- + +## 1. Primitives + +### 1.1 Field separator + +Fields are joined with **U+001F**, the ASCII Unit Separator (`"\x1f"`), in the exact order each tier +lists. Nothing else separates them; there is no prefix, suffix, length prefix, or trailing separator. + +U+001F was chosen over a printable glyph because a printable separator can occur inside a snippet or a +symbol name and silently move a field boundary — hashing `("a‖b", "c")` and `("a", "b‖c")` would +collide. U+001F cannot appear in normalized source text. + +### 1.2 Field guard + +Before joining, **every** field is checked. A field containing any character in `U+0000`–`U+001F` or +`U+007F` (C0 controls and DEL) is **rejected with an error**; the digest is not computed. U+001F is the +case that matters, but the whole control range is rejected, because none of these can legitimately +appear in a canonicalised path, rule id, symbol path, route template, purl, advisory id, or normalized +match — and a newline in one of them means the caller passed raw, uncanonicalised text. + +An empty **field list** is rejected. An empty **field value** is legal wherever the tier says so +(`enclosing_symbol_path`, `param_name`), and is hashed as a zero-length field, which keeps the field +count constant. + +### 1.3 Digest + + DIGEST = lowercase hex-encoded SHA-256 of the UTF-8 bytes of the joined string + +Exactly **64 lowercase hex characters. Never truncated.** Uppercase hex is not a valid digest and must +be rejected rather than folded — a store that accepted both would hold two rows for one finding and +defeat `UNIQUE (target_id, fingerprint)`. + +All string handling is on **UTF-8 bytes**. No Unicode normalization (NFC/NFD) is applied anywhere; see +§9. + +--- + +## 2. The four tiers + +Exactly four tiers exist. Each is a fixed, ordered field list. A wrong field *order* produces a +perfectly valid-looking 64-hex digest that is silently incompatible, so the order below is normative. + +### 2.1 Tier SAST — `evidence_class ∈ {sast_reachable, sast_static_only}` + + sha256( target_id ␟ "sast" ␟ rule_id_versioned ␟ repo_relpath + ␟ enclosing_symbol_path ␟ normalized_match ␟ ordinal ) + +Seven fields. + +| # | Field | Derivation | Empty allowed | +|---|---|---|---| +| 1 | `target_id` | verbatim | no | +| 2 | `"sast"` | the literal string `sast`, for **both** evidence classes | — | +| 3 | `rule_id_versioned` | verbatim, e.g. `opengrep.go.sqli@2026.07.1` | no | +| 4 | `repo_relpath` | §7.1 `CanonicalRepoRelPath` | no (nor after canonicalisation) | +| 5 | `enclosing_symbol_path` | verbatim, e.g. `pkg/mod.py::ClassA.method_b` | **yes** | +| 6 | `normalized_match` | §3, from the raw snippet | no (nor after normalization) | +| 7 | `ordinal` | §4, rendered as a base-10 integer with no padding and no sign | no; negative is rejected | + +Field 2 is the literal `sast` and **not** the evidence class, so a finding upgraded from +`sast_static_only` to `sast_reachable` keeps its identity. + +`enclosing_symbol_path` may be empty because a match in top-level module code, a config file, or a +template has no enclosing symbol, and rejecting those would make them unfingerprintable. + +**Never hashed by this tier:** line number, column number, the literal (non-normalized) snippet text, +`advisory_id`, the evidence class, any timestamp. + +### 2.2 Tiers SCA and HOST — one formula, parameterised + + sha256( target_id ␟ detector_kind ␟ advisory_id ␟ purl_base ␟ locator ) + +Five fields. `detector_kind` is the literal `sca` for repository dependencies and `host` for operating +system packages. It is the only thing keeping a repo dependency and a host package with the same +advisory apart, and it must be present: without it a host finding (`remediable_by_agent=false`) could +upsert over an agent-remediable dependency finding. + +| # | Field | Derivation | +|---|---|---| +| 1 | `target_id` | verbatim | +| 2 | `detector_kind` | the literal `sca` or `host` | +| 3 | `advisory_id` | **verbatim, not case-folded** — GHSA identifiers mix case meaningfully and folding would fork identity against the advisory table | +| 4 | `purl_base` | §7.2 `PurlBase` | +| 5 | `locator` | SCA: `manifest_relpath` through §7.1. Host: `":"`, §7.3 | + +**Never hashed by these tiers:** the version string. Bumping `1.2.3` → `1.2.4` while still inside the +vulnerable range must not mint a new finding; resolution is proved by re-evaluating `advisory_affects`, +never by an identity change. `PurlBase` strips the version defensively even if a caller passes it. + +### 2.3 Tier DAST — `evidence_class = dast_confirmed` + + sha256( target_id ␟ "dast" ␟ rule_id_versioned ␟ http_method ␟ route_template + ␟ injection_point ␟ param_name ␟ evidence_class_detail ) + +Eight fields. + +| # | Field | Derivation | Empty allowed | +|---|---|---|---| +| 1 | `target_id` | verbatim | no | +| 2 | `"dast"` | the literal string `dast` | — | +| 3 | `rule_id_versioned` | verbatim, e.g. `nuclei:CVE-2021-44228@a1b2c3d` | no | +| 4 | `http_method` | §7.4: trimmed, then upper-cased | no | +| 5 | `route_template` | §6 `CanonicalRouteTemplate` — a **derived** value | no (nor after derivation) | +| 6 | `injection_point` | one of `query`, `body`, `header`, `cookie`, `path`; any other value is rejected | no | +| 7 | `param_name` | verbatim. A parameter **name**, never a parameter **value** | **yes** | +| 8 | `evidence_class_detail` | one of `responseStackTrace`, `statusCodeFlip`, `dbErrorString`, `timingSideChannel`, `reflectedPayload`, `other`; any other value is rejected | no | + +Fields 6 and 8 are independent facts — *where* the payload went in versus *how* the defect was +observed. An SQL injection proved by a database error string and one proved by a timing side channel on +the same parameter are different findings with different remediation evidence. + +`param_name` may be empty: a whole-body, raw-request, or path-segment injection has no single named +parameter. + +**Never hashed by this tier:** host, port, scheme, the concrete payload string, any session token, any +timestamp. + +--- + +## 3. `normalized_match` — the complete algorithm + +This section is the one CRITIQUE-01 proved was under-specified. It is stated here in full. A +re-implementation that follows §3.1–§3.6 and the word list in §3.5 reproduces the committed SAST +goldens exactly. + +The algorithm is **one left-to-right pass over Unicode code points**, with no lookbehind beyond the +output buffer and no lookahead beyond skipping spaces. It is a single language-agnostic lexer, not N +parsers: Anvil's SAST tier is an opengrep subprocess that returns text, and the language is not +reliably known at fingerprint time. Determinism, not semantic perfection, is the property identity +needs. + +### 3.1 Preprocessing + +1. Replace every `"\r\n"` with `"\n"`. +2. Replace every remaining `"\r"` with `"\n"`. + +Both replacements are global and are applied in that order. (Order matters only in that step 1 must +precede step 2, or a CRLF would become two newlines.) + +The result is then decoded as a sequence of Unicode code points and scanned. Two output helpers are +used below: + +- **emit(s)** appends the characters of `s` to the output. +- **emitSpace()** appends a single space **only if** the output is non-empty and does not already end + in a space. It never produces two consecutive spaces and never produces a leading space. + +### 3.2 The scan + +At each position, the **first** matching rule below is applied. The rules are ordered; the order is +normative. + +| # | Condition at the current character `c` | Action | +|---|---|---| +| 1 | `c` is whitespace (Unicode `IsSpace`: `\t \n \v \f \r`, space, U+0085, U+00A0, and the Unicode space separators) | consume the whole whitespace run; `emitSpace()` | +| 2 | `c` is `/` and the next character is `/` | consume to the next `\n` (not consuming it) or to end of input; `emitSpace()` | +| 3 | `c` is `#` | consume to the next `\n` (not consuming it) or to end of input; `emitSpace()` | +| 4 | `c` is `/` and the next character is `*` | consume through the closing `*/`; if there is none, consume to end of input; `emitSpace()` | +| 5 | `c` is `"`, `'` or `` ` `` | consume the string literal per §3.3; `emit("")` | +| 6 | `c` is an ASCII digit `0`–`9` | consume the number token per §3.4; `emit("")` | +| 7 | `c` is an identifier-start character | consume the identifier and dispose of it per §3.6 | +| 8 | anything else | append `c` verbatim; advance one character | + +**Identifier-start** is: `_`, `$`, or any character with Unicode general category L (`IsLetter`). +**Identifier-part** is: `_`, `$`, any letter, or any digit (`IsDigit`, i.e. Unicode category Nd — not +only ASCII). + +Note the consequences, which are accepted and stable: + +- `#` is a line comment, so a C/C++ preprocessor directive or a C# `#region` inside a snippet is + dropped. Match snippets rarely contain them. +- `'` is a string delimiter, so a Rust lifetime (`'static`) or a Lisp quote consumes to the next `'`. +- Rule 3 is tested before rule 4 is reachable for `#`, and rule 2 before rule 4 for `//`, so `//*` is a + line comment, not a block comment. + +### 3.3 String literals + +Having consumed the opening delimiter `q` (one of `"`, `'`, `` ` ``), scan forward: + +- If the current character is `\` **and `q` is not** `` ` ``, skip **two** characters (the backslash and + whatever follows it, including a closing delimiter or another backslash) and continue. +- If the current character is `q`, consume it and stop. +- Otherwise consume one character and continue. +- If input ends first, stop. + +Backslash escapes are therefore honoured inside `"` and `'` and **not** inside `` ` `` — matching Go's +raw string literals, where a backslash has no special meaning. + +Exactly one `` is emitted per literal, regardless of its contents or length. No space is emitted +around it. + +### 3.4 Number tokens + +Having seen an ASCII digit, consume characters while **either**: + +- the character is a letter (`IsLetter`), a digit (`IsDigit`), `_`, or `.`; **or** +- the character is `+` or `-` **and** the immediately preceding *input* character was `e` or `E`. + +Stop at the first character satisfying neither. Exactly one `` is emitted. + +This grammar deliberately makes each of the following a **single** token: `0xFF`, `1_000`, `3.14f`, +`1e-9`, `1E+10`, `100L`, `0b1010`, `12.5e-3`. It also means `1.2.3` is one token, and that a number +immediately followed by a letter (`42px`) is one token. A number can never *start* a token that begins +with a letter, because rule 7 would have matched first. + +### 3.5 Identifier disposition — the rules the summary omitted + +Having consumed an identifier `word`, apply the **first** matching clause: + +| Clause | Condition | Emit | Why | +|---|---|---|---| +| **(a)** | `word` is in the reserved-word list below | `word` verbatim | A keyword is not a name a refactor renames. Abstracting `for`, `return` or `int` would erase the statement's shape and merge structurally different code. | +| **(b)** | the output so far, **ignoring trailing spaces**, ends in `.`, `->`, or `::` | `word` verbatim | `word` is a member, field, method or qualified name — API surface, not churn. Abstracting it would normalise `request.getParameter(userInput)` and `config.getName(key)` to the same string, destroying nearly all discriminating power and pushing the whole burden of distinguishing findings onto `ordinal`, the least stable field in the tier. | +| **(c)** | the next non-space **input** characters are `::` | `word` verbatim | The **left** operand of `::` is a namespace or type name in every language that has the operator (C++, Rust, PHP, Ruby) — never a local variable. Abstracting it would violate "replace local identifiers" outright **and** collapse `Ns::Helper(v)` and `Other::Helper(v)` — two calls into two different namespaces — onto one digest. | +| **(d)** | the next non-space **input** character is `(` | `word` verbatim | `word` is a callee name: the sink the rule actually matched. `exec(x)` and `spawn(x)` are different findings. | +| **(e)** | otherwise | `$N` | `word` is a local. `N` counts **distinct spellings** in first-occurrence order starting at 1, and every later occurrence of the same spelling maps to the same `$N`. This is what survives a rename. | + +Clause (b) asks about the **output**; clauses (c) and (d) ask about the **input**. That asymmetry is +deliberate: (b) looks at what has already been decided (a `.` survives step 8 verbatim, so it is +visible in the output), while (c) and (d) look ahead at text not yet scanned. + +**Why (b) treats `.` and `->` differently from (c)'s `::`.** The *left* operand of `.` or `->` is +usually a receiver bound to a local (`db.Query(q)`, `p->Field`), so it **is** abstracted — a lexer +cannot tell a Go package qualifier from a receiver variable, and guessing wrong in the other direction +would fork the digest of unchanged code, which is the worse failure. The *left* operand of `::` is +never a local, so it is preserved. Both operands to the *right* of any of the three are preserved by +(b). + +The metavariable counter and the spelling→`$N` map are **per call** to the normalizer: they start empty +for every snippet. `$1` in one finding has no relationship to `$1` in another. + +The map must never be iterated. Numbering comes from a counter advanced in source order. (In Go, map +iteration order is randomised per process; a re-implementation that numbered by iterating a hash map +would produce a stable-but-wrong order within a run and a different one in the next, which is the +cross-process nondeterminism `TestCorpusDigestsAreStableAcrossProcesses` exists to catch.) + +#### The reserved-word list, verbatim + +This is a **union** across the languages Anvil's SAST tier covers (Go, Java, C#, C/C++, +JavaScript/TypeScript, Python, Ruby, PHP): keywords, literal keywords and self-references, and +primitive/built-in type names. It is a union on purpose — a per-language list would require knowing the +language at fingerprint time, which the opengrep subprocess boundary does not reliably give us, and a +wrong language guess would change the digest of unchanged code. The union is a fixed, deterministic +function of the token text alone. + +The cost is accepted: an identifier named `class` in a language where `class` is not reserved is +preserved rather than abstracted. That is stable under re-scan, which is the property that matters. + +Matching is **exact and case-sensitive**. `None` is in the list; `NONE` is not. Whitespace-separated, +sorted in byte order (uppercase before lowercase), **193 entries**: + + +```text +False NULL None True abstract and any as +assert async await base begin bigint bool boolean +break byte case catch chan char checked class +clone cls complex128 complex64 const constexpr continue debugger +decimal declare def default defer del delete do +double echo elif else elseif elsif end endforeach +endif endwhile ensure enum error except exit explicit +export extends extern fallthrough false final finally float +float32 float64 fn for foreach friend from func +function global go goto if implements implicit import +in include include_once inline instanceof insteadof int int16 +int32 int64 int8 interface internal iota is keyof +lambda let lock long match module mutable namespace +native never new nil none nonlocal not null +nullptr object operator or out override package params +pass print private protected public raise range readonly +redo ref register require require_once require_relative rescue retry +return rune sbyte sealed select self short signed +sizeof stackalloc static strictfp string struct super switch +symbol synchronized template this throw throws trait transient +true try type typedef typeof uint uint16 uint32 +uint64 uint8 uintptr ulong unchecked undefined union unknown +unless unsafe unsigned until use ushort using var +virtual void volatile wchar_t when while with xor +yield +``` + + +`fingerprint_spec_test.go` asserts this block is exactly the set in `fingerprint.go`, is sorted, and +has no duplicates. + +### 3.6 Final trim + +The output has leading and trailing **spaces** removed (`TrimSpace` over the whole result). Because +`emitSpace()` never emits doubled spaces and no other rule emits whitespace, the result contains no +`\t`, `\n`, `\r`, and no run of two spaces — which is what lets it pass §1.2's field guard. + +A snippet that normalises to the **empty string** (comments and whitespace only) is **rejected**: it +carries no identity. + +### 3.7 Worked examples + +Each of these is pinned by a test. + +| Input | Output | +|---|---| +| `rows, err := db.Query("SELECT * FROM users WHERE name = '" + name + "'")` | `$1, $2 := $3.Query( + $4 + )` | +| `os.system("rm -rf " + path)` | `$1.system( + $2)` | +| `a = b + a + b` | `$1 = $2 + $1 + $2` | +| `rows := db.Query(userInput)` | `$1 := $2.Query($3)` | +| `for i := range items { return nil }` | `for $1 := range $2 { return nil }` | +| `value = compute(x) # trailing note` | `$1 = compute($2)` | +| `a = 0xFF + 1_000 + 3.14f + 1e-9` | `$1 = + + + ` | +| `s = "a \" b" + t` | `$1 = + $2` | +| `p->Field = Ns::Helper(v)` | `$1->Field = Ns::Helper($2)` | +| `std::vector v = Foo::Bar::make(x)` | `std::vector $1 = Foo::Bar::make($2)` | +| `Ns::Helper(v)` | `Ns::Helper($1)` | +| `Other::Helper(v)` | `Other::Helper($1)` | +| `totalCount := len(itemsList)` | `$1 := len($2)` | +| `a\n/* one\n two */\nb` | `$1 $2` | +| ` // nothing here \n\n` | *(empty — rejected)* | + +Trace of the first row, to remove all doubt: + +1. `rows` — identifier, not reserved; output is empty so (b) fails; next non-space is `,` so (c) and + (d) fail → `$1`. +2. `,` → verbatim. Space → one space. +3. `err` → `$2`. Space, `:`, `=` → verbatim (`:=` is two applications of rule 8). Space. +4. `db` — next non-space is `.`, not `::` or `(` → `$3`. +5. `.` → verbatim. +6. `Query` — output ends in `.` → clause (b) → `Query` verbatim. (Clause (d) would also have fired.) +7. `(` → verbatim. +8. `"SELECT * FROM users WHERE name = '"` — a double-quoted string; the `'` inside is ordinary + content → ``. +9. ` + ` → space, `+`, space. `name` → `$4` (a new spelling). +10. ` + ` then `"'"` → ``, then `)` → verbatim. + +--- + +## 4. `ordinal` and its grouping key + +`ordinal` is the **0-based index of this match among all matches of the same rule in the same file +whose `normalized_match` is identical**. It exists because without it two identical macro-expanded or +generated call sites in one file hash identically and the second finding is **lost** on upsert against +`UNIQUE (target_id, fingerprint)`. Losing a finding is worse than churning one. + +**Grouping key** — four components, joined with U+001F for comparison purposes only (never hashed as a +group): + + target_id ␟ rule_id_versioned ␟ CanonicalRepoRelPath(repo_relpath) ␟ normalized_match + +`target_id` is included because the specification's grouping is implicitly per-target; passing two +targets' candidates in one batch would otherwise cross-index them. + +**Ordering within a group** — ascending by, in order: + +1. source line number, +2. source column number, +3. the candidate's original index in the batch (a stable tiebreak). + +Line and column are used **only** for this ordering. They are never hashed and never reach the digest. +The sort must be stable. + +**Known limitation, inherited from the specification rather than chosen here.** Inserting a third +identical call site above two existing ones shifts their ordinals and therefore their digests. The +alternative — dropping the ordinal — silently loses one of two findings on upsert, which is worse. +research/07 §3's matching cascade (exact hit, then rule+path+line_hash, then rule+symbol_hash) is what +recovers identity in that case; the fingerprint alone cannot. CRITIQUE-01 findings 4 and 5 raise two +sharper forms of this and are **not ruled on** — see §10. + +--- + +## 5. `repo_relpath`, `purl_base`, `locator`, `http_method` + +*(§6 covers `route_template` separately; it is the longest.)* + +### 5.1 `CanonicalRepoRelPath` — see §7.1 +### 5.2 `PurlBase` — see §7.2 +### 5.3 host locator — see §7.3 +### 5.4 `http_method` — see §7.4 + +--- + +## 6. `route_template` — a DERIVED value + +The specification has always defined this field as derived: *"numeric/UUID/hash path segments replaced +with a placeholder token."* CRITIQUE-01 finding 2 proved the derivation was not implemented, and that +no fixture could detect it because all three DAST fixtures arrived pre-templated. + +**Ruling (2026-08-08): area 40 owns the fingerprint, so area 40 canonicalises.** A DAST producer emits +whatever route it observed — concrete or already templated in its own syntax. `CanonicalRouteTemplate` +derives the hashed template. This keeps **one owner**. If templating were the producer's job, two +producers seeing one defect at `/api/users/12345/orders` would emit `/api/users/12345/orders`, +`/api/users/{id}/orders` and `/api/users/:id/orders` — three digests, one defect, no error, regression +matching silently dead. This matters more than the SAST case because the DAST tier is what earns +"verified fixed" under `plan/00-SPINE.md` S7, and a reproduction that cannot be matched to its prior +finding cannot prove a fix. + +### 6.1 The placeholder token + + + +One frozen token for every volatile segment class. Angle brackets are chosen for the same reason +`` uses them: RFC 3986 excludes `<` and `>` from every production a path segment can use, so they +must be percent-encoded to appear in a real URL, and a literal segment therefore cannot collide with +the placeholder. `` is also itself recognised as an already-templated segment (§6.3, rule P), +which makes `CanonicalRouteTemplate` **idempotent** — a record read out of the store and +re-fingerprinted keeps its identity. + +### 6.2 Steps, in order + +1. If the input contains `?` or `#`, drop everything from the **first** occurrence of either. A query + string or fragment in a "template" carries concrete values — exactly what templating removes — and + `injection_point` plus `param_name` already record which query parameter was targeted. +2. Replace every `\` with `/`. +3. While the string contains `//`, replace every `//` with `/`. +4. If the string is now empty, the result is empty (and the DAST tier rejects it). +5. If the string does not start with `/`, prepend `/`. +6. If the string is longer than one character, remove a single trailing `/`. +7. If the string is exactly `/`, the result is `/`. +8. Otherwise split the string after the leading `/` on `/`, and replace every **volatile** segment + (§6.3) with ``. Re-join with `/` and prepend `/`. + +Case is preserved on non-volatile segments: URL paths are case-sensitive and `/Search` is a different +route from `/search` on most servers. The UUID and hex predicates in §6.3 are themselves +case-insensitive, because the same identifier rendered in upper hex is the same identifier. + +Percent-encoding is **not** decoded. Decoding could introduce a `/` and change the segment structure, +and a producer that percent-encodes a whole segment has emitted a different route. + +### 6.3 Which segments are volatile + +A segment is volatile if **any** of the following holds. Rules are evaluated in this order; the outcome +is the same token either way, but the order makes each rule independently testable. An empty segment is +never volatile. + +| Rule | Name | Predicate | +|---|---|---| +| **P** | already templated | length ≥ 2 **and** (starts `{` and ends `}`) **or** (starts `<` and ends `>`) **or** starts `:` | +| **N** | numeric | non-empty and every character is an ASCII digit `0`–`9` | +| **U** | UUID | length exactly 36; `-` at 0-based offsets 8, 13, 18 and 23; every other character an ASCII hex digit (`0-9A-Fa-f`) | +| **H** | long hex | length ≥ **16** and every character an ASCII hex digit | +| **O** | long opaque | length ≥ **20**, every character ASCII alphanumeric (`A-Za-z0-9`), containing **at least one digit and at least one letter** | + +**Rule P** accepts all three placeholder syntaxes in common use — OpenAPI/ASP.NET `{id}`, +Express/Rails/Sinatra `:id`, Flask/Werkzeug `` (including converter prefixes like +``) — and normalises them onto the same token as a concrete segment. It also ignores the +placeholder's **name**: `{userId}` and `{id}` are the same token. This is mandatory, not cosmetic: a +DAST crawler, an OpenAPI document checked into the repo, and a route table exported from a framework +will disagree about which syntax and which name to use for the same route, and that disagreement must +not fork identity. + +### 6.4 The thresholds, and why they are conservative + +**The governing asymmetry.** Over-templating merges two genuinely distinct routes into one identity and +loses a finding on upsert against `UNIQUE (target_id, fingerprint)` — silently. Under-templating only +leaves a volatile route un-merged, which the DAST producer can still repair by emitting `{id}` itself. +Under-templating is the recoverable direction, so both thresholds sit well above any plausible +human-authored path segment. + +**`routeHexSegmentMinLen = 16`** (rule H). The hex alphabet's letters are only `a`–`f`; a 16-character +English word drawn from `{a,b,c,d,e,f}` plus digits does not exist — the longest such words +(`defaced`, `cabbage`) are seven letters. Meanwhile every hash form Anvil will meet in a URL clears it: +MD5 is 32, SHA-1 is 40, SHA-256 is 64, a dash-free UUID is 32. A **short git object id (7–12 +characters) is deliberately NOT templated**, because 7 hex characters is also a plausible slug. + +**`routeOpaqueSegmentMinLen = 20`** (rule O). Measured against the longest plausible single-word route +segments: `recommendations` (15), `misrepresentation` (17), `internationalization` (20). None contains +a digit — which is why the **digit requirement carries most of the safety here**, not the length alone. +Real opaque tokens clear the bar comfortably: a base64url session token is 22+ characters and a base32 +token is 26+. + +Rule O's three restrictions each buy something specific, and dropping any of them over-templates: + +- **Alphanumeric only** (no `-`, `_`, `.`) keeps slugs out. `release-notes-2026-08` is 21 characters + and carries a digit; it is route structure, not an identifier, and merging every dated release note + onto one digest is exactly the over-templating failure. The cost is accepted: a **base64url token + containing `-` or `_` is left un-templated**. +- **A digit is required**, which excludes the long all-letter words that do reach 20 characters. +- **A letter is required**, so a purely numeric run is attributed to rule N rather than rule O. + +### 6.5 Worked examples + +Volatile (all → ``): + + 12345 0 4192 + 3f2504e0-4f89-11d3-9a0c-0305e82c3301 3F2504E0-4F89-11D3-9A0C-0305E82C3301 + 3f2504e04f8911d39a0c0305e82c3301 (32 hex, rule H) + e3b0c44298fc1c14 (16 hex, rule H at the threshold) + da39a3ee5e6b4b0d3255bfef95601890afd80709 + dXNlcjEyMzQ1Njc4OTAxMg (22 alnum with digits, rule O) + {id} {userId} :id :userId + +Preserved (never templated): + + v1 v2 me users api latest oauth2 utf8 Search + internationalization (20 letters, no digit) + recommendations + release-notes-2026-08 (hyphenated slug) + user_profile_settings (underscored slug) + deadbeef (8 hex, below 16) + cafebabecafebab (15 hex, below 16) + a1b2c3d (short git object id) + report.pdf 2026-08-08 + +Whole routes: + +| Input | Output | +|---|---| +| `/api/v1/users/12345/orders` | `/api/v1/users//orders` | +| `/api/v1/users/{id}/orders` | `/api/v1/users//orders` | +| `/api/v1/users/:userId/orders` | `/api/v1/users//orders` | +| `api//v1/users/12345//orders/?debug=1` | `/api/v1/users//orders` | +| `/12345/orders/6789` | `//orders/` | +| `/api/v1/users/me/orders` | `/api/v1/users/me/orders` | +| `/` | `/` | + +`testdata/fingerprint_corpus/dast-04-idor-concrete-numeric-and-uuid-segments.json` is the fixture that +proves the derivation happens, with twelve mutations that must all produce one digest. + +--- + +## 7. The remaining canonicalisations + +Each of these was applied by the implementation but absent from the summary text (CRITIQUE-01 finding +9). They are normative. + +### 7.1 `CanonicalRepoRelPath` + +Applied to SAST `repo_relpath` and to the SCA locator (`manifest_relpath`). In order: + +1. Replace every `\` with `/`. +2. While the string contains `//`, replace every `//` with `/`. +3. While the string starts with `./`, remove those two characters. +4. Remove a single leading `/` if present. +5. Remove a single trailing `/` if present. + +So `./cmd/x.go`, `/cmd/x.go`, `cmd\x.go` and `cmd/x.go` are one path. + +It deliberately does **not** case-fold — POSIX paths are case-sensitive and folding would merge two +genuinely distinct files. It does **not** resolve `..` — a path escaping the repo root is the caller's +bug and must not be silently rewritten into a different file. See §9 for what this does *not* achieve. + +### 7.2 `PurlBase` + +Reduces a package URL to its version-free base, `pkg:type/namespace/name`. + +1. Trim surrounding whitespace. Reject an empty string. +2. The first four characters must be `pkg:`, compared **case-insensitively**; otherwise reject. +3. Take everything after `pkg:` and truncate it at the first `#` (subpath), then at the first `?` + (qualifiers), then at the first `@` (version) — in that order. +4. Remove a single trailing `/`. Reject if nothing remains. +5. Lower-case everything up to the first `/` (the type). Reject if there is no `/` (a type but no name). +6. The result is `"pkg:"` + the processed remainder. + +The scheme and type are lower-cased because the purl specification defines both as case-insensitive +with a lowercase canonical form. The **namespace and name are left alone**, because their +case-sensitivity is type-dependent and folding them could merge two distinct packages. + +Truncating at the first raw `@` is safe against namespaced packages: purl requires a literal `@` inside +a namespace or name to be percent-encoded as `%40` (as in `pkg:npm/%40angular/core@13.0.0`), so the +first raw `@` can only be the version delimiter. + +This step is the **enforcement point** for "the version string is never hashed": a caller who passes +the full versioned purl by mistake still gets a version-free fingerprint. + +### 7.3 Host locator + + lower(trim(package_manager)) + ":" + trim(host_identifier) + +The manager is lower-cased because `APT` and `apt` are the same manager and a case difference between +two scanner versions would fork every host finding at once. The identifier is **not** case-folded and +is otherwise verbatim: architecture and suffixes are meaningful to the manager (`openssl:amd64`). + +A `package_manager` containing `:` is **rejected** — it is the locator's own delimiter, and allowing it +would make `a:b` + `c` indistinguishable from `a` + `b:c`. An empty manager or identifier (after +trimming) is rejected. + +### 7.4 `http_method` + +Trim surrounding whitespace, then upper-case. Reject if empty before trimming-and-checking, or if the +trimmed result contains any internal whitespace (it must be a single token). + +RFC 9110 methods are case-sensitive and canonically uppercase; a producer sending `get` must not fork +identity from one sending `GET`. + +Case folding uses the Unicode default case mappings, which are locale-independent. There is no +Turkish-I hazard. + +--- + +## 8. Machine-checked constants + +`fingerprint_spec_test.go` parses this block and asserts every value equals the corresponding constant +in `fingerprint.go` / `contract.go`. + + +```text +FingerprintAlgV1 = anvil-fp/v1 +FingerprintFieldSeparator = U+001F +FingerprintDigestHexLen = 64 +NormalizedStringToken = +NormalizedNumberToken = +NormalizedMetavarPrefix = $ +NormalizedRouteSegmentToken = +routeHexSegmentMinLen = 16 +routeOpaqueSegmentMinLen = 20 +``` + + +--- + +## 9. What this algorithm does NOT do + +Stated so the gaps are visible rather than assumed covered. + +- **No Unicode normalization.** A path or symbol containing a precomposed versus decomposed accented + character hashes differently. This is the macOS-checkout case (CRITIQUE-01 finding 8c). +- **No case folding of repo paths.** A Windows producer may legitimately report `Internal/API/Store.go` + where a Linux producer reports `internal/api/store.go`. Different digests (finding 8a). +- **Backslash rewriting is lossy on POSIX.** A real Linux file named `a\b.go` canonicalises onto + `a/b.go` and collides with the genuinely different file at that path (finding 8b). +- **`NormalizeMatch` does not strip C0 controls.** A snippet containing `NUL`, `BEL`, `ESC` or a raw + `\x1f` survives normalization and is then rejected by §1.2, so the finding cannot be hashed at all + (finding 7). +- **`rule_id_versioned` is hashed with its ruleset version.** An opengrep or nuclei-templates release + bump re-mints every SAST and DAST fingerprint at once (finding 6). +- **`enclosing_symbol_path` is hashed but is not in the ordinal grouping key**, so deleting an + unrelated function above a match can churn that match's ordinal and therefore its digest (finding 4). +- **Two semantically different sinks can share a `normalized_match`** (`exec(cmd)` and `exec(other)` + both give `exec($1)`), leaving `ordinal` as the only discriminator; swapping the two lines swaps the + two findings' identities (finding 5). +- **`primaryLocationLineHash` is not produced here.** It is not an `anvil-fp/v1` tier; it is a separate, + line-dependent partial fingerprint for GitHub code-scanning de-duplication (finding 3). + +None of these is ruled on by this document. See §10. + +--- + +## 10. Provenance + +| Date | Change | +|---|---| +| 2026-08-08 | Document created. Discharges the R.3 blocker-1 ruling: `normalized_match` was defined only in Go, so R.16's mandated independent oracle could not reproduce the SAST goldens and a second producer implementing from the written text would diverge silently. §3 is the algorithm written down completely; §5 and §7 close CRITIQUE-01 finding 9. | +| 2026-08-08 | §6 added. Discharges the R.3 blocker-2 ruling: `route_template` was specified as derived and derived by nobody. Templating is implemented in `CanonicalRouteTemplate` and owned by area 40. Two DAST goldens moved as a result — `dast-01` `ca801b8d…` → `199c3b5f…` and `dast-03` `84fe311d…` → `5fc15c55…`; `dast-02` and every SAST, SCA and host golden are unchanged. Nothing had been stored under the old digests, so this is a correction of an unimplemented clause, not a `v2` event. | + +**Open, not ruled on.** CRITIQUE-01 findings 3, 4, 5, 6, 7, 8, 10 and 11 remain open. They are listed +in §9 so no downstream area assumes they are handled. Findings 4, 5 and 6 in particular each require an +orchestrator ruling before `anvil-fp/v1` is treated as final, and findings 4 and 6 would be `v2` events +if ruled in favour of the critic. + +--- + +## Appendix Z — Where this specification is still under-determined + +Recorded by the orchestrator on 2026-08-07, from the independent-oracle verification that closed +blocker 1. **These are not known bugs. They are places where the prose admits more than one reading and +the corpus happens not to distinguish them** — which is the same defect as blocker 1, one level down and +not yet triggered. + +The verification that closed blocker 1 was real: an implementer working from this document alone, +forbidden from reading `fingerprint.go`, reproduced all 8 committed digests and 42/42 mutations on the +first run with no iteration. But **MATCH means this document is sufficient for what the corpus +exercises**, and the corpus exercises a narrow slice. Anyone extending the corpus, and `R.16` in +particular, should resolve these first — each one is cheap to settle now and expensive to discover as a +digest divergence later. + +| # | Where | The ambiguity | What the oracle guessed | +|---|---|---|---| +| Z1 | §3.2 rule 1 | *"the Unicode space separators"* names category **Zs**, but Go's `unicode.IsSpace` implements the **White_Space** property, which also includes U+2028 (Zl) and U+2029 (Zp). | Included both. **Unverified** — no fixture contains non-ASCII whitespace. | +| Z2 | §3.5 clauses (c), (d) | *"the next non-space input characters"* — "non-space" is never defined and need not mean the same class as rule 1's "whitespace". | Used the full whitespace class, so `Ns\n::Helper(v)` preserves `Ns`. A narrower reading abstracts it. | +| Z3 | §6.3 rule P | *"length >= 2 and (starts { and ends }) or (starts < and ends >) or starts :"* — and/or precedence in prose is genuinely ambiguous. | Read as `len>=2 AND (A or B or C)`. The alternative parse differs for the one-character segment `:`. | +| Z4 | §4 (ordinals) | **Not exercised by the corpus at all.** Every SAST fixture supplies a pre-computed `ordinal` in its `input`; no fixture supplies a batch of candidates with line/column for an implementer to derive ordinals from. | Nothing — untested end to end. An independent implementation could get the grouping key wrong and still pass every current fixture. | +| Z5 | §3 generally | Only two snippet shapes appear (a Go `.`-selector call and a Python `.`-selector call) plus comment/CRLF/rename variants. **Unexercised:** §3.2 rule 4 block comments, §3.3 backtick raw strings and their no-escape rule, §3.4 `` entirely, and most of the reserved-word list. | Nothing — those paths have no fixture. | +| Z6 | fixture schema | The fixture JSON uses `evidence_signal` where the spec's field is `evidence_class_detail`, and `repo_rel_path` where the spec says `repo_relpath`. The spec never states the fixture schema, so the mapping is inferred by eye. | Mapped by inspection. Unambiguous in practice, but it is an undocumented interface. | + +**Z4 is the one that matters most.** The ordinal grouping key is what keeps a finding's identity stable +when an unrelated edit moves it, and it is the single least-tested part of the algorithm. It also +already carries an unresolved `CRITIQUE-01` finding (the key omits `enclosing_symbol_path`, so an edit +elsewhere in the same file can churn a live finding's identity). Settling Z4 and that finding together +is the obvious next move on this algorithm. + +**Resolving any of these is an `anvil-fp/v2` event if it changes a digest, and a v1 clarification if it +does not.** Determine which before editing, not after. diff --git a/internal/record/fingerprint.go b/internal/record/fingerprint.go new file mode 100644 index 0000000..a84ddc9 --- /dev/null +++ b/internal/record/fingerprint.go @@ -0,0 +1,1429 @@ +package record + +// fingerprint.go — the ONE anvil-fp/v1 algorithm. +// +// =========================================================================== +// THE CONFLICT THIS FILE RESOLVES +// =========================================================================== +// +// research/07-database-design.md §3 ("Fingerprint scheme: anvil-fp/v1 — a +// four-tier cascade") and research/18-unified-audit-record.md ("Stable +// identity — the rule that makes regression checking work") each specified a +// DIFFERENT algorithm under the SAME name, `anvil-fp/v1`. They disagree on +// the field set, the separator, the digest length, and the normalization +// depth. +// +// plan/00-SPINE.md S6 states the consequence plainly: "One fingerprint +// algorithm, defined once, in the record. Two branches specified different +// /v1 algorithms under the same name; two producers emitting different hashes +// means regression matching silently fails forever." Silently is the +// operative word — nothing surfaces the failure. Every finding looks new on +// every scan, `finding.first_seen_at` never stabilises, the regression check +// `UNIQUE (target_id, fingerprint)` never fires, and "verified fixed" can +// never be proved, all without a single error being logged. +// +// plan/40-record-and-storage.md, "Fingerprint Specification", is the +// orchestrator's resolution. THAT TEXT — not either research branch verbatim +// — is what this file implements. The seven contested points and the reason +// each was decided that way: +// +// 1. SEPARATOR — research/07's explicit U+001F wins over research/18's +// undefined "‖" glyph. A printable separator can occur inside a snippet +// or a symbol name and silently move a field boundary: hashing +// ("a‖b", "c") and ("a", "b‖c") would collide. U+001F cannot appear in +// normalized source text, and Digest rejects it (and every other C0 +// control character) outright rather than trusting that claim. +// +// 2. ORDINAL — research/07's `ordinal` is adopted; research/18's SAST +// formula lacks it. Without it, two identical macro-expanded or generated +// call sites in one file hash identically and the second finding is lost +// on upsert against `UNIQUE (target_id, fingerprint)`. Losing a finding +// is worse than churning one. +// +// 3. advisory_id IS EXCLUDED from the SAST hash — research/07 includes it; +// research/18 does not; the resolution follows research/18. Identity +// tracks "this exact defect in this exact code". If ingestion later +// attaches a specific advisory to a sink previously linked only to a +// generic CWE, that reclassification must not fork the finding's +// identity. SastInput therefore has no advisory field at all — the +// exclusion is enforced by the type, not by discipline. +// +// 4. NO TRUNCATION — research/07 truncates to 32 hex chars "for storage". +// The resolution does not truncate. SQLite's cost for a 64- vs +// 32-character TEXT column is negligible, and halving a cryptographic +// digest without a forcing constraint buys nothing and adds collision +// risk. Digest always returns FingerprintDigestHexLen (64) characters. +// +// 5. NORMALIZATION DEPTH — research/07's metavariable abstraction +// (literals to /, local identifiers to positional $1..$N) is +// adopted over research/18's whitespace-and-comments-only normalization, +// because it is the one directly modelled on Semgrep's `match_based_id` +// — the cited, externally-verified mechanism for surviving reindentation +// and metavariable-only edits (research/07 [S3]). +// +// 6. DAST EVIDENCE SIGNAL — research/18's `evidenceClass` (HOW the defect +// was observed) is a real distinguishing signal absent from research/07's +// Tier D. It is hashed ALONGSIDE research/07's injection_point/param_name +// (WHERE the payload went in), because the two are independent facts. +// See EvidenceSignal and InjectionPoint in contract.go. +// +// 7. HOST TIER — neither branch defined a tier for host/package findings, +// but plan/00-SPINE.md S1 gives Lane A "dependency and host findings" and +// S6 requires `remediable_by_agent=false` on all of them, so they need an +// identity. research/07's Tier C is generalised by parameterising the +// hashed detector kind over {sca, host} rather than inventing an +// unrelated scheme. +// +// research/18's Tier B (CodeQL's `primaryLocationLineHash`) is deliberately +// NOT implemented here. It is not an anvil-fp/v1 tier; it is a separate, +// line-DEPENDENT partial fingerprint whose only purpose is GitHub code +// scanning de-duplication. It lives under +// PartialFingerprintPrimaryLocationLineHash and is owned by the GitHub +// projection (R.14). Computing it here would put a line number one import +// away from this file. +// +// =========================================================================== +// THE VERSION STRING IS NEVER HASHED +// =========================================================================== +// +// For the SCA and host tiers this is the single most load-bearing exclusion, +// and it is the reason PurlBase exists rather than the caller passing a purl +// straight through: +// +// Bumping 1.2.3 to 1.2.4 while still inside the vulnerable range must not +// mint a new finding. +// +// If the version were hashed, every patch-level dependency bump would resolve +// the old finding and open an identical new one. `first_seen_at` would reset, +// the age-based ranking would reset, any suppression keyed on the fingerprint +// would silently stop applying, and a maintainer bumping a version WITHOUT +// leaving the vulnerable range would see the alert disappear and reappear as +// "new" — indistinguishable from an actual fix. Resolution is proved by +// re-evaluating `advisory_affects`, never by an identity change. +// +// PurlBase enforces this defensively: it truncates a purl at the first +// version, qualifier, or subpath delimiter, so a caller who passes the full +// versioned purl by mistake still gets a version-free fingerprint. The +// ScaInput/HostInput structs also have no Version field, so there is no +// in-band way to hash one. +// +// =========================================================================== +// WHAT IS EXCLUDED BY CONSTRUCTION +// =========================================================================== +// +// The tier input structs below deliberately have NO field for anything the +// specification forbids hashing. This is a type-level guarantee, not a +// convention, and TestInputStructsCannotCarryForbiddenFields asserts it +// reflectively so a later edit cannot quietly add one: +// +// line number, column number — absent from SastInput entirely; any +// unrelated edit above a match changes them +// (research/07 §3: "the single most important +// rule"). +// raw snippet text — SastInput.Snippet is normalized before it +// is hashed; the literal text never reaches +// Digest. +// advisory_id (SAST tier) — see resolution point 3. +// host, port, scheme (DAST) — absent from DastInput; a redeployed +// container, a rotated ephemeral port, or a +// staging-to-prod move must not fork +// identity. +// payload string (DAST) — absent from DastInput; a fuzzer varying +// its payload must not create N findings for +// one bug. +// timestamps — absent from every input struct. +// version string (SCA/host) — absent, and stripped defensively by +// PurlBase. +// evidence_class (SAST tier) — the SAST tier hashes the literal "sast", +// not the evidence class, so a finding +// upgraded from sast_static_only to +// sast_reachable keeps its identity. +// +// =========================================================================== +// THE AUTHORITATIVE SPECIFICATION IS internal/record/FINGERPRINT-SPEC.md +// =========================================================================== +// +// R.3's CRITIQUE-01.md proved that the four-clause `normalized_match` text in +// plan/40-record-and-storage.md is NOT sufficient to reproduce this file's +// digests: a re-implementation written from that text alone emits +// 55e27b07... where the committed golden for sast-01 is 13c60ccf... . The +// orchestrator ruled (2026-08-08) that the implementation is right and the +// specification was incomplete, and that the fix is to write the +// specification down completely, IN TREE. +// +// internal/record/FINGERPRINT-SPEC.md is that document. It is the +// authoritative definition of anvil-fp/v1: every normalization step in +// NormalizeMatch in the order applied, the reserved-word list verbatim, the +// identifier-preservation rules and their reasons, the route-segment +// templating patterns and thresholds, the separator, the join order, the hash, +// and the ordinal grouping key. plan/ is gitignored; a second producer working +// from a clone can read FINGERPRINT-SPEC.md and nothing else and still emit +// byte-identical digests. +// +// fingerprint_spec_test.go keeps the document honest: it asserts that the +// reserved-word list and the algorithm constants printed in the document are +// exactly the ones in this file. A specification that can drift from the code +// silently is the same defect one level up. +// +// =========================================================================== +// CONFORMANCE (R.16) +// =========================================================================== +// +// testdata/fingerprint_corpus/*.json is the fixed corpus. Every fixture +// carries its complete ordered `hashed_fields` list and its `expected_digest`, +// so R.16's offline oracle can re-derive the digest from the fixture alone — +// join with U+001F, SHA-256, lowercase hex — without importing or reading any +// Go code. R.16 must NOT copy `expected_digest`; it must recompute it from +// FINGERPRINT-SPEC.md (NOT from plan/40-record-and-storage.md, whose algorithm +// text is a summary and was proved insufficient by CRITIQUE-01) and assert +// equality. +// That mutual check is the mechanism that would have caught research/07 and +// research/18 shipping two different /v1 algorithms under one name. +// +// Sources: plan/40-record-and-storage.md ("Fingerprint Specification"); +// plan/00-SPINE.md S1, S6, S7; plan/IMPLEMENTATION-PLAN.md §6 (this file +// declares no shared enum — it consumes contract.go's DetectorKind, +// InjectionPoint and EvidenceSignal); research/07-database-design.md §3; +// research/18-unified-audit-record.md ("Stable identity"). + +import ( + "crypto/sha256" + "encoding/hex" + "fmt" + "sort" + "strconv" + "strings" + "unicode" +) + +// --------------------------------------------------------------------------- +// Algorithm constants that R.16's independent oracle must reproduce +// --------------------------------------------------------------------------- + +// The separator (FingerprintFieldSeparator, U+001F) and the digest length +// (FingerprintDigestHexLen, 64) live in contract.go, because they are wire +// contract rather than implementation. The tokens below are algorithm detail +// but must still be reproduced byte for byte by any re-implementation, so +// they are exported and named. +const ( + // NormalizedStringToken replaces every string literal in a normalized + // SAST match. Chosen with angle brackets because they cannot appear in + // an identifier, so a source-level `` cannot be confused with the + // placeholder in any language Anvil scans. + NormalizedStringToken = "" + + // NormalizedNumberToken replaces every numeric literal in a normalized + // SAST match. + NormalizedNumberToken = "" + + // NormalizedMetavarPrefix prefixes the positional metavariables that + // replace local identifiers: $1, $2, ... in first-occurrence order. + NormalizedMetavarPrefix = "$" + + // NormalizedRouteSegmentToken replaces every volatile path segment in a + // canonicalised DAST route template — a numeric id, a UUID, a long + // hex/base32/base64-ish opaque token, and any segment a producer has + // already templated in its own syntax ("{id}", ":id", ""). + // + // Angle brackets are chosen for the same reason as NormalizedStringToken: + // RFC 3986 excludes '<' and '>' from every production a path segment can + // use, so they must be percent-encoded to appear in a real URL. A literal + // segment therefore cannot be confused with the placeholder, and + // CanonicalRouteTemplate is idempotent — "" is itself recognised as + // an already-templated segment and maps to itself. + NormalizedRouteSegmentToken = "" + + // tierTokenSast and tierTokenDast are the literal tier discriminators + // hashed in field position 2 of their tiers. They are string literals + // rather than DetectorKind values on purpose: the SAST tier hashes + // "sast" for BOTH sast_reachable and sast_static_only findings, so this + // token must not be confused with the evidence class. + tierTokenSast = "sast" + tierTokenDast = "dast" +) + +// Route-segment templating thresholds. These are part of anvil-fp/v1: changing +// either one changes every DAST digest whose route carries a segment near the +// boundary, and is therefore an anvil-fp/v2 event, not a tuning knob. +// +// The governing asymmetry, from the R.3 ruling: OVER-templating merges two +// genuinely distinct routes into one identity and silently loses a finding on +// upsert against UNIQUE (target_id, fingerprint); UNDER-templating only leaves +// a volatile route un-merged, which the DAST producer can still fix by +// emitting "{id}" itself. Under-templating is the recoverable failure, so both +// thresholds are set high enough that no plausible human-authored path segment +// reaches them. +const ( + // routeHexSegmentMinLen is the length at or above which an all-hex segment + // is treated as an opaque identifier. 16 is chosen because the hex + // alphabet's letters are only a-f: a 16-character English word drawn from + // {a,b,c,d,e,f} plus digits does not exist (the longest such words — + // "defaced", "cabbage" — are seven letters), while every hash form Anvil + // will meet in a URL is at or above it: MD5 is 32, SHA-1 is 40, SHA-256 is + // 64, a dash-free UUID is 32, and a short git object id is 7-12 and so is + // deliberately NOT templated (7 hex characters is also a plausible slug). + routeHexSegmentMinLen = 16 + + // routeOpaqueSegmentMinLen is the length at or above which a mixed + // alphanumeric segment is treated as an opaque identifier. 20 is chosen + // against the longest plausible single-word route segments — + // "recommendations" (15), "internationalization" (20), "misrepresentation" + // (17) — none of which contains a digit, which is why the digit + // requirement in isLongOpaqueRouteSegment carries most of the safety here. + // A base64url session token is 22+ characters and a base32 token is 26+, + // so real opaque tokens clear the bar comfortably. + routeOpaqueSegmentMinLen = 20 +) + +// --------------------------------------------------------------------------- +// Errors +// --------------------------------------------------------------------------- + +// FingerprintError reports an input that cannot be fingerprinted. It names +// the tier and the field, because a bare "invalid input" tells a producer +// nothing about which of eight positional fields it got wrong — and a +// producer that silently emits a wrong digest is exactly the failure this +// package exists to prevent, so every tier function fails loudly rather than +// hashing a degraded input. +type FingerprintError struct { + Tier string // "sast", "dast", "sca", "host", or "" for the shared primitive + Field string // the specification's field name, e.g. "repo_relpath" + Msg string +} + +func (e *FingerprintError) Error() string { + if e.Tier == "" { + return fmt.Sprintf("anvil-fp/v1: %s: %s", e.Field, e.Msg) + } + return fmt.Sprintf("anvil-fp/v1 tier %s: %s: %s", e.Tier, e.Field, e.Msg) +} + +func fpErr(tier, field, msg string) error { + return &FingerprintError{Tier: tier, Field: field, Msg: msg} +} + +// fpRequire returns v, or an error naming the field if v is empty. +func fpRequire(tier, field, v string) (string, error) { + if v == "" { + return "", fpErr(tier, field, "must not be empty") + } + return v, nil +} + +// --------------------------------------------------------------------------- +// The shared primitive: join, guard, hash +// --------------------------------------------------------------------------- + +// Digest joins fields with FingerprintFieldSeparator (U+001F) and returns the +// lowercase hex SHA-256 of the result: exactly FingerprintDigestHexLen (64) +// characters, never truncated. +// +// It rejects any field containing a C0 control character or DEL. U+001F is +// the important case — a field carrying the separator would silently move a +// field boundary and let two different findings collide — but the whole +// control range is rejected because none of them can legitimately appear in +// a canonicalised path, rule id, symbol path, route template, purl, advisory +// id, or normalized match, and a newline in one of those means the caller has +// passed raw, uncanonicalised text. +// +// The exported tier functions below are the only sanctioned field lists. +// Digest is exported for the store and for debugging tooling, not so callers +// can invent a fingerprint shape of their own. +func Digest(fields ...string) (string, error) { + if len(fields) == 0 { + return "", fpErr("", "fields", "at least one field is required") + } + for i, f := range fields { + if idx := strings.IndexFunc(f, isForbiddenControl); idx >= 0 { + what := "a C0 control character" + if f[idx] == FingerprintFieldSeparator[0] { + what = "the U+001F field separator" + } + return "", fpErr("", "fields["+strconv.Itoa(i)+"]", + "contains "+what+"; this would move a field boundary and let two distinct findings collide") + } + } + sum := sha256.Sum256([]byte(strings.Join(fields, FingerprintFieldSeparator))) + return hex.EncodeToString(sum[:]), nil +} + +func isForbiddenControl(r rune) bool { + return r < 0x20 || r == 0x7f +} + +// ValidateDigest reports whether s is a well-formed anvil-fp/v1 digest: +// exactly 64 lowercase hexadecimal characters. Uppercase hex is rejected +// rather than folded, because a store that accepts both would hold two rows +// for one finding and defeat UNIQUE (target_id, fingerprint). +func ValidateDigest(s string) error { + if len(s) != FingerprintDigestHexLen { + return fpErr("", "digest", + fmt.Sprintf("must be exactly %d hex characters, got %d (the digest is never truncated)", + FingerprintDigestHexLen, len(s))) + } + for i := 0; i < len(s); i++ { + c := s[i] + if (c >= '0' && c <= '9') || (c >= 'a' && c <= 'f') { + continue + } + return fpErr("", "digest", "must be lowercase hexadecimal; found "+strconv.QuoteRune(rune(c))+ + " at offset "+strconv.Itoa(i)) + } + return nil +} + +// --------------------------------------------------------------------------- +// Tier SAST +// --------------------------------------------------------------------------- + +// SastInput is the complete input to the SAST fingerprint tier, which covers +// EvidenceClassSastReachable and EvidenceClassSastStaticOnly alike. +// +// There is no line, column, advisory, or evidence-class field here, and there +// must never be one: each of those changes for reasons unrelated to the +// defect's identity. The raw Snippet is accepted only so NormalizeMatch can +// abstract it; the literal text is never hashed. +type SastInput struct { + // TargetID is the stable identifier of the scanned target + // (`target.target_id` rendered as text). Required. + TargetID string + + // RuleIDVersioned is the detector rule id including its ruleset version, + // e.g. "opengrep.go.lang.security.audit.sqli@2026.07". Required. The + // version component is part of identity here: a rule that changed what + // it matches is a different rule, and conflating them would silently + // migrate findings between rule semantics. + RuleIDVersioned string + + // RepoRelPath is the repository-root-relative POSIX path of the file + // containing the match. Required. Canonicalised by CanonicalRepoRelPath + // before hashing so a Windows producer and a Linux producer agree. + RepoRelPath string + + // EnclosingSymbolPath is the fully-qualified enclosing symbol, e.g. + // "pkg/mod.py::ClassA.method_b". MAY be empty: a match in top-level + // module code, a config file, or a template has no enclosing symbol, and + // rejecting those would make them unfingerprintable. An empty value is + // hashed as an empty field, which keeps the field count constant. + EnclosingSymbolPath string + + // Snippet is the RAW matched source text. It is passed through + // NormalizeMatch and only the normalized form is hashed. + Snippet string + + // Ordinal is the 0-based index of this match among all matches of the + // same RuleIDVersioned in the same RepoRelPath whose normalized match is + // IDENTICAL. Use AssignSastOrdinals to compute it. Without it, two + // identical generated or macro-expanded call sites in one file collide + // and one finding is lost on upsert. + Ordinal int +} + +// SastFields returns the ordered field list the SAST tier hashes, exactly as +// plan/40-record-and-storage.md specifies: +// +// target_id ␟ "sast" ␟ rule_id_versioned ␟ repo_relpath +// ␟ enclosing_symbol_path ␟ normalized_match ␟ ordinal +// +// Exported so a test or R.16's conformance harness can assert the field list +// itself, not merely the digest — a wrong field ORDER produces a perfectly +// valid-looking 64-hex digest that is silently incompatible. +func SastFields(in SastInput) ([]string, error) { + const tier = "sast" + + targetID, err := fpRequire(tier, "target_id", in.TargetID) + if err != nil { + return nil, err + } + ruleID, err := fpRequire(tier, "rule_id_versioned", in.RuleIDVersioned) + if err != nil { + return nil, err + } + if in.RepoRelPath == "" { + return nil, fpErr(tier, "repo_relpath", "must not be empty") + } + relPath := CanonicalRepoRelPath(in.RepoRelPath) + if relPath == "" { + return nil, fpErr(tier, "repo_relpath", "canonicalises to the empty string") + } + if in.Snippet == "" { + return nil, fpErr(tier, "normalized_match", "snippet must not be empty") + } + normalized := NormalizeMatch(in.Snippet) + if normalized == "" { + return nil, fpErr(tier, "normalized_match", + "snippet normalises to the empty string (comments and whitespace only); it carries no identity") + } + if in.Ordinal < 0 { + return nil, fpErr(tier, "ordinal", "must not be negative") + } + + return []string{ + targetID, + tierTokenSast, + ruleID, + relPath, + in.EnclosingSymbolPath, + normalized, + strconv.Itoa(in.Ordinal), + }, nil +} + +// Sast returns the anvil-fp/v1 digest for a first-party source finding. +func Sast(in SastInput) (string, error) { + fields, err := SastFields(in) + if err != nil { + return "", err + } + return Digest(fields...) +} + +// --------------------------------------------------------------------------- +// Tier SCA / HOST — one formula, parameterised by detector kind +// --------------------------------------------------------------------------- + +// ScaInput is the input to the SCA tier: a repository dependency that matched +// a vulnerable version range. +// +// There is deliberately no Version field. See "THE VERSION STRING IS NEVER +// HASHED" at the top of this file. +type ScaInput struct { + // TargetID is the scanned target's identifier. Required. + TargetID string + + // AdvisoryID is the canonical advisory identifier as stored in + // `advisory.advisory_id` (e.g. "CVE-2021-44228", "GHSA-jfh8-c2jp-5v3q"). + // Required, and hashed VERBATIM: advisory identifiers are not + // case-normalised here because GHSA identifiers mix cases meaningfully + // and folding them would fork identity against the advisory table. + AdvisoryID string + + // Purl is the package URL. It MAY carry a version, qualifiers, or a + // subpath; PurlBase strips all three before hashing. + Purl string + + // ManifestRelPath is the repo-relative path of the manifest or lockfile + // that declared the dependency, e.g. "services/api/go.mod". Required: + // the same vulnerable package pulled in by two different manifests in a + // monorepo is two findings with two different owners and two different + // fixes. + ManifestRelPath string +} + +// HostInput is the input to the host tier: an operating-system package on the +// scanned host that matched a vulnerable version range. +// +// plan/00-SPINE.md S7 makes the host agent read-only, so every host finding +// carries `remediable_by_agent=false` (enforced by the `finding` table's +// CHECK constraint, not here). It still needs a stable identity so that a +// host finding can be tracked, suppressed, and reported as resolved. +type HostInput struct { + // TargetID is the scanned target's identifier. Required. + TargetID string + + // AdvisoryID is the canonical advisory identifier. Required, verbatim. + AdvisoryID string + + // Purl is the package URL for the host package, e.g. + // "pkg:deb/debian/openssl". Version, qualifiers and subpath are stripped. + Purl string + + // PackageManager is the host package manager, e.g. "apt", "apk", "rpm", + // "dpkg". Required. Lowercased before hashing, because "APT" and "apt" + // are the same manager and a case difference between two scanner + // versions would fork every host finding at once. + PackageManager string + + // HostIdentifier is the package identity as the manager names it, e.g. + // "openssl" or "openssl:amd64". Required, verbatim: architecture and + // suffixes are meaningful to the manager. + HostIdentifier string +} + +// HostLocator composes the host tier's `locator` field, +// ":" (e.g. "apt:openssl"), applying the +// documented lowercasing of the manager segment. +func HostLocator(packageManager, hostIdentifier string) string { + return strings.ToLower(strings.TrimSpace(packageManager)) + ":" + strings.TrimSpace(hostIdentifier) +} + +// packageFields is the single implementation of the shared SCA/host formula: +// +// target_id ␟ detector_kind ␟ advisory_id ␟ purl_base ␟ locator +// +// Written once and parameterised rather than copied, so the two tiers cannot +// drift apart the way research/07 and research/18 did. +func packageFields(tier string, kind DetectorKind, targetID, advisoryID, purl, locator string) ([]string, error) { + tid, err := fpRequire(tier, "target_id", targetID) + if err != nil { + return nil, err + } + adv, err := fpRequire(tier, "advisory_id", advisoryID) + if err != nil { + return nil, err + } + if purl == "" { + return nil, fpErr(tier, "purl_base", "purl must not be empty") + } + base, err := PurlBase(purl) + if err != nil { + return nil, err + } + if locator == "" { + return nil, fpErr(tier, "locator", "must not be empty") + } + return []string{tid, string(kind), adv, base, locator}, nil +} + +// ScaFields returns the ordered field list the SCA tier hashes. +func ScaFields(in ScaInput) ([]string, error) { + const tier = "sca" + if in.ManifestRelPath == "" { + return nil, fpErr(tier, "locator", "manifest_relpath must not be empty") + } + locator := CanonicalRepoRelPath(in.ManifestRelPath) + if locator == "" { + return nil, fpErr(tier, "locator", "manifest_relpath canonicalises to the empty string") + } + return packageFields(tier, DetectorKindSCA, in.TargetID, in.AdvisoryID, in.Purl, locator) +} + +// Sca returns the anvil-fp/v1 digest for a repository dependency finding. +func Sca(in ScaInput) (string, error) { + fields, err := ScaFields(in) + if err != nil { + return "", err + } + return Digest(fields...) +} + +// HostFields returns the ordered field list the host tier hashes. +func HostFields(in HostInput) ([]string, error) { + const tier = "host" + if strings.TrimSpace(in.PackageManager) == "" { + return nil, fpErr(tier, "locator", "package_manager must not be empty") + } + if strings.ContainsRune(in.PackageManager, ':') { + return nil, fpErr(tier, "locator", + "package_manager must not contain ':'; it is the locator's own delimiter") + } + if strings.TrimSpace(in.HostIdentifier) == "" { + return nil, fpErr(tier, "locator", "host_identifier must not be empty") + } + return packageFields(tier, DetectorKindHost, in.TargetID, in.AdvisoryID, in.Purl, + HostLocator(in.PackageManager, in.HostIdentifier)) +} + +// Host returns the anvil-fp/v1 digest for a host package finding. +func Host(in HostInput) (string, error) { + fields, err := HostFields(in) + if err != nil { + return "", err + } + return Digest(fields...) +} + +// --------------------------------------------------------------------------- +// Tier DAST +// --------------------------------------------------------------------------- + +// DastInput is the input to the DAST tier. +// +// There is no host, port, scheme, payload, session token, or timestamp field, +// and there must never be one. A redeployed container, a rotated ephemeral +// port, or a staging-to-prod move must not fork identity, and a fuzzer +// varying its payload must not mint N findings for one bug. +type DastInput struct { + // TargetID is the scanned target's identifier. Required. + TargetID string + + // RuleIDVersioned is the detector rule id including its version, e.g. + // "nuclei:CVE-2021-44228@a1b2c3d". Required. + RuleIDVersioned string + + // HTTPMethod is the request method. Required; uppercased before hashing + // (RFC 9110 methods are case-sensitive and canonically uppercase, and a + // producer sending "get" must not fork identity from one sending "GET"). + HTTPMethod string + + // RouteTemplate is the observed request path. Required. It may be the + // CONCRETE path the producer requested ("/api/users/12345/orders") or a + // path the producer has already templated in any of the three common + // syntaxes ("{id}", ":id", ""); CanonicalRouteTemplate derives the + // hashed template from either, so the caller does not have to agree with + // any other caller about which form to use. It also strips any query + // string or fragment — those carry concrete values, which is exactly what + // templating exists to remove. + // + // Templating is deliberately NOT the producer's job. See + // CanonicalRouteTemplate for the ruling and the reason. + RouteTemplate string + + // InjectionPoint is WHERE the payload was injected. Required; must be a + // legal InjectionPoint literal from contract.go. + InjectionPoint InjectionPoint + + // ParamName is the name of the injected parameter. MAY be empty: a + // whole-body or raw-request injection has no single named parameter, and + // rejecting those would make them unfingerprintable. This is a parameter + // NAME, never a parameter value. + ParamName string + + // EvidenceSignal is HOW the vulnerability was observed — research/18's + // contribution to this tier. Required; must be a legal EvidenceSignal + // literal from contract.go. It is independent of InjectionPoint: an SQL + // injection proved by a database error string and one proved by a timing + // side channel on the same parameter are different findings with + // different remediation evidence. + EvidenceSignal EvidenceSignal +} + +// DastFields returns the ordered field list the DAST tier hashes, exactly as +// plan/40-record-and-storage.md specifies: +// +// target_id ␟ "dast" ␟ rule_id_versioned ␟ http_method ␟ route_template +// ␟ injection_point ␟ param_name ␟ evidence_class_detail +func DastFields(in DastInput) ([]string, error) { + const tier = "dast" + + targetID, err := fpRequire(tier, "target_id", in.TargetID) + if err != nil { + return nil, err + } + ruleID, err := fpRequire(tier, "rule_id_versioned", in.RuleIDVersioned) + if err != nil { + return nil, err + } + if strings.TrimSpace(in.HTTPMethod) == "" { + return nil, fpErr(tier, "http_method", "must not be empty") + } + method := strings.ToUpper(strings.TrimSpace(in.HTTPMethod)) + if strings.ContainsFunc(method, unicode.IsSpace) { + return nil, fpErr(tier, "http_method", "must be a single token") + } + if in.RouteTemplate == "" { + return nil, fpErr(tier, "route_template", "must not be empty") + } + route := CanonicalRouteTemplate(in.RouteTemplate) + if route == "" { + return nil, fpErr(tier, "route_template", "canonicalises to the empty string") + } + if err := ValidateInjectionPoint(string(in.InjectionPoint)); err != nil { + return nil, fpErr(tier, "injection_point", err.Error()) + } + if err := ValidateEvidenceSignal(string(in.EvidenceSignal)); err != nil { + return nil, fpErr(tier, "evidence_class_detail", err.Error()) + } + + return []string{ + targetID, + tierTokenDast, + ruleID, + method, + route, + string(in.InjectionPoint), + in.ParamName, + string(in.EvidenceSignal), + }, nil +} + +// Dast returns the anvil-fp/v1 digest for a dynamically-confirmed finding. +func Dast(in DastInput) (string, error) { + fields, err := DastFields(in) + if err != nil { + return "", err + } + return Digest(fields...) +} + +// --------------------------------------------------------------------------- +// Canonicalisation helpers +// --------------------------------------------------------------------------- + +// CanonicalRepoRelPath normalises a repository-relative path to the single +// POSIX form the specification assumes ("repo_relpath — POSIX, +// repo-root-relative", research/07 §3): +// +// - backslashes become forward slashes, so a Windows producer and a Linux +// producer scanning the same repository agree; +// - runs of slashes collapse to one; +// - a leading "./" or "/" is removed, so "./cmd/x.go", "/cmd/x.go" and +// "cmd/x.go" are one path; +// - a trailing slash is removed. +// +// It deliberately does NOT case-fold: POSIX paths are case-sensitive, and +// folding would merge two genuinely distinct files on a case-sensitive +// checkout. It also does not resolve ".." — a path escaping the repo root is +// the caller's bug and must not be silently rewritten into a different file. +func CanonicalRepoRelPath(p string) string { + p = strings.ReplaceAll(p, "\\", "/") + for strings.Contains(p, "//") { + p = strings.ReplaceAll(p, "//", "/") + } + for strings.HasPrefix(p, "./") { + p = p[2:] + } + p = strings.TrimPrefix(p, "/") + p = strings.TrimSuffix(p, "/") + return p +} + +// CanonicalRouteTemplate DERIVES the hashed `route_template` from whatever +// route a DAST producer observed. The specification defines the field as a +// derived value — "numeric/UUID/hash path segments replaced with a placeholder +// token" — and this function is where that derivation happens. +// +// WHY IT HAPPENS HERE AND NOT IN THE PRODUCER (R.3 ruling, 2026-08-08). Area +// 40 owns the fingerprint, so area 40 canonicalises. A DAST producer emits +// whatever route it observed; if templating were the producer's job, then two +// producers seeing one defect at /api/users/12345/orders would emit +// "/api/users/12345/orders", "/api/users/{id}/orders" and +// "/api/users/:id/orders" — three digests, one defect, no error, regression +// matching silently dead. The DAST tier is the one that earns "verified fixed" +// under plan/00-SPINE.md S7, and a reproduction that cannot be matched to its +// prior finding cannot prove a fix. +// +// The steps, in order: +// +// 1. anything from the first '?' or '#' is dropped. A query string or +// fragment in a "template" carries concrete values — the very thing +// templating removes — and injection_point plus param_name already record +// which query parameter was targeted; +// 2. backslashes become forward slashes; +// 3. runs of slashes collapse to one; +// 4. a leading '/' is added if absent; +// 5. a trailing '/' is removed, except on the root path "/"; +// 6. every VOLATILE path segment is replaced by NormalizedRouteSegmentToken. +// A segment is volatile when it is already templated in a producer's own +// syntax ("{id}", ":id", ""), is all ASCII digits, is a UUID, is a +// long all-hex run, or is a long mixed alphanumeric run containing a +// digit. See isVolatileRouteSegment for the exact predicates. +// +// Case is preserved on non-volatile segments: URL paths are case-sensitive and +// "/Search" is a different route from "/search" on most servers. The UUID and +// hex predicates are themselves case-insensitive, because the same identifier +// rendered in upper and lower hex is the same identifier. +// +// Percent-encoding is deliberately NOT decoded: decoding could introduce a '/' +// and change the segment structure, and a producer that percent-encodes a +// whole segment has emitted a different route. +func CanonicalRouteTemplate(route string) string { + if i := strings.IndexAny(route, "?#"); i >= 0 { + route = route[:i] + } + route = strings.ReplaceAll(route, "\\", "/") + for strings.Contains(route, "//") { + route = strings.ReplaceAll(route, "//", "/") + } + if route == "" { + return "" + } + if !strings.HasPrefix(route, "/") { + route = "/" + route + } + if len(route) > 1 { + route = strings.TrimSuffix(route, "/") + } + if route == "/" { + return "/" + } + + segs := strings.Split(route[1:], "/") + for i, s := range segs { + if isVolatileRouteSegment(s) { + segs[i] = NormalizedRouteSegmentToken + } + } + return "/" + strings.Join(segs, "/") +} + +// isVolatileRouteSegment reports whether one path segment carries a concrete +// instance identifier rather than route structure, and must therefore be +// replaced by NormalizedRouteSegmentToken. +// +// Every predicate is conservative by construction — see the threshold +// constants for the asymmetry that motivates it. +func isVolatileRouteSegment(s string) bool { + switch { + case s == "": + return false + case isRoutePlaceholderSegment(s): + return true + case isAllASCIIDigits(s): + return true + case isUUIDRouteSegment(s): + return true + case isLongHexRouteSegment(s): + return true + case isLongOpaqueRouteSegment(s): + return true + default: + return false + } +} + +// isRoutePlaceholderSegment recognises a segment a producer has ALREADY +// templated, in any of the three syntaxes in common use: OpenAPI/ASP.NET +// "{id}", Express/Rails/Sinatra ":id", and Flask/Werkzeug "". +// +// Normalising all three onto one token is the point: a DAST crawler, an +// OpenAPI document checked into the repo, and a route table exported from a +// framework will disagree about which syntax to use for the same route, and +// that disagreement must not fork identity. +// +// The "" form also makes CanonicalRouteTemplate idempotent, because +// NormalizedRouteSegmentToken is itself of that form. +func isRoutePlaceholderSegment(s string) bool { + if len(s) < 2 { + return false + } + if s[0] == '{' && s[len(s)-1] == '}' { + return true + } + if s[0] == '<' && s[len(s)-1] == '>' { + return true + } + return s[0] == ':' +} + +func isAllASCIIDigits(s string) bool { + if s == "" { + return false + } + for i := 0; i < len(s); i++ { + if s[i] < '0' || s[i] > '9' { + return false + } + } + return true +} + +func isASCIIHexDigit(c byte) bool { + return (c >= '0' && c <= '9') || (c >= 'a' && c <= 'f') || (c >= 'A' && c <= 'F') +} + +// isUUIDRouteSegment recognises the canonical 8-4-4-4-12 hyphenated form, +// case-insensitively. Braced ("{...}") and URN ("urn:uuid:...") forms are not +// recognised here: the first is already caught by isRoutePlaceholderSegment +// and the second is not a bare path segment. +func isUUIDRouteSegment(s string) bool { + if len(s) != 36 { + return false + } + for i := 0; i < len(s); i++ { + switch i { + case 8, 13, 18, 23: + if s[i] != '-' { + return false + } + default: + if !isASCIIHexDigit(s[i]) { + return false + } + } + } + return true +} + +// isLongHexRouteSegment recognises a run of at least routeHexSegmentMinLen hex +// characters: a dash-free UUID, an MD5/SHA digest, an object id. +func isLongHexRouteSegment(s string) bool { + if len(s) < routeHexSegmentMinLen { + return false + } + for i := 0; i < len(s); i++ { + if !isASCIIHexDigit(s[i]) { + return false + } + } + return true +} + +// isLongOpaqueRouteSegment recognises a base32/base64-ish opaque token: at +// least routeOpaqueSegmentMinLen characters, ASCII alphanumeric throughout, +// and carrying BOTH a digit and a letter. +// +// The three restrictions each buy something specific, and dropping any of them +// over-templates: +// +// - alphanumeric only (no '-', '_', '.') keeps slugs out. A hyphenated slug +// like "release-notes-2026-08" is 21 characters and carries a digit; it is +// route structure, not an identifier, and merging every dated release note +// onto one digest would be exactly the over-templating failure the ruling +// warns about. Base64url tokens that use '-' or '_' are therefore left +// un-templated: under-templating is the recoverable direction. +// - a digit is required, which is what excludes the long all-letter words +// that do reach 20 characters ("internationalization"). +// - a letter is required, so that a purely numeric run is attributed to the +// all-digits rule rather than this one; the outcome is the same token, but +// the two rules stay independently testable. +func isLongOpaqueRouteSegment(s string) bool { + if len(s) < routeOpaqueSegmentMinLen { + return false + } + hasDigit, hasLetter := false, false + for i := 0; i < len(s); i++ { + c := s[i] + switch { + case c >= '0' && c <= '9': + hasDigit = true + case (c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z'): + hasLetter = true + default: + return false + } + } + return hasDigit && hasLetter +} + +// PurlBase reduces a package URL to its version-free base: +// "pkg:type/namespace/name". It strips, in order, any subpath ('#'), any +// qualifiers ('?'), and any version ('@'). +// +// This is the enforcement point for the rule that the version string is NEVER +// hashed. A caller passing "pkg:npm/lodash@4.17.20" and a caller passing +// "pkg:npm/lodash" produce the same fingerprint, so bumping a dependency +// inside the vulnerable range does not mint a new finding. +// +// The '@' scan starts after "pkg:" and is safe against namespaced packages: +// the purl specification requires a literal '@' inside a namespace or name to +// be percent-encoded as "%40" (as in "pkg:npm/%40angular/core@13.0.0"), so +// the first raw '@' can only be the version delimiter. +// +// The scheme and type segments are lowercased because the purl specification +// defines both as case-insensitive with a lowercase canonical form; the +// namespace and name are left alone because their case-sensitivity is +// type-dependent and folding them could merge two distinct packages. +func PurlBase(purl string) (string, error) { + p := strings.TrimSpace(purl) + if p == "" { + return "", fpErr("", "purl_base", "must not be empty") + } + if len(p) < 4 || !strings.EqualFold(p[:4], "pkg:") { + return "", fpErr("", "purl_base", "must be a package URL beginning with \"pkg:\", got "+strconv.Quote(purl)) + } + + rest := p[4:] + if i := strings.IndexByte(rest, '#'); i >= 0 { + rest = rest[:i] + } + if i := strings.IndexByte(rest, '?'); i >= 0 { + rest = rest[:i] + } + if i := strings.IndexByte(rest, '@'); i >= 0 { + rest = rest[:i] + } + rest = strings.TrimSuffix(rest, "/") + if rest == "" { + return "", fpErr("", "purl_base", "carries no type or name after \"pkg:\": "+strconv.Quote(purl)) + } + + // Lowercase only the type segment (everything up to the first '/'). + if i := strings.IndexByte(rest, '/'); i >= 0 { + rest = strings.ToLower(rest[:i]) + rest[i:] + } else { + return "", fpErr("", "purl_base", "carries a type but no name: "+strconv.Quote(purl)) + } + + return "pkg:" + rest, nil +} + +// --------------------------------------------------------------------------- +// Ordinal assignment +// --------------------------------------------------------------------------- + +// SastCandidate pairs a SastInput with the source position used ONLY to order +// identical matches deterministically. Line and Column are never hashed and +// never reach Digest; they exist because "the 0-based index of this match +// among identical matches" needs a total order, and source order is the only +// one a producer and a re-scan both agree on. +// +// KNOWN LIMITATION, inherited from the specification rather than chosen here: +// inserting a THIRD identical call site above two existing ones shifts their +// ordinals and therefore their digests. research/07 §3's matching cascade +// (exact hit, then rule+path+line_hash, then rule+symbol_hash) is what +// recovers identity in that case; the fingerprint alone cannot. The +// alternative — dropping the ordinal — silently LOSES one of the two findings +// on upsert against UNIQUE (target_id, fingerprint), which is worse. +type SastCandidate struct { + Input SastInput + Line int // ordering only; never hashed + Column int // ordering only; never hashed +} + +// AssignSastOrdinals returns a copy of cands' inputs with Ordinal set, in the +// SAME order as cands. Ordinals are assigned per +// (TargetID, RuleIDVersioned, RepoRelPath, normalized match) group in +// ascending (Line, Column, original index) order. +// +// The group key adds TargetID to the specification's (rule, path, normalized) +// key because the specification's grouping is implicitly per-target; passing +// two targets' candidates in one slice would otherwise cross-index them. +// +// Any input whose fields are invalid is returned as an error rather than +// silently given ordinal 0, because a bad ordinal is invisible in the digest. +func AssignSastOrdinals(cands []SastCandidate) ([]SastInput, error) { + type keyed struct { + idx int + key string + } + + out := make([]SastInput, len(cands)) + order := make([]keyed, 0, len(cands)) + + for i, c := range cands { + // Validate through the real field builder so an input that cannot be + // fingerprinted is rejected here rather than at hash time. + if _, err := SastFields(c.Input); err != nil { + return nil, fmt.Errorf("candidate %d: %w", i, err) + } + out[i] = c.Input + order = append(order, keyed{ + idx: i, + key: strings.Join([]string{ + c.Input.TargetID, + c.Input.RuleIDVersioned, + CanonicalRepoRelPath(c.Input.RepoRelPath), + NormalizeMatch(c.Input.Snippet), + }, FingerprintFieldSeparator), + }) + } + + sort.SliceStable(order, func(a, b int) bool { + if order[a].key != order[b].key { + return order[a].key < order[b].key + } + ca, cb := cands[order[a].idx], cands[order[b].idx] + if ca.Line != cb.Line { + return ca.Line < cb.Line + } + if ca.Column != cb.Column { + return ca.Column < cb.Column + } + return order[a].idx < order[b].idx + }) + + prevKey := "" + ordinal := 0 + for i, k := range order { + if i == 0 || k.key != prevKey { + ordinal = 0 + prevKey = k.key + } + out[k.idx].Ordinal = ordinal + ordinal++ + } + + return out, nil +} + +// --------------------------------------------------------------------------- +// Match normalization +// --------------------------------------------------------------------------- + +// NormalizeMatch abstracts a raw matched source snippet into the form the +// SAST tier hashes. It is modelled on Semgrep's `match_based_id` +// (research/07 §3 [S3]), which is the externally-verified mechanism for +// surviving reindentation and metavariable-only edits. +// +// The algorithm, in one pass, left to right. Any re-implementation (R.16) +// must reproduce it exactly: +// +// 1. CRLF and CR are folded to LF. +// 2. A whitespace run emits a single space. +// 3. A line comment ("//..." or "#..." to end of line) and a block comment +// ("/*...*/", unterminated meaning to end of input) are dropped and emit +// a single space in their place. +// 4. A string literal delimited by ", ' or ` emits NormalizedStringToken. +// Backslash escapes are honoured inside " and ' but not inside `. +// 5. A token starting with an ASCII digit emits NormalizedNumberToken. It +// consumes following letters, digits, '_' and '.', plus a '+' or '-' +// immediately after an 'e' or 'E', so 0xFF, 1_000, 3.14f and 1e-9 are all +// one token. +// 6. An identifier ([\p{L}_$][\p{L}\p{N}_$]*) emits: +// a. itself, if it is a reserved word (the union list below); +// b. itself, if the preceding non-space output ends in ".", "->" or "::" +// — it is a member or qualified name, which is API surface, not churn; +// c. itself, if the next non-space input is "::" — it is a namespace or +// type qualifier, which is API surface for the same reason; +// d. itself, if the next non-space input character is "(" — it is a +// callee name, which is the sink the rule actually matched; +// e. otherwise "$N", where N counts distinct such identifiers in +// first-occurrence order, with every later occurrence of the same +// spelling mapping to the same "$N". +// 7. Any other character is emitted verbatim. +// 8. The result is trimmed of leading and trailing spaces. +// +// WHY 6(b), 6(c) AND 6(d) EXIST — the specification says "replace LOCAL +// identifiers", and this is what "local" is taken to mean. Replacing every +// identifier would normalise `request.getParameter(userInput)` and +// `config.getName(key)` to the same string, destroying nearly all +// discriminating power and pushing the whole burden of distinguishing +// findings onto `ordinal`, which is the least stable field in the tier. +// Preserving member, qualifier and callee names keeps the sink identifiable +// while still abstracting the variable names that a refactor renames. +// +// WHY 6(c) TREATS "::" DIFFERENTLY FROM "." AND "->". The LEFT operand of +// "::" is a namespace or type name in every language that has the operator +// (C++, Rust, PHP, Ruby) — it is never a local variable, so abstracting it +// violates "replace local identifiers" outright and collapses +// `Ns::Helper(v)` and `Other::Helper(v)` — two calls into two different +// namespaces — onto one digest. The left operand of "." or "->" is usually a +// receiver bound to a local (`db.Query(q)`, `p->Field`), so it is abstracted; +// a lexer cannot tell a Go package qualifier from a receiver variable, and +// guessing wrong in the other direction would fork the digest of unchanged +// code, which is the worse failure. +// +// KNOWN, ACCEPTED LIMITATIONS. This is one language-agnostic lexer, not N +// parsers, because Anvil's SAST tier is an opengrep subprocess that returns +// text (plan/00-SPINE.md S12: opengrep "is an OCaml CLI with zero bindings in +// any language"). Determinism, not semantic perfection, is what identity +// needs — the same snippet must normalise the same way on every scan, and it +// does: +// +// - '#' is treated as a line comment, so a C/C++ preprocessor directive or +// a C# '#region' inside a snippet is dropped. Match snippets rarely +// contain them. +// - "'" is treated as a string delimiter, so a Rust lifetime ('static) or a +// Lisp quote consumes to the next "'". +// - the reserved-word list is a UNION across languages, so an identifier +// named `class` in a language where it is not reserved is preserved +// rather than abstracted. +// +// Each of these is stable under re-scan, which is the property that matters. +func NormalizeMatch(snippet string) string { + src := strings.ReplaceAll(snippet, "\r\n", "\n") + src = strings.ReplaceAll(src, "\r", "\n") + + in := []rune(src) + n := len(in) + out := make([]rune, 0, n) + + emit := func(s string) { + out = append(out, []rune(s)...) + } + emitSpace := func() { + if len(out) > 0 && out[len(out)-1] != ' ' { + out = append(out, ' ') + } + } + // endsWithSelector reports whether the output so far ends in a member or + // qualified-name selector, ignoring trailing spaces. + endsWithSelector := func() bool { + end := len(out) + for end > 0 && out[end-1] == ' ' { + end-- + } + if end == 0 { + return false + } + if out[end-1] == '.' { + return true + } + if end >= 2 && out[end-2] == '-' && out[end-1] == '>' { + return true + } + if end >= 2 && out[end-2] == ':' && out[end-1] == ':' { + return true + } + return false + } + + metavars := make(map[string]string) + nextMetavar := 1 + + i := 0 + for i < n { + c := in[i] + + switch { + case unicode.IsSpace(c): + for i < n && unicode.IsSpace(in[i]) { + i++ + } + emitSpace() + + case c == '/' && i+1 < n && in[i+1] == '/': + for i < n && in[i] != '\n' { + i++ + } + emitSpace() + + case c == '#': + for i < n && in[i] != '\n' { + i++ + } + emitSpace() + + case c == '/' && i+1 < n && in[i+1] == '*': + i += 2 + for i < n { + if in[i] == '*' && i+1 < n && in[i+1] == '/' { + i += 2 + break + } + i++ + } + emitSpace() + + case c == '"' || c == '\'' || c == '`': + quote := c + i++ + for i < n { + if in[i] == '\\' && quote != '`' { + i += 2 + continue + } + if in[i] == quote { + i++ + break + } + i++ + } + emit(NormalizedStringToken) + + case c >= '0' && c <= '9': + for i < n { + r := in[i] + if unicode.IsLetter(r) || unicode.IsDigit(r) || r == '_' || r == '.' { + i++ + continue + } + if (r == '+' || r == '-') && i > 0 && (in[i-1] == 'e' || in[i-1] == 'E') { + i++ + continue + } + break + } + emit(NormalizedNumberToken) + + case isIdentStart(c): + j := i + for j < n && isIdentPart(in[j]) { + j++ + } + word := string(in[i:j]) + i = j + + switch { + case fingerprintReservedWords[word]: + emit(word) + case endsWithSelector(): + emit(word) + case nextNonSpaceIsScopeResolution(in, i): + emit(word) + case nextNonSpaceIsCallOpen(in, i): + emit(word) + default: + mv, ok := metavars[word] + if !ok { + mv = NormalizedMetavarPrefix + strconv.Itoa(nextMetavar) + nextMetavar++ + metavars[word] = mv + } + emit(mv) + } + + default: + out = append(out, c) + i++ + } + } + + return strings.TrimSpace(string(out)) +} + +func isIdentStart(r rune) bool { + return r == '_' || r == '$' || unicode.IsLetter(r) +} + +func isIdentPart(r rune) bool { + return r == '_' || r == '$' || unicode.IsLetter(r) || unicode.IsDigit(r) +} + +// nextNonSpaceIsCallOpen reports whether the next non-space character at or +// after i is '(' — i.e. the identifier just consumed is a callee name. +func nextNonSpaceIsCallOpen(in []rune, i int) bool { + for i < len(in) && unicode.IsSpace(in[i]) { + i++ + } + return i < len(in) && in[i] == '(' +} + +// nextNonSpaceIsScopeResolution reports whether the next non-space characters +// at or after i are "::" — i.e. the identifier just consumed is the namespace +// or type qualifier on the left of a scope-resolution operator. See rule 6(c) +// in NormalizeMatch's contract: that operand is never a local variable, so +// abstracting it would both violate "replace local identifiers" and merge +// calls into two different namespaces onto one digest. +func nextNonSpaceIsScopeResolution(in []rune, i int) bool { + for i < len(in) && unicode.IsSpace(in[i]) { + i++ + } + return i+1 < len(in) && in[i] == ':' && in[i+1] == ':' +} + +// fingerprintReservedWords is the union of keywords, literal keywords and +// primitive type names across the languages Anvil's SAST tier covers (Go, +// Java, C#, C/C++, JavaScript/TypeScript, Python, Ruby, PHP). Identifiers in +// this set are preserved verbatim rather than abstracted to $N. +// +// It is a UNION on purpose. A per-language list would require knowing the +// language at fingerprint time, which the opengrep subprocess boundary does +// not reliably give us, and a wrong language guess would change the digest of +// unchanged code. A union is a fixed, deterministic function of the token +// text alone. Adding or removing an entry CHANGES EVERY SAST FINGERPRINT and +// is therefore an anvil-fp/v2 event, not a maintenance edit. +var fingerprintReservedWords = map[string]bool{ + // Literal keywords and self-references, across all languages. + "true": true, "false": true, "null": true, "nil": true, "none": true, + "None": true, "True": true, "False": true, "NULL": true, "nullptr": true, + "undefined": true, "self": true, "this": true, "super": true, "cls": true, + "iota": true, "base": true, + + // Control flow and declaration keywords. + "abstract": true, "and": true, "as": true, "assert": true, "async": true, + "await": true, "begin": true, "break": true, "case": true, "catch": true, + "chan": true, "checked": true, "class": true, "clone": true, "const": true, + "constexpr": true, "continue": true, "debugger": true, "declare": true, + "def": true, "default": true, "defer": true, "del": true, "delete": true, + "do": true, "echo": true, "elif": true, "else": true, "elseif": true, + "elsif": true, "end": true, "endforeach": true, "endif": true, + "endwhile": true, "ensure": true, "enum": true, "except": true, + "exit": true, "explicit": true, "export": true, "extends": true, + "extern": true, "fallthrough": true, "final": true, "finally": true, + "fn": true, "for": true, "foreach": true, "friend": true, "from": true, + "func": true, "function": true, "global": true, "go": true, "goto": true, + "if": true, "implements": true, "implicit": true, "import": true, + "in": true, "include": true, "include_once": true, "inline": true, + "instanceof": true, "insteadof": true, "interface": true, "internal": true, + "is": true, "keyof": true, "lambda": true, "let": true, "lock": true, + "match": true, "module": true, "mutable": true, "namespace": true, + "native": true, "new": true, "nonlocal": true, "not": true, + "operator": true, "or": true, "out": true, "override": true, + "package": true, "params": true, "pass": true, "print": true, + "private": true, "protected": true, "public": true, "raise": true, + "range": true, "readonly": true, "redo": true, "ref": true, + "register": true, "require": true, "require_once": true, + "require_relative": true, "rescue": true, "retry": true, "return": true, + "sealed": true, "select": true, "signed": true, "sizeof": true, + "stackalloc": true, "static": true, "strictfp": true, "struct": true, + "switch": true, "synchronized": true, "template": true, "throw": true, + "throws": true, "trait": true, "transient": true, "try": true, + "type": true, "typedef": true, "typeof": true, "unchecked": true, + "union": true, "unless": true, "unsafe": true, "unsigned": true, + "until": true, "use": true, "using": true, "var": true, "virtual": true, + "volatile": true, "when": true, "while": true, "with": true, "xor": true, + "yield": true, + + // Primitive and built-in type names. + "any": true, "bigint": true, "bool": true, "boolean": true, "byte": true, + "char": true, "complex64": true, "complex128": true, "decimal": true, + "double": true, "error": true, "float": true, "float32": true, + "float64": true, "int": true, "int8": true, "int16": true, "int32": true, + "int64": true, "long": true, "never": true, "object": true, "rune": true, + "sbyte": true, "short": true, "string": true, "symbol": true, + "uint": true, "uint8": true, "uint16": true, "uint32": true, + "uint64": true, "uintptr": true, "ulong": true, "unknown": true, + "ushort": true, "void": true, "wchar_t": true, +} diff --git a/internal/record/fingerprint_spec_test.go b/internal/record/fingerprint_spec_test.go new file mode 100644 index 0000000..9a7bbea --- /dev/null +++ b/internal/record/fingerprint_spec_test.go @@ -0,0 +1,349 @@ +package record + +// fingerprint_spec_test.go — the test that keeps FINGERPRINT-SPEC.md honest. +// +// R.3's CRITIQUE-01.md, blocker 1: `normalized_match` was defined only in Go. +// The critic re-implemented plan/40-record-and-storage.md's four-clause spec +// text in Python and got 55e27b07… where the committed golden for sast-01 is +// 13c60ccf…. The orchestrator ruled that the implementation is right and the +// specification incomplete, and that the fix is to write the specification +// down completely and IN TREE — plan/ is gitignored, and a second producer +// working from a clone must be able to read it. +// +// internal/record/FINGERPRINT-SPEC.md is that document. This file is the +// second half of the ruling: "add a test that keeps the spec honest … A spec +// that can drift from the code silently is the same defect one level up." +// +// What is checkable mechanically is checked here: +// +// - the reserved-word list in the document is EXACTLY the one in +// fingerprint.go — same set, no extra, no missing, sorted, no duplicates, +// and the count the prose claims; +// - every algorithm constant the document prints equals the constant in the +// code; +// - the document still carries the anvil-fp/v2 rule, which is the sentence +// that stops someone treating any of the above as a maintenance edit. +// +// What is NOT checkable mechanically — the prose describing the scan order, +// the identifier-disposition clauses and their reasons — is guarded instead by +// the corpus lock in fingerprint_test.go and, eventually, by R.16's +// independent oracle re-implemented FROM THIS DOCUMENT. + +import ( + "fmt" + "os" + "regexp" + "sort" + "strconv" + "strings" + "testing" +) + +const fingerprintSpecPath = "FINGERPRINT-SPEC.md" + +// specBlock returns the lines between "" and +// "", with markdown code fences removed. The markers are +// HTML comments so they are invisible when the document is rendered but +// unambiguous to parse. +func specBlock(t *testing.T, doc, name string) []string { + t.Helper() + + begin := "" + end := "" + + i := strings.Index(doc, begin) + if i < 0 { + t.Fatalf("%s: missing marker %q; the machine-checked %s block is how this "+ + "document is kept from drifting away from fingerprint.go", + fingerprintSpecPath, begin, name) + } + rest := doc[i+len(begin):] + j := strings.Index(rest, end) + if j < 0 { + t.Fatalf("%s: marker %q has no matching %q", fingerprintSpecPath, begin, end) + } + + var out []string + for _, line := range strings.Split(rest[:j], "\n") { + if strings.HasPrefix(strings.TrimSpace(line), "```") { + continue + } + if strings.TrimSpace(line) == "" { + continue + } + out = append(out, line) + } + if len(out) == 0 { + t.Fatalf("%s: block %s is empty", fingerprintSpecPath, name) + } + return out +} + +func readFingerprintSpec(t *testing.T) string { + t.Helper() + b, err := os.ReadFile(fingerprintSpecPath) + if err != nil { + t.Fatalf("reading %s: %v\nThis document is the authoritative definition of anvil-fp/v1 "+ + "and must be in tree: plan/ is gitignored, so a second producer working from a clone "+ + "has nothing else to implement from.", fingerprintSpecPath, err) + } + return strings.ReplaceAll(string(b), "\r\n", "\n") +} + +// TestSpecReservedWordListMatchesTheCode is the ruling's named test. The +// reserved-word list is the single largest undocumented input to +// normalized_match — ~200 entries, every one of which changes the digest of +// any snippet containing it — and it is the reason the critic's from-the-text +// re-implementation diverged. +func TestSpecReservedWordListMatchesTheCode(t *testing.T) { + doc := readFingerprintSpec(t) + lines := specBlock(t, doc, "ANVIL-FP-RESERVED-WORDS") + + var documented []string + for _, line := range lines { + documented = append(documented, strings.Fields(line)...) + } + + // No duplicates. A duplicate is invisible in a set comparison but means + // the human-readable count is wrong and the list has been edited carelessly. + seen := map[string]bool{} + for _, w := range documented { + if seen[w] { + t.Errorf("%s: reserved word %q is listed twice", fingerprintSpecPath, w) + } + seen[w] = true + } + + // Sorted in byte order. Not a correctness property of the algorithm, but it + // makes an added or removed entry a one-line diff instead of a hunt. + if !sort.StringsAreSorted(documented) { + t.Errorf("%s: the reserved-word block is not sorted in byte order; "+ + "sort it so that adding or removing an entry is a readable diff", fingerprintSpecPath) + } + + // The set must be EXACTLY fingerprint.go's. + inCode := make(map[string]bool, len(fingerprintReservedWords)) + for w, v := range fingerprintReservedWords { + if !v { + t.Fatalf("fingerprintReservedWords[%q] is false; every entry must be true", w) + } + inCode[w] = true + } + + var missing, extra []string + for w := range inCode { + if !seen[w] { + missing = append(missing, w) + } + } + for w := range seen { + if !inCode[w] { + extra = append(extra, w) + } + } + sort.Strings(missing) + sort.Strings(extra) + + if len(missing) > 0 { + t.Errorf("%s: %d reserved word(s) are in fingerprint.go but NOT documented: %q\n"+ + "A second producer implementing from this document would abstract them to $N and "+ + "emit a different digest for unchanged code.", + fingerprintSpecPath, len(missing), missing) + } + if len(extra) > 0 { + t.Errorf("%s: %d reserved word(s) are documented but NOT in fingerprint.go: %q\n"+ + "A second producer would preserve them verbatim where this implementation "+ + "abstracts them.", + fingerprintSpecPath, len(extra), extra) + } + + // The count the prose claims must be the real one. + countRe := regexp.MustCompile(`\*\*(\d+) entries\*\*`) + m := countRe.FindStringSubmatch(doc) + if m == nil { + t.Fatalf("%s: the reserved-word section must state the entry count as \"**N entries**\"", + fingerprintSpecPath) + } + claimed, err := strconv.Atoi(m[1]) + if err != nil { + t.Fatalf("%s: unparseable entry count %q", fingerprintSpecPath, m[1]) + } + if claimed != len(inCode) { + t.Errorf("%s: claims %d reserved words, fingerprint.go has %d", + fingerprintSpecPath, claimed, len(inCode)) + } + if len(documented) != len(inCode) { + t.Errorf("%s: block lists %d words, fingerprint.go has %d", + fingerprintSpecPath, len(documented), len(inCode)) + } +} + +// TestSpecConstantsMatchTheCode covers every named value a re-implementation +// must reproduce byte for byte: the separator, the digest length, the four +// normalization tokens, and the two route-templating thresholds. A wrong +// threshold is the subtlest of these — it produces a perfectly valid digest +// that merges or splits routes differently from every other producer. +func TestSpecConstantsMatchTheCode(t *testing.T) { + doc := readFingerprintSpec(t) + lines := specBlock(t, doc, "ANVIL-FP-CONSTANTS") + + documented := map[string]string{} + for _, line := range lines { + k, v, ok := strings.Cut(line, "=") + if !ok { + t.Fatalf("%s: constants block line %q is not \"name = value\"", fingerprintSpecPath, line) + } + k, v = strings.TrimSpace(k), strings.TrimSpace(v) + if _, dup := documented[k]; dup { + t.Errorf("%s: constant %q is listed twice", fingerprintSpecPath, k) + } + documented[k] = v + } + + // "U+001F" rather than a literal control character: the separator cannot be + // written literally into a text document without becoming invisible, and an + // invisible separator in the one document that defines it is how the + // research/07-vs-research/18 conflict happened in the first place. + inCode := map[string]string{ + "FingerprintAlgV1": FingerprintAlgV1, + "FingerprintFieldSeparator": "U+001F", + "FingerprintDigestHexLen": strconv.Itoa(FingerprintDigestHexLen), + "NormalizedStringToken": NormalizedStringToken, + "NormalizedNumberToken": NormalizedNumberToken, + "NormalizedMetavarPrefix": NormalizedMetavarPrefix, + "NormalizedRouteSegmentToken": NormalizedRouteSegmentToken, + "routeHexSegmentMinLen": strconv.Itoa(routeHexSegmentMinLen), + "routeOpaqueSegmentMinLen": strconv.Itoa(routeOpaqueSegmentMinLen), + } + + for name, want := range inCode { + got, ok := documented[name] + if !ok { + t.Errorf("%s: constant %s is not documented; a re-implementation cannot reproduce it", + fingerprintSpecPath, name) + continue + } + if got != want { + t.Errorf("%s: documents %s = %q, the code has %q", fingerprintSpecPath, name, got, want) + } + } + for name := range documented { + if _, ok := inCode[name]; !ok { + t.Errorf("%s: documents constant %s, which does not exist in the code", + fingerprintSpecPath, name) + } + } + + // The separator spelling must actually denote U+001F, not merely agree with + // a string in this test. + if FingerprintFieldSeparator != "\x1f" { + t.Fatalf("FingerprintFieldSeparator = %q, want U+001F", FingerprintFieldSeparator) + } +} + +// TestSpecCarriesTheVersioningRule. fingerprint.go's fingerprintReservedWords +// comment already says that adding or removing an entry "CHANGES EVERY SAST +// FINGERPRINT and is therefore an anvil-fp/v2 event, not a maintenance edit". +// The ruling requires the document to agree, because the document is what a +// second producer reads, and someone editing a word list in a markdown file +// feels far more casual than someone editing a hash function. +func TestSpecCarriesTheVersioningRule(t *testing.T) { + doc := readFingerprintSpec(t) + required := []string{ + "Any change to this document's algorithm is an `anvil-fp/v2` event, never a `v1` edit.", + "anvil-fp/v1", + } + for _, s := range required { + if !strings.Contains(doc, s) { + t.Errorf("%s must contain %q", fingerprintSpecPath, s) + } + } +} + +// TestSpecDocumentsEveryExportedAlgorithmSurface is a cheap guard against the +// failure mode this whole document exists to prevent: a new normalization step +// or tier helper is added to fingerprint.go and nobody writes it down, so the +// spec silently stops being the definition. It names the surfaces a +// re-implementation must know about and asserts each appears in the text. +func TestSpecDocumentsEveryExportedAlgorithmSurface(t *testing.T) { + doc := readFingerprintSpec(t) + surfaces := []string{ + "NormalizeMatch", + "CanonicalRepoRelPath", + "CanonicalRouteTemplate", + "PurlBase", + "ordinal", + "normalized_match", + "route_template", + "rule_id_versioned", + "enclosing_symbol_path", + "injection_point", + "evidence_class_detail", + "purl_base", + "locator", + "http_method", + "target_id", + "detector_kind", + "advisory_id", + "repo_relpath", + "param_name", + } + for _, s := range surfaces { + if !strings.Contains(doc, s) { + t.Errorf("%s does not mention %q; a re-implementation would not know it exists", + fingerprintSpecPath, s) + } + } +} + +// TestSpecWorkedRouteExamplesHold re-runs the §6.5 "Whole routes" examples +// against the implementation. They are transcribed here rather than parsed out +// of the markdown table, so the assertion is on the BEHAVIOUR the document +// promises; if the document's table is edited without the code changing (or +// vice versa) this test and TestCanonicalRouteTemplate disagree with each +// other, which is the signal. +func TestSpecWorkedRouteExamplesHold(t *testing.T) { + examples := [][2]string{ + {"/api/v1/users/12345/orders", "/api/v1/users//orders"}, + {"/api/v1/users/{id}/orders", "/api/v1/users//orders"}, + {"/api/v1/users/:userId/orders", "/api/v1/users//orders"}, + {"api//v1/users/12345//orders/?debug=1", "/api/v1/users//orders"}, + {"/12345/orders/6789", "//orders/"}, + {"/api/v1/users/me/orders", "/api/v1/users/me/orders"}, + {"/", "/"}, + } + for _, ex := range examples { + if got := CanonicalRouteTemplate(ex[0]); got != ex[1] { + t.Errorf("%s section 6.5 promises CanonicalRouteTemplate(%q) = %q, got %q", + fingerprintSpecPath, ex[0], ex[1], got) + } + } + + // And the §3.7 normalization examples, same reasoning. + norms := [][2]string{ + {`rows, err := db.Query("SELECT * FROM users WHERE name = '" + name + "'")`, + `$1, $2 := $3.Query( + $4 + )`}, + {`os.system("rm -rf " + path)`, `$1.system( + $2)`}, + {`a = b + a + b`, `$1 = $2 + $1 + $2`}, + {`for i := range items { return nil }`, `for $1 := range $2 { return nil }`}, + {`a = 0xFF + 1_000 + 3.14f + 1e-9`, `$1 = + + + `}, + {`p->Field = Ns::Helper(v)`, `$1->Field = Ns::Helper($2)`}, + {`std::vector v = Foo::Bar::make(x)`, `std::vector $1 = Foo::Bar::make($2)`}, + {`Ns::Helper(v)`, `Ns::Helper($1)`}, + {`Other::Helper(v)`, `Other::Helper($1)`}, + {`totalCount := len(itemsList)`, `$1 := len($2)`}, + } + for _, ex := range norms { + if got := NormalizeMatch(ex[0]); got != ex[1] { + t.Errorf("%s section 3.7 promises NormalizeMatch(%q) = %q, got %q", + fingerprintSpecPath, ex[0], ex[1], got) + } + } + + // Sanity: the two examples the document uses to argue rule 6(c) must not + // collide, or the argument in the document is false. + if NormalizeMatch("Ns::Helper(v)") == NormalizeMatch("Other::Helper(v)") { + t.Fatal("Ns::Helper and Other::Helper normalise identically; " + + fmt.Sprintf("%s clause (c)'s stated reason does not hold", fingerprintSpecPath)) + } +} diff --git a/internal/record/fingerprint_test.go b/internal/record/fingerprint_test.go new file mode 100644 index 0000000..f90da9e --- /dev/null +++ b/internal/record/fingerprint_test.go @@ -0,0 +1,1464 @@ +package record + +// fingerprint_test.go — R.2's own tests. +// +// These are NOT the conformance test. R.16 owns that +// (internal/record/fingerprint_conformance_test.go plus +// testdata/fingerprint_corpus/*.golden), and its oracle must be an offline +// re-implementation of internal/record/FINGERPRINT-SPEC.md that never imports +// this package. What is here is: +// +// - a corpus lock: every fixture's committed hashed_fields and +// expected_digest must still be what the implementation produces; +// - determinism across two consecutive runs (R.2's stop condition) AND +// across two separate OS processes, which is the form that can actually +// detect a per-process source of nondeterminism such as map iteration +// order; +// - the mutation tests that prove the forbidden inputs are not hashed: +// line/column numbers for SAST, the version string for SCA and host, +// host/port/payload for DAST; +// - a frozen-shape test on the four input structs, so a later edit cannot +// quietly add a field that reintroduces a volatile input. +// +// THE COMMITTED GOLDENS WERE NOT PRODUCED BY THIS PACKAGE. Each fixture's +// `hashed_fields` list was derived by hand from the algorithm text — now +// internal/record/FINGERPRINT-SPEC.md, which is the authoritative and COMPLETE +// definition; plan/40-record-and-storage.md's four-clause summary was proved +// insufficient by CRITIQUE-01 — and its `expected_digest` was computed from +// that list by an offline script that only joins with U+001F and SHA-256s. +// The lock below is therefore a genuine two-sided check, and there is +// deliberately no test here that can regenerate it — see the note further +// down. +// +// TWO DAST GOLDENS WERE RE-DERIVED ON 2026-08-08 under the R.3 blocker-2 +// ruling, when CanonicalRouteTemplate began deriving `route_template` as the +// specification always said it should: +// +// dast-01 ca801b8d… → 199c3b5f… route_template "/api/v1/users/{id}/orders" +// → "/api/v1/users//orders" +// dast-03 84fe311d… → 5fc15c55… route_template "/files/{path}" +// → "/files/" +// +// dast-02 ("/search") was unaffected, and its untouched golden reproducing +// exactly under the same offline script is the control that shows the script +// was not quietly rewritten to agree with the Go code. No SAST, SCA or host +// digest moved. Nothing had been stored under the old digests — anvil-fp/v1 +// has not shipped — so this was a correction of an unimplemented spec clause, +// not an anvil-fp/v2 event. +// +// R.16: do NOT derive the .golden files from `expected_digest` either. Even +// though it is independent of the Go code, copying it would make the +// conformance gate a transcription check rather than a re-derivation. + +import ( + "bytes" + "encoding/json" + "fmt" + "os" + "os/exec" + "path/filepath" + "reflect" + "regexp" + "sort" + "strings" + "testing" +) + +const corpusDir = "../../testdata/fingerprint_corpus" + +// --------------------------------------------------------------------------- +// Corpus loading +// --------------------------------------------------------------------------- + +type corpusMutation struct { + Name string `json:"name"` + Description string `json:"description"` + Input json.RawMessage `json:"input"` + NotHashed json.RawMessage `json:"not_hashed,omitempty"` +} + +type corpusFixture struct { + ID string `json:"id"` + Tier string `json:"tier"` + Description string `json:"description"` + Input json.RawMessage `json:"input"` + NotHashed json.RawMessage `json:"not_hashed,omitempty"` + HashedFields []string `json:"hashed_fields"` + ExpectedDigest string `json:"expected_digest"` + Mutations []corpusMutation `json:"mutations,omitempty"` + Notes []string `json:"notes,omitempty"` + + path string +} + +type jsonSastInput struct { + TargetID string `json:"target_id"` + RuleIDVersioned string `json:"rule_id_versioned"` + RepoRelPath string `json:"repo_rel_path"` + EnclosingSymbolPath string `json:"enclosing_symbol_path"` + Snippet string `json:"snippet"` + Ordinal int `json:"ordinal"` +} + +type jsonScaInput struct { + TargetID string `json:"target_id"` + AdvisoryID string `json:"advisory_id"` + Purl string `json:"purl"` + ManifestRelPath string `json:"manifest_rel_path"` +} + +type jsonHostInput struct { + TargetID string `json:"target_id"` + AdvisoryID string `json:"advisory_id"` + Purl string `json:"purl"` + PackageManager string `json:"package_manager"` + HostIdentifier string `json:"host_identifier"` +} + +type jsonDastInput struct { + TargetID string `json:"target_id"` + RuleIDVersioned string `json:"rule_id_versioned"` + HTTPMethod string `json:"http_method"` + RouteTemplate string `json:"route_template"` + InjectionPoint string `json:"injection_point"` + ParamName string `json:"param_name"` + EvidenceSignal string `json:"evidence_signal"` +} + +// strictUnmarshal rejects unknown keys, so a misspelled fixture key fails the +// test instead of silently defaulting a hashed field to the empty string and +// locking in a wrong digest. +func strictUnmarshal(raw json.RawMessage, dst any) error { + dec := json.NewDecoder(bytes.NewReader(raw)) + dec.DisallowUnknownFields() + return dec.Decode(dst) +} + +// fieldsFor decodes a fixture input for the named tier and returns the exact +// ordered field list that tier hashes. +func fieldsFor(tier string, raw json.RawMessage) ([]string, error) { + switch tier { + case "sast": + var in jsonSastInput + if err := strictUnmarshal(raw, &in); err != nil { + return nil, err + } + return SastFields(SastInput{ + TargetID: in.TargetID, + RuleIDVersioned: in.RuleIDVersioned, + RepoRelPath: in.RepoRelPath, + EnclosingSymbolPath: in.EnclosingSymbolPath, + Snippet: in.Snippet, + Ordinal: in.Ordinal, + }) + case "sca": + var in jsonScaInput + if err := strictUnmarshal(raw, &in); err != nil { + return nil, err + } + return ScaFields(ScaInput{ + TargetID: in.TargetID, + AdvisoryID: in.AdvisoryID, + Purl: in.Purl, + ManifestRelPath: in.ManifestRelPath, + }) + case "host": + var in jsonHostInput + if err := strictUnmarshal(raw, &in); err != nil { + return nil, err + } + return HostFields(HostInput{ + TargetID: in.TargetID, + AdvisoryID: in.AdvisoryID, + Purl: in.Purl, + PackageManager: in.PackageManager, + HostIdentifier: in.HostIdentifier, + }) + case "dast": + var in jsonDastInput + if err := strictUnmarshal(raw, &in); err != nil { + return nil, err + } + return DastFields(DastInput{ + TargetID: in.TargetID, + RuleIDVersioned: in.RuleIDVersioned, + HTTPMethod: in.HTTPMethod, + RouteTemplate: in.RouteTemplate, + InjectionPoint: InjectionPoint(in.InjectionPoint), + ParamName: in.ParamName, + EvidenceSignal: EvidenceSignal(in.EvidenceSignal), + }) + default: + return nil, &FingerprintError{Field: "tier", Msg: "unknown tier " + tier} + } +} + +// digestFor is the tier-dispatching equivalent of the exported Sast/Sca/ +// Host/Dast entry points, used so a fixture can be driven by its declared +// tier. The public entry points are exercised directly in +// TestExportedTierEntryPointsAgreeWithFieldBuilders. +func digestFor(tier string, raw json.RawMessage) (string, error) { + fields, err := fieldsFor(tier, raw) + if err != nil { + return "", err + } + return Digest(fields...) +} + +func loadCorpus(t *testing.T) []corpusFixture { + t.Helper() + + paths, err := filepath.Glob(filepath.Join(corpusDir, "*.json")) + if err != nil { + t.Fatalf("globbing corpus: %v", err) + } + if len(paths) == 0 { + t.Fatalf("no fixtures found in %s; the fixed corpus is mandatory (plan/00-SPINE.md S6)", corpusDir) + } + sort.Strings(paths) + + out := make([]corpusFixture, 0, len(paths)) + for _, p := range paths { + b, err := os.ReadFile(p) + if err != nil { + t.Fatalf("reading %s: %v", p, err) + } + var f corpusFixture + dec := json.NewDecoder(bytes.NewReader(b)) + dec.DisallowUnknownFields() + if err := dec.Decode(&f); err != nil { + t.Fatalf("decoding %s: %v", p, err) + } + f.path = p + if f.ID == "" || f.Tier == "" { + t.Fatalf("%s: fixture must declare id and tier", p) + } + out = append(out, f) + } + return out +} + +// --------------------------------------------------------------------------- +// The corpus lock +// --------------------------------------------------------------------------- + +func TestCorpusCoverage(t *testing.T) { + // plan/40-record-and-storage.md R.2: "at minimum 2 SAST, 2 DAST, 1 SCA, + // 1 host". A tier with no fixture is a tier nothing guards. + want := map[string]int{"sast": 2, "dast": 2, "sca": 1, "host": 1} + got := map[string]int{} + for _, f := range loadCorpus(t) { + got[f.Tier]++ + } + for tier, min := range want { + if got[tier] < min { + t.Errorf("tier %q: corpus has %d fixtures, the specification requires at least %d", + tier, got[tier], min) + } + } +} + +func TestCorpusFixturesProduceTheirDocumentedDigest(t *testing.T) { + for _, f := range loadCorpus(t) { + t.Run(f.ID, func(t *testing.T) { + if f.ExpectedDigest == "" { + t.Fatalf("%s: expected_digest is empty; every fixture must carry a golden "+ + "derived independently of this implementation", f.path) + } + if err := ValidateDigest(f.ExpectedDigest); err != nil { + t.Fatalf("%s: committed expected_digest is malformed: %v", f.path, err) + } + + fields, err := fieldsFor(f.Tier, f.Input) + if err != nil { + t.Fatalf("%s: building fields: %v", f.path, err) + } + if !reflect.DeepEqual(fields, f.HashedFields) { + t.Errorf("%s: hashed field list drifted.\n got: %q\n want: %q", + f.path, fields, f.HashedFields) + } + + got, err := Digest(fields...) + if err != nil { + t.Fatalf("%s: hashing: %v", f.path, err) + } + if got != f.ExpectedDigest { + t.Errorf("%s: digest changed.\n got: %s\n want: %s\n"+ + "A changed digest means every stored finding for this tier loses its identity; "+ + "this is an anvil-fp/v2 event, not a fixture update.", + f.path, got, f.ExpectedDigest) + } + if err := ValidateDigest(got); err != nil { + t.Errorf("%s: computed digest is malformed: %v", f.path, err) + } + }) + } +} + +// TestCorpusDigestsAreStableAcrossTwoConsecutiveRuns is R.2's stop condition +// verbatim. It catches map-iteration order or any other nondeterminism +// leaking into the digest — NormalizeMatch uses a map for metavariable +// assignment, and if that assignment ever became iteration-ordered this test +// would fail. +func TestCorpusDigestsAreStableAcrossTwoConsecutiveRuns(t *testing.T) { + for _, f := range loadCorpus(t) { + first, err := digestFor(f.Tier, f.Input) + if err != nil { + t.Fatalf("%s: run 1: %v", f.path, err) + } + for run := 2; run <= 8; run++ { + next, err := digestFor(f.Tier, f.Input) + if err != nil { + t.Fatalf("%s: run %d: %v", f.path, run, err) + } + if next != first { + t.Fatalf("%s: digest is not deterministic: run 1 = %s, run %d = %s", + f.path, first, run, next) + } + } + } +} + +// crossProcessEnv puts this test binary into "child" mode: it prints one +// crossProcessMarker line per fixture and exits, so the parent can compare a +// digest computed in a DIFFERENT OS process against its own. +const ( + crossProcessEnv = "ANVIL_FINGERPRINT_CROSS_PROCESS_CHILD" + crossProcessMarker = "ANVIL-FP-DIGEST\t" +) + +// TestCorpusDigestsAreStableAcrossProcesses is the stronger form of the stop +// condition. Repeating a computation inside ONE process cannot detect the +// failure that actually matters here, because the things that would break +// cross-process stability are all per-process constants: Go's map iteration +// seed (re-randomised per process, so an unsorted range over a map returns a +// stable-but-wrong order within a run and a different one in the next), +// pointer addresses, and any address-space or locale state. Two producers +// emitting different digests is exactly the failure 00-SPINE.md S6 says fails +// silently forever, and the two producers are two processes. +// +// The child is this same test binary re-executed with crossProcessEnv set, +// which needs no build step and no new dependency. +func TestCorpusDigestsAreStableAcrossProcesses(t *testing.T) { + fixtures := loadCorpus(t) + + if os.Getenv(crossProcessEnv) == "1" { + for _, f := range fixtures { + d, err := digestFor(f.Tier, f.Input) + if err != nil { + fmt.Printf("%s%s\tERROR: %v\n", crossProcessMarker, f.ID, err) + continue + } + fmt.Printf("%s%s\t%s\n", crossProcessMarker, f.ID, d) + } + return + } + + want := make(map[string]string, len(fixtures)) + for _, f := range fixtures { + d, err := digestFor(f.Tier, f.Input) + if err != nil { + t.Fatalf("%s: parent process: %v", f.path, err) + } + want[f.ID] = d + } + + cmd := exec.Command(os.Args[0], + "-test.run=^TestCorpusDigestsAreStableAcrossProcesses$", + "-test.count=1") + cmd.Env = append(os.Environ(), crossProcessEnv+"=1") + out, err := cmd.CombinedOutput() + if err != nil { + t.Fatalf("re-executing the test binary as a child process failed: %v\n%s", err, out) + } + + got := make(map[string]string, len(fixtures)) + for _, line := range strings.Split(string(out), "\n") { + line = strings.TrimSpace(line) + if !strings.HasPrefix(line, strings.TrimSpace(crossProcessMarker)) { + continue + } + parts := strings.Split(line, "\t") + if len(parts) != 3 { + t.Fatalf("malformed child output line %q", line) + } + got[parts[1]] = parts[2] + } + + if len(got) != len(want) { + t.Fatalf("child reported %d digests, parent computed %d\nchild output:\n%s", + len(got), len(want), out) + } + for id, w := range want { + if got[id] != w { + t.Errorf("fixture %s: digest differs between processes.\n parent: %s\n child: %s\n"+ + "anvil-fp/v1 must be a pure function of its hashed fields; two producers "+ + "emitting different digests breaks regression matching silently and forever.", + id, w, got[id]) + } + } +} + +// TestCorpusMutationsDoNotChangeTheDigest is the mutation test R.2's +// validation requirement names. Every fixture's `mutations` array holds inputs +// that differ ONLY in fields the specification forbids hashing — line and +// column numbers, indentation, comments, the dependency version string, the +// HTTP method's case, a query string on a route template. Each must produce +// the fixture's digest unchanged. +func TestCorpusMutationsDoNotChangeTheDigest(t *testing.T) { + total := 0 + for _, f := range loadCorpus(t) { + for _, m := range f.Mutations { + total++ + t.Run(f.ID+"/"+m.Name, func(t *testing.T) { + got, err := digestFor(f.Tier, m.Input) + if err != nil { + t.Fatalf("%s mutation %q: %v", f.path, m.Name, err) + } + if got != f.ExpectedDigest { + t.Errorf("%s mutation %q (%s) changed the digest.\n got: %s\n want: %s", + f.path, m.Name, m.Description, got, f.ExpectedDigest) + } + }) + } + } + if total == 0 { + t.Fatal("no mutations in the corpus; the volatile-field exclusions are then untested") + } +} + +// TestCorpusFixturesHaveDistinctDigests guards the failure that motivates the +// ordinal field: two fixtures that ought to be different findings must not +// share an identity. +func TestCorpusFixturesHaveDistinctDigests(t *testing.T) { + seen := map[string]string{} + for _, f := range loadCorpus(t) { + d, err := digestFor(f.Tier, f.Input) + if err != nil { + t.Fatalf("%s: %v", f.path, err) + } + if prev, ok := seen[d]; ok { + t.Errorf("fixtures %q and %q collide on digest %s", prev, f.ID, d) + } + seen[d] = f.ID + } +} + +// THERE IS DELIBERATELY NO "UPDATE THE GOLDENS" TEST HERE, AND ONE MUST NOT BE +// ADDED. +// +// A golden that the implementation under test can regenerate proves nothing: +// the next time a digest changes, the cheapest way to make this package green +// is to re-seal the fixtures, and the change that was supposed to be caught +// ships silently. That is not a hypothetical — research/07 and research/18 +// shipped two different algorithms under one `anvil-fp/v1` name and nothing +// surfaced it, which is why this corpus exists at all. +// +// The committed `hashed_fields` lists were derived BY HAND from the algorithm +// text in plan/40-record-and-storage.md, and each `expected_digest` was then +// computed from that list by an offline script that joins with U+001F and +// SHA-256s — no Go code involved. To change a fixture, redo that derivation. +// To change a digest, ship anvil-fp/v2 and follow the dual-write migration +// protocol; do not edit the golden. + +// --------------------------------------------------------------------------- +// The exclusions, tested directly rather than only through fixtures +// --------------------------------------------------------------------------- + +// TestSastDigestIgnoresPositionAndFormatting is the specific mutation +// assertion R.2 requires: "a line-number-only change to a SAST fixture leaves +// the digest unchanged". SastInput has no line field at all, so the strongest +// form of the test is to change everything a line-number change implies — +// leading blank lines, indentation, line breaks, and a comment naming the old +// line — and show the digest holds. +func TestSastDigestIgnoresPositionAndFormatting(t *testing.T) { + base := SastInput{ + TargetID: "t-0001", + RuleIDVersioned: "opengrep.go.sqli@2026.07.1", + RepoRelPath: "internal/api/store.go", + EnclosingSymbolPath: "internal/api/store.go::Store.Find", + Snippet: "q := \"SELECT * FROM u WHERE n = '\" + name + \"'\"\nrows, err := db.Query(q)", + Ordinal: 0, + } + want, err := Sast(base) + if err != nil { + t.Fatalf("base: %v", err) + } + + cases := []struct { + name string + snippet string + }{ + { + name: "shifted down by 40 lines", + snippet: strings.Repeat("\n", 40) + base.Snippet, + }, + { + name: "reindented from tabs to eight spaces", + snippet: " q := \"SELECT * FROM u WHERE n = '\" + name + \"'\"\n rows, err := db.Query(q)", + }, + { + name: "CRLF line endings", + snippet: strings.ReplaceAll(base.Snippet, "\n", "\r\n"), + }, + { + name: "a comment naming the old line number added", + snippet: "// was line 128, now line 173\n" + base.Snippet + " // moved", + }, + { + name: "collapsed onto one line", + snippet: strings.ReplaceAll(base.Snippet, "\n", " "), + }, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + m := base + m.Snippet = tc.snippet + got, err := Sast(m) + if err != nil { + t.Fatalf("%v", err) + } + if got != want { + t.Errorf("digest changed under a position/formatting-only edit\n got: %s\n want: %s\n normalized: %q vs %q", + got, want, NormalizeMatch(tc.snippet), NormalizeMatch(base.Snippet)) + } + }) + } +} + +// TestSastDigestChangesWhenTheCodeChanges is the counterpart: normalization +// must not be so aggressive that a genuinely different sink hashes the same. +func TestSastDigestChangesWhenTheCodeChanges(t *testing.T) { + base := SastInput{ + TargetID: "t-0001", + RuleIDVersioned: "opengrep.go.sqli@2026.07.1", + RepoRelPath: "internal/api/store.go", + EnclosingSymbolPath: "internal/api/store.go::Store.Find", + Snippet: "rows, err := db.Query(q)", + } + baseDigest, err := Sast(base) + if err != nil { + t.Fatalf("%v", err) + } + + variants := map[string]func(*SastInput){ + "different callee": func(in *SastInput) { in.Snippet = "rows, err := db.Exec(q)" }, + "different receiver field": func(in *SastInput) { in.Snippet = "rows, err := db.conn.Query(q)" }, + "different rule version": func(in *SastInput) { in.RuleIDVersioned = "opengrep.go.sqli@2026.08.1" }, + "different file": func(in *SastInput) { in.RepoRelPath = "internal/api/other.go" }, + "different symbol": func(in *SastInput) { in.EnclosingSymbolPath = "internal/api/store.go::Store.List" }, + "different target": func(in *SastInput) { in.TargetID = "t-0002" }, + "different ordinal": func(in *SastInput) { in.Ordinal = 1 }, + } + for name, mutate := range variants { + t.Run(name, func(t *testing.T) { + m := base + mutate(&m) + got, err := Sast(m) + if err != nil { + t.Fatalf("%v", err) + } + if got == baseDigest { + t.Errorf("%s did not change the digest; two distinct findings would share one identity", name) + } + }) + } +} + +// TestPackageDigestIgnoresTheVersionString is the rule the specification +// calls out most emphatically: "bumping 1.2.3 to 1.2.4 while still inside the +// vulnerable range must not mint a new finding". +func TestPackageDigestIgnoresTheVersionString(t *testing.T) { + sca := ScaInput{ + TargetID: "t-0001", + AdvisoryID: "CVE-2021-44228", + Purl: "pkg:maven/org.apache.logging.log4j/log4j-core", + ManifestRelPath: "services/api/pom.xml", + } + want, err := Sca(sca) + if err != nil { + t.Fatalf("%v", err) + } + + for _, purl := range []string{ + "pkg:maven/org.apache.logging.log4j/log4j-core@2.14.0", + "pkg:maven/org.apache.logging.log4j/log4j-core@2.14.1", + "pkg:maven/org.apache.logging.log4j/log4j-core@2.15.0", + "pkg:maven/org.apache.logging.log4j/log4j-core@2.14.1?type=jar", + "pkg:maven/org.apache.logging.log4j/log4j-core@2.14.1?type=jar#sub/path", + "PKG:MAVEN/org.apache.logging.log4j/log4j-core@2.14.1", + } { + m := sca + m.Purl = purl + got, err := Sca(m) + if err != nil { + t.Fatalf("%s: %v", purl, err) + } + if got != want { + t.Errorf("purl %q changed the SCA digest; the version string must never be hashed\n got: %s\n want: %s", + purl, got, want) + } + } + + host := HostInput{ + TargetID: "t-0001", + AdvisoryID: "CVE-2022-0778", + Purl: "pkg:deb/debian/openssl", + PackageManager: "apt", + HostIdentifier: "openssl", + } + hostWant, err := Host(host) + if err != nil { + t.Fatalf("%v", err) + } + for _, purl := range []string{ + "pkg:deb/debian/openssl@1.1.1n-0+deb11u3", + "pkg:deb/debian/openssl@1.1.1n-0+deb11u4", + "pkg:deb/debian/openssl@1.1.1w-0+deb11u1?arch=amd64", + } { + m := host + m.Purl = purl + got, err := Host(m) + if err != nil { + t.Fatalf("%s: %v", purl, err) + } + if got != hostWant { + t.Errorf("purl %q changed the host digest; the version string must never be hashed", purl) + } + } +} + +// TestScaAndHostTiersDoNotCollide: the two tiers share one formula, so the +// detector-kind field is the only thing keeping a repo dependency and a host +// package with the same advisory apart. If it were dropped, a host finding +// (remediable_by_agent=false, plan/00-SPINE.md S7) could upsert over an +// agent-remediable dependency finding. +func TestScaAndHostTiersDoNotCollide(t *testing.T) { + scaDigest, err := Sca(ScaInput{ + TargetID: "t-1", + AdvisoryID: "CVE-2022-0778", + Purl: "pkg:deb/debian/openssl", + ManifestRelPath: "apt:openssl", + }) + if err != nil { + t.Fatalf("%v", err) + } + hostDigest, err := Host(HostInput{ + TargetID: "t-1", + AdvisoryID: "CVE-2022-0778", + Purl: "pkg:deb/debian/openssl", + PackageManager: "apt", + HostIdentifier: "openssl", + }) + if err != nil { + t.Fatalf("%v", err) + } + if scaDigest == hostDigest { + t.Fatal("sca and host tiers collided on identical locators; detector_kind is not being hashed") + } +} + +// TestDastDigestIgnoresVolatileTransportDetail. Host, port, scheme and the +// concrete payload are absent from DastInput by construction; what remains +// testable is that the canonicalisation drops the volatile parts a producer +// might still smuggle in through the route template, and that method case +// does not fork identity. +func TestDastDigestIgnoresVolatileTransportDetail(t *testing.T) { + base := DastInput{ + TargetID: "t-0001", + RuleIDVersioned: "nuclei:sqli-error-based@a1b2c3d", + HTTPMethod: "POST", + RouteTemplate: "/api/v1/users/{id}/orders", + InjectionPoint: InjectionPointBody, + ParamName: "sortBy", + EvidenceSignal: EvidenceSignalDBErrorString, + } + want, err := Dast(base) + if err != nil { + t.Fatalf("%v", err) + } + + cases := map[string]func(*DastInput){ + "lowercase method": func(in *DastInput) { in.HTTPMethod = "post" }, + "method with whitespace": func(in *DastInput) { in.HTTPMethod = " POST " }, + "route with query string": func(in *DastInput) { in.RouteTemplate = "/api/v1/users/{id}/orders?debug=1&payload=%27" }, + "route with fragment": func(in *DastInput) { in.RouteTemplate = "/api/v1/users/{id}/orders#frag" }, + "route with trailing slash": func(in *DastInput) { + in.RouteTemplate = "/api/v1/users/{id}/orders/" + }, + "route with duplicate slashes": func(in *DastInput) { + in.RouteTemplate = "//api//v1/users/{id}/orders" + }, + "route without leading slash": func(in *DastInput) { in.RouteTemplate = "api/v1/users/{id}/orders" }, + + // The blocker-2 cases: the concrete route the scanner actually + // requested, and the two other syntaxes a second producer would use + // for the same route. All four are one finding. + "route with a concrete numeric id": func(in *DastInput) { in.RouteTemplate = "/api/v1/users/12345/orders" }, + "route with a different numeric id": func(in *DastInput) { in.RouteTemplate = "/api/v1/users/4192/orders" }, + "route with a concrete uuid": func(in *DastInput) { + in.RouteTemplate = "/api/v1/users/3f2504e0-4f89-11d3-9a0c-0305e82c3301/orders" + }, + "route with a concrete sha1 object id": func(in *DastInput) { + in.RouteTemplate = "/api/v1/users/da39a3ee5e6b4b0d3255bfef95601890afd80709/orders" + }, + "route in express placeholder syntax": func(in *DastInput) { in.RouteTemplate = "/api/v1/users/:id/orders" }, + "route in flask placeholder syntax": func(in *DastInput) { + in.RouteTemplate = "/api/v1/users//orders" + }, + "route already carrying the canonical token": func(in *DastInput) { + in.RouteTemplate = "/api/v1/users/" + NormalizedRouteSegmentToken + "/orders" + }, + "route with a differently NAMED placeholder": func(in *DastInput) { + in.RouteTemplate = "/api/v1/users/{userId}/orders" + }, + } + for name, mutate := range cases { + t.Run(name, func(t *testing.T) { + m := base + mutate(&m) + got, err := Dast(m) + if err != nil { + t.Fatalf("%v", err) + } + if got != want { + t.Errorf("%s changed the DAST digest\n got: %s\n want: %s", name, got, want) + } + }) + } +} + +// TestDastRouteTemplatingKeepsDistinctRoutesDistinct is the counterpart to the +// blocker-2 cases above. Route templating exists to merge one defect's +// instances; it must not merge two defects. Losing a finding on upsert against +// UNIQUE (target_id, fingerprint) is worse than churning one, and it is silent. +func TestDastRouteTemplatingKeepsDistinctRoutesDistinct(t *testing.T) { + base := DastInput{ + TargetID: "t-0001", + RuleIDVersioned: "nuclei:idor@a1b2c3d", + HTTPMethod: "GET", + RouteTemplate: "/api/v1/users/12345/orders", + InjectionPoint: InjectionPointPath, + EvidenceSignal: EvidenceSignalStatusCodeFlip, + } + + routes := []string{ + "/api/v1/users/12345/orders", + "/api/v1/users/me/orders", // "me" is a sentinel, not an id + "/api/v1/users/12345/invoices", // a different collection + "/api/v2/users/12345/orders", // a different API version + "/api/v1/users/12345", // a shorter route + "/api/v1/admins/12345/orders", // a different resource + } + seen := map[string]string{} + for _, r := range routes { + m := base + m.RouteTemplate = r + d, err := Dast(m) + if err != nil { + t.Fatalf("%s: %v", r, err) + } + if prev, ok := seen[d]; ok { + t.Errorf("routes %q and %q share digest %s; one of the two findings is lost on upsert", + prev, r, d) + } + seen[d] = r + } +} + +// TestDastInjectionPointAndEvidenceSignalAreIndependent. The whole reason +// research/18's evidenceClass was merged into research/07's Tier D is that +// WHERE the payload went in and HOW the defect was observed are different +// facts. Each must move the digest on its own. +func TestDastInjectionPointAndEvidenceSignalAreIndependent(t *testing.T) { + base := DastInput{ + TargetID: "t-0001", + RuleIDVersioned: "nuclei:sqli@a1b2c3d", + HTTPMethod: "GET", + RouteTemplate: "/search", + InjectionPoint: InjectionPointQuery, + ParamName: "q", + EvidenceSignal: EvidenceSignalDBErrorString, + } + baseDigest, err := Dast(base) + if err != nil { + t.Fatalf("%v", err) + } + + byPoint := base + byPoint.InjectionPoint = InjectionPointHeader + d1, err := Dast(byPoint) + if err != nil { + t.Fatalf("%v", err) + } + bySignal := base + bySignal.EvidenceSignal = EvidenceSignalTimingSideChannel + d2, err := Dast(bySignal) + if err != nil { + t.Fatalf("%v", err) + } + + if d1 == baseDigest { + t.Error("changing injection_point alone did not change the digest") + } + if d2 == baseDigest { + t.Error("changing evidence_class_detail alone did not change the digest") + } + if d1 == d2 { + t.Error("injection_point and evidence_class_detail are not independent fields") + } +} + +// TestInputStructsHaveTheirFrozenShape pins the exact field set of each tier +// input struct. Adding a field is how a volatile input (a line number, a +// version, a host) gets reintroduced, and it would silently change every +// digest that field participates in. If this test fails, the change is an +// anvil-fp/v2 event and an amendment to plan/40-record-and-storage.md, not a +// local edit. +func TestInputStructsHaveTheirFrozenShape(t *testing.T) { + cases := []struct { + name string + typ reflect.Type + fields []string + forbidden []string + }{ + { + name: "SastInput", + typ: reflect.TypeOf(SastInput{}), + fields: []string{"TargetID", "RuleIDVersioned", "RepoRelPath", + "EnclosingSymbolPath", "Snippet", "Ordinal"}, + // "version" is absent from this list on purpose: the RULE + // version is deliberately hashed. Line, column, advisory and + // evidence class are not. + forbidden: []string{"line", "column", "advisory", "evidence", "timestamp", "region"}, + }, + { + name: "ScaInput", + typ: reflect.TypeOf(ScaInput{}), + fields: []string{"TargetID", "AdvisoryID", "Purl", "ManifestRelPath"}, + forbidden: []string{"version", "line", "timestamp"}, + }, + { + name: "HostInput", + typ: reflect.TypeOf(HostInput{}), + fields: []string{"TargetID", "AdvisoryID", "Purl", "PackageManager", "HostIdentifier"}, + forbidden: []string{"version", "line", "timestamp"}, + }, + { + name: "DastInput", + typ: reflect.TypeOf(DastInput{}), + fields: []string{"TargetID", "RuleIDVersioned", "HTTPMethod", "RouteTemplate", + "InjectionPoint", "ParamName", "EvidenceSignal"}, + forbidden: []string{"host", "port", "scheme", "payload", "token", "cookieval", "timestamp", "line"}, + }, + } + + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + var got []string + for i := 0; i < tc.typ.NumField(); i++ { + got = append(got, tc.typ.Field(i).Name) + } + if !reflect.DeepEqual(got, tc.fields) { + t.Fatalf("field set changed.\n got: %v\n want: %v\n"+ + "Adding or removing a hashed input is an anvil-fp/v2 event.", got, tc.fields) + } + for _, f := range got { + lower := strings.ToLower(f) + for _, bad := range tc.forbidden { + if strings.Contains(lower, bad) { + t.Errorf("field %s contains forbidden substring %q; "+ + "the specification forbids hashing it", f, bad) + } + } + } + }) + } +} + +// --------------------------------------------------------------------------- +// Primitive, canonicalisation and normalization behaviour +// --------------------------------------------------------------------------- + +var hexDigest = regexp.MustCompile(`^[0-9a-f]{64}$`) + +func TestDigestShapeAndSeparator(t *testing.T) { + d, err := Digest("a", "b", "c") + if err != nil { + t.Fatalf("%v", err) + } + if !hexDigest.MatchString(d) { + t.Errorf("digest %q is not 64 lowercase hex characters", d) + } + if len(d) != FingerprintDigestHexLen { + t.Errorf("digest length %d, want %d (never truncated)", len(d), FingerprintDigestHexLen) + } + + // The separator, not concatenation: ("ab","c") and ("a","bc") must differ. + d1, _ := Digest("ab", "c") + d2, _ := Digest("a", "bc") + if d1 == d2 { + t.Error("fields are being concatenated without a separator") + } + + if _, err := Digest("ok", "bad\x1ffield"); err == nil { + t.Error("a field containing U+001F must be rejected: it moves a field boundary") + } + if _, err := Digest("ok", "bad\nfield"); err == nil { + t.Error("a field containing a control character must be rejected") + } + if _, err := Digest(); err == nil { + t.Error("an empty field list must be rejected") + } +} + +func TestValidateDigest(t *testing.T) { + good, err := Digest("x") + if err != nil { + t.Fatalf("%v", err) + } + if err := ValidateDigest(good); err != nil { + t.Errorf("a freshly computed digest failed validation: %v", err) + } + bad := map[string]string{ + "truncated to 32": good[:32], + "uppercase hex": strings.ToUpper(good), + "empty": "", + "non-hex at end": good[:63] + "g", + "one char too long": good + "0", + } + for name, v := range bad { + if err := ValidateDigest(v); err == nil { + t.Errorf("%s: expected rejection, got none", name) + } + } +} + +func TestCanonicalRepoRelPath(t *testing.T) { + cases := map[string]string{ + "internal/api/store.go": "internal/api/store.go", + `internal\api\store.go`: "internal/api/store.go", + "./internal/api/store.go": "internal/api/store.go", + "/internal/api/store.go": "internal/api/store.go", + "internal//api///store.go": "internal/api/store.go", + ".//internal/api/store.go": "internal/api/store.go", + "internal/api/": "internal/api", + `.\services\api\pom.xml`: "services/api/pom.xml", + "Internal/API/Store.go": "Internal/API/Store.go", // case preserved + } + for in, want := range cases { + if got := CanonicalRepoRelPath(in); got != want { + t.Errorf("CanonicalRepoRelPath(%q) = %q, want %q", in, got, want) + } + } +} + +func TestCanonicalRouteTemplate(t *testing.T) { + const tok = NormalizedRouteSegmentToken + + cases := map[string]string{ + // Structural canonicalisation. + "/api/v1/users/{id}": "/api/v1/users/" + tok, + "api/v1/users/{id}": "/api/v1/users/" + tok, + "/api/v1/users/{id}/": "/api/v1/users/" + tok, + "//api//v1/users/{id}": "/api/v1/users/" + tok, + "/api/v1/users/{id}?a=1": "/api/v1/users/" + tok, + "/api/v1/users/{id}#frag": "/api/v1/users/" + tok, + "/": "/", + "/Search": "/Search", // case preserved + + // Templating: the four volatile-segment classes. + "/api/v1/users/12345/orders": "/api/v1/users/" + tok + "/orders", + "/orders/0": "/orders/" + tok, + "/u/3f2504e0-4f89-11d3-9a0c-0305e82c3301": "/u/" + tok, + "/u/3F2504E0-4F89-11D3-9A0C-0305E82C3301": "/u/" + tok, + "/blob/e3b0c44298fc1c14": "/blob/" + tok, // 16 hex, the threshold exactly + "/blob/3f2504e04f8911d39a0c0305e82c3301": "/blob/" + tok, // dash-free UUID + "/s/dXNlcjEyMzQ1Njc4OTAxMg": "/s/" + tok, // 22-char alnum with digits + + // Templating: the three producer placeholder syntaxes, all onto one token. + "/api/users/{userId}/orders": "/api/users/" + tok + "/orders", + "/api/users/:userId/orders": "/api/users/" + tok + "/orders", + "/api/users//orders": "/api/users/" + tok + "/orders", + + // Idempotence: the canonical token is itself an already-templated segment. + "/api/users/" + tok + "/orders": "/api/users/" + tok + "/orders", + + // Every segment is examined, not only the last. + "/12345/orders/6789": "/" + tok + "/orders/" + tok, + } + for in, want := range cases { + if got := CanonicalRouteTemplate(in); got != want { + t.Errorf("CanonicalRouteTemplate(%q) = %q, want %q", in, got, want) + } + } + + // Idempotence as a property, not only on the one case above: a record read + // out of the store and re-fingerprinted must not fork its own identity. + for in := range cases { + once := CanonicalRouteTemplate(in) + if twice := CanonicalRouteTemplate(once); twice != once { + t.Errorf("CanonicalRouteTemplate is not idempotent on %q: %q then %q", in, once, twice) + } + } +} + +// TestCanonicalRouteTemplateDoesNotOverTemplate is the other half of the +// blocker-2 ruling and the more important half. Over-templating merges two +// genuinely distinct routes onto one digest, and one of the two findings is +// then LOST on upsert against UNIQUE (target_id, fingerprint) — silently. +// Under-templating only leaves a volatile route un-merged, which the DAST +// producer can still repair by emitting "{id}" itself. +// +// Every segment below must survive canonicalisation untouched. +func TestCanonicalRouteTemplateDoesNotOverTemplate(t *testing.T) { + preserved := []string{ + "v1", // a version segment: has a digit, but is 2 chars + "v2", // + "me", // the sentinel that is NOT an id + "users", // + "oauth2", // digit-bearing but short + "utf8", // + "api", // + "latest", // + "internationalization", // 20 letters, no digit — the long-word case + "recommendations", // + "release-notes-2026-08", // 21 chars with a digit, but hyphenated: a slug + "user_profile_settings", // 21 chars, underscored: a slug + "deadbeef", // 8 hex — below the 16-char hex threshold + "a1b2c3d", // a short git object id, 7 chars + "cafebabecafebab", // 15 hex — one short of the threshold + "report.pdf", // a filename + "2026-08-08", // an ISO date: hyphenated, so not opaque + "Search", // + } + for _, seg := range preserved { + in := "/api/" + seg + "/x" + want := "/api/" + seg + "/x" + if got := CanonicalRouteTemplate(in); got != want { + t.Errorf("segment %q was templated but must be preserved: CanonicalRouteTemplate(%q) = %q", + seg, in, got) + } + } + + // And the routes that must stay distinct from each other end to end. + distinct := []string{ + "/api/v1/users/12345/orders", + "/api/v1/users/me/orders", + "/api/v1/users/12345/invoices", + "/api/v2/users/12345/orders", + "/api/v1/users/12345/orders/12345", + } + seen := map[string]string{} + for _, r := range distinct { + c := CanonicalRouteTemplate(r) + if prev, ok := seen[c]; ok { + t.Errorf("routes %q and %q both canonicalise to %q; two distinct routes would share one identity", + prev, r, c) + } + seen[c] = r + } +} + +func TestPurlBase(t *testing.T) { + ok := map[string]string{ + "pkg:npm/lodash": "pkg:npm/lodash", + "pkg:npm/lodash@4.17.20": "pkg:npm/lodash", + "pkg:NPM/lodash@4.17.20": "pkg:npm/lodash", + "pkg:npm/%40angular/core@13.0.0": "pkg:npm/%40angular/core", + "pkg:deb/debian/openssl@1.1.1n?a=b": "pkg:deb/debian/openssl", + "pkg:golang/golang.org/x/net@v0.0.1": "pkg:golang/golang.org/x/net", + "PKG:maven/org.example/lib@1.0#s": "pkg:maven/org.example/lib", + " pkg:pypi/django@3.2.4 ": "pkg:pypi/django", + } + for in, want := range ok { + got, err := PurlBase(in) + if err != nil { + t.Errorf("PurlBase(%q): unexpected error %v", in, err) + continue + } + if got != want { + t.Errorf("PurlBase(%q) = %q, want %q", in, got, want) + } + } + + for _, bad := range []string{"", " ", "npm/lodash", "pkg:", "pkg:npm", "pkg:@1.0"} { + if got, err := PurlBase(bad); err == nil { + t.Errorf("PurlBase(%q) = %q, want an error", bad, got) + } + } +} + +func TestNormalizeMatch(t *testing.T) { + cases := []struct { + name string + in string + want string + }{ + { + name: "comments stripped, whitespace collapsed, literals abstracted", + in: "// lookup\nq := \"SELECT \" + name + \"'\" /* inline */ + 42\n", + want: "$1 := + $2 + + ", + }, + { + name: "same identifier maps to the same metavariable", + in: "a = b + a + b", + want: "$1 = $2 + $1 + $2", + }, + { + name: "callee and member names are preserved", + in: "rows := db.Query(userInput)", + want: "$1 := $2.Query($3)", + }, + { + name: "keywords are preserved", + in: "for i := range items { return nil }", + want: "for $1 := range $2 { return nil }", + }, + { + name: "hash line comments are stripped", + in: "value = compute(x) # trailing note", + want: "$1 = compute($2)", + }, + { + name: "numeric literal forms collapse to one token", + in: "a = 0xFF + 1_000 + 3.14f + 1e-9", + want: "$1 = + + + ", + }, + { + name: "escaped quotes do not terminate a string", + in: `s = "a \" b" + t`, + want: "$1 = + $2", + }, + { + // The receiver `p` is a local and is abstracted; `Field` is a + // member and `Ns`/`Helper` are the qualifier and callee of a + // scope-resolved call, which are API surface, not churn. + name: "arrow and scope selectors preserve member names", + in: "p->Field = Ns::Helper(v)", + want: "$1->Field = Ns::Helper($2)", + }, + { + name: "a scope-resolution chain is preserved end to end", + in: "std::vector v = Foo::Bar::make(x)", + want: "std::vector $1 = Foo::Bar::make($2)", + }, + { + // These two cases exist as a pair: calls into two different + // namespaces must NOT normalise to the same string, which is what + // abstracting the left operand of "::" would do. + name: "namespace qualifier Ns is preserved", + in: "Ns::Helper(v)", + want: "Ns::Helper($1)", + }, + { + name: "namespace qualifier Other is preserved and differs from Ns", + in: "Other::Helper(v)", + want: "Other::Helper($1)", + }, + { + name: "renaming a local does not change the shape", + in: "totalCount := len(itemsList)", + want: "$1 := len($2)", + }, + { + name: "renaming the same local differently yields the same shape", + in: "n := len(xs)", + want: "$1 := len($2)", + }, + { + name: "block comment spanning lines is dropped", + in: "a\n/* one\n two */\nb", + want: "$1 $2", + }, + { + name: "only comments and whitespace normalises to empty", + in: " // nothing here \n\n", + want: "", + }, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + if got := NormalizeMatch(tc.in); got != tc.want { + t.Errorf("NormalizeMatch(%q)\n got: %q\n want: %q", tc.in, got, tc.want) + } + }) + } + + // The normalized form must never contain the separator or a control + // character, or Digest would reject it. + for _, tc := range cases { + if strings.ContainsAny(NormalizeMatch(tc.in), "\x00\x1f\n\r\t") { + t.Errorf("normalized output of %q contains a control character", tc.name) + } + } +} + +func TestNormalizeMatchIsDeterministic(t *testing.T) { + src := "handler := func(w http.ResponseWriter, r *http.Request) { io.WriteString(w, r.URL.Query().Get(\"q\")) }" + first := NormalizeMatch(src) + for i := 0; i < 200; i++ { + if got := NormalizeMatch(src); got != first { + t.Fatalf("NormalizeMatch is not deterministic: %q vs %q", got, first) + } + } + if first == "" { + t.Fatal("normalization produced an empty result for a real snippet") + } +} + +// --------------------------------------------------------------------------- +// Ordinal assignment +// --------------------------------------------------------------------------- + +func TestAssignSastOrdinals(t *testing.T) { + mk := func(path, snippet string, line int) SastCandidate { + return SastCandidate{ + Input: SastInput{ + TargetID: "t-1", + RuleIDVersioned: "r@1", + RepoRelPath: path, + EnclosingSymbolPath: "sym", + Snippet: snippet, + }, + Line: line, + } + } + + // Five matches, given out of source order, spanning two files. + // + // THE GROUPING KEY IS THE NORMALIZED MATCH, NOT THE RAW SNIPPET. The + // specification defines ordinal as "the 0-based index of this match among + // all matches of the same rule_id in the same repo_relpath whose + // NORMALIZED_MATCH is IDENTICAL", and "exec(cmd)" and "exec(other)" both + // normalise to `exec($1)` — `exec` is a callee name so it is preserved, + // and the argument is a local so it becomes $1. All four a.go candidates + // are therefore ONE group; only b.go is separate. + cands := []SastCandidate{ + mk("a.go", "exec(cmd)", 90), + mk("a.go", "exec(cmd)", 10), + mk("b.go", "exec(cmd)", 5), + mk("a.go", "exec(other)", 50), + mk("a.go", "exec(cmd)", 50), + } + + got, err := AssignSastOrdinals(cands) + if err != nil { + t.Fatalf("%v", err) + } + if len(got) != len(cands) { + t.Fatalf("got %d inputs, want %d", len(got), len(cands)) + } + + // Within the a.go group, ordinals follow ascending (line, column, original + // index): line 10 -> 0, line 50 (index 3) -> 1, line 50 (index 4) -> 2, + // line 90 -> 3. The single b.go candidate is a group of one -> 0. The + // returned slice keeps the caller's original order, so the expectation is + // stated in that order. + want := []int{3, 0, 0, 1, 2} + for i := range want { + if got[i].Ordinal != want[i] { + t.Errorf("candidate %d (line %d): ordinal %d, want %d", + i, cands[i].Line, got[i].Ordinal, want[i]) + } + } + + // The collapse of "exec(cmd)" and "exec(other)" onto one normalized match + // is precisely the collision ordinal exists to break: without it the four + // a.go candidates would share one digest and three findings would be lost + // on upsert against UNIQUE (target_id, fingerprint). Assert the resulting + // digests are all distinct. + seen := map[string]int{} + for i, in := range got { + d, err := Sast(in) + if err != nil { + t.Fatalf("candidate %d: %v", i, err) + } + if prev, ok := seen[d]; ok { + t.Errorf("candidates %d and %d collide on digest %s", prev, i, d) + } + seen[d] = i + } + + // A run over the same candidates must produce the same ordinals. + again, err := AssignSastOrdinals(cands) + if err != nil { + t.Fatalf("%v", err) + } + if !reflect.DeepEqual(got, again) { + t.Error("AssignSastOrdinals is not deterministic") + } + + if _, err := AssignSastOrdinals([]SastCandidate{{Input: SastInput{}}}); err == nil { + t.Error("an unfingerprintable candidate must be rejected, not given ordinal 0") + } +} + +// --------------------------------------------------------------------------- +// Validation +// --------------------------------------------------------------------------- + +func TestTierValidationRejectsIncompleteInput(t *testing.T) { + if _, err := Sast(SastInput{RuleIDVersioned: "r", RepoRelPath: "a.go", Snippet: "x"}); err == nil { + t.Error("sast: empty target_id must be rejected") + } + if _, err := Sast(SastInput{TargetID: "t", RepoRelPath: "a.go", Snippet: "x"}); err == nil { + t.Error("sast: empty rule_id_versioned must be rejected") + } + if _, err := Sast(SastInput{TargetID: "t", RuleIDVersioned: "r", Snippet: "x"}); err == nil { + t.Error("sast: empty repo_relpath must be rejected") + } + if _, err := Sast(SastInput{TargetID: "t", RuleIDVersioned: "r", RepoRelPath: "a.go", Snippet: "// only a comment"}); err == nil { + t.Error("sast: a snippet that normalises to nothing must be rejected") + } + if _, err := Sast(SastInput{TargetID: "t", RuleIDVersioned: "r", RepoRelPath: "a.go", Snippet: "x", Ordinal: -1}); err == nil { + t.Error("sast: a negative ordinal must be rejected") + } + // An empty enclosing symbol is legal: top-level code and config files + // have none, and rejecting them would make them unfingerprintable. + if _, err := Sast(SastInput{TargetID: "t", RuleIDVersioned: "r", RepoRelPath: "a.go", Snippet: "exec(x)"}); err != nil { + t.Errorf("sast: an empty enclosing_symbol_path must be accepted, got %v", err) + } + + if _, err := Sca(ScaInput{TargetID: "t", Purl: "pkg:npm/a", ManifestRelPath: "p.json"}); err == nil { + t.Error("sca: empty advisory_id must be rejected") + } + if _, err := Sca(ScaInput{TargetID: "t", AdvisoryID: "CVE-1", ManifestRelPath: "p.json"}); err == nil { + t.Error("sca: empty purl must be rejected") + } + if _, err := Sca(ScaInput{TargetID: "t", AdvisoryID: "CVE-1", Purl: "pkg:npm/a"}); err == nil { + t.Error("sca: empty manifest_relpath must be rejected") + } + + if _, err := Host(HostInput{TargetID: "t", AdvisoryID: "CVE-1", Purl: "pkg:deb/d/o", HostIdentifier: "o"}); err == nil { + t.Error("host: empty package_manager must be rejected") + } + if _, err := Host(HostInput{TargetID: "t", AdvisoryID: "CVE-1", Purl: "pkg:deb/d/o", PackageManager: "a:b", HostIdentifier: "o"}); err == nil { + t.Error("host: a package_manager containing ':' must be rejected") + } + + badDast := DastInput{ + TargetID: "t", RuleIDVersioned: "r", HTTPMethod: "GET", RouteTemplate: "/x", + InjectionPoint: InjectionPoint("QUERY"), EvidenceSignal: EvidenceSignalOther, + } + if _, err := Dast(badDast); err == nil { + t.Error("dast: a SCREAMING_CASE injection point must be rejected; the record's convention is lowercase") + } + badDast.InjectionPoint = InjectionPointQuery + badDast.EvidenceSignal = EvidenceSignal("db_error_string") + if _, err := Dast(badDast); err == nil { + t.Error("dast: an unfrozen evidence signal literal must be rejected") + } + badDast.EvidenceSignal = EvidenceSignalDBErrorString + badDast.HTTPMethod = "" + if _, err := Dast(badDast); err == nil { + t.Error("dast: empty http_method must be rejected") + } + // An empty param name is legal: a whole-body or raw-request injection has + // no single named parameter. + badDast.HTTPMethod = "POST" + badDast.ParamName = "" + if _, err := Dast(badDast); err != nil { + t.Errorf("dast: an empty param_name must be accepted, got %v", err) + } +} + +// TestExportedTierEntryPointsAgreeWithFieldBuilders keeps the two exported +// surfaces from drifting: Sast must be exactly Digest(SastFields(...)), and so +// on for each tier. +func TestExportedTierEntryPointsAgreeWithFieldBuilders(t *testing.T) { + sastIn := SastInput{TargetID: "t", RuleIDVersioned: "r@1", RepoRelPath: "a.go", Snippet: "exec(x)"} + scaIn := ScaInput{TargetID: "t", AdvisoryID: "CVE-1", Purl: "pkg:npm/a@1.0", ManifestRelPath: "p.json"} + hostIn := HostInput{TargetID: "t", AdvisoryID: "CVE-1", Purl: "pkg:deb/d/o@1", PackageManager: "apt", HostIdentifier: "o"} + dastIn := DastInput{TargetID: "t", RuleIDVersioned: "r@1", HTTPMethod: "GET", RouteTemplate: "/x", + InjectionPoint: InjectionPointQuery, ParamName: "q", EvidenceSignal: EvidenceSignalReflectedPayload} + + pairs := []struct { + name string + fields func() ([]string, error) + digest func() (string, error) + want int + }{ + {"sast", func() ([]string, error) { return SastFields(sastIn) }, func() (string, error) { return Sast(sastIn) }, 7}, + {"sca", func() ([]string, error) { return ScaFields(scaIn) }, func() (string, error) { return Sca(scaIn) }, 5}, + {"host", func() ([]string, error) { return HostFields(hostIn) }, func() (string, error) { return Host(hostIn) }, 5}, + {"dast", func() ([]string, error) { return DastFields(dastIn) }, func() (string, error) { return Dast(dastIn) }, 8}, + } + for _, p := range pairs { + t.Run(p.name, func(t *testing.T) { + fields, err := p.fields() + if err != nil { + t.Fatalf("%v", err) + } + if len(fields) != p.want { + t.Errorf("tier %s hashes %d fields, the specification lists %d", p.name, len(fields), p.want) + } + viaFields, err := Digest(fields...) + if err != nil { + t.Fatalf("%v", err) + } + direct, err := p.digest() + if err != nil { + t.Fatalf("%v", err) + } + if viaFields != direct { + t.Errorf("tier %s: Digest(Fields(in)) = %s but the entry point returned %s", + p.name, viaFields, direct) + } + }) + } +} + +// TestTiersUseTheirSpecifiedDiscriminator asserts the literal tier token in +// field position 2, because a swapped discriminator is invisible in a digest +// but makes every producer incompatible. +func TestTiersUseTheirSpecifiedDiscriminator(t *testing.T) { + sastFields, err := SastFields(SastInput{TargetID: "t", RuleIDVersioned: "r", RepoRelPath: "a.go", Snippet: "exec(x)"}) + if err != nil { + t.Fatalf("%v", err) + } + if sastFields[1] != "sast" { + t.Errorf("sast tier discriminator = %q, want \"sast\"", sastFields[1]) + } + dastFields, err := DastFields(DastInput{TargetID: "t", RuleIDVersioned: "r", HTTPMethod: "GET", + RouteTemplate: "/x", InjectionPoint: InjectionPointQuery, EvidenceSignal: EvidenceSignalOther}) + if err != nil { + t.Fatalf("%v", err) + } + if dastFields[1] != "dast" { + t.Errorf("dast tier discriminator = %q, want \"dast\"", dastFields[1]) + } + scaFields, err := ScaFields(ScaInput{TargetID: "t", AdvisoryID: "C", Purl: "pkg:npm/a", ManifestRelPath: "p"}) + if err != nil { + t.Fatalf("%v", err) + } + if scaFields[1] != string(DetectorKindSCA) { + t.Errorf("sca tier discriminator = %q, want %q", scaFields[1], DetectorKindSCA) + } + hostFields, err := HostFields(HostInput{TargetID: "t", AdvisoryID: "C", Purl: "pkg:deb/d/o", + PackageManager: "apt", HostIdentifier: "o"}) + if err != nil { + t.Fatalf("%v", err) + } + if hostFields[1] != string(DetectorKindHost) { + t.Errorf("host tier discriminator = %q, want %q", hostFields[1], DetectorKindHost) + } + if hostFields[4] != "apt:openssl" && hostFields[4] != "apt:o" { + t.Errorf("host locator = %q, want \":\"", hostFields[4]) + } +} + +// TestSeparatorConstantIsUnitSeparator is a belt-and-braces check on the one +// byte the whole scheme depends on. contract.go owns the constant; if it ever +// changed, every stored fingerprint would be orphaned silently. +func TestSeparatorConstantIsUnitSeparator(t *testing.T) { + if FingerprintFieldSeparator != "\x1f" { + t.Fatalf("FingerprintFieldSeparator = %q, want U+001F", FingerprintFieldSeparator) + } + if FingerprintDigestHexLen != 64 { + t.Fatalf("FingerprintDigestHexLen = %d, want 64 (never truncated)", FingerprintDigestHexLen) + } + if FingerprintAlgV1 != "anvil-fp/v1" { + t.Fatalf("FingerprintAlgV1 = %q, want \"anvil-fp/v1\"", FingerprintAlgV1) + } +} diff --git a/internal/store/ddl.go b/internal/store/ddl.go new file mode 100644 index 0000000..7eb2cb6 --- /dev/null +++ b/internal/store/ddl.go @@ -0,0 +1,165 @@ +// Package store owns Anvil's single SQLite store of record. +// +// plan/00-SPINE.md S1 collapsed the originally-specified "8-hour buffer file" +// into one SQLite database, a `handoff` table, and a regenerable tmpfs packet +// that is never a source of truth. This package holds the DDL for that +// database. plan/IMPLEMENTATION-PLAN.md §6 rulings G9 and G10 make schema.sql +// the ONLY definition of the `handoff` table anywhere in Anvil: area 70's O.3 +// migration and area 60's rival `anvil_ledger` are both folded into it. +// +// This file is a thin, dependency-free wrapper. It embeds the DDL, exposes the +// connection pragmas the DDL deliberately does not contain, and offers just +// enough read-only introspection for R.5's migration ledger and for the test +// that proves the SQL vocabularies and internal/record's Go enums have not +// drifted apart. It opens no database and executes no statement. +package store + +import ( + "crypto/sha256" + _ "embed" // for the //go:embed directive on schemaSQL + "encoding/hex" + "fmt" + "regexp" + "strings" +) + +//go:embed schema.sql +var schemaSQL string + +// MaxDurableTextBytes is the byte cap enforced by schema.sql's +// finding_occurrence triggers on `message` and `evidence_ref`. +// +// research/07 Risk #13: if a raw snippet or a DAST request/response body is +// copied into a durable text column it outlives the payload purge forever, and +// those are exactly the fields most likely to carry a secret. The cap is far +// below internal/record's smallest inline body cap (8 KiB) so that no body can +// be smuggled into a durable column even at its minimum size, and ample for a +// rule title plus a pointer into the sealed payload. +// +// Changing this constant alone does nothing: the value is enforced by SQL. +// ddl_test.go asserts the trigger and this constant agree. +const MaxDurableTextBytes = 2048 + +// Schema returns the complete DDL for schema version 1, exactly as committed +// in schema.sql. +// +// It contains no PRAGMA statement, by design. `PRAGMA journal_mode = WAL` +// cannot run inside a transaction and R.5 applies this DDL inside +// BEGIN...COMMIT. Use ConnectionPragmas for the settings that must be applied +// per connection instead. +func Schema() string { return schemaSQL } + +// SchemaSHA256 returns the lowercase hex SHA-256 of the embedded DDL, over its +// exact committed bytes with no normalisation. R.5's migration ledger records +// a checksum; this is the value for the initial migration, and it changes if +// so much as a comment in schema.sql changes, which is the intended +// sensitivity for a frozen interface. +func SchemaSHA256() string { + sum := sha256.Sum256([]byte(schemaSQL)) + return hex.EncodeToString(sum[:]) +} + +// ConnectionPragmas returns the pragmas from plan/40-record-and-storage.md's +// Store Schema section, in the order they must be applied. +// +// Every one of these is per connection, not per database file, so they must be +// re-applied on every connection the pool opens — `foreign_keys` above all, +// which SQLite leaves OFF by default and which this schema depends on. R.5 +// applies them before any other store operation, after its network-mount and +// FTS5 guards. +// +// The `synchronous = NORMAL` value resolves a research conflict on purpose: +// research/07 recommends NORMAL as the documented standard WAL pairing, while +// research/08 recommends FULL without any WAL-specific justification. NORMAL +// with WAL is crash-safe for the database file; the risk it accepts is losing +// the most recent commits on power loss, not corruption. +func ConnectionPragmas() []string { + return []string{ + "PRAGMA journal_mode = WAL", + "PRAGMA foreign_keys = ON", + "PRAGMA busy_timeout = 10000", + "PRAGMA synchronous = NORMAL", + "PRAGMA wal_autocheckpoint = 1000", + } +} + +var tableRE = regexp.MustCompile(`(?m)^CREATE\s+(?:VIRTUAL\s+)?TABLE\s+([A-Za-z_][A-Za-z0-9_]*)`) + +// Tables returns every table name schema.sql creates, ordinary and virtual +// alike, in file order. It is a parse of the committed DDL, not a hand-kept +// list, so a table added to schema.sql without a corresponding smoke-test +// insert fails ddl_test.go rather than going unexercised. +func Tables() []string { + matches := tableRE.FindAllStringSubmatch(schemaSQL, -1) + names := make([]string, 0, len(matches)) + for _, m := range matches { + names = append(names, m[1]) + } + return names +} + +// CheckConstraint returns the text of the named CHECK constraint's expression, +// without the enclosing parentheses. +// +// Every CHECK in schema.sql is named. research/07 Risk #15 records that +// batch-recreate tooling silently drops UNNAMED check constraints, which on a +// security tool's schema is an integrity regression with no error message; +// naming them is that risk's mitigation, and it also makes the constraints +// addressable from a test. +func CheckConstraint(name string) (string, error) { + anchor := regexp.MustCompile(`CONSTRAINT\s+` + regexp.QuoteMeta(name) + `\s+CHECK\s*\(`) + loc := anchor.FindStringIndex(schemaSQL) + if loc == nil { + return "", fmt.Errorf("store: no CHECK constraint named %q in schema.sql", name) + } + // loc[1] is the index just past the opening parenthesis. Scan forward to + // its match, tracking nesting and single-quoted literals so that a + // parenthesis inside a string literal cannot end the scan early. + depth := 1 + inLiteral := false + for i := loc[1]; i < len(schemaSQL); i++ { + switch c := schemaSQL[i]; { + case c == '\'': + // Doubled '' inside a literal is an escaped quote; toggling twice + // leaves the state correct, so no special case is needed. + inLiteral = !inLiteral + case inLiteral: + // Parentheses inside a literal are data, not structure. + case c == '(': + depth++ + case c == ')': + depth-- + if depth == 0 { + return schemaSQL[loc[1]:i], nil + } + } + } + return "", fmt.Errorf("store: CHECK constraint %q has an unbalanced expression in schema.sql", name) +} + +var literalRE = regexp.MustCompile(`'((?:[^']|'')*)'`) + +// EnumCheckValues returns the SQL string literals of the named CHECK +// constraint, in the order they appear. +// +// This exists because of a real constraint on how the vocabulary is written +// down. plan/IMPLEMENTATION-PLAN.md §6 freezes the enums in +// internal/record/contract.go and forbids any second declaration; but a SQL +// CHECK constraint cannot reference a Go constant, and templating the DDL at +// run time would mean schema.sql was no longer a file that applies to an empty +// database on its own — which its own stop condition requires. So the literals +// are written once more, here, and ddl_test.go asserts set equality with +// internal/record's values for every one of them. A drift is a failed test, +// not an integration surprise in Phase 5. +func EnumCheckValues(name string) ([]string, error) { + expr, err := CheckConstraint(name) + if err != nil { + return nil, err + } + matches := literalRE.FindAllStringSubmatch(expr, -1) + values := make([]string, 0, len(matches)) + for _, m := range matches { + values = append(values, strings.ReplaceAll(m[1], "''", "'")) + } + return values, nil +} diff --git a/internal/store/ddl_test.go b/internal/store/ddl_test.go new file mode 100644 index 0000000..68b909f --- /dev/null +++ b/internal/store/ddl_test.go @@ -0,0 +1,608 @@ +package store + +import ( + "database/sql" + "strconv" + "strings" + "testing" + + "github.com/Susquehanna-Syntax/Anvil/internal/record" + + _ "modernc.org/sqlite" // cgo-free driver, plan/00-SPINE.md S12 +) + +// newDB applies ConnectionPragmas and then schema.sql to a fresh in-memory +// database, exactly as R.5's migration will, and returns it. +// +// MaxOpenConns is pinned to 1 because each new connection to ":memory:" is a +// different, empty database. +func newDB(t *testing.T) *sql.DB { + t.Helper() + + db, err := sql.Open("sqlite", ":memory:") + if err != nil { + t.Fatalf("sql.Open: %v", err) + } + t.Cleanup(func() { _ = db.Close() }) + db.SetMaxOpenConns(1) + + for _, p := range ConnectionPragmas() { + // journal_mode is a no-op on an in-memory database (it reports + // "memory"), which is why the WAL guarantees themselves are R.5's to + // verify on a real file. The rest apply normally. + if _, err := db.Exec(p); err != nil { + t.Fatalf("pragma %q: %v", p, err) + } + } + if _, err := db.Exec(Schema()); err != nil { + t.Fatalf("applying schema.sql: %v", err) + } + return db +} + +const ( + // 64 lowercase hex characters, the only shape ck_*_fingerprint_hex admits. + fpA = "a1b2c3d4e5f60718293a4b5c6d7e8f90a1b2c3d4e5f60718293a4b5c6d7e8f90" + sha256Hex = "0011223344556677889900112233445566778899001122334455667788990011" +) + +// seedAll inserts exactly one FK-valid row into every table schema.sql +// creates, then selects each row back. This is R.4's mandated smoke test. +func seedAll(t *testing.T, db *sql.DB) { + t.Helper() + + type seed struct { + table string + insert string + args []any + verify string + } + + seeds := []seed{ + { + table: "target", + insert: `INSERT INTO target (target_id, kind, locator) VALUES (1, 'repo', 'https://example.invalid/r.git')`, + verify: `SELECT locator FROM target WHERE target_id = 1`, + }, + { + table: "trigger_policy", + insert: `INSERT INTO trigger_policy (policy_id, target_id, kind, spec, scan_depth) VALUES (1, 1, 'cron', '0 3 * * *', 'full')`, + verify: `SELECT spec FROM trigger_policy WHERE policy_id = 1`, + }, + { + table: "ingest_watermark", + insert: `INSERT INTO ingest_watermark (source, cursor, etag, last_success_at) VALUES ('osv', 'c1', 'e1', '2026-08-08T00:00:00Z')`, + verify: `SELECT cursor FROM ingest_watermark WHERE source = 'osv'`, + }, + { + table: "advisory", + insert: `INSERT INTO advisory (advisory_id, source, published_at, modified_at, severity, summary, content_hash, ingested_at) + VALUES ('CVE-2026-0001', 'osv', '2026-01-01T00:00:00Z', '2026-01-02T00:00:00Z', 'high', 'sqli in widget', 'h1', '2026-08-08T00:00:00Z')`, + verify: `SELECT summary FROM advisory WHERE advisory_id = 'CVE-2026-0001'`, + }, + { + table: "advisory_alias", + insert: `INSERT INTO advisory_alias (advisory_id, alias_id) VALUES ('CVE-2026-0001', 'GHSA-aaaa-bbbb-cccc')`, + verify: `SELECT alias_id FROM advisory_alias WHERE advisory_id = 'CVE-2026-0001'`, + }, + { + table: "advisory_fts", + insert: `INSERT INTO advisory_fts (rowid, summary, details, aliases) + SELECT rowid, summary, '', 'CVE-2026-0001' FROM advisory WHERE advisory_id = 'CVE-2026-0001'`, + // rowid only, not a column value: `content='advisory'` makes any + // column read go back to `advisory`, which has no `aliases` + // column. See the KNOWN DEFECT note above this table in + // schema.sql. Writes and rowid-only MATCH queries are the part + // that works, and the part R.4 pins here. + verify: `SELECT rowid FROM advisory_fts WHERE advisory_fts MATCH 'sqli'`, + }, + { + table: "component", + insert: `INSERT INTO component (component_id, ecosystem, name, purl_base) VALUES (1, 'pypi', 'requests', 'pkg:pypi/requests')`, + verify: `SELECT purl_base FROM component WHERE component_id = 1`, + }, + { + table: "advisory_affects", + insert: `INSERT INTO advisory_affects (advisory_id, component_id, introduced, fixed, range_kind) VALUES ('CVE-2026-0001', 1, '2.0.0', '2.31.0', 'semver')`, + verify: `SELECT fixed FROM advisory_affects WHERE advisory_id = 'CVE-2026-0001' AND component_id = 1`, + }, + { + table: "scan_run", + insert: `INSERT INTO scan_run (scan_run_id, target_id, policy_id, trigger_ref, commit_sha, started_at, status, ruleset_version) + VALUES (1, 1, 1, 'v1.0.0', 'deadbeef', '2026-08-08T00:00:00Z', 'ok', 'rs/1')`, + verify: `SELECT status FROM scan_run WHERE scan_run_id = 1`, + }, + { + table: "audit_record", + insert: `INSERT INTO audit_record (audit_record_id, scan_run_id, schema_version, state, sast_status, sast_sealed_at, + dast_status, dast_coverage_json, target_provenance, deadline_at, payload_sha256, created_at) + VALUES (1, 1, ?, 'both_sealed', 'sealed', '2026-08-08T01:00:00Z', + 'completed_clean', '{"probedCount":1}', 'booted_clean', '2026-08-08T08:00:00Z', ?, '2026-08-08T00:00:00Z')`, + args: []any{record.SchemaVersion, sha256Hex}, + verify: `SELECT state FROM audit_record WHERE audit_record_id = 1`, + }, + { + table: "code_location", + insert: `INSERT INTO code_location (location_id, repo_relpath, start_line, end_line, symbol, symbol_kind, blob_sha, snippet_hash) + VALUES (1, 'src/app/db.py', 41, 43, 'src/app/db.py::Repo.query', 'method', 'b1', 's1')`, + verify: `SELECT repo_relpath FROM code_location WHERE location_id = 1`, + }, + { + table: "finding", + insert: `INSERT INTO finding (finding_id, target_id, fingerprint, detector, evidence_class, rule_id, + remediable_by_agent, advisory_id, component_id, severity, title, state, + first_seen_scan, first_seen_at) + VALUES (1, 1, ?, 'sast', 'sast_reachable', 'anvil.py.sqli/v3', 0, 'CVE-2026-0001', 1, 'high', 'SQL injection', 'open', 1, '2026-08-08T00:30:00Z')`, + args: []any{fpA}, + verify: `SELECT fingerprint FROM finding WHERE finding_id = 1`, + }, + { + table: "finding_fingerprint", + insert: `INSERT INTO finding_fingerprint (finding_id, kind, alg, value) VALUES (1, 'primary', ?, ?)`, + args: []any{record.FingerprintAlgV1, fpA}, + verify: `SELECT value FROM finding_fingerprint WHERE finding_id = 1 AND kind = 'primary'`, + }, + { + table: "finding_occurrence", + insert: `INSERT INTO finding_occurrence (occurrence_id, finding_id, scan_run_id, location_id, confidence, message, + evidence_ref, advisory_as_of, advisory_staleness_seconds, advisory_parse_degraded) + VALUES (1, 1, 1, 1, 0.9, 'rule anvil.py.sqli/v3 matched', 'runs/0/results/0', '2026-08-07T00:00:00Z', 86400, 0)`, + verify: `SELECT message FROM finding_occurrence WHERE occurrence_id = 1`, + }, + { + table: "finding_state_event", + insert: `INSERT INTO finding_state_event (event_id, finding_id, scan_run_id, from_state, to_state, cause, at) VALUES (1, 1, 1, NULL, 'open', 'first_seen', '2026-08-08T00:30:00Z')`, + verify: `SELECT to_state FROM finding_state_event WHERE event_id = 1`, + }, + { + table: "fix_attempt", + insert: `INSERT INTO fix_attempt (fix_attempt_id, finding_id, audit_record_id, agent_model_id, started_at, status, patch_ref, branch_name, pr_url) + VALUES (1, 1, 1, 'model/x', '2026-08-08T02:00:00Z', 'proposed', 'refs/anvil/fix/1', 'anvil/fix-1', 'https://example.invalid/pr/1')`, + verify: `SELECT status FROM fix_attempt WHERE fix_attempt_id = 1`, + }, + { + table: "verification", + insert: `INSERT INTO verification (verification_id, fix_attempt_id, kind, scan_run_id, result, details_json, verified_at) + VALUES (1, 1, 'rescan_dast', 1, 'pass', '{"reproFailed":true}', '2026-08-08T03:00:00Z')`, + verify: `SELECT result FROM verification WHERE verification_id = 1`, + }, + { + table: "suppression", + insert: `INSERT INTO suppression (suppression_id, target_id, match_kind, match_value, classification, justification, created_by, created_at, active) + VALUES (1, 1, 'fingerprint', ?, 'accepted_risk', 'compensating control', 'operator', '2026-08-08T00:00:00Z', 1)`, + args: []any{fpA}, + verify: `SELECT classification FROM suppression WHERE suppression_id = 1`, + }, + { + table: "file_state", + insert: `INSERT INTO file_state (target_id, repo_relpath, blob_sha, ruleset_version, last_scan_id) VALUES (1, 'src/app/db.py', 'b1', 'rs/1', 1)`, + verify: `SELECT blob_sha FROM file_state WHERE target_id = 1 AND repo_relpath = 'src/app/db.py'`, + }, + { + table: "handoff", + insert: `INSERT INTO handoff (handoff_id, finding_id, audit_record_id, fingerprint, group_id, state, consumption_class, + claimed_by, lease_expires_at, attempts, max_attempts, idempotency_key, created_at, updated_at) + VALUES (1, 1, 1, ?, 'grp-1', 'leased', 'requires_dynamic_confirmation', + 'worker-1', '2026-08-08T00:20:00Z', 0, 2, ?, '2026-08-08T00:00:00Z', '2026-08-08T00:00:00Z')`, + args: []any{fpA, sha256Hex}, + verify: `SELECT state FROM handoff WHERE handoff_id = 1`, + }, + { + table: "schema_migration", + insert: `INSERT INTO schema_migration (version, name, checksum, applied_at) VALUES (1, '0001_init', ?, '2026-08-08T00:00:00Z')`, + args: []any{SchemaSHA256()}, + verify: `SELECT checksum FROM schema_migration WHERE version = 1`, + }, + } + + seeded := make(map[string]bool, len(seeds)) + for _, s := range seeds { + if _, err := db.Exec(s.insert, s.args...); err != nil { + t.Fatalf("insert into %s: %v", s.table, err) + } + var got string + if err := db.QueryRow(s.verify).Scan(&got); err != nil { + t.Fatalf("select back from %s: %v", s.table, err) + } + if got == "" { + t.Errorf("%s: selected an empty value back", s.table) + } + seeded[s.table] = true + } + + // The seed list is checked against the DDL rather than against itself, so + // a table added to schema.sql without a smoke insert fails here. + for _, table := range Tables() { + if !seeded[table] { + t.Errorf("table %q is created by schema.sql but never exercised by the smoke test", table) + } + } + if len(seeded) != len(Tables()) { + t.Errorf("smoke test seeds %d tables, schema.sql creates %d", len(seeded), len(Tables())) + } +} + +func TestSchemaAppliesAndSmokeInsertsRoundTrip(t *testing.T) { + db := newDB(t) + seedAll(t, db) + + var fk int + if err := db.QueryRow(`PRAGMA foreign_keys`).Scan(&fk); err != nil { + t.Fatalf("PRAGMA foreign_keys: %v", err) + } + if fk != 1 { + t.Fatalf("PRAGMA foreign_keys = %d, want 1: the schema depends on FK enforcement", fk) + } +} + +// enumStrings widens any ~string enum slice from internal/record. +func enumStrings[T ~string](vals []T) []string { + out := make([]string, len(vals)) + for i, v := range vals { + out[i] = string(v) + } + return out +} + +// enumChecks maps every CHECK constraint in schema.sql that names a vocabulary +// to the internal/record function that owns it. plan/IMPLEMENTATION-PLAN.md §6 +// makes internal/record the single declaration site; SQL cannot reference a Go +// constant, so this table is the seam, and this test is what keeps the seam +// honest. +var enumChecks = []struct { + constraint string + table string + column string + want []string +}{ + {"ck_scan_run_status", "scan_run", "status", enumStrings(record.ScanRunStatusValues())}, + {"ck_audit_record_state", "audit_record", "state", enumStrings(record.StateValues())}, + {"ck_audit_record_sast_status", "audit_record", "sast_status", enumStrings(record.HalfStatusValues())}, + {"ck_audit_record_dast_status", "audit_record", "dast_status", enumStrings(record.DastStatusValues())}, + {"ck_audit_record_target_provenance", "audit_record", "target_provenance", enumStrings(record.TargetProvenanceValues())}, + {"ck_finding_detector", "finding", "detector", enumStrings(record.DetectorKindValues())}, + {"ck_finding_evidence_class", "finding", "evidence_class", enumStrings(record.EvidenceClassValues())}, + {"ck_finding_verdict", "finding", "verdict", enumStrings(record.VerdictValues())}, + {"ck_finding_state", "finding", "state", enumStrings(record.FindingStateValues())}, + {"ck_handoff_state", "handoff", "state", enumStrings(record.HandoffStateValues())}, + {"ck_handoff_consumption_class", "handoff", "consumption_class", enumStrings(record.ConsumptionClassValues())}, +} + +func TestEnumCheckConstraintsMatchContractLiteralForLiteral(t *testing.T) { + for _, c := range enumChecks { + got, err := EnumCheckValues(c.constraint) + if err != nil { + t.Errorf("%s: %v", c.constraint, err) + continue + } + if len(got) != len(c.want) { + t.Errorf("%s (%s.%s): SQL admits %d literals %v, internal/record declares %d %v", + c.constraint, c.table, c.column, len(got), got, len(c.want), c.want) + continue + } + for i := range got { + if got[i] != c.want[i] { + t.Errorf("%s (%s.%s): literal %d is %q in SQL, %q in internal/record", + c.constraint, c.table, c.column, i, got[i], c.want[i]) + } + } + } +} + +// TestEnumCheckConstraintsEnforceContractValues proves the agreement at run +// time, not only by reading the DDL: every literal internal/record declares is +// accepted by the column, and a literal it does not declare is rejected. +func TestEnumCheckConstraintsEnforceContractValues(t *testing.T) { + db := newDB(t) + seedAll(t, db) + + for _, c := range enumChecks { + update := `UPDATE ` + c.table + ` SET ` + c.column + ` = ? WHERE rowid = (SELECT MIN(rowid) FROM ` + c.table + `)` + for _, v := range c.want { + if _, err := db.Exec(update, v); err != nil { + t.Errorf("%s.%s rejected %q, which internal/record declares legal: %v", c.table, c.column, v, err) + } + } + if _, err := db.Exec(update, "not_a_declared_literal"); err == nil { + t.Errorf("%s.%s accepted an undeclared literal; %s is not enforcing the vocabulary", + c.table, c.column, c.constraint) + } + } +} + +// TestHandoffCarriesAllThirteenDispositions is the G10 regression guard. Area +// X's `anvil_ledger` was deleted and its four extra dispositions folded into +// handoff.state; if that set ever shrinks, X.9 writes a disposition this +// column cannot hold and the ready-set index re-leases the finding forever. +func TestHandoffCarriesAllThirteenDispositions(t *testing.T) { + got, err := EnumCheckValues("ck_handoff_state") + if err != nil { + t.Fatalf("ck_handoff_state: %v", err) + } + if len(got) != 13 { + t.Fatalf("ck_handoff_state admits %d states %v, want the 13 of IMPLEMENTATION-PLAN.md §6", len(got), got) + } + for _, needed := range []record.HandoffState{ + record.HandoffStateSkippedBudget, + record.HandoffStateFixedIncidentally, + record.HandoffStateSplitRequired, + record.HandoffStateWithdrawn, + record.HandoffStateSuperseded, + } { + if !contains(got, string(needed)) { + t.Errorf("ck_handoff_state is missing %q, one of the dispositions collapsed in from anvil_ledger", needed) + } + } +} + +// TestNoSecondDispositionTable guards the S1 spine rule directly: one durable +// table carrying finding dispositions, not two. +func TestNoSecondDispositionTable(t *testing.T) { + for _, table := range Tables() { + if strings.Contains(table, "ledger") { + t.Errorf("schema.sql creates %q; §6 G10 collapses every finding disposition into handoff.state", table) + } + } + // No other column may admit a handoff disposition. `audit_record.state` is + // the audit's own lifecycle and `finding.state` is the durable finding + // lifecycle; neither is a queue disposition, and the overlap that would + // signal a second ledger is a literal only handoff.state should carry. + for _, c := range enumChecks { + if c.table == "handoff" { + continue + } + for _, v := range c.want { + if v == string(record.HandoffStateSkippedBudget) || v == string(record.HandoffStateLeased) { + t.Errorf("%s.%s admits the handoff disposition %q; dispositions live only in handoff.state", c.table, c.column, v) + } + } + } +} + +func TestConsumptionClassHasNoDefault(t *testing.T) { + db := newDB(t) + + notNull, dflt := columnInfo(t, db, "handoff", "consumption_class") + if !notNull { + t.Error("handoff.consumption_class must be NOT NULL: the static-only vs requires-dynamic-confirmation gate has no other home in the schema") + } + if dflt.Valid { + t.Errorf("handoff.consumption_class has DEFAULT %q; a default silently grants every row the permissive value, which 00-SPINE.md S7 forbids", dflt.String) + } +} + +// TestColumnDefaultsMatchContract keeps the DDL's literal defaults tied to the +// Go constants they mirror. +func TestColumnDefaultsMatchContract(t *testing.T) { + db := newDB(t) + + cases := []struct { + table, column, want string + }{ + {"finding", "fingerprint_alg", "'" + record.FingerprintAlgV1 + "'"}, + {"finding", "verdict", "'" + string(record.VerdictTruePositive) + "'"}, + {"audit_record", "dast_status", "'" + string(record.DastStatusNotRun) + "'"}, + {"audit_record", "claim_timeout_seconds", strconv.Itoa(record.DefaultClaimTimeoutSeconds)}, + {"handoff", "state", "'" + string(record.HandoffStateReady) + "'"}, + } + for _, c := range cases { + _, dflt := columnInfo(t, db, c.table, c.column) + if !dflt.Valid { + t.Errorf("%s.%s has no DEFAULT, want %s", c.table, c.column, c.want) + continue + } + if dflt.String != c.want { + t.Errorf("%s.%s DEFAULT is %s, internal/record says %s", c.table, c.column, dflt.String, c.want) + } + } +} + +// TestFingerprintColumnsEnforceContractDigestLength ties the SQL length check +// to record.FingerprintDigestHexLen. CRITIQUE-01 §9 records that the digest is +// 64 lowercase hex characters and is never truncated; a truncating writer must +// fail at the column, not silently halve the collision resistance. +func TestFingerprintColumnsEnforceContractDigestLength(t *testing.T) { + db := newDB(t) + seedAll(t, db) + + exact := strings.Repeat("a", record.FingerprintDigestHexLen) + short := strings.Repeat("a", record.FingerprintDigestHexLen-1) + long := strings.Repeat("a", record.FingerprintDigestHexLen+1) + upper := strings.Repeat("A", record.FingerprintDigestHexLen) + nonHex := strings.Repeat("z", record.FingerprintDigestHexLen) + + for _, table := range []string{"finding", "handoff"} { + update := `UPDATE ` + table + ` SET fingerprint = ? WHERE rowid = 1` + if _, err := db.Exec(update, exact); err != nil { + t.Errorf("%s.fingerprint rejected a full %d-hex digest: %v", table, record.FingerprintDigestHexLen, err) + } + for name, bad := range map[string]string{"truncated": short, "over-long": long, "uppercase": upper, "non-hex": nonHex} { + if _, err := db.Exec(update, bad); err == nil { + t.Errorf("%s.fingerprint accepted a %s digest", table, name) + } + } + // Restore for the next table's FK-valid state. + if _, err := db.Exec(update, exact); err != nil { + t.Fatalf("restoring %s.fingerprint: %v", table, err) + } + } +} + +// TestHostFindingsAreNeverAgentRemediable is 00-SPINE.md S7 in the schema: the +// host agent is read-only, "no package manager in a mutating mode, not behind +// a flag." +func TestHostFindingsAreNeverAgentRemediable(t *testing.T) { + db := newDB(t) + seedAll(t, db) + + _, err := db.Exec(`INSERT INTO finding (finding_id, target_id, fingerprint, detector, evidence_class, rule_id, + remediable_by_agent, severity, title, state, first_seen_scan, first_seen_at) + VALUES (2, 1, ?, 'host', 'host', 'anvil.host.pkg/v1', 1, 'high', 'vulnerable host package', 'open', 1, '2026-08-08T00:30:00Z')`, + strings.Repeat("b", record.FingerprintDigestHexLen)) + if err == nil { + t.Error("a host finding was accepted with remediable_by_agent = 1") + } +} + +// TestDurableTextCapRejectsOversizedMessage is research/07 Risk #13's named +// mitigation: a raw snippet or DAST body copied into a durable column outlives +// the payload purge forever. +func TestDurableTextCapRejectsOversizedMessage(t *testing.T) { + db := newDB(t) + seedAll(t, db) + + for _, column := range []string{"message", "evidence_ref"} { + update := `UPDATE finding_occurrence SET ` + column + ` = ? WHERE occurrence_id = 1` + if _, err := db.Exec(update, strings.Repeat("x", MaxDurableTextBytes)); err != nil { + t.Errorf("finding_occurrence.%s rejected a value at exactly the cap: %v", column, err) + } + if _, err := db.Exec(update, strings.Repeat("x", MaxDurableTextBytes+1)); err == nil { + t.Errorf("finding_occurrence.%s accepted a value one byte over the cap", column) + } + } + + // The same cap must hold on INSERT, not only on UPDATE. + _, err := db.Exec(`INSERT INTO finding_occurrence (occurrence_id, finding_id, scan_run_id, message) + VALUES (2, 1, 1, ?)`, strings.Repeat("x", MaxDurableTextBytes+1)) + if err == nil { + t.Error("finding_occurrence accepted an oversized message on INSERT") + } +} + +func TestForeignKeysAreEnforced(t *testing.T) { + db := newDB(t) + seedAll(t, db) + + _, err := db.Exec(`INSERT INTO handoff (handoff_id, finding_id, audit_record_id, fingerprint, consumption_class, created_at, updated_at) + VALUES (99, 4242, 1, ?, 'static_only', '2026-08-08T00:00:00Z', '2026-08-08T00:00:00Z')`, fpA) + if err == nil { + t.Error("handoff accepted a row referencing a finding that does not exist") + } +} + +// TestLeasedRowMustNameItsHolder: a row in state 'leased' with no claimed_by +// is unreclaimable, because the reaper's requeue rule keys on the holder and +// the lease clock. +func TestLeasedRowMustNameItsHolder(t *testing.T) { + db := newDB(t) + seedAll(t, db) + + _, err := db.Exec(`UPDATE handoff SET state = 'leased', claimed_by = NULL WHERE handoff_id = 1`) + if err == nil { + t.Error("handoff accepted state = 'leased' with a NULL claimed_by") + } +} + +// TestSchemaCarriesNoPragma: R.5 applies this DDL inside BEGIN...COMMIT, and +// `PRAGMA journal_mode = WAL` cannot run inside a transaction. The pragmas +// live in ConnectionPragmas instead, because they are per connection anyway. +func TestSchemaCarriesNoPragma(t *testing.T) { + if strings.Contains(strings.ToUpper(stripSQLComments(Schema())), "PRAGMA") { + t.Error("schema.sql contains a PRAGMA statement; it cannot then be applied inside a transaction") + } + want := []string{"journal_mode = WAL", "foreign_keys = ON", "busy_timeout = 10000", "synchronous = NORMAL", "wal_autocheckpoint = 1000"} + got := ConnectionPragmas() + if len(got) != len(want) { + t.Fatalf("ConnectionPragmas returned %d pragmas %v, want %d", len(got), got, len(want)) + } + for i := range want { + if !strings.Contains(got[i], want[i]) { + t.Errorf("pragma %d is %q, want it to set %q", i, got[i], want[i]) + } + } +} + +// TestEveryResearch07TableSurvives: R.4 may not drop a research/07 table +// without logging the reason, and this schema drops none. +func TestEveryResearch07TableSurvives(t *testing.T) { + carriedForward := []string{ + "advisory", "advisory_alias", "advisory_fts", "component", "advisory_affects", + "target", "trigger_policy", "ingest_watermark", "scan_run", "audit_record", + "code_location", "finding", "finding_fingerprint", "finding_occurrence", + "finding_state_event", "fix_attempt", "verification", "suppression", + "file_state", "schema_migration", + } + created := Tables() + for _, want := range carriedForward { + if !contains(created, want) { + t.Errorf("research/07 table %q is missing from schema.sql", want) + } + } + if !contains(created, "handoff") { + t.Error("the handoff table is missing; S1 collapses the buffer into it") + } +} + +func TestSchemaSHA256IsStableAndFullLength(t *testing.T) { + first := SchemaSHA256() + if len(first) != record.FingerprintDigestHexLen { + t.Errorf("SchemaSHA256 returned %d characters, want %d", len(first), record.FingerprintDigestHexLen) + } + if second := SchemaSHA256(); first != second { + t.Errorf("SchemaSHA256 is not stable: %q then %q", first, second) + } +} + +func TestCheckConstraintReportsMissingNames(t *testing.T) { + if _, err := CheckConstraint("ck_not_a_real_constraint"); err == nil { + t.Error("CheckConstraint reported success for a constraint that does not exist") + } + if _, err := EnumCheckValues("ck_not_a_real_constraint"); err == nil { + t.Error("EnumCheckValues reported success for a constraint that does not exist") + } +} + +// ---- helpers ---- + +func columnInfo(t *testing.T, db *sql.DB, table, column string) (notNull bool, dflt sql.NullString) { + t.Helper() + + rows, err := db.Query(`SELECT name, "notnull", dflt_value FROM pragma_table_info(?)`, table) + if err != nil { + t.Fatalf("pragma_table_info(%s): %v", table, err) + } + defer func() { _ = rows.Close() }() + + for rows.Next() { + var name string + var nn int + var d sql.NullString + if err := rows.Scan(&name, &nn, &d); err != nil { + t.Fatalf("scanning pragma_table_info(%s): %v", table, err) + } + if name == column { + if err := rows.Err(); err != nil { + t.Fatalf("pragma_table_info(%s): %v", table, err) + } + return nn != 0, d + } + } + if err := rows.Err(); err != nil { + t.Fatalf("pragma_table_info(%s): %v", table, err) + } + t.Fatalf("%s has no column %q", table, column) + return false, sql.NullString{} +} + +func contains(haystack []string, needle string) bool { + for _, s := range haystack { + if s == needle { + return true + } + } + return false +} + +// stripSQLComments removes `-- ...` line comments so that a PRAGMA mentioned +// in a comment does not fail TestSchemaCarriesNoPragma. +func stripSQLComments(sqlText string) string { + var b strings.Builder + for _, line := range strings.Split(sqlText, "\n") { + if i := strings.Index(line, "--"); i >= 0 { + line = line[:i] + } + b.WriteString(line) + b.WriteByte('\n') + } + return b.String() +} diff --git a/internal/store/guards.go b/internal/store/guards.go new file mode 100644 index 0000000..6d5ac28 --- /dev/null +++ b/internal/store/guards.go @@ -0,0 +1,457 @@ +// Startup guards for the Anvil store (step R.5). +// +// plan/40-record-and-storage.md, "Startup guards (R.5), both mandatory": +// refuse to start if the data directory is on a network-mounted filesystem, +// and refuse to start if an FTS5 smoke-test virtual table cannot be created. +// Both must run before any other store operation; CheckStartup runs them in +// the required order and Migrate calls CheckFTS5 itself so that a caller who +// builds its own *sql.DB cannot skip it. +// +// Neither guard trusts a version number, a build tag, or a claim in a +// document. plan/00-SPINE.md S12 calls modernc.org/sqlite's FTS5 support +// "orchestrator-verified", but the research trail behind that +// (plan/spine-c-language.md C5-C7) grades its own evidence B, +// "absence-of-evidence, not evidence-of-absence". A dependency bump can drop a +// build-time feature without any signal at all, and the only thing that +// catches that is executing the feature at every process start. So CheckFTS5 +// creates a real FTS5 table, writes a real row, and runs a real MATCH. + +package store + +import ( + "bufio" + "context" + "database/sql" + "errors" + "fmt" + "io/fs" + "os" + "path/filepath" + "runtime" + "sort" + "strings" + "sync/atomic" +) + +// ErrNetworkMount reports that the data directory is on a filesystem where +// SQLite's write-ahead log is documented not to work. +// +// research/07-database-design.md Risk #4: "WAL does not work over a network +// filesystem", and homelab users routinely put data directories on NFS or SMB. +// The failure is silent corruption of the store of record, not an error at +// mount time, which is why this is a refusal and not a warning. +var ErrNetworkMount = errors.New("store: data directory is on a network filesystem") + +// ErrNoFTS5 reports that the SQLite build behind the *sql.DB cannot create or +// query an FTS5 virtual table. schema.sql's advisory_fts table needs it. +var ErrNoFTS5 = errors.New("store: SQLite FTS5 is unavailable") + +// CheckStartup runs both mandatory guards in the order the store needs them, +// and is the single call a process should make before any other store +// operation. +// +// The mount check comes first because it is answerable from the filesystem +// alone: if it fails there is no reason to have opened a database at all. +func CheckStartup(dataDir string, db *sql.DB) error { + if err := CheckNetworkMount(dataDir); err != nil { + return err + } + return CheckFTS5(db) +} + +// --------------------------------------------------------------------------- +// Guard 1 — network-mounted data directory +// --------------------------------------------------------------------------- + +// networkFilesystemTypes is the set of filesystem type names that mean "this +// data lives on another host". A type in this set is a hard refusal. +// +// The set is deliberately explicit rather than a heuristic: guessing from the +// mount source would refuse loopback-mounted images and other perfectly local +// setups, and a false refusal at startup is as user-hostile as a missed one is +// dangerous. +var networkFilesystemTypes = map[string]bool{ + "9p": true, // Plan 9 / virtio-9p, the usual VM share + "afpfs": true, + "afs": true, + "beegfs": true, + "ceph": true, + "cifs": true, + "coda": true, + "davfs": true, + "gfs2": true, + "glusterfs": true, + "lustre": true, + "ncpfs": true, + "nfs": true, + "nfs4": true, + "ocfs2": true, + "orangefs": true, + "pvfs2": true, + "smb2": true, + "smb3": true, + "smbfs": true, + "unc": true, // synthesised below for a Windows \\server\share path + "vboxsf": true, + "virtiofs": true, +} + +// networkFUSEBackends covers FUSE mounts, whose type is reported as +// "fuse.". The FUSE layer itself is local; the backend is what +// decides. +var networkFUSEBackends = map[string]bool{ + "cephfs": true, + "davfs": true, + "glusterfs": true, + "gvfsd-fuse": true, + "rclone": true, + "s3fs": true, + "sshfs": true, +} + +// fsTypeProbe reports the filesystem type backing path, or "" when the host +// gives no way to tell. It is a variable-shaped seam so guards_test.go can +// inject a fake network filesystem type without needing a real NFS server. +type fsTypeProbe func(path string) (string, error) + +// CheckNetworkMount refuses a data directory that sits on a network +// filesystem. +// +// KNOWN LIMIT, stated plainly because a guard that overstates its coverage is +// worse than one that does not exist. Filesystem type detection here is pure +// Go with no cgo and no build-tagged files (plan/00-SPINE.md S12), which means +// it reads /proc/self/mountinfo (falling back to /proc/mounts). On Linux — +// Anvil's deployment target — that is authoritative. On Windows it detects a +// UNC path but CANNOT see that a mapped drive letter such as Z: points at a +// share, because that needs GetDriveTypeW and therefore a windows-only source +// file this step does not own. On any host where the type cannot be +// determined the guard passes: refusing every path we cannot classify would +// make Anvil unstartable on Windows and macOS, and a guard nobody can get past +// gets deleted. Treat a positive result as reliable and a negative one as +// "not proven local". +func CheckNetworkMount(path string) error { + return checkNetworkMountWith(path, probeFilesystemType) +} + +func checkNetworkMountWith(path string, probe fsTypeProbe) error { + if strings.TrimSpace(path) == "" { + return fmt.Errorf("%w: no data directory was given", ErrNetworkMount) + } + fsType, err := probe(path) + if err != nil { + // The host offered a mount table and it could not be read or parsed. + // That is not "unknown", it is broken, and staying quiet about it is + // exactly the silence this guard exists to prevent. + return fmt.Errorf("store: cannot determine the filesystem type of data directory %q: %w", path, err) + } + if fsType == "" { + return nil // undeterminable; see the CheckNetworkMount doc comment + } + if !isNetworkFilesystemType(fsType) { + return nil + } + return fmt.Errorf("%w: %q is on a %q filesystem. SQLite's write-ahead log "+ + "does not work over a network filesystem, so leaving it here risks silent "+ + "corruption of the store of record rather than an error you would notice "+ + "(research/07-database-design.md Risk #4). Move the Anvil data directory to "+ + "local storage, or mount local storage at this path", + ErrNetworkMount, path, fsType) +} + +func isNetworkFilesystemType(fsType string) bool { + fsType = strings.ToLower(strings.TrimSpace(fsType)) + if networkFilesystemTypes[fsType] { + return true + } + if backend, ok := strings.CutPrefix(fsType, "fuse."); ok { + return networkFUSEBackends[backend] + } + return false +} + +// probeFilesystemType is the real probe: a UNC check on Windows, then the +// Linux mount table. +func probeFilesystemType(path string) (string, error) { + if runtime.GOOS == "windows" && isUNCPath(path) { + return "unc", nil + } + return mountTableFilesystemType(path) +} + +// isUNCPath reports whether path names a Windows network share directly, as +// \\server\share\... — the one network location Windows exposes without a +// syscall. +func isUNCPath(path string) bool { + p := strings.ReplaceAll(path, "/", `\`) + if !strings.HasPrefix(p, `\\`) { + return false + } + if strings.HasPrefix(p, `\\?\`) || strings.HasPrefix(p, `\\.\`) { + return false // device / extended-length namespace, not a share + } + rest := strings.Trim(p[2:], `\`) + host, share, ok := strings.Cut(rest, `\`) + return ok && host != "" && strings.TrimLeft(share, `\`) != "" +} + +// mountTableFilesystemType returns the type of the most specific mount point +// containing path, or "" if this host publishes no mount table. +func mountTableFilesystemType(path string) (string, error) { + target, err := resolveForMountLookup(path) + if err != nil { + return "", err + } + + mounts, err := readMountTable() + if err != nil { + return "", err + } + if len(mounts) == 0 { + return "", nil + } + + // Longest matching mount point wins: /mnt/nas is more specific than /, and + // a data directory under it is on the share, not on the root filesystem. + sort.SliceStable(mounts, func(i, j int) bool { + return len(mounts[i].point) > len(mounts[j].point) + }) + for _, m := range mounts { + if pathHasPrefix(m.point, target) { + return m.fsType, nil + } + } + return "", nil +} + +// resolveForMountLookup makes path absolute and resolves it against the +// nearest ancestor that actually exists, because the data directory is +// routinely created after this guard runs. +func resolveForMountLookup(path string) (string, error) { + abs, err := filepath.Abs(path) + if err != nil { + return "", fmt.Errorf("resolving %q: %w", path, err) + } + existing := abs + for { + if _, err := os.Lstat(existing); err == nil { + break + } + parent := filepath.Dir(existing) + if parent == existing { + return filepath.Clean(abs), nil + } + existing = parent + } + // A symlink can cross a mount boundary; the mount that matters is the one + // holding the real directory. Best effort — an unresolvable link is not a + // reason to refuse startup. + if resolved, err := filepath.EvalSymlinks(existing); err == nil { + suffix := strings.TrimPrefix(abs, existing) + return filepath.Clean(resolved + suffix), nil + } + return filepath.Clean(abs), nil +} + +type mountEntry struct { + point string + fsType string +} + +// readMountTable parses /proc/self/mountinfo, falling back to /proc/mounts. +// +// mountinfo is preferred because its mount-point field is unambiguous even +// when a filesystem is bind-mounted or has a non-root subtree mounted, both of +// which /proc/mounts renders confusingly. +func readMountTable() ([]mountEntry, error) { + entries, err := parseMountFile("/proc/self/mountinfo", parseMountinfoLine) + if err == nil { + return entries, nil + } + if !errors.Is(err, fs.ErrNotExist) { + return nil, err + } + entries, err = parseMountFile("/proc/mounts", parseProcMountsLine) + if err == nil { + return entries, nil + } + if errors.Is(err, fs.ErrNotExist) { + return nil, nil // no mount table on this host: undeterminable, not an error + } + return nil, err +} + +func parseMountFile(name string, parse func(string) (mountEntry, bool)) ([]mountEntry, error) { + f, err := os.Open(name) //nolint:gosec // fixed, non-user-supplied kernel path + if err != nil { + return nil, err + } + defer func() { _ = f.Close() }() + + var entries []mountEntry + scanner := bufio.NewScanner(f) + scanner.Buffer(make([]byte, 0, 64*1024), 1024*1024) + for scanner.Scan() { + if e, ok := parse(scanner.Text()); ok { + entries = append(entries, e) + } + } + if err := scanner.Err(); err != nil { + return nil, fmt.Errorf("reading %s: %w", name, err) + } + return entries, nil +} + +// parseMountinfoLine reads one /proc/self/mountinfo record. Layout: +// +// ID PARENT MAJ:MIN ROOT MOUNTPOINT OPTIONS [OPTIONAL...] - FSTYPE SOURCE SUPEROPTS +// +// The optional-fields run is variable length and terminated by a lone "-", so +// the separator has to be located rather than assumed at a fixed index. +func parseMountinfoLine(line string) (mountEntry, bool) { + fields := strings.Fields(line) + sep := -1 + for i, f := range fields { + if f == "-" { + sep = i + break + } + } + if sep < 5 || sep+1 >= len(fields) { + return mountEntry{}, false + } + return mountEntry{ + point: unescapeMountField(fields[4]), + fsType: unescapeMountField(fields[sep+1]), + }, true +} + +// parseProcMountsLine reads one /proc/mounts record: +// +// SOURCE MOUNTPOINT FSTYPE OPTIONS DUMP PASS +func parseProcMountsLine(line string) (mountEntry, bool) { + fields := strings.Fields(line) + if len(fields) < 3 { + return mountEntry{}, false + } + return mountEntry{ + point: unescapeMountField(fields[1]), + fsType: unescapeMountField(fields[2]), + }, true +} + +// unescapeMountField undoes the kernel's octal escaping of space (\040), tab +// (\011), newline (\012) and backslash (\134) in mount paths. Without it a +// data directory containing a space silently fails to match its own mount. +func unescapeMountField(s string) string { + if !strings.ContainsRune(s, '\\') { + return s + } + var b strings.Builder + b.Grow(len(s)) + for i := 0; i < len(s); i++ { + if s[i] != '\\' || i+3 >= len(s) { + b.WriteByte(s[i]) + continue + } + var v int + valid := true + for _, c := range []byte(s[i+1 : i+4]) { + if c < '0' || c > '7' { + valid = false + break + } + v = v*8 + int(c-'0') + } + if !valid || v > 0xff { + b.WriteByte(s[i]) + continue + } + b.WriteByte(byte(v)) + i += 3 + } + return b.String() +} + +// pathHasPrefix reports whether target is mountPoint or lies beneath it, +// comparing whole path elements so that /mnt/nas does not match /mnt/nasty. +func pathHasPrefix(mountPoint, target string) bool { + mp := filepath.Clean(mountPoint) + tg := filepath.Clean(target) + if mp == tg { + return true + } + if !strings.HasSuffix(mp, string(filepath.Separator)) { + mp += string(filepath.Separator) + } + return strings.HasPrefix(tg, mp) +} + +// --------------------------------------------------------------------------- +// Guard 2 — FTS5 availability +// --------------------------------------------------------------------------- + +// fts5ProbeSeq keeps concurrent or repeated probes on separate table names, so +// one probe can never observe another's leftovers and report a false pass. +var fts5ProbeSeq atomic.Uint64 + +// CheckFTS5 verifies, by doing it, that this database can create an FTS5 +// virtual table, index a row into it, and match that row back. +// +// It is deliberately not a version check, a build-tag check, or a lookup in +// pragma_compile_options: those report intent, and this guard exists to catch +// the case where intent and reality have come apart — a dependency bump that +// drops FTS5, or a driver that accepts the DDL and then indexes nothing. Both +// of those are silent failures at startup and loud, data-losing ones at first +// query against schema.sql's advisory_fts table. +// +// The probe table is created in the `temp` schema on a single pinned +// connection and dropped again, so it never touches the store file. +func CheckFTS5(db *sql.DB) error { + return CheckFTS5Context(context.Background(), db) +} + +// CheckFTS5Context is CheckFTS5 with a caller-supplied context. The +// context-free signature is the one plan/40-record-and-storage.md's R.5 packet +// specifies, so it stays; this is the form startup code with a deadline wants. +func CheckFTS5Context(ctx context.Context, db *sql.DB) error { + if db == nil { + return fmt.Errorf("%w: no database handle was given to the guard", ErrNoFTS5) + } + + // One pinned connection: `temp` objects are per connection, so creating on + // one and querying on another would test nothing and leak a table. + conn, err := db.Conn(ctx) + if err != nil { + return fmt.Errorf("store: FTS5 guard could not acquire a connection: %w", err) + } + defer func() { _ = conn.Close() }() + + table := fmt.Sprintf("anvil_fts5_probe_%d", fts5ProbeSeq.Add(1)) + drop := "DROP TABLE IF EXISTS temp." + table + _, _ = conn.ExecContext(ctx, drop) + defer func() { _, _ = conn.ExecContext(ctx, drop) }() + + const remedy = "Anvil's store needs FTS5 for schema.sql's advisory_fts table. " + + "plan/00-SPINE.md S12 pins modernc.org/sqlite precisely because it bundles " + + "FTS5; if this fails, the SQLite build behind this binary changed and the " + + "store cannot be opened safely" + + if _, err := conn.ExecContext(ctx, "CREATE VIRTUAL TABLE temp."+table+" USING fts5(body)"); err != nil { + return fmt.Errorf("%w: CREATE VIRTUAL TABLE ... USING fts5 was rejected: %v. %s", ErrNoFTS5, err, remedy) + } + if _, err := conn.ExecContext(ctx, "INSERT INTO temp."+table+"(body) VALUES ('anvil fts5 startup probe')"); err != nil { + return fmt.Errorf("%w: the FTS5 table was created but would not accept a row: %v. %s", ErrNoFTS5, err, remedy) + } + + var matched int + query := "SELECT count(*) FROM temp." + table + " WHERE " + table + " MATCH 'startup'" + if err := conn.QueryRowContext(ctx, query).Scan(&matched); err != nil { + return fmt.Errorf("%w: the FTS5 table was created and written but MATCH failed: %v. %s", ErrNoFTS5, err, remedy) + } + if matched != 1 { + return fmt.Errorf("%w: MATCH 'startup' returned %d rows, want exactly 1 — this SQLite build "+ + "accepts FTS5 syntax without indexing anything, which is the silent-failure case the guard "+ + "exists to catch. %s", ErrNoFTS5, matched, remedy) + } + return nil +} diff --git a/internal/store/guards_test.go b/internal/store/guards_test.go new file mode 100644 index 0000000..d93e1b7 --- /dev/null +++ b/internal/store/guards_test.go @@ -0,0 +1,317 @@ +package store + +import ( + "context" + "database/sql" + "database/sql/driver" + "errors" + "io" + "path/filepath" + "strings" + "testing" + + _ "modernc.org/sqlite" // cgo-free driver, plan/00-SPINE.md S12 +) + +// --------------------------------------------------------------------------- +// Guard 1 — CheckNetworkMount +// --------------------------------------------------------------------------- + +// TestCheckNetworkMountRefusesInjectedNetworkFilesystems is the R.5 packet's +// "CheckNetworkMount fails against an injected fake network filesystem type" +// evidence item. The probe is the seam; no NFS server is involved. +func TestCheckNetworkMountRefusesInjectedNetworkFilesystems(t *testing.T) { + refused := []string{ + "nfs", "nfs4", "cifs", "smb3", "smbfs", "9p", "ceph", "glusterfs", + "lustre", "afs", "davfs", "vboxsf", "virtiofs", "unc", + "fuse.sshfs", "fuse.davfs", "fuse.s3fs", "fuse.rclone", + "NFS4", " cifs ", // case and whitespace must not smuggle a mount past + } + for _, fsType := range refused { + t.Run(strings.TrimSpace(fsType), func(t *testing.T) { + probe := func(string) (string, error) { return fsType, nil } + err := checkNetworkMountWith("/srv/anvil", probe) + if err == nil { + t.Fatalf("CheckNetworkMount accepted a %q data directory", fsType) + } + if !errors.Is(err, ErrNetworkMount) { + t.Fatalf("error does not wrap ErrNetworkMount: %v", err) + } + // "Fails loudly" means the operator can act on the message: it has + // to name the path and say what to do. + for _, want := range []string{"/srv/anvil", "write-ahead log", "local storage"} { + if !strings.Contains(err.Error(), want) { + t.Errorf("error message omits %q: %v", want, err) + } + } + }) + } +} + +func TestCheckNetworkMountAcceptsLocalFilesystems(t *testing.T) { + // fuseblk and fuse.gocryptfs are local; refusing them would be a false + // positive that pushes users to delete the guard. + for _, fsType := range []string{"ext4", "xfs", "btrfs", "zfs", "apfs", "ntfs", "overlay", "tmpfs", "fuseblk", "fuse.gocryptfs"} { + probe := func(string) (string, error) { return fsType, nil } + if err := checkNetworkMountWith("/srv/anvil", probe); err != nil { + t.Errorf("CheckNetworkMount refused local filesystem %q: %v", fsType, err) + } + } +} + +func TestCheckNetworkMountPassesWhenTypeIsUndeterminable(t *testing.T) { + probe := func(string) (string, error) { return "", nil } + if err := checkNetworkMountWith("/srv/anvil", probe); err != nil { + t.Fatalf("an undeterminable filesystem type must not block startup: %v", err) + } +} + +// TestCheckNetworkMountSurfacesProbeFailures pins the difference between "this +// host has no mount table" (pass) and "this host has one and it could not be +// read" (fail). The second is the case where staying quiet would hide exactly +// what the guard is for. +func TestCheckNetworkMountSurfacesProbeFailures(t *testing.T) { + sentinel := errors.New("mount table is unreadable") + probe := func(string) (string, error) { return "", sentinel } + err := checkNetworkMountWith("/srv/anvil", probe) + if !errors.Is(err, sentinel) { + t.Fatalf("probe failure was swallowed: %v", err) + } +} + +func TestCheckNetworkMountRejectsEmptyPath(t *testing.T) { + if err := CheckNetworkMount(" "); !errors.Is(err, ErrNetworkMount) { + t.Fatalf("empty data directory should be refused, got %v", err) + } +} + +// TestCheckNetworkMountOnRealHostAcceptsTempDir exercises the real probe end +// to end. A temp directory is local on every machine this is expected to run +// on, so a refusal here means the probe misreads its own host. +func TestCheckNetworkMountOnRealHostAcceptsTempDir(t *testing.T) { + if err := CheckNetworkMount(t.TempDir()); err != nil { + t.Fatalf("real probe refused a local temp directory: %v", err) + } + // A directory that does not exist yet must resolve through its nearest + // existing ancestor: the data directory is created after this guard runs. + if err := CheckNetworkMount(filepath.Join(t.TempDir(), "anvil", "data")); err != nil { + t.Fatalf("real probe refused a not-yet-created data directory: %v", err) + } +} + +func TestUNCPathDetection(t *testing.T) { + unc := []string{`\\nas\anvil`, `\\nas\anvil\data`, `//nas/anvil`} + for _, p := range unc { + if !isUNCPath(p) { + t.Errorf("isUNCPath(%q) = false, want true", p) + } + } + notUNC := []string{`C:\anvil`, `/srv/anvil`, `\\?\C:\anvil`, `\\.\PIPE\x`, `\\nas`, `\\nas\`, ``} + for _, p := range notUNC { + if isUNCPath(p) { + t.Errorf("isUNCPath(%q) = true, want false", p) + } + } +} + +func TestMountTableParsing(t *testing.T) { + mountinfo := "36 35 98:0 / /srv rw,noatime shared:1 - ext4 /dev/root rw\n" + + "41 36 0:41 / /srv/anvil\\040data rw - nfs4 nas:/export/anvil rw\n" + + "42 36 0:42 / /srv/anvilx rw - ext4 /dev/sdb rw\n" + var got []mountEntry + for _, line := range strings.Split(strings.TrimSuffix(mountinfo, "\n"), "\n") { + e, ok := parseMountinfoLine(line) + if !ok { + t.Fatalf("failed to parse mountinfo line %q", line) + } + got = append(got, e) + } + if got[1].point != "/srv/anvil data" || got[1].fsType != "nfs4" { + t.Fatalf("octal-escaped mount point mis-parsed: %+v", got[1]) + } + + // /proc/mounts has a different field order; both feed the same lookup. + e, ok := parseProcMountsLine(`nas:/export /srv/anvil\040data nfs4 rw,relatime 0 0`) + if !ok || e.point != "/srv/anvil data" || e.fsType != "nfs4" { + t.Fatalf("/proc/mounts line mis-parsed: %+v ok=%v", e, ok) + } + + // A mountinfo line with no "-" separator is malformed, not a mount. + if _, ok := parseMountinfoLine("36 35 98:0 / /srv rw ext4 /dev/root rw"); ok { + t.Fatal("a mountinfo line without the optional-fields separator must not parse") + } +} + +// TestLongestMountPointWins is the whole reason the mount table is sorted: +// a share mounted under the root filesystem must not be reported as the root +// filesystem's type. +func TestLongestMountPointWins(t *testing.T) { + if pathHasPrefix("/srv/anvil", "/srv/anvilx/data") { + t.Fatal("/srv/anvil must not match /srv/anvilx/data — whole path elements only") + } + if !pathHasPrefix("/srv/anvil", "/srv/anvil") { + t.Fatal("a mount point is a prefix of itself") + } + if !pathHasPrefix("/", "/srv/anvil") { + t.Fatal("the root filesystem contains everything") + } +} + +// --------------------------------------------------------------------------- +// Guard 2 — CheckFTS5 +// --------------------------------------------------------------------------- + +// TestCheckFTS5PassesAgainstTheRealDriver is the positive control. It is also +// the check the R.5 packet actually cares about at run time: if a future +// modernc.org/sqlite bump drops FTS5, this test goes red in CI on the same +// commit that bumps the dependency. +func TestCheckFTS5PassesAgainstTheRealDriver(t *testing.T) { + db := openMemory(t) + if err := CheckFTS5(db); err != nil { + t.Fatalf("FTS5 guard failed against modernc.org/sqlite: %v", err) + } + // Repeat runs must not collide on the probe table name, and must leave no + // trace behind on the pooled connection. + if err := CheckFTS5(db); err != nil { + t.Fatalf("second FTS5 probe failed: %v", err) + } + var leftovers int + err := db.QueryRow(`SELECT count(*) FROM temp.sqlite_schema WHERE name LIKE 'anvil_fts5_probe%'`).Scan(&leftovers) + if err != nil { + t.Fatalf("counting temp objects: %v", err) + } + if leftovers != 0 { + t.Fatalf("FTS5 probe left %d temp objects behind", leftovers) + } +} + +func TestCheckFTS5RejectsNilHandle(t *testing.T) { + if err := CheckFTS5(nil); !errors.Is(err, ErrNoFTS5) { + t.Fatalf("nil handle should be refused with ErrNoFTS5, got %v", err) + } +} + +// TestCheckFTS5FailsLoudlyWithoutFTS5 stands in for the packet's "build tag +// that disables FTS5". modernc.org/sqlite exposes no such tag — its FTS5 is +// compiled in unconditionally — so the absent capability is injected at the +// database/sql driver layer instead, which exercises the identical code path +// in CheckFTS5 including the error it produces. +func TestCheckFTS5FailsLoudlyWithoutFTS5(t *testing.T) { + db, err := sql.Open(driverNoFTS5, "") + if err != nil { + t.Fatalf("sql.Open: %v", err) + } + t.Cleanup(func() { _ = db.Close() }) + + err = CheckFTS5(db) + if err == nil { + t.Fatal("CheckFTS5 passed against a driver with no fts5 module") + } + if !errors.Is(err, ErrNoFTS5) { + t.Fatalf("error does not wrap ErrNoFTS5: %v", err) + } + for _, want := range []string{"CREATE VIRTUAL TABLE", "no such module: fts5", "advisory_fts"} { + if !strings.Contains(err.Error(), want) { + t.Errorf("error message omits %q: %v", want, err) + } + } +} + +// TestCheckFTS5FailsWhenMatchSilentlyIndexesNothing is the "not silently" half +// of the packet's requirement, and the reason this guard runs a real MATCH +// rather than stopping at a successful CREATE. A build that accepts the DDL +// and indexes nothing would pass a syntax-only probe and then return zero +// advisories forever. +func TestCheckFTS5FailsWhenMatchSilentlyIndexesNothing(t *testing.T) { + db, err := sql.Open(driverSilentFTS5, "") + if err != nil { + t.Fatalf("sql.Open: %v", err) + } + t.Cleanup(func() { _ = db.Close() }) + + err = CheckFTS5(db) + if err == nil { + t.Fatal("CheckFTS5 passed against a driver whose FTS5 indexes nothing") + } + if !errors.Is(err, ErrNoFTS5) { + t.Fatalf("error does not wrap ErrNoFTS5: %v", err) + } + if !strings.Contains(err.Error(), "returned 0 rows") { + t.Errorf("error message does not say what went wrong: %v", err) + } +} + +func TestCheckStartupRunsBothGuards(t *testing.T) { + // A refused mount short-circuits before the database is touched. + if err := CheckStartup("", nil); !errors.Is(err, ErrNetworkMount) { + t.Fatalf("CheckStartup should fail on the mount guard first, got %v", err) + } + // A good directory still has to clear the FTS5 guard. + if err := CheckStartup(t.TempDir(), nil); !errors.Is(err, ErrNoFTS5) { + t.Fatalf("CheckStartup should reach the FTS5 guard, got %v", err) + } + if err := CheckStartup(t.TempDir(), openMemory(t)); err != nil { + t.Fatalf("CheckStartup failed on a healthy local store: %v", err) + } +} + +// --------------------------------------------------------------------------- +// Test fixtures: two database/sql drivers with broken FTS5 +// --------------------------------------------------------------------------- + +const ( + driverNoFTS5 = "anvil-test-no-fts5" + driverSilentFTS5 = "anvil-test-silent-fts5" +) + +func init() { + sql.Register(driverNoFTS5, fakeDriver{silent: false}) + sql.Register(driverSilentFTS5, fakeDriver{silent: true}) +} + +// fakeDriver is the smallest database/sql driver that can answer CheckFTS5's +// three statements. silent=false rejects the CREATE the way a SQLite build +// without the fts5 module does; silent=true accepts everything and then +// matches nothing. +type fakeDriver struct{ silent bool } + +func (d fakeDriver) Open(string) (driver.Conn, error) { return fakeConn{silent: d.silent}, nil } + +type fakeConn struct{ silent bool } + +func (c fakeConn) Prepare(string) (driver.Stmt, error) { + return nil, errors.New("fake driver: Prepare is not implemented; use the context methods") +} +func (c fakeConn) Close() error { return nil } +func (c fakeConn) Begin() (driver.Tx, error) { return nil, errors.New("fake driver: no transactions") } + +func (c fakeConn) ExecContext(_ context.Context, query string, _ []driver.NamedValue) (driver.Result, error) { + if !c.silent && strings.Contains(query, "USING fts5") { + return nil, errors.New("no such module: fts5") + } + return driver.RowsAffected(0), nil +} + +func (c fakeConn) QueryContext(_ context.Context, query string, _ []driver.NamedValue) (driver.Rows, error) { + if strings.Contains(query, "MATCH") { + return &fakeRows{cols: []string{"count(*)"}, vals: [][]driver.Value{{int64(0)}}}, nil + } + return &fakeRows{cols: []string{"x"}}, nil +} + +type fakeRows struct { + cols []string + vals [][]driver.Value + next int +} + +func (r *fakeRows) Columns() []string { return r.cols } +func (r *fakeRows) Close() error { return nil } +func (r *fakeRows) Next(dest []driver.Value) error { + if r.next >= len(r.vals) { + return io.EOF + } + copy(dest, r.vals[r.next]) + r.next++ + return nil +} diff --git a/internal/store/migrate.go b/internal/store/migrate.go new file mode 100644 index 0000000..6c25db4 --- /dev/null +++ b/internal/store/migrate.go @@ -0,0 +1,477 @@ +// Forward-only numbered migrations with a checksummed ledger (step R.5). +// +// research/07-database-design.md §7: "numbered forward-only SQL + +// `PRAGMA user_version` + a checksummed ledger". No Alembic, no goose, no +// sqlx-cli, nothing for a self-hoster to install — plan/00-SPINE.md S12 makes +// this Go-only, and the migrations are embedded in the binary. +// +// Three properties are load-bearing and each has a test: +// +// 1. Each migration runs inside BEGIN ... COMMIT and bumps `user_version` in +// the SAME transaction, so a failure leaves the schema untouched rather +// than half-applied. +// 2. Every applied migration's checksum is re-verified at startup against the +// embedded text. A mismatch is a refusal, not a re-run and not a skip: it +// means the database was built by a different definition of the schema +// than the one this binary carries, and either guess about what to do next +// corrupts the store of record. +// 3. Forward-only. There are no down migrations. The rollback story is +// `VACUUM INTO 'anvil-pre-v{N}.db'` taken before the first migration +// touches an existing database. + +package store + +import ( + "context" + "crypto/sha256" + "database/sql" + "embed" + "encoding/hex" + "errors" + "fmt" + "io/fs" + "os" + "path" + "path/filepath" + "regexp" + "sort" + "strconv" + "strings" + "sync" + "time" +) + +//go:embed migrations/*.sql +var migrationFS embed.FS + +const migrationsDir = "migrations" + +// includeDirectivePrefix introduces the one preprocessor directive a migration +// file may contain. See migrations/0001_init.sql for why it exists: schema.sql +// is a frozen interface with exactly one copy in this repository, and a +// migration references it rather than duplicating it. +const includeDirectivePrefix = "-- @anvil:include " + +// ErrMigrationLedger reports that the database's migration history and this +// binary's embedded migrations disagree. It is always a refusal to proceed. +var ErrMigrationLedger = errors.New("store: migration ledger mismatch") + +// ErrSnapshotRequired reports that migrating an already-populated database was +// attempted without somewhere to put the pre-migration snapshot. +var ErrSnapshotRequired = errors.New("store: pre-migration snapshot directory required") + +var migrationFilenameRE = regexp.MustCompile(`^(\d{4})_([a-z0-9]+(?:_[a-z0-9]+)*)\.sql$`) + +// Migration is one numbered, embedded, forward-only migration. +type Migration struct { + // Version is the number in the filename; versions are contiguous from 1. + Version int + // Name is the descriptive part of the filename, e.g. "init". + Name string + // Filename is the name as embedded, e.g. "0001_init.sql". + Filename string + // SQL is the fully expanded text that will be executed: the file's bytes + // with every include directive replaced by the included content. + SQL string + // Checksum is the lowercase hex SHA-256 of SQL. This is the value written + // to and re-verified against schema_migration.checksum. + Checksum string +} + +var loadMigrations = sync.OnceValues(func() ([]Migration, error) { + entries, err := fs.ReadDir(migrationFS, migrationsDir) + if err != nil { + return nil, fmt.Errorf("store: reading embedded migrations: %w", err) + } + + migrations := make([]Migration, 0, len(entries)) + for _, e := range entries { + if e.IsDir() { + return nil, fmt.Errorf("store: %s/%s is a directory; migrations are flat", migrationsDir, e.Name()) + } + m := migrationFilenameRE.FindStringSubmatch(e.Name()) + if m == nil { + return nil, fmt.Errorf("store: migration filename %q does not match NNNN_lower_snake_name.sql", e.Name()) + } + version, err := strconv.Atoi(m[1]) + if err != nil { // unreachable: the regexp already proved four digits + return nil, fmt.Errorf("store: migration %q has an unparseable version: %w", e.Name(), err) + } + raw, err := migrationFS.ReadFile(path.Join(migrationsDir, e.Name())) + if err != nil { + return nil, fmt.Errorf("store: reading migration %q: %w", e.Name(), err) + } + expanded, err := expandIncludes(e.Name(), string(raw)) + if err != nil { + return nil, err + } + sum := sha256.Sum256([]byte(expanded)) + migrations = append(migrations, Migration{ + Version: version, + Name: m[2], + Filename: e.Name(), + SQL: expanded, + Checksum: hex.EncodeToString(sum[:]), + }) + } + + sort.Slice(migrations, func(i, j int) bool { return migrations[i].Version < migrations[j].Version }) + for i, mig := range migrations { + if want := i + 1; mig.Version != want { + return nil, fmt.Errorf("store: migration versions must be contiguous from 0001; "+ + "expected %04d at position %d but found %q", want, i+1, mig.Filename) + } + } + if len(migrations) == 0 { + return nil, errors.New("store: no migrations are embedded; the store cannot be created") + } + return migrations, nil +}) + +// Migrations returns every embedded migration in ascending version order. +// +// It fails rather than returning a partial list if the filenames are malformed +// or the numbering has a gap or duplicate, because "migrate what parses" is +// how a store ends up at a version nobody can describe. +func Migrations() ([]Migration, error) { + migrations, err := loadMigrations() + if err != nil { + return nil, err + } + return append([]Migration(nil), migrations...), nil +} + +// expandIncludes replaces each `-- @anvil:include ` line with the +// content of the named file. Today the only legal target is schema.sql, whose +// bytes come from the string ddl.go embeds — so there is exactly one copy of +// the schema in the repository and the migration cannot drift from it. +func expandIncludes(filename, src string) (string, error) { + if !strings.Contains(src, includeDirectivePrefix) { + return src, nil + } + + var out strings.Builder + out.Grow(len(src) + len(schemaSQL)) + rest := src + lineNo := 0 + for len(rest) > 0 { + line := rest + if i := strings.IndexByte(rest, '\n'); i >= 0 { + line, rest = rest[:i+1], rest[i+1:] + } else { + rest = "" + } + lineNo++ + + trimmed := strings.TrimSpace(line) + if !strings.HasPrefix(trimmed, includeDirectivePrefix) { + out.WriteString(line) + continue + } + target := strings.TrimSpace(strings.TrimPrefix(trimmed, includeDirectivePrefix)) + if target != "schema.sql" { + return "", fmt.Errorf("store: %s line %d includes %q; the only includable file is schema.sql", + filename, lineNo, target) + } + out.WriteString(Schema()) + if !strings.HasSuffix(Schema(), "\n") { + out.WriteString("\n") + } + } + return out.String(), nil +} + +// appliedMigration is one row of the schema_migration ledger. +type appliedMigration struct { + Version int + Name string + Checksum string + AppliedAt string +} + +// Migrate brings db up to the latest embedded schema version and returns the +// versions it applied, in order. An already-current database returns an empty +// slice and touches nothing. +// +// snapshotDir is where the `VACUUM INTO 'anvil-pre-v{N}.db'` pre-migration +// snapshot is written. It may be empty ONLY when the database has no applied +// migrations — a fresh database has nothing to lose, so demanding a snapshot +// of it would just teach callers to pass a junk directory. Upgrading a +// populated database without one is refused: research/07-database-design.md §7 +// makes the snapshot the entire replacement for down migrations, so skipping +// it means the upgrade has no rollback path at all. +// +// CheckFTS5 runs first, on the same handle. plan/40-record-and-storage.md +// requires the guards before any other store operation, and applying +// schema.sql is a store operation — its advisory_fts table would otherwise +// fail deep inside the transaction with a driver-level error instead of the +// guard's explanation. +func Migrate(ctx context.Context, db *sql.DB, snapshotDir string) ([]int, error) { + migrations, err := Migrations() + if err != nil { + return nil, err + } + return migrateWith(ctx, db, migrations, snapshotDir) +} + +// migrateWith is Migrate over an explicit migration list. It exists so +// migrate_test.go can exercise ordering, mid-sequence failure and the snapshot +// rule against synthetic later versions: those paths cannot otherwise be +// tested while 0001 is the only embedded migration, and leaving them untested +// until the first real 0002 means discovering them during an upgrade. +func migrateWith(ctx context.Context, db *sql.DB, migrations []Migration, snapshotDir string) ([]int, error) { + if db == nil { + return nil, errors.New("store: Migrate needs a database handle") + } + if err := CheckFTS5Context(ctx, db); err != nil { + return nil, err + } + + applied, err := verifyLedger(ctx, db, migrations) + if err != nil { + return nil, err + } + + pending := migrations[len(applied):] + if len(pending) == 0 { + return nil, nil + } + + if len(applied) > 0 { + if strings.TrimSpace(snapshotDir) == "" { + return nil, fmt.Errorf("%w: this database is at schema version %d and %d migration(s) are "+ + "pending. Migrations are forward-only and there are no down migrations "+ + "(research/07-database-design.md §7), so a VACUUM INTO snapshot is the only rollback "+ + "path; pass a directory to write it to", ErrSnapshotRequired, len(applied), len(pending)) + } + dest, err := snapshotPath(snapshotDir, len(applied)) + if err != nil { + return nil, err + } + if err := Snapshot(ctx, db, dest); err != nil { + return nil, err + } + } + + appliedNow := make([]int, 0, len(pending)) + for _, m := range pending { + if err := applyOne(ctx, db, m); err != nil { + return appliedNow, err + } + appliedNow = append(appliedNow, m.Version) + } + return appliedNow, nil +} + +// verifyLedger reads schema_migration and PRAGMA user_version and proves they +// agree with each other and with the embedded migrations. It returns the +// applied migrations in version order. +// +// Every disagreement here is fatal. The three it names are the ones +// research/07 §7 says actually happen to self-hosted tools: a hand-edited +// migration, a downgraded binary, and a database that was interrupted or +// touched by something that is not this code. +func verifyLedger(ctx context.Context, db *sql.DB, migrations []Migration) ([]appliedMigration, error) { + userVersion, err := SchemaVersion(ctx, db) + if err != nil { + return nil, err + } + + hasLedger, err := tableExists(ctx, db, "schema_migration") + if err != nil { + return nil, err + } + if !hasLedger { + if userVersion != 0 { + return nil, fmt.Errorf("%w: PRAGMA user_version is %d but there is no schema_migration table. "+ + "This database was not created by Anvil, or its ledger was dropped; refusing to guess which "+ + "migrations it has", ErrMigrationLedger, userVersion) + } + return nil, nil + } + + applied, err := readLedger(ctx, db) + if err != nil { + return nil, err + } + if len(applied) == 0 { + if userVersion != 0 { + return nil, fmt.Errorf("%w: PRAGMA user_version is %d but schema_migration is empty", + ErrMigrationLedger, userVersion) + } + return nil, nil + } + + byVersion := make(map[int]Migration, len(migrations)) + for _, m := range migrations { + byVersion[m.Version] = m + } + + for i, row := range applied { + if want := i + 1; row.Version != want { + return nil, fmt.Errorf("%w: schema_migration jumps from version %d to %d. The history has a "+ + "gap, so the schema in this file cannot be reconstructed from the migrations in this binary", + ErrMigrationLedger, want-1, row.Version) + } + embedded, ok := byVersion[row.Version] + if !ok { + return nil, fmt.Errorf("%w: the database has applied migration %04d (%q) which this binary does "+ + "not contain. This is an older Anvil opening a newer store; migrations are forward-only, so "+ + "downgrading is not supported. Run the newer binary, or restore an anvil-pre-v*.db snapshot", + ErrMigrationLedger, row.Version, row.Name) + } + if row.Name != embedded.Name { + return nil, fmt.Errorf("%w: migration %04d was applied as %q but this binary calls it %q", + ErrMigrationLedger, row.Version, row.Name, embedded.Name) + } + if row.Checksum != embedded.Checksum { + return nil, fmt.Errorf("%w: migration %s no longer matches what was applied to this database "+ + "(ledger recorded %s, this binary carries %s). Either the migration file or schema.sql was "+ + "edited after the fact, or this binary's schema differs from the one that built this store. "+ + "Refusing to start: re-running it would apply DDL twice and skipping it would run against a "+ + "schema that was never applied", + ErrMigrationLedger, embedded.Filename, row.Checksum, embedded.Checksum) + } + } + + if maxApplied := applied[len(applied)-1].Version; userVersion != maxApplied { + return nil, fmt.Errorf("%w: PRAGMA user_version is %d but the highest applied migration is %d. "+ + "They are written in one transaction, so this means the file was modified outside Anvil", + ErrMigrationLedger, userVersion, maxApplied) + } + return applied, nil +} + +// applyOne runs a single migration inside one transaction, together with the +// user_version bump and the ledger row. +// +// SQLite DDL is transactional, so a failure anywhere in here leaves the schema +// exactly as it was — which is the reason this is one transaction and not +// three statements. +func applyOne(ctx context.Context, db *sql.DB, m Migration) error { + tx, err := db.BeginTx(ctx, nil) + if err != nil { + return fmt.Errorf("store: migration %s: BEGIN failed: %w", m.Filename, err) + } + committed := false + defer func() { + if !committed { + _ = tx.Rollback() + } + }() + + if _, err := tx.ExecContext(ctx, m.SQL); err != nil { + return fmt.Errorf("store: migration %s failed and was rolled back; the schema is unchanged: %w", m.Filename, err) + } + + // PRAGMA takes no bound parameters, so the version is formatted in. It is + // an int parsed from a \d{4} filename match, not caller input. + if _, err := tx.ExecContext(ctx, "PRAGMA user_version = "+strconv.Itoa(m.Version)); err != nil { + return fmt.Errorf("store: migration %s: bumping user_version failed and was rolled back: %w", m.Filename, err) + } + + if _, err := tx.ExecContext(ctx, + `INSERT INTO schema_migration (version, name, checksum, applied_at) VALUES (?, ?, ?, ?)`, + m.Version, m.Name, m.Checksum, time.Now().UTC().Format(time.RFC3339Nano), + ); err != nil { + return fmt.Errorf("store: migration %s: recording the ledger row failed and was rolled back: %w", m.Filename, err) + } + + if err := tx.Commit(); err != nil { + return fmt.Errorf("store: migration %s: COMMIT failed; the schema is unchanged: %w", m.Filename, err) + } + committed = true + return nil +} + +// Snapshot writes a consistent copy of the whole database to dest using +// `VACUUM INTO`, which research/07-database-design.md §7 makes the substitute +// for down migrations. +// +// dest must not already exist: SQLite refuses to overwrite, and so does this — +// silently replacing the only pre-upgrade copy of a store would be the worst +// possible way to be helpful. +func Snapshot(ctx context.Context, db *sql.DB, dest string) error { + if db == nil { + return errors.New("store: Snapshot needs a database handle") + } + if strings.TrimSpace(dest) == "" { + return errors.New("store: Snapshot needs a destination path") + } + // VACUUM cannot run inside a transaction, so this goes to db directly. The + // path is a SQL string literal because SQLite does not bind parameters in + // VACUUM INTO; doubling embedded quotes is the whole escaping rule for a + // SQLite string literal. + quoted := "'" + strings.ReplaceAll(dest, "'", "''") + "'" + if _, err := db.ExecContext(ctx, "VACUUM INTO "+quoted); err != nil { + return fmt.Errorf("store: pre-migration snapshot to %q failed; refusing to migrate without one: %w", dest, err) + } + return nil +} + +// snapshotPath builds research/07's `anvil-pre-v{N}.db` name inside dir, +// creating dir if needed. If that name is taken — a second upgrade attempt +// after a failure — a timestamp is appended rather than clobbering the +// existing snapshot. +func snapshotPath(dir string, currentVersion int) (string, error) { + if err := os.MkdirAll(dir, 0o750); err != nil { + return "", fmt.Errorf("store: creating snapshot directory %q: %w", dir, err) + } + base := filepath.Join(dir, fmt.Sprintf("anvil-pre-v%d.db", currentVersion)) + if _, err := os.Lstat(base); errors.Is(err, fs.ErrNotExist) { + return base, nil + } + return filepath.Join(dir, fmt.Sprintf("anvil-pre-v%d-%d.db", currentVersion, time.Now().UTC().UnixNano())), nil +} + +// SchemaVersion returns the database's PRAGMA user_version, which is 0 for a +// database no migration has touched. +func SchemaVersion(ctx context.Context, db *sql.DB) (int, error) { + var v int + if err := db.QueryRowContext(ctx, "PRAGMA user_version").Scan(&v); err != nil { + return 0, fmt.Errorf("store: reading PRAGMA user_version: %w", err) + } + return v, nil +} + +// LatestVersion returns the highest embedded migration version — the schema +// version a fully migrated database will report. +func LatestVersion() (int, error) { + migrations, err := loadMigrations() + if err != nil { + return 0, err + } + return migrations[len(migrations)-1].Version, nil +} + +func tableExists(ctx context.Context, db *sql.DB, name string) (bool, error) { + var n int + err := db.QueryRowContext(ctx, + `SELECT count(*) FROM sqlite_schema WHERE type = 'table' AND name = ?`, name).Scan(&n) + if err != nil { + return false, fmt.Errorf("store: looking for table %q: %w", name, err) + } + return n > 0, nil +} + +func readLedger(ctx context.Context, db *sql.DB) ([]appliedMigration, error) { + rows, err := db.QueryContext(ctx, + `SELECT version, name, checksum, applied_at FROM schema_migration ORDER BY version`) + if err != nil { + return nil, fmt.Errorf("store: reading schema_migration: %w", err) + } + defer func() { _ = rows.Close() }() + + var applied []appliedMigration + for rows.Next() { + var a appliedMigration + if err := rows.Scan(&a.Version, &a.Name, &a.Checksum, &a.AppliedAt); err != nil { + return nil, fmt.Errorf("store: scanning schema_migration: %w", err) + } + applied = append(applied, a) + } + if err := rows.Err(); err != nil { + return nil, fmt.Errorf("store: iterating schema_migration: %w", err) + } + return applied, nil +} diff --git a/internal/store/migrate_test.go b/internal/store/migrate_test.go new file mode 100644 index 0000000..a6f4daf --- /dev/null +++ b/internal/store/migrate_test.go @@ -0,0 +1,556 @@ +package store + +import ( + "context" + "crypto/sha256" + "database/sql" + "encoding/hex" + "errors" + "fmt" + "os" + "path/filepath" + "strings" + "testing" + + _ "modernc.org/sqlite" // cgo-free driver, plan/00-SPINE.md S12 +) + +// openMemory returns an empty in-memory database with ConnectionPragmas +// applied. MaxOpenConns is 1 because every new connection to ":memory:" is a +// different, empty database — and because the pragmas are per connection. +func openMemory(t *testing.T) *sql.DB { + t.Helper() + db, err := sql.Open("sqlite", ":memory:") + if err != nil { + t.Fatalf("sql.Open: %v", err) + } + t.Cleanup(func() { _ = db.Close() }) + db.SetMaxOpenConns(1) + applyPragmas(t, db) + return db +} + +// openOnDisk returns an empty file-backed database and its directory. The +// snapshot path has to be exercised against a real file: VACUUM INTO on a +// database that never touched a filesystem would prove nothing about the +// rollback story it replaces. +func openOnDisk(t *testing.T) (*sql.DB, string) { + t.Helper() + dir := t.TempDir() + db, err := sql.Open("sqlite", filepath.Join(dir, "anvil.db")) + if err != nil { + t.Fatalf("sql.Open: %v", err) + } + t.Cleanup(func() { _ = db.Close() }) + db.SetMaxOpenConns(1) + applyPragmas(t, db) + return db, dir +} + +func applyPragmas(t *testing.T, db *sql.DB) { + t.Helper() + for _, p := range ConnectionPragmas() { + if _, err := db.Exec(p); err != nil { + t.Fatalf("pragma %q: %v", p, err) + } + } +} + +// schemaObjects renders every object SQLite actually created, which is the +// tool-free equivalent of the packet's `sqlite3 .schema` diff — and a stricter +// one, since it compares the stored DDL text rather than a pretty-printed +// rendering of it. +func schemaObjects(t *testing.T, db *sql.DB) string { + t.Helper() + rows, err := db.Query(`SELECT type, name, tbl_name, COALESCE(sql, '') FROM sqlite_schema ORDER BY type, name`) + if err != nil { + t.Fatalf("reading sqlite_schema: %v", err) + } + defer func() { _ = rows.Close() }() + + var b strings.Builder + for rows.Next() { + var typ, name, tbl, ddl string + if err := rows.Scan(&typ, &name, &tbl, &ddl); err != nil { + t.Fatalf("scanning sqlite_schema: %v", err) + } + fmt.Fprintf(&b, "%s\t%s\t%s\n%s\n--\n", typ, name, tbl, ddl) + } + if err := rows.Err(); err != nil { + t.Fatalf("iterating sqlite_schema: %v", err) + } + return b.String() +} + +func ledgerRows(t *testing.T, db *sql.DB) []appliedMigration { + t.Helper() + applied, err := readLedger(context.Background(), db) + if err != nil { + t.Fatalf("readLedger: %v", err) + } + return applied +} + +func mustSchemaVersion(t *testing.T, db *sql.DB) int { + t.Helper() + v, err := SchemaVersion(context.Background(), db) + if err != nil { + t.Fatalf("SchemaVersion: %v", err) + } + return v +} + +// --------------------------------------------------------------------------- +// The embedded migration set +// --------------------------------------------------------------------------- + +func TestEmbeddedMigrationsAreWellFormed(t *testing.T) { + migrations, err := Migrations() + if err != nil { + t.Fatalf("Migrations: %v", err) + } + if len(migrations) == 0 { + t.Fatal("no migrations are embedded") + } + for i, m := range migrations { + if m.Version != i+1 { + t.Fatalf("migration %d is version %d; versions must be contiguous from 1", i, m.Version) + } + if len(m.Checksum) != 64 { + t.Errorf("%s: checksum %q is not a 64-character SHA-256 hex digest", m.Filename, m.Checksum) + } + if strings.Contains(m.SQL, includeDirectivePrefix) { + t.Errorf("%s: an include directive survived expansion", m.Filename) + } + sum := sha256.Sum256([]byte(m.SQL)) + if want := hex.EncodeToString(sum[:]); m.Checksum != want { + t.Errorf("%s: checksum %s does not digest the expanded SQL (%s)", m.Filename, m.Checksum, want) + } + } + + latest, err := LatestVersion() + if err != nil { + t.Fatalf("LatestVersion: %v", err) + } + if latest != migrations[len(migrations)-1].Version { + t.Fatalf("LatestVersion = %d, want %d", latest, migrations[len(migrations)-1].Version) + } +} + +// TestInitMigrationEmbedsSchemaByteForByte is the structural half of the +// packet's byte-for-byte requirement: 0001 does not carry a copy of the DDL +// that could drift from schema.sql, it carries schema.sql's exact bytes. +func TestInitMigrationEmbedsSchemaByteForByte(t *testing.T) { + migrations, err := Migrations() + if err != nil { + t.Fatalf("Migrations: %v", err) + } + init := migrations[0] + if init.Name != "init" || init.Filename != "0001_init.sql" { + t.Fatalf("migration 1 is %q (%s), want init (0001_init.sql)", init.Name, init.Filename) + } + if !strings.Contains(init.SQL, Schema()) { + t.Fatal("0001_init.sql does not contain schema.sql verbatim after expansion") + } + // Anything outside the included schema must be comment, so the migration + // cannot quietly add DDL that schema.sql does not describe. + extra := strings.ReplaceAll(init.SQL, Schema(), "") + for i, line := range strings.Split(extra, "\n") { + if trimmed := strings.TrimSpace(line); trimmed != "" && !strings.HasPrefix(trimmed, "--") { + t.Fatalf("0001_init.sql line %d adds DDL outside schema.sql: %q", i+1, trimmed) + } + } +} + +func TestIncludeExpansion(t *testing.T) { + got, err := expandIncludes("test.sql", "-- header\n"+includeDirectivePrefix+"schema.sql\n-- footer\n") + if err != nil { + t.Fatalf("expandIncludes: %v", err) + } + if want := "-- header\n" + Schema() + "-- footer\n"; got != want { + t.Fatalf("expansion did not splice schema.sql in place:\n%q", got[:min(len(got), 200)]) + } + + if _, err := expandIncludes("test.sql", includeDirectivePrefix+"secrets.sql\n"); err == nil { + t.Fatal("expandIncludes accepted an include target other than schema.sql") + } +} + +// --------------------------------------------------------------------------- +// Applying migrations +// --------------------------------------------------------------------------- + +// TestMigrateBuildsExactlyR4Schema is the packet's schema-equivalence evidence +// item: a database built by the migration runner and a database built by +// applying schema.sql directly must be indistinguishable, object for object. +func TestMigrateBuildsExactlyR4Schema(t *testing.T) { + migrated := openMemory(t) + applied, err := Migrate(context.Background(), migrated, "") + if err != nil { + t.Fatalf("Migrate: %v", err) + } + if len(applied) != 1 || applied[0] != 1 { + t.Fatalf("Migrate applied %v, want [1]", applied) + } + + direct := openMemory(t) + if _, err := direct.Exec(Schema()); err != nil { + t.Fatalf("applying schema.sql directly: %v", err) + } + + if got, want := schemaObjects(t, migrated), schemaObjects(t, direct); got != want { + t.Fatalf("migrated schema differs from schema.sql applied directly:\n--- migrated ---\n%s\n--- direct ---\n%s", got, want) + } + + if v := mustSchemaVersion(t, migrated); v != 1 { + t.Fatalf("PRAGMA user_version = %d after migrating, want 1", v) + } + + rows := ledgerRows(t, migrated) + if len(rows) != 1 { + t.Fatalf("schema_migration has %d rows, want 1", len(rows)) + } + migrations, err := Migrations() + if err != nil { + t.Fatalf("Migrations: %v", err) + } + if rows[0].Version != 1 || rows[0].Name != "init" || rows[0].Checksum != migrations[0].Checksum { + t.Fatalf("ledger row %+v does not describe %s", rows[0], migrations[0].Filename) + } + if rows[0].AppliedAt == "" { + t.Fatal("ledger row has no applied_at timestamp") + } +} + +func TestMigrateIsANoOpWhenCurrent(t *testing.T) { + db := openMemory(t) + ctx := context.Background() + if _, err := Migrate(ctx, db, ""); err != nil { + t.Fatalf("first Migrate: %v", err) + } + before := ledgerRows(t, db) + schemaBefore := schemaObjects(t, db) + + applied, err := Migrate(ctx, db, "") + if err != nil { + t.Fatalf("second Migrate: %v", err) + } + if len(applied) != 0 { + t.Fatalf("re-running Migrate applied %v, want nothing", applied) + } + after := ledgerRows(t, db) + if len(after) != len(before) || after[0].AppliedAt != before[0].AppliedAt { + t.Fatalf("re-running Migrate rewrote the ledger: %+v -> %+v", before, after) + } + if schemaObjects(t, db) != schemaBefore { + t.Fatal("re-running Migrate changed the schema") + } +} + +// TestMigrateAppliesInNumberedOrderInsideTransactions injects later versions, +// which is the only way to test ordering and transactionality while 0001 is +// the only real migration. The alternative is finding out during the first +// upgrade a user ever performs. +func TestMigrateAppliesInNumberedOrderInsideTransactions(t *testing.T) { + db, dir := openOnDisk(t) + ctx := context.Background() + + migrations := withSynthetic(t, + synthetic(2, "second", "CREATE TABLE second_step (x INTEGER);"), + synthetic(3, "third", "CREATE TABLE third_step (y INTEGER REFERENCES second_step(x));"), + ) + + applied, err := migrateWith(ctx, db, migrations, dir) + if err != nil { + t.Fatalf("migrateWith: %v", err) + } + if len(applied) != 3 || applied[0] != 1 || applied[1] != 2 || applied[2] != 3 { + t.Fatalf("applied %v, want [1 2 3] in order", applied) + } + if v := mustSchemaVersion(t, db); v != 3 { + t.Fatalf("user_version = %d, want 3", v) + } + if rows := ledgerRows(t, db); len(rows) != 3 { + t.Fatalf("ledger has %d rows, want 3", len(rows)) + } + // third_step references second_step; it could not have been created first. + var n int + if err := db.QueryRow(`SELECT count(*) FROM sqlite_schema WHERE name IN ('second_step','third_step')`).Scan(&n); err != nil { + t.Fatalf("counting new tables: %v", err) + } + if n != 2 { + t.Fatalf("expected both synthetic tables, found %d", n) + } +} + +// TestFailedMigrationLeavesSchemaUntouched is why each migration runs inside +// BEGIN ... COMMIT with the user_version bump in the same transaction. +func TestFailedMigrationLeavesSchemaUntouched(t *testing.T) { + db := openMemory(t) + ctx := context.Background() + + migrations := withSynthetic(t, synthetic(2, "broken", + "CREATE TABLE half_applied (x INTEGER);\nCREATE TABLE half_applied (x INTEGER);")) + + applied, err := migrateWith(ctx, db, migrations, t.TempDir()) + if err == nil { + t.Fatal("a migration with invalid DDL was accepted") + } + if len(applied) != 1 || applied[0] != 1 { + t.Fatalf("applied %v, want [1] — 0001 succeeded and 0002 must not count", applied) + } + if v := mustSchemaVersion(t, db); v != 1 { + t.Fatalf("user_version = %d after a failed migration, want 1", v) + } + if rows := ledgerRows(t, db); len(rows) != 1 { + t.Fatalf("ledger has %d rows after a failed migration, want 1", len(rows)) + } + var n int + if err := db.QueryRow(`SELECT count(*) FROM sqlite_schema WHERE name = 'half_applied'`).Scan(&n); err != nil { + t.Fatalf("looking for the rolled-back table: %v", err) + } + if n != 0 { + t.Fatal("half_applied survived a rolled-back migration") + } +} + +// --------------------------------------------------------------------------- +// The ledger refuses to guess +// --------------------------------------------------------------------------- + +// TestMigrateRefusesEditedMigration is the packet's stop condition: "a +// hand-edited migration file with a mismatched checksum causes migrate.go to +// refuse to start". The ledger row is edited rather than the embedded file +// because the embedded file cannot be edited at run time; the comparison +// migrate.go performs is the same one either way. +func TestMigrateRefusesEditedMigration(t *testing.T) { + db := openMemory(t) + ctx := context.Background() + if _, err := Migrate(ctx, db, ""); err != nil { + t.Fatalf("Migrate: %v", err) + } + + const tampered = "ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff" + if _, err := db.Exec(`UPDATE schema_migration SET checksum = ? WHERE version = 1`, tampered); err != nil { + t.Fatalf("tampering with the ledger: %v", err) + } + + _, err := Migrate(ctx, db, "") + if err == nil { + t.Fatal("Migrate started against a database whose migration checksum does not match") + } + if !errors.Is(err, ErrMigrationLedger) { + t.Fatalf("error does not wrap ErrMigrationLedger: %v", err) + } + for _, want := range []string{"0001_init.sql", tampered, "Refusing to start"} { + if !strings.Contains(err.Error(), want) { + t.Errorf("error message omits %q: %v", want, err) + } + } +} + +func TestMigrateRefusesADowngrade(t *testing.T) { + db := openMemory(t) + ctx := context.Background() + if _, err := Migrate(ctx, db, ""); err != nil { + t.Fatalf("Migrate: %v", err) + } + // Simulate an older binary opening a store a newer one migrated. + if _, err := db.Exec( + `INSERT INTO schema_migration (version, name, checksum, applied_at) VALUES (2, 'future', 'x', '2026-01-01T00:00:00Z')`, + ); err != nil { + t.Fatalf("seeding a future migration: %v", err) + } + if _, err := db.Exec(`PRAGMA user_version = 2`); err != nil { + t.Fatalf("bumping user_version: %v", err) + } + + _, err := Migrate(ctx, db, "") + if !errors.Is(err, ErrMigrationLedger) { + t.Fatalf("Migrate accepted a newer store, got %v", err) + } + if !strings.Contains(err.Error(), "forward-only") { + t.Errorf("error does not explain that downgrades are unsupported: %v", err) + } +} + +func TestMigrateRefusesUserVersionLedgerDisagreement(t *testing.T) { + db := openMemory(t) + ctx := context.Background() + if _, err := Migrate(ctx, db, ""); err != nil { + t.Fatalf("Migrate: %v", err) + } + if _, err := db.Exec(`PRAGMA user_version = 9`); err != nil { + t.Fatalf("bumping user_version: %v", err) + } + if _, err := Migrate(ctx, db, ""); !errors.Is(err, ErrMigrationLedger) { + t.Fatalf("Migrate accepted user_version 9 with one applied migration, got %v", err) + } +} + +func TestMigrateRefusesAPopulatedDatabaseWithNoLedger(t *testing.T) { + db := openMemory(t) + if _, err := db.Exec(`PRAGMA user_version = 4`); err != nil { + t.Fatalf("bumping user_version: %v", err) + } + _, err := Migrate(context.Background(), db, "") + if !errors.Is(err, ErrMigrationLedger) { + t.Fatalf("Migrate accepted a foreign database with no schema_migration table, got %v", err) + } +} + +func TestMigrateRefusesAGapInTheLedger(t *testing.T) { + db := openMemory(t) + ctx := context.Background() + migrations := withSynthetic(t, + synthetic(2, "second", "CREATE TABLE second_step (x INTEGER);"), + synthetic(3, "third", "CREATE TABLE third_step (y INTEGER);"), + ) + if _, err := migrateWith(ctx, db, migrations, t.TempDir()); err != nil { + t.Fatalf("migrateWith: %v", err) + } + if _, err := db.Exec(`DELETE FROM schema_migration WHERE version = 2`); err != nil { + t.Fatalf("punching a hole in the ledger: %v", err) + } + if _, err := migrateWith(ctx, db, migrations, t.TempDir()); !errors.Is(err, ErrMigrationLedger) { + t.Fatalf("Migrate accepted a ledger with a gap, got %v", err) + } +} + +// --------------------------------------------------------------------------- +// Forward-only means the snapshot is mandatory +// --------------------------------------------------------------------------- + +func TestMigrateRefusesToUpgradeWithoutASnapshotDirectory(t *testing.T) { + db, _ := openOnDisk(t) + ctx := context.Background() + if _, err := Migrate(ctx, db, ""); err != nil { + t.Fatalf("initial Migrate: %v", err) + } + + migrations := withSynthetic(t, synthetic(2, "second", "CREATE TABLE second_step (x INTEGER);")) + _, err := migrateWith(ctx, db, migrations, "") + if !errors.Is(err, ErrSnapshotRequired) { + t.Fatalf("Migrate upgraded a populated database with no snapshot directory, got %v", err) + } + if !strings.Contains(err.Error(), "forward-only") { + t.Errorf("error does not explain why the snapshot is mandatory: %v", err) + } + if v := mustSchemaVersion(t, db); v != 1 { + t.Fatalf("a refused upgrade changed user_version to %d", v) + } +} + +// TestSnapshotIsTakenBeforeAnUpgrade proves the replacement for down +// migrations actually exists on disk and is a readable database at the +// pre-upgrade version — an unopenable file would be a rollback path in name +// only. +func TestSnapshotIsTakenBeforeAnUpgrade(t *testing.T) { + db, dir := openOnDisk(t) + ctx := context.Background() + if _, err := Migrate(ctx, db, ""); err != nil { + t.Fatalf("initial Migrate: %v", err) + } + + snapDir := filepath.Join(dir, "snapshots") + migrations := withSynthetic(t, synthetic(2, "second", "CREATE TABLE second_step (x INTEGER);")) + if _, err := migrateWith(ctx, db, migrations, snapDir); err != nil { + t.Fatalf("migrateWith: %v", err) + } + + snap := filepath.Join(snapDir, "anvil-pre-v1.db") + if _, err := os.Stat(snap); err != nil { + t.Fatalf("no pre-migration snapshot at %s: %v", snap, err) + } + + restored, err := sql.Open("sqlite", snap) + if err != nil { + t.Fatalf("reopening the snapshot: %v", err) + } + defer func() { _ = restored.Close() }() + restored.SetMaxOpenConns(1) + + if v := mustSchemaVersion(t, restored); v != 1 { + t.Fatalf("snapshot is at user_version %d, want the pre-upgrade version 1", v) + } + var n int + if err := restored.QueryRow(`SELECT count(*) FROM sqlite_schema WHERE name = 'second_step'`).Scan(&n); err != nil { + t.Fatalf("querying the snapshot: %v", err) + } + if n != 0 { + t.Fatal("the snapshot contains the migration it was supposed to precede") + } +} + +func TestSnapshotRefusesToOverwriteAnExistingFile(t *testing.T) { + db, dir := openOnDisk(t) + ctx := context.Background() + if _, err := Migrate(ctx, db, ""); err != nil { + t.Fatalf("Migrate: %v", err) + } + dest := filepath.Join(dir, "snap.db") + if err := Snapshot(ctx, db, dest); err != nil { + t.Fatalf("Snapshot: %v", err) + } + if err := Snapshot(ctx, db, dest); err == nil { + t.Fatal("Snapshot overwrote an existing snapshot") + } + + // snapshotPath must therefore never hand back a name that is already + // taken, or a retried upgrade would fail on its own previous snapshot. + first, err := snapshotPath(dir, 1) + if err != nil { + t.Fatalf("snapshotPath: %v", err) + } + if err := os.WriteFile(first, []byte("occupied"), 0o600); err != nil { + t.Fatalf("occupying the snapshot name: %v", err) + } + second, err := snapshotPath(dir, 1) + if err != nil { + t.Fatalf("snapshotPath: %v", err) + } + if second == first { + t.Fatalf("snapshotPath returned the occupied name %q twice", first) + } +} + +func TestSnapshotRejectsEmptyArguments(t *testing.T) { + db := openMemory(t) + if err := Snapshot(context.Background(), db, " "); err == nil { + t.Fatal("Snapshot accepted an empty destination") + } + if err := Snapshot(context.Background(), nil, "x.db"); err == nil { + t.Fatal("Snapshot accepted a nil database") + } +} + +func TestMigrateRejectsNilDatabase(t *testing.T) { + if _, err := Migrate(context.Background(), nil, ""); err == nil { + t.Fatal("Migrate accepted a nil database handle") + } +} + +// --------------------------------------------------------------------------- +// Synthetic migrations +// --------------------------------------------------------------------------- + +func synthetic(version int, name, body string) Migration { + sum := sha256.Sum256([]byte(body)) + return Migration{ + Version: version, + Name: name, + Filename: fmt.Sprintf("%04d_%s.sql", version, name), + SQL: body, + Checksum: hex.EncodeToString(sum[:]), + } +} + +// withSynthetic appends test-only migrations after the real embedded set. +func withSynthetic(t *testing.T, extra ...Migration) []Migration { + t.Helper() + migrations, err := Migrations() + if err != nil { + t.Fatalf("Migrations: %v", err) + } + return append(migrations, extra...) +} diff --git a/internal/store/migrations/0001_init.sql b/internal/store/migrations/0001_init.sql new file mode 100644 index 0000000..80353ef --- /dev/null +++ b/internal/store/migrations/0001_init.sql @@ -0,0 +1,33 @@ +-- 0001_init — the initial Anvil schema. +-- +-- This migration deliberately contains no DDL of its own. Its body is R.4's +-- internal/store/schema.sql, included verbatim by the directive below and +-- expanded by migrate.go at load time from the string ddl.go already embeds. +-- +-- WHY AN INCLUDE AND NOT A COPY. plan/40-record-and-storage.md declares +-- schema.sql a frozen interface and R.4 built ddl.go explicitly as the "Go +-- wrapper exposing the DDL as an embedded string for R.5's migrations". A +-- second, byte-identical copy of 26 KB of DDL in this directory would be a +-- second definition of that interface: the two would drift the first time +-- someone edited one of them, and the drift would be invisible until a fresh +-- install and an upgraded install disagreed about the shape of a table. There +-- is exactly one copy of the schema in this repository, and this is a pointer +-- to it. +-- +-- CHECKSUM SENSITIVITY. migrate.go checksums the EXPANDED text — this file's +-- bytes with the directive line replaced by schema.sql's bytes. So editing +-- either file changes migration 0001's checksum, and a database that has +-- already applied the old text refuses to start rather than silently running +-- on a schema nobody applied. That is the whole point of the ledger. +-- +-- FORWARD-ONLY. There is no 0001_init.down.sql and there never will be, per +-- research/07-database-design.md §7: the rollback story is the +-- `VACUUM INTO 'anvil-pre-v{N}.db'` snapshot migrate.go takes before touching +-- an existing database, not reversible DDL nobody ever tests. +-- +-- NO PRAGMA HERE. migrate.go runs every migration inside BEGIN ... COMMIT and +-- `PRAGMA journal_mode = WAL` cannot run in a transaction. Connection pragmas +-- are ddl.go's ConnectionPragmas(); `PRAGMA user_version` is set by migrate.go +-- inside the same transaction as this file's DDL. + +-- @anvil:include schema.sql diff --git a/internal/store/schema.sql b/internal/store/schema.sql new file mode 100644 index 0000000..69b147d --- /dev/null +++ b/internal/store/schema.sql @@ -0,0 +1,459 @@ +-- Anvil store of record — complete DDL for schema version 1 (step R.4). +-- +-- plan/00-SPINE.md S1 collapses the originally-specified "8-hour buffer file" +-- into ONE SQLite database plus a `handoff` table plus a regenerable tmpfs +-- packet. There is no second durable buffer file and no second durable table +-- carrying finding dispositions: plan/IMPLEMENTATION-PLAN.md §6 rulings G9 and +-- G10 make THIS FILE the only definition of `handoff` anywhere in Anvil. +-- Area 70's O.3 migration and area 60's `anvil_ledger` are both deleted and +-- folded in here. +-- +-- SOURCES, in precedence order: +-- 1. plan/40-record-and-storage.md, "Store Schema" — authoritative for +-- `scan_run`, `audit_record`, `finding`, `finding_occurrence`, `handoff`. +-- 2. plan/IMPLEMENTATION-PLAN.md §6 (G9, G10) — supersedes (1) where they +-- disagree: `handoff.state` carries all thirteen dispositions and +-- `handoff.consumption_class` is present. +-- 3. research/07-database-design.md §2 — carried forward unchanged for every +-- other table. No research/07 table is dropped. +-- +-- WHAT IS DELIBERATELY NOT IN THIS FILE: +-- +-- * PRAGMA statements. `PRAGMA journal_mode = WAL` cannot run inside a +-- transaction, and R.5 applies this file inside `BEGIN ... COMMIT`. The +-- connection pragmas plan/40-record-and-storage.md specifies are exposed +-- by ddl.go as ConnectionPragmas() and are applied per connection, before +-- any other store operation. `foreign_keys` in particular is per +-- connection and OFF by default; this schema depends on it being ON. +-- * `PRAGMA user_version` and the migration ledger's contents. R.5 owns +-- both; the `schema_migration` table itself is created here. +-- +-- CHECK-CONSTRAINT POLICY. A CHECK constraint that names literals freezes a +-- vocabulary, and plan/IMPLEMENTATION-PLAN.md §6 documents what happens when +-- two areas freeze the same vocabulary differently: one area writes a literal +-- another area's NOT NULL column cannot accept. So enum CHECKs appear here +-- ONLY where internal/record/contract.go owns the enum, and ddl_test.go +-- asserts every such constraint agrees literal-for-literal with the Go values. +-- Columns whose comment names a vocabulary that contract.go does NOT own +-- (severity, ecosystem, suppression classification, ...) are left +-- unconstrained on purpose; constraining them here would be area 40 inventing +-- vocabulary for another area, which is the defect §6 exists to stop. +-- +-- Every CHECK is NAMED. research/07 Risk #15: batch-recreate tooling silently +-- drops unnamed CHECK constraints, which on a security tool's schema is an +-- integrity regression with no error message. + +-- ============ ADVISORY DOMAIN (feeds owned by area 20) ============ +CREATE TABLE advisory ( + advisory_id TEXT PRIMARY KEY, -- 'CVE-2026-1234' | 'GHSA-xxxx-yyyy-zzzz' + source TEXT NOT NULL, -- 'osv' | 'ghsa' | 'nvd' | ... + published_at TEXT NOT NULL, -- ISO-8601 UTC + modified_at TEXT NOT NULL, + severity TEXT, -- 'critical'|'high'|'medium'|'low'|'none' + cvss_vector TEXT, + cvss_score REAL, + summary TEXT NOT NULL, + details TEXT, + raw_json BLOB, -- zstd-compressed original + content_hash TEXT NOT NULL, -- sha256 of canonical form; drives delta ingest + ingested_at TEXT NOT NULL +); + +CREATE TABLE advisory_alias ( -- CVE <-> GHSA <-> OSV + advisory_id TEXT NOT NULL REFERENCES advisory(advisory_id) ON DELETE CASCADE, + alias_id TEXT NOT NULL, + PRIMARY KEY (advisory_id, alias_id) +); +CREATE INDEX idx_alias_lookup ON advisory_alias(alias_id); + +-- BM25 over advisory text. External-content table: no duplicate storage of +-- summary/details. Query: +-- ... WHERE advisory_fts MATCH ? ORDER BY bm25(advisory_fts, 4.0, 1.0, 8.0) +-- with `aliases` weighted highest so a literal 'CVE-2026-1234' beats prose. +-- +-- This is also the schema's built-in FTS5 availability probe: if the driver +-- lacks FTS5, applying this file fails loudly here rather than at first query. +-- R.5's CheckFTS5 guard still runs independently at every process start. +-- +-- KNOWN DEFECT, CARRIED FORWARD DELIBERATELY AND REPORTED, NOT SILENTLY +-- PATCHED. `content='advisory'` makes FTS5 read column values back from +-- `advisory`, but `advisory` has no `aliases` column — aliases are a +-- one-to-many in `advisory_alias`. So `INSERT INTO advisory_fts(advisory_fts) +-- VALUES('rebuild')`, any `SELECT FROM advisory_fts`, and any +-- snippet()/highlight() call fail with: +-- SQL logic error: no such column: T.aliases +-- Verified empirically on modernc.org/sqlite v1.56.0. Writes into +-- advisory_fts and rowid-only MATCH queries work; ddl_test.go exercises +-- exactly those. R.4 does not invent a fix because the two candidate repairs +-- (point `content=` at a view that projects the aliases, or drop `aliases` +-- from the FTS columns and re-derive the bm25 weights) both change an +-- interface area 20's ingestion owns. Flagged to the orchestrator. +CREATE VIRTUAL TABLE advisory_fts USING fts5( + summary, details, aliases, + content='advisory', content_rowid='rowid', + tokenize='porter unicode61' +); + +-- ============ COMPONENT DOMAIN ============ +CREATE TABLE component ( + component_id INTEGER PRIMARY KEY, + ecosystem TEXT NOT NULL, -- 'npm'|'pypi'|'cargo'|'go'|'deb'|'rpm'|'maven' + name TEXT NOT NULL, + purl_base TEXT NOT NULL, -- 'pkg:pypi/requests' (no version) + UNIQUE (ecosystem, name) +); + +CREATE TABLE advisory_affects ( + advisory_id TEXT NOT NULL REFERENCES advisory(advisory_id) ON DELETE CASCADE, + component_id INTEGER NOT NULL REFERENCES component(component_id), + introduced TEXT, -- version range endpoints, ecosystem-native + fixed TEXT, + range_kind TEXT NOT NULL, -- 'semver'|'ecosystem'|'git' + PRIMARY KEY (advisory_id, component_id, introduced, fixed) +); +CREATE INDEX idx_affects_component ON advisory_affects(component_id, advisory_id); + +-- ============ TARGET + TRIGGER (never hard-coded) ============ +CREATE TABLE target ( + target_id INTEGER PRIMARY KEY, + kind TEXT NOT NULL, -- 'repo'|'host' + locator TEXT NOT NULL UNIQUE, -- clone URL or hostname + config_json TEXT NOT NULL DEFAULT '{}', + CONSTRAINT ck_target_config_json CHECK (json_valid(config_json)) +); + +CREATE TABLE trigger_policy ( -- HARD CONSTRAINT: policy lives in data, not code + policy_id INTEGER PRIMARY KEY, + target_id INTEGER NOT NULL REFERENCES target(target_id) ON DELETE CASCADE, + kind TEXT NOT NULL, -- 'cron'|'github_action'|'webhook'|'manual' + spec TEXT NOT NULL, -- cron expr | tag glob 'v*.0.0' | event name + scan_depth TEXT NOT NULL, -- 'full'|'incremental'|'sca_only' + enabled INTEGER NOT NULL DEFAULT 1, + config_json TEXT NOT NULL DEFAULT '{}', + CONSTRAINT ck_trigger_policy_enabled_bool CHECK (enabled IN (0, 1)), + CONSTRAINT ck_trigger_policy_config_json CHECK (json_valid(config_json)) +); +CREATE INDEX idx_policy_enabled ON trigger_policy(target_id, kind) WHERE enabled = 1; + +CREATE TABLE ingest_watermark ( -- delta scraping cursors (feeds = area 20) + source TEXT PRIMARY KEY, + cursor TEXT, + etag TEXT, + last_success_at TEXT +); + +-- ============ SCAN RUN — unchanged shape from research/07 ============ +CREATE TABLE scan_run ( + scan_run_id INTEGER PRIMARY KEY, + target_id INTEGER NOT NULL REFERENCES target(target_id), + policy_id INTEGER REFERENCES trigger_policy(policy_id), + trigger_ref TEXT, -- tag / commit / cron fire id + commit_sha TEXT, + started_at TEXT NOT NULL, -- anvil/deadline.deadlineAt is computed from THIS, never last write + finished_at TEXT, + status TEXT NOT NULL, -- scan_run.status, owned by internal/record + sast_engine_ver TEXT, + dast_engine_ver TEXT, + ruleset_version TEXT NOT NULL, + advisory_snapshot TEXT, -- max(advisory.modified_at) at scan time + rollup_hash TEXT, -- sha256 over sorted finding fingerprints + CONSTRAINT ck_scan_run_status CHECK (status IN ('running', 'ok', 'failed', 'partial')) +); +CREATE INDEX idx_scan_target_time ON scan_run(target_id, started_at DESC); +CREATE INDEX idx_scan_running ON scan_run(target_id) WHERE status = 'running'; + +-- ============ AUDIT RECORD — the collapsed store (S1: no second durable buffer file) ============ +CREATE TABLE audit_record ( + audit_record_id INTEGER PRIMARY KEY, + scan_run_id INTEGER NOT NULL UNIQUE REFERENCES scan_run(scan_run_id), + schema_version TEXT NOT NULL, -- anvil/schemaVersion + audit_version INTEGER NOT NULL DEFAULT 1, -- anvil/version; a bump triggers R.11's queue re-cut + state TEXT NOT NULL, -- anvil/state + sast_status TEXT, -- per-half status (anvil/status) + sast_sealed_at TEXT, + dast_status TEXT NOT NULL DEFAULT 'not_run', -- S6: never NULL, never silently 'completed_clean' + dast_sealed_at TEXT, + dast_coverage_json TEXT, -- S6: {probedCount, inventoryUnionCount, inventoryProvenanceMix} + target_provenance TEXT NOT NULL, -- S6: a target that failed to boot must be distinguishable from scanned clean + deadline_at TEXT NOT NULL, -- = scan_run.started_at + claim_timeout_seconds. NEVER recomputed. + claim_timeout_seconds INTEGER NOT NULL DEFAULT 28800, -- 8h default, configurable. A CLAIM timeout, not a deletion policy. + dast_deadline_seconds INTEGER, -- S6: configurable, independent clock from claim_timeout_seconds + payload BLOB, -- zstd(canonical SARIF JSON); NULLed by the reaper at deadline_at + payload_sha256 TEXT NOT NULL, -- survives payload deletion: proof of what was handed over + created_at TEXT NOT NULL, + consumed_at TEXT, + purged_at TEXT, + CONSTRAINT ck_audit_record_state CHECK ( + state IN ('collecting', 'sast_sealed', 'dast_sealed', 'both_sealed', 'consumed', 'expired')), + CONSTRAINT ck_audit_record_sast_status CHECK ( + sast_status IS NULL OR sast_status IN ('running', 'sealed', 'failed', 'timed_out', 'skipped')), + CONSTRAINT ck_audit_record_dast_status CHECK ( + dast_status IN ('not_run', 'skipped_no_manifest', 'running', 'completed_clean', + 'completed_findings', 'completed_partial', 'target_boot_failed', + 'target_unreachable', 'timed_out')), + CONSTRAINT ck_audit_record_target_provenance CHECK ( + target_provenance IN ('booted_clean', 'boot_failed', 'build_failed', + 'no_target_declared', 'unreachable_at_scan_time')), + CONSTRAINT ck_audit_record_payload_sha256_hex CHECK ( + length(payload_sha256) = 64 AND payload_sha256 NOT GLOB '*[^0-9a-f]*'), + CONSTRAINT ck_audit_record_claim_timeout_positive CHECK (claim_timeout_seconds > 0), + CONSTRAINT ck_audit_record_dast_deadline_positive CHECK ( + dast_deadline_seconds IS NULL OR dast_deadline_seconds > 0), + CONSTRAINT ck_audit_record_audit_version_positive CHECK (audit_version >= 1), + CONSTRAINT ck_audit_record_dast_coverage_json CHECK ( + dast_coverage_json IS NULL OR json_valid(dast_coverage_json)) +); +CREATE INDEX idx_audit_deadline ON audit_record(deadline_at) WHERE purged_at IS NULL; +CREATE INDEX idx_audit_state ON audit_record(state); + +-- ============ CODE LOCATION ============ +CREATE TABLE code_location ( + location_id INTEGER PRIMARY KEY, + repo_relpath TEXT NOT NULL, -- POSIX separators, repo-root-relative, never absolute + start_line INTEGER, + end_line INTEGER, + start_col INTEGER, + end_col INTEGER, + symbol TEXT, -- 'pkg/mod.py::ClassA.method_b' + symbol_kind TEXT, -- 'function'|'method'|'class'|'module' + blob_sha TEXT, -- git blob hash of the file at scan time + snippet_hash TEXT -- sha256 of NORMALISED snippet; never the raw snippet +); +CREATE INDEX idx_loc_path ON code_location(repo_relpath, start_line); + +-- ============ FINDING: the stable identity ============ +-- Extended from research/07 with the S6/S24 fields (evidence_class, verdict, +-- remediable_by_agent). `finding_id` is the permanent key; `fingerprint` is a +-- lookup key that may be re-derived, because no vendor guarantees fingerprint +-- stability (research/07 Risk #1). +CREATE TABLE finding ( + finding_id INTEGER PRIMARY KEY, + target_id INTEGER NOT NULL REFERENCES target(target_id), + fingerprint TEXT NOT NULL, -- anvil-fp/v1, full 64-hex SHA-256, never truncated + fingerprint_alg TEXT NOT NULL DEFAULT 'anvil-fp/v1', + detector TEXT NOT NULL, -- finding.detector + evidence_class TEXT NOT NULL, -- anvil/evidenceClass + rule_id TEXT NOT NULL, -- versioned: 'anvil.py.sqli/v3' + verdict TEXT NOT NULL DEFAULT 'true_positive', -- S6: anvil/verdict + remediable_by_agent INTEGER NOT NULL, -- S6: 0/1; host findings are ALWAYS 0 (S7 read-only host agent) + advisory_id TEXT REFERENCES advisory(advisory_id), + component_id INTEGER REFERENCES component(component_id), + severity TEXT NOT NULL, + title TEXT NOT NULL, + state TEXT NOT NULL, -- finding.state + first_seen_scan INTEGER NOT NULL REFERENCES scan_run(scan_run_id), + first_seen_at TEXT NOT NULL, + last_seen_scan INTEGER REFERENCES scan_run(scan_run_id), + last_seen_at TEXT, + resolved_at TEXT, + resolved_by_fix INTEGER, -- -> fix_attempt(fix_attempt_id); intentionally not an FK, see note below + UNIQUE (target_id, fingerprint), -- THE regression-check index + CONSTRAINT ck_finding_detector CHECK (detector IN ('sast', 'dast', 'sca', 'host')), + CONSTRAINT ck_finding_evidence_class CHECK ( + evidence_class IN ('dast_confirmed', 'sast_reachable', 'sast_static_only', 'sca', 'host')), + CONSTRAINT ck_finding_verdict CHECK ( + verdict IN ('true_positive', 'false_positive', 'insufficient_context')), + CONSTRAINT ck_finding_state CHECK (state IN ('open', 'resolved', 'suppressed', 'regressed')), + CONSTRAINT ck_finding_remediable_bool CHECK (remediable_by_agent IN (0, 1)), + CONSTRAINT ck_finding_host_not_remediable CHECK (detector != 'host' OR remediable_by_agent = 0), + CONSTRAINT ck_finding_fingerprint_hex CHECK ( + length(fingerprint) = 64 AND fingerprint NOT GLOB '*[^0-9a-f]*') +); +CREATE INDEX idx_finding_state ON finding(target_id, state, severity); +CREATE INDEX idx_finding_evclass ON finding(target_id, evidence_class); +CREATE INDEX idx_finding_verdict ON finding(verdict) WHERE verdict != 'true_positive'; +CREATE INDEX idx_finding_adv ON finding(advisory_id) WHERE advisory_id IS NOT NULL; +CREATE INDEX idx_finding_comp ON finding(component_id) WHERE component_id IS NOT NULL; +-- NOTE on `resolved_by_fix`: research/07 §2 and plan/40's Store Schema both +-- declare it a bare INTEGER, not a REFERENCES clause, because `fix_attempt` +-- also references `finding` and a mutual FK pair cannot be satisfied by either +-- insert order without deferred constraints. Carried forward as specified. + +-- Multi-fingerprint side table -> fuzzy fallback + algorithm migration +-- without data loss. +CREATE TABLE finding_fingerprint ( + finding_id INTEGER NOT NULL REFERENCES finding(finding_id) ON DELETE CASCADE, + kind TEXT NOT NULL, -- 'primary'|'line_hash'|'symbol_hash'|'purl_advisory'|'dast_route' + alg TEXT NOT NULL, -- 'anvil-fp/v1' | 'lineHash/v1' | ... + value TEXT NOT NULL, + PRIMARY KEY (finding_id, kind, alg) +); +CREATE INDEX idx_fp_lookup ON finding_fingerprint(kind, value); + +-- ============ OCCURRENCE: one row per (finding, scan) ============ +-- Extended with the S6 advisory-staleness fields. +CREATE TABLE finding_occurrence ( + occurrence_id INTEGER PRIMARY KEY, + finding_id INTEGER NOT NULL REFERENCES finding(finding_id) ON DELETE CASCADE, + scan_run_id INTEGER NOT NULL REFERENCES scan_run(scan_run_id) ON DELETE CASCADE, + location_id INTEGER REFERENCES code_location(location_id), + confidence REAL, + message TEXT, -- hashes/pointers only; oversized inserts rejected by trigger + evidence_ref TEXT, -- pointer INTO the sealed payload; never the raw request/response + advisory_as_of TEXT, -- S6: as_of + advisory_staleness_seconds INTEGER, -- S6: staleness_seconds + advisory_parse_degraded INTEGER NOT NULL DEFAULT 0, -- S6: parse_degraded + UNIQUE (finding_id, scan_run_id), + CONSTRAINT ck_occurrence_parse_degraded_bool CHECK (advisory_parse_degraded IN (0, 1)) +); +CREATE INDEX idx_occ_scan ON finding_occurrence(scan_run_id, finding_id); -- drives the delta query + +-- research/07 Risk #13, verbatim: "The 8-hour rule is defeatable by careless +-- denormalisation. If raw snippets or DAST request/response bodies get copied +-- into finding_occurrence.message, they outlive the purge forever." These two +-- triggers are that risk's named mitigation. The cap is deliberately far below +-- internal/record's smallest inline body cap (8 KiB) so that no request or +-- response body can be smuggled into a durable column even at its minimum +-- size, while remaining ample for a rule title plus a pointer. +CREATE TRIGGER trg_occurrence_durable_text_cap_insert +BEFORE INSERT ON finding_occurrence +FOR EACH ROW WHEN + length(CAST(COALESCE(NEW.message, '') AS BLOB)) > 2048 + OR length(CAST(COALESCE(NEW.evidence_ref, '') AS BLOB)) > 2048 +BEGIN + SELECT RAISE(ABORT, + 'finding_occurrence: message/evidence_ref exceed the durable-text cap; durable tables store hashes and pointers only (research/07 Risk #13)'); +END; + +CREATE TRIGGER trg_occurrence_durable_text_cap_update +BEFORE UPDATE ON finding_occurrence +FOR EACH ROW WHEN + length(CAST(COALESCE(NEW.message, '') AS BLOB)) > 2048 + OR length(CAST(COALESCE(NEW.evidence_ref, '') AS BLOB)) > 2048 +BEGIN + SELECT RAISE(ABORT, + 'finding_occurrence: message/evidence_ref exceed the durable-text cap; durable tables store hashes and pointers only (research/07 Risk #13)'); +END; + +-- ============ APPEND-ONLY STATE HISTORY ============ +CREATE TABLE finding_state_event ( + event_id INTEGER PRIMARY KEY, + finding_id INTEGER NOT NULL REFERENCES finding(finding_id) ON DELETE CASCADE, + scan_run_id INTEGER REFERENCES scan_run(scan_run_id), + from_state TEXT, + to_state TEXT NOT NULL, + cause TEXT NOT NULL, -- 'first_seen'|'absent_in_scan'|'fix_verified'|'regression'|'suppressed'|'expired' + at TEXT NOT NULL +); +CREATE INDEX idx_state_hist ON finding_state_event(finding_id, at DESC); + +-- ============ FIX + VERIFICATION ============ +CREATE TABLE fix_attempt ( + fix_attempt_id INTEGER PRIMARY KEY, + finding_id INTEGER NOT NULL REFERENCES finding(finding_id), + audit_record_id INTEGER REFERENCES audit_record(audit_record_id), + agent_model_id TEXT NOT NULL, + started_at TEXT NOT NULL, + finished_at TEXT, + status TEXT NOT NULL, -- 'proposed'|'applied'|'rejected'|'failed' + patch_ref TEXT, -- git ref / blob sha, not the diff text + branch_name TEXT, + pr_url TEXT +); +CREATE INDEX idx_fix_finding ON fix_attempt(finding_id, started_at DESC); + +CREATE TABLE verification ( + verification_id INTEGER PRIMARY KEY, + fix_attempt_id INTEGER NOT NULL REFERENCES fix_attempt(fix_attempt_id) ON DELETE CASCADE, + kind TEXT NOT NULL, -- 'rescan_sast'|'rescan_dast'|'build'|'unit_tests' + scan_run_id INTEGER REFERENCES scan_run(scan_run_id), + result TEXT NOT NULL, -- 'pass'|'fail'|'inconclusive' + details_json TEXT, + verified_at TEXT NOT NULL, + CONSTRAINT ck_verification_details_json CHECK (details_json IS NULL OR json_valid(details_json)) +); +CREATE INDEX idx_verif_fix ON verification(fix_attempt_id); + +-- ============ SUPPRESSION / FALSE POSITIVE ============ +CREATE TABLE suppression ( + suppression_id INTEGER PRIMARY KEY, + target_id INTEGER NOT NULL REFERENCES target(target_id) ON DELETE CASCADE, + match_kind TEXT NOT NULL, -- 'fingerprint'|'rule'|'path_glob'|'advisory'|'component' + match_value TEXT NOT NULL, + classification TEXT NOT NULL, -- 'false_positive'|'accepted_risk'|'not_exploitable'|'wont_fix' + justification TEXT NOT NULL, + created_by TEXT NOT NULL, + created_at TEXT NOT NULL, + expires_at TEXT, -- expiring suppressions prevent permanent blindness + active INTEGER NOT NULL DEFAULT 1, + CONSTRAINT ck_suppression_active_bool CHECK (active IN (0, 1)) +); +CREATE INDEX idx_supp_match ON suppression(target_id, match_kind, match_value) WHERE active = 1; + +-- ============ INCREMENTAL-SCAN CACHE (this is where compute is actually saved) ============ +CREATE TABLE file_state ( + target_id INTEGER NOT NULL REFERENCES target(target_id) ON DELETE CASCADE, + repo_relpath TEXT NOT NULL, + blob_sha TEXT NOT NULL, + ruleset_version TEXT NOT NULL, + last_scan_id INTEGER NOT NULL REFERENCES scan_run(scan_run_id), + PRIMARY KEY (target_id, repo_relpath) +); + +-- ============ HANDOFF — the collapsed buffer replacement (S1: state/lease/attempts/expiry) ============ +-- +-- ONE TABLE, ONE STATE COLUMN. plan/IMPLEMENTATION-PLAN.md §6 G10 traced the +-- concrete bug that a second table produces: area X wrote `skipped_budget` to +-- its own `anvil_ledger` while area 40's ready-set index still saw the finding +-- as 'ready', so the finding was re-leased forever. `anvil_ledger` is deleted; +-- its four extra dispositions (fixed_incidentally, split_required, withdrawn, +-- superseded) are values of `state` here. +-- +-- `consumption_class` arrives from area 70's O.3 (§6 G9). Nothing else in this +-- schema can express the gate research/21 §5 requires: `static_only` findings +-- are claimable once the SAST half is sealed, `requires_dynamic_confirmation` +-- findings must wait on the DAST half. It is NOT NULL with NO DEFAULT on +-- purpose — a default would silently grant every row the permissive value, and +-- plan/00-SPINE.md S7 says only a DAST reproduction earns "verified fixed". +-- +-- TWO INDEPENDENT CLOCKS, NEVER CONFLATED (S1): +-- handoff.lease_expires_at 15-30 min, heartbeat-renewed, governs ONE +-- coding-agent attempt. +-- audit_record.claim_timeout_seconds default 8h, governs how long an +-- UNCLAIMED finding stays eligible. At its +-- expiry the reaper drops only the tmpfs +-- packet and NULLs audit_record.payload; +-- this row, `finding`, and +-- `finding_state_event` are never deleted. +CREATE TABLE handoff ( + handoff_id INTEGER PRIMARY KEY, + finding_id INTEGER NOT NULL REFERENCES finding(finding_id) ON DELETE CASCADE, + audit_record_id INTEGER NOT NULL REFERENCES audit_record(audit_record_id), + fingerprint TEXT NOT NULL, -- denormalised for the reaper's WHERE clause + group_id TEXT, -- fix-group id, assigned by the coding-agent consumption pipeline + state TEXT NOT NULL DEFAULT 'ready', + consumption_class TEXT NOT NULL, -- from O.3 per §6 G9; no default, see note above + claimed_by TEXT, -- worker_id (O.3's lease_owner) + lease_expires_at TEXT, -- claim lease. NOT audit_record.claim_timeout_seconds. + attempts INTEGER NOT NULL DEFAULT 0, + max_attempts INTEGER NOT NULL DEFAULT 2, + idempotency_key TEXT UNIQUE, -- sha256(audit_id || finding_fingerprint || base_commit_sha); mirrors the git trailer + created_at TEXT NOT NULL, + updated_at TEXT NOT NULL, + UNIQUE (finding_id, audit_record_id), + CONSTRAINT ck_handoff_state CHECK ( + state IN ('ready', 'leased', 'validated', 'failed_validation', 'failed_format', + 'skipped_budget', 'false_positive', 'regression_introduced', + 'fixed_incidentally', 'split_required', 'withdrawn', 'superseded', + 'expired')), + CONSTRAINT ck_handoff_consumption_class CHECK ( + consumption_class IN ('static_only', 'requires_dynamic_confirmation')), + CONSTRAINT ck_handoff_fingerprint_hex CHECK ( + length(fingerprint) = 64 AND fingerprint NOT GLOB '*[^0-9a-f]*'), + CONSTRAINT ck_handoff_attempts_nonneg CHECK (attempts >= 0 AND max_attempts >= 0), + CONSTRAINT ck_handoff_lease_requires_holder CHECK ( + state != 'leased' OR (claimed_by IS NOT NULL AND lease_expires_at IS NOT NULL)) +); +CREATE INDEX idx_handoff_ready ON handoff(state) WHERE state = 'ready'; +CREATE INDEX idx_handoff_lease ON handoff(lease_expires_at) WHERE state = 'leased'; +CREATE INDEX idx_handoff_fp ON handoff(fingerprint); + +-- ============ MIGRATIONS ============ +-- Rows are written by R.5's migrate.go, in the same transaction that applies +-- the migration and bumps PRAGMA user_version. +CREATE TABLE schema_migration ( + version INTEGER PRIMARY KEY, + name TEXT NOT NULL, + checksum TEXT NOT NULL, + applied_at TEXT NOT NULL +); diff --git a/testdata/fingerprint_corpus/dast-01-sqli-error-based-body.json b/testdata/fingerprint_corpus/dast-01-sqli-error-based-body.json new file mode 100644 index 0000000..25258bc --- /dev/null +++ b/testdata/fingerprint_corpus/dast-01-sqli-error-based-body.json @@ -0,0 +1,148 @@ +{ + "id": "dast-01-sqli-error-based-body", + "tier": "dast", + "description": "A dynamically-confirmed SQL injection: payload injected into a JSON body parameter, observed as a database error string in the response. The baseline DAST fixture.", + "input": { + "target_id": "t-0003", + "rule_id_versioned": "nuclei:sqli-error-based@2026.07.1", + "http_method": "POST", + "route_template": "/api/v1/users/{id}/orders", + "injection_point": "body", + "param_name": "sortBy", + "evidence_signal": "dbErrorString" + }, + "not_hashed": { + "scheme": "https", + "host": "staging-7f3a.ephemeral.internal", + "port": 8443, + "concrete_url": "https://staging-7f3a.ephemeral.internal:8443/api/v1/users/4192/orders", + "payload": "' OR 1=1--", + "response_time_ms": 1204, + "observed_at": "2026-08-08T12:04:11Z", + "comment": "plan/40-record-and-storage.md: 'NEVER hashed: host, port, scheme, the concrete payload string, any timestamp.' A redeployed container, a rotated ephemeral port, or a staging-to-prod move must not fork identity, and a fuzzer varying its payload must not mint N findings for one bug." + }, + "hashed_fields": [ + "t-0003", + "dast", + "nuclei:sqli-error-based@2026.07.1", + "POST", + "/api/v1/users//orders", + "body", + "sortBy", + "dbErrorString" + ], + "expected_digest": "199c3b5fa4615b4caa74c05394d08e78cc84d2d2c40ffd3178f268c02bf7db90", + "mutations": [ + { + "name": "http_method_lowercased", + "description": "A producer emitting 'post' must not fork identity from one emitting 'POST'; the method is uppercased before hashing.", + "input": { + "target_id": "t-0003", + "rule_id_versioned": "nuclei:sqli-error-based@2026.07.1", + "http_method": "post", + "route_template": "/api/v1/users/{id}/orders", + "injection_point": "body", + "param_name": "sortBy", + "evidence_signal": "dbErrorString" + } + }, + { + "name": "http_method_padded", + "description": "Whitespace from a parsed request line is trimmed.", + "input": { + "target_id": "t-0003", + "rule_id_versioned": "nuclei:sqli-error-based@2026.07.1", + "http_method": " POST ", + "route_template": "/api/v1/users/{id}/orders", + "injection_point": "body", + "param_name": "sortBy", + "evidence_signal": "dbErrorString" + } + }, + { + "name": "route_template_carrying_the_payload_in_a_query_string", + "description": "The single most important DAST canonicalisation. A producer that smuggles the concrete payload into the 'template' would otherwise mint one finding per fuzzer iteration; everything from the first '?' is dropped.", + "input": { + "target_id": "t-0003", + "rule_id_versioned": "nuclei:sqli-error-based@2026.07.1", + "http_method": "POST", + "route_template": "/api/v1/users/{id}/orders?debug=1&sortBy=%27%20OR%201%3D1--", + "injection_point": "body", + "param_name": "sortBy", + "evidence_signal": "dbErrorString" + } + }, + { + "name": "route_template_carrying_a_fragment", + "description": "Everything from the first '#' is dropped for the same reason.", + "input": { + "target_id": "t-0003", + "rule_id_versioned": "nuclei:sqli-error-based@2026.07.1", + "http_method": "POST", + "route_template": "/api/v1/users/{id}/orders#results", + "injection_point": "body", + "param_name": "sortBy", + "evidence_signal": "dbErrorString" + } + }, + { + "name": "route_template_with_a_trailing_slash", + "description": "A trailing slash is removed (except on the root path).", + "input": { + "target_id": "t-0003", + "rule_id_versioned": "nuclei:sqli-error-based@2026.07.1", + "http_method": "POST", + "route_template": "/api/v1/users/{id}/orders/", + "injection_point": "body", + "param_name": "sortBy", + "evidence_signal": "dbErrorString" + } + }, + { + "name": "route_template_with_duplicate_slashes_and_no_leading_slash", + "description": "Slash runs collapse and a leading slash is added, so a crawler joining path segments naively agrees with one that does not.", + "input": { + "target_id": "t-0003", + "rule_id_versioned": "nuclei:sqli-error-based@2026.07.1", + "http_method": "POST", + "route_template": "api//v1/users/{id}//orders", + "injection_point": "body", + "param_name": "sortBy", + "evidence_signal": "dbErrorString" + } + }, + { + "name": "route_template_carrying_the_concrete_numeric_id_this_scan_requested", + "description": "The route this fixture's not_hashed.concrete_url actually carried. CanonicalRouteTemplate replaces the all-digit segment '4192' with , so the producer does not have to template the route itself. Before the R.3 blocker-2 ruling this input produced a THIRD digest for the same defect.", + "input": { + "target_id": "t-0003", + "rule_id_versioned": "nuclei:sqli-error-based@2026.07.1", + "http_method": "POST", + "route_template": "/api/v1/users/4192/orders", + "injection_point": "body", + "param_name": "sortBy", + "evidence_signal": "dbErrorString" + } + }, + { + "name": "route_template_in_express_placeholder_syntax", + "description": "A producer reading the repository's own Express route table emits ':id'; a producer reading its OpenAPI document emits '{id}'. Both normalise to , so the two producers agree.", + "input": { + "target_id": "t-0003", + "rule_id_versioned": "nuclei:sqli-error-based@2026.07.1", + "http_method": "POST", + "route_template": "/api/v1/users/:id/orders", + "injection_point": "body", + "param_name": "sortBy", + "evidence_signal": "dbErrorString" + } + } + ], + "notes": [ + "Field order is target_id, \"dast\", rule_id_versioned, http_method, route_template, injection_point, param_name, evidence_class_detail.", + "DIGEST CHANGED 2026-08-08 under the R.3 blocker-2 ruling, from ca801b8d64fdabf43aad112e3b62c53211cf369a66683c12352081357eb9d125. The hashed route_template moved from '/api/v1/users/{id}/orders' to '/api/v1/users//orders' because CanonicalRouteTemplate now DERIVES the template, as the specification always said it should ('route_template := numeric/UUID/hash path segments replaced with a placeholder token'). Nothing was stored under the old digest - anvil-fp/v1 has not shipped - so this is a correction, not a v2 event. See internal/record/FINGERPRINT-SPEC.md section 6.", + "injection_point (WHERE the payload went in) and evidence_class_detail (HOW the defect was observed) are independent hashed fields. research/07 had only the former and research/18 only the latter; the resolution hashes both, because an SQL injection proved by a database error string and one proved by a timing side channel on the same parameter are different findings with different remediation evidence.", + "param_name is a parameter NAME, never a parameter VALUE.", + "Both literals are lowercase snake_case-free tokens frozen in R.1's contract.go (InjectionPoint, EvidenceSignal); an unfrozen or SCREAMING_CASE literal is rejected rather than hashed." + ] +} diff --git a/testdata/fingerprint_corpus/dast-02-xss-reflected-query.json b/testdata/fingerprint_corpus/dast-02-xss-reflected-query.json new file mode 100644 index 0000000..79c2fd8 --- /dev/null +++ b/testdata/fingerprint_corpus/dast-02-xss-reflected-query.json @@ -0,0 +1,78 @@ +{ + "id": "dast-02-xss-reflected-query", + "tier": "dast", + "description": "A reflected cross-site scripting finding: payload injected into a query parameter, observed by seeing the payload reflected in the response. Differs from dast-01 in every discriminating field, including both the injection point and the evidence signal.", + "input": { + "target_id": "t-0003", + "rule_id_versioned": "nuclei:xss-reflected@2026.07.1", + "http_method": "GET", + "route_template": "/search", + "injection_point": "query", + "param_name": "q", + "evidence_signal": "reflectedPayload" + }, + "not_hashed": { + "scheme": "http", + "host": "127.0.0.1", + "port": 41337, + "payload": "", + "response_status": 200, + "observed_at": "2026-08-08T12:06:02Z" + }, + "hashed_fields": [ + "t-0003", + "dast", + "nuclei:xss-reflected@2026.07.1", + "GET", + "/search", + "query", + "q", + "reflectedPayload" + ], + "expected_digest": "bbe1a200328dfd7415d4a209384515a7ae01cb0494a896e69ba2bf565299c489", + "mutations": [ + { + "name": "http_method_lowercased", + "description": "Method case does not fork identity.", + "input": { + "target_id": "t-0003", + "rule_id_versioned": "nuclei:xss-reflected@2026.07.1", + "http_method": "get", + "route_template": "/search", + "injection_point": "query", + "param_name": "q", + "evidence_signal": "reflectedPayload" + } + }, + { + "name": "route_template_carrying_the_injected_query_parameter", + "description": "The query string is dropped: injection_point and param_name already record which query parameter was targeted, so keeping it would hash the payload value as well.", + "input": { + "target_id": "t-0003", + "rule_id_versioned": "nuclei:xss-reflected@2026.07.1", + "http_method": "GET", + "route_template": "/search?q=%3Cscript%3Ealert(1)%3C%2Fscript%3E", + "injection_point": "query", + "param_name": "q", + "evidence_signal": "reflectedPayload" + } + }, + { + "name": "route_template_with_a_trailing_slash", + "description": "'/search/' and '/search' are one route.", + "input": { + "target_id": "t-0003", + "rule_id_versioned": "nuclei:xss-reflected@2026.07.1", + "http_method": "GET", + "route_template": "/search/", + "injection_point": "query", + "param_name": "q", + "evidence_signal": "reflectedPayload" + } + } + ], + "notes": [ + "Paired with dast-01 to demonstrate that two DAST findings on one target stay distinct. Sharing a target_id with dast-01 and dast-03 is deliberate: the target is not what separates them.", + "Route templates are case-sensitive and are NOT folded: URL paths are case-sensitive, and '/Search' is a different route from '/search' on most servers." + ] +} diff --git a/testdata/fingerprint_corpus/dast-03-path-traversal-no-param-name.json b/testdata/fingerprint_corpus/dast-03-path-traversal-no-param-name.json new file mode 100644 index 0000000..1ba82e1 --- /dev/null +++ b/testdata/fingerprint_corpus/dast-03-path-traversal-no-param-name.json @@ -0,0 +1,105 @@ +{ + "id": "dast-03-path-traversal-no-param-name", + "tier": "dast", + "description": "A path-traversal finding injected into a templated path segment. There is no named parameter, so this fixture pins the EMPTY param_name case, and it observes the defect by a stack trace in the response body rather than by reflection.", + "input": { + "target_id": "t-0003", + "rule_id_versioned": "nuclei:path-traversal@2026.07.1", + "http_method": "GET", + "route_template": "/files/{path}", + "injection_point": "path", + "param_name": "", + "evidence_signal": "responseStackTrace" + }, + "not_hashed": { + "scheme": "http", + "host": "127.0.0.1", + "port": 41337, + "payload": "../../../../etc/passwd", + "response_body_sha256": "referenced, never inlined - plan/00-SPINE.md S7 names the DAST response body the highest-risk field", + "observed_at": "2026-08-08T12:07:44Z" + }, + "hashed_fields": [ + "t-0003", + "dast", + "nuclei:path-traversal@2026.07.1", + "GET", + "/files/", + "path", + "", + "responseStackTrace" + ], + "expected_digest": "5fc15c556531a9a7b6d6fe4a56c36a5383bbc2a0555c4e6ba60e686c0dd21550", + "mutations": [ + { + "name": "http_method_lowercased", + "description": "Method case does not fork identity.", + "input": { + "target_id": "t-0003", + "rule_id_versioned": "nuclei:path-traversal@2026.07.1", + "http_method": "get", + "route_template": "/files/{path}", + "injection_point": "path", + "param_name": "", + "evidence_signal": "responseStackTrace" + } + }, + { + "name": "route_template_with_duplicate_leading_slashes", + "description": "Slash runs collapse to one.", + "input": { + "target_id": "t-0003", + "rule_id_versioned": "nuclei:path-traversal@2026.07.1", + "http_method": "GET", + "route_template": "//files/{path}", + "injection_point": "path", + "param_name": "", + "evidence_signal": "responseStackTrace" + } + }, + { + "name": "route_template_carrying_the_traversal_payload_as_a_query_string", + "description": "The payload smuggled into the template is dropped with the rest of the query string.", + "input": { + "target_id": "t-0003", + "rule_id_versioned": "nuclei:path-traversal@2026.07.1", + "http_method": "GET", + "route_template": "/files/{path}?f=..%2F..%2F..%2Fetc%2Fpasswd", + "injection_point": "path", + "param_name": "", + "evidence_signal": "responseStackTrace" + } + }, + { + "name": "route_template_in_flask_placeholder_syntax", + "description": "Flask and Werkzeug spell a variable segment ''. It normalises to the same as '{path}' and ':path'.", + "input": { + "target_id": "t-0003", + "rule_id_versioned": "nuclei:path-traversal@2026.07.1", + "http_method": "GET", + "route_template": "/files/", + "injection_point": "path", + "param_name": "", + "evidence_signal": "responseStackTrace" + } + }, + { + "name": "route_template_already_carrying_the_canonical_placeholder_token", + "description": "Idempotence: feeding a canonicalised route back through CanonicalRouteTemplate must be a fixed point, or a record round-tripped through the store would fork its own identity.", + "input": { + "target_id": "t-0003", + "rule_id_versioned": "nuclei:path-traversal@2026.07.1", + "http_method": "GET", + "route_template": "/files/", + "injection_point": "path", + "param_name": "", + "evidence_signal": "responseStackTrace" + } + } + ], + "notes": [ + "An empty param_name is legal and is hashed as an empty field, keeping the DAST field count at exactly eight. A whole-body, raw-request, or path-segment injection has no single named parameter, and rejecting those would make them unfingerprintable.", + "DIGEST CHANGED 2026-08-08 under the R.3 blocker-2 ruling, from 84fe311debbc87f852e1f4e17a3242d071169448c5852a00b1b32316fb30b036. The hashed route_template moved from '/files/{path}' to '/files/' because CanonicalRouteTemplate now normalises an already-templated segment onto the one frozen placeholder token rather than trusting the producer's choice of syntax. Nothing was stored under the old digest - anvil-fp/v1 has not shipped. See internal/record/FINGERPRINT-SPEC.md section 6.", + "The input's '{path}' was the ONLY normalization the old corpus exercised on this field, and it exercised it by arriving pre-templated - which is exactly why CRITIQUE-01 could not detect that CanonicalRouteTemplate did no templating at all. dast-04 is the fixture that closes that hole, with a concrete numeric id and a concrete UUID." + ] +} diff --git a/testdata/fingerprint_corpus/dast-04-idor-concrete-numeric-and-uuid-segments.json b/testdata/fingerprint_corpus/dast-04-idor-concrete-numeric-and-uuid-segments.json new file mode 100644 index 0000000..96208a8 --- /dev/null +++ b/testdata/fingerprint_corpus/dast-04-idor-concrete-numeric-and-uuid-segments.json @@ -0,0 +1,200 @@ +{ + "id": "dast-04-idor-concrete-numeric-and-uuid-segments", + "tier": "dast", + "description": "An insecure-direct-object-reference finding whose observed route carries BOTH a concrete numeric id and a concrete UUID. This is the fixture CRITIQUE-01 said the corpus was missing: dast-01 through dast-03 all arrived pre-templated, so no fixture could tell whether CanonicalRouteTemplate templated anything at all. Every mutation below is a different concrete instance of the SAME defect and must produce the SAME digest.", + "input": { + "target_id": "t-0003", + "rule_id_versioned": "nuclei:idor-object-reference@2026.07.1", + "http_method": "GET", + "route_template": "/api/v1/users/12345/orders/3f2504e0-4f89-11d3-9a0c-0305e82c3301", + "injection_point": "path", + "param_name": "", + "evidence_signal": "statusCodeFlip" + }, + "not_hashed": { + "scheme": "https", + "host": "staging-7f3a.ephemeral.internal", + "port": 8443, + "concrete_url": "https://staging-7f3a.ephemeral.internal:8443/api/v1/users/12345/orders/3f2504e0-4f89-11d3-9a0c-0305e82c3301", + "authenticated_as": "user 4192", + "observed_status_flip": "403 -> 200", + "observed_at": "2026-08-08T16:41:09Z", + "comment": "The numeric id and the UUID are volatile in the strongest sense: the next scan against a freshly seeded ephemeral target will observe different ones for the same defect. If they were hashed, every scan would open a new finding and 'verified fixed' could never be proved - which is precisely why plan/00-SPINE.md S7's fix-verification gate depends on this tier." + }, + "hashed_fields": [ + "t-0003", + "dast", + "nuclei:idor-object-reference@2026.07.1", + "GET", + "/api/v1/users//orders/", + "path", + "", + "statusCodeFlip" + ], + "expected_digest": "84f96a6be8790384b52dda96908acc0ca9c493ada224d996f065d9eb3be4605b", + "mutations": [ + { + "name": "different_numeric_id", + "description": "THE mutation the R.3 ruling demands. A second scan lands on user 98765 instead of user 12345; it is the same defect and must keep the same identity. Without route templating this input produced a different digest and the finding was reported resolved and re-opened as new.", + "input": { + "target_id": "t-0003", + "rule_id_versioned": "nuclei:idor-object-reference@2026.07.1", + "http_method": "GET", + "route_template": "/api/v1/users/98765/orders/3f2504e0-4f89-11d3-9a0c-0305e82c3301", + "injection_point": "path", + "param_name": "", + "evidence_signal": "statusCodeFlip" + } + }, + { + "name": "different_uuid", + "description": "The order id is a freshly minted UUID on every seeded run.", + "input": { + "target_id": "t-0003", + "rule_id_versioned": "nuclei:idor-object-reference@2026.07.1", + "http_method": "GET", + "route_template": "/api/v1/users/12345/orders/9e107d9d-372b-b682-6bd8-1d3542a419d6", + "injection_point": "path", + "param_name": "", + "evidence_signal": "statusCodeFlip" + } + }, + { + "name": "both_ids_different_at_once", + "description": "The realistic re-scan: nothing about the concrete instance survives, and the identity still holds.", + "input": { + "target_id": "t-0003", + "rule_id_versioned": "nuclei:idor-object-reference@2026.07.1", + "http_method": "GET", + "route_template": "/api/v1/users/1/orders/00000000-0000-0000-0000-000000000000", + "injection_point": "path", + "param_name": "", + "evidence_signal": "statusCodeFlip" + } + }, + { + "name": "uuid_uppercased", + "description": "The UUID predicate is case-insensitive: the same identifier rendered in upper hex is the same identifier.", + "input": { + "target_id": "t-0003", + "rule_id_versioned": "nuclei:idor-object-reference@2026.07.1", + "http_method": "GET", + "route_template": "/api/v1/users/12345/orders/3F2504E0-4F89-11D3-9A0C-0305E82C3301", + "injection_point": "path", + "param_name": "", + "evidence_signal": "statusCodeFlip" + } + }, + { + "name": "uuid_without_dashes", + "description": "A 32-character dash-free UUID is caught by the long-hex rule (>= 16 hex characters) rather than the UUID rule. Both land on the same token, so a service that renders ids with dashes and one that renders them without do not fork identity.", + "input": { + "target_id": "t-0003", + "rule_id_versioned": "nuclei:idor-object-reference@2026.07.1", + "http_method": "GET", + "route_template": "/api/v1/users/12345/orders/3f2504e04f8911d39a0c0305e82c3301", + "injection_point": "path", + "param_name": "", + "evidence_signal": "statusCodeFlip" + } + }, + { + "name": "sha256_hex_object_id", + "description": "A content-addressed order id: 64 hex characters, also caught by the long-hex rule.", + "input": { + "target_id": "t-0003", + "rule_id_versioned": "nuclei:idor-object-reference@2026.07.1", + "http_method": "GET", + "route_template": "/api/v1/users/12345/orders/e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", + "injection_point": "path", + "param_name": "", + "evidence_signal": "statusCodeFlip" + } + }, + { + "name": "opaque_base64ish_order_token", + "description": "A 22-character alphanumeric token carrying both letters and digits, caught by the long-opaque rule (>= 20 characters). Note it is alphanumeric only: a base64url token containing '-' or '_' is deliberately NOT templated, because hyphenated slugs are route structure and over-templating merges distinct routes.", + "input": { + "target_id": "t-0003", + "rule_id_versioned": "nuclei:idor-object-reference@2026.07.1", + "http_method": "GET", + "route_template": "/api/v1/users/12345/orders/dXNlcjEyMzQ1Njc4OTAxMg", + "injection_point": "path", + "param_name": "", + "evidence_signal": "statusCodeFlip" + } + }, + { + "name": "openapi_curly_placeholders", + "description": "The repository's own OpenAPI document spells both segments '{userId}' and '{orderId}'. Different placeholder NAMES must not fork identity either - the token replaces the whole segment, name included.", + "input": { + "target_id": "t-0003", + "rule_id_versioned": "nuclei:idor-object-reference@2026.07.1", + "http_method": "GET", + "route_template": "/api/v1/users/{userId}/orders/{orderId}", + "injection_point": "path", + "param_name": "", + "evidence_signal": "statusCodeFlip" + } + }, + { + "name": "express_colon_placeholders", + "description": "Express, Rails and Sinatra spell the same route ':userId'/':orderId'.", + "input": { + "target_id": "t-0003", + "rule_id_versioned": "nuclei:idor-object-reference@2026.07.1", + "http_method": "GET", + "route_template": "/api/v1/users/:userId/orders/:orderId", + "injection_point": "path", + "param_name": "", + "evidence_signal": "statusCodeFlip" + } + }, + { + "name": "flask_angle_placeholders_with_converters", + "description": "Flask and Werkzeug spell it ''/'', converter prefix and all.", + "input": { + "target_id": "t-0003", + "rule_id_versioned": "nuclei:idor-object-reference@2026.07.1", + "http_method": "GET", + "route_template": "/api/v1/users//orders/", + "injection_point": "path", + "param_name": "", + "evidence_signal": "statusCodeFlip" + } + }, + { + "name": "already_carrying_the_canonical_placeholder_token", + "description": "Idempotence. A canonicalised route fed back through CanonicalRouteTemplate is a fixed point, so a record read out of the store and re-fingerprinted keeps its identity.", + "input": { + "target_id": "t-0003", + "rule_id_versioned": "nuclei:idor-object-reference@2026.07.1", + "http_method": "GET", + "route_template": "/api/v1/users//orders/", + "injection_point": "path", + "param_name": "", + "evidence_signal": "statusCodeFlip" + } + }, + { + "name": "concrete_ids_with_query_string_trailing_slash_and_duplicate_slashes", + "description": "Templating composes with the pre-existing canonicalisation: query string dropped, slash runs collapsed, trailing slash removed, leading slash added, and only then are the segments templated.", + "input": { + "target_id": "t-0003", + "rule_id_versioned": "nuclei:idor-object-reference@2026.07.1", + "http_method": "get", + "route_template": "api//v1/users/12345//orders/3f2504e0-4f89-11d3-9a0c-0305e82c3301/?debug=1&as=4192", + "injection_point": "path", + "param_name": "", + "evidence_signal": "statusCodeFlip" + } + } + ], + "notes": [ + "This fixture exists because of CRITIQUE-01 blocker 2: 'All three DAST fixtures arrive already templated, so the corpus cannot detect this. There is no test anywhere that feeds a concrete numeric or UUID segment.'", + "route_template is a DERIVED field. Area 40 owns the fingerprint, so area 40 canonicalises: area D emits whatever route it observed and CanonicalRouteTemplate templates it. Pushing templating to the producer would give the field two owners and let two producers emit three digests for one defect.", + "The placeholder token is , frozen. Angle brackets cannot appear unencoded in a URI path (RFC 3986), so no literal segment can collide with it, and is itself recognised as an already-templated segment, which makes the function idempotent.", + "The templating rules are conservative on purpose. Over-templating merges two distinct routes into one identity and loses a finding on upsert; under-templating only leaves a volatile route un-merged, which a producer can still fix by emitting '{id}' itself. The exact predicates and their length thresholds are in internal/record/FINGERPRINT-SPEC.md section 6.", + "Note the deliberate NON-mutations: '/api/v1/users/me/orders/...' and '/api/v1/users/12345/invoices/...' are DIFFERENT routes and must produce different digests. TestCanonicalRouteTemplateDoesNotOverTemplate in fingerprint_test.go pins the segments that must survive untouched." + ] +} diff --git a/testdata/fingerprint_corpus/host-01-openssl-debian.json b/testdata/fingerprint_corpus/host-01-openssl-debian.json new file mode 100644 index 0000000..03964af --- /dev/null +++ b/testdata/fingerprint_corpus/host-01-openssl-debian.json @@ -0,0 +1,88 @@ +{ + "id": "host-01-openssl-debian", + "tier": "host", + "description": "An operating-system package on the scanned host that matched a vulnerable version range. Host findings are read-only for the agent (plan/00-SPINE.md S7, remediable_by_agent=false) but still need a stable identity so they can be tracked, suppressed, and reported as resolved.", + "input": { + "target_id": "t-0002", + "advisory_id": "CVE-2022-0778", + "purl": "pkg:deb/debian/openssl@1.1.1n-0+deb11u3", + "package_manager": "apt", + "host_identifier": "openssl:amd64" + }, + "not_hashed": { + "installed_version": "1.1.1n-0+deb11u3", + "fixed_version": "1.1.1n-0+deb11u4", + "hostname": "build-runner-04", + "scanned_at": "2026-08-08T12:00:00Z" + }, + "hashed_fields": [ + "t-0002", + "host", + "CVE-2022-0778", + "pkg:deb/debian/openssl", + "apt:openssl:amd64" + ], + "expected_digest": "c6d2c1a505849bf987d9a03a5a3ce8ab6053a9ae5e0d154d332b464bb69e5953", + "mutations": [ + { + "name": "version_bumped", + "description": "A Debian point release. The version lives only in the purl and is stripped before hashing.", + "input": { + "target_id": "t-0002", + "advisory_id": "CVE-2022-0778", + "purl": "pkg:deb/debian/openssl@1.1.1w-0+deb11u1", + "package_manager": "apt", + "host_identifier": "openssl:amd64" + } + }, + { + "name": "purl_carrying_an_arch_qualifier", + "description": "Qualifiers are stripped with the version.", + "input": { + "target_id": "t-0002", + "advisory_id": "CVE-2022-0778", + "purl": "pkg:deb/debian/openssl@1.1.1n-0+deb11u3?arch=amd64", + "package_manager": "apt", + "host_identifier": "openssl:amd64" + } + }, + { + "name": "package_manager_uppercased", + "description": "'APT' and 'apt' are the same manager. The manager segment of the locator is lowercased, because a case difference between two scanner versions would otherwise fork every host finding at once.", + "input": { + "target_id": "t-0002", + "advisory_id": "CVE-2022-0778", + "purl": "pkg:deb/debian/openssl@1.1.1n-0+deb11u3", + "package_manager": "APT", + "host_identifier": "openssl:amd64" + } + }, + { + "name": "package_manager_and_identifier_padded", + "description": "Surrounding whitespace from a parsed CLI table is trimmed.", + "input": { + "target_id": "t-0002", + "advisory_id": "CVE-2022-0778", + "purl": "pkg:deb/debian/openssl@1.1.1n-0+deb11u3", + "package_manager": " apt ", + "host_identifier": " openssl:amd64 " + } + }, + { + "name": "purl_already_version_free", + "description": "PurlBase is idempotent, so a producer that already stripped the version agrees with one that did not.", + "input": { + "target_id": "t-0002", + "advisory_id": "CVE-2022-0778", + "purl": "pkg:deb/debian/openssl", + "package_manager": "apt", + "host_identifier": "openssl:amd64" + } + } + ], + "notes": [ + "The host tier shares the SCA formula, parameterised by detector_kind. Its locator is ':'.", + "host_identifier keeps its architecture suffix verbatim: 'openssl:amd64' and 'openssl:i386' are two installed packages the manager treats separately, so they are two findings.", + "The locator therefore contains more than one ':'. That is fine - it is a composed field, not a parsed one - but package_manager itself is rejected if it contains ':', because that would make the composition ambiguous." + ] +} diff --git a/testdata/fingerprint_corpus/sast-01-go-sql-string-concat.json b/testdata/fingerprint_corpus/sast-01-go-sql-string-concat.json new file mode 100644 index 0000000..a5504c4 --- /dev/null +++ b/testdata/fingerprint_corpus/sast-01-go-sql-string-concat.json @@ -0,0 +1,103 @@ +{ + "id": "sast-01-go-sql-string-concat", + "tier": "sast", + "description": "Go SQL injection by string concatenation, inside a named method. The baseline SAST fixture: a non-empty enclosing symbol, ordinal 0, two string literals and one abstracted local flowing into a preserved callee name.", + "input": { + "target_id": "t-0001", + "rule_id_versioned": "opengrep.go.lang.security.audit.sqli-string-concat@2026.07.1", + "repo_rel_path": "internal/api/store.go", + "enclosing_symbol_path": "internal/api/store.go::Store.FindByName", + "snippet": "rows, err := db.Query(\"SELECT * FROM users WHERE name = '\" + name + \"'\")", + "ordinal": 0 + }, + "not_hashed": { + "line": 128, + "column": 9, + "end_line": 128, + "advisory_id": "CWE-89", + "evidence_class": "sast_static_only", + "scanned_at": "2026-08-08T12:00:00Z", + "comment": "Every value in this object was available to the producer and MUST NOT appear in the hash input. plan/40-record-and-storage.md: 'NEVER hashed: line number, column number, the literal (non-normalized) snippet text, advisory_id.'" + }, + "hashed_fields": [ + "t-0001", + "sast", + "opengrep.go.lang.security.audit.sqli-string-concat@2026.07.1", + "internal/api/store.go", + "internal/api/store.go::Store.FindByName", + "$1, $2 := $3.Query( + $4 + )", + "0" + ], + "expected_digest": "13c60ccf8ec84530e075db4190005b03bc87050210065398fd48088683bb1aa6", + "mutations": [ + { + "name": "shifted_down_three_lines_and_reindented", + "description": "The match moved down the file and was reindented. This is the line-number-only mutation R.2's validation requires: SastInput carries no line field, so the strongest available form is to change everything a line move implies.", + "input": { + "target_id": "t-0001", + "rule_id_versioned": "opengrep.go.lang.security.audit.sqli-string-concat@2026.07.1", + "repo_rel_path": "internal/api/store.go", + "enclosing_symbol_path": "internal/api/store.go::Store.FindByName", + "snippet": "\n\n\n rows, err := db.Query(\"SELECT * FROM users WHERE name = '\" + name + \"'\")\n", + "ordinal": 0 + }, + "not_hashed": { + "line": 131, + "column": 9 + } + }, + { + "name": "crlf_endings_and_trailing_comment_naming_the_old_line", + "description": "A Windows checkout plus a reviewer comment that literally records the old line number. Comments are stripped and CRLF is folded to LF before hashing.", + "input": { + "target_id": "t-0001", + "rule_id_versioned": "opengrep.go.lang.security.audit.sqli-string-concat@2026.07.1", + "repo_rel_path": "internal/api/store.go", + "enclosing_symbol_path": "internal/api/store.go::Store.FindByName", + "snippet": "rows, err := db.Query(\"SELECT * FROM users WHERE name = '\" + name + \"'\") // was line 128\r\n", + "ordinal": 0 + } + }, + { + "name": "leading_comment_line_added", + "description": "A triage annotation added above the match. Comments carry no identity.", + "input": { + "target_id": "t-0001", + "rule_id_versioned": "opengrep.go.lang.security.audit.sqli-string-concat@2026.07.1", + "repo_rel_path": "internal/api/store.go", + "enclosing_symbol_path": "internal/api/store.go::Store.FindByName", + "snippet": "// SECURITY: reviewed 2026-01-14, tracked as ANV-412\nrows, err := db.Query(\"SELECT * FROM users WHERE name = '\" + name + \"'\")", + "ordinal": 0 + } + }, + { + "name": "locals_renamed_and_literal_text_changed", + "description": "A pure rename refactor plus a different SQL string. This is the metavariable-abstraction property the specification adopts from Semgrep's match_based_id: local identifiers become positional $1..$N and string literals become , so neither a rename nor a literal edit forks the finding's identity.", + "input": { + "target_id": "t-0001", + "rule_id_versioned": "opengrep.go.lang.security.audit.sqli-string-concat@2026.07.1", + "repo_rel_path": "internal/api/store.go", + "enclosing_symbol_path": "internal/api/store.go::Store.FindByName", + "snippet": "records, e := conn.Query(\"SELECT * FROM accounts WHERE label = '\" + userName + \"'\")", + "ordinal": 0 + } + }, + { + "name": "repo_rel_path_in_windows_form", + "description": "The same file reported by a Windows producer. repo_relpath is canonicalised to POSIX form before hashing, so two producers scanning one repository agree.", + "input": { + "target_id": "t-0001", + "rule_id_versioned": "opengrep.go.lang.security.audit.sqli-string-concat@2026.07.1", + "repo_rel_path": ".\\internal\\api\\store.go", + "enclosing_symbol_path": "internal/api/store.go::Store.FindByName", + "snippet": "rows, err := db.Query(\"SELECT * FROM users WHERE name = '\" + name + \"'\")", + "ordinal": 0 + } + } + ], + "notes": [ + "Field order is target_id, \"sast\", rule_id_versioned, repo_relpath, enclosing_symbol_path, normalized_match, ordinal.", + "The tier token in position 2 is the literal \"sast\" for both sast_reachable and sast_static_only findings, so promoting a static-only finding to reachable does not change its identity.", + "Normalization of the snippet: `db` is a receiver bound to a local, so it is abstracted to $3; `Query` follows a '.' selector so it is preserved as API surface." + ] +} diff --git a/testdata/fingerprint_corpus/sast-02-python-shell-command.json b/testdata/fingerprint_corpus/sast-02-python-shell-command.json new file mode 100644 index 0000000..b0f0631 --- /dev/null +++ b/testdata/fingerprint_corpus/sast-02-python-shell-command.json @@ -0,0 +1,89 @@ +{ + "id": "sast-02-python-shell-command", + "tier": "sast", + "description": "Python command injection at module level: the SECOND of two byte-identical call sites in one file, so this fixture exercises ordinal 1 and an EMPTY enclosing_symbol_path (top-level module code has no enclosing symbol).", + "input": { + "target_id": "t-0001", + "rule_id_versioned": "opengrep.python.lang.security.audit.dangerous-system-call@2026.07.1", + "repo_rel_path": "scripts/deploy.py", + "enclosing_symbol_path": "", + "snippet": "os.system(\"rm -rf \" + path)", + "ordinal": 1 + }, + "not_hashed": { + "line": 41, + "column": 5, + "advisory_id": "CWE-78", + "evidence_class": "sast_reachable", + "sibling_call_site_line": 12, + "comment": "The identical call site at line 12 gets ordinal 0 and this one gets ordinal 1. Without the ordinal field the two would share a digest and one finding would be lost on upsert against UNIQUE (target_id, fingerprint)." + }, + "hashed_fields": [ + "t-0001", + "sast", + "opengrep.python.lang.security.audit.dangerous-system-call@2026.07.1", + "scripts/deploy.py", + "", + "$1.system( + $2)", + "1" + ], + "expected_digest": "d2239ee40c3f9da1cc9db3417962e3794b0c5baca0409015ff331a0aea095242", + "mutations": [ + { + "name": "shifted_down_and_reindented_into_a_block", + "description": "The statement moved and gained indentation. Neither line position nor indentation is hashed.", + "input": { + "target_id": "t-0001", + "rule_id_versioned": "opengrep.python.lang.security.audit.dangerous-system-call@2026.07.1", + "repo_rel_path": "scripts/deploy.py", + "enclosing_symbol_path": "", + "snippet": "\n\n os.system(\"rm -rf \" + path)\n", + "ordinal": 1 + }, + "not_hashed": { + "line": 58, + "column": 9 + } + }, + { + "name": "hash_line_comment_appended", + "description": "A '#' comment is stripped like any other comment, which is why the normalizer treats '#' as a line comment across languages.", + "input": { + "target_id": "t-0001", + "rule_id_versioned": "opengrep.python.lang.security.audit.dangerous-system-call@2026.07.1", + "repo_rel_path": "scripts/deploy.py", + "enclosing_symbol_path": "", + "snippet": "os.system(\"rm -rf \" + path) # noqa: was line 41", + "ordinal": 1 + } + }, + { + "name": "local_renamed_and_literal_changed", + "description": "`path` renamed and the shell string edited; both are abstracted, so identity holds.", + "input": { + "target_id": "t-0001", + "rule_id_versioned": "opengrep.python.lang.security.audit.dangerous-system-call@2026.07.1", + "repo_rel_path": "scripts/deploy.py", + "enclosing_symbol_path": "", + "snippet": "os.system(\"rm -rf --one-file-system \" + target_dir)", + "ordinal": 1 + } + }, + { + "name": "repo_rel_path_with_dot_slash_prefix", + "description": "'./scripts/deploy.py' and 'scripts/deploy.py' are one path after canonicalisation.", + "input": { + "target_id": "t-0001", + "rule_id_versioned": "opengrep.python.lang.security.audit.dangerous-system-call@2026.07.1", + "repo_rel_path": "./scripts/deploy.py", + "enclosing_symbol_path": "", + "snippet": "os.system(\"rm -rf \" + path)", + "ordinal": 1 + } + } + ], + "notes": [ + "An empty enclosing_symbol_path is hashed as an empty field rather than dropped, so the SAST tier always joins exactly seven fields. Dropping it instead would let a match with no symbol collide with a shorter field list.", + "`os` is abstracted to $1: a lexer cannot distinguish a module qualifier from a receiver variable, and guessing wrong in the other direction would fork the digest of unchanged code." + ] +} diff --git a/testdata/fingerprint_corpus/sca-01-log4shell-maven.json b/testdata/fingerprint_corpus/sca-01-log4shell-maven.json new file mode 100644 index 0000000..47aef2d --- /dev/null +++ b/testdata/fingerprint_corpus/sca-01-log4shell-maven.json @@ -0,0 +1,102 @@ +{ + "id": "sca-01-log4shell-maven", + "tier": "sca", + "description": "A repository dependency (Log4Shell, declared in a monorepo service's pom.xml) that matched a vulnerable version range. This is the fixture that pins the single most load-bearing exclusion in the specification: the version string is never hashed.", + "input": { + "target_id": "t-0001", + "advisory_id": "GHSA-jfh8-c2jp-5v3q", + "purl": "pkg:maven/org.apache.logging.log4j/log4j-core@2.14.1", + "manifest_rel_path": "services/api/pom.xml" + }, + "not_hashed": { + "resolved_version": "2.14.1", + "fixed_version": "2.15.0", + "first_seen_at": "2026-08-08T12:00:00Z", + "comment": "plan/40-record-and-storage.md: 'NEVER hashed: the version string - bumping 1.2.3->1.2.4 while still inside the vulnerable range must not mint a new finding; resolution is proved by re-evaluating advisory_affects, not by identity change.'" + }, + "hashed_fields": [ + "t-0001", + "sca", + "GHSA-jfh8-c2jp-5v3q", + "pkg:maven/org.apache.logging.log4j/log4j-core", + "services/api/pom.xml" + ], + "expected_digest": "c3208f00ab0f426a1c2cbb2ccf0033881bb4bf6cb506763dea0840f03dd24cd8", + "mutations": [ + { + "name": "version_bumped_inside_the_vulnerable_range", + "description": "2.14.1 -> 2.14.0. Still vulnerable, still the same finding. If this changed the digest, first_seen_at would reset and any fingerprint-keyed suppression would silently stop applying.", + "input": { + "target_id": "t-0001", + "advisory_id": "GHSA-jfh8-c2jp-5v3q", + "purl": "pkg:maven/org.apache.logging.log4j/log4j-core@2.14.0", + "manifest_rel_path": "services/api/pom.xml" + } + }, + { + "name": "version_bumped_out_of_the_vulnerable_range", + "description": "2.14.1 -> 2.17.1. The identity is still the same; resolution is proved by re-evaluating advisory_affects, never by the fingerprint changing.", + "input": { + "target_id": "t-0001", + "advisory_id": "GHSA-jfh8-c2jp-5v3q", + "purl": "pkg:maven/org.apache.logging.log4j/log4j-core@2.17.1", + "manifest_rel_path": "services/api/pom.xml" + } + }, + { + "name": "purl_carrying_qualifiers", + "description": "Qualifiers are stripped by PurlBase along with the version.", + "input": { + "target_id": "t-0001", + "advisory_id": "GHSA-jfh8-c2jp-5v3q", + "purl": "pkg:maven/org.apache.logging.log4j/log4j-core@2.14.1?type=jar&classifier=sources", + "manifest_rel_path": "services/api/pom.xml" + } + }, + { + "name": "purl_carrying_qualifiers_and_a_subpath", + "description": "Subpath ('#') is stripped as well.", + "input": { + "target_id": "t-0001", + "advisory_id": "GHSA-jfh8-c2jp-5v3q", + "purl": "pkg:maven/org.apache.logging.log4j/log4j-core@2.14.1?type=jar#META-INF/MANIFEST.MF", + "manifest_rel_path": "services/api/pom.xml" + } + }, + { + "name": "purl_scheme_and_type_uppercased", + "description": "The purl specification defines the scheme and type as case-insensitive with a lowercase canonical form, so 'PKG:MAVEN/...' and 'pkg:maven/...' are one package. The namespace and name are NOT folded, because their case-sensitivity is type-dependent.", + "input": { + "target_id": "t-0001", + "advisory_id": "GHSA-jfh8-c2jp-5v3q", + "purl": "PKG:MAVEN/org.apache.logging.log4j/log4j-core@2.14.1", + "manifest_rel_path": "services/api/pom.xml" + } + }, + { + "name": "purl_already_version_free", + "description": "A producer that strips the version itself gets the same digest as one that does not; PurlBase is idempotent.", + "input": { + "target_id": "t-0001", + "advisory_id": "GHSA-jfh8-c2jp-5v3q", + "purl": "pkg:maven/org.apache.logging.log4j/log4j-core", + "manifest_rel_path": "services/api/pom.xml" + } + }, + { + "name": "manifest_path_in_windows_form", + "description": "The locator is canonicalised to POSIX form like any other repo-relative path.", + "input": { + "target_id": "t-0001", + "advisory_id": "GHSA-jfh8-c2jp-5v3q", + "purl": "pkg:maven/org.apache.logging.log4j/log4j-core@2.14.1", + "manifest_rel_path": "services\\api\\pom.xml" + } + } + ], + "notes": [ + "Field order is target_id, detector_kind, advisory_id, purl_base, locator. For the SCA tier the locator is the manifest or lockfile path: the same vulnerable package pulled in by two manifests in a monorepo is two findings, with two owners and two fixes.", + "advisory_id is hashed verbatim, not case-folded. GHSA identifiers mix case meaningfully and folding them would fork identity against the advisory table.", + "detector_kind is what keeps this fixture from colliding with host-01 when a repository dependency and a host package share an advisory." + ] +}