diff --git a/internal/record/CRITIQUE-01.md b/internal/record/CRITIQUE-01.md new file mode 100644 index 0000000..9be719b --- /dev/null +++ b/internal/record/CRITIQUE-01.md @@ -0,0 +1,585 @@ +# CRITIQUE-01 — R.3, critique of R.1 (Record Field Contract) and R.2 (Fingerprint Specification) + +**Step:** R.3 · **Reviewed:** `internal/record/contract.go`, `internal/record/CONTRACT.md`, +`schemas/anvil-record-v1.schema.json`, `internal/record/fingerprint.go`, +`internal/record/fingerprint_test.go`, `testdata/fingerprint_corpus/*.json`, against +`plan/40-record-and-storage.md` (Record Field Contract + Fingerprint Specification), +`plan/00-SPINE.md` S1/S6/S7/S12, and `plan/IMPLEMENTATION-PLAN.md` §6. + +**Date:** 2026-08-08 · **Branch:** `feat/phase1-record-store` · **No code was modified by this pass.** + +--- + +## 0. WHICH GUARANTEE THIS DOCUMENT ACTUALLY PROVIDES — READ THIS FIRST + +**This was a SAME-FAMILY critic.** The critic and both implementers (R.1, R.2) are Anthropic models. +`plan/00-ROUTING.md` originally required a **different model family** here precisely so that a shared +blind spot could not survive review; the owner withdrew external routes on 2026-08-07 because running +them means copying `plan/` — deliberately private — to a third-party provider. See the OWNER DECISION +block at the top of `00-ROUTING.md`. + +**A later reader must not record this as a cross-family critique.** The cross-family guarantee +`00-ROUTING.md` asked for on data-integrity work has *not* been obtained, and nothing in this document +supplies it. What compensation was applied: + +- Findings were sought by refutation, not assessment: the implementations were assumed wrong. +- Every claim below was checked against the **files**, not against the implementers' prose. The + implementers' reasoning in doc comments is extensive and persuasive; it was treated as a claim to be + falsified, and in three places (findings 2, 3, 8) the doc comment states something the code does not do. +- Determinism was proved **empirically in two separate OS processes**, not by rereading the code and not + by trusting reported output. +- The one property that most needed an outside check — *can an independent re-implementation of the + written specification reproduce the committed digests?* — was answered by **writing that + re-implementation** (in Python, from the plan text, without reading `fingerprint.go`'s lexer) and + comparing digests. It cannot. That is finding 1, and it is the finding a same-family reviewer was most + likely to miss by nodding along with the Go code. + +**A residual same-family risk remains and cannot be closed here:** anything both the implementer and this +critic consider self-evidently correct about hashing, canonicalisation or SARIF was not challenged by an +outside vocabulary. + +--- + +## 1. VERDICT + +| Question R.3 must answer | Verdict | +|---|---| +| Every `00-SPINE.md` S6 field present in the contract | **PASS** (§3, field by field) | +| Fingerprint determinism | **PASS** — proved cross-process (§4) | +| Fingerprint excludes volatile fields | **PASS with one reservation** — package version, line, column, host, port, scheme, payload, timestamp all excluded and proved; the *ruleset* version is hashed and its churn is unbounded (finding 6) | +| Six frozen enums (§6) match, no second copy | **PASS** — literal by literal, three sources agree (§5) | +| Goldens are committed and compared, not regenerated | **PASS** (§6) | +| `CONTRACT.md` ⇄ `contract.go` ⇄ schema agree | **PASS** on all six enums and all S6 fields; one open decision unrecorded (finding 11) | +| **Is `anvil-fp/v1` "one algorithm, defined once" as S6 requires?** | **FAIL** — findings 1 and 2 | + +**Overall: FAIL.** Two blockers. Per the packet's stop condition, R.4 must not start until they are ruled +on. Note carefully that **neither blocker is a bug in the Go code**: the code is clean, well-tested and +internally consistent. Both are failures of the *written specification* to be the thing S6 says it must +be — the single, reproducible definition. Fixing them is an orchestrator ruling on +`plan/40-record-and-storage.md`, not a rewrite of `fingerprint.go`. + +--- + +## 2. GATE OUTPUT — real, re-run by this critic, `-count=1` throughout + +``` +$ gofmt -l . +(no output) + +$ go vet ./... +(no output) + +$ go build ./... +(no output) + +$ go test -count=1 ./... +ok github.com/Susquehanna-Syntax/Anvil/cmd/anvil 0.348s +? github.com/Susquehanna-Syntax/Anvil/cmd/anvil-dast [no test files] +? github.com/Susquehanna-Syntax/Anvil/internal/buildpin [no test files] +ok github.com/Susquehanna-Syntax/Anvil/internal/record 0.677s + +$ go test -count=1 -v ./internal/record/ | grep -c "^--- PASS" +129 +``` + +R.1's `contract_test.go` still passes alongside R.2's additions. Go 1.26.5. + +--- + +## 3. `00-SPINE.md` S6 — one verdict per required field + +S6's required set, in the order S6 lists it. "Where" cites the Go symbol; every row was additionally +checked to exist in `schemas/anvil-record-v1.schema.json` and to be described identically in +`CONTRACT.md`. + +| # | S6 field | Verdict | One-line reason | +|---|---|---|---| +| 1 | `anvil/state` | **PASS** | `State` + `PropAuditState`; six literals exactly as §6 froze them. | +| 2 | `anvil/version` | **PASS** | `PropAuditVersion = "anvil/version"`, present in schema and CONTRACT.md; correctly **not** hashed by any fingerprint tier. | +| 3 | per-half `status` | **PASS** | `HalfStatus` + `PropRunStatus`; `running\|sealed\|failed\|timed_out\|skipped`, matching §6's G5 ruling including the added `timed_out`. | +| 4 | per-half `sealedAt` | **PASS** | `PropRunSealedAt`; required in schema on a sealed half. | +| 5 | `anvil/trust` on every string originating outside Anvil | **PASS** | `Trust` with the three S6 literals, carried as an *object* not a bare enum (CONTRACT.md deviation 3) so one result can hold several provenances; `LegalForExternalString()` enforces the rule at the type level. The richer container is a strict improvement over the plan's table and does not change the literals. | +| 6 | `dast_status` | **PASS** | `DastStatus` + `PropAuditDastStatus`; all nine §6 literals, `skipped_no_manifest` kept distinct from `not_run` as G3+G6 requires. | +| 7 | `dast_coverage` | **PASS** | `PropRunDastCoverage`. | +| 8 | `target_provenance` | **PASS** | `TargetProvenance`, five literals; the G4+G7 split is honoured — the boot/reachability meaning is kept here and D's provisioning-path enum is the separate `TargetProvisioning` (`ephemeral_manifest\|live_url_authorized`). Both fields exist; neither was merged. | +| 9 | `remediable_by_agent` (host findings `false`) | **PASS** | `PropResultRemediableByAgent`; `Validate()` rejects `EvidenceClassHost` with `RemediableByAgent == true` (contract.go ~line 2317). Enforced in code, per S7's instruction. | +| 10 | `INSUFFICIENT_CONTEXT` as a verdict, not a confidence float | **PASS** | `VerdictInsufficientContext = "insufficient_context"`; a separate `anvil/confidence` exists but the verdict is a first-class enum value. | +| 11 | `as_of` | **PASS** | `Advisory.AsOf`; `Validate()` rejects a zero value when an advisory is linked. | +| 12 | `staleness_seconds` | **PASS** | `Advisory.StalenessSeconds`; non-negative enforced. | +| 13 | `parse_degraded` | **PASS** | `parseDegraded` in contract.go, schema and CONTRACT.md. | +| 14 | `endpoint_coverage` | **PASS** | `endpointCoverage` in contract.go and schema. | +| 15 | `inventory_provenance` | **PASS** | `InventoryProvenance` (`runtime_spec\|repo_spec\|static_extraction\|crawl`), in all three sources. | +| 16 | sanitizer state on any reproducer | **PASS** | `Repro.Env.Sanitizers []string`, `json:"sanitizers"`, schema-required; `Validate()` rejects `nil` and demands an empty array for a stock build, which is the right call — `null` and "no sanitizers" are different claims. | +| 17 | ASLR state on any reproducer | **PASS** | `Repro.Env.AslrEnabled bool`, `json:"aslrEnabled"`, schema-required. Key spelling identical in all three sources. | +| 18 | **One fingerprint algorithm, defined once, in the record** | **FAIL** | See findings 1 and 2. The algorithm is defined once *in Go*; the written specification is not sufficient to reproduce it, and one specified step of it is not implemented at all. | +| 19 | Ship a conformance test asserting identical digests on a fixed corpus | **PASS for R.2, at risk for R.16** | The corpus exists (7 fixtures, exceeding the 2/2/1/1 minimum) and is asserted. R.16's *independent* conformance oracle cannot currently be written to pass — finding 1. | + +S6's closing "Ordering" paragraph (re-cut the queue on every version bump; reserve a configurable +fraction of budget for late DAST arrivals) is a scheduler requirement, not a record field, and is not +in R.1's or R.2's scope. **It is not checked here and must not be assumed covered.** + +--- + +## 4. DETERMINISM — hunted first, proved empirically + +### 4.1 Static hunt + +`fingerprint.go` was read in full for the usual leaks. Grep, then reading: + +``` +$ grep -n "time\.\|rand\.\|filepath\.\|os\.\|runtime\.\|unsafe\.\|sync\.\|GOOS\|PathSeparator" internal/record/fingerprint.go +819: // fingerprinted is rejected here rather than at hash time. <- a comment, not a call + +$ grep -n "range " internal/record/fingerprint.go +262: for i, f := range fields <- slice +817: for i, c := range cands <- slice +851: for i, k := range order <- slice +``` + +- **Map iteration:** the only two maps are `metavars` (NormalizeMatch) and `fingerprintReservedWords`. + Both are read by key lookup only; neither is ranged over. Metavariable numbering comes from + `nextMetavar`, a counter advanced in source order, not from map order. **Clean.** +- **Time / randomness / pointers / addresses:** none reachable. `Digest` is a pure function of its + arguments. **Clean.** +- **Sorting:** `AssignSastOrdinals` uses `sort.SliceStable` with a total order (group key, then Line, + then Column, then original index), so ties cannot reorder. **Clean.** +- **Locale-dependent case folding:** `strings.ToUpper`/`ToLower` are used (HTTP method, purl type, host + package manager). Go's non-`Special` casers are Unicode-default and locale-independent — there is no + Turkish-I hazard. **Clean.** (Unicode *normalization* is a different matter — finding 8.) + +### 4.2 Empirical proof, in two separate OS processes + +Not two calls in one process — the failure mode that matters (per-process map seed, address-space state) +is invisible that way. The test binary was compiled once and executed **twice as independent +processes**, each printing the digest of every corpus fixture: + +``` +$ go test -count=1 -c -o $SCRATCH/record.test.exe ./internal/record +$ cd internal/record +$ ANVIL_FINGERPRINT_CROSS_PROCESS_CHILD=1 $SCRATCH/record.test.exe \ + -test.run='^TestCorpusDigestsAreStableAcrossProcesses$' -test.count=1 | grep ANVIL-FP > procA.txt +$ ANVIL_FINGERPRINT_CROSS_PROCESS_CHILD=1 $SCRATCH/record.test.exe \ + -test.run='^TestCorpusDigestsAreStableAcrossProcesses$' -test.count=1 | grep ANVIL-FP > procB.txt + +ANVIL-FP-DIGEST dast-01-sqli-error-based-body ca801b8d64fdabf43aad112e3b62c53211cf369a66683c12352081357eb9d125 +ANVIL-FP-DIGEST dast-02-xss-reflected-query bbe1a200328dfd7415d4a209384515a7ae01cb0494a896e69ba2bf565299c489 +ANVIL-FP-DIGEST dast-03-path-traversal-no-param-name 84fe311debbc87f852e1f4e17a3242d071169448c5852a00b1b32316fb30b036 +ANVIL-FP-DIGEST host-01-openssl-debian c6d2c1a505849bf987d9a03a5a3ce8ab6053a9ae5e0d154d332b464bb69e5953 +ANVIL-FP-DIGEST sast-01-go-sql-string-concat 13c60ccf8ec84530e075db4190005b03bc87050210065398fd48088683bb1aa6 +ANVIL-FP-DIGEST sast-02-python-shell-command d2239ee40c3f9da1cc9db3417962e3794b0c5baca0409015ff331a0aea095242 +ANVIL-FP-DIGEST sca-01-log4shell-maven c3208f00ab0f426a1c2cbb2ccf0033881bb4bf6cb506763dea0840f03dd24cd8 + +$ diff procA.txt procB.txt +IDENTICAL +``` + +Every digest also equals the `expected_digest` committed in the corresponding fixture. + +**Verdict: determinism PASS.** R.2's own `TestCorpusDigestsAreStableAcrossProcesses` is a genuine +cross-process check and not theatre; this critic re-ran the same property independently of it. + +### 4.3 Windows vs POSIX path separators + +Anvil targets Linux and is being developed on Windows, so a separator leak would be invisible on one +platform. Checked specifically: + +- `CanonicalRepoRelPath` and `CanonicalRouteTemplate` are **pure string manipulation**. Neither imports + `path/filepath`, `os.PathSeparator`, nor anything platform-conditional. The whole file imports only + `crypto/sha256`, `encoding/hex`, `fmt`, `sort`, `strconv`, `strings`, `unicode`. +- `sast-01`'s `repo_rel_path_in_windows_form` mutation (`.\internal\api\store.go`) and `sca-01`'s + `manifest_path_in_windows_form` mutation (`services\api\pom.xml`) both reproduce the base digest, and + they passed here on Windows. +- The one hazard the conversion introduces is the reverse direction — see finding 8(b). + +**No separator leak. PASS.** + +--- + +## 5. THE SIX FROZEN ENUMS — compared literal by literal against §6 + +`plan/IMPLEMENTATION-PLAN.md` §6's enum block was compared character by character against +`internal/record/contract.go`, `schemas/anvil-record-v1.schema.json` and `internal/record/CONTRACT.md`. + +| Enum | §6 literals | contract.go | schema `$defs` | CONTRACT.md | +|---|---|---|---|---| +| `anvil/state` | 6 | ✅ identical | ✅ identical | ✅ identical | +| `anvil/status` (per half) | 5 | ✅ | ✅ | ✅ | +| `anvil/dastStatus` | 9 | ✅ | ✅ | ✅ | +| `anvil/target.provenance` | 5 | ✅ | ✅ | ✅ | +| `anvil/target.provisioning` | 2 | ✅ | ✅ | ✅ | +| `anvil/verdict` | 3 | ✅ | ✅ | ✅ | +| `handoff.state` | 13 | ✅ | ✅ | ✅ | + +No drift, no omission, no extra literal, no case difference, no ordering difference in any of the three +sources. §6's "lowercase snake_case is the record's convention" holds for all six. + +**Did R.2 introduce a second, drifting copy? No.** `fingerprint.go` declares **no enum type**. It +consumes `DetectorKindSCA`, `DetectorKindHost`, `InjectionPoint` and `EvidenceSignal` from +`contract.go`, and validates the latter two through `ValidateInjectionPoint`/`ValidateEvidenceSignal` +rather than re-listing their literals. `TestTierValidationRejectsIncompleteInput` proves a +SCREAMING_CASE injection point and an off-spec `db_error_string` are both rejected. The one blemish is +two private duplicated string literals — finding 10, minor, and **not** a §6 violation because they are +not an enum declaration. + +`InjectionPoint` and `EvidenceSignal` are camelCase (`dbErrorString`), which looks like a violation of +§6's snake_case sentence. It is not: neither is one of the six frozen enums, both come from the +Fingerprint Specification's own camelCase text, and `CONTRACT.md` line 443–446 records the reconciliation +explicitly. Consistent across all three sources. **No finding.** + +--- + +## 6. GOLDENS — do they regenerate themselves? + +**No. PASS.** + +- `testdata/fingerprint_corpus/*.json` are committed data files carrying `hashed_fields` and + `expected_digest`. `TestCorpusFixturesProduceTheirDocumentedDigest` compares the implementation's field + list **and** digest against them; nothing writes back. +- There is **no `-update` flag, no `os.WriteFile`, no golden-regeneration path** anywhere in + `fingerprint_test.go`. The file contains an explicit, correct block (lines 429–444) forbidding one. +- `strictUnmarshal` uses `DisallowUnknownFields`, so a misspelled fixture key fails loudly instead of + silently defaulting a hashed field to `""` and locking in a wrong digest. That is the right paranoia. +- Independent check by this critic: each fixture's `expected_digest` was recomputed **outside Go**, in + Python, as `sha256(U+001F.join(hashed_fields))`. All seven matched, and all seven matched the Go + output in §4.2. So the fixtures are internally consistent and the Go code agrees with them. + +**The reservation:** that check proves `expected_digest = H(hashed_fields)`. It does **not** prove +`hashed_fields` is what the *written specification* produces from `input`. That is exactly what R.16 +exists to prove, and finding 1 shows it currently cannot. + +--- + +## 7. NUMBERED GAPS + +### BLOCKER 1 — `normalized_match` is defined only in Go, so R.16's mandated independent oracle cannot reproduce the SAST goldens + +`plan/40-record-and-storage.md` defines the SAST tier's hardest field in four clauses: + +``` +normalized_match := strip comments; collapse whitespace runs to a single space; + replace string/numeric literals with /; + replace local identifiers with positional $1..$N in first-occurrence order. +``` + +`fingerprint.go`'s `NormalizeMatch` implements those four **plus at least nine rules the specification +never states**: a ~200-entry cross-language reserved-word list held verbatim (lines 1120–1172); +identifiers after `.`, `->` or `::` held verbatim; identifiers before `::` held verbatim; identifiers +before `(` held verbatim (lines 1051–1068); CRLF folding; which comment syntaxes exist (`//`, `#`, +`/*…*/`); which string delimiters exist (`"`, `'`, `` ` ``) and where backslash escapes apply; the +number-token grammar (`0xFF`, `1_000`, `3.14f`, `1e-9` are each one token); and a final trim. + +This critic re-implemented the **specification text only** in Python — no reserved-word list, no +selector/callee/scope exceptions, everything else deliberately matched to the Go choices so the +divergence is attributable to the identifier rule alone — and rehashed the committed fixtures: + +``` +sast-01-go-sql-string-concat + spec-text normalized_match : '$1, $2 := $3.$4( + $5 + )' + committed normalized_match : '$1, $2 := $3.Query( + $4 + )' + spec-text digest : 55e27b07806178f7afd55b9178523ac45dd92b5e9aefcb00b2011f6d4cea2eed + committed expected_digest : 13c60ccf8ec84530e075db4190005b03bc87050210065398fd48088683bb1aa6 + *** DIVERGES *** + +sast-02-python-shell-command + spec-text normalized_match : '$1.$2( + $3)' + committed normalized_match : '$1.system( + $2)' + spec-text digest : 3ef752ff8836e5edacf8a8f8e38b895d41d816183147604d2445c884db7d09c0 + committed expected_digest : d2239ee40c3f9da1cc9db3417962e3794b0c5baca0409015ff331a0aea095242 + *** DIVERGES *** +``` + +**Why this is a blocker and not a nit.** R.16's packet requires an oracle that "re-implements the +algorithm text above from scratch, **not** by importing `internal/record/fingerprint.go`, so the oracle +is independent". Any such oracle produces the digests on the left. R.16 then has exactly two ways to go +green, and the plan forbids both: read `fingerprint.go` (destroys independence) or copy +`expected_digest` (the test file's own header at lines 31–33 forbids it, correctly). The spine-mandated +conformance gate is therefore unsatisfiable as written. + +It is also the S6 failure itself, one level up. S6's rule is "**One fingerprint algorithm, defined +once, in the record.**" Today the authoritative definition of `normalized_match` is Go source. A second +producer — the plan's whole premise is that there will be more than one — implementing from +`plan/40-record-and-storage.md` will emit different digests and nothing will surface it. The header +comment of `fingerprint_test.go` asserts the `hashed_fields` "were derived BY HAND from the algorithm +text"; for the two SAST fixtures that derivation must have used knowledge not present in the algorithm +text, and this critic could not reproduce it. **Claim not supported by the artifact.** + +**Proposed fix (orchestrator ruling, not a code change).** Amend the Fingerprint Specification to carry +the full normalization: the reserved-word list verbatim, the selector/callee/scope-resolution rules, the +comment syntaxes, the string delimiters and escape handling, the number-token grammar, CRLF folding and +the trim — and state that any edit to that list is an `anvil-fp/v2` event, which `fingerprint.go` line +1118 already says. Alternatively rule the other way and re-route R.2 to the literal four-clause text, +accepting the loss of discriminating power the implementer argues against at lines 900–907. Either +ruling is defensible; leaving it undecided is not, because R.16 will hit it and the cheapest thing R.16 +can do is quietly weaken its own oracle. + +--- + +### BLOCKER 2 — the DAST tier's only specified normalization is not implemented, and no fixture covers it + +The specification defines the DAST tier's route field as a **derived** value: + +``` +route_template := numeric/UUID/hash path segments replaced with a placeholder token. +``` + +`CanonicalRouteTemplate` (fingerprint.go:702–720) does not do this. It strips a query string or +fragment, collapses slash runs, adds a leading slash and trims a trailing one. It performs **zero** +segment templating. `DastInput`'s doc comment (line 575) reassigns the work to the caller — +"RouteTemplate is the path with volatile segments templated" — which the plan does not say. + +Consequence, and it is precisely S6's named failure: two producers observing the same defect at +`/api/users/12345/orders` will emit `/api/users/12345/orders`, `/api/users/{id}/orders` and +`/api/users/:id/orders` respectively. Three digests, one defect, no error, regression matching silently +dead. Worse than the SAST case, because the DAST tier is the one that earns "verified fixed" under S7 — +a DAST reproduction that cannot be matched to its prior finding cannot prove a fix. + +All three DAST fixtures (`dast-01`, `dast-02`, `dast-03`) arrive **already templated** +(`/api/v1/users/{id}/orders`, `/search`, `/files/{path}`), so the corpus cannot detect this. There is no +test anywhere that feeds a concrete numeric or UUID segment. + +**Proposed fix:** implement the templating in `CanonicalRouteTemplate` — replace segments matching +all-digits, a UUID, and a long hex/base32/base64 run with a single frozen placeholder token, with the +patterns written into the specification — and add a DAST fixture whose input carries +`/api/v1/users/12345/orders` and a UUID segment, with a mutation proving a different id yields the same +digest. If the orchestrator instead rules that templating belongs to the DAST producer (area D), then +`plan/40-record-and-storage.md` must say so, D's packet must own it with a named test, and the +specification's `route_template :=` line must be struck — otherwise two areas each believe the other +does it. + +--- + +### MAJOR 3 — `primaryLocationLineHash` is required by the contract, assigned to the fingerprint engine by the plan, and implemented by nobody + +- `plan/40-record-and-storage.md` line 653 lists `result.partialFingerprints["primaryLocationLineHash"]` + as **"required when a physical location exists"**, producer column **"fingerprint engine"**, consumer + column "GitHub upload path only (R.14)". +- `contract.go:2328` makes it a hard `Validate()` failure: + `partialFingerprints["primaryLocationLineHash"] is required when a physical code location exists`. +- `fingerprint.go:77–83` declines to implement it and reassigns production to R.14: *"It is not an + anvil-fp/v1 tier … it lives under `PartialFingerprintPrimaryLocationLineHash` and is owned by the + GitHub projection (R.14)."* + +The technical argument for keeping a line-dependent hash out of this file is good. But the plan's +producer column says fingerprint engine, and R.2 unilaterally moved it. Net effect **today**: no code in +the tree can construct a SARIF result with a physical code location that passes `Validate()`. R.4 (the +store) and every SAST producer will hit this. + +**Proposed fix:** an explicit ruling on the owner, plus whichever of these follows — R.2 ships a +separate `PrimaryLocationLineHash(...)` helper in its own file (identity untouched, no line number one +import from `Digest`), or R.14's packet is amended to own it and `plan/40`'s producer column is corrected +to say so. Do not leave `Validate()` enforcing a field with no producer. + +--- + +### MAJOR 4 — the ordinal group key omits `enclosing_symbol_path`, so an unrelated edit elsewhere in the file churns a finding's identity + +`AssignSastOrdinals` (fingerprint.go:824–833) groups on +`TargetID | RuleIDVersioned | CanonicalRepoRelPath | NormalizeMatch(Snippet)` — faithfully following the +specification's "index among all matches of the same `rule_id` in the same `repo_relpath` whose +`normalized_match` is IDENTICAL". `enclosing_symbol_path` is **hashed** but is **not** in the group key. + +Failing scenario: + +1. `store.go` contains `func A() { … exec(x) … }` at line 10 and `func B() { … exec(y) … }` at line 50. + Both normalise to `exec($1)`. +2. They are already distinguishable — `enclosing_symbol_path` differs — so they cannot collide. + Nevertheless they land in one ordinal group: A gets 0, B gets 1. +3. Someone deletes `func A` for unrelated reasons. +4. B's ordinal drops to 1 → 0. **B's digest changes.** B is reported resolved and re-opened as new. + `first_seen_at` resets, age-based ranking resets, any suppression keyed on B's fingerprint silently + stops applying, and a `handoff` row keyed on the old digest is orphaned. + +Adding `enclosing_symbol_path` to the group key removes the churn entirely and loses nothing: within one +symbol, ordinal still breaks the duplicate-call-site collision that motivated it (`fingerprint.go:37–41`). +`TestAssignSastOrdinals` sets `EnclosingSymbolPath: "sym"` on every candidate, so the case is untested. +This requires a one-line specification amendment as well as the code change, since the group key is +spec text. + +--- + +### MAJOR 5 — normalization is aggressive enough that two different sinks share a `normalized_match`, and identity then rests on the least stable field + +The implementer's own test documents it (`fingerprint_test.go:1063–1076`): `exec(cmd)` and `exec(other)` +both normalise to `exec($1)`, because `exec` is a preserved callee and the argument is abstracted. + +That is intended for renames. The consequence is not: two *semantically different* call sites — one +executing `cmd`, one executing `other` — are distinguished **only by `ordinal`**, which is derived from +line order. Swap the two lines and the two findings **swap identities**. Not churn — misattribution. +A triage verdict, a suppression, a `handoff` row and a "verified fixed" claim all transfer to the wrong +finding, and every field of both records still looks internally consistent. + +`fingerprint.go:784–790` records the ordinal's instability as a known limitation and points at +research/07's matching cascade to recover. That cascade does not help here: both findings are live, both +digests exist, and the cascade has nothing to disambiguate on. No test asserts against the swap. + +**Proposed fix:** either narrow abstraction so an argument identifier's *role* survives (e.g. abstract +only identifiers appearing more than once, or keep the first argument of a preserved callee), or accept +the risk explicitly in the specification and require the store to alert when two findings in one +(rule, path, normalized) group exchange ordinals between scans. Silent acceptance is the one option that +should not survive. + +--- + +### MAJOR 6 — a ruleset version bump re-mints every SAST and DAST fingerprint, with no bound and no migration + +`rule_id_versioned` is hashed in both the SAST (fingerprint.go:387–395) and DAST (638–647) tiers, per the +specification. `fingerprint_test.go:529` pins `"different rule version"` as *required* to change the +digest, and `fingerprint.go:319–321` argues the case: "a rule that changed what it matches is a different +rule." + +The argument is sound for a rule whose semantics changed. It is applied, however, to a token that moves +on the **ruleset's** release cadence — the fixtures use `opengrep…@2026.07.1` and +`nuclei:…@2026.07.1`. opengrep and nuclei-templates ship routinely; S7 already requires nuclei-templates +to be "pinned by commit SHA and diffed before promotion", i.e. bumped deliberately and often. Every such +bump resolves and re-opens **the entire SAST and DAST finding population at once**, resetting +`first_seen_at`, resetting age-based ranking, dropping every fingerprint-keyed suppression, and orphaning +every `handoff` row — with no error anywhere. The plan's only migration protocol (dual-write `v1` and +`v2` for one retention cycle) covers **algorithm** version changes, not rule-version changes, so nothing +absorbs this. + +R.3's remit is explicitly "exclusion of volatile fields". A token that turns over on a weekly-to-monthly +cadence, driven by an upstream project rather than by the scanned code, is volatile. + +**Proposed fix (one of):** hash the rule id **without** its version and carry the version as an unhashed +result attribute — the rule's *identity* is stable even when its ruleset ships; or extend the dual-write +migration protocol to cover ruleset bumps, with the store matching `old_rule_version OR new_rule_version` +for one cycle; or record an explicit accepted-risk ruling in `plan/40-record-and-storage.md` stating that +ruleset bumps mass-reset finding identity and naming what compensates. The current position — hash it, +pin the behaviour in a test, say nothing about the consequence — is the one that fails silently. + +--- + +### MINOR 7 — a C0 control byte in a snippet makes a finding unfingerprintable instead of being normalized away + +`NormalizeMatch` collapses whitespace and drops comments, but any other character falls through to +`out = append(out, c)` verbatim (fingerprint.go:1070–1073). A C0 control that is not +`unicode.IsSpace` — `NUL`, `BEL`, `ESC`, `\x1f` itself — therefore survives normalization, and `Digest` +then rejects the whole field (fingerprint.go:262–271). `Sast` returns an error and the finding **cannot +be hashed at all**. + +`Digest`'s rejection is right — a field carrying `U+001F` would move a field boundary. The gap is that +`NormalizeMatch` does not guarantee its output is hashable, so the failure lands on the producer as a +hard error. If any caller drops errored findings (the natural thing to do in a scan loop), a real +vulnerability on a line containing a stray control byte becomes invisible. S7's threat model makes this +worth closing: an attacker who can land a byte in a source file, a generated file or a vendored blob can +make the finding on that line unreportable. + +**Proposed fix:** have `NormalizeMatch` strip C0 and DEL the way it strips comments, and add a test that +`Sast` succeeds on a snippet containing `\x00` and `\x1f`. Keep `Digest`'s rejection as the backstop. + +*Confidence note: confirmed by reading the code path, **not executed** — this critic's packet forbids +adding a test file. `unicode.IsSpace` covers `\t \n \v \f \r`, space, U+0085 and U+00A0, and not `\x00` +or `\x07`, so those reach the `default` branch.* + +--- + +### MINOR 8 — `CanonicalRepoRelPath`'s stated goal is not achieved, and its backslash rewrite can merge two distinct POSIX files + +The doc comment (fingerprint.go:669–671) claims backslash conversion exists "so a Windows producer and a +Linux producer scanning the same repository agree". Three ways they still do not: + +- **(a) Case.** The function deliberately does not case-fold (lines 674–677), with a correct reason. + But Windows filesystems are case-insensitive, so a Windows producer may legitimately report + `Internal/API/Store.go` where a Linux producer reports `internal/api/store.go`. Different digests. + `TestCanonicalRepoRelPath` pins case preservation as intended, which pins the divergence too. +- **(b) Backslash is a legal POSIX filename character.** A real Linux file named `a\b.go` canonicalises + onto `a/b.go` and collides with the genuinely different file at that path. Rare, but it is a + *collision* — two findings, one identity, one lost on upsert — which the whole ordinal mechanism + exists to prevent. +- **(c) Unicode form.** No NFC/NFD normalization anywhere. A path or symbol containing a precomposed + vs decomposed accented character hashes differently, which is the macOS-checkout case. + +**Proposed fix:** narrow the doc claim to what is true ("separator form only"), or add NFC normalization +and decide the case question explicitly. (b) is cheap to fix — reject a path containing a backslash on +POSIX rather than rewriting it — but needs a ruling, since the Windows-producer fixture depends on the +rewrite. + +--- + +### MINOR 9 — five undocumented canonicalizations sit between fixture input and hashed field + +Each is defensible; none appears in `plan/40-record-and-storage.md`'s algorithm text, and each is another +way R.16's oracle will diverge (same root cause as finding 1, listed separately because writing them +down is nearly free): + +| Canonicalization | Where | In the spec text? | +|---|---|---| +| HTTP method upper-cased and trimmed | fingerprint.go:617–623 | no | +| Route query string / fragment stripped, slashes collapsed, leading `/` added, trailing `/` trimmed | 702–720 | no | +| purl scheme and type lower-cased | 736–739, 764–769 | no | +| purl subpath / qualifiers / version stripped | 750–758 | version yes, `?`/`#` no | +| host package manager lower-cased and trimmed; `:` forbidden in it | 472–474, 529–535 | no | +| repo path: backslashes, slash runs, leading `./` and `/`, trailing `/` | 678–689 | no | + +The corpus *tests* all of them via mutations, which is good, and pins them against future drift. It just +does not make them reproducible from the specification. + +--- + +### MINOR 10 — two of four tiers take their hashed discriminator from a private literal, two from the contract enum + +`tierTokenSast = "sast"` and `tierTokenDast = "dast"` (fingerprint.go:195–201) duplicate the values of +`DetectorKindSast` and `DetectorKindDast`, while the SCA and host tiers hash `string(kind)` taken from +the contract enum (line 501). The stated reason is to avoid confusing the tier token with the *evidence +class* — but `DetectorKind` is not the evidence class; `EvidenceClass` is, and it is a separate type with +`sast_reachable`/`sast_static_only`. `DetectorKind` already means exactly what the tier token means. + +Not a §6 violation (no enum is declared, no shared vocabulary is redefined) and not a bug today — +`TestTiersUseTheirSpecifiedDiscriminator` asserts both spellings. It is a second copy of a hashed +literal, in the one file whose entire premise is that a second copy of a hashed value is how +`anvil-fp/v1` got two meanings in the first place. + +**Proposed fix:** use `string(DetectorKindSast)` / `string(DetectorKindDast)` and keep the comment +explaining why the tier token is not the evidence class. + +--- + +### MINOR 11 — an R.2 decision that `CONTRACT.md` explicitly assigns to R.2 was never recorded + +`CONTRACT.md` deviation 2 (lines 429–432): *"`partialFingerprints["regionSha256"]` reserved … **R.2 +decides whether to populate it**, and R.2 may strike it if the algorithm has no use for it."* + +`fingerprint.go` never mentions `regionSha256`. The decision is neither made nor struck; it has silently +become nobody's. `research/24` names `fingerprint.region_sha256` non-negotiable for the coding-agent +handoff, so R.9/R.10 will meet it again with no ruling to consult. + +**Proposed fix:** R.2 (or the orchestrator) records one sentence in `CONTRACT.md` — populated by X, or +struck — before R.4. + +--- + +## 8. WHAT THIS PASS DID *NOT* CHECK + +Stated so the gap is visible rather than assumed covered: + +- **No cross-family review was obtained.** See §0. +- `contract.go` is ~104 KB. Its **six frozen enums, every S6 field, and the validation rules touching + them** were read closely. Its SARIF projection, JSON round-trip and the remainder of `Validate()` were + not audited line by line; `contract_test.go`'s 129 passing assertions were re-run, not re-derived. +- The **schema** was compared for enum literals and S6 field presence. It was not validated against a + JSON Schema meta-schema, and no instance document was validated against it by this pass. +- S6's **ordering / budget-reservation** requirement is out of R.1's and R.2's scope and was not checked + anywhere. +- Finding 7 is reasoned from the code path, not executed — this packet forbids adding a test file. +- The claim that the corpus `hashed_fields` were "derived by hand from the algorithm text" is + **unverifiable from the artifact** and, for the two SAST fixtures, is contradicted by finding 1's + experiment. Treat the goldens as pinning `fingerprint.go`'s behaviour, which is still worth having. + +--- + +## 9. WHAT R.4 MAY AND MAY NOT ASSUME + +**May assume, now proved:** the six frozen enums are correct and singly-sourced across Go, schema and +prose; every S6 field exists; digests are 64 lowercase hex, never truncated, deterministic across +processes and platforms; the package version, line, column, host, port, scheme, payload and timestamps +are all excluded by construction and by test; `UNIQUE (target_id, fingerprint)` is safe to build on for +the SCA and host tiers. + +**May not assume:** that a second producer implementing from `plan/40-record-and-storage.md` will emit +the same SAST or DAST digest (findings 1, 2); that `route_template` is normalized by anything +(finding 2); that a SAST finding's identity survives an unrelated deletion in the same file (finding 4) +or a ruleset bump (finding 6); that `primaryLocationLineHash` has a producer (finding 3). + +Findings 1 and 2 are blockers. Per the packet's stop condition, R.1 or R.2 — or, more likely here, the +Fingerprint Specification section of `plan/40-record-and-storage.md` — must be rerouted and re-reviewed +to all-PASS before R.4 starts. diff --git a/internal/record/FINGERPRINT-SPEC.md b/internal/record/FINGERPRINT-SPEC.md new file mode 100644 index 0000000..df5c43b --- /dev/null +++ b/internal/record/FINGERPRINT-SPEC.md @@ -0,0 +1,720 @@ +# FINGERPRINT-SPEC.md — the authoritative definition of `anvil-fp/v1` + +**Status:** normative. **Version:** `anvil-fp/v1`. **Last amended:** 2026-08-08. + +This document is the single definition of Anvil's finding-identity algorithm. It is deliberately +**in-tree**: `plan/` is gitignored, so a second producer working from a clone of this repository must +be able to read the complete algorithm here and nowhere else, and emit byte-identical digests. + +`plan/40-record-and-storage.md`'s "Fingerprint Specification" section remains the record of *why* the +research-branch conflict was resolved the way it was. It is a **summary**, not the definition. +`internal/record/CRITIQUE-01.md` (finding 1) proved the summary insufficient by re-implementing its +four-clause `normalized_match` text in Python and obtaining `55e27b07…` where the committed golden for +`sast-01-go-sql-string-concat` is `13c60ccf…`. The orchestrator ruled on 2026-08-08 that the +implementation was right and the specification incomplete, and that the fix was to write the +specification down completely. This file is that ruling discharged. **Where this document and +`plan/40-record-and-storage.md` disagree, this document governs.** + +`internal/record/fingerprint.go` implements it. `internal/record/fingerprint_spec_test.go` asserts that +the machine-checkable parts of this document — the reserved-word list and the algorithm constants — +are exactly the ones in the code, so the two cannot drift apart silently. + +--- + +## 0. The versioning rule — read before editing anything below + +**Every rule in this document is load-bearing on stored identity.** Changing *any* of it — +adding one reserved word, moving one threshold by one, reordering two normalization steps, altering a +token's spelling — changes digests that are already stored, and a changed digest means a finding is +reported resolved and re-opened as new: `first_seen_at` resets, age-based ranking resets, every +fingerprint-keyed suppression silently stops applying, and every `handoff` row keyed on the old digest +is orphaned. Nothing logs an error when this happens. That is the exact failure `plan/00-SPINE.md` S6 +exists to prevent: *"two producers emitting different hashes means regression matching silently fails +forever."* + +> **Any change to this document's algorithm is an `anvil-fp/v2` event, never a `v1` edit.** + +`fingerprint.go`'s `fingerprintReservedWords` comment says the same thing, and the two must keep saying +it. Shipping `v2` means: bump the algorithm name, dual-write both `v1` and `v2` values into +`finding_fingerprint` for one full retention cycle, match on `v1 OR v2` during that cycle, then retire +`v1`. Do not edit a golden digest to make a test pass. + +The **prose, rationale and examples** in this document may be improved freely. Only the rules may not. + +--- + +## 1. Primitives + +### 1.1 Field separator + +Fields are joined with **U+001F**, the ASCII Unit Separator (`"\x1f"`), in the exact order each tier +lists. Nothing else separates them; there is no prefix, suffix, length prefix, or trailing separator. + +U+001F was chosen over a printable glyph because a printable separator can occur inside a snippet or a +symbol name and silently move a field boundary — hashing `("a‖b", "c")` and `("a", "b‖c")` would +collide. U+001F cannot appear in normalized source text. + +### 1.2 Field guard + +Before joining, **every** field is checked. A field containing any character in `U+0000`–`U+001F` or +`U+007F` (C0 controls and DEL) is **rejected with an error**; the digest is not computed. U+001F is the +case that matters, but the whole control range is rejected, because none of these can legitimately +appear in a canonicalised path, rule id, symbol path, route template, purl, advisory id, or normalized +match — and a newline in one of them means the caller passed raw, uncanonicalised text. + +An empty **field list** is rejected. An empty **field value** is legal wherever the tier says so +(`enclosing_symbol_path`, `param_name`), and is hashed as a zero-length field, which keeps the field +count constant. + +### 1.3 Digest + + DIGEST = lowercase hex-encoded SHA-256 of the UTF-8 bytes of the joined string + +Exactly **64 lowercase hex characters. Never truncated.** Uppercase hex is not a valid digest and must +be rejected rather than folded — a store that accepted both would hold two rows for one finding and +defeat `UNIQUE (target_id, fingerprint)`. + +All string handling is on **UTF-8 bytes**. No Unicode normalization (NFC/NFD) is applied anywhere; see +§9. + +--- + +## 2. The four tiers + +Exactly four tiers exist. Each is a fixed, ordered field list. A wrong field *order* produces a +perfectly valid-looking 64-hex digest that is silently incompatible, so the order below is normative. + +### 2.1 Tier SAST — `evidence_class ∈ {sast_reachable, sast_static_only}` + + sha256( target_id ␟ "sast" ␟ rule_id_versioned ␟ repo_relpath + ␟ enclosing_symbol_path ␟ normalized_match ␟ ordinal ) + +Seven fields. + +| # | Field | Derivation | Empty allowed | +|---|---|---|---| +| 1 | `target_id` | verbatim | no | +| 2 | `"sast"` | the literal string `sast`, for **both** evidence classes | — | +| 3 | `rule_id_versioned` | verbatim, e.g. `opengrep.go.sqli@2026.07.1` | no | +| 4 | `repo_relpath` | §7.1 `CanonicalRepoRelPath` | no (nor after canonicalisation) | +| 5 | `enclosing_symbol_path` | verbatim, e.g. `pkg/mod.py::ClassA.method_b` | **yes** | +| 6 | `normalized_match` | §3, from the raw snippet | no (nor after normalization) | +| 7 | `ordinal` | §4, rendered as a base-10 integer with no padding and no sign | no; negative is rejected | + +Field 2 is the literal `sast` and **not** the evidence class, so a finding upgraded from +`sast_static_only` to `sast_reachable` keeps its identity. + +`enclosing_symbol_path` may be empty because a match in top-level module code, a config file, or a +template has no enclosing symbol, and rejecting those would make them unfingerprintable. + +**Never hashed by this tier:** line number, column number, the literal (non-normalized) snippet text, +`advisory_id`, the evidence class, any timestamp. + +### 2.2 Tiers SCA and HOST — one formula, parameterised + + sha256( target_id ␟ detector_kind ␟ advisory_id ␟ purl_base ␟ locator ) + +Five fields. `detector_kind` is the literal `sca` for repository dependencies and `host` for operating +system packages. It is the only thing keeping a repo dependency and a host package with the same +advisory apart, and it must be present: without it a host finding (`remediable_by_agent=false`) could +upsert over an agent-remediable dependency finding. + +| # | Field | Derivation | +|---|---|---| +| 1 | `target_id` | verbatim | +| 2 | `detector_kind` | the literal `sca` or `host` | +| 3 | `advisory_id` | **verbatim, not case-folded** — GHSA identifiers mix case meaningfully and folding would fork identity against the advisory table | +| 4 | `purl_base` | §7.2 `PurlBase` | +| 5 | `locator` | SCA: `manifest_relpath` through §7.1. Host: `":"`, §7.3 | + +**Never hashed by these tiers:** the version string. Bumping `1.2.3` → `1.2.4` while still inside the +vulnerable range must not mint a new finding; resolution is proved by re-evaluating `advisory_affects`, +never by an identity change. `PurlBase` strips the version defensively even if a caller passes it. + +### 2.3 Tier DAST — `evidence_class = dast_confirmed` + + sha256( target_id ␟ "dast" ␟ rule_id_versioned ␟ http_method ␟ route_template + ␟ injection_point ␟ param_name ␟ evidence_class_detail ) + +Eight fields. + +| # | Field | Derivation | Empty allowed | +|---|---|---|---| +| 1 | `target_id` | verbatim | no | +| 2 | `"dast"` | the literal string `dast` | — | +| 3 | `rule_id_versioned` | verbatim, e.g. `nuclei:CVE-2021-44228@a1b2c3d` | no | +| 4 | `http_method` | §7.4: trimmed, then upper-cased | no | +| 5 | `route_template` | §6 `CanonicalRouteTemplate` — a **derived** value | no (nor after derivation) | +| 6 | `injection_point` | one of `query`, `body`, `header`, `cookie`, `path`; any other value is rejected | no | +| 7 | `param_name` | verbatim. A parameter **name**, never a parameter **value** | **yes** | +| 8 | `evidence_class_detail` | one of `responseStackTrace`, `statusCodeFlip`, `dbErrorString`, `timingSideChannel`, `reflectedPayload`, `other`; any other value is rejected | no | + +Fields 6 and 8 are independent facts — *where* the payload went in versus *how* the defect was +observed. An SQL injection proved by a database error string and one proved by a timing side channel on +the same parameter are different findings with different remediation evidence. + +`param_name` may be empty: a whole-body, raw-request, or path-segment injection has no single named +parameter. + +**Never hashed by this tier:** host, port, scheme, the concrete payload string, any session token, any +timestamp. + +--- + +## 3. `normalized_match` — the complete algorithm + +This section is the one CRITIQUE-01 proved was under-specified. It is stated here in full. A +re-implementation that follows §3.1–§3.6 and the word list in §3.5 reproduces the committed SAST +goldens exactly. + +The algorithm is **one left-to-right pass over Unicode code points**, with no lookbehind beyond the +output buffer and no lookahead beyond skipping spaces. It is a single language-agnostic lexer, not N +parsers: Anvil's SAST tier is an opengrep subprocess that returns text, and the language is not +reliably known at fingerprint time. Determinism, not semantic perfection, is the property identity +needs. + +### 3.1 Preprocessing + +1. Replace every `"\r\n"` with `"\n"`. +2. Replace every remaining `"\r"` with `"\n"`. + +Both replacements are global and are applied in that order. (Order matters only in that step 1 must +precede step 2, or a CRLF would become two newlines.) + +The result is then decoded as a sequence of Unicode code points and scanned. Two output helpers are +used below: + +- **emit(s)** appends the characters of `s` to the output. +- **emitSpace()** appends a single space **only if** the output is non-empty and does not already end + in a space. It never produces two consecutive spaces and never produces a leading space. + +### 3.2 The scan + +At each position, the **first** matching rule below is applied. The rules are ordered; the order is +normative. + +| # | Condition at the current character `c` | Action | +|---|---|---| +| 1 | `c` is whitespace (Unicode `IsSpace`: `\t \n \v \f \r`, space, U+0085, U+00A0, and the Unicode space separators) | consume the whole whitespace run; `emitSpace()` | +| 2 | `c` is `/` and the next character is `/` | consume to the next `\n` (not consuming it) or to end of input; `emitSpace()` | +| 3 | `c` is `#` | consume to the next `\n` (not consuming it) or to end of input; `emitSpace()` | +| 4 | `c` is `/` and the next character is `*` | consume through the closing `*/`; if there is none, consume to end of input; `emitSpace()` | +| 5 | `c` is `"`, `'` or `` ` `` | consume the string literal per §3.3; `emit("")` | +| 6 | `c` is an ASCII digit `0`–`9` | consume the number token per §3.4; `emit("")` | +| 7 | `c` is an identifier-start character | consume the identifier and dispose of it per §3.6 | +| 8 | anything else | append `c` verbatim; advance one character | + +**Identifier-start** is: `_`, `$`, or any character with Unicode general category L (`IsLetter`). +**Identifier-part** is: `_`, `$`, any letter, or any digit (`IsDigit`, i.e. Unicode category Nd — not +only ASCII). + +Note the consequences, which are accepted and stable: + +- `#` is a line comment, so a C/C++ preprocessor directive or a C# `#region` inside a snippet is + dropped. Match snippets rarely contain them. +- `'` is a string delimiter, so a Rust lifetime (`'static`) or a Lisp quote consumes to the next `'`. +- Rule 3 is tested before rule 4 is reachable for `#`, and rule 2 before rule 4 for `//`, so `//*` is a + line comment, not a block comment. + +### 3.3 String literals + +Having consumed the opening delimiter `q` (one of `"`, `'`, `` ` ``), scan forward: + +- If the current character is `\` **and `q` is not** `` ` ``, skip **two** characters (the backslash and + whatever follows it, including a closing delimiter or another backslash) and continue. +- If the current character is `q`, consume it and stop. +- Otherwise consume one character and continue. +- If input ends first, stop. + +Backslash escapes are therefore honoured inside `"` and `'` and **not** inside `` ` `` — matching Go's +raw string literals, where a backslash has no special meaning. + +Exactly one `` is emitted per literal, regardless of its contents or length. No space is emitted +around it. + +### 3.4 Number tokens + +Having seen an ASCII digit, consume characters while **either**: + +- the character is a letter (`IsLetter`), a digit (`IsDigit`), `_`, or `.`; **or** +- the character is `+` or `-` **and** the immediately preceding *input* character was `e` or `E`. + +Stop at the first character satisfying neither. Exactly one `` is emitted. + +This grammar deliberately makes each of the following a **single** token: `0xFF`, `1_000`, `3.14f`, +`1e-9`, `1E+10`, `100L`, `0b1010`, `12.5e-3`. It also means `1.2.3` is one token, and that a number +immediately followed by a letter (`42px`) is one token. A number can never *start* a token that begins +with a letter, because rule 7 would have matched first. + +### 3.5 Identifier disposition — the rules the summary omitted + +Having consumed an identifier `word`, apply the **first** matching clause: + +| Clause | Condition | Emit | Why | +|---|---|---|---| +| **(a)** | `word` is in the reserved-word list below | `word` verbatim | A keyword is not a name a refactor renames. Abstracting `for`, `return` or `int` would erase the statement's shape and merge structurally different code. | +| **(b)** | the output so far, **ignoring trailing spaces**, ends in `.`, `->`, or `::` | `word` verbatim | `word` is a member, field, method or qualified name — API surface, not churn. Abstracting it would normalise `request.getParameter(userInput)` and `config.getName(key)` to the same string, destroying nearly all discriminating power and pushing the whole burden of distinguishing findings onto `ordinal`, the least stable field in the tier. | +| **(c)** | the next non-space **input** characters are `::` | `word` verbatim | The **left** operand of `::` is a namespace or type name in every language that has the operator (C++, Rust, PHP, Ruby) — never a local variable. Abstracting it would violate "replace local identifiers" outright **and** collapse `Ns::Helper(v)` and `Other::Helper(v)` — two calls into two different namespaces — onto one digest. | +| **(d)** | the next non-space **input** character is `(` | `word` verbatim | `word` is a callee name: the sink the rule actually matched. `exec(x)` and `spawn(x)` are different findings. | +| **(e)** | otherwise | `$N` | `word` is a local. `N` counts **distinct spellings** in first-occurrence order starting at 1, and every later occurrence of the same spelling maps to the same `$N`. This is what survives a rename. | + +Clause (b) asks about the **output**; clauses (c) and (d) ask about the **input**. That asymmetry is +deliberate: (b) looks at what has already been decided (a `.` survives step 8 verbatim, so it is +visible in the output), while (c) and (d) look ahead at text not yet scanned. + +**Why (b) treats `.` and `->` differently from (c)'s `::`.** The *left* operand of `.` or `->` is +usually a receiver bound to a local (`db.Query(q)`, `p->Field`), so it **is** abstracted — a lexer +cannot tell a Go package qualifier from a receiver variable, and guessing wrong in the other direction +would fork the digest of unchanged code, which is the worse failure. The *left* operand of `::` is +never a local, so it is preserved. Both operands to the *right* of any of the three are preserved by +(b). + +The metavariable counter and the spelling→`$N` map are **per call** to the normalizer: they start empty +for every snippet. `$1` in one finding has no relationship to `$1` in another. + +The map must never be iterated. Numbering comes from a counter advanced in source order. (In Go, map +iteration order is randomised per process; a re-implementation that numbered by iterating a hash map +would produce a stable-but-wrong order within a run and a different one in the next, which is the +cross-process nondeterminism `TestCorpusDigestsAreStableAcrossProcesses` exists to catch.) + +#### The reserved-word list, verbatim + +This is a **union** across the languages Anvil's SAST tier covers (Go, Java, C#, C/C++, +JavaScript/TypeScript, Python, Ruby, PHP): keywords, literal keywords and self-references, and +primitive/built-in type names. It is a union on purpose — a per-language list would require knowing the +language at fingerprint time, which the opengrep subprocess boundary does not reliably give us, and a +wrong language guess would change the digest of unchanged code. The union is a fixed, deterministic +function of the token text alone. + +The cost is accepted: an identifier named `class` in a language where `class` is not reserved is +preserved rather than abstracted. That is stable under re-scan, which is the property that matters. + +Matching is **exact and case-sensitive**. `None` is in the list; `NONE` is not. Whitespace-separated, +sorted in byte order (uppercase before lowercase), **193 entries**: + + +```text +False NULL None True abstract and any as +assert async await base begin bigint bool boolean +break byte case catch chan char checked class +clone cls complex128 complex64 const constexpr continue debugger +decimal declare def default defer del delete do +double echo elif else elseif elsif end endforeach +endif endwhile ensure enum error except exit explicit +export extends extern fallthrough false final finally float +float32 float64 fn for foreach friend from func +function global go goto if implements implicit import +in include include_once inline instanceof insteadof int int16 +int32 int64 int8 interface internal iota is keyof +lambda let lock long match module mutable namespace +native never new nil none nonlocal not null +nullptr object operator or out override package params +pass print private protected public raise range readonly +redo ref register require require_once require_relative rescue retry +return rune sbyte sealed select self short signed +sizeof stackalloc static strictfp string struct super switch +symbol synchronized template this throw throws trait transient +true try type typedef typeof uint uint16 uint32 +uint64 uint8 uintptr ulong unchecked undefined union unknown +unless unsafe unsigned until use ushort using var +virtual void volatile wchar_t when while with xor +yield +``` + + +`fingerprint_spec_test.go` asserts this block is exactly the set in `fingerprint.go`, is sorted, and +has no duplicates. + +### 3.6 Final trim + +The output has leading and trailing **spaces** removed (`TrimSpace` over the whole result). Because +`emitSpace()` never emits doubled spaces and no other rule emits whitespace, the result contains no +`\t`, `\n`, `\r`, and no run of two spaces — which is what lets it pass §1.2's field guard. + +A snippet that normalises to the **empty string** (comments and whitespace only) is **rejected**: it +carries no identity. + +### 3.7 Worked examples + +Each of these is pinned by a test. + +| Input | Output | +|---|---| +| `rows, err := db.Query("SELECT * FROM users WHERE name = '" + name + "'")` | `$1, $2 := $3.Query( + $4 + )` | +| `os.system("rm -rf " + path)` | `$1.system( + $2)` | +| `a = b + a + b` | `$1 = $2 + $1 + $2` | +| `rows := db.Query(userInput)` | `$1 := $2.Query($3)` | +| `for i := range items { return nil }` | `for $1 := range $2 { return nil }` | +| `value = compute(x) # trailing note` | `$1 = compute($2)` | +| `a = 0xFF + 1_000 + 3.14f + 1e-9` | `$1 = + + + ` | +| `s = "a \" b" + t` | `$1 = + $2` | +| `p->Field = Ns::Helper(v)` | `$1->Field = Ns::Helper($2)` | +| `std::vector v = Foo::Bar::make(x)` | `std::vector $1 = Foo::Bar::make($2)` | +| `Ns::Helper(v)` | `Ns::Helper($1)` | +| `Other::Helper(v)` | `Other::Helper($1)` | +| `totalCount := len(itemsList)` | `$1 := len($2)` | +| `a\n/* one\n two */\nb` | `$1 $2` | +| ` // nothing here \n\n` | *(empty — rejected)* | + +Trace of the first row, to remove all doubt: + +1. `rows` — identifier, not reserved; output is empty so (b) fails; next non-space is `,` so (c) and + (d) fail → `$1`. +2. `,` → verbatim. Space → one space. +3. `err` → `$2`. Space, `:`, `=` → verbatim (`:=` is two applications of rule 8). Space. +4. `db` — next non-space is `.`, not `::` or `(` → `$3`. +5. `.` → verbatim. +6. `Query` — output ends in `.` → clause (b) → `Query` verbatim. (Clause (d) would also have fired.) +7. `(` → verbatim. +8. `"SELECT * FROM users WHERE name = '"` — a double-quoted string; the `'` inside is ordinary + content → ``. +9. ` + ` → space, `+`, space. `name` → `$4` (a new spelling). +10. ` + ` then `"'"` → ``, then `)` → verbatim. + +--- + +## 4. `ordinal` and its grouping key + +`ordinal` is the **0-based index of this match among all matches of the same rule in the same file +whose `normalized_match` is identical**. It exists because without it two identical macro-expanded or +generated call sites in one file hash identically and the second finding is **lost** on upsert against +`UNIQUE (target_id, fingerprint)`. Losing a finding is worse than churning one. + +**Grouping key** — four components, joined with U+001F for comparison purposes only (never hashed as a +group): + + target_id ␟ rule_id_versioned ␟ CanonicalRepoRelPath(repo_relpath) ␟ normalized_match + +`target_id` is included because the specification's grouping is implicitly per-target; passing two +targets' candidates in one batch would otherwise cross-index them. + +**Ordering within a group** — ascending by, in order: + +1. source line number, +2. source column number, +3. the candidate's original index in the batch (a stable tiebreak). + +Line and column are used **only** for this ordering. They are never hashed and never reach the digest. +The sort must be stable. + +**Known limitation, inherited from the specification rather than chosen here.** Inserting a third +identical call site above two existing ones shifts their ordinals and therefore their digests. The +alternative — dropping the ordinal — silently loses one of two findings on upsert, which is worse. +research/07 §3's matching cascade (exact hit, then rule+path+line_hash, then rule+symbol_hash) is what +recovers identity in that case; the fingerprint alone cannot. CRITIQUE-01 findings 4 and 5 raise two +sharper forms of this and are **not ruled on** — see §10. + +--- + +## 5. `repo_relpath`, `purl_base`, `locator`, `http_method` + +*(§6 covers `route_template` separately; it is the longest.)* + +### 5.1 `CanonicalRepoRelPath` — see §7.1 +### 5.2 `PurlBase` — see §7.2 +### 5.3 host locator — see §7.3 +### 5.4 `http_method` — see §7.4 + +--- + +## 6. `route_template` — a DERIVED value + +The specification has always defined this field as derived: *"numeric/UUID/hash path segments replaced +with a placeholder token."* CRITIQUE-01 finding 2 proved the derivation was not implemented, and that +no fixture could detect it because all three DAST fixtures arrived pre-templated. + +**Ruling (2026-08-08): area 40 owns the fingerprint, so area 40 canonicalises.** A DAST producer emits +whatever route it observed — concrete or already templated in its own syntax. `CanonicalRouteTemplate` +derives the hashed template. This keeps **one owner**. If templating were the producer's job, two +producers seeing one defect at `/api/users/12345/orders` would emit `/api/users/12345/orders`, +`/api/users/{id}/orders` and `/api/users/:id/orders` — three digests, one defect, no error, regression +matching silently dead. This matters more than the SAST case because the DAST tier is what earns +"verified fixed" under `plan/00-SPINE.md` S7, and a reproduction that cannot be matched to its prior +finding cannot prove a fix. + +### 6.1 The placeholder token + + + +One frozen token for every volatile segment class. Angle brackets are chosen for the same reason +`` uses them: RFC 3986 excludes `<` and `>` from every production a path segment can use, so they +must be percent-encoded to appear in a real URL, and a literal segment therefore cannot collide with +the placeholder. `` is also itself recognised as an already-templated segment (§6.3, rule P), +which makes `CanonicalRouteTemplate` **idempotent** — a record read out of the store and +re-fingerprinted keeps its identity. + +### 6.2 Steps, in order + +1. If the input contains `?` or `#`, drop everything from the **first** occurrence of either. A query + string or fragment in a "template" carries concrete values — exactly what templating removes — and + `injection_point` plus `param_name` already record which query parameter was targeted. +2. Replace every `\` with `/`. +3. While the string contains `//`, replace every `//` with `/`. +4. If the string is now empty, the result is empty (and the DAST tier rejects it). +5. If the string does not start with `/`, prepend `/`. +6. If the string is longer than one character, remove a single trailing `/`. +7. If the string is exactly `/`, the result is `/`. +8. Otherwise split the string after the leading `/` on `/`, and replace every **volatile** segment + (§6.3) with ``. Re-join with `/` and prepend `/`. + +Case is preserved on non-volatile segments: URL paths are case-sensitive and `/Search` is a different +route from `/search` on most servers. The UUID and hex predicates in §6.3 are themselves +case-insensitive, because the same identifier rendered in upper hex is the same identifier. + +Percent-encoding is **not** decoded. Decoding could introduce a `/` and change the segment structure, +and a producer that percent-encodes a whole segment has emitted a different route. + +### 6.3 Which segments are volatile + +A segment is volatile if **any** of the following holds. Rules are evaluated in this order; the outcome +is the same token either way, but the order makes each rule independently testable. An empty segment is +never volatile. + +| Rule | Name | Predicate | +|---|---|---| +| **P** | already templated | length ≥ 2 **and** (starts `{` and ends `}`) **or** (starts `<` and ends `>`) **or** starts `:` | +| **N** | numeric | non-empty and every character is an ASCII digit `0`–`9` | +| **U** | UUID | length exactly 36; `-` at 0-based offsets 8, 13, 18 and 23; every other character an ASCII hex digit (`0-9A-Fa-f`) | +| **H** | long hex | length ≥ **16** and every character an ASCII hex digit | +| **O** | long opaque | length ≥ **20**, every character ASCII alphanumeric (`A-Za-z0-9`), containing **at least one digit and at least one letter** | + +**Rule P** accepts all three placeholder syntaxes in common use — OpenAPI/ASP.NET `{id}`, +Express/Rails/Sinatra `:id`, Flask/Werkzeug `` (including converter prefixes like +``) — and normalises them onto the same token as a concrete segment. It also ignores the +placeholder's **name**: `{userId}` and `{id}` are the same token. This is mandatory, not cosmetic: a +DAST crawler, an OpenAPI document checked into the repo, and a route table exported from a framework +will disagree about which syntax and which name to use for the same route, and that disagreement must +not fork identity. + +### 6.4 The thresholds, and why they are conservative + +**The governing asymmetry.** Over-templating merges two genuinely distinct routes into one identity and +loses a finding on upsert against `UNIQUE (target_id, fingerprint)` — silently. Under-templating only +leaves a volatile route un-merged, which the DAST producer can still repair by emitting `{id}` itself. +Under-templating is the recoverable direction, so both thresholds sit well above any plausible +human-authored path segment. + +**`routeHexSegmentMinLen = 16`** (rule H). The hex alphabet's letters are only `a`–`f`; a 16-character +English word drawn from `{a,b,c,d,e,f}` plus digits does not exist — the longest such words +(`defaced`, `cabbage`) are seven letters. Meanwhile every hash form Anvil will meet in a URL clears it: +MD5 is 32, SHA-1 is 40, SHA-256 is 64, a dash-free UUID is 32. A **short git object id (7–12 +characters) is deliberately NOT templated**, because 7 hex characters is also a plausible slug. + +**`routeOpaqueSegmentMinLen = 20`** (rule O). Measured against the longest plausible single-word route +segments: `recommendations` (15), `misrepresentation` (17), `internationalization` (20). None contains +a digit — which is why the **digit requirement carries most of the safety here**, not the length alone. +Real opaque tokens clear the bar comfortably: a base64url session token is 22+ characters and a base32 +token is 26+. + +Rule O's three restrictions each buy something specific, and dropping any of them over-templates: + +- **Alphanumeric only** (no `-`, `_`, `.`) keeps slugs out. `release-notes-2026-08` is 21 characters + and carries a digit; it is route structure, not an identifier, and merging every dated release note + onto one digest is exactly the over-templating failure. The cost is accepted: a **base64url token + containing `-` or `_` is left un-templated**. +- **A digit is required**, which excludes the long all-letter words that do reach 20 characters. +- **A letter is required**, so a purely numeric run is attributed to rule N rather than rule O. + +### 6.5 Worked examples + +Volatile (all → ``): + + 12345 0 4192 + 3f2504e0-4f89-11d3-9a0c-0305e82c3301 3F2504E0-4F89-11D3-9A0C-0305E82C3301 + 3f2504e04f8911d39a0c0305e82c3301 (32 hex, rule H) + e3b0c44298fc1c14 (16 hex, rule H at the threshold) + da39a3ee5e6b4b0d3255bfef95601890afd80709 + dXNlcjEyMzQ1Njc4OTAxMg (22 alnum with digits, rule O) + {id} {userId} :id :userId + +Preserved (never templated): + + v1 v2 me users api latest oauth2 utf8 Search + internationalization (20 letters, no digit) + recommendations + release-notes-2026-08 (hyphenated slug) + user_profile_settings (underscored slug) + deadbeef (8 hex, below 16) + cafebabecafebab (15 hex, below 16) + a1b2c3d (short git object id) + report.pdf 2026-08-08 + +Whole routes: + +| Input | Output | +|---|---| +| `/api/v1/users/12345/orders` | `/api/v1/users//orders` | +| `/api/v1/users/{id}/orders` | `/api/v1/users//orders` | +| `/api/v1/users/:userId/orders` | `/api/v1/users//orders` | +| `api//v1/users/12345//orders/?debug=1` | `/api/v1/users//orders` | +| `/12345/orders/6789` | `//orders/` | +| `/api/v1/users/me/orders` | `/api/v1/users/me/orders` | +| `/` | `/` | + +`testdata/fingerprint_corpus/dast-04-idor-concrete-numeric-and-uuid-segments.json` is the fixture that +proves the derivation happens, with twelve mutations that must all produce one digest. + +--- + +## 7. The remaining canonicalisations + +Each of these was applied by the implementation but absent from the summary text (CRITIQUE-01 finding +9). They are normative. + +### 7.1 `CanonicalRepoRelPath` + +Applied to SAST `repo_relpath` and to the SCA locator (`manifest_relpath`). In order: + +1. Replace every `\` with `/`. +2. While the string contains `//`, replace every `//` with `/`. +3. While the string starts with `./`, remove those two characters. +4. Remove a single leading `/` if present. +5. Remove a single trailing `/` if present. + +So `./cmd/x.go`, `/cmd/x.go`, `cmd\x.go` and `cmd/x.go` are one path. + +It deliberately does **not** case-fold — POSIX paths are case-sensitive and folding would merge two +genuinely distinct files. It does **not** resolve `..` — a path escaping the repo root is the caller's +bug and must not be silently rewritten into a different file. See §9 for what this does *not* achieve. + +### 7.2 `PurlBase` + +Reduces a package URL to its version-free base, `pkg:type/namespace/name`. + +1. Trim surrounding whitespace. Reject an empty string. +2. The first four characters must be `pkg:`, compared **case-insensitively**; otherwise reject. +3. Take everything after `pkg:` and truncate it at the first `#` (subpath), then at the first `?` + (qualifiers), then at the first `@` (version) — in that order. +4. Remove a single trailing `/`. Reject if nothing remains. +5. Lower-case everything up to the first `/` (the type). Reject if there is no `/` (a type but no name). +6. The result is `"pkg:"` + the processed remainder. + +The scheme and type are lower-cased because the purl specification defines both as case-insensitive +with a lowercase canonical form. The **namespace and name are left alone**, because their +case-sensitivity is type-dependent and folding them could merge two distinct packages. + +Truncating at the first raw `@` is safe against namespaced packages: purl requires a literal `@` inside +a namespace or name to be percent-encoded as `%40` (as in `pkg:npm/%40angular/core@13.0.0`), so the +first raw `@` can only be the version delimiter. + +This step is the **enforcement point** for "the version string is never hashed": a caller who passes +the full versioned purl by mistake still gets a version-free fingerprint. + +### 7.3 Host locator + + lower(trim(package_manager)) + ":" + trim(host_identifier) + +The manager is lower-cased because `APT` and `apt` are the same manager and a case difference between +two scanner versions would fork every host finding at once. The identifier is **not** case-folded and +is otherwise verbatim: architecture and suffixes are meaningful to the manager (`openssl:amd64`). + +A `package_manager` containing `:` is **rejected** — it is the locator's own delimiter, and allowing it +would make `a:b` + `c` indistinguishable from `a` + `b:c`. An empty manager or identifier (after +trimming) is rejected. + +### 7.4 `http_method` + +Trim surrounding whitespace, then upper-case. Reject if empty before trimming-and-checking, or if the +trimmed result contains any internal whitespace (it must be a single token). + +RFC 9110 methods are case-sensitive and canonically uppercase; a producer sending `get` must not fork +identity from one sending `GET`. + +Case folding uses the Unicode default case mappings, which are locale-independent. There is no +Turkish-I hazard. + +--- + +## 8. Machine-checked constants + +`fingerprint_spec_test.go` parses this block and asserts every value equals the corresponding constant +in `fingerprint.go` / `contract.go`. + + +```text +FingerprintAlgV1 = anvil-fp/v1 +FingerprintFieldSeparator = U+001F +FingerprintDigestHexLen = 64 +NormalizedStringToken = +NormalizedNumberToken = +NormalizedMetavarPrefix = $ +NormalizedRouteSegmentToken = +routeHexSegmentMinLen = 16 +routeOpaqueSegmentMinLen = 20 +``` + + +--- + +## 9. What this algorithm does NOT do + +Stated so the gaps are visible rather than assumed covered. + +- **No Unicode normalization.** A path or symbol containing a precomposed versus decomposed accented + character hashes differently. This is the macOS-checkout case (CRITIQUE-01 finding 8c). +- **No case folding of repo paths.** A Windows producer may legitimately report `Internal/API/Store.go` + where a Linux producer reports `internal/api/store.go`. Different digests (finding 8a). +- **Backslash rewriting is lossy on POSIX.** A real Linux file named `a\b.go` canonicalises onto + `a/b.go` and collides with the genuinely different file at that path (finding 8b). +- **`NormalizeMatch` does not strip C0 controls.** A snippet containing `NUL`, `BEL`, `ESC` or a raw + `\x1f` survives normalization and is then rejected by §1.2, so the finding cannot be hashed at all + (finding 7). +- **`rule_id_versioned` is hashed with its ruleset version.** An opengrep or nuclei-templates release + bump re-mints every SAST and DAST fingerprint at once (finding 6). +- **`enclosing_symbol_path` is hashed but is not in the ordinal grouping key**, so deleting an + unrelated function above a match can churn that match's ordinal and therefore its digest (finding 4). +- **Two semantically different sinks can share a `normalized_match`** (`exec(cmd)` and `exec(other)` + both give `exec($1)`), leaving `ordinal` as the only discriminator; swapping the two lines swaps the + two findings' identities (finding 5). +- **`primaryLocationLineHash` is not produced here.** It is not an `anvil-fp/v1` tier; it is a separate, + line-dependent partial fingerprint for GitHub code-scanning de-duplication (finding 3). + +None of these is ruled on by this document. See §10. + +--- + +## 10. Provenance + +| Date | Change | +|---|---| +| 2026-08-08 | Document created. Discharges the R.3 blocker-1 ruling: `normalized_match` was defined only in Go, so R.16's mandated independent oracle could not reproduce the SAST goldens and a second producer implementing from the written text would diverge silently. §3 is the algorithm written down completely; §5 and §7 close CRITIQUE-01 finding 9. | +| 2026-08-08 | §6 added. Discharges the R.3 blocker-2 ruling: `route_template` was specified as derived and derived by nobody. Templating is implemented in `CanonicalRouteTemplate` and owned by area 40. Two DAST goldens moved as a result — `dast-01` `ca801b8d…` → `199c3b5f…` and `dast-03` `84fe311d…` → `5fc15c55…`; `dast-02` and every SAST, SCA and host golden are unchanged. Nothing had been stored under the old digests, so this is a correction of an unimplemented clause, not a `v2` event. | + +**Open, not ruled on.** CRITIQUE-01 findings 3, 4, 5, 6, 7, 8, 10 and 11 remain open. They are listed +in §9 so no downstream area assumes they are handled. Findings 4, 5 and 6 in particular each require an +orchestrator ruling before `anvil-fp/v1` is treated as final, and findings 4 and 6 would be `v2` events +if ruled in favour of the critic. + +--- + +## Appendix Z — Where this specification is still under-determined + +Recorded by the orchestrator on 2026-08-07, from the independent-oracle verification that closed +blocker 1. **These are not known bugs. They are places where the prose admits more than one reading and +the corpus happens not to distinguish them** — which is the same defect as blocker 1, one level down and +not yet triggered. + +The verification that closed blocker 1 was real: an implementer working from this document alone, +forbidden from reading `fingerprint.go`, reproduced all 8 committed digests and 42/42 mutations on the +first run with no iteration. But **MATCH means this document is sufficient for what the corpus +exercises**, and the corpus exercises a narrow slice. Anyone extending the corpus, and `R.16` in +particular, should resolve these first — each one is cheap to settle now and expensive to discover as a +digest divergence later. + +| # | Where | The ambiguity | What the oracle guessed | +|---|---|---|---| +| Z1 | §3.2 rule 1 | *"the Unicode space separators"* names category **Zs**, but Go's `unicode.IsSpace` implements the **White_Space** property, which also includes U+2028 (Zl) and U+2029 (Zp). | Included both. **Unverified** — no fixture contains non-ASCII whitespace. | +| Z2 | §3.5 clauses (c), (d) | *"the next non-space input characters"* — "non-space" is never defined and need not mean the same class as rule 1's "whitespace". | Used the full whitespace class, so `Ns\n::Helper(v)` preserves `Ns`. A narrower reading abstracts it. | +| Z3 | §6.3 rule P | *"length >= 2 and (starts { and ends }) or (starts < and ends >) or starts :"* — and/or precedence in prose is genuinely ambiguous. | Read as `len>=2 AND (A or B or C)`. The alternative parse differs for the one-character segment `:`. | +| Z4 | §4 (ordinals) | **Not exercised by the corpus at all.** Every SAST fixture supplies a pre-computed `ordinal` in its `input`; no fixture supplies a batch of candidates with line/column for an implementer to derive ordinals from. | Nothing — untested end to end. An independent implementation could get the grouping key wrong and still pass every current fixture. | +| Z5 | §3 generally | Only two snippet shapes appear (a Go `.`-selector call and a Python `.`-selector call) plus comment/CRLF/rename variants. **Unexercised:** §3.2 rule 4 block comments, §3.3 backtick raw strings and their no-escape rule, §3.4 `` entirely, and most of the reserved-word list. | Nothing — those paths have no fixture. | +| Z6 | fixture schema | The fixture JSON uses `evidence_signal` where the spec's field is `evidence_class_detail`, and `repo_rel_path` where the spec says `repo_relpath`. The spec never states the fixture schema, so the mapping is inferred by eye. | Mapped by inspection. Unambiguous in practice, but it is an undocumented interface. | + +**Z4 is the one that matters most.** The ordinal grouping key is what keeps a finding's identity stable +when an unrelated edit moves it, and it is the single least-tested part of the algorithm. It also +already carries an unresolved `CRITIQUE-01` finding (the key omits `enclosing_symbol_path`, so an edit +elsewhere in the same file can churn a live finding's identity). Settling Z4 and that finding together +is the obvious next move on this algorithm. + +**Resolving any of these is an `anvil-fp/v2` event if it changes a digest, and a v1 clarification if it +does not.** Determine which before editing, not after. diff --git a/internal/record/fingerprint.go b/internal/record/fingerprint.go new file mode 100644 index 0000000..a84ddc9 --- /dev/null +++ b/internal/record/fingerprint.go @@ -0,0 +1,1429 @@ +package record + +// fingerprint.go — the ONE anvil-fp/v1 algorithm. +// +// =========================================================================== +// THE CONFLICT THIS FILE RESOLVES +// =========================================================================== +// +// research/07-database-design.md §3 ("Fingerprint scheme: anvil-fp/v1 — a +// four-tier cascade") and research/18-unified-audit-record.md ("Stable +// identity — the rule that makes regression checking work") each specified a +// DIFFERENT algorithm under the SAME name, `anvil-fp/v1`. They disagree on +// the field set, the separator, the digest length, and the normalization +// depth. +// +// plan/00-SPINE.md S6 states the consequence plainly: "One fingerprint +// algorithm, defined once, in the record. Two branches specified different +// /v1 algorithms under the same name; two producers emitting different hashes +// means regression matching silently fails forever." Silently is the +// operative word — nothing surfaces the failure. Every finding looks new on +// every scan, `finding.first_seen_at` never stabilises, the regression check +// `UNIQUE (target_id, fingerprint)` never fires, and "verified fixed" can +// never be proved, all without a single error being logged. +// +// plan/40-record-and-storage.md, "Fingerprint Specification", is the +// orchestrator's resolution. THAT TEXT — not either research branch verbatim +// — is what this file implements. The seven contested points and the reason +// each was decided that way: +// +// 1. SEPARATOR — research/07's explicit U+001F wins over research/18's +// undefined "‖" glyph. A printable separator can occur inside a snippet +// or a symbol name and silently move a field boundary: hashing +// ("a‖b", "c") and ("a", "b‖c") would collide. U+001F cannot appear in +// normalized source text, and Digest rejects it (and every other C0 +// control character) outright rather than trusting that claim. +// +// 2. ORDINAL — research/07's `ordinal` is adopted; research/18's SAST +// formula lacks it. Without it, two identical macro-expanded or generated +// call sites in one file hash identically and the second finding is lost +// on upsert against `UNIQUE (target_id, fingerprint)`. Losing a finding +// is worse than churning one. +// +// 3. advisory_id IS EXCLUDED from the SAST hash — research/07 includes it; +// research/18 does not; the resolution follows research/18. Identity +// tracks "this exact defect in this exact code". If ingestion later +// attaches a specific advisory to a sink previously linked only to a +// generic CWE, that reclassification must not fork the finding's +// identity. SastInput therefore has no advisory field at all — the +// exclusion is enforced by the type, not by discipline. +// +// 4. NO TRUNCATION — research/07 truncates to 32 hex chars "for storage". +// The resolution does not truncate. SQLite's cost for a 64- vs +// 32-character TEXT column is negligible, and halving a cryptographic +// digest without a forcing constraint buys nothing and adds collision +// risk. Digest always returns FingerprintDigestHexLen (64) characters. +// +// 5. NORMALIZATION DEPTH — research/07's metavariable abstraction +// (literals to /, local identifiers to positional $1..$N) is +// adopted over research/18's whitespace-and-comments-only normalization, +// because it is the one directly modelled on Semgrep's `match_based_id` +// — the cited, externally-verified mechanism for surviving reindentation +// and metavariable-only edits (research/07 [S3]). +// +// 6. DAST EVIDENCE SIGNAL — research/18's `evidenceClass` (HOW the defect +// was observed) is a real distinguishing signal absent from research/07's +// Tier D. It is hashed ALONGSIDE research/07's injection_point/param_name +// (WHERE the payload went in), because the two are independent facts. +// See EvidenceSignal and InjectionPoint in contract.go. +// +// 7. HOST TIER — neither branch defined a tier for host/package findings, +// but plan/00-SPINE.md S1 gives Lane A "dependency and host findings" and +// S6 requires `remediable_by_agent=false` on all of them, so they need an +// identity. research/07's Tier C is generalised by parameterising the +// hashed detector kind over {sca, host} rather than inventing an +// unrelated scheme. +// +// research/18's Tier B (CodeQL's `primaryLocationLineHash`) is deliberately +// NOT implemented here. It is not an anvil-fp/v1 tier; it is a separate, +// line-DEPENDENT partial fingerprint whose only purpose is GitHub code +// scanning de-duplication. It lives under +// PartialFingerprintPrimaryLocationLineHash and is owned by the GitHub +// projection (R.14). Computing it here would put a line number one import +// away from this file. +// +// =========================================================================== +// THE VERSION STRING IS NEVER HASHED +// =========================================================================== +// +// For the SCA and host tiers this is the single most load-bearing exclusion, +// and it is the reason PurlBase exists rather than the caller passing a purl +// straight through: +// +// Bumping 1.2.3 to 1.2.4 while still inside the vulnerable range must not +// mint a new finding. +// +// If the version were hashed, every patch-level dependency bump would resolve +// the old finding and open an identical new one. `first_seen_at` would reset, +// the age-based ranking would reset, any suppression keyed on the fingerprint +// would silently stop applying, and a maintainer bumping a version WITHOUT +// leaving the vulnerable range would see the alert disappear and reappear as +// "new" — indistinguishable from an actual fix. Resolution is proved by +// re-evaluating `advisory_affects`, never by an identity change. +// +// PurlBase enforces this defensively: it truncates a purl at the first +// version, qualifier, or subpath delimiter, so a caller who passes the full +// versioned purl by mistake still gets a version-free fingerprint. The +// ScaInput/HostInput structs also have no Version field, so there is no +// in-band way to hash one. +// +// =========================================================================== +// WHAT IS EXCLUDED BY CONSTRUCTION +// =========================================================================== +// +// The tier input structs below deliberately have NO field for anything the +// specification forbids hashing. This is a type-level guarantee, not a +// convention, and TestInputStructsCannotCarryForbiddenFields asserts it +// reflectively so a later edit cannot quietly add one: +// +// line number, column number — absent from SastInput entirely; any +// unrelated edit above a match changes them +// (research/07 §3: "the single most important +// rule"). +// raw snippet text — SastInput.Snippet is normalized before it +// is hashed; the literal text never reaches +// Digest. +// advisory_id (SAST tier) — see resolution point 3. +// host, port, scheme (DAST) — absent from DastInput; a redeployed +// container, a rotated ephemeral port, or a +// staging-to-prod move must not fork +// identity. +// payload string (DAST) — absent from DastInput; a fuzzer varying +// its payload must not create N findings for +// one bug. +// timestamps — absent from every input struct. +// version string (SCA/host) — absent, and stripped defensively by +// PurlBase. +// evidence_class (SAST tier) — the SAST tier hashes the literal "sast", +// not the evidence class, so a finding +// upgraded from sast_static_only to +// sast_reachable keeps its identity. +// +// =========================================================================== +// THE AUTHORITATIVE SPECIFICATION IS internal/record/FINGERPRINT-SPEC.md +// =========================================================================== +// +// R.3's CRITIQUE-01.md proved that the four-clause `normalized_match` text in +// plan/40-record-and-storage.md is NOT sufficient to reproduce this file's +// digests: a re-implementation written from that text alone emits +// 55e27b07... where the committed golden for sast-01 is 13c60ccf... . The +// orchestrator ruled (2026-08-08) that the implementation is right and the +// specification was incomplete, and that the fix is to write the +// specification down completely, IN TREE. +// +// internal/record/FINGERPRINT-SPEC.md is that document. It is the +// authoritative definition of anvil-fp/v1: every normalization step in +// NormalizeMatch in the order applied, the reserved-word list verbatim, the +// identifier-preservation rules and their reasons, the route-segment +// templating patterns and thresholds, the separator, the join order, the hash, +// and the ordinal grouping key. plan/ is gitignored; a second producer working +// from a clone can read FINGERPRINT-SPEC.md and nothing else and still emit +// byte-identical digests. +// +// fingerprint_spec_test.go keeps the document honest: it asserts that the +// reserved-word list and the algorithm constants printed in the document are +// exactly the ones in this file. A specification that can drift from the code +// silently is the same defect one level up. +// +// =========================================================================== +// CONFORMANCE (R.16) +// =========================================================================== +// +// testdata/fingerprint_corpus/*.json is the fixed corpus. Every fixture +// carries its complete ordered `hashed_fields` list and its `expected_digest`, +// so R.16's offline oracle can re-derive the digest from the fixture alone — +// join with U+001F, SHA-256, lowercase hex — without importing or reading any +// Go code. R.16 must NOT copy `expected_digest`; it must recompute it from +// FINGERPRINT-SPEC.md (NOT from plan/40-record-and-storage.md, whose algorithm +// text is a summary and was proved insufficient by CRITIQUE-01) and assert +// equality. +// That mutual check is the mechanism that would have caught research/07 and +// research/18 shipping two different /v1 algorithms under one name. +// +// Sources: plan/40-record-and-storage.md ("Fingerprint Specification"); +// plan/00-SPINE.md S1, S6, S7; plan/IMPLEMENTATION-PLAN.md §6 (this file +// declares no shared enum — it consumes contract.go's DetectorKind, +// InjectionPoint and EvidenceSignal); research/07-database-design.md §3; +// research/18-unified-audit-record.md ("Stable identity"). + +import ( + "crypto/sha256" + "encoding/hex" + "fmt" + "sort" + "strconv" + "strings" + "unicode" +) + +// --------------------------------------------------------------------------- +// Algorithm constants that R.16's independent oracle must reproduce +// --------------------------------------------------------------------------- + +// The separator (FingerprintFieldSeparator, U+001F) and the digest length +// (FingerprintDigestHexLen, 64) live in contract.go, because they are wire +// contract rather than implementation. The tokens below are algorithm detail +// but must still be reproduced byte for byte by any re-implementation, so +// they are exported and named. +const ( + // NormalizedStringToken replaces every string literal in a normalized + // SAST match. Chosen with angle brackets because they cannot appear in + // an identifier, so a source-level `` cannot be confused with the + // placeholder in any language Anvil scans. + NormalizedStringToken = "" + + // NormalizedNumberToken replaces every numeric literal in a normalized + // SAST match. + NormalizedNumberToken = "" + + // NormalizedMetavarPrefix prefixes the positional metavariables that + // replace local identifiers: $1, $2, ... in first-occurrence order. + NormalizedMetavarPrefix = "$" + + // NormalizedRouteSegmentToken replaces every volatile path segment in a + // canonicalised DAST route template — a numeric id, a UUID, a long + // hex/base32/base64-ish opaque token, and any segment a producer has + // already templated in its own syntax ("{id}", ":id", ""). + // + // Angle brackets are chosen for the same reason as NormalizedStringToken: + // RFC 3986 excludes '<' and '>' from every production a path segment can + // use, so they must be percent-encoded to appear in a real URL. A literal + // segment therefore cannot be confused with the placeholder, and + // CanonicalRouteTemplate is idempotent — "" is itself recognised as + // an already-templated segment and maps to itself. + NormalizedRouteSegmentToken = "" + + // tierTokenSast and tierTokenDast are the literal tier discriminators + // hashed in field position 2 of their tiers. They are string literals + // rather than DetectorKind values on purpose: the SAST tier hashes + // "sast" for BOTH sast_reachable and sast_static_only findings, so this + // token must not be confused with the evidence class. + tierTokenSast = "sast" + tierTokenDast = "dast" +) + +// Route-segment templating thresholds. These are part of anvil-fp/v1: changing +// either one changes every DAST digest whose route carries a segment near the +// boundary, and is therefore an anvil-fp/v2 event, not a tuning knob. +// +// The governing asymmetry, from the R.3 ruling: OVER-templating merges two +// genuinely distinct routes into one identity and silently loses a finding on +// upsert against UNIQUE (target_id, fingerprint); UNDER-templating only leaves +// a volatile route un-merged, which the DAST producer can still fix by +// emitting "{id}" itself. Under-templating is the recoverable failure, so both +// thresholds are set high enough that no plausible human-authored path segment +// reaches them. +const ( + // routeHexSegmentMinLen is the length at or above which an all-hex segment + // is treated as an opaque identifier. 16 is chosen because the hex + // alphabet's letters are only a-f: a 16-character English word drawn from + // {a,b,c,d,e,f} plus digits does not exist (the longest such words — + // "defaced", "cabbage" — are seven letters), while every hash form Anvil + // will meet in a URL is at or above it: MD5 is 32, SHA-1 is 40, SHA-256 is + // 64, a dash-free UUID is 32, and a short git object id is 7-12 and so is + // deliberately NOT templated (7 hex characters is also a plausible slug). + routeHexSegmentMinLen = 16 + + // routeOpaqueSegmentMinLen is the length at or above which a mixed + // alphanumeric segment is treated as an opaque identifier. 20 is chosen + // against the longest plausible single-word route segments — + // "recommendations" (15), "internationalization" (20), "misrepresentation" + // (17) — none of which contains a digit, which is why the digit + // requirement in isLongOpaqueRouteSegment carries most of the safety here. + // A base64url session token is 22+ characters and a base32 token is 26+, + // so real opaque tokens clear the bar comfortably. + routeOpaqueSegmentMinLen = 20 +) + +// --------------------------------------------------------------------------- +// Errors +// --------------------------------------------------------------------------- + +// FingerprintError reports an input that cannot be fingerprinted. It names +// the tier and the field, because a bare "invalid input" tells a producer +// nothing about which of eight positional fields it got wrong — and a +// producer that silently emits a wrong digest is exactly the failure this +// package exists to prevent, so every tier function fails loudly rather than +// hashing a degraded input. +type FingerprintError struct { + Tier string // "sast", "dast", "sca", "host", or "" for the shared primitive + Field string // the specification's field name, e.g. "repo_relpath" + Msg string +} + +func (e *FingerprintError) Error() string { + if e.Tier == "" { + return fmt.Sprintf("anvil-fp/v1: %s: %s", e.Field, e.Msg) + } + return fmt.Sprintf("anvil-fp/v1 tier %s: %s: %s", e.Tier, e.Field, e.Msg) +} + +func fpErr(tier, field, msg string) error { + return &FingerprintError{Tier: tier, Field: field, Msg: msg} +} + +// fpRequire returns v, or an error naming the field if v is empty. +func fpRequire(tier, field, v string) (string, error) { + if v == "" { + return "", fpErr(tier, field, "must not be empty") + } + return v, nil +} + +// --------------------------------------------------------------------------- +// The shared primitive: join, guard, hash +// --------------------------------------------------------------------------- + +// Digest joins fields with FingerprintFieldSeparator (U+001F) and returns the +// lowercase hex SHA-256 of the result: exactly FingerprintDigestHexLen (64) +// characters, never truncated. +// +// It rejects any field containing a C0 control character or DEL. U+001F is +// the important case — a field carrying the separator would silently move a +// field boundary and let two different findings collide — but the whole +// control range is rejected because none of them can legitimately appear in +// a canonicalised path, rule id, symbol path, route template, purl, advisory +// id, or normalized match, and a newline in one of those means the caller has +// passed raw, uncanonicalised text. +// +// The exported tier functions below are the only sanctioned field lists. +// Digest is exported for the store and for debugging tooling, not so callers +// can invent a fingerprint shape of their own. +func Digest(fields ...string) (string, error) { + if len(fields) == 0 { + return "", fpErr("", "fields", "at least one field is required") + } + for i, f := range fields { + if idx := strings.IndexFunc(f, isForbiddenControl); idx >= 0 { + what := "a C0 control character" + if f[idx] == FingerprintFieldSeparator[0] { + what = "the U+001F field separator" + } + return "", fpErr("", "fields["+strconv.Itoa(i)+"]", + "contains "+what+"; this would move a field boundary and let two distinct findings collide") + } + } + sum := sha256.Sum256([]byte(strings.Join(fields, FingerprintFieldSeparator))) + return hex.EncodeToString(sum[:]), nil +} + +func isForbiddenControl(r rune) bool { + return r < 0x20 || r == 0x7f +} + +// ValidateDigest reports whether s is a well-formed anvil-fp/v1 digest: +// exactly 64 lowercase hexadecimal characters. Uppercase hex is rejected +// rather than folded, because a store that accepts both would hold two rows +// for one finding and defeat UNIQUE (target_id, fingerprint). +func ValidateDigest(s string) error { + if len(s) != FingerprintDigestHexLen { + return fpErr("", "digest", + fmt.Sprintf("must be exactly %d hex characters, got %d (the digest is never truncated)", + FingerprintDigestHexLen, len(s))) + } + for i := 0; i < len(s); i++ { + c := s[i] + if (c >= '0' && c <= '9') || (c >= 'a' && c <= 'f') { + continue + } + return fpErr("", "digest", "must be lowercase hexadecimal; found "+strconv.QuoteRune(rune(c))+ + " at offset "+strconv.Itoa(i)) + } + return nil +} + +// --------------------------------------------------------------------------- +// Tier SAST +// --------------------------------------------------------------------------- + +// SastInput is the complete input to the SAST fingerprint tier, which covers +// EvidenceClassSastReachable and EvidenceClassSastStaticOnly alike. +// +// There is no line, column, advisory, or evidence-class field here, and there +// must never be one: each of those changes for reasons unrelated to the +// defect's identity. The raw Snippet is accepted only so NormalizeMatch can +// abstract it; the literal text is never hashed. +type SastInput struct { + // TargetID is the stable identifier of the scanned target + // (`target.target_id` rendered as text). Required. + TargetID string + + // RuleIDVersioned is the detector rule id including its ruleset version, + // e.g. "opengrep.go.lang.security.audit.sqli@2026.07". Required. The + // version component is part of identity here: a rule that changed what + // it matches is a different rule, and conflating them would silently + // migrate findings between rule semantics. + RuleIDVersioned string + + // RepoRelPath is the repository-root-relative POSIX path of the file + // containing the match. Required. Canonicalised by CanonicalRepoRelPath + // before hashing so a Windows producer and a Linux producer agree. + RepoRelPath string + + // EnclosingSymbolPath is the fully-qualified enclosing symbol, e.g. + // "pkg/mod.py::ClassA.method_b". MAY be empty: a match in top-level + // module code, a config file, or a template has no enclosing symbol, and + // rejecting those would make them unfingerprintable. An empty value is + // hashed as an empty field, which keeps the field count constant. + EnclosingSymbolPath string + + // Snippet is the RAW matched source text. It is passed through + // NormalizeMatch and only the normalized form is hashed. + Snippet string + + // Ordinal is the 0-based index of this match among all matches of the + // same RuleIDVersioned in the same RepoRelPath whose normalized match is + // IDENTICAL. Use AssignSastOrdinals to compute it. Without it, two + // identical generated or macro-expanded call sites in one file collide + // and one finding is lost on upsert. + Ordinal int +} + +// SastFields returns the ordered field list the SAST tier hashes, exactly as +// plan/40-record-and-storage.md specifies: +// +// target_id ␟ "sast" ␟ rule_id_versioned ␟ repo_relpath +// ␟ enclosing_symbol_path ␟ normalized_match ␟ ordinal +// +// Exported so a test or R.16's conformance harness can assert the field list +// itself, not merely the digest — a wrong field ORDER produces a perfectly +// valid-looking 64-hex digest that is silently incompatible. +func SastFields(in SastInput) ([]string, error) { + const tier = "sast" + + targetID, err := fpRequire(tier, "target_id", in.TargetID) + if err != nil { + return nil, err + } + ruleID, err := fpRequire(tier, "rule_id_versioned", in.RuleIDVersioned) + if err != nil { + return nil, err + } + if in.RepoRelPath == "" { + return nil, fpErr(tier, "repo_relpath", "must not be empty") + } + relPath := CanonicalRepoRelPath(in.RepoRelPath) + if relPath == "" { + return nil, fpErr(tier, "repo_relpath", "canonicalises to the empty string") + } + if in.Snippet == "" { + return nil, fpErr(tier, "normalized_match", "snippet must not be empty") + } + normalized := NormalizeMatch(in.Snippet) + if normalized == "" { + return nil, fpErr(tier, "normalized_match", + "snippet normalises to the empty string (comments and whitespace only); it carries no identity") + } + if in.Ordinal < 0 { + return nil, fpErr(tier, "ordinal", "must not be negative") + } + + return []string{ + targetID, + tierTokenSast, + ruleID, + relPath, + in.EnclosingSymbolPath, + normalized, + strconv.Itoa(in.Ordinal), + }, nil +} + +// Sast returns the anvil-fp/v1 digest for a first-party source finding. +func Sast(in SastInput) (string, error) { + fields, err := SastFields(in) + if err != nil { + return "", err + } + return Digest(fields...) +} + +// --------------------------------------------------------------------------- +// Tier SCA / HOST — one formula, parameterised by detector kind +// --------------------------------------------------------------------------- + +// ScaInput is the input to the SCA tier: a repository dependency that matched +// a vulnerable version range. +// +// There is deliberately no Version field. See "THE VERSION STRING IS NEVER +// HASHED" at the top of this file. +type ScaInput struct { + // TargetID is the scanned target's identifier. Required. + TargetID string + + // AdvisoryID is the canonical advisory identifier as stored in + // `advisory.advisory_id` (e.g. "CVE-2021-44228", "GHSA-jfh8-c2jp-5v3q"). + // Required, and hashed VERBATIM: advisory identifiers are not + // case-normalised here because GHSA identifiers mix cases meaningfully + // and folding them would fork identity against the advisory table. + AdvisoryID string + + // Purl is the package URL. It MAY carry a version, qualifiers, or a + // subpath; PurlBase strips all three before hashing. + Purl string + + // ManifestRelPath is the repo-relative path of the manifest or lockfile + // that declared the dependency, e.g. "services/api/go.mod". Required: + // the same vulnerable package pulled in by two different manifests in a + // monorepo is two findings with two different owners and two different + // fixes. + ManifestRelPath string +} + +// HostInput is the input to the host tier: an operating-system package on the +// scanned host that matched a vulnerable version range. +// +// plan/00-SPINE.md S7 makes the host agent read-only, so every host finding +// carries `remediable_by_agent=false` (enforced by the `finding` table's +// CHECK constraint, not here). It still needs a stable identity so that a +// host finding can be tracked, suppressed, and reported as resolved. +type HostInput struct { + // TargetID is the scanned target's identifier. Required. + TargetID string + + // AdvisoryID is the canonical advisory identifier. Required, verbatim. + AdvisoryID string + + // Purl is the package URL for the host package, e.g. + // "pkg:deb/debian/openssl". Version, qualifiers and subpath are stripped. + Purl string + + // PackageManager is the host package manager, e.g. "apt", "apk", "rpm", + // "dpkg". Required. Lowercased before hashing, because "APT" and "apt" + // are the same manager and a case difference between two scanner + // versions would fork every host finding at once. + PackageManager string + + // HostIdentifier is the package identity as the manager names it, e.g. + // "openssl" or "openssl:amd64". Required, verbatim: architecture and + // suffixes are meaningful to the manager. + HostIdentifier string +} + +// HostLocator composes the host tier's `locator` field, +// ":" (e.g. "apt:openssl"), applying the +// documented lowercasing of the manager segment. +func HostLocator(packageManager, hostIdentifier string) string { + return strings.ToLower(strings.TrimSpace(packageManager)) + ":" + strings.TrimSpace(hostIdentifier) +} + +// packageFields is the single implementation of the shared SCA/host formula: +// +// target_id ␟ detector_kind ␟ advisory_id ␟ purl_base ␟ locator +// +// Written once and parameterised rather than copied, so the two tiers cannot +// drift apart the way research/07 and research/18 did. +func packageFields(tier string, kind DetectorKind, targetID, advisoryID, purl, locator string) ([]string, error) { + tid, err := fpRequire(tier, "target_id", targetID) + if err != nil { + return nil, err + } + adv, err := fpRequire(tier, "advisory_id", advisoryID) + if err != nil { + return nil, err + } + if purl == "" { + return nil, fpErr(tier, "purl_base", "purl must not be empty") + } + base, err := PurlBase(purl) + if err != nil { + return nil, err + } + if locator == "" { + return nil, fpErr(tier, "locator", "must not be empty") + } + return []string{tid, string(kind), adv, base, locator}, nil +} + +// ScaFields returns the ordered field list the SCA tier hashes. +func ScaFields(in ScaInput) ([]string, error) { + const tier = "sca" + if in.ManifestRelPath == "" { + return nil, fpErr(tier, "locator", "manifest_relpath must not be empty") + } + locator := CanonicalRepoRelPath(in.ManifestRelPath) + if locator == "" { + return nil, fpErr(tier, "locator", "manifest_relpath canonicalises to the empty string") + } + return packageFields(tier, DetectorKindSCA, in.TargetID, in.AdvisoryID, in.Purl, locator) +} + +// Sca returns the anvil-fp/v1 digest for a repository dependency finding. +func Sca(in ScaInput) (string, error) { + fields, err := ScaFields(in) + if err != nil { + return "", err + } + return Digest(fields...) +} + +// HostFields returns the ordered field list the host tier hashes. +func HostFields(in HostInput) ([]string, error) { + const tier = "host" + if strings.TrimSpace(in.PackageManager) == "" { + return nil, fpErr(tier, "locator", "package_manager must not be empty") + } + if strings.ContainsRune(in.PackageManager, ':') { + return nil, fpErr(tier, "locator", + "package_manager must not contain ':'; it is the locator's own delimiter") + } + if strings.TrimSpace(in.HostIdentifier) == "" { + return nil, fpErr(tier, "locator", "host_identifier must not be empty") + } + return packageFields(tier, DetectorKindHost, in.TargetID, in.AdvisoryID, in.Purl, + HostLocator(in.PackageManager, in.HostIdentifier)) +} + +// Host returns the anvil-fp/v1 digest for a host package finding. +func Host(in HostInput) (string, error) { + fields, err := HostFields(in) + if err != nil { + return "", err + } + return Digest(fields...) +} + +// --------------------------------------------------------------------------- +// Tier DAST +// --------------------------------------------------------------------------- + +// DastInput is the input to the DAST tier. +// +// There is no host, port, scheme, payload, session token, or timestamp field, +// and there must never be one. A redeployed container, a rotated ephemeral +// port, or a staging-to-prod move must not fork identity, and a fuzzer +// varying its payload must not mint N findings for one bug. +type DastInput struct { + // TargetID is the scanned target's identifier. Required. + TargetID string + + // RuleIDVersioned is the detector rule id including its version, e.g. + // "nuclei:CVE-2021-44228@a1b2c3d". Required. + RuleIDVersioned string + + // HTTPMethod is the request method. Required; uppercased before hashing + // (RFC 9110 methods are case-sensitive and canonically uppercase, and a + // producer sending "get" must not fork identity from one sending "GET"). + HTTPMethod string + + // RouteTemplate is the observed request path. Required. It may be the + // CONCRETE path the producer requested ("/api/users/12345/orders") or a + // path the producer has already templated in any of the three common + // syntaxes ("{id}", ":id", ""); CanonicalRouteTemplate derives the + // hashed template from either, so the caller does not have to agree with + // any other caller about which form to use. It also strips any query + // string or fragment — those carry concrete values, which is exactly what + // templating exists to remove. + // + // Templating is deliberately NOT the producer's job. See + // CanonicalRouteTemplate for the ruling and the reason. + RouteTemplate string + + // InjectionPoint is WHERE the payload was injected. Required; must be a + // legal InjectionPoint literal from contract.go. + InjectionPoint InjectionPoint + + // ParamName is the name of the injected parameter. MAY be empty: a + // whole-body or raw-request injection has no single named parameter, and + // rejecting those would make them unfingerprintable. This is a parameter + // NAME, never a parameter value. + ParamName string + + // EvidenceSignal is HOW the vulnerability was observed — research/18's + // contribution to this tier. Required; must be a legal EvidenceSignal + // literal from contract.go. It is independent of InjectionPoint: an SQL + // injection proved by a database error string and one proved by a timing + // side channel on the same parameter are different findings with + // different remediation evidence. + EvidenceSignal EvidenceSignal +} + +// DastFields returns the ordered field list the DAST tier hashes, exactly as +// plan/40-record-and-storage.md specifies: +// +// target_id ␟ "dast" ␟ rule_id_versioned ␟ http_method ␟ route_template +// ␟ injection_point ␟ param_name ␟ evidence_class_detail +func DastFields(in DastInput) ([]string, error) { + const tier = "dast" + + targetID, err := fpRequire(tier, "target_id", in.TargetID) + if err != nil { + return nil, err + } + ruleID, err := fpRequire(tier, "rule_id_versioned", in.RuleIDVersioned) + if err != nil { + return nil, err + } + if strings.TrimSpace(in.HTTPMethod) == "" { + return nil, fpErr(tier, "http_method", "must not be empty") + } + method := strings.ToUpper(strings.TrimSpace(in.HTTPMethod)) + if strings.ContainsFunc(method, unicode.IsSpace) { + return nil, fpErr(tier, "http_method", "must be a single token") + } + if in.RouteTemplate == "" { + return nil, fpErr(tier, "route_template", "must not be empty") + } + route := CanonicalRouteTemplate(in.RouteTemplate) + if route == "" { + return nil, fpErr(tier, "route_template", "canonicalises to the empty string") + } + if err := ValidateInjectionPoint(string(in.InjectionPoint)); err != nil { + return nil, fpErr(tier, "injection_point", err.Error()) + } + if err := ValidateEvidenceSignal(string(in.EvidenceSignal)); err != nil { + return nil, fpErr(tier, "evidence_class_detail", err.Error()) + } + + return []string{ + targetID, + tierTokenDast, + ruleID, + method, + route, + string(in.InjectionPoint), + in.ParamName, + string(in.EvidenceSignal), + }, nil +} + +// Dast returns the anvil-fp/v1 digest for a dynamically-confirmed finding. +func Dast(in DastInput) (string, error) { + fields, err := DastFields(in) + if err != nil { + return "", err + } + return Digest(fields...) +} + +// --------------------------------------------------------------------------- +// Canonicalisation helpers +// --------------------------------------------------------------------------- + +// CanonicalRepoRelPath normalises a repository-relative path to the single +// POSIX form the specification assumes ("repo_relpath — POSIX, +// repo-root-relative", research/07 §3): +// +// - backslashes become forward slashes, so a Windows producer and a Linux +// producer scanning the same repository agree; +// - runs of slashes collapse to one; +// - a leading "./" or "/" is removed, so "./cmd/x.go", "/cmd/x.go" and +// "cmd/x.go" are one path; +// - a trailing slash is removed. +// +// It deliberately does NOT case-fold: POSIX paths are case-sensitive, and +// folding would merge two genuinely distinct files on a case-sensitive +// checkout. It also does not resolve ".." — a path escaping the repo root is +// the caller's bug and must not be silently rewritten into a different file. +func CanonicalRepoRelPath(p string) string { + p = strings.ReplaceAll(p, "\\", "/") + for strings.Contains(p, "//") { + p = strings.ReplaceAll(p, "//", "/") + } + for strings.HasPrefix(p, "./") { + p = p[2:] + } + p = strings.TrimPrefix(p, "/") + p = strings.TrimSuffix(p, "/") + return p +} + +// CanonicalRouteTemplate DERIVES the hashed `route_template` from whatever +// route a DAST producer observed. The specification defines the field as a +// derived value — "numeric/UUID/hash path segments replaced with a placeholder +// token" — and this function is where that derivation happens. +// +// WHY IT HAPPENS HERE AND NOT IN THE PRODUCER (R.3 ruling, 2026-08-08). Area +// 40 owns the fingerprint, so area 40 canonicalises. A DAST producer emits +// whatever route it observed; if templating were the producer's job, then two +// producers seeing one defect at /api/users/12345/orders would emit +// "/api/users/12345/orders", "/api/users/{id}/orders" and +// "/api/users/:id/orders" — three digests, one defect, no error, regression +// matching silently dead. The DAST tier is the one that earns "verified fixed" +// under plan/00-SPINE.md S7, and a reproduction that cannot be matched to its +// prior finding cannot prove a fix. +// +// The steps, in order: +// +// 1. anything from the first '?' or '#' is dropped. A query string or +// fragment in a "template" carries concrete values — the very thing +// templating removes — and injection_point plus param_name already record +// which query parameter was targeted; +// 2. backslashes become forward slashes; +// 3. runs of slashes collapse to one; +// 4. a leading '/' is added if absent; +// 5. a trailing '/' is removed, except on the root path "/"; +// 6. every VOLATILE path segment is replaced by NormalizedRouteSegmentToken. +// A segment is volatile when it is already templated in a producer's own +// syntax ("{id}", ":id", ""), is all ASCII digits, is a UUID, is a +// long all-hex run, or is a long mixed alphanumeric run containing a +// digit. See isVolatileRouteSegment for the exact predicates. +// +// Case is preserved on non-volatile segments: URL paths are case-sensitive and +// "/Search" is a different route from "/search" on most servers. The UUID and +// hex predicates are themselves case-insensitive, because the same identifier +// rendered in upper and lower hex is the same identifier. +// +// Percent-encoding is deliberately NOT decoded: decoding could introduce a '/' +// and change the segment structure, and a producer that percent-encodes a +// whole segment has emitted a different route. +func CanonicalRouteTemplate(route string) string { + if i := strings.IndexAny(route, "?#"); i >= 0 { + route = route[:i] + } + route = strings.ReplaceAll(route, "\\", "/") + for strings.Contains(route, "//") { + route = strings.ReplaceAll(route, "//", "/") + } + if route == "" { + return "" + } + if !strings.HasPrefix(route, "/") { + route = "/" + route + } + if len(route) > 1 { + route = strings.TrimSuffix(route, "/") + } + if route == "/" { + return "/" + } + + segs := strings.Split(route[1:], "/") + for i, s := range segs { + if isVolatileRouteSegment(s) { + segs[i] = NormalizedRouteSegmentToken + } + } + return "/" + strings.Join(segs, "/") +} + +// isVolatileRouteSegment reports whether one path segment carries a concrete +// instance identifier rather than route structure, and must therefore be +// replaced by NormalizedRouteSegmentToken. +// +// Every predicate is conservative by construction — see the threshold +// constants for the asymmetry that motivates it. +func isVolatileRouteSegment(s string) bool { + switch { + case s == "": + return false + case isRoutePlaceholderSegment(s): + return true + case isAllASCIIDigits(s): + return true + case isUUIDRouteSegment(s): + return true + case isLongHexRouteSegment(s): + return true + case isLongOpaqueRouteSegment(s): + return true + default: + return false + } +} + +// isRoutePlaceholderSegment recognises a segment a producer has ALREADY +// templated, in any of the three syntaxes in common use: OpenAPI/ASP.NET +// "{id}", Express/Rails/Sinatra ":id", and Flask/Werkzeug "". +// +// Normalising all three onto one token is the point: a DAST crawler, an +// OpenAPI document checked into the repo, and a route table exported from a +// framework will disagree about which syntax to use for the same route, and +// that disagreement must not fork identity. +// +// The "" form also makes CanonicalRouteTemplate idempotent, because +// NormalizedRouteSegmentToken is itself of that form. +func isRoutePlaceholderSegment(s string) bool { + if len(s) < 2 { + return false + } + if s[0] == '{' && s[len(s)-1] == '}' { + return true + } + if s[0] == '<' && s[len(s)-1] == '>' { + return true + } + return s[0] == ':' +} + +func isAllASCIIDigits(s string) bool { + if s == "" { + return false + } + for i := 0; i < len(s); i++ { + if s[i] < '0' || s[i] > '9' { + return false + } + } + return true +} + +func isASCIIHexDigit(c byte) bool { + return (c >= '0' && c <= '9') || (c >= 'a' && c <= 'f') || (c >= 'A' && c <= 'F') +} + +// isUUIDRouteSegment recognises the canonical 8-4-4-4-12 hyphenated form, +// case-insensitively. Braced ("{...}") and URN ("urn:uuid:...") forms are not +// recognised here: the first is already caught by isRoutePlaceholderSegment +// and the second is not a bare path segment. +func isUUIDRouteSegment(s string) bool { + if len(s) != 36 { + return false + } + for i := 0; i < len(s); i++ { + switch i { + case 8, 13, 18, 23: + if s[i] != '-' { + return false + } + default: + if !isASCIIHexDigit(s[i]) { + return false + } + } + } + return true +} + +// isLongHexRouteSegment recognises a run of at least routeHexSegmentMinLen hex +// characters: a dash-free UUID, an MD5/SHA digest, an object id. +func isLongHexRouteSegment(s string) bool { + if len(s) < routeHexSegmentMinLen { + return false + } + for i := 0; i < len(s); i++ { + if !isASCIIHexDigit(s[i]) { + return false + } + } + return true +} + +// isLongOpaqueRouteSegment recognises a base32/base64-ish opaque token: at +// least routeOpaqueSegmentMinLen characters, ASCII alphanumeric throughout, +// and carrying BOTH a digit and a letter. +// +// The three restrictions each buy something specific, and dropping any of them +// over-templates: +// +// - alphanumeric only (no '-', '_', '.') keeps slugs out. A hyphenated slug +// like "release-notes-2026-08" is 21 characters and carries a digit; it is +// route structure, not an identifier, and merging every dated release note +// onto one digest would be exactly the over-templating failure the ruling +// warns about. Base64url tokens that use '-' or '_' are therefore left +// un-templated: under-templating is the recoverable direction. +// - a digit is required, which is what excludes the long all-letter words +// that do reach 20 characters ("internationalization"). +// - a letter is required, so that a purely numeric run is attributed to the +// all-digits rule rather than this one; the outcome is the same token, but +// the two rules stay independently testable. +func isLongOpaqueRouteSegment(s string) bool { + if len(s) < routeOpaqueSegmentMinLen { + return false + } + hasDigit, hasLetter := false, false + for i := 0; i < len(s); i++ { + c := s[i] + switch { + case c >= '0' && c <= '9': + hasDigit = true + case (c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z'): + hasLetter = true + default: + return false + } + } + return hasDigit && hasLetter +} + +// PurlBase reduces a package URL to its version-free base: +// "pkg:type/namespace/name". It strips, in order, any subpath ('#'), any +// qualifiers ('?'), and any version ('@'). +// +// This is the enforcement point for the rule that the version string is NEVER +// hashed. A caller passing "pkg:npm/lodash@4.17.20" and a caller passing +// "pkg:npm/lodash" produce the same fingerprint, so bumping a dependency +// inside the vulnerable range does not mint a new finding. +// +// The '@' scan starts after "pkg:" and is safe against namespaced packages: +// the purl specification requires a literal '@' inside a namespace or name to +// be percent-encoded as "%40" (as in "pkg:npm/%40angular/core@13.0.0"), so +// the first raw '@' can only be the version delimiter. +// +// The scheme and type segments are lowercased because the purl specification +// defines both as case-insensitive with a lowercase canonical form; the +// namespace and name are left alone because their case-sensitivity is +// type-dependent and folding them could merge two distinct packages. +func PurlBase(purl string) (string, error) { + p := strings.TrimSpace(purl) + if p == "" { + return "", fpErr("", "purl_base", "must not be empty") + } + if len(p) < 4 || !strings.EqualFold(p[:4], "pkg:") { + return "", fpErr("", "purl_base", "must be a package URL beginning with \"pkg:\", got "+strconv.Quote(purl)) + } + + rest := p[4:] + if i := strings.IndexByte(rest, '#'); i >= 0 { + rest = rest[:i] + } + if i := strings.IndexByte(rest, '?'); i >= 0 { + rest = rest[:i] + } + if i := strings.IndexByte(rest, '@'); i >= 0 { + rest = rest[:i] + } + rest = strings.TrimSuffix(rest, "/") + if rest == "" { + return "", fpErr("", "purl_base", "carries no type or name after \"pkg:\": "+strconv.Quote(purl)) + } + + // Lowercase only the type segment (everything up to the first '/'). + if i := strings.IndexByte(rest, '/'); i >= 0 { + rest = strings.ToLower(rest[:i]) + rest[i:] + } else { + return "", fpErr("", "purl_base", "carries a type but no name: "+strconv.Quote(purl)) + } + + return "pkg:" + rest, nil +} + +// --------------------------------------------------------------------------- +// Ordinal assignment +// --------------------------------------------------------------------------- + +// SastCandidate pairs a SastInput with the source position used ONLY to order +// identical matches deterministically. Line and Column are never hashed and +// never reach Digest; they exist because "the 0-based index of this match +// among identical matches" needs a total order, and source order is the only +// one a producer and a re-scan both agree on. +// +// KNOWN LIMITATION, inherited from the specification rather than chosen here: +// inserting a THIRD identical call site above two existing ones shifts their +// ordinals and therefore their digests. research/07 §3's matching cascade +// (exact hit, then rule+path+line_hash, then rule+symbol_hash) is what +// recovers identity in that case; the fingerprint alone cannot. The +// alternative — dropping the ordinal — silently LOSES one of the two findings +// on upsert against UNIQUE (target_id, fingerprint), which is worse. +type SastCandidate struct { + Input SastInput + Line int // ordering only; never hashed + Column int // ordering only; never hashed +} + +// AssignSastOrdinals returns a copy of cands' inputs with Ordinal set, in the +// SAME order as cands. Ordinals are assigned per +// (TargetID, RuleIDVersioned, RepoRelPath, normalized match) group in +// ascending (Line, Column, original index) order. +// +// The group key adds TargetID to the specification's (rule, path, normalized) +// key because the specification's grouping is implicitly per-target; passing +// two targets' candidates in one slice would otherwise cross-index them. +// +// Any input whose fields are invalid is returned as an error rather than +// silently given ordinal 0, because a bad ordinal is invisible in the digest. +func AssignSastOrdinals(cands []SastCandidate) ([]SastInput, error) { + type keyed struct { + idx int + key string + } + + out := make([]SastInput, len(cands)) + order := make([]keyed, 0, len(cands)) + + for i, c := range cands { + // Validate through the real field builder so an input that cannot be + // fingerprinted is rejected here rather than at hash time. + if _, err := SastFields(c.Input); err != nil { + return nil, fmt.Errorf("candidate %d: %w", i, err) + } + out[i] = c.Input + order = append(order, keyed{ + idx: i, + key: strings.Join([]string{ + c.Input.TargetID, + c.Input.RuleIDVersioned, + CanonicalRepoRelPath(c.Input.RepoRelPath), + NormalizeMatch(c.Input.Snippet), + }, FingerprintFieldSeparator), + }) + } + + sort.SliceStable(order, func(a, b int) bool { + if order[a].key != order[b].key { + return order[a].key < order[b].key + } + ca, cb := cands[order[a].idx], cands[order[b].idx] + if ca.Line != cb.Line { + return ca.Line < cb.Line + } + if ca.Column != cb.Column { + return ca.Column < cb.Column + } + return order[a].idx < order[b].idx + }) + + prevKey := "" + ordinal := 0 + for i, k := range order { + if i == 0 || k.key != prevKey { + ordinal = 0 + prevKey = k.key + } + out[k.idx].Ordinal = ordinal + ordinal++ + } + + return out, nil +} + +// --------------------------------------------------------------------------- +// Match normalization +// --------------------------------------------------------------------------- + +// NormalizeMatch abstracts a raw matched source snippet into the form the +// SAST tier hashes. It is modelled on Semgrep's `match_based_id` +// (research/07 §3 [S3]), which is the externally-verified mechanism for +// surviving reindentation and metavariable-only edits. +// +// The algorithm, in one pass, left to right. Any re-implementation (R.16) +// must reproduce it exactly: +// +// 1. CRLF and CR are folded to LF. +// 2. A whitespace run emits a single space. +// 3. A line comment ("//..." or "#..." to end of line) and a block comment +// ("/*...*/", unterminated meaning to end of input) are dropped and emit +// a single space in their place. +// 4. A string literal delimited by ", ' or ` emits NormalizedStringToken. +// Backslash escapes are honoured inside " and ' but not inside `. +// 5. A token starting with an ASCII digit emits NormalizedNumberToken. It +// consumes following letters, digits, '_' and '.', plus a '+' or '-' +// immediately after an 'e' or 'E', so 0xFF, 1_000, 3.14f and 1e-9 are all +// one token. +// 6. An identifier ([\p{L}_$][\p{L}\p{N}_$]*) emits: +// a. itself, if it is a reserved word (the union list below); +// b. itself, if the preceding non-space output ends in ".", "->" or "::" +// — it is a member or qualified name, which is API surface, not churn; +// c. itself, if the next non-space input is "::" — it is a namespace or +// type qualifier, which is API surface for the same reason; +// d. itself, if the next non-space input character is "(" — it is a +// callee name, which is the sink the rule actually matched; +// e. otherwise "$N", where N counts distinct such identifiers in +// first-occurrence order, with every later occurrence of the same +// spelling mapping to the same "$N". +// 7. Any other character is emitted verbatim. +// 8. The result is trimmed of leading and trailing spaces. +// +// WHY 6(b), 6(c) AND 6(d) EXIST — the specification says "replace LOCAL +// identifiers", and this is what "local" is taken to mean. Replacing every +// identifier would normalise `request.getParameter(userInput)` and +// `config.getName(key)` to the same string, destroying nearly all +// discriminating power and pushing the whole burden of distinguishing +// findings onto `ordinal`, which is the least stable field in the tier. +// Preserving member, qualifier and callee names keeps the sink identifiable +// while still abstracting the variable names that a refactor renames. +// +// WHY 6(c) TREATS "::" DIFFERENTLY FROM "." AND "->". The LEFT operand of +// "::" is a namespace or type name in every language that has the operator +// (C++, Rust, PHP, Ruby) — it is never a local variable, so abstracting it +// violates "replace local identifiers" outright and collapses +// `Ns::Helper(v)` and `Other::Helper(v)` — two calls into two different +// namespaces — onto one digest. The left operand of "." or "->" is usually a +// receiver bound to a local (`db.Query(q)`, `p->Field`), so it is abstracted; +// a lexer cannot tell a Go package qualifier from a receiver variable, and +// guessing wrong in the other direction would fork the digest of unchanged +// code, which is the worse failure. +// +// KNOWN, ACCEPTED LIMITATIONS. This is one language-agnostic lexer, not N +// parsers, because Anvil's SAST tier is an opengrep subprocess that returns +// text (plan/00-SPINE.md S12: opengrep "is an OCaml CLI with zero bindings in +// any language"). Determinism, not semantic perfection, is what identity +// needs — the same snippet must normalise the same way on every scan, and it +// does: +// +// - '#' is treated as a line comment, so a C/C++ preprocessor directive or +// a C# '#region' inside a snippet is dropped. Match snippets rarely +// contain them. +// - "'" is treated as a string delimiter, so a Rust lifetime ('static) or a +// Lisp quote consumes to the next "'". +// - the reserved-word list is a UNION across languages, so an identifier +// named `class` in a language where it is not reserved is preserved +// rather than abstracted. +// +// Each of these is stable under re-scan, which is the property that matters. +func NormalizeMatch(snippet string) string { + src := strings.ReplaceAll(snippet, "\r\n", "\n") + src = strings.ReplaceAll(src, "\r", "\n") + + in := []rune(src) + n := len(in) + out := make([]rune, 0, n) + + emit := func(s string) { + out = append(out, []rune(s)...) + } + emitSpace := func() { + if len(out) > 0 && out[len(out)-1] != ' ' { + out = append(out, ' ') + } + } + // endsWithSelector reports whether the output so far ends in a member or + // qualified-name selector, ignoring trailing spaces. + endsWithSelector := func() bool { + end := len(out) + for end > 0 && out[end-1] == ' ' { + end-- + } + if end == 0 { + return false + } + if out[end-1] == '.' { + return true + } + if end >= 2 && out[end-2] == '-' && out[end-1] == '>' { + return true + } + if end >= 2 && out[end-2] == ':' && out[end-1] == ':' { + return true + } + return false + } + + metavars := make(map[string]string) + nextMetavar := 1 + + i := 0 + for i < n { + c := in[i] + + switch { + case unicode.IsSpace(c): + for i < n && unicode.IsSpace(in[i]) { + i++ + } + emitSpace() + + case c == '/' && i+1 < n && in[i+1] == '/': + for i < n && in[i] != '\n' { + i++ + } + emitSpace() + + case c == '#': + for i < n && in[i] != '\n' { + i++ + } + emitSpace() + + case c == '/' && i+1 < n && in[i+1] == '*': + i += 2 + for i < n { + if in[i] == '*' && i+1 < n && in[i+1] == '/' { + i += 2 + break + } + i++ + } + emitSpace() + + case c == '"' || c == '\'' || c == '`': + quote := c + i++ + for i < n { + if in[i] == '\\' && quote != '`' { + i += 2 + continue + } + if in[i] == quote { + i++ + break + } + i++ + } + emit(NormalizedStringToken) + + case c >= '0' && c <= '9': + for i < n { + r := in[i] + if unicode.IsLetter(r) || unicode.IsDigit(r) || r == '_' || r == '.' { + i++ + continue + } + if (r == '+' || r == '-') && i > 0 && (in[i-1] == 'e' || in[i-1] == 'E') { + i++ + continue + } + break + } + emit(NormalizedNumberToken) + + case isIdentStart(c): + j := i + for j < n && isIdentPart(in[j]) { + j++ + } + word := string(in[i:j]) + i = j + + switch { + case fingerprintReservedWords[word]: + emit(word) + case endsWithSelector(): + emit(word) + case nextNonSpaceIsScopeResolution(in, i): + emit(word) + case nextNonSpaceIsCallOpen(in, i): + emit(word) + default: + mv, ok := metavars[word] + if !ok { + mv = NormalizedMetavarPrefix + strconv.Itoa(nextMetavar) + nextMetavar++ + metavars[word] = mv + } + emit(mv) + } + + default: + out = append(out, c) + i++ + } + } + + return strings.TrimSpace(string(out)) +} + +func isIdentStart(r rune) bool { + return r == '_' || r == '$' || unicode.IsLetter(r) +} + +func isIdentPart(r rune) bool { + return r == '_' || r == '$' || unicode.IsLetter(r) || unicode.IsDigit(r) +} + +// nextNonSpaceIsCallOpen reports whether the next non-space character at or +// after i is '(' — i.e. the identifier just consumed is a callee name. +func nextNonSpaceIsCallOpen(in []rune, i int) bool { + for i < len(in) && unicode.IsSpace(in[i]) { + i++ + } + return i < len(in) && in[i] == '(' +} + +// nextNonSpaceIsScopeResolution reports whether the next non-space characters +// at or after i are "::" — i.e. the identifier just consumed is the namespace +// or type qualifier on the left of a scope-resolution operator. See rule 6(c) +// in NormalizeMatch's contract: that operand is never a local variable, so +// abstracting it would both violate "replace local identifiers" and merge +// calls into two different namespaces onto one digest. +func nextNonSpaceIsScopeResolution(in []rune, i int) bool { + for i < len(in) && unicode.IsSpace(in[i]) { + i++ + } + return i+1 < len(in) && in[i] == ':' && in[i+1] == ':' +} + +// fingerprintReservedWords is the union of keywords, literal keywords and +// primitive type names across the languages Anvil's SAST tier covers (Go, +// Java, C#, C/C++, JavaScript/TypeScript, Python, Ruby, PHP). Identifiers in +// this set are preserved verbatim rather than abstracted to $N. +// +// It is a UNION on purpose. A per-language list would require knowing the +// language at fingerprint time, which the opengrep subprocess boundary does +// not reliably give us, and a wrong language guess would change the digest of +// unchanged code. A union is a fixed, deterministic function of the token +// text alone. Adding or removing an entry CHANGES EVERY SAST FINGERPRINT and +// is therefore an anvil-fp/v2 event, not a maintenance edit. +var fingerprintReservedWords = map[string]bool{ + // Literal keywords and self-references, across all languages. + "true": true, "false": true, "null": true, "nil": true, "none": true, + "None": true, "True": true, "False": true, "NULL": true, "nullptr": true, + "undefined": true, "self": true, "this": true, "super": true, "cls": true, + "iota": true, "base": true, + + // Control flow and declaration keywords. + "abstract": true, "and": true, "as": true, "assert": true, "async": true, + "await": true, "begin": true, "break": true, "case": true, "catch": true, + "chan": true, "checked": true, "class": true, "clone": true, "const": true, + "constexpr": true, "continue": true, "debugger": true, "declare": true, + "def": true, "default": true, "defer": true, "del": true, "delete": true, + "do": true, "echo": true, "elif": true, "else": true, "elseif": true, + "elsif": true, "end": true, "endforeach": true, "endif": true, + "endwhile": true, "ensure": true, "enum": true, "except": true, + "exit": true, "explicit": true, "export": true, "extends": true, + "extern": true, "fallthrough": true, "final": true, "finally": true, + "fn": true, "for": true, "foreach": true, "friend": true, "from": true, + "func": true, "function": true, "global": true, "go": true, "goto": true, + "if": true, "implements": true, "implicit": true, "import": true, + "in": true, "include": true, "include_once": true, "inline": true, + "instanceof": true, "insteadof": true, "interface": true, "internal": true, + "is": true, "keyof": true, "lambda": true, "let": true, "lock": true, + "match": true, "module": true, "mutable": true, "namespace": true, + "native": true, "new": true, "nonlocal": true, "not": true, + "operator": true, "or": true, "out": true, "override": true, + "package": true, "params": true, "pass": true, "print": true, + "private": true, "protected": true, "public": true, "raise": true, + "range": true, "readonly": true, "redo": true, "ref": true, + "register": true, "require": true, "require_once": true, + "require_relative": true, "rescue": true, "retry": true, "return": true, + "sealed": true, "select": true, "signed": true, "sizeof": true, + "stackalloc": true, "static": true, "strictfp": true, "struct": true, + "switch": true, "synchronized": true, "template": true, "throw": true, + "throws": true, "trait": true, "transient": true, "try": true, + "type": true, "typedef": true, "typeof": true, "unchecked": true, + "union": true, "unless": true, "unsafe": true, "unsigned": true, + "until": true, "use": true, "using": true, "var": true, "virtual": true, + "volatile": true, "when": true, "while": true, "with": true, "xor": true, + "yield": true, + + // Primitive and built-in type names. + "any": true, "bigint": true, "bool": true, "boolean": true, "byte": true, + "char": true, "complex64": true, "complex128": true, "decimal": true, + "double": true, "error": true, "float": true, "float32": true, + "float64": true, "int": true, "int8": true, "int16": true, "int32": true, + "int64": true, "long": true, "never": true, "object": true, "rune": true, + "sbyte": true, "short": true, "string": true, "symbol": true, + "uint": true, "uint8": true, "uint16": true, "uint32": true, + "uint64": true, "uintptr": true, "ulong": true, "unknown": true, + "ushort": true, "void": true, "wchar_t": true, +} diff --git a/internal/record/fingerprint_spec_test.go b/internal/record/fingerprint_spec_test.go new file mode 100644 index 0000000..9a7bbea --- /dev/null +++ b/internal/record/fingerprint_spec_test.go @@ -0,0 +1,349 @@ +package record + +// fingerprint_spec_test.go — the test that keeps FINGERPRINT-SPEC.md honest. +// +// R.3's CRITIQUE-01.md, blocker 1: `normalized_match` was defined only in Go. +// The critic re-implemented plan/40-record-and-storage.md's four-clause spec +// text in Python and got 55e27b07… where the committed golden for sast-01 is +// 13c60ccf…. The orchestrator ruled that the implementation is right and the +// specification incomplete, and that the fix is to write the specification +// down completely and IN TREE — plan/ is gitignored, and a second producer +// working from a clone must be able to read it. +// +// internal/record/FINGERPRINT-SPEC.md is that document. This file is the +// second half of the ruling: "add a test that keeps the spec honest … A spec +// that can drift from the code silently is the same defect one level up." +// +// What is checkable mechanically is checked here: +// +// - the reserved-word list in the document is EXACTLY the one in +// fingerprint.go — same set, no extra, no missing, sorted, no duplicates, +// and the count the prose claims; +// - every algorithm constant the document prints equals the constant in the +// code; +// - the document still carries the anvil-fp/v2 rule, which is the sentence +// that stops someone treating any of the above as a maintenance edit. +// +// What is NOT checkable mechanically — the prose describing the scan order, +// the identifier-disposition clauses and their reasons — is guarded instead by +// the corpus lock in fingerprint_test.go and, eventually, by R.16's +// independent oracle re-implemented FROM THIS DOCUMENT. + +import ( + "fmt" + "os" + "regexp" + "sort" + "strconv" + "strings" + "testing" +) + +const fingerprintSpecPath = "FINGERPRINT-SPEC.md" + +// specBlock returns the lines between "" and +// "", with markdown code fences removed. The markers are +// HTML comments so they are invisible when the document is rendered but +// unambiguous to parse. +func specBlock(t *testing.T, doc, name string) []string { + t.Helper() + + begin := "" + end := "" + + i := strings.Index(doc, begin) + if i < 0 { + t.Fatalf("%s: missing marker %q; the machine-checked %s block is how this "+ + "document is kept from drifting away from fingerprint.go", + fingerprintSpecPath, begin, name) + } + rest := doc[i+len(begin):] + j := strings.Index(rest, end) + if j < 0 { + t.Fatalf("%s: marker %q has no matching %q", fingerprintSpecPath, begin, end) + } + + var out []string + for _, line := range strings.Split(rest[:j], "\n") { + if strings.HasPrefix(strings.TrimSpace(line), "```") { + continue + } + if strings.TrimSpace(line) == "" { + continue + } + out = append(out, line) + } + if len(out) == 0 { + t.Fatalf("%s: block %s is empty", fingerprintSpecPath, name) + } + return out +} + +func readFingerprintSpec(t *testing.T) string { + t.Helper() + b, err := os.ReadFile(fingerprintSpecPath) + if err != nil { + t.Fatalf("reading %s: %v\nThis document is the authoritative definition of anvil-fp/v1 "+ + "and must be in tree: plan/ is gitignored, so a second producer working from a clone "+ + "has nothing else to implement from.", fingerprintSpecPath, err) + } + return strings.ReplaceAll(string(b), "\r\n", "\n") +} + +// TestSpecReservedWordListMatchesTheCode is the ruling's named test. The +// reserved-word list is the single largest undocumented input to +// normalized_match — ~200 entries, every one of which changes the digest of +// any snippet containing it — and it is the reason the critic's from-the-text +// re-implementation diverged. +func TestSpecReservedWordListMatchesTheCode(t *testing.T) { + doc := readFingerprintSpec(t) + lines := specBlock(t, doc, "ANVIL-FP-RESERVED-WORDS") + + var documented []string + for _, line := range lines { + documented = append(documented, strings.Fields(line)...) + } + + // No duplicates. A duplicate is invisible in a set comparison but means + // the human-readable count is wrong and the list has been edited carelessly. + seen := map[string]bool{} + for _, w := range documented { + if seen[w] { + t.Errorf("%s: reserved word %q is listed twice", fingerprintSpecPath, w) + } + seen[w] = true + } + + // Sorted in byte order. Not a correctness property of the algorithm, but it + // makes an added or removed entry a one-line diff instead of a hunt. + if !sort.StringsAreSorted(documented) { + t.Errorf("%s: the reserved-word block is not sorted in byte order; "+ + "sort it so that adding or removing an entry is a readable diff", fingerprintSpecPath) + } + + // The set must be EXACTLY fingerprint.go's. + inCode := make(map[string]bool, len(fingerprintReservedWords)) + for w, v := range fingerprintReservedWords { + if !v { + t.Fatalf("fingerprintReservedWords[%q] is false; every entry must be true", w) + } + inCode[w] = true + } + + var missing, extra []string + for w := range inCode { + if !seen[w] { + missing = append(missing, w) + } + } + for w := range seen { + if !inCode[w] { + extra = append(extra, w) + } + } + sort.Strings(missing) + sort.Strings(extra) + + if len(missing) > 0 { + t.Errorf("%s: %d reserved word(s) are in fingerprint.go but NOT documented: %q\n"+ + "A second producer implementing from this document would abstract them to $N and "+ + "emit a different digest for unchanged code.", + fingerprintSpecPath, len(missing), missing) + } + if len(extra) > 0 { + t.Errorf("%s: %d reserved word(s) are documented but NOT in fingerprint.go: %q\n"+ + "A second producer would preserve them verbatim where this implementation "+ + "abstracts them.", + fingerprintSpecPath, len(extra), extra) + } + + // The count the prose claims must be the real one. + countRe := regexp.MustCompile(`\*\*(\d+) entries\*\*`) + m := countRe.FindStringSubmatch(doc) + if m == nil { + t.Fatalf("%s: the reserved-word section must state the entry count as \"**N entries**\"", + fingerprintSpecPath) + } + claimed, err := strconv.Atoi(m[1]) + if err != nil { + t.Fatalf("%s: unparseable entry count %q", fingerprintSpecPath, m[1]) + } + if claimed != len(inCode) { + t.Errorf("%s: claims %d reserved words, fingerprint.go has %d", + fingerprintSpecPath, claimed, len(inCode)) + } + if len(documented) != len(inCode) { + t.Errorf("%s: block lists %d words, fingerprint.go has %d", + fingerprintSpecPath, len(documented), len(inCode)) + } +} + +// TestSpecConstantsMatchTheCode covers every named value a re-implementation +// must reproduce byte for byte: the separator, the digest length, the four +// normalization tokens, and the two route-templating thresholds. A wrong +// threshold is the subtlest of these — it produces a perfectly valid digest +// that merges or splits routes differently from every other producer. +func TestSpecConstantsMatchTheCode(t *testing.T) { + doc := readFingerprintSpec(t) + lines := specBlock(t, doc, "ANVIL-FP-CONSTANTS") + + documented := map[string]string{} + for _, line := range lines { + k, v, ok := strings.Cut(line, "=") + if !ok { + t.Fatalf("%s: constants block line %q is not \"name = value\"", fingerprintSpecPath, line) + } + k, v = strings.TrimSpace(k), strings.TrimSpace(v) + if _, dup := documented[k]; dup { + t.Errorf("%s: constant %q is listed twice", fingerprintSpecPath, k) + } + documented[k] = v + } + + // "U+001F" rather than a literal control character: the separator cannot be + // written literally into a text document without becoming invisible, and an + // invisible separator in the one document that defines it is how the + // research/07-vs-research/18 conflict happened in the first place. + inCode := map[string]string{ + "FingerprintAlgV1": FingerprintAlgV1, + "FingerprintFieldSeparator": "U+001F", + "FingerprintDigestHexLen": strconv.Itoa(FingerprintDigestHexLen), + "NormalizedStringToken": NormalizedStringToken, + "NormalizedNumberToken": NormalizedNumberToken, + "NormalizedMetavarPrefix": NormalizedMetavarPrefix, + "NormalizedRouteSegmentToken": NormalizedRouteSegmentToken, + "routeHexSegmentMinLen": strconv.Itoa(routeHexSegmentMinLen), + "routeOpaqueSegmentMinLen": strconv.Itoa(routeOpaqueSegmentMinLen), + } + + for name, want := range inCode { + got, ok := documented[name] + if !ok { + t.Errorf("%s: constant %s is not documented; a re-implementation cannot reproduce it", + fingerprintSpecPath, name) + continue + } + if got != want { + t.Errorf("%s: documents %s = %q, the code has %q", fingerprintSpecPath, name, got, want) + } + } + for name := range documented { + if _, ok := inCode[name]; !ok { + t.Errorf("%s: documents constant %s, which does not exist in the code", + fingerprintSpecPath, name) + } + } + + // The separator spelling must actually denote U+001F, not merely agree with + // a string in this test. + if FingerprintFieldSeparator != "\x1f" { + t.Fatalf("FingerprintFieldSeparator = %q, want U+001F", FingerprintFieldSeparator) + } +} + +// TestSpecCarriesTheVersioningRule. fingerprint.go's fingerprintReservedWords +// comment already says that adding or removing an entry "CHANGES EVERY SAST +// FINGERPRINT and is therefore an anvil-fp/v2 event, not a maintenance edit". +// The ruling requires the document to agree, because the document is what a +// second producer reads, and someone editing a word list in a markdown file +// feels far more casual than someone editing a hash function. +func TestSpecCarriesTheVersioningRule(t *testing.T) { + doc := readFingerprintSpec(t) + required := []string{ + "Any change to this document's algorithm is an `anvil-fp/v2` event, never a `v1` edit.", + "anvil-fp/v1", + } + for _, s := range required { + if !strings.Contains(doc, s) { + t.Errorf("%s must contain %q", fingerprintSpecPath, s) + } + } +} + +// TestSpecDocumentsEveryExportedAlgorithmSurface is a cheap guard against the +// failure mode this whole document exists to prevent: a new normalization step +// or tier helper is added to fingerprint.go and nobody writes it down, so the +// spec silently stops being the definition. It names the surfaces a +// re-implementation must know about and asserts each appears in the text. +func TestSpecDocumentsEveryExportedAlgorithmSurface(t *testing.T) { + doc := readFingerprintSpec(t) + surfaces := []string{ + "NormalizeMatch", + "CanonicalRepoRelPath", + "CanonicalRouteTemplate", + "PurlBase", + "ordinal", + "normalized_match", + "route_template", + "rule_id_versioned", + "enclosing_symbol_path", + "injection_point", + "evidence_class_detail", + "purl_base", + "locator", + "http_method", + "target_id", + "detector_kind", + "advisory_id", + "repo_relpath", + "param_name", + } + for _, s := range surfaces { + if !strings.Contains(doc, s) { + t.Errorf("%s does not mention %q; a re-implementation would not know it exists", + fingerprintSpecPath, s) + } + } +} + +// TestSpecWorkedRouteExamplesHold re-runs the §6.5 "Whole routes" examples +// against the implementation. They are transcribed here rather than parsed out +// of the markdown table, so the assertion is on the BEHAVIOUR the document +// promises; if the document's table is edited without the code changing (or +// vice versa) this test and TestCanonicalRouteTemplate disagree with each +// other, which is the signal. +func TestSpecWorkedRouteExamplesHold(t *testing.T) { + examples := [][2]string{ + {"/api/v1/users/12345/orders", "/api/v1/users//orders"}, + {"/api/v1/users/{id}/orders", "/api/v1/users//orders"}, + {"/api/v1/users/:userId/orders", "/api/v1/users//orders"}, + {"api//v1/users/12345//orders/?debug=1", "/api/v1/users//orders"}, + {"/12345/orders/6789", "//orders/"}, + {"/api/v1/users/me/orders", "/api/v1/users/me/orders"}, + {"/", "/"}, + } + for _, ex := range examples { + if got := CanonicalRouteTemplate(ex[0]); got != ex[1] { + t.Errorf("%s section 6.5 promises CanonicalRouteTemplate(%q) = %q, got %q", + fingerprintSpecPath, ex[0], ex[1], got) + } + } + + // And the §3.7 normalization examples, same reasoning. + norms := [][2]string{ + {`rows, err := db.Query("SELECT * FROM users WHERE name = '" + name + "'")`, + `$1, $2 := $3.Query( + $4 + )`}, + {`os.system("rm -rf " + path)`, `$1.system( + $2)`}, + {`a = b + a + b`, `$1 = $2 + $1 + $2`}, + {`for i := range items { return nil }`, `for $1 := range $2 { return nil }`}, + {`a = 0xFF + 1_000 + 3.14f + 1e-9`, `$1 = + + + `}, + {`p->Field = Ns::Helper(v)`, `$1->Field = Ns::Helper($2)`}, + {`std::vector v = Foo::Bar::make(x)`, `std::vector $1 = Foo::Bar::make($2)`}, + {`Ns::Helper(v)`, `Ns::Helper($1)`}, + {`Other::Helper(v)`, `Other::Helper($1)`}, + {`totalCount := len(itemsList)`, `$1 := len($2)`}, + } + for _, ex := range norms { + if got := NormalizeMatch(ex[0]); got != ex[1] { + t.Errorf("%s section 3.7 promises NormalizeMatch(%q) = %q, got %q", + fingerprintSpecPath, ex[0], ex[1], got) + } + } + + // Sanity: the two examples the document uses to argue rule 6(c) must not + // collide, or the argument in the document is false. + if NormalizeMatch("Ns::Helper(v)") == NormalizeMatch("Other::Helper(v)") { + t.Fatal("Ns::Helper and Other::Helper normalise identically; " + + fmt.Sprintf("%s clause (c)'s stated reason does not hold", fingerprintSpecPath)) + } +} diff --git a/internal/record/fingerprint_test.go b/internal/record/fingerprint_test.go new file mode 100644 index 0000000..f90da9e --- /dev/null +++ b/internal/record/fingerprint_test.go @@ -0,0 +1,1464 @@ +package record + +// fingerprint_test.go — R.2's own tests. +// +// These are NOT the conformance test. R.16 owns that +// (internal/record/fingerprint_conformance_test.go plus +// testdata/fingerprint_corpus/*.golden), and its oracle must be an offline +// re-implementation of internal/record/FINGERPRINT-SPEC.md that never imports +// this package. What is here is: +// +// - a corpus lock: every fixture's committed hashed_fields and +// expected_digest must still be what the implementation produces; +// - determinism across two consecutive runs (R.2's stop condition) AND +// across two separate OS processes, which is the form that can actually +// detect a per-process source of nondeterminism such as map iteration +// order; +// - the mutation tests that prove the forbidden inputs are not hashed: +// line/column numbers for SAST, the version string for SCA and host, +// host/port/payload for DAST; +// - a frozen-shape test on the four input structs, so a later edit cannot +// quietly add a field that reintroduces a volatile input. +// +// THE COMMITTED GOLDENS WERE NOT PRODUCED BY THIS PACKAGE. Each fixture's +// `hashed_fields` list was derived by hand from the algorithm text — now +// internal/record/FINGERPRINT-SPEC.md, which is the authoritative and COMPLETE +// definition; plan/40-record-and-storage.md's four-clause summary was proved +// insufficient by CRITIQUE-01 — and its `expected_digest` was computed from +// that list by an offline script that only joins with U+001F and SHA-256s. +// The lock below is therefore a genuine two-sided check, and there is +// deliberately no test here that can regenerate it — see the note further +// down. +// +// TWO DAST GOLDENS WERE RE-DERIVED ON 2026-08-08 under the R.3 blocker-2 +// ruling, when CanonicalRouteTemplate began deriving `route_template` as the +// specification always said it should: +// +// dast-01 ca801b8d… → 199c3b5f… route_template "/api/v1/users/{id}/orders" +// → "/api/v1/users//orders" +// dast-03 84fe311d… → 5fc15c55… route_template "/files/{path}" +// → "/files/" +// +// dast-02 ("/search") was unaffected, and its untouched golden reproducing +// exactly under the same offline script is the control that shows the script +// was not quietly rewritten to agree with the Go code. No SAST, SCA or host +// digest moved. Nothing had been stored under the old digests — anvil-fp/v1 +// has not shipped — so this was a correction of an unimplemented spec clause, +// not an anvil-fp/v2 event. +// +// R.16: do NOT derive the .golden files from `expected_digest` either. Even +// though it is independent of the Go code, copying it would make the +// conformance gate a transcription check rather than a re-derivation. + +import ( + "bytes" + "encoding/json" + "fmt" + "os" + "os/exec" + "path/filepath" + "reflect" + "regexp" + "sort" + "strings" + "testing" +) + +const corpusDir = "../../testdata/fingerprint_corpus" + +// --------------------------------------------------------------------------- +// Corpus loading +// --------------------------------------------------------------------------- + +type corpusMutation struct { + Name string `json:"name"` + Description string `json:"description"` + Input json.RawMessage `json:"input"` + NotHashed json.RawMessage `json:"not_hashed,omitempty"` +} + +type corpusFixture struct { + ID string `json:"id"` + Tier string `json:"tier"` + Description string `json:"description"` + Input json.RawMessage `json:"input"` + NotHashed json.RawMessage `json:"not_hashed,omitempty"` + HashedFields []string `json:"hashed_fields"` + ExpectedDigest string `json:"expected_digest"` + Mutations []corpusMutation `json:"mutations,omitempty"` + Notes []string `json:"notes,omitempty"` + + path string +} + +type jsonSastInput struct { + TargetID string `json:"target_id"` + RuleIDVersioned string `json:"rule_id_versioned"` + RepoRelPath string `json:"repo_rel_path"` + EnclosingSymbolPath string `json:"enclosing_symbol_path"` + Snippet string `json:"snippet"` + Ordinal int `json:"ordinal"` +} + +type jsonScaInput struct { + TargetID string `json:"target_id"` + AdvisoryID string `json:"advisory_id"` + Purl string `json:"purl"` + ManifestRelPath string `json:"manifest_rel_path"` +} + +type jsonHostInput struct { + TargetID string `json:"target_id"` + AdvisoryID string `json:"advisory_id"` + Purl string `json:"purl"` + PackageManager string `json:"package_manager"` + HostIdentifier string `json:"host_identifier"` +} + +type jsonDastInput struct { + TargetID string `json:"target_id"` + RuleIDVersioned string `json:"rule_id_versioned"` + HTTPMethod string `json:"http_method"` + RouteTemplate string `json:"route_template"` + InjectionPoint string `json:"injection_point"` + ParamName string `json:"param_name"` + EvidenceSignal string `json:"evidence_signal"` +} + +// strictUnmarshal rejects unknown keys, so a misspelled fixture key fails the +// test instead of silently defaulting a hashed field to the empty string and +// locking in a wrong digest. +func strictUnmarshal(raw json.RawMessage, dst any) error { + dec := json.NewDecoder(bytes.NewReader(raw)) + dec.DisallowUnknownFields() + return dec.Decode(dst) +} + +// fieldsFor decodes a fixture input for the named tier and returns the exact +// ordered field list that tier hashes. +func fieldsFor(tier string, raw json.RawMessage) ([]string, error) { + switch tier { + case "sast": + var in jsonSastInput + if err := strictUnmarshal(raw, &in); err != nil { + return nil, err + } + return SastFields(SastInput{ + TargetID: in.TargetID, + RuleIDVersioned: in.RuleIDVersioned, + RepoRelPath: in.RepoRelPath, + EnclosingSymbolPath: in.EnclosingSymbolPath, + Snippet: in.Snippet, + Ordinal: in.Ordinal, + }) + case "sca": + var in jsonScaInput + if err := strictUnmarshal(raw, &in); err != nil { + return nil, err + } + return ScaFields(ScaInput{ + TargetID: in.TargetID, + AdvisoryID: in.AdvisoryID, + Purl: in.Purl, + ManifestRelPath: in.ManifestRelPath, + }) + case "host": + var in jsonHostInput + if err := strictUnmarshal(raw, &in); err != nil { + return nil, err + } + return HostFields(HostInput{ + TargetID: in.TargetID, + AdvisoryID: in.AdvisoryID, + Purl: in.Purl, + PackageManager: in.PackageManager, + HostIdentifier: in.HostIdentifier, + }) + case "dast": + var in jsonDastInput + if err := strictUnmarshal(raw, &in); err != nil { + return nil, err + } + return DastFields(DastInput{ + TargetID: in.TargetID, + RuleIDVersioned: in.RuleIDVersioned, + HTTPMethod: in.HTTPMethod, + RouteTemplate: in.RouteTemplate, + InjectionPoint: InjectionPoint(in.InjectionPoint), + ParamName: in.ParamName, + EvidenceSignal: EvidenceSignal(in.EvidenceSignal), + }) + default: + return nil, &FingerprintError{Field: "tier", Msg: "unknown tier " + tier} + } +} + +// digestFor is the tier-dispatching equivalent of the exported Sast/Sca/ +// Host/Dast entry points, used so a fixture can be driven by its declared +// tier. The public entry points are exercised directly in +// TestExportedTierEntryPointsAgreeWithFieldBuilders. +func digestFor(tier string, raw json.RawMessage) (string, error) { + fields, err := fieldsFor(tier, raw) + if err != nil { + return "", err + } + return Digest(fields...) +} + +func loadCorpus(t *testing.T) []corpusFixture { + t.Helper() + + paths, err := filepath.Glob(filepath.Join(corpusDir, "*.json")) + if err != nil { + t.Fatalf("globbing corpus: %v", err) + } + if len(paths) == 0 { + t.Fatalf("no fixtures found in %s; the fixed corpus is mandatory (plan/00-SPINE.md S6)", corpusDir) + } + sort.Strings(paths) + + out := make([]corpusFixture, 0, len(paths)) + for _, p := range paths { + b, err := os.ReadFile(p) + if err != nil { + t.Fatalf("reading %s: %v", p, err) + } + var f corpusFixture + dec := json.NewDecoder(bytes.NewReader(b)) + dec.DisallowUnknownFields() + if err := dec.Decode(&f); err != nil { + t.Fatalf("decoding %s: %v", p, err) + } + f.path = p + if f.ID == "" || f.Tier == "" { + t.Fatalf("%s: fixture must declare id and tier", p) + } + out = append(out, f) + } + return out +} + +// --------------------------------------------------------------------------- +// The corpus lock +// --------------------------------------------------------------------------- + +func TestCorpusCoverage(t *testing.T) { + // plan/40-record-and-storage.md R.2: "at minimum 2 SAST, 2 DAST, 1 SCA, + // 1 host". A tier with no fixture is a tier nothing guards. + want := map[string]int{"sast": 2, "dast": 2, "sca": 1, "host": 1} + got := map[string]int{} + for _, f := range loadCorpus(t) { + got[f.Tier]++ + } + for tier, min := range want { + if got[tier] < min { + t.Errorf("tier %q: corpus has %d fixtures, the specification requires at least %d", + tier, got[tier], min) + } + } +} + +func TestCorpusFixturesProduceTheirDocumentedDigest(t *testing.T) { + for _, f := range loadCorpus(t) { + t.Run(f.ID, func(t *testing.T) { + if f.ExpectedDigest == "" { + t.Fatalf("%s: expected_digest is empty; every fixture must carry a golden "+ + "derived independently of this implementation", f.path) + } + if err := ValidateDigest(f.ExpectedDigest); err != nil { + t.Fatalf("%s: committed expected_digest is malformed: %v", f.path, err) + } + + fields, err := fieldsFor(f.Tier, f.Input) + if err != nil { + t.Fatalf("%s: building fields: %v", f.path, err) + } + if !reflect.DeepEqual(fields, f.HashedFields) { + t.Errorf("%s: hashed field list drifted.\n got: %q\n want: %q", + f.path, fields, f.HashedFields) + } + + got, err := Digest(fields...) + if err != nil { + t.Fatalf("%s: hashing: %v", f.path, err) + } + if got != f.ExpectedDigest { + t.Errorf("%s: digest changed.\n got: %s\n want: %s\n"+ + "A changed digest means every stored finding for this tier loses its identity; "+ + "this is an anvil-fp/v2 event, not a fixture update.", + f.path, got, f.ExpectedDigest) + } + if err := ValidateDigest(got); err != nil { + t.Errorf("%s: computed digest is malformed: %v", f.path, err) + } + }) + } +} + +// TestCorpusDigestsAreStableAcrossTwoConsecutiveRuns is R.2's stop condition +// verbatim. It catches map-iteration order or any other nondeterminism +// leaking into the digest — NormalizeMatch uses a map for metavariable +// assignment, and if that assignment ever became iteration-ordered this test +// would fail. +func TestCorpusDigestsAreStableAcrossTwoConsecutiveRuns(t *testing.T) { + for _, f := range loadCorpus(t) { + first, err := digestFor(f.Tier, f.Input) + if err != nil { + t.Fatalf("%s: run 1: %v", f.path, err) + } + for run := 2; run <= 8; run++ { + next, err := digestFor(f.Tier, f.Input) + if err != nil { + t.Fatalf("%s: run %d: %v", f.path, run, err) + } + if next != first { + t.Fatalf("%s: digest is not deterministic: run 1 = %s, run %d = %s", + f.path, first, run, next) + } + } + } +} + +// crossProcessEnv puts this test binary into "child" mode: it prints one +// crossProcessMarker line per fixture and exits, so the parent can compare a +// digest computed in a DIFFERENT OS process against its own. +const ( + crossProcessEnv = "ANVIL_FINGERPRINT_CROSS_PROCESS_CHILD" + crossProcessMarker = "ANVIL-FP-DIGEST\t" +) + +// TestCorpusDigestsAreStableAcrossProcesses is the stronger form of the stop +// condition. Repeating a computation inside ONE process cannot detect the +// failure that actually matters here, because the things that would break +// cross-process stability are all per-process constants: Go's map iteration +// seed (re-randomised per process, so an unsorted range over a map returns a +// stable-but-wrong order within a run and a different one in the next), +// pointer addresses, and any address-space or locale state. Two producers +// emitting different digests is exactly the failure 00-SPINE.md S6 says fails +// silently forever, and the two producers are two processes. +// +// The child is this same test binary re-executed with crossProcessEnv set, +// which needs no build step and no new dependency. +func TestCorpusDigestsAreStableAcrossProcesses(t *testing.T) { + fixtures := loadCorpus(t) + + if os.Getenv(crossProcessEnv) == "1" { + for _, f := range fixtures { + d, err := digestFor(f.Tier, f.Input) + if err != nil { + fmt.Printf("%s%s\tERROR: %v\n", crossProcessMarker, f.ID, err) + continue + } + fmt.Printf("%s%s\t%s\n", crossProcessMarker, f.ID, d) + } + return + } + + want := make(map[string]string, len(fixtures)) + for _, f := range fixtures { + d, err := digestFor(f.Tier, f.Input) + if err != nil { + t.Fatalf("%s: parent process: %v", f.path, err) + } + want[f.ID] = d + } + + cmd := exec.Command(os.Args[0], + "-test.run=^TestCorpusDigestsAreStableAcrossProcesses$", + "-test.count=1") + cmd.Env = append(os.Environ(), crossProcessEnv+"=1") + out, err := cmd.CombinedOutput() + if err != nil { + t.Fatalf("re-executing the test binary as a child process failed: %v\n%s", err, out) + } + + got := make(map[string]string, len(fixtures)) + for _, line := range strings.Split(string(out), "\n") { + line = strings.TrimSpace(line) + if !strings.HasPrefix(line, strings.TrimSpace(crossProcessMarker)) { + continue + } + parts := strings.Split(line, "\t") + if len(parts) != 3 { + t.Fatalf("malformed child output line %q", line) + } + got[parts[1]] = parts[2] + } + + if len(got) != len(want) { + t.Fatalf("child reported %d digests, parent computed %d\nchild output:\n%s", + len(got), len(want), out) + } + for id, w := range want { + if got[id] != w { + t.Errorf("fixture %s: digest differs between processes.\n parent: %s\n child: %s\n"+ + "anvil-fp/v1 must be a pure function of its hashed fields; two producers "+ + "emitting different digests breaks regression matching silently and forever.", + id, w, got[id]) + } + } +} + +// TestCorpusMutationsDoNotChangeTheDigest is the mutation test R.2's +// validation requirement names. Every fixture's `mutations` array holds inputs +// that differ ONLY in fields the specification forbids hashing — line and +// column numbers, indentation, comments, the dependency version string, the +// HTTP method's case, a query string on a route template. Each must produce +// the fixture's digest unchanged. +func TestCorpusMutationsDoNotChangeTheDigest(t *testing.T) { + total := 0 + for _, f := range loadCorpus(t) { + for _, m := range f.Mutations { + total++ + t.Run(f.ID+"/"+m.Name, func(t *testing.T) { + got, err := digestFor(f.Tier, m.Input) + if err != nil { + t.Fatalf("%s mutation %q: %v", f.path, m.Name, err) + } + if got != f.ExpectedDigest { + t.Errorf("%s mutation %q (%s) changed the digest.\n got: %s\n want: %s", + f.path, m.Name, m.Description, got, f.ExpectedDigest) + } + }) + } + } + if total == 0 { + t.Fatal("no mutations in the corpus; the volatile-field exclusions are then untested") + } +} + +// TestCorpusFixturesHaveDistinctDigests guards the failure that motivates the +// ordinal field: two fixtures that ought to be different findings must not +// share an identity. +func TestCorpusFixturesHaveDistinctDigests(t *testing.T) { + seen := map[string]string{} + for _, f := range loadCorpus(t) { + d, err := digestFor(f.Tier, f.Input) + if err != nil { + t.Fatalf("%s: %v", f.path, err) + } + if prev, ok := seen[d]; ok { + t.Errorf("fixtures %q and %q collide on digest %s", prev, f.ID, d) + } + seen[d] = f.ID + } +} + +// THERE IS DELIBERATELY NO "UPDATE THE GOLDENS" TEST HERE, AND ONE MUST NOT BE +// ADDED. +// +// A golden that the implementation under test can regenerate proves nothing: +// the next time a digest changes, the cheapest way to make this package green +// is to re-seal the fixtures, and the change that was supposed to be caught +// ships silently. That is not a hypothetical — research/07 and research/18 +// shipped two different algorithms under one `anvil-fp/v1` name and nothing +// surfaced it, which is why this corpus exists at all. +// +// The committed `hashed_fields` lists were derived BY HAND from the algorithm +// text in plan/40-record-and-storage.md, and each `expected_digest` was then +// computed from that list by an offline script that joins with U+001F and +// SHA-256s — no Go code involved. To change a fixture, redo that derivation. +// To change a digest, ship anvil-fp/v2 and follow the dual-write migration +// protocol; do not edit the golden. + +// --------------------------------------------------------------------------- +// The exclusions, tested directly rather than only through fixtures +// --------------------------------------------------------------------------- + +// TestSastDigestIgnoresPositionAndFormatting is the specific mutation +// assertion R.2 requires: "a line-number-only change to a SAST fixture leaves +// the digest unchanged". SastInput has no line field at all, so the strongest +// form of the test is to change everything a line-number change implies — +// leading blank lines, indentation, line breaks, and a comment naming the old +// line — and show the digest holds. +func TestSastDigestIgnoresPositionAndFormatting(t *testing.T) { + base := SastInput{ + TargetID: "t-0001", + RuleIDVersioned: "opengrep.go.sqli@2026.07.1", + RepoRelPath: "internal/api/store.go", + EnclosingSymbolPath: "internal/api/store.go::Store.Find", + Snippet: "q := \"SELECT * FROM u WHERE n = '\" + name + \"'\"\nrows, err := db.Query(q)", + Ordinal: 0, + } + want, err := Sast(base) + if err != nil { + t.Fatalf("base: %v", err) + } + + cases := []struct { + name string + snippet string + }{ + { + name: "shifted down by 40 lines", + snippet: strings.Repeat("\n", 40) + base.Snippet, + }, + { + name: "reindented from tabs to eight spaces", + snippet: " q := \"SELECT * FROM u WHERE n = '\" + name + \"'\"\n rows, err := db.Query(q)", + }, + { + name: "CRLF line endings", + snippet: strings.ReplaceAll(base.Snippet, "\n", "\r\n"), + }, + { + name: "a comment naming the old line number added", + snippet: "// was line 128, now line 173\n" + base.Snippet + " // moved", + }, + { + name: "collapsed onto one line", + snippet: strings.ReplaceAll(base.Snippet, "\n", " "), + }, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + m := base + m.Snippet = tc.snippet + got, err := Sast(m) + if err != nil { + t.Fatalf("%v", err) + } + if got != want { + t.Errorf("digest changed under a position/formatting-only edit\n got: %s\n want: %s\n normalized: %q vs %q", + got, want, NormalizeMatch(tc.snippet), NormalizeMatch(base.Snippet)) + } + }) + } +} + +// TestSastDigestChangesWhenTheCodeChanges is the counterpart: normalization +// must not be so aggressive that a genuinely different sink hashes the same. +func TestSastDigestChangesWhenTheCodeChanges(t *testing.T) { + base := SastInput{ + TargetID: "t-0001", + RuleIDVersioned: "opengrep.go.sqli@2026.07.1", + RepoRelPath: "internal/api/store.go", + EnclosingSymbolPath: "internal/api/store.go::Store.Find", + Snippet: "rows, err := db.Query(q)", + } + baseDigest, err := Sast(base) + if err != nil { + t.Fatalf("%v", err) + } + + variants := map[string]func(*SastInput){ + "different callee": func(in *SastInput) { in.Snippet = "rows, err := db.Exec(q)" }, + "different receiver field": func(in *SastInput) { in.Snippet = "rows, err := db.conn.Query(q)" }, + "different rule version": func(in *SastInput) { in.RuleIDVersioned = "opengrep.go.sqli@2026.08.1" }, + "different file": func(in *SastInput) { in.RepoRelPath = "internal/api/other.go" }, + "different symbol": func(in *SastInput) { in.EnclosingSymbolPath = "internal/api/store.go::Store.List" }, + "different target": func(in *SastInput) { in.TargetID = "t-0002" }, + "different ordinal": func(in *SastInput) { in.Ordinal = 1 }, + } + for name, mutate := range variants { + t.Run(name, func(t *testing.T) { + m := base + mutate(&m) + got, err := Sast(m) + if err != nil { + t.Fatalf("%v", err) + } + if got == baseDigest { + t.Errorf("%s did not change the digest; two distinct findings would share one identity", name) + } + }) + } +} + +// TestPackageDigestIgnoresTheVersionString is the rule the specification +// calls out most emphatically: "bumping 1.2.3 to 1.2.4 while still inside the +// vulnerable range must not mint a new finding". +func TestPackageDigestIgnoresTheVersionString(t *testing.T) { + sca := ScaInput{ + TargetID: "t-0001", + AdvisoryID: "CVE-2021-44228", + Purl: "pkg:maven/org.apache.logging.log4j/log4j-core", + ManifestRelPath: "services/api/pom.xml", + } + want, err := Sca(sca) + if err != nil { + t.Fatalf("%v", err) + } + + for _, purl := range []string{ + "pkg:maven/org.apache.logging.log4j/log4j-core@2.14.0", + "pkg:maven/org.apache.logging.log4j/log4j-core@2.14.1", + "pkg:maven/org.apache.logging.log4j/log4j-core@2.15.0", + "pkg:maven/org.apache.logging.log4j/log4j-core@2.14.1?type=jar", + "pkg:maven/org.apache.logging.log4j/log4j-core@2.14.1?type=jar#sub/path", + "PKG:MAVEN/org.apache.logging.log4j/log4j-core@2.14.1", + } { + m := sca + m.Purl = purl + got, err := Sca(m) + if err != nil { + t.Fatalf("%s: %v", purl, err) + } + if got != want { + t.Errorf("purl %q changed the SCA digest; the version string must never be hashed\n got: %s\n want: %s", + purl, got, want) + } + } + + host := HostInput{ + TargetID: "t-0001", + AdvisoryID: "CVE-2022-0778", + Purl: "pkg:deb/debian/openssl", + PackageManager: "apt", + HostIdentifier: "openssl", + } + hostWant, err := Host(host) + if err != nil { + t.Fatalf("%v", err) + } + for _, purl := range []string{ + "pkg:deb/debian/openssl@1.1.1n-0+deb11u3", + "pkg:deb/debian/openssl@1.1.1n-0+deb11u4", + "pkg:deb/debian/openssl@1.1.1w-0+deb11u1?arch=amd64", + } { + m := host + m.Purl = purl + got, err := Host(m) + if err != nil { + t.Fatalf("%s: %v", purl, err) + } + if got != hostWant { + t.Errorf("purl %q changed the host digest; the version string must never be hashed", purl) + } + } +} + +// TestScaAndHostTiersDoNotCollide: the two tiers share one formula, so the +// detector-kind field is the only thing keeping a repo dependency and a host +// package with the same advisory apart. If it were dropped, a host finding +// (remediable_by_agent=false, plan/00-SPINE.md S7) could upsert over an +// agent-remediable dependency finding. +func TestScaAndHostTiersDoNotCollide(t *testing.T) { + scaDigest, err := Sca(ScaInput{ + TargetID: "t-1", + AdvisoryID: "CVE-2022-0778", + Purl: "pkg:deb/debian/openssl", + ManifestRelPath: "apt:openssl", + }) + if err != nil { + t.Fatalf("%v", err) + } + hostDigest, err := Host(HostInput{ + TargetID: "t-1", + AdvisoryID: "CVE-2022-0778", + Purl: "pkg:deb/debian/openssl", + PackageManager: "apt", + HostIdentifier: "openssl", + }) + if err != nil { + t.Fatalf("%v", err) + } + if scaDigest == hostDigest { + t.Fatal("sca and host tiers collided on identical locators; detector_kind is not being hashed") + } +} + +// TestDastDigestIgnoresVolatileTransportDetail. Host, port, scheme and the +// concrete payload are absent from DastInput by construction; what remains +// testable is that the canonicalisation drops the volatile parts a producer +// might still smuggle in through the route template, and that method case +// does not fork identity. +func TestDastDigestIgnoresVolatileTransportDetail(t *testing.T) { + base := DastInput{ + TargetID: "t-0001", + RuleIDVersioned: "nuclei:sqli-error-based@a1b2c3d", + HTTPMethod: "POST", + RouteTemplate: "/api/v1/users/{id}/orders", + InjectionPoint: InjectionPointBody, + ParamName: "sortBy", + EvidenceSignal: EvidenceSignalDBErrorString, + } + want, err := Dast(base) + if err != nil { + t.Fatalf("%v", err) + } + + cases := map[string]func(*DastInput){ + "lowercase method": func(in *DastInput) { in.HTTPMethod = "post" }, + "method with whitespace": func(in *DastInput) { in.HTTPMethod = " POST " }, + "route with query string": func(in *DastInput) { in.RouteTemplate = "/api/v1/users/{id}/orders?debug=1&payload=%27" }, + "route with fragment": func(in *DastInput) { in.RouteTemplate = "/api/v1/users/{id}/orders#frag" }, + "route with trailing slash": func(in *DastInput) { + in.RouteTemplate = "/api/v1/users/{id}/orders/" + }, + "route with duplicate slashes": func(in *DastInput) { + in.RouteTemplate = "//api//v1/users/{id}/orders" + }, + "route without leading slash": func(in *DastInput) { in.RouteTemplate = "api/v1/users/{id}/orders" }, + + // The blocker-2 cases: the concrete route the scanner actually + // requested, and the two other syntaxes a second producer would use + // for the same route. All four are one finding. + "route with a concrete numeric id": func(in *DastInput) { in.RouteTemplate = "/api/v1/users/12345/orders" }, + "route with a different numeric id": func(in *DastInput) { in.RouteTemplate = "/api/v1/users/4192/orders" }, + "route with a concrete uuid": func(in *DastInput) { + in.RouteTemplate = "/api/v1/users/3f2504e0-4f89-11d3-9a0c-0305e82c3301/orders" + }, + "route with a concrete sha1 object id": func(in *DastInput) { + in.RouteTemplate = "/api/v1/users/da39a3ee5e6b4b0d3255bfef95601890afd80709/orders" + }, + "route in express placeholder syntax": func(in *DastInput) { in.RouteTemplate = "/api/v1/users/:id/orders" }, + "route in flask placeholder syntax": func(in *DastInput) { + in.RouteTemplate = "/api/v1/users//orders" + }, + "route already carrying the canonical token": func(in *DastInput) { + in.RouteTemplate = "/api/v1/users/" + NormalizedRouteSegmentToken + "/orders" + }, + "route with a differently NAMED placeholder": func(in *DastInput) { + in.RouteTemplate = "/api/v1/users/{userId}/orders" + }, + } + for name, mutate := range cases { + t.Run(name, func(t *testing.T) { + m := base + mutate(&m) + got, err := Dast(m) + if err != nil { + t.Fatalf("%v", err) + } + if got != want { + t.Errorf("%s changed the DAST digest\n got: %s\n want: %s", name, got, want) + } + }) + } +} + +// TestDastRouteTemplatingKeepsDistinctRoutesDistinct is the counterpart to the +// blocker-2 cases above. Route templating exists to merge one defect's +// instances; it must not merge two defects. Losing a finding on upsert against +// UNIQUE (target_id, fingerprint) is worse than churning one, and it is silent. +func TestDastRouteTemplatingKeepsDistinctRoutesDistinct(t *testing.T) { + base := DastInput{ + TargetID: "t-0001", + RuleIDVersioned: "nuclei:idor@a1b2c3d", + HTTPMethod: "GET", + RouteTemplate: "/api/v1/users/12345/orders", + InjectionPoint: InjectionPointPath, + EvidenceSignal: EvidenceSignalStatusCodeFlip, + } + + routes := []string{ + "/api/v1/users/12345/orders", + "/api/v1/users/me/orders", // "me" is a sentinel, not an id + "/api/v1/users/12345/invoices", // a different collection + "/api/v2/users/12345/orders", // a different API version + "/api/v1/users/12345", // a shorter route + "/api/v1/admins/12345/orders", // a different resource + } + seen := map[string]string{} + for _, r := range routes { + m := base + m.RouteTemplate = r + d, err := Dast(m) + if err != nil { + t.Fatalf("%s: %v", r, err) + } + if prev, ok := seen[d]; ok { + t.Errorf("routes %q and %q share digest %s; one of the two findings is lost on upsert", + prev, r, d) + } + seen[d] = r + } +} + +// TestDastInjectionPointAndEvidenceSignalAreIndependent. The whole reason +// research/18's evidenceClass was merged into research/07's Tier D is that +// WHERE the payload went in and HOW the defect was observed are different +// facts. Each must move the digest on its own. +func TestDastInjectionPointAndEvidenceSignalAreIndependent(t *testing.T) { + base := DastInput{ + TargetID: "t-0001", + RuleIDVersioned: "nuclei:sqli@a1b2c3d", + HTTPMethod: "GET", + RouteTemplate: "/search", + InjectionPoint: InjectionPointQuery, + ParamName: "q", + EvidenceSignal: EvidenceSignalDBErrorString, + } + baseDigest, err := Dast(base) + if err != nil { + t.Fatalf("%v", err) + } + + byPoint := base + byPoint.InjectionPoint = InjectionPointHeader + d1, err := Dast(byPoint) + if err != nil { + t.Fatalf("%v", err) + } + bySignal := base + bySignal.EvidenceSignal = EvidenceSignalTimingSideChannel + d2, err := Dast(bySignal) + if err != nil { + t.Fatalf("%v", err) + } + + if d1 == baseDigest { + t.Error("changing injection_point alone did not change the digest") + } + if d2 == baseDigest { + t.Error("changing evidence_class_detail alone did not change the digest") + } + if d1 == d2 { + t.Error("injection_point and evidence_class_detail are not independent fields") + } +} + +// TestInputStructsHaveTheirFrozenShape pins the exact field set of each tier +// input struct. Adding a field is how a volatile input (a line number, a +// version, a host) gets reintroduced, and it would silently change every +// digest that field participates in. If this test fails, the change is an +// anvil-fp/v2 event and an amendment to plan/40-record-and-storage.md, not a +// local edit. +func TestInputStructsHaveTheirFrozenShape(t *testing.T) { + cases := []struct { + name string + typ reflect.Type + fields []string + forbidden []string + }{ + { + name: "SastInput", + typ: reflect.TypeOf(SastInput{}), + fields: []string{"TargetID", "RuleIDVersioned", "RepoRelPath", + "EnclosingSymbolPath", "Snippet", "Ordinal"}, + // "version" is absent from this list on purpose: the RULE + // version is deliberately hashed. Line, column, advisory and + // evidence class are not. + forbidden: []string{"line", "column", "advisory", "evidence", "timestamp", "region"}, + }, + { + name: "ScaInput", + typ: reflect.TypeOf(ScaInput{}), + fields: []string{"TargetID", "AdvisoryID", "Purl", "ManifestRelPath"}, + forbidden: []string{"version", "line", "timestamp"}, + }, + { + name: "HostInput", + typ: reflect.TypeOf(HostInput{}), + fields: []string{"TargetID", "AdvisoryID", "Purl", "PackageManager", "HostIdentifier"}, + forbidden: []string{"version", "line", "timestamp"}, + }, + { + name: "DastInput", + typ: reflect.TypeOf(DastInput{}), + fields: []string{"TargetID", "RuleIDVersioned", "HTTPMethod", "RouteTemplate", + "InjectionPoint", "ParamName", "EvidenceSignal"}, + forbidden: []string{"host", "port", "scheme", "payload", "token", "cookieval", "timestamp", "line"}, + }, + } + + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + var got []string + for i := 0; i < tc.typ.NumField(); i++ { + got = append(got, tc.typ.Field(i).Name) + } + if !reflect.DeepEqual(got, tc.fields) { + t.Fatalf("field set changed.\n got: %v\n want: %v\n"+ + "Adding or removing a hashed input is an anvil-fp/v2 event.", got, tc.fields) + } + for _, f := range got { + lower := strings.ToLower(f) + for _, bad := range tc.forbidden { + if strings.Contains(lower, bad) { + t.Errorf("field %s contains forbidden substring %q; "+ + "the specification forbids hashing it", f, bad) + } + } + } + }) + } +} + +// --------------------------------------------------------------------------- +// Primitive, canonicalisation and normalization behaviour +// --------------------------------------------------------------------------- + +var hexDigest = regexp.MustCompile(`^[0-9a-f]{64}$`) + +func TestDigestShapeAndSeparator(t *testing.T) { + d, err := Digest("a", "b", "c") + if err != nil { + t.Fatalf("%v", err) + } + if !hexDigest.MatchString(d) { + t.Errorf("digest %q is not 64 lowercase hex characters", d) + } + if len(d) != FingerprintDigestHexLen { + t.Errorf("digest length %d, want %d (never truncated)", len(d), FingerprintDigestHexLen) + } + + // The separator, not concatenation: ("ab","c") and ("a","bc") must differ. + d1, _ := Digest("ab", "c") + d2, _ := Digest("a", "bc") + if d1 == d2 { + t.Error("fields are being concatenated without a separator") + } + + if _, err := Digest("ok", "bad\x1ffield"); err == nil { + t.Error("a field containing U+001F must be rejected: it moves a field boundary") + } + if _, err := Digest("ok", "bad\nfield"); err == nil { + t.Error("a field containing a control character must be rejected") + } + if _, err := Digest(); err == nil { + t.Error("an empty field list must be rejected") + } +} + +func TestValidateDigest(t *testing.T) { + good, err := Digest("x") + if err != nil { + t.Fatalf("%v", err) + } + if err := ValidateDigest(good); err != nil { + t.Errorf("a freshly computed digest failed validation: %v", err) + } + bad := map[string]string{ + "truncated to 32": good[:32], + "uppercase hex": strings.ToUpper(good), + "empty": "", + "non-hex at end": good[:63] + "g", + "one char too long": good + "0", + } + for name, v := range bad { + if err := ValidateDigest(v); err == nil { + t.Errorf("%s: expected rejection, got none", name) + } + } +} + +func TestCanonicalRepoRelPath(t *testing.T) { + cases := map[string]string{ + "internal/api/store.go": "internal/api/store.go", + `internal\api\store.go`: "internal/api/store.go", + "./internal/api/store.go": "internal/api/store.go", + "/internal/api/store.go": "internal/api/store.go", + "internal//api///store.go": "internal/api/store.go", + ".//internal/api/store.go": "internal/api/store.go", + "internal/api/": "internal/api", + `.\services\api\pom.xml`: "services/api/pom.xml", + "Internal/API/Store.go": "Internal/API/Store.go", // case preserved + } + for in, want := range cases { + if got := CanonicalRepoRelPath(in); got != want { + t.Errorf("CanonicalRepoRelPath(%q) = %q, want %q", in, got, want) + } + } +} + +func TestCanonicalRouteTemplate(t *testing.T) { + const tok = NormalizedRouteSegmentToken + + cases := map[string]string{ + // Structural canonicalisation. + "/api/v1/users/{id}": "/api/v1/users/" + tok, + "api/v1/users/{id}": "/api/v1/users/" + tok, + "/api/v1/users/{id}/": "/api/v1/users/" + tok, + "//api//v1/users/{id}": "/api/v1/users/" + tok, + "/api/v1/users/{id}?a=1": "/api/v1/users/" + tok, + "/api/v1/users/{id}#frag": "/api/v1/users/" + tok, + "/": "/", + "/Search": "/Search", // case preserved + + // Templating: the four volatile-segment classes. + "/api/v1/users/12345/orders": "/api/v1/users/" + tok + "/orders", + "/orders/0": "/orders/" + tok, + "/u/3f2504e0-4f89-11d3-9a0c-0305e82c3301": "/u/" + tok, + "/u/3F2504E0-4F89-11D3-9A0C-0305E82C3301": "/u/" + tok, + "/blob/e3b0c44298fc1c14": "/blob/" + tok, // 16 hex, the threshold exactly + "/blob/3f2504e04f8911d39a0c0305e82c3301": "/blob/" + tok, // dash-free UUID + "/s/dXNlcjEyMzQ1Njc4OTAxMg": "/s/" + tok, // 22-char alnum with digits + + // Templating: the three producer placeholder syntaxes, all onto one token. + "/api/users/{userId}/orders": "/api/users/" + tok + "/orders", + "/api/users/:userId/orders": "/api/users/" + tok + "/orders", + "/api/users//orders": "/api/users/" + tok + "/orders", + + // Idempotence: the canonical token is itself an already-templated segment. + "/api/users/" + tok + "/orders": "/api/users/" + tok + "/orders", + + // Every segment is examined, not only the last. + "/12345/orders/6789": "/" + tok + "/orders/" + tok, + } + for in, want := range cases { + if got := CanonicalRouteTemplate(in); got != want { + t.Errorf("CanonicalRouteTemplate(%q) = %q, want %q", in, got, want) + } + } + + // Idempotence as a property, not only on the one case above: a record read + // out of the store and re-fingerprinted must not fork its own identity. + for in := range cases { + once := CanonicalRouteTemplate(in) + if twice := CanonicalRouteTemplate(once); twice != once { + t.Errorf("CanonicalRouteTemplate is not idempotent on %q: %q then %q", in, once, twice) + } + } +} + +// TestCanonicalRouteTemplateDoesNotOverTemplate is the other half of the +// blocker-2 ruling and the more important half. Over-templating merges two +// genuinely distinct routes onto one digest, and one of the two findings is +// then LOST on upsert against UNIQUE (target_id, fingerprint) — silently. +// Under-templating only leaves a volatile route un-merged, which the DAST +// producer can still repair by emitting "{id}" itself. +// +// Every segment below must survive canonicalisation untouched. +func TestCanonicalRouteTemplateDoesNotOverTemplate(t *testing.T) { + preserved := []string{ + "v1", // a version segment: has a digit, but is 2 chars + "v2", // + "me", // the sentinel that is NOT an id + "users", // + "oauth2", // digit-bearing but short + "utf8", // + "api", // + "latest", // + "internationalization", // 20 letters, no digit — the long-word case + "recommendations", // + "release-notes-2026-08", // 21 chars with a digit, but hyphenated: a slug + "user_profile_settings", // 21 chars, underscored: a slug + "deadbeef", // 8 hex — below the 16-char hex threshold + "a1b2c3d", // a short git object id, 7 chars + "cafebabecafebab", // 15 hex — one short of the threshold + "report.pdf", // a filename + "2026-08-08", // an ISO date: hyphenated, so not opaque + "Search", // + } + for _, seg := range preserved { + in := "/api/" + seg + "/x" + want := "/api/" + seg + "/x" + if got := CanonicalRouteTemplate(in); got != want { + t.Errorf("segment %q was templated but must be preserved: CanonicalRouteTemplate(%q) = %q", + seg, in, got) + } + } + + // And the routes that must stay distinct from each other end to end. + distinct := []string{ + "/api/v1/users/12345/orders", + "/api/v1/users/me/orders", + "/api/v1/users/12345/invoices", + "/api/v2/users/12345/orders", + "/api/v1/users/12345/orders/12345", + } + seen := map[string]string{} + for _, r := range distinct { + c := CanonicalRouteTemplate(r) + if prev, ok := seen[c]; ok { + t.Errorf("routes %q and %q both canonicalise to %q; two distinct routes would share one identity", + prev, r, c) + } + seen[c] = r + } +} + +func TestPurlBase(t *testing.T) { + ok := map[string]string{ + "pkg:npm/lodash": "pkg:npm/lodash", + "pkg:npm/lodash@4.17.20": "pkg:npm/lodash", + "pkg:NPM/lodash@4.17.20": "pkg:npm/lodash", + "pkg:npm/%40angular/core@13.0.0": "pkg:npm/%40angular/core", + "pkg:deb/debian/openssl@1.1.1n?a=b": "pkg:deb/debian/openssl", + "pkg:golang/golang.org/x/net@v0.0.1": "pkg:golang/golang.org/x/net", + "PKG:maven/org.example/lib@1.0#s": "pkg:maven/org.example/lib", + " pkg:pypi/django@3.2.4 ": "pkg:pypi/django", + } + for in, want := range ok { + got, err := PurlBase(in) + if err != nil { + t.Errorf("PurlBase(%q): unexpected error %v", in, err) + continue + } + if got != want { + t.Errorf("PurlBase(%q) = %q, want %q", in, got, want) + } + } + + for _, bad := range []string{"", " ", "npm/lodash", "pkg:", "pkg:npm", "pkg:@1.0"} { + if got, err := PurlBase(bad); err == nil { + t.Errorf("PurlBase(%q) = %q, want an error", bad, got) + } + } +} + +func TestNormalizeMatch(t *testing.T) { + cases := []struct { + name string + in string + want string + }{ + { + name: "comments stripped, whitespace collapsed, literals abstracted", + in: "// lookup\nq := \"SELECT \" + name + \"'\" /* inline */ + 42\n", + want: "$1 := + $2 + + ", + }, + { + name: "same identifier maps to the same metavariable", + in: "a = b + a + b", + want: "$1 = $2 + $1 + $2", + }, + { + name: "callee and member names are preserved", + in: "rows := db.Query(userInput)", + want: "$1 := $2.Query($3)", + }, + { + name: "keywords are preserved", + in: "for i := range items { return nil }", + want: "for $1 := range $2 { return nil }", + }, + { + name: "hash line comments are stripped", + in: "value = compute(x) # trailing note", + want: "$1 = compute($2)", + }, + { + name: "numeric literal forms collapse to one token", + in: "a = 0xFF + 1_000 + 3.14f + 1e-9", + want: "$1 = + + + ", + }, + { + name: "escaped quotes do not terminate a string", + in: `s = "a \" b" + t`, + want: "$1 = + $2", + }, + { + // The receiver `p` is a local and is abstracted; `Field` is a + // member and `Ns`/`Helper` are the qualifier and callee of a + // scope-resolved call, which are API surface, not churn. + name: "arrow and scope selectors preserve member names", + in: "p->Field = Ns::Helper(v)", + want: "$1->Field = Ns::Helper($2)", + }, + { + name: "a scope-resolution chain is preserved end to end", + in: "std::vector v = Foo::Bar::make(x)", + want: "std::vector $1 = Foo::Bar::make($2)", + }, + { + // These two cases exist as a pair: calls into two different + // namespaces must NOT normalise to the same string, which is what + // abstracting the left operand of "::" would do. + name: "namespace qualifier Ns is preserved", + in: "Ns::Helper(v)", + want: "Ns::Helper($1)", + }, + { + name: "namespace qualifier Other is preserved and differs from Ns", + in: "Other::Helper(v)", + want: "Other::Helper($1)", + }, + { + name: "renaming a local does not change the shape", + in: "totalCount := len(itemsList)", + want: "$1 := len($2)", + }, + { + name: "renaming the same local differently yields the same shape", + in: "n := len(xs)", + want: "$1 := len($2)", + }, + { + name: "block comment spanning lines is dropped", + in: "a\n/* one\n two */\nb", + want: "$1 $2", + }, + { + name: "only comments and whitespace normalises to empty", + in: " // nothing here \n\n", + want: "", + }, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + if got := NormalizeMatch(tc.in); got != tc.want { + t.Errorf("NormalizeMatch(%q)\n got: %q\n want: %q", tc.in, got, tc.want) + } + }) + } + + // The normalized form must never contain the separator or a control + // character, or Digest would reject it. + for _, tc := range cases { + if strings.ContainsAny(NormalizeMatch(tc.in), "\x00\x1f\n\r\t") { + t.Errorf("normalized output of %q contains a control character", tc.name) + } + } +} + +func TestNormalizeMatchIsDeterministic(t *testing.T) { + src := "handler := func(w http.ResponseWriter, r *http.Request) { io.WriteString(w, r.URL.Query().Get(\"q\")) }" + first := NormalizeMatch(src) + for i := 0; i < 200; i++ { + if got := NormalizeMatch(src); got != first { + t.Fatalf("NormalizeMatch is not deterministic: %q vs %q", got, first) + } + } + if first == "" { + t.Fatal("normalization produced an empty result for a real snippet") + } +} + +// --------------------------------------------------------------------------- +// Ordinal assignment +// --------------------------------------------------------------------------- + +func TestAssignSastOrdinals(t *testing.T) { + mk := func(path, snippet string, line int) SastCandidate { + return SastCandidate{ + Input: SastInput{ + TargetID: "t-1", + RuleIDVersioned: "r@1", + RepoRelPath: path, + EnclosingSymbolPath: "sym", + Snippet: snippet, + }, + Line: line, + } + } + + // Five matches, given out of source order, spanning two files. + // + // THE GROUPING KEY IS THE NORMALIZED MATCH, NOT THE RAW SNIPPET. The + // specification defines ordinal as "the 0-based index of this match among + // all matches of the same rule_id in the same repo_relpath whose + // NORMALIZED_MATCH is IDENTICAL", and "exec(cmd)" and "exec(other)" both + // normalise to `exec($1)` — `exec` is a callee name so it is preserved, + // and the argument is a local so it becomes $1. All four a.go candidates + // are therefore ONE group; only b.go is separate. + cands := []SastCandidate{ + mk("a.go", "exec(cmd)", 90), + mk("a.go", "exec(cmd)", 10), + mk("b.go", "exec(cmd)", 5), + mk("a.go", "exec(other)", 50), + mk("a.go", "exec(cmd)", 50), + } + + got, err := AssignSastOrdinals(cands) + if err != nil { + t.Fatalf("%v", err) + } + if len(got) != len(cands) { + t.Fatalf("got %d inputs, want %d", len(got), len(cands)) + } + + // Within the a.go group, ordinals follow ascending (line, column, original + // index): line 10 -> 0, line 50 (index 3) -> 1, line 50 (index 4) -> 2, + // line 90 -> 3. The single b.go candidate is a group of one -> 0. The + // returned slice keeps the caller's original order, so the expectation is + // stated in that order. + want := []int{3, 0, 0, 1, 2} + for i := range want { + if got[i].Ordinal != want[i] { + t.Errorf("candidate %d (line %d): ordinal %d, want %d", + i, cands[i].Line, got[i].Ordinal, want[i]) + } + } + + // The collapse of "exec(cmd)" and "exec(other)" onto one normalized match + // is precisely the collision ordinal exists to break: without it the four + // a.go candidates would share one digest and three findings would be lost + // on upsert against UNIQUE (target_id, fingerprint). Assert the resulting + // digests are all distinct. + seen := map[string]int{} + for i, in := range got { + d, err := Sast(in) + if err != nil { + t.Fatalf("candidate %d: %v", i, err) + } + if prev, ok := seen[d]; ok { + t.Errorf("candidates %d and %d collide on digest %s", prev, i, d) + } + seen[d] = i + } + + // A run over the same candidates must produce the same ordinals. + again, err := AssignSastOrdinals(cands) + if err != nil { + t.Fatalf("%v", err) + } + if !reflect.DeepEqual(got, again) { + t.Error("AssignSastOrdinals is not deterministic") + } + + if _, err := AssignSastOrdinals([]SastCandidate{{Input: SastInput{}}}); err == nil { + t.Error("an unfingerprintable candidate must be rejected, not given ordinal 0") + } +} + +// --------------------------------------------------------------------------- +// Validation +// --------------------------------------------------------------------------- + +func TestTierValidationRejectsIncompleteInput(t *testing.T) { + if _, err := Sast(SastInput{RuleIDVersioned: "r", RepoRelPath: "a.go", Snippet: "x"}); err == nil { + t.Error("sast: empty target_id must be rejected") + } + if _, err := Sast(SastInput{TargetID: "t", RepoRelPath: "a.go", Snippet: "x"}); err == nil { + t.Error("sast: empty rule_id_versioned must be rejected") + } + if _, err := Sast(SastInput{TargetID: "t", RuleIDVersioned: "r", Snippet: "x"}); err == nil { + t.Error("sast: empty repo_relpath must be rejected") + } + if _, err := Sast(SastInput{TargetID: "t", RuleIDVersioned: "r", RepoRelPath: "a.go", Snippet: "// only a comment"}); err == nil { + t.Error("sast: a snippet that normalises to nothing must be rejected") + } + if _, err := Sast(SastInput{TargetID: "t", RuleIDVersioned: "r", RepoRelPath: "a.go", Snippet: "x", Ordinal: -1}); err == nil { + t.Error("sast: a negative ordinal must be rejected") + } + // An empty enclosing symbol is legal: top-level code and config files + // have none, and rejecting them would make them unfingerprintable. + if _, err := Sast(SastInput{TargetID: "t", RuleIDVersioned: "r", RepoRelPath: "a.go", Snippet: "exec(x)"}); err != nil { + t.Errorf("sast: an empty enclosing_symbol_path must be accepted, got %v", err) + } + + if _, err := Sca(ScaInput{TargetID: "t", Purl: "pkg:npm/a", ManifestRelPath: "p.json"}); err == nil { + t.Error("sca: empty advisory_id must be rejected") + } + if _, err := Sca(ScaInput{TargetID: "t", AdvisoryID: "CVE-1", ManifestRelPath: "p.json"}); err == nil { + t.Error("sca: empty purl must be rejected") + } + if _, err := Sca(ScaInput{TargetID: "t", AdvisoryID: "CVE-1", Purl: "pkg:npm/a"}); err == nil { + t.Error("sca: empty manifest_relpath must be rejected") + } + + if _, err := Host(HostInput{TargetID: "t", AdvisoryID: "CVE-1", Purl: "pkg:deb/d/o", HostIdentifier: "o"}); err == nil { + t.Error("host: empty package_manager must be rejected") + } + if _, err := Host(HostInput{TargetID: "t", AdvisoryID: "CVE-1", Purl: "pkg:deb/d/o", PackageManager: "a:b", HostIdentifier: "o"}); err == nil { + t.Error("host: a package_manager containing ':' must be rejected") + } + + badDast := DastInput{ + TargetID: "t", RuleIDVersioned: "r", HTTPMethod: "GET", RouteTemplate: "/x", + InjectionPoint: InjectionPoint("QUERY"), EvidenceSignal: EvidenceSignalOther, + } + if _, err := Dast(badDast); err == nil { + t.Error("dast: a SCREAMING_CASE injection point must be rejected; the record's convention is lowercase") + } + badDast.InjectionPoint = InjectionPointQuery + badDast.EvidenceSignal = EvidenceSignal("db_error_string") + if _, err := Dast(badDast); err == nil { + t.Error("dast: an unfrozen evidence signal literal must be rejected") + } + badDast.EvidenceSignal = EvidenceSignalDBErrorString + badDast.HTTPMethod = "" + if _, err := Dast(badDast); err == nil { + t.Error("dast: empty http_method must be rejected") + } + // An empty param name is legal: a whole-body or raw-request injection has + // no single named parameter. + badDast.HTTPMethod = "POST" + badDast.ParamName = "" + if _, err := Dast(badDast); err != nil { + t.Errorf("dast: an empty param_name must be accepted, got %v", err) + } +} + +// TestExportedTierEntryPointsAgreeWithFieldBuilders keeps the two exported +// surfaces from drifting: Sast must be exactly Digest(SastFields(...)), and so +// on for each tier. +func TestExportedTierEntryPointsAgreeWithFieldBuilders(t *testing.T) { + sastIn := SastInput{TargetID: "t", RuleIDVersioned: "r@1", RepoRelPath: "a.go", Snippet: "exec(x)"} + scaIn := ScaInput{TargetID: "t", AdvisoryID: "CVE-1", Purl: "pkg:npm/a@1.0", ManifestRelPath: "p.json"} + hostIn := HostInput{TargetID: "t", AdvisoryID: "CVE-1", Purl: "pkg:deb/d/o@1", PackageManager: "apt", HostIdentifier: "o"} + dastIn := DastInput{TargetID: "t", RuleIDVersioned: "r@1", HTTPMethod: "GET", RouteTemplate: "/x", + InjectionPoint: InjectionPointQuery, ParamName: "q", EvidenceSignal: EvidenceSignalReflectedPayload} + + pairs := []struct { + name string + fields func() ([]string, error) + digest func() (string, error) + want int + }{ + {"sast", func() ([]string, error) { return SastFields(sastIn) }, func() (string, error) { return Sast(sastIn) }, 7}, + {"sca", func() ([]string, error) { return ScaFields(scaIn) }, func() (string, error) { return Sca(scaIn) }, 5}, + {"host", func() ([]string, error) { return HostFields(hostIn) }, func() (string, error) { return Host(hostIn) }, 5}, + {"dast", func() ([]string, error) { return DastFields(dastIn) }, func() (string, error) { return Dast(dastIn) }, 8}, + } + for _, p := range pairs { + t.Run(p.name, func(t *testing.T) { + fields, err := p.fields() + if err != nil { + t.Fatalf("%v", err) + } + if len(fields) != p.want { + t.Errorf("tier %s hashes %d fields, the specification lists %d", p.name, len(fields), p.want) + } + viaFields, err := Digest(fields...) + if err != nil { + t.Fatalf("%v", err) + } + direct, err := p.digest() + if err != nil { + t.Fatalf("%v", err) + } + if viaFields != direct { + t.Errorf("tier %s: Digest(Fields(in)) = %s but the entry point returned %s", + p.name, viaFields, direct) + } + }) + } +} + +// TestTiersUseTheirSpecifiedDiscriminator asserts the literal tier token in +// field position 2, because a swapped discriminator is invisible in a digest +// but makes every producer incompatible. +func TestTiersUseTheirSpecifiedDiscriminator(t *testing.T) { + sastFields, err := SastFields(SastInput{TargetID: "t", RuleIDVersioned: "r", RepoRelPath: "a.go", Snippet: "exec(x)"}) + if err != nil { + t.Fatalf("%v", err) + } + if sastFields[1] != "sast" { + t.Errorf("sast tier discriminator = %q, want \"sast\"", sastFields[1]) + } + dastFields, err := DastFields(DastInput{TargetID: "t", RuleIDVersioned: "r", HTTPMethod: "GET", + RouteTemplate: "/x", InjectionPoint: InjectionPointQuery, EvidenceSignal: EvidenceSignalOther}) + if err != nil { + t.Fatalf("%v", err) + } + if dastFields[1] != "dast" { + t.Errorf("dast tier discriminator = %q, want \"dast\"", dastFields[1]) + } + scaFields, err := ScaFields(ScaInput{TargetID: "t", AdvisoryID: "C", Purl: "pkg:npm/a", ManifestRelPath: "p"}) + if err != nil { + t.Fatalf("%v", err) + } + if scaFields[1] != string(DetectorKindSCA) { + t.Errorf("sca tier discriminator = %q, want %q", scaFields[1], DetectorKindSCA) + } + hostFields, err := HostFields(HostInput{TargetID: "t", AdvisoryID: "C", Purl: "pkg:deb/d/o", + PackageManager: "apt", HostIdentifier: "o"}) + if err != nil { + t.Fatalf("%v", err) + } + if hostFields[1] != string(DetectorKindHost) { + t.Errorf("host tier discriminator = %q, want %q", hostFields[1], DetectorKindHost) + } + if hostFields[4] != "apt:openssl" && hostFields[4] != "apt:o" { + t.Errorf("host locator = %q, want \":\"", hostFields[4]) + } +} + +// TestSeparatorConstantIsUnitSeparator is a belt-and-braces check on the one +// byte the whole scheme depends on. contract.go owns the constant; if it ever +// changed, every stored fingerprint would be orphaned silently. +func TestSeparatorConstantIsUnitSeparator(t *testing.T) { + if FingerprintFieldSeparator != "\x1f" { + t.Fatalf("FingerprintFieldSeparator = %q, want U+001F", FingerprintFieldSeparator) + } + if FingerprintDigestHexLen != 64 { + t.Fatalf("FingerprintDigestHexLen = %d, want 64 (never truncated)", FingerprintDigestHexLen) + } + if FingerprintAlgV1 != "anvil-fp/v1" { + t.Fatalf("FingerprintAlgV1 = %q, want \"anvil-fp/v1\"", FingerprintAlgV1) + } +} diff --git a/internal/store/ddl.go b/internal/store/ddl.go new file mode 100644 index 0000000..7eb2cb6 --- /dev/null +++ b/internal/store/ddl.go @@ -0,0 +1,165 @@ +// Package store owns Anvil's single SQLite store of record. +// +// plan/00-SPINE.md S1 collapsed the originally-specified "8-hour buffer file" +// into one SQLite database, a `handoff` table, and a regenerable tmpfs packet +// that is never a source of truth. This package holds the DDL for that +// database. plan/IMPLEMENTATION-PLAN.md §6 rulings G9 and G10 make schema.sql +// the ONLY definition of the `handoff` table anywhere in Anvil: area 70's O.3 +// migration and area 60's rival `anvil_ledger` are both folded into it. +// +// This file is a thin, dependency-free wrapper. It embeds the DDL, exposes the +// connection pragmas the DDL deliberately does not contain, and offers just +// enough read-only introspection for R.5's migration ledger and for the test +// that proves the SQL vocabularies and internal/record's Go enums have not +// drifted apart. It opens no database and executes no statement. +package store + +import ( + "crypto/sha256" + _ "embed" // for the //go:embed directive on schemaSQL + "encoding/hex" + "fmt" + "regexp" + "strings" +) + +//go:embed schema.sql +var schemaSQL string + +// MaxDurableTextBytes is the byte cap enforced by schema.sql's +// finding_occurrence triggers on `message` and `evidence_ref`. +// +// research/07 Risk #13: if a raw snippet or a DAST request/response body is +// copied into a durable text column it outlives the payload purge forever, and +// those are exactly the fields most likely to carry a secret. The cap is far +// below internal/record's smallest inline body cap (8 KiB) so that no body can +// be smuggled into a durable column even at its minimum size, and ample for a +// rule title plus a pointer into the sealed payload. +// +// Changing this constant alone does nothing: the value is enforced by SQL. +// ddl_test.go asserts the trigger and this constant agree. +const MaxDurableTextBytes = 2048 + +// Schema returns the complete DDL for schema version 1, exactly as committed +// in schema.sql. +// +// It contains no PRAGMA statement, by design. `PRAGMA journal_mode = WAL` +// cannot run inside a transaction and R.5 applies this DDL inside +// BEGIN...COMMIT. Use ConnectionPragmas for the settings that must be applied +// per connection instead. +func Schema() string { return schemaSQL } + +// SchemaSHA256 returns the lowercase hex SHA-256 of the embedded DDL, over its +// exact committed bytes with no normalisation. R.5's migration ledger records +// a checksum; this is the value for the initial migration, and it changes if +// so much as a comment in schema.sql changes, which is the intended +// sensitivity for a frozen interface. +func SchemaSHA256() string { + sum := sha256.Sum256([]byte(schemaSQL)) + return hex.EncodeToString(sum[:]) +} + +// ConnectionPragmas returns the pragmas from plan/40-record-and-storage.md's +// Store Schema section, in the order they must be applied. +// +// Every one of these is per connection, not per database file, so they must be +// re-applied on every connection the pool opens — `foreign_keys` above all, +// which SQLite leaves OFF by default and which this schema depends on. R.5 +// applies them before any other store operation, after its network-mount and +// FTS5 guards. +// +// The `synchronous = NORMAL` value resolves a research conflict on purpose: +// research/07 recommends NORMAL as the documented standard WAL pairing, while +// research/08 recommends FULL without any WAL-specific justification. NORMAL +// with WAL is crash-safe for the database file; the risk it accepts is losing +// the most recent commits on power loss, not corruption. +func ConnectionPragmas() []string { + return []string{ + "PRAGMA journal_mode = WAL", + "PRAGMA foreign_keys = ON", + "PRAGMA busy_timeout = 10000", + "PRAGMA synchronous = NORMAL", + "PRAGMA wal_autocheckpoint = 1000", + } +} + +var tableRE = regexp.MustCompile(`(?m)^CREATE\s+(?:VIRTUAL\s+)?TABLE\s+([A-Za-z_][A-Za-z0-9_]*)`) + +// Tables returns every table name schema.sql creates, ordinary and virtual +// alike, in file order. It is a parse of the committed DDL, not a hand-kept +// list, so a table added to schema.sql without a corresponding smoke-test +// insert fails ddl_test.go rather than going unexercised. +func Tables() []string { + matches := tableRE.FindAllStringSubmatch(schemaSQL, -1) + names := make([]string, 0, len(matches)) + for _, m := range matches { + names = append(names, m[1]) + } + return names +} + +// CheckConstraint returns the text of the named CHECK constraint's expression, +// without the enclosing parentheses. +// +// Every CHECK in schema.sql is named. research/07 Risk #15 records that +// batch-recreate tooling silently drops UNNAMED check constraints, which on a +// security tool's schema is an integrity regression with no error message; +// naming them is that risk's mitigation, and it also makes the constraints +// addressable from a test. +func CheckConstraint(name string) (string, error) { + anchor := regexp.MustCompile(`CONSTRAINT\s+` + regexp.QuoteMeta(name) + `\s+CHECK\s*\(`) + loc := anchor.FindStringIndex(schemaSQL) + if loc == nil { + return "", fmt.Errorf("store: no CHECK constraint named %q in schema.sql", name) + } + // loc[1] is the index just past the opening parenthesis. Scan forward to + // its match, tracking nesting and single-quoted literals so that a + // parenthesis inside a string literal cannot end the scan early. + depth := 1 + inLiteral := false + for i := loc[1]; i < len(schemaSQL); i++ { + switch c := schemaSQL[i]; { + case c == '\'': + // Doubled '' inside a literal is an escaped quote; toggling twice + // leaves the state correct, so no special case is needed. + inLiteral = !inLiteral + case inLiteral: + // Parentheses inside a literal are data, not structure. + case c == '(': + depth++ + case c == ')': + depth-- + if depth == 0 { + return schemaSQL[loc[1]:i], nil + } + } + } + return "", fmt.Errorf("store: CHECK constraint %q has an unbalanced expression in schema.sql", name) +} + +var literalRE = regexp.MustCompile(`'((?:[^']|'')*)'`) + +// EnumCheckValues returns the SQL string literals of the named CHECK +// constraint, in the order they appear. +// +// This exists because of a real constraint on how the vocabulary is written +// down. plan/IMPLEMENTATION-PLAN.md §6 freezes the enums in +// internal/record/contract.go and forbids any second declaration; but a SQL +// CHECK constraint cannot reference a Go constant, and templating the DDL at +// run time would mean schema.sql was no longer a file that applies to an empty +// database on its own — which its own stop condition requires. So the literals +// are written once more, here, and ddl_test.go asserts set equality with +// internal/record's values for every one of them. A drift is a failed test, +// not an integration surprise in Phase 5. +func EnumCheckValues(name string) ([]string, error) { + expr, err := CheckConstraint(name) + if err != nil { + return nil, err + } + matches := literalRE.FindAllStringSubmatch(expr, -1) + values := make([]string, 0, len(matches)) + for _, m := range matches { + values = append(values, strings.ReplaceAll(m[1], "''", "'")) + } + return values, nil +} diff --git a/internal/store/ddl_test.go b/internal/store/ddl_test.go new file mode 100644 index 0000000..68b909f --- /dev/null +++ b/internal/store/ddl_test.go @@ -0,0 +1,608 @@ +package store + +import ( + "database/sql" + "strconv" + "strings" + "testing" + + "github.com/Susquehanna-Syntax/Anvil/internal/record" + + _ "modernc.org/sqlite" // cgo-free driver, plan/00-SPINE.md S12 +) + +// newDB applies ConnectionPragmas and then schema.sql to a fresh in-memory +// database, exactly as R.5's migration will, and returns it. +// +// MaxOpenConns is pinned to 1 because each new connection to ":memory:" is a +// different, empty database. +func newDB(t *testing.T) *sql.DB { + t.Helper() + + db, err := sql.Open("sqlite", ":memory:") + if err != nil { + t.Fatalf("sql.Open: %v", err) + } + t.Cleanup(func() { _ = db.Close() }) + db.SetMaxOpenConns(1) + + for _, p := range ConnectionPragmas() { + // journal_mode is a no-op on an in-memory database (it reports + // "memory"), which is why the WAL guarantees themselves are R.5's to + // verify on a real file. The rest apply normally. + if _, err := db.Exec(p); err != nil { + t.Fatalf("pragma %q: %v", p, err) + } + } + if _, err := db.Exec(Schema()); err != nil { + t.Fatalf("applying schema.sql: %v", err) + } + return db +} + +const ( + // 64 lowercase hex characters, the only shape ck_*_fingerprint_hex admits. + fpA = "a1b2c3d4e5f60718293a4b5c6d7e8f90a1b2c3d4e5f60718293a4b5c6d7e8f90" + sha256Hex = "0011223344556677889900112233445566778899001122334455667788990011" +) + +// seedAll inserts exactly one FK-valid row into every table schema.sql +// creates, then selects each row back. This is R.4's mandated smoke test. +func seedAll(t *testing.T, db *sql.DB) { + t.Helper() + + type seed struct { + table string + insert string + args []any + verify string + } + + seeds := []seed{ + { + table: "target", + insert: `INSERT INTO target (target_id, kind, locator) VALUES (1, 'repo', 'https://example.invalid/r.git')`, + verify: `SELECT locator FROM target WHERE target_id = 1`, + }, + { + table: "trigger_policy", + insert: `INSERT INTO trigger_policy (policy_id, target_id, kind, spec, scan_depth) VALUES (1, 1, 'cron', '0 3 * * *', 'full')`, + verify: `SELECT spec FROM trigger_policy WHERE policy_id = 1`, + }, + { + table: "ingest_watermark", + insert: `INSERT INTO ingest_watermark (source, cursor, etag, last_success_at) VALUES ('osv', 'c1', 'e1', '2026-08-08T00:00:00Z')`, + verify: `SELECT cursor FROM ingest_watermark WHERE source = 'osv'`, + }, + { + table: "advisory", + insert: `INSERT INTO advisory (advisory_id, source, published_at, modified_at, severity, summary, content_hash, ingested_at) + VALUES ('CVE-2026-0001', 'osv', '2026-01-01T00:00:00Z', '2026-01-02T00:00:00Z', 'high', 'sqli in widget', 'h1', '2026-08-08T00:00:00Z')`, + verify: `SELECT summary FROM advisory WHERE advisory_id = 'CVE-2026-0001'`, + }, + { + table: "advisory_alias", + insert: `INSERT INTO advisory_alias (advisory_id, alias_id) VALUES ('CVE-2026-0001', 'GHSA-aaaa-bbbb-cccc')`, + verify: `SELECT alias_id FROM advisory_alias WHERE advisory_id = 'CVE-2026-0001'`, + }, + { + table: "advisory_fts", + insert: `INSERT INTO advisory_fts (rowid, summary, details, aliases) + SELECT rowid, summary, '', 'CVE-2026-0001' FROM advisory WHERE advisory_id = 'CVE-2026-0001'`, + // rowid only, not a column value: `content='advisory'` makes any + // column read go back to `advisory`, which has no `aliases` + // column. See the KNOWN DEFECT note above this table in + // schema.sql. Writes and rowid-only MATCH queries are the part + // that works, and the part R.4 pins here. + verify: `SELECT rowid FROM advisory_fts WHERE advisory_fts MATCH 'sqli'`, + }, + { + table: "component", + insert: `INSERT INTO component (component_id, ecosystem, name, purl_base) VALUES (1, 'pypi', 'requests', 'pkg:pypi/requests')`, + verify: `SELECT purl_base FROM component WHERE component_id = 1`, + }, + { + table: "advisory_affects", + insert: `INSERT INTO advisory_affects (advisory_id, component_id, introduced, fixed, range_kind) VALUES ('CVE-2026-0001', 1, '2.0.0', '2.31.0', 'semver')`, + verify: `SELECT fixed FROM advisory_affects WHERE advisory_id = 'CVE-2026-0001' AND component_id = 1`, + }, + { + table: "scan_run", + insert: `INSERT INTO scan_run (scan_run_id, target_id, policy_id, trigger_ref, commit_sha, started_at, status, ruleset_version) + VALUES (1, 1, 1, 'v1.0.0', 'deadbeef', '2026-08-08T00:00:00Z', 'ok', 'rs/1')`, + verify: `SELECT status FROM scan_run WHERE scan_run_id = 1`, + }, + { + table: "audit_record", + insert: `INSERT INTO audit_record (audit_record_id, scan_run_id, schema_version, state, sast_status, sast_sealed_at, + dast_status, dast_coverage_json, target_provenance, deadline_at, payload_sha256, created_at) + VALUES (1, 1, ?, 'both_sealed', 'sealed', '2026-08-08T01:00:00Z', + 'completed_clean', '{"probedCount":1}', 'booted_clean', '2026-08-08T08:00:00Z', ?, '2026-08-08T00:00:00Z')`, + args: []any{record.SchemaVersion, sha256Hex}, + verify: `SELECT state FROM audit_record WHERE audit_record_id = 1`, + }, + { + table: "code_location", + insert: `INSERT INTO code_location (location_id, repo_relpath, start_line, end_line, symbol, symbol_kind, blob_sha, snippet_hash) + VALUES (1, 'src/app/db.py', 41, 43, 'src/app/db.py::Repo.query', 'method', 'b1', 's1')`, + verify: `SELECT repo_relpath FROM code_location WHERE location_id = 1`, + }, + { + table: "finding", + insert: `INSERT INTO finding (finding_id, target_id, fingerprint, detector, evidence_class, rule_id, + remediable_by_agent, advisory_id, component_id, severity, title, state, + first_seen_scan, first_seen_at) + VALUES (1, 1, ?, 'sast', 'sast_reachable', 'anvil.py.sqli/v3', 0, 'CVE-2026-0001', 1, 'high', 'SQL injection', 'open', 1, '2026-08-08T00:30:00Z')`, + args: []any{fpA}, + verify: `SELECT fingerprint FROM finding WHERE finding_id = 1`, + }, + { + table: "finding_fingerprint", + insert: `INSERT INTO finding_fingerprint (finding_id, kind, alg, value) VALUES (1, 'primary', ?, ?)`, + args: []any{record.FingerprintAlgV1, fpA}, + verify: `SELECT value FROM finding_fingerprint WHERE finding_id = 1 AND kind = 'primary'`, + }, + { + table: "finding_occurrence", + insert: `INSERT INTO finding_occurrence (occurrence_id, finding_id, scan_run_id, location_id, confidence, message, + evidence_ref, advisory_as_of, advisory_staleness_seconds, advisory_parse_degraded) + VALUES (1, 1, 1, 1, 0.9, 'rule anvil.py.sqli/v3 matched', 'runs/0/results/0', '2026-08-07T00:00:00Z', 86400, 0)`, + verify: `SELECT message FROM finding_occurrence WHERE occurrence_id = 1`, + }, + { + table: "finding_state_event", + insert: `INSERT INTO finding_state_event (event_id, finding_id, scan_run_id, from_state, to_state, cause, at) VALUES (1, 1, 1, NULL, 'open', 'first_seen', '2026-08-08T00:30:00Z')`, + verify: `SELECT to_state FROM finding_state_event WHERE event_id = 1`, + }, + { + table: "fix_attempt", + insert: `INSERT INTO fix_attempt (fix_attempt_id, finding_id, audit_record_id, agent_model_id, started_at, status, patch_ref, branch_name, pr_url) + VALUES (1, 1, 1, 'model/x', '2026-08-08T02:00:00Z', 'proposed', 'refs/anvil/fix/1', 'anvil/fix-1', 'https://example.invalid/pr/1')`, + verify: `SELECT status FROM fix_attempt WHERE fix_attempt_id = 1`, + }, + { + table: "verification", + insert: `INSERT INTO verification (verification_id, fix_attempt_id, kind, scan_run_id, result, details_json, verified_at) + VALUES (1, 1, 'rescan_dast', 1, 'pass', '{"reproFailed":true}', '2026-08-08T03:00:00Z')`, + verify: `SELECT result FROM verification WHERE verification_id = 1`, + }, + { + table: "suppression", + insert: `INSERT INTO suppression (suppression_id, target_id, match_kind, match_value, classification, justification, created_by, created_at, active) + VALUES (1, 1, 'fingerprint', ?, 'accepted_risk', 'compensating control', 'operator', '2026-08-08T00:00:00Z', 1)`, + args: []any{fpA}, + verify: `SELECT classification FROM suppression WHERE suppression_id = 1`, + }, + { + table: "file_state", + insert: `INSERT INTO file_state (target_id, repo_relpath, blob_sha, ruleset_version, last_scan_id) VALUES (1, 'src/app/db.py', 'b1', 'rs/1', 1)`, + verify: `SELECT blob_sha FROM file_state WHERE target_id = 1 AND repo_relpath = 'src/app/db.py'`, + }, + { + table: "handoff", + insert: `INSERT INTO handoff (handoff_id, finding_id, audit_record_id, fingerprint, group_id, state, consumption_class, + claimed_by, lease_expires_at, attempts, max_attempts, idempotency_key, created_at, updated_at) + VALUES (1, 1, 1, ?, 'grp-1', 'leased', 'requires_dynamic_confirmation', + 'worker-1', '2026-08-08T00:20:00Z', 0, 2, ?, '2026-08-08T00:00:00Z', '2026-08-08T00:00:00Z')`, + args: []any{fpA, sha256Hex}, + verify: `SELECT state FROM handoff WHERE handoff_id = 1`, + }, + { + table: "schema_migration", + insert: `INSERT INTO schema_migration (version, name, checksum, applied_at) VALUES (1, '0001_init', ?, '2026-08-08T00:00:00Z')`, + args: []any{SchemaSHA256()}, + verify: `SELECT checksum FROM schema_migration WHERE version = 1`, + }, + } + + seeded := make(map[string]bool, len(seeds)) + for _, s := range seeds { + if _, err := db.Exec(s.insert, s.args...); err != nil { + t.Fatalf("insert into %s: %v", s.table, err) + } + var got string + if err := db.QueryRow(s.verify).Scan(&got); err != nil { + t.Fatalf("select back from %s: %v", s.table, err) + } + if got == "" { + t.Errorf("%s: selected an empty value back", s.table) + } + seeded[s.table] = true + } + + // The seed list is checked against the DDL rather than against itself, so + // a table added to schema.sql without a smoke insert fails here. + for _, table := range Tables() { + if !seeded[table] { + t.Errorf("table %q is created by schema.sql but never exercised by the smoke test", table) + } + } + if len(seeded) != len(Tables()) { + t.Errorf("smoke test seeds %d tables, schema.sql creates %d", len(seeded), len(Tables())) + } +} + +func TestSchemaAppliesAndSmokeInsertsRoundTrip(t *testing.T) { + db := newDB(t) + seedAll(t, db) + + var fk int + if err := db.QueryRow(`PRAGMA foreign_keys`).Scan(&fk); err != nil { + t.Fatalf("PRAGMA foreign_keys: %v", err) + } + if fk != 1 { + t.Fatalf("PRAGMA foreign_keys = %d, want 1: the schema depends on FK enforcement", fk) + } +} + +// enumStrings widens any ~string enum slice from internal/record. +func enumStrings[T ~string](vals []T) []string { + out := make([]string, len(vals)) + for i, v := range vals { + out[i] = string(v) + } + return out +} + +// enumChecks maps every CHECK constraint in schema.sql that names a vocabulary +// to the internal/record function that owns it. plan/IMPLEMENTATION-PLAN.md §6 +// makes internal/record the single declaration site; SQL cannot reference a Go +// constant, so this table is the seam, and this test is what keeps the seam +// honest. +var enumChecks = []struct { + constraint string + table string + column string + want []string +}{ + {"ck_scan_run_status", "scan_run", "status", enumStrings(record.ScanRunStatusValues())}, + {"ck_audit_record_state", "audit_record", "state", enumStrings(record.StateValues())}, + {"ck_audit_record_sast_status", "audit_record", "sast_status", enumStrings(record.HalfStatusValues())}, + {"ck_audit_record_dast_status", "audit_record", "dast_status", enumStrings(record.DastStatusValues())}, + {"ck_audit_record_target_provenance", "audit_record", "target_provenance", enumStrings(record.TargetProvenanceValues())}, + {"ck_finding_detector", "finding", "detector", enumStrings(record.DetectorKindValues())}, + {"ck_finding_evidence_class", "finding", "evidence_class", enumStrings(record.EvidenceClassValues())}, + {"ck_finding_verdict", "finding", "verdict", enumStrings(record.VerdictValues())}, + {"ck_finding_state", "finding", "state", enumStrings(record.FindingStateValues())}, + {"ck_handoff_state", "handoff", "state", enumStrings(record.HandoffStateValues())}, + {"ck_handoff_consumption_class", "handoff", "consumption_class", enumStrings(record.ConsumptionClassValues())}, +} + +func TestEnumCheckConstraintsMatchContractLiteralForLiteral(t *testing.T) { + for _, c := range enumChecks { + got, err := EnumCheckValues(c.constraint) + if err != nil { + t.Errorf("%s: %v", c.constraint, err) + continue + } + if len(got) != len(c.want) { + t.Errorf("%s (%s.%s): SQL admits %d literals %v, internal/record declares %d %v", + c.constraint, c.table, c.column, len(got), got, len(c.want), c.want) + continue + } + for i := range got { + if got[i] != c.want[i] { + t.Errorf("%s (%s.%s): literal %d is %q in SQL, %q in internal/record", + c.constraint, c.table, c.column, i, got[i], c.want[i]) + } + } + } +} + +// TestEnumCheckConstraintsEnforceContractValues proves the agreement at run +// time, not only by reading the DDL: every literal internal/record declares is +// accepted by the column, and a literal it does not declare is rejected. +func TestEnumCheckConstraintsEnforceContractValues(t *testing.T) { + db := newDB(t) + seedAll(t, db) + + for _, c := range enumChecks { + update := `UPDATE ` + c.table + ` SET ` + c.column + ` = ? WHERE rowid = (SELECT MIN(rowid) FROM ` + c.table + `)` + for _, v := range c.want { + if _, err := db.Exec(update, v); err != nil { + t.Errorf("%s.%s rejected %q, which internal/record declares legal: %v", c.table, c.column, v, err) + } + } + if _, err := db.Exec(update, "not_a_declared_literal"); err == nil { + t.Errorf("%s.%s accepted an undeclared literal; %s is not enforcing the vocabulary", + c.table, c.column, c.constraint) + } + } +} + +// TestHandoffCarriesAllThirteenDispositions is the G10 regression guard. Area +// X's `anvil_ledger` was deleted and its four extra dispositions folded into +// handoff.state; if that set ever shrinks, X.9 writes a disposition this +// column cannot hold and the ready-set index re-leases the finding forever. +func TestHandoffCarriesAllThirteenDispositions(t *testing.T) { + got, err := EnumCheckValues("ck_handoff_state") + if err != nil { + t.Fatalf("ck_handoff_state: %v", err) + } + if len(got) != 13 { + t.Fatalf("ck_handoff_state admits %d states %v, want the 13 of IMPLEMENTATION-PLAN.md §6", len(got), got) + } + for _, needed := range []record.HandoffState{ + record.HandoffStateSkippedBudget, + record.HandoffStateFixedIncidentally, + record.HandoffStateSplitRequired, + record.HandoffStateWithdrawn, + record.HandoffStateSuperseded, + } { + if !contains(got, string(needed)) { + t.Errorf("ck_handoff_state is missing %q, one of the dispositions collapsed in from anvil_ledger", needed) + } + } +} + +// TestNoSecondDispositionTable guards the S1 spine rule directly: one durable +// table carrying finding dispositions, not two. +func TestNoSecondDispositionTable(t *testing.T) { + for _, table := range Tables() { + if strings.Contains(table, "ledger") { + t.Errorf("schema.sql creates %q; §6 G10 collapses every finding disposition into handoff.state", table) + } + } + // No other column may admit a handoff disposition. `audit_record.state` is + // the audit's own lifecycle and `finding.state` is the durable finding + // lifecycle; neither is a queue disposition, and the overlap that would + // signal a second ledger is a literal only handoff.state should carry. + for _, c := range enumChecks { + if c.table == "handoff" { + continue + } + for _, v := range c.want { + if v == string(record.HandoffStateSkippedBudget) || v == string(record.HandoffStateLeased) { + t.Errorf("%s.%s admits the handoff disposition %q; dispositions live only in handoff.state", c.table, c.column, v) + } + } + } +} + +func TestConsumptionClassHasNoDefault(t *testing.T) { + db := newDB(t) + + notNull, dflt := columnInfo(t, db, "handoff", "consumption_class") + if !notNull { + t.Error("handoff.consumption_class must be NOT NULL: the static-only vs requires-dynamic-confirmation gate has no other home in the schema") + } + if dflt.Valid { + t.Errorf("handoff.consumption_class has DEFAULT %q; a default silently grants every row the permissive value, which 00-SPINE.md S7 forbids", dflt.String) + } +} + +// TestColumnDefaultsMatchContract keeps the DDL's literal defaults tied to the +// Go constants they mirror. +func TestColumnDefaultsMatchContract(t *testing.T) { + db := newDB(t) + + cases := []struct { + table, column, want string + }{ + {"finding", "fingerprint_alg", "'" + record.FingerprintAlgV1 + "'"}, + {"finding", "verdict", "'" + string(record.VerdictTruePositive) + "'"}, + {"audit_record", "dast_status", "'" + string(record.DastStatusNotRun) + "'"}, + {"audit_record", "claim_timeout_seconds", strconv.Itoa(record.DefaultClaimTimeoutSeconds)}, + {"handoff", "state", "'" + string(record.HandoffStateReady) + "'"}, + } + for _, c := range cases { + _, dflt := columnInfo(t, db, c.table, c.column) + if !dflt.Valid { + t.Errorf("%s.%s has no DEFAULT, want %s", c.table, c.column, c.want) + continue + } + if dflt.String != c.want { + t.Errorf("%s.%s DEFAULT is %s, internal/record says %s", c.table, c.column, dflt.String, c.want) + } + } +} + +// TestFingerprintColumnsEnforceContractDigestLength ties the SQL length check +// to record.FingerprintDigestHexLen. CRITIQUE-01 §9 records that the digest is +// 64 lowercase hex characters and is never truncated; a truncating writer must +// fail at the column, not silently halve the collision resistance. +func TestFingerprintColumnsEnforceContractDigestLength(t *testing.T) { + db := newDB(t) + seedAll(t, db) + + exact := strings.Repeat("a", record.FingerprintDigestHexLen) + short := strings.Repeat("a", record.FingerprintDigestHexLen-1) + long := strings.Repeat("a", record.FingerprintDigestHexLen+1) + upper := strings.Repeat("A", record.FingerprintDigestHexLen) + nonHex := strings.Repeat("z", record.FingerprintDigestHexLen) + + for _, table := range []string{"finding", "handoff"} { + update := `UPDATE ` + table + ` SET fingerprint = ? WHERE rowid = 1` + if _, err := db.Exec(update, exact); err != nil { + t.Errorf("%s.fingerprint rejected a full %d-hex digest: %v", table, record.FingerprintDigestHexLen, err) + } + for name, bad := range map[string]string{"truncated": short, "over-long": long, "uppercase": upper, "non-hex": nonHex} { + if _, err := db.Exec(update, bad); err == nil { + t.Errorf("%s.fingerprint accepted a %s digest", table, name) + } + } + // Restore for the next table's FK-valid state. + if _, err := db.Exec(update, exact); err != nil { + t.Fatalf("restoring %s.fingerprint: %v", table, err) + } + } +} + +// TestHostFindingsAreNeverAgentRemediable is 00-SPINE.md S7 in the schema: the +// host agent is read-only, "no package manager in a mutating mode, not behind +// a flag." +func TestHostFindingsAreNeverAgentRemediable(t *testing.T) { + db := newDB(t) + seedAll(t, db) + + _, err := db.Exec(`INSERT INTO finding (finding_id, target_id, fingerprint, detector, evidence_class, rule_id, + remediable_by_agent, severity, title, state, first_seen_scan, first_seen_at) + VALUES (2, 1, ?, 'host', 'host', 'anvil.host.pkg/v1', 1, 'high', 'vulnerable host package', 'open', 1, '2026-08-08T00:30:00Z')`, + strings.Repeat("b", record.FingerprintDigestHexLen)) + if err == nil { + t.Error("a host finding was accepted with remediable_by_agent = 1") + } +} + +// TestDurableTextCapRejectsOversizedMessage is research/07 Risk #13's named +// mitigation: a raw snippet or DAST body copied into a durable column outlives +// the payload purge forever. +func TestDurableTextCapRejectsOversizedMessage(t *testing.T) { + db := newDB(t) + seedAll(t, db) + + for _, column := range []string{"message", "evidence_ref"} { + update := `UPDATE finding_occurrence SET ` + column + ` = ? WHERE occurrence_id = 1` + if _, err := db.Exec(update, strings.Repeat("x", MaxDurableTextBytes)); err != nil { + t.Errorf("finding_occurrence.%s rejected a value at exactly the cap: %v", column, err) + } + if _, err := db.Exec(update, strings.Repeat("x", MaxDurableTextBytes+1)); err == nil { + t.Errorf("finding_occurrence.%s accepted a value one byte over the cap", column) + } + } + + // The same cap must hold on INSERT, not only on UPDATE. + _, err := db.Exec(`INSERT INTO finding_occurrence (occurrence_id, finding_id, scan_run_id, message) + VALUES (2, 1, 1, ?)`, strings.Repeat("x", MaxDurableTextBytes+1)) + if err == nil { + t.Error("finding_occurrence accepted an oversized message on INSERT") + } +} + +func TestForeignKeysAreEnforced(t *testing.T) { + db := newDB(t) + seedAll(t, db) + + _, err := db.Exec(`INSERT INTO handoff (handoff_id, finding_id, audit_record_id, fingerprint, consumption_class, created_at, updated_at) + VALUES (99, 4242, 1, ?, 'static_only', '2026-08-08T00:00:00Z', '2026-08-08T00:00:00Z')`, fpA) + if err == nil { + t.Error("handoff accepted a row referencing a finding that does not exist") + } +} + +// TestLeasedRowMustNameItsHolder: a row in state 'leased' with no claimed_by +// is unreclaimable, because the reaper's requeue rule keys on the holder and +// the lease clock. +func TestLeasedRowMustNameItsHolder(t *testing.T) { + db := newDB(t) + seedAll(t, db) + + _, err := db.Exec(`UPDATE handoff SET state = 'leased', claimed_by = NULL WHERE handoff_id = 1`) + if err == nil { + t.Error("handoff accepted state = 'leased' with a NULL claimed_by") + } +} + +// TestSchemaCarriesNoPragma: R.5 applies this DDL inside BEGIN...COMMIT, and +// `PRAGMA journal_mode = WAL` cannot run inside a transaction. The pragmas +// live in ConnectionPragmas instead, because they are per connection anyway. +func TestSchemaCarriesNoPragma(t *testing.T) { + if strings.Contains(strings.ToUpper(stripSQLComments(Schema())), "PRAGMA") { + t.Error("schema.sql contains a PRAGMA statement; it cannot then be applied inside a transaction") + } + want := []string{"journal_mode = WAL", "foreign_keys = ON", "busy_timeout = 10000", "synchronous = NORMAL", "wal_autocheckpoint = 1000"} + got := ConnectionPragmas() + if len(got) != len(want) { + t.Fatalf("ConnectionPragmas returned %d pragmas %v, want %d", len(got), got, len(want)) + } + for i := range want { + if !strings.Contains(got[i], want[i]) { + t.Errorf("pragma %d is %q, want it to set %q", i, got[i], want[i]) + } + } +} + +// TestEveryResearch07TableSurvives: R.4 may not drop a research/07 table +// without logging the reason, and this schema drops none. +func TestEveryResearch07TableSurvives(t *testing.T) { + carriedForward := []string{ + "advisory", "advisory_alias", "advisory_fts", "component", "advisory_affects", + "target", "trigger_policy", "ingest_watermark", "scan_run", "audit_record", + "code_location", "finding", "finding_fingerprint", "finding_occurrence", + "finding_state_event", "fix_attempt", "verification", "suppression", + "file_state", "schema_migration", + } + created := Tables() + for _, want := range carriedForward { + if !contains(created, want) { + t.Errorf("research/07 table %q is missing from schema.sql", want) + } + } + if !contains(created, "handoff") { + t.Error("the handoff table is missing; S1 collapses the buffer into it") + } +} + +func TestSchemaSHA256IsStableAndFullLength(t *testing.T) { + first := SchemaSHA256() + if len(first) != record.FingerprintDigestHexLen { + t.Errorf("SchemaSHA256 returned %d characters, want %d", len(first), record.FingerprintDigestHexLen) + } + if second := SchemaSHA256(); first != second { + t.Errorf("SchemaSHA256 is not stable: %q then %q", first, second) + } +} + +func TestCheckConstraintReportsMissingNames(t *testing.T) { + if _, err := CheckConstraint("ck_not_a_real_constraint"); err == nil { + t.Error("CheckConstraint reported success for a constraint that does not exist") + } + if _, err := EnumCheckValues("ck_not_a_real_constraint"); err == nil { + t.Error("EnumCheckValues reported success for a constraint that does not exist") + } +} + +// ---- helpers ---- + +func columnInfo(t *testing.T, db *sql.DB, table, column string) (notNull bool, dflt sql.NullString) { + t.Helper() + + rows, err := db.Query(`SELECT name, "notnull", dflt_value FROM pragma_table_info(?)`, table) + if err != nil { + t.Fatalf("pragma_table_info(%s): %v", table, err) + } + defer func() { _ = rows.Close() }() + + for rows.Next() { + var name string + var nn int + var d sql.NullString + if err := rows.Scan(&name, &nn, &d); err != nil { + t.Fatalf("scanning pragma_table_info(%s): %v", table, err) + } + if name == column { + if err := rows.Err(); err != nil { + t.Fatalf("pragma_table_info(%s): %v", table, err) + } + return nn != 0, d + } + } + if err := rows.Err(); err != nil { + t.Fatalf("pragma_table_info(%s): %v", table, err) + } + t.Fatalf("%s has no column %q", table, column) + return false, sql.NullString{} +} + +func contains(haystack []string, needle string) bool { + for _, s := range haystack { + if s == needle { + return true + } + } + return false +} + +// stripSQLComments removes `-- ...` line comments so that a PRAGMA mentioned +// in a comment does not fail TestSchemaCarriesNoPragma. +func stripSQLComments(sqlText string) string { + var b strings.Builder + for _, line := range strings.Split(sqlText, "\n") { + if i := strings.Index(line, "--"); i >= 0 { + line = line[:i] + } + b.WriteString(line) + b.WriteByte('\n') + } + return b.String() +} diff --git a/internal/store/guards.go b/internal/store/guards.go new file mode 100644 index 0000000..6d5ac28 --- /dev/null +++ b/internal/store/guards.go @@ -0,0 +1,457 @@ +// Startup guards for the Anvil store (step R.5). +// +// plan/40-record-and-storage.md, "Startup guards (R.5), both mandatory": +// refuse to start if the data directory is on a network-mounted filesystem, +// and refuse to start if an FTS5 smoke-test virtual table cannot be created. +// Both must run before any other store operation; CheckStartup runs them in +// the required order and Migrate calls CheckFTS5 itself so that a caller who +// builds its own *sql.DB cannot skip it. +// +// Neither guard trusts a version number, a build tag, or a claim in a +// document. plan/00-SPINE.md S12 calls modernc.org/sqlite's FTS5 support +// "orchestrator-verified", but the research trail behind that +// (plan/spine-c-language.md C5-C7) grades its own evidence B, +// "absence-of-evidence, not evidence-of-absence". A dependency bump can drop a +// build-time feature without any signal at all, and the only thing that +// catches that is executing the feature at every process start. So CheckFTS5 +// creates a real FTS5 table, writes a real row, and runs a real MATCH. + +package store + +import ( + "bufio" + "context" + "database/sql" + "errors" + "fmt" + "io/fs" + "os" + "path/filepath" + "runtime" + "sort" + "strings" + "sync/atomic" +) + +// ErrNetworkMount reports that the data directory is on a filesystem where +// SQLite's write-ahead log is documented not to work. +// +// research/07-database-design.md Risk #4: "WAL does not work over a network +// filesystem", and homelab users routinely put data directories on NFS or SMB. +// The failure is silent corruption of the store of record, not an error at +// mount time, which is why this is a refusal and not a warning. +var ErrNetworkMount = errors.New("store: data directory is on a network filesystem") + +// ErrNoFTS5 reports that the SQLite build behind the *sql.DB cannot create or +// query an FTS5 virtual table. schema.sql's advisory_fts table needs it. +var ErrNoFTS5 = errors.New("store: SQLite FTS5 is unavailable") + +// CheckStartup runs both mandatory guards in the order the store needs them, +// and is the single call a process should make before any other store +// operation. +// +// The mount check comes first because it is answerable from the filesystem +// alone: if it fails there is no reason to have opened a database at all. +func CheckStartup(dataDir string, db *sql.DB) error { + if err := CheckNetworkMount(dataDir); err != nil { + return err + } + return CheckFTS5(db) +} + +// --------------------------------------------------------------------------- +// Guard 1 — network-mounted data directory +// --------------------------------------------------------------------------- + +// networkFilesystemTypes is the set of filesystem type names that mean "this +// data lives on another host". A type in this set is a hard refusal. +// +// The set is deliberately explicit rather than a heuristic: guessing from the +// mount source would refuse loopback-mounted images and other perfectly local +// setups, and a false refusal at startup is as user-hostile as a missed one is +// dangerous. +var networkFilesystemTypes = map[string]bool{ + "9p": true, // Plan 9 / virtio-9p, the usual VM share + "afpfs": true, + "afs": true, + "beegfs": true, + "ceph": true, + "cifs": true, + "coda": true, + "davfs": true, + "gfs2": true, + "glusterfs": true, + "lustre": true, + "ncpfs": true, + "nfs": true, + "nfs4": true, + "ocfs2": true, + "orangefs": true, + "pvfs2": true, + "smb2": true, + "smb3": true, + "smbfs": true, + "unc": true, // synthesised below for a Windows \\server\share path + "vboxsf": true, + "virtiofs": true, +} + +// networkFUSEBackends covers FUSE mounts, whose type is reported as +// "fuse.". The FUSE layer itself is local; the backend is what +// decides. +var networkFUSEBackends = map[string]bool{ + "cephfs": true, + "davfs": true, + "glusterfs": true, + "gvfsd-fuse": true, + "rclone": true, + "s3fs": true, + "sshfs": true, +} + +// fsTypeProbe reports the filesystem type backing path, or "" when the host +// gives no way to tell. It is a variable-shaped seam so guards_test.go can +// inject a fake network filesystem type without needing a real NFS server. +type fsTypeProbe func(path string) (string, error) + +// CheckNetworkMount refuses a data directory that sits on a network +// filesystem. +// +// KNOWN LIMIT, stated plainly because a guard that overstates its coverage is +// worse than one that does not exist. Filesystem type detection here is pure +// Go with no cgo and no build-tagged files (plan/00-SPINE.md S12), which means +// it reads /proc/self/mountinfo (falling back to /proc/mounts). On Linux — +// Anvil's deployment target — that is authoritative. On Windows it detects a +// UNC path but CANNOT see that a mapped drive letter such as Z: points at a +// share, because that needs GetDriveTypeW and therefore a windows-only source +// file this step does not own. On any host where the type cannot be +// determined the guard passes: refusing every path we cannot classify would +// make Anvil unstartable on Windows and macOS, and a guard nobody can get past +// gets deleted. Treat a positive result as reliable and a negative one as +// "not proven local". +func CheckNetworkMount(path string) error { + return checkNetworkMountWith(path, probeFilesystemType) +} + +func checkNetworkMountWith(path string, probe fsTypeProbe) error { + if strings.TrimSpace(path) == "" { + return fmt.Errorf("%w: no data directory was given", ErrNetworkMount) + } + fsType, err := probe(path) + if err != nil { + // The host offered a mount table and it could not be read or parsed. + // That is not "unknown", it is broken, and staying quiet about it is + // exactly the silence this guard exists to prevent. + return fmt.Errorf("store: cannot determine the filesystem type of data directory %q: %w", path, err) + } + if fsType == "" { + return nil // undeterminable; see the CheckNetworkMount doc comment + } + if !isNetworkFilesystemType(fsType) { + return nil + } + return fmt.Errorf("%w: %q is on a %q filesystem. SQLite's write-ahead log "+ + "does not work over a network filesystem, so leaving it here risks silent "+ + "corruption of the store of record rather than an error you would notice "+ + "(research/07-database-design.md Risk #4). Move the Anvil data directory to "+ + "local storage, or mount local storage at this path", + ErrNetworkMount, path, fsType) +} + +func isNetworkFilesystemType(fsType string) bool { + fsType = strings.ToLower(strings.TrimSpace(fsType)) + if networkFilesystemTypes[fsType] { + return true + } + if backend, ok := strings.CutPrefix(fsType, "fuse."); ok { + return networkFUSEBackends[backend] + } + return false +} + +// probeFilesystemType is the real probe: a UNC check on Windows, then the +// Linux mount table. +func probeFilesystemType(path string) (string, error) { + if runtime.GOOS == "windows" && isUNCPath(path) { + return "unc", nil + } + return mountTableFilesystemType(path) +} + +// isUNCPath reports whether path names a Windows network share directly, as +// \\server\share\... — the one network location Windows exposes without a +// syscall. +func isUNCPath(path string) bool { + p := strings.ReplaceAll(path, "/", `\`) + if !strings.HasPrefix(p, `\\`) { + return false + } + if strings.HasPrefix(p, `\\?\`) || strings.HasPrefix(p, `\\.\`) { + return false // device / extended-length namespace, not a share + } + rest := strings.Trim(p[2:], `\`) + host, share, ok := strings.Cut(rest, `\`) + return ok && host != "" && strings.TrimLeft(share, `\`) != "" +} + +// mountTableFilesystemType returns the type of the most specific mount point +// containing path, or "" if this host publishes no mount table. +func mountTableFilesystemType(path string) (string, error) { + target, err := resolveForMountLookup(path) + if err != nil { + return "", err + } + + mounts, err := readMountTable() + if err != nil { + return "", err + } + if len(mounts) == 0 { + return "", nil + } + + // Longest matching mount point wins: /mnt/nas is more specific than /, and + // a data directory under it is on the share, not on the root filesystem. + sort.SliceStable(mounts, func(i, j int) bool { + return len(mounts[i].point) > len(mounts[j].point) + }) + for _, m := range mounts { + if pathHasPrefix(m.point, target) { + return m.fsType, nil + } + } + return "", nil +} + +// resolveForMountLookup makes path absolute and resolves it against the +// nearest ancestor that actually exists, because the data directory is +// routinely created after this guard runs. +func resolveForMountLookup(path string) (string, error) { + abs, err := filepath.Abs(path) + if err != nil { + return "", fmt.Errorf("resolving %q: %w", path, err) + } + existing := abs + for { + if _, err := os.Lstat(existing); err == nil { + break + } + parent := filepath.Dir(existing) + if parent == existing { + return filepath.Clean(abs), nil + } + existing = parent + } + // A symlink can cross a mount boundary; the mount that matters is the one + // holding the real directory. Best effort — an unresolvable link is not a + // reason to refuse startup. + if resolved, err := filepath.EvalSymlinks(existing); err == nil { + suffix := strings.TrimPrefix(abs, existing) + return filepath.Clean(resolved + suffix), nil + } + return filepath.Clean(abs), nil +} + +type mountEntry struct { + point string + fsType string +} + +// readMountTable parses /proc/self/mountinfo, falling back to /proc/mounts. +// +// mountinfo is preferred because its mount-point field is unambiguous even +// when a filesystem is bind-mounted or has a non-root subtree mounted, both of +// which /proc/mounts renders confusingly. +func readMountTable() ([]mountEntry, error) { + entries, err := parseMountFile("/proc/self/mountinfo", parseMountinfoLine) + if err == nil { + return entries, nil + } + if !errors.Is(err, fs.ErrNotExist) { + return nil, err + } + entries, err = parseMountFile("/proc/mounts", parseProcMountsLine) + if err == nil { + return entries, nil + } + if errors.Is(err, fs.ErrNotExist) { + return nil, nil // no mount table on this host: undeterminable, not an error + } + return nil, err +} + +func parseMountFile(name string, parse func(string) (mountEntry, bool)) ([]mountEntry, error) { + f, err := os.Open(name) //nolint:gosec // fixed, non-user-supplied kernel path + if err != nil { + return nil, err + } + defer func() { _ = f.Close() }() + + var entries []mountEntry + scanner := bufio.NewScanner(f) + scanner.Buffer(make([]byte, 0, 64*1024), 1024*1024) + for scanner.Scan() { + if e, ok := parse(scanner.Text()); ok { + entries = append(entries, e) + } + } + if err := scanner.Err(); err != nil { + return nil, fmt.Errorf("reading %s: %w", name, err) + } + return entries, nil +} + +// parseMountinfoLine reads one /proc/self/mountinfo record. Layout: +// +// ID PARENT MAJ:MIN ROOT MOUNTPOINT OPTIONS [OPTIONAL...] - FSTYPE SOURCE SUPEROPTS +// +// The optional-fields run is variable length and terminated by a lone "-", so +// the separator has to be located rather than assumed at a fixed index. +func parseMountinfoLine(line string) (mountEntry, bool) { + fields := strings.Fields(line) + sep := -1 + for i, f := range fields { + if f == "-" { + sep = i + break + } + } + if sep < 5 || sep+1 >= len(fields) { + return mountEntry{}, false + } + return mountEntry{ + point: unescapeMountField(fields[4]), + fsType: unescapeMountField(fields[sep+1]), + }, true +} + +// parseProcMountsLine reads one /proc/mounts record: +// +// SOURCE MOUNTPOINT FSTYPE OPTIONS DUMP PASS +func parseProcMountsLine(line string) (mountEntry, bool) { + fields := strings.Fields(line) + if len(fields) < 3 { + return mountEntry{}, false + } + return mountEntry{ + point: unescapeMountField(fields[1]), + fsType: unescapeMountField(fields[2]), + }, true +} + +// unescapeMountField undoes the kernel's octal escaping of space (\040), tab +// (\011), newline (\012) and backslash (\134) in mount paths. Without it a +// data directory containing a space silently fails to match its own mount. +func unescapeMountField(s string) string { + if !strings.ContainsRune(s, '\\') { + return s + } + var b strings.Builder + b.Grow(len(s)) + for i := 0; i < len(s); i++ { + if s[i] != '\\' || i+3 >= len(s) { + b.WriteByte(s[i]) + continue + } + var v int + valid := true + for _, c := range []byte(s[i+1 : i+4]) { + if c < '0' || c > '7' { + valid = false + break + } + v = v*8 + int(c-'0') + } + if !valid || v > 0xff { + b.WriteByte(s[i]) + continue + } + b.WriteByte(byte(v)) + i += 3 + } + return b.String() +} + +// pathHasPrefix reports whether target is mountPoint or lies beneath it, +// comparing whole path elements so that /mnt/nas does not match /mnt/nasty. +func pathHasPrefix(mountPoint, target string) bool { + mp := filepath.Clean(mountPoint) + tg := filepath.Clean(target) + if mp == tg { + return true + } + if !strings.HasSuffix(mp, string(filepath.Separator)) { + mp += string(filepath.Separator) + } + return strings.HasPrefix(tg, mp) +} + +// --------------------------------------------------------------------------- +// Guard 2 — FTS5 availability +// --------------------------------------------------------------------------- + +// fts5ProbeSeq keeps concurrent or repeated probes on separate table names, so +// one probe can never observe another's leftovers and report a false pass. +var fts5ProbeSeq atomic.Uint64 + +// CheckFTS5 verifies, by doing it, that this database can create an FTS5 +// virtual table, index a row into it, and match that row back. +// +// It is deliberately not a version check, a build-tag check, or a lookup in +// pragma_compile_options: those report intent, and this guard exists to catch +// the case where intent and reality have come apart — a dependency bump that +// drops FTS5, or a driver that accepts the DDL and then indexes nothing. Both +// of those are silent failures at startup and loud, data-losing ones at first +// query against schema.sql's advisory_fts table. +// +// The probe table is created in the `temp` schema on a single pinned +// connection and dropped again, so it never touches the store file. +func CheckFTS5(db *sql.DB) error { + return CheckFTS5Context(context.Background(), db) +} + +// CheckFTS5Context is CheckFTS5 with a caller-supplied context. The +// context-free signature is the one plan/40-record-and-storage.md's R.5 packet +// specifies, so it stays; this is the form startup code with a deadline wants. +func CheckFTS5Context(ctx context.Context, db *sql.DB) error { + if db == nil { + return fmt.Errorf("%w: no database handle was given to the guard", ErrNoFTS5) + } + + // One pinned connection: `temp` objects are per connection, so creating on + // one and querying on another would test nothing and leak a table. + conn, err := db.Conn(ctx) + if err != nil { + return fmt.Errorf("store: FTS5 guard could not acquire a connection: %w", err) + } + defer func() { _ = conn.Close() }() + + table := fmt.Sprintf("anvil_fts5_probe_%d", fts5ProbeSeq.Add(1)) + drop := "DROP TABLE IF EXISTS temp." + table + _, _ = conn.ExecContext(ctx, drop) + defer func() { _, _ = conn.ExecContext(ctx, drop) }() + + const remedy = "Anvil's store needs FTS5 for schema.sql's advisory_fts table. " + + "plan/00-SPINE.md S12 pins modernc.org/sqlite precisely because it bundles " + + "FTS5; if this fails, the SQLite build behind this binary changed and the " + + "store cannot be opened safely" + + if _, err := conn.ExecContext(ctx, "CREATE VIRTUAL TABLE temp."+table+" USING fts5(body)"); err != nil { + return fmt.Errorf("%w: CREATE VIRTUAL TABLE ... USING fts5 was rejected: %v. %s", ErrNoFTS5, err, remedy) + } + if _, err := conn.ExecContext(ctx, "INSERT INTO temp."+table+"(body) VALUES ('anvil fts5 startup probe')"); err != nil { + return fmt.Errorf("%w: the FTS5 table was created but would not accept a row: %v. %s", ErrNoFTS5, err, remedy) + } + + var matched int + query := "SELECT count(*) FROM temp." + table + " WHERE " + table + " MATCH 'startup'" + if err := conn.QueryRowContext(ctx, query).Scan(&matched); err != nil { + return fmt.Errorf("%w: the FTS5 table was created and written but MATCH failed: %v. %s", ErrNoFTS5, err, remedy) + } + if matched != 1 { + return fmt.Errorf("%w: MATCH 'startup' returned %d rows, want exactly 1 — this SQLite build "+ + "accepts FTS5 syntax without indexing anything, which is the silent-failure case the guard "+ + "exists to catch. %s", ErrNoFTS5, matched, remedy) + } + return nil +} diff --git a/internal/store/guards_test.go b/internal/store/guards_test.go new file mode 100644 index 0000000..d93e1b7 --- /dev/null +++ b/internal/store/guards_test.go @@ -0,0 +1,317 @@ +package store + +import ( + "context" + "database/sql" + "database/sql/driver" + "errors" + "io" + "path/filepath" + "strings" + "testing" + + _ "modernc.org/sqlite" // cgo-free driver, plan/00-SPINE.md S12 +) + +// --------------------------------------------------------------------------- +// Guard 1 — CheckNetworkMount +// --------------------------------------------------------------------------- + +// TestCheckNetworkMountRefusesInjectedNetworkFilesystems is the R.5 packet's +// "CheckNetworkMount fails against an injected fake network filesystem type" +// evidence item. The probe is the seam; no NFS server is involved. +func TestCheckNetworkMountRefusesInjectedNetworkFilesystems(t *testing.T) { + refused := []string{ + "nfs", "nfs4", "cifs", "smb3", "smbfs", "9p", "ceph", "glusterfs", + "lustre", "afs", "davfs", "vboxsf", "virtiofs", "unc", + "fuse.sshfs", "fuse.davfs", "fuse.s3fs", "fuse.rclone", + "NFS4", " cifs ", // case and whitespace must not smuggle a mount past + } + for _, fsType := range refused { + t.Run(strings.TrimSpace(fsType), func(t *testing.T) { + probe := func(string) (string, error) { return fsType, nil } + err := checkNetworkMountWith("/srv/anvil", probe) + if err == nil { + t.Fatalf("CheckNetworkMount accepted a %q data directory", fsType) + } + if !errors.Is(err, ErrNetworkMount) { + t.Fatalf("error does not wrap ErrNetworkMount: %v", err) + } + // "Fails loudly" means the operator can act on the message: it has + // to name the path and say what to do. + for _, want := range []string{"/srv/anvil", "write-ahead log", "local storage"} { + if !strings.Contains(err.Error(), want) { + t.Errorf("error message omits %q: %v", want, err) + } + } + }) + } +} + +func TestCheckNetworkMountAcceptsLocalFilesystems(t *testing.T) { + // fuseblk and fuse.gocryptfs are local; refusing them would be a false + // positive that pushes users to delete the guard. + for _, fsType := range []string{"ext4", "xfs", "btrfs", "zfs", "apfs", "ntfs", "overlay", "tmpfs", "fuseblk", "fuse.gocryptfs"} { + probe := func(string) (string, error) { return fsType, nil } + if err := checkNetworkMountWith("/srv/anvil", probe); err != nil { + t.Errorf("CheckNetworkMount refused local filesystem %q: %v", fsType, err) + } + } +} + +func TestCheckNetworkMountPassesWhenTypeIsUndeterminable(t *testing.T) { + probe := func(string) (string, error) { return "", nil } + if err := checkNetworkMountWith("/srv/anvil", probe); err != nil { + t.Fatalf("an undeterminable filesystem type must not block startup: %v", err) + } +} + +// TestCheckNetworkMountSurfacesProbeFailures pins the difference between "this +// host has no mount table" (pass) and "this host has one and it could not be +// read" (fail). The second is the case where staying quiet would hide exactly +// what the guard is for. +func TestCheckNetworkMountSurfacesProbeFailures(t *testing.T) { + sentinel := errors.New("mount table is unreadable") + probe := func(string) (string, error) { return "", sentinel } + err := checkNetworkMountWith("/srv/anvil", probe) + if !errors.Is(err, sentinel) { + t.Fatalf("probe failure was swallowed: %v", err) + } +} + +func TestCheckNetworkMountRejectsEmptyPath(t *testing.T) { + if err := CheckNetworkMount(" "); !errors.Is(err, ErrNetworkMount) { + t.Fatalf("empty data directory should be refused, got %v", err) + } +} + +// TestCheckNetworkMountOnRealHostAcceptsTempDir exercises the real probe end +// to end. A temp directory is local on every machine this is expected to run +// on, so a refusal here means the probe misreads its own host. +func TestCheckNetworkMountOnRealHostAcceptsTempDir(t *testing.T) { + if err := CheckNetworkMount(t.TempDir()); err != nil { + t.Fatalf("real probe refused a local temp directory: %v", err) + } + // A directory that does not exist yet must resolve through its nearest + // existing ancestor: the data directory is created after this guard runs. + if err := CheckNetworkMount(filepath.Join(t.TempDir(), "anvil", "data")); err != nil { + t.Fatalf("real probe refused a not-yet-created data directory: %v", err) + } +} + +func TestUNCPathDetection(t *testing.T) { + unc := []string{`\\nas\anvil`, `\\nas\anvil\data`, `//nas/anvil`} + for _, p := range unc { + if !isUNCPath(p) { + t.Errorf("isUNCPath(%q) = false, want true", p) + } + } + notUNC := []string{`C:\anvil`, `/srv/anvil`, `\\?\C:\anvil`, `\\.\PIPE\x`, `\\nas`, `\\nas\`, ``} + for _, p := range notUNC { + if isUNCPath(p) { + t.Errorf("isUNCPath(%q) = true, want false", p) + } + } +} + +func TestMountTableParsing(t *testing.T) { + mountinfo := "36 35 98:0 / /srv rw,noatime shared:1 - ext4 /dev/root rw\n" + + "41 36 0:41 / /srv/anvil\\040data rw - nfs4 nas:/export/anvil rw\n" + + "42 36 0:42 / /srv/anvilx rw - ext4 /dev/sdb rw\n" + var got []mountEntry + for _, line := range strings.Split(strings.TrimSuffix(mountinfo, "\n"), "\n") { + e, ok := parseMountinfoLine(line) + if !ok { + t.Fatalf("failed to parse mountinfo line %q", line) + } + got = append(got, e) + } + if got[1].point != "/srv/anvil data" || got[1].fsType != "nfs4" { + t.Fatalf("octal-escaped mount point mis-parsed: %+v", got[1]) + } + + // /proc/mounts has a different field order; both feed the same lookup. + e, ok := parseProcMountsLine(`nas:/export /srv/anvil\040data nfs4 rw,relatime 0 0`) + if !ok || e.point != "/srv/anvil data" || e.fsType != "nfs4" { + t.Fatalf("/proc/mounts line mis-parsed: %+v ok=%v", e, ok) + } + + // A mountinfo line with no "-" separator is malformed, not a mount. + if _, ok := parseMountinfoLine("36 35 98:0 / /srv rw ext4 /dev/root rw"); ok { + t.Fatal("a mountinfo line without the optional-fields separator must not parse") + } +} + +// TestLongestMountPointWins is the whole reason the mount table is sorted: +// a share mounted under the root filesystem must not be reported as the root +// filesystem's type. +func TestLongestMountPointWins(t *testing.T) { + if pathHasPrefix("/srv/anvil", "/srv/anvilx/data") { + t.Fatal("/srv/anvil must not match /srv/anvilx/data — whole path elements only") + } + if !pathHasPrefix("/srv/anvil", "/srv/anvil") { + t.Fatal("a mount point is a prefix of itself") + } + if !pathHasPrefix("/", "/srv/anvil") { + t.Fatal("the root filesystem contains everything") + } +} + +// --------------------------------------------------------------------------- +// Guard 2 — CheckFTS5 +// --------------------------------------------------------------------------- + +// TestCheckFTS5PassesAgainstTheRealDriver is the positive control. It is also +// the check the R.5 packet actually cares about at run time: if a future +// modernc.org/sqlite bump drops FTS5, this test goes red in CI on the same +// commit that bumps the dependency. +func TestCheckFTS5PassesAgainstTheRealDriver(t *testing.T) { + db := openMemory(t) + if err := CheckFTS5(db); err != nil { + t.Fatalf("FTS5 guard failed against modernc.org/sqlite: %v", err) + } + // Repeat runs must not collide on the probe table name, and must leave no + // trace behind on the pooled connection. + if err := CheckFTS5(db); err != nil { + t.Fatalf("second FTS5 probe failed: %v", err) + } + var leftovers int + err := db.QueryRow(`SELECT count(*) FROM temp.sqlite_schema WHERE name LIKE 'anvil_fts5_probe%'`).Scan(&leftovers) + if err != nil { + t.Fatalf("counting temp objects: %v", err) + } + if leftovers != 0 { + t.Fatalf("FTS5 probe left %d temp objects behind", leftovers) + } +} + +func TestCheckFTS5RejectsNilHandle(t *testing.T) { + if err := CheckFTS5(nil); !errors.Is(err, ErrNoFTS5) { + t.Fatalf("nil handle should be refused with ErrNoFTS5, got %v", err) + } +} + +// TestCheckFTS5FailsLoudlyWithoutFTS5 stands in for the packet's "build tag +// that disables FTS5". modernc.org/sqlite exposes no such tag — its FTS5 is +// compiled in unconditionally — so the absent capability is injected at the +// database/sql driver layer instead, which exercises the identical code path +// in CheckFTS5 including the error it produces. +func TestCheckFTS5FailsLoudlyWithoutFTS5(t *testing.T) { + db, err := sql.Open(driverNoFTS5, "") + if err != nil { + t.Fatalf("sql.Open: %v", err) + } + t.Cleanup(func() { _ = db.Close() }) + + err = CheckFTS5(db) + if err == nil { + t.Fatal("CheckFTS5 passed against a driver with no fts5 module") + } + if !errors.Is(err, ErrNoFTS5) { + t.Fatalf("error does not wrap ErrNoFTS5: %v", err) + } + for _, want := range []string{"CREATE VIRTUAL TABLE", "no such module: fts5", "advisory_fts"} { + if !strings.Contains(err.Error(), want) { + t.Errorf("error message omits %q: %v", want, err) + } + } +} + +// TestCheckFTS5FailsWhenMatchSilentlyIndexesNothing is the "not silently" half +// of the packet's requirement, and the reason this guard runs a real MATCH +// rather than stopping at a successful CREATE. A build that accepts the DDL +// and indexes nothing would pass a syntax-only probe and then return zero +// advisories forever. +func TestCheckFTS5FailsWhenMatchSilentlyIndexesNothing(t *testing.T) { + db, err := sql.Open(driverSilentFTS5, "") + if err != nil { + t.Fatalf("sql.Open: %v", err) + } + t.Cleanup(func() { _ = db.Close() }) + + err = CheckFTS5(db) + if err == nil { + t.Fatal("CheckFTS5 passed against a driver whose FTS5 indexes nothing") + } + if !errors.Is(err, ErrNoFTS5) { + t.Fatalf("error does not wrap ErrNoFTS5: %v", err) + } + if !strings.Contains(err.Error(), "returned 0 rows") { + t.Errorf("error message does not say what went wrong: %v", err) + } +} + +func TestCheckStartupRunsBothGuards(t *testing.T) { + // A refused mount short-circuits before the database is touched. + if err := CheckStartup("", nil); !errors.Is(err, ErrNetworkMount) { + t.Fatalf("CheckStartup should fail on the mount guard first, got %v", err) + } + // A good directory still has to clear the FTS5 guard. + if err := CheckStartup(t.TempDir(), nil); !errors.Is(err, ErrNoFTS5) { + t.Fatalf("CheckStartup should reach the FTS5 guard, got %v", err) + } + if err := CheckStartup(t.TempDir(), openMemory(t)); err != nil { + t.Fatalf("CheckStartup failed on a healthy local store: %v", err) + } +} + +// --------------------------------------------------------------------------- +// Test fixtures: two database/sql drivers with broken FTS5 +// --------------------------------------------------------------------------- + +const ( + driverNoFTS5 = "anvil-test-no-fts5" + driverSilentFTS5 = "anvil-test-silent-fts5" +) + +func init() { + sql.Register(driverNoFTS5, fakeDriver{silent: false}) + sql.Register(driverSilentFTS5, fakeDriver{silent: true}) +} + +// fakeDriver is the smallest database/sql driver that can answer CheckFTS5's +// three statements. silent=false rejects the CREATE the way a SQLite build +// without the fts5 module does; silent=true accepts everything and then +// matches nothing. +type fakeDriver struct{ silent bool } + +func (d fakeDriver) Open(string) (driver.Conn, error) { return fakeConn{silent: d.silent}, nil } + +type fakeConn struct{ silent bool } + +func (c fakeConn) Prepare(string) (driver.Stmt, error) { + return nil, errors.New("fake driver: Prepare is not implemented; use the context methods") +} +func (c fakeConn) Close() error { return nil } +func (c fakeConn) Begin() (driver.Tx, error) { return nil, errors.New("fake driver: no transactions") } + +func (c fakeConn) ExecContext(_ context.Context, query string, _ []driver.NamedValue) (driver.Result, error) { + if !c.silent && strings.Contains(query, "USING fts5") { + return nil, errors.New("no such module: fts5") + } + return driver.RowsAffected(0), nil +} + +func (c fakeConn) QueryContext(_ context.Context, query string, _ []driver.NamedValue) (driver.Rows, error) { + if strings.Contains(query, "MATCH") { + return &fakeRows{cols: []string{"count(*)"}, vals: [][]driver.Value{{int64(0)}}}, nil + } + return &fakeRows{cols: []string{"x"}}, nil +} + +type fakeRows struct { + cols []string + vals [][]driver.Value + next int +} + +func (r *fakeRows) Columns() []string { return r.cols } +func (r *fakeRows) Close() error { return nil } +func (r *fakeRows) Next(dest []driver.Value) error { + if r.next >= len(r.vals) { + return io.EOF + } + copy(dest, r.vals[r.next]) + r.next++ + return nil +} diff --git a/internal/store/migrate.go b/internal/store/migrate.go new file mode 100644 index 0000000..6c25db4 --- /dev/null +++ b/internal/store/migrate.go @@ -0,0 +1,477 @@ +// Forward-only numbered migrations with a checksummed ledger (step R.5). +// +// research/07-database-design.md §7: "numbered forward-only SQL + +// `PRAGMA user_version` + a checksummed ledger". No Alembic, no goose, no +// sqlx-cli, nothing for a self-hoster to install — plan/00-SPINE.md S12 makes +// this Go-only, and the migrations are embedded in the binary. +// +// Three properties are load-bearing and each has a test: +// +// 1. Each migration runs inside BEGIN ... COMMIT and bumps `user_version` in +// the SAME transaction, so a failure leaves the schema untouched rather +// than half-applied. +// 2. Every applied migration's checksum is re-verified at startup against the +// embedded text. A mismatch is a refusal, not a re-run and not a skip: it +// means the database was built by a different definition of the schema +// than the one this binary carries, and either guess about what to do next +// corrupts the store of record. +// 3. Forward-only. There are no down migrations. The rollback story is +// `VACUUM INTO 'anvil-pre-v{N}.db'` taken before the first migration +// touches an existing database. + +package store + +import ( + "context" + "crypto/sha256" + "database/sql" + "embed" + "encoding/hex" + "errors" + "fmt" + "io/fs" + "os" + "path" + "path/filepath" + "regexp" + "sort" + "strconv" + "strings" + "sync" + "time" +) + +//go:embed migrations/*.sql +var migrationFS embed.FS + +const migrationsDir = "migrations" + +// includeDirectivePrefix introduces the one preprocessor directive a migration +// file may contain. See migrations/0001_init.sql for why it exists: schema.sql +// is a frozen interface with exactly one copy in this repository, and a +// migration references it rather than duplicating it. +const includeDirectivePrefix = "-- @anvil:include " + +// ErrMigrationLedger reports that the database's migration history and this +// binary's embedded migrations disagree. It is always a refusal to proceed. +var ErrMigrationLedger = errors.New("store: migration ledger mismatch") + +// ErrSnapshotRequired reports that migrating an already-populated database was +// attempted without somewhere to put the pre-migration snapshot. +var ErrSnapshotRequired = errors.New("store: pre-migration snapshot directory required") + +var migrationFilenameRE = regexp.MustCompile(`^(\d{4})_([a-z0-9]+(?:_[a-z0-9]+)*)\.sql$`) + +// Migration is one numbered, embedded, forward-only migration. +type Migration struct { + // Version is the number in the filename; versions are contiguous from 1. + Version int + // Name is the descriptive part of the filename, e.g. "init". + Name string + // Filename is the name as embedded, e.g. "0001_init.sql". + Filename string + // SQL is the fully expanded text that will be executed: the file's bytes + // with every include directive replaced by the included content. + SQL string + // Checksum is the lowercase hex SHA-256 of SQL. This is the value written + // to and re-verified against schema_migration.checksum. + Checksum string +} + +var loadMigrations = sync.OnceValues(func() ([]Migration, error) { + entries, err := fs.ReadDir(migrationFS, migrationsDir) + if err != nil { + return nil, fmt.Errorf("store: reading embedded migrations: %w", err) + } + + migrations := make([]Migration, 0, len(entries)) + for _, e := range entries { + if e.IsDir() { + return nil, fmt.Errorf("store: %s/%s is a directory; migrations are flat", migrationsDir, e.Name()) + } + m := migrationFilenameRE.FindStringSubmatch(e.Name()) + if m == nil { + return nil, fmt.Errorf("store: migration filename %q does not match NNNN_lower_snake_name.sql", e.Name()) + } + version, err := strconv.Atoi(m[1]) + if err != nil { // unreachable: the regexp already proved four digits + return nil, fmt.Errorf("store: migration %q has an unparseable version: %w", e.Name(), err) + } + raw, err := migrationFS.ReadFile(path.Join(migrationsDir, e.Name())) + if err != nil { + return nil, fmt.Errorf("store: reading migration %q: %w", e.Name(), err) + } + expanded, err := expandIncludes(e.Name(), string(raw)) + if err != nil { + return nil, err + } + sum := sha256.Sum256([]byte(expanded)) + migrations = append(migrations, Migration{ + Version: version, + Name: m[2], + Filename: e.Name(), + SQL: expanded, + Checksum: hex.EncodeToString(sum[:]), + }) + } + + sort.Slice(migrations, func(i, j int) bool { return migrations[i].Version < migrations[j].Version }) + for i, mig := range migrations { + if want := i + 1; mig.Version != want { + return nil, fmt.Errorf("store: migration versions must be contiguous from 0001; "+ + "expected %04d at position %d but found %q", want, i+1, mig.Filename) + } + } + if len(migrations) == 0 { + return nil, errors.New("store: no migrations are embedded; the store cannot be created") + } + return migrations, nil +}) + +// Migrations returns every embedded migration in ascending version order. +// +// It fails rather than returning a partial list if the filenames are malformed +// or the numbering has a gap or duplicate, because "migrate what parses" is +// how a store ends up at a version nobody can describe. +func Migrations() ([]Migration, error) { + migrations, err := loadMigrations() + if err != nil { + return nil, err + } + return append([]Migration(nil), migrations...), nil +} + +// expandIncludes replaces each `-- @anvil:include ` line with the +// content of the named file. Today the only legal target is schema.sql, whose +// bytes come from the string ddl.go embeds — so there is exactly one copy of +// the schema in the repository and the migration cannot drift from it. +func expandIncludes(filename, src string) (string, error) { + if !strings.Contains(src, includeDirectivePrefix) { + return src, nil + } + + var out strings.Builder + out.Grow(len(src) + len(schemaSQL)) + rest := src + lineNo := 0 + for len(rest) > 0 { + line := rest + if i := strings.IndexByte(rest, '\n'); i >= 0 { + line, rest = rest[:i+1], rest[i+1:] + } else { + rest = "" + } + lineNo++ + + trimmed := strings.TrimSpace(line) + if !strings.HasPrefix(trimmed, includeDirectivePrefix) { + out.WriteString(line) + continue + } + target := strings.TrimSpace(strings.TrimPrefix(trimmed, includeDirectivePrefix)) + if target != "schema.sql" { + return "", fmt.Errorf("store: %s line %d includes %q; the only includable file is schema.sql", + filename, lineNo, target) + } + out.WriteString(Schema()) + if !strings.HasSuffix(Schema(), "\n") { + out.WriteString("\n") + } + } + return out.String(), nil +} + +// appliedMigration is one row of the schema_migration ledger. +type appliedMigration struct { + Version int + Name string + Checksum string + AppliedAt string +} + +// Migrate brings db up to the latest embedded schema version and returns the +// versions it applied, in order. An already-current database returns an empty +// slice and touches nothing. +// +// snapshotDir is where the `VACUUM INTO 'anvil-pre-v{N}.db'` pre-migration +// snapshot is written. It may be empty ONLY when the database has no applied +// migrations — a fresh database has nothing to lose, so demanding a snapshot +// of it would just teach callers to pass a junk directory. Upgrading a +// populated database without one is refused: research/07-database-design.md §7 +// makes the snapshot the entire replacement for down migrations, so skipping +// it means the upgrade has no rollback path at all. +// +// CheckFTS5 runs first, on the same handle. plan/40-record-and-storage.md +// requires the guards before any other store operation, and applying +// schema.sql is a store operation — its advisory_fts table would otherwise +// fail deep inside the transaction with a driver-level error instead of the +// guard's explanation. +func Migrate(ctx context.Context, db *sql.DB, snapshotDir string) ([]int, error) { + migrations, err := Migrations() + if err != nil { + return nil, err + } + return migrateWith(ctx, db, migrations, snapshotDir) +} + +// migrateWith is Migrate over an explicit migration list. It exists so +// migrate_test.go can exercise ordering, mid-sequence failure and the snapshot +// rule against synthetic later versions: those paths cannot otherwise be +// tested while 0001 is the only embedded migration, and leaving them untested +// until the first real 0002 means discovering them during an upgrade. +func migrateWith(ctx context.Context, db *sql.DB, migrations []Migration, snapshotDir string) ([]int, error) { + if db == nil { + return nil, errors.New("store: Migrate needs a database handle") + } + if err := CheckFTS5Context(ctx, db); err != nil { + return nil, err + } + + applied, err := verifyLedger(ctx, db, migrations) + if err != nil { + return nil, err + } + + pending := migrations[len(applied):] + if len(pending) == 0 { + return nil, nil + } + + if len(applied) > 0 { + if strings.TrimSpace(snapshotDir) == "" { + return nil, fmt.Errorf("%w: this database is at schema version %d and %d migration(s) are "+ + "pending. Migrations are forward-only and there are no down migrations "+ + "(research/07-database-design.md §7), so a VACUUM INTO snapshot is the only rollback "+ + "path; pass a directory to write it to", ErrSnapshotRequired, len(applied), len(pending)) + } + dest, err := snapshotPath(snapshotDir, len(applied)) + if err != nil { + return nil, err + } + if err := Snapshot(ctx, db, dest); err != nil { + return nil, err + } + } + + appliedNow := make([]int, 0, len(pending)) + for _, m := range pending { + if err := applyOne(ctx, db, m); err != nil { + return appliedNow, err + } + appliedNow = append(appliedNow, m.Version) + } + return appliedNow, nil +} + +// verifyLedger reads schema_migration and PRAGMA user_version and proves they +// agree with each other and with the embedded migrations. It returns the +// applied migrations in version order. +// +// Every disagreement here is fatal. The three it names are the ones +// research/07 §7 says actually happen to self-hosted tools: a hand-edited +// migration, a downgraded binary, and a database that was interrupted or +// touched by something that is not this code. +func verifyLedger(ctx context.Context, db *sql.DB, migrations []Migration) ([]appliedMigration, error) { + userVersion, err := SchemaVersion(ctx, db) + if err != nil { + return nil, err + } + + hasLedger, err := tableExists(ctx, db, "schema_migration") + if err != nil { + return nil, err + } + if !hasLedger { + if userVersion != 0 { + return nil, fmt.Errorf("%w: PRAGMA user_version is %d but there is no schema_migration table. "+ + "This database was not created by Anvil, or its ledger was dropped; refusing to guess which "+ + "migrations it has", ErrMigrationLedger, userVersion) + } + return nil, nil + } + + applied, err := readLedger(ctx, db) + if err != nil { + return nil, err + } + if len(applied) == 0 { + if userVersion != 0 { + return nil, fmt.Errorf("%w: PRAGMA user_version is %d but schema_migration is empty", + ErrMigrationLedger, userVersion) + } + return nil, nil + } + + byVersion := make(map[int]Migration, len(migrations)) + for _, m := range migrations { + byVersion[m.Version] = m + } + + for i, row := range applied { + if want := i + 1; row.Version != want { + return nil, fmt.Errorf("%w: schema_migration jumps from version %d to %d. The history has a "+ + "gap, so the schema in this file cannot be reconstructed from the migrations in this binary", + ErrMigrationLedger, want-1, row.Version) + } + embedded, ok := byVersion[row.Version] + if !ok { + return nil, fmt.Errorf("%w: the database has applied migration %04d (%q) which this binary does "+ + "not contain. This is an older Anvil opening a newer store; migrations are forward-only, so "+ + "downgrading is not supported. Run the newer binary, or restore an anvil-pre-v*.db snapshot", + ErrMigrationLedger, row.Version, row.Name) + } + if row.Name != embedded.Name { + return nil, fmt.Errorf("%w: migration %04d was applied as %q but this binary calls it %q", + ErrMigrationLedger, row.Version, row.Name, embedded.Name) + } + if row.Checksum != embedded.Checksum { + return nil, fmt.Errorf("%w: migration %s no longer matches what was applied to this database "+ + "(ledger recorded %s, this binary carries %s). Either the migration file or schema.sql was "+ + "edited after the fact, or this binary's schema differs from the one that built this store. "+ + "Refusing to start: re-running it would apply DDL twice and skipping it would run against a "+ + "schema that was never applied", + ErrMigrationLedger, embedded.Filename, row.Checksum, embedded.Checksum) + } + } + + if maxApplied := applied[len(applied)-1].Version; userVersion != maxApplied { + return nil, fmt.Errorf("%w: PRAGMA user_version is %d but the highest applied migration is %d. "+ + "They are written in one transaction, so this means the file was modified outside Anvil", + ErrMigrationLedger, userVersion, maxApplied) + } + return applied, nil +} + +// applyOne runs a single migration inside one transaction, together with the +// user_version bump and the ledger row. +// +// SQLite DDL is transactional, so a failure anywhere in here leaves the schema +// exactly as it was — which is the reason this is one transaction and not +// three statements. +func applyOne(ctx context.Context, db *sql.DB, m Migration) error { + tx, err := db.BeginTx(ctx, nil) + if err != nil { + return fmt.Errorf("store: migration %s: BEGIN failed: %w", m.Filename, err) + } + committed := false + defer func() { + if !committed { + _ = tx.Rollback() + } + }() + + if _, err := tx.ExecContext(ctx, m.SQL); err != nil { + return fmt.Errorf("store: migration %s failed and was rolled back; the schema is unchanged: %w", m.Filename, err) + } + + // PRAGMA takes no bound parameters, so the version is formatted in. It is + // an int parsed from a \d{4} filename match, not caller input. + if _, err := tx.ExecContext(ctx, "PRAGMA user_version = "+strconv.Itoa(m.Version)); err != nil { + return fmt.Errorf("store: migration %s: bumping user_version failed and was rolled back: %w", m.Filename, err) + } + + if _, err := tx.ExecContext(ctx, + `INSERT INTO schema_migration (version, name, checksum, applied_at) VALUES (?, ?, ?, ?)`, + m.Version, m.Name, m.Checksum, time.Now().UTC().Format(time.RFC3339Nano), + ); err != nil { + return fmt.Errorf("store: migration %s: recording the ledger row failed and was rolled back: %w", m.Filename, err) + } + + if err := tx.Commit(); err != nil { + return fmt.Errorf("store: migration %s: COMMIT failed; the schema is unchanged: %w", m.Filename, err) + } + committed = true + return nil +} + +// Snapshot writes a consistent copy of the whole database to dest using +// `VACUUM INTO`, which research/07-database-design.md §7 makes the substitute +// for down migrations. +// +// dest must not already exist: SQLite refuses to overwrite, and so does this — +// silently replacing the only pre-upgrade copy of a store would be the worst +// possible way to be helpful. +func Snapshot(ctx context.Context, db *sql.DB, dest string) error { + if db == nil { + return errors.New("store: Snapshot needs a database handle") + } + if strings.TrimSpace(dest) == "" { + return errors.New("store: Snapshot needs a destination path") + } + // VACUUM cannot run inside a transaction, so this goes to db directly. The + // path is a SQL string literal because SQLite does not bind parameters in + // VACUUM INTO; doubling embedded quotes is the whole escaping rule for a + // SQLite string literal. + quoted := "'" + strings.ReplaceAll(dest, "'", "''") + "'" + if _, err := db.ExecContext(ctx, "VACUUM INTO "+quoted); err != nil { + return fmt.Errorf("store: pre-migration snapshot to %q failed; refusing to migrate without one: %w", dest, err) + } + return nil +} + +// snapshotPath builds research/07's `anvil-pre-v{N}.db` name inside dir, +// creating dir if needed. If that name is taken — a second upgrade attempt +// after a failure — a timestamp is appended rather than clobbering the +// existing snapshot. +func snapshotPath(dir string, currentVersion int) (string, error) { + if err := os.MkdirAll(dir, 0o750); err != nil { + return "", fmt.Errorf("store: creating snapshot directory %q: %w", dir, err) + } + base := filepath.Join(dir, fmt.Sprintf("anvil-pre-v%d.db", currentVersion)) + if _, err := os.Lstat(base); errors.Is(err, fs.ErrNotExist) { + return base, nil + } + return filepath.Join(dir, fmt.Sprintf("anvil-pre-v%d-%d.db", currentVersion, time.Now().UTC().UnixNano())), nil +} + +// SchemaVersion returns the database's PRAGMA user_version, which is 0 for a +// database no migration has touched. +func SchemaVersion(ctx context.Context, db *sql.DB) (int, error) { + var v int + if err := db.QueryRowContext(ctx, "PRAGMA user_version").Scan(&v); err != nil { + return 0, fmt.Errorf("store: reading PRAGMA user_version: %w", err) + } + return v, nil +} + +// LatestVersion returns the highest embedded migration version — the schema +// version a fully migrated database will report. +func LatestVersion() (int, error) { + migrations, err := loadMigrations() + if err != nil { + return 0, err + } + return migrations[len(migrations)-1].Version, nil +} + +func tableExists(ctx context.Context, db *sql.DB, name string) (bool, error) { + var n int + err := db.QueryRowContext(ctx, + `SELECT count(*) FROM sqlite_schema WHERE type = 'table' AND name = ?`, name).Scan(&n) + if err != nil { + return false, fmt.Errorf("store: looking for table %q: %w", name, err) + } + return n > 0, nil +} + +func readLedger(ctx context.Context, db *sql.DB) ([]appliedMigration, error) { + rows, err := db.QueryContext(ctx, + `SELECT version, name, checksum, applied_at FROM schema_migration ORDER BY version`) + if err != nil { + return nil, fmt.Errorf("store: reading schema_migration: %w", err) + } + defer func() { _ = rows.Close() }() + + var applied []appliedMigration + for rows.Next() { + var a appliedMigration + if err := rows.Scan(&a.Version, &a.Name, &a.Checksum, &a.AppliedAt); err != nil { + return nil, fmt.Errorf("store: scanning schema_migration: %w", err) + } + applied = append(applied, a) + } + if err := rows.Err(); err != nil { + return nil, fmt.Errorf("store: iterating schema_migration: %w", err) + } + return applied, nil +} diff --git a/internal/store/migrate_test.go b/internal/store/migrate_test.go new file mode 100644 index 0000000..a6f4daf --- /dev/null +++ b/internal/store/migrate_test.go @@ -0,0 +1,556 @@ +package store + +import ( + "context" + "crypto/sha256" + "database/sql" + "encoding/hex" + "errors" + "fmt" + "os" + "path/filepath" + "strings" + "testing" + + _ "modernc.org/sqlite" // cgo-free driver, plan/00-SPINE.md S12 +) + +// openMemory returns an empty in-memory database with ConnectionPragmas +// applied. MaxOpenConns is 1 because every new connection to ":memory:" is a +// different, empty database — and because the pragmas are per connection. +func openMemory(t *testing.T) *sql.DB { + t.Helper() + db, err := sql.Open("sqlite", ":memory:") + if err != nil { + t.Fatalf("sql.Open: %v", err) + } + t.Cleanup(func() { _ = db.Close() }) + db.SetMaxOpenConns(1) + applyPragmas(t, db) + return db +} + +// openOnDisk returns an empty file-backed database and its directory. The +// snapshot path has to be exercised against a real file: VACUUM INTO on a +// database that never touched a filesystem would prove nothing about the +// rollback story it replaces. +func openOnDisk(t *testing.T) (*sql.DB, string) { + t.Helper() + dir := t.TempDir() + db, err := sql.Open("sqlite", filepath.Join(dir, "anvil.db")) + if err != nil { + t.Fatalf("sql.Open: %v", err) + } + t.Cleanup(func() { _ = db.Close() }) + db.SetMaxOpenConns(1) + applyPragmas(t, db) + return db, dir +} + +func applyPragmas(t *testing.T, db *sql.DB) { + t.Helper() + for _, p := range ConnectionPragmas() { + if _, err := db.Exec(p); err != nil { + t.Fatalf("pragma %q: %v", p, err) + } + } +} + +// schemaObjects renders every object SQLite actually created, which is the +// tool-free equivalent of the packet's `sqlite3 .schema` diff — and a stricter +// one, since it compares the stored DDL text rather than a pretty-printed +// rendering of it. +func schemaObjects(t *testing.T, db *sql.DB) string { + t.Helper() + rows, err := db.Query(`SELECT type, name, tbl_name, COALESCE(sql, '') FROM sqlite_schema ORDER BY type, name`) + if err != nil { + t.Fatalf("reading sqlite_schema: %v", err) + } + defer func() { _ = rows.Close() }() + + var b strings.Builder + for rows.Next() { + var typ, name, tbl, ddl string + if err := rows.Scan(&typ, &name, &tbl, &ddl); err != nil { + t.Fatalf("scanning sqlite_schema: %v", err) + } + fmt.Fprintf(&b, "%s\t%s\t%s\n%s\n--\n", typ, name, tbl, ddl) + } + if err := rows.Err(); err != nil { + t.Fatalf("iterating sqlite_schema: %v", err) + } + return b.String() +} + +func ledgerRows(t *testing.T, db *sql.DB) []appliedMigration { + t.Helper() + applied, err := readLedger(context.Background(), db) + if err != nil { + t.Fatalf("readLedger: %v", err) + } + return applied +} + +func mustSchemaVersion(t *testing.T, db *sql.DB) int { + t.Helper() + v, err := SchemaVersion(context.Background(), db) + if err != nil { + t.Fatalf("SchemaVersion: %v", err) + } + return v +} + +// --------------------------------------------------------------------------- +// The embedded migration set +// --------------------------------------------------------------------------- + +func TestEmbeddedMigrationsAreWellFormed(t *testing.T) { + migrations, err := Migrations() + if err != nil { + t.Fatalf("Migrations: %v", err) + } + if len(migrations) == 0 { + t.Fatal("no migrations are embedded") + } + for i, m := range migrations { + if m.Version != i+1 { + t.Fatalf("migration %d is version %d; versions must be contiguous from 1", i, m.Version) + } + if len(m.Checksum) != 64 { + t.Errorf("%s: checksum %q is not a 64-character SHA-256 hex digest", m.Filename, m.Checksum) + } + if strings.Contains(m.SQL, includeDirectivePrefix) { + t.Errorf("%s: an include directive survived expansion", m.Filename) + } + sum := sha256.Sum256([]byte(m.SQL)) + if want := hex.EncodeToString(sum[:]); m.Checksum != want { + t.Errorf("%s: checksum %s does not digest the expanded SQL (%s)", m.Filename, m.Checksum, want) + } + } + + latest, err := LatestVersion() + if err != nil { + t.Fatalf("LatestVersion: %v", err) + } + if latest != migrations[len(migrations)-1].Version { + t.Fatalf("LatestVersion = %d, want %d", latest, migrations[len(migrations)-1].Version) + } +} + +// TestInitMigrationEmbedsSchemaByteForByte is the structural half of the +// packet's byte-for-byte requirement: 0001 does not carry a copy of the DDL +// that could drift from schema.sql, it carries schema.sql's exact bytes. +func TestInitMigrationEmbedsSchemaByteForByte(t *testing.T) { + migrations, err := Migrations() + if err != nil { + t.Fatalf("Migrations: %v", err) + } + init := migrations[0] + if init.Name != "init" || init.Filename != "0001_init.sql" { + t.Fatalf("migration 1 is %q (%s), want init (0001_init.sql)", init.Name, init.Filename) + } + if !strings.Contains(init.SQL, Schema()) { + t.Fatal("0001_init.sql does not contain schema.sql verbatim after expansion") + } + // Anything outside the included schema must be comment, so the migration + // cannot quietly add DDL that schema.sql does not describe. + extra := strings.ReplaceAll(init.SQL, Schema(), "") + for i, line := range strings.Split(extra, "\n") { + if trimmed := strings.TrimSpace(line); trimmed != "" && !strings.HasPrefix(trimmed, "--") { + t.Fatalf("0001_init.sql line %d adds DDL outside schema.sql: %q", i+1, trimmed) + } + } +} + +func TestIncludeExpansion(t *testing.T) { + got, err := expandIncludes("test.sql", "-- header\n"+includeDirectivePrefix+"schema.sql\n-- footer\n") + if err != nil { + t.Fatalf("expandIncludes: %v", err) + } + if want := "-- header\n" + Schema() + "-- footer\n"; got != want { + t.Fatalf("expansion did not splice schema.sql in place:\n%q", got[:min(len(got), 200)]) + } + + if _, err := expandIncludes("test.sql", includeDirectivePrefix+"secrets.sql\n"); err == nil { + t.Fatal("expandIncludes accepted an include target other than schema.sql") + } +} + +// --------------------------------------------------------------------------- +// Applying migrations +// --------------------------------------------------------------------------- + +// TestMigrateBuildsExactlyR4Schema is the packet's schema-equivalence evidence +// item: a database built by the migration runner and a database built by +// applying schema.sql directly must be indistinguishable, object for object. +func TestMigrateBuildsExactlyR4Schema(t *testing.T) { + migrated := openMemory(t) + applied, err := Migrate(context.Background(), migrated, "") + if err != nil { + t.Fatalf("Migrate: %v", err) + } + if len(applied) != 1 || applied[0] != 1 { + t.Fatalf("Migrate applied %v, want [1]", applied) + } + + direct := openMemory(t) + if _, err := direct.Exec(Schema()); err != nil { + t.Fatalf("applying schema.sql directly: %v", err) + } + + if got, want := schemaObjects(t, migrated), schemaObjects(t, direct); got != want { + t.Fatalf("migrated schema differs from schema.sql applied directly:\n--- migrated ---\n%s\n--- direct ---\n%s", got, want) + } + + if v := mustSchemaVersion(t, migrated); v != 1 { + t.Fatalf("PRAGMA user_version = %d after migrating, want 1", v) + } + + rows := ledgerRows(t, migrated) + if len(rows) != 1 { + t.Fatalf("schema_migration has %d rows, want 1", len(rows)) + } + migrations, err := Migrations() + if err != nil { + t.Fatalf("Migrations: %v", err) + } + if rows[0].Version != 1 || rows[0].Name != "init" || rows[0].Checksum != migrations[0].Checksum { + t.Fatalf("ledger row %+v does not describe %s", rows[0], migrations[0].Filename) + } + if rows[0].AppliedAt == "" { + t.Fatal("ledger row has no applied_at timestamp") + } +} + +func TestMigrateIsANoOpWhenCurrent(t *testing.T) { + db := openMemory(t) + ctx := context.Background() + if _, err := Migrate(ctx, db, ""); err != nil { + t.Fatalf("first Migrate: %v", err) + } + before := ledgerRows(t, db) + schemaBefore := schemaObjects(t, db) + + applied, err := Migrate(ctx, db, "") + if err != nil { + t.Fatalf("second Migrate: %v", err) + } + if len(applied) != 0 { + t.Fatalf("re-running Migrate applied %v, want nothing", applied) + } + after := ledgerRows(t, db) + if len(after) != len(before) || after[0].AppliedAt != before[0].AppliedAt { + t.Fatalf("re-running Migrate rewrote the ledger: %+v -> %+v", before, after) + } + if schemaObjects(t, db) != schemaBefore { + t.Fatal("re-running Migrate changed the schema") + } +} + +// TestMigrateAppliesInNumberedOrderInsideTransactions injects later versions, +// which is the only way to test ordering and transactionality while 0001 is +// the only real migration. The alternative is finding out during the first +// upgrade a user ever performs. +func TestMigrateAppliesInNumberedOrderInsideTransactions(t *testing.T) { + db, dir := openOnDisk(t) + ctx := context.Background() + + migrations := withSynthetic(t, + synthetic(2, "second", "CREATE TABLE second_step (x INTEGER);"), + synthetic(3, "third", "CREATE TABLE third_step (y INTEGER REFERENCES second_step(x));"), + ) + + applied, err := migrateWith(ctx, db, migrations, dir) + if err != nil { + t.Fatalf("migrateWith: %v", err) + } + if len(applied) != 3 || applied[0] != 1 || applied[1] != 2 || applied[2] != 3 { + t.Fatalf("applied %v, want [1 2 3] in order", applied) + } + if v := mustSchemaVersion(t, db); v != 3 { + t.Fatalf("user_version = %d, want 3", v) + } + if rows := ledgerRows(t, db); len(rows) != 3 { + t.Fatalf("ledger has %d rows, want 3", len(rows)) + } + // third_step references second_step; it could not have been created first. + var n int + if err := db.QueryRow(`SELECT count(*) FROM sqlite_schema WHERE name IN ('second_step','third_step')`).Scan(&n); err != nil { + t.Fatalf("counting new tables: %v", err) + } + if n != 2 { + t.Fatalf("expected both synthetic tables, found %d", n) + } +} + +// TestFailedMigrationLeavesSchemaUntouched is why each migration runs inside +// BEGIN ... COMMIT with the user_version bump in the same transaction. +func TestFailedMigrationLeavesSchemaUntouched(t *testing.T) { + db := openMemory(t) + ctx := context.Background() + + migrations := withSynthetic(t, synthetic(2, "broken", + "CREATE TABLE half_applied (x INTEGER);\nCREATE TABLE half_applied (x INTEGER);")) + + applied, err := migrateWith(ctx, db, migrations, t.TempDir()) + if err == nil { + t.Fatal("a migration with invalid DDL was accepted") + } + if len(applied) != 1 || applied[0] != 1 { + t.Fatalf("applied %v, want [1] — 0001 succeeded and 0002 must not count", applied) + } + if v := mustSchemaVersion(t, db); v != 1 { + t.Fatalf("user_version = %d after a failed migration, want 1", v) + } + if rows := ledgerRows(t, db); len(rows) != 1 { + t.Fatalf("ledger has %d rows after a failed migration, want 1", len(rows)) + } + var n int + if err := db.QueryRow(`SELECT count(*) FROM sqlite_schema WHERE name = 'half_applied'`).Scan(&n); err != nil { + t.Fatalf("looking for the rolled-back table: %v", err) + } + if n != 0 { + t.Fatal("half_applied survived a rolled-back migration") + } +} + +// --------------------------------------------------------------------------- +// The ledger refuses to guess +// --------------------------------------------------------------------------- + +// TestMigrateRefusesEditedMigration is the packet's stop condition: "a +// hand-edited migration file with a mismatched checksum causes migrate.go to +// refuse to start". The ledger row is edited rather than the embedded file +// because the embedded file cannot be edited at run time; the comparison +// migrate.go performs is the same one either way. +func TestMigrateRefusesEditedMigration(t *testing.T) { + db := openMemory(t) + ctx := context.Background() + if _, err := Migrate(ctx, db, ""); err != nil { + t.Fatalf("Migrate: %v", err) + } + + const tampered = "ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff" + if _, err := db.Exec(`UPDATE schema_migration SET checksum = ? WHERE version = 1`, tampered); err != nil { + t.Fatalf("tampering with the ledger: %v", err) + } + + _, err := Migrate(ctx, db, "") + if err == nil { + t.Fatal("Migrate started against a database whose migration checksum does not match") + } + if !errors.Is(err, ErrMigrationLedger) { + t.Fatalf("error does not wrap ErrMigrationLedger: %v", err) + } + for _, want := range []string{"0001_init.sql", tampered, "Refusing to start"} { + if !strings.Contains(err.Error(), want) { + t.Errorf("error message omits %q: %v", want, err) + } + } +} + +func TestMigrateRefusesADowngrade(t *testing.T) { + db := openMemory(t) + ctx := context.Background() + if _, err := Migrate(ctx, db, ""); err != nil { + t.Fatalf("Migrate: %v", err) + } + // Simulate an older binary opening a store a newer one migrated. + if _, err := db.Exec( + `INSERT INTO schema_migration (version, name, checksum, applied_at) VALUES (2, 'future', 'x', '2026-01-01T00:00:00Z')`, + ); err != nil { + t.Fatalf("seeding a future migration: %v", err) + } + if _, err := db.Exec(`PRAGMA user_version = 2`); err != nil { + t.Fatalf("bumping user_version: %v", err) + } + + _, err := Migrate(ctx, db, "") + if !errors.Is(err, ErrMigrationLedger) { + t.Fatalf("Migrate accepted a newer store, got %v", err) + } + if !strings.Contains(err.Error(), "forward-only") { + t.Errorf("error does not explain that downgrades are unsupported: %v", err) + } +} + +func TestMigrateRefusesUserVersionLedgerDisagreement(t *testing.T) { + db := openMemory(t) + ctx := context.Background() + if _, err := Migrate(ctx, db, ""); err != nil { + t.Fatalf("Migrate: %v", err) + } + if _, err := db.Exec(`PRAGMA user_version = 9`); err != nil { + t.Fatalf("bumping user_version: %v", err) + } + if _, err := Migrate(ctx, db, ""); !errors.Is(err, ErrMigrationLedger) { + t.Fatalf("Migrate accepted user_version 9 with one applied migration, got %v", err) + } +} + +func TestMigrateRefusesAPopulatedDatabaseWithNoLedger(t *testing.T) { + db := openMemory(t) + if _, err := db.Exec(`PRAGMA user_version = 4`); err != nil { + t.Fatalf("bumping user_version: %v", err) + } + _, err := Migrate(context.Background(), db, "") + if !errors.Is(err, ErrMigrationLedger) { + t.Fatalf("Migrate accepted a foreign database with no schema_migration table, got %v", err) + } +} + +func TestMigrateRefusesAGapInTheLedger(t *testing.T) { + db := openMemory(t) + ctx := context.Background() + migrations := withSynthetic(t, + synthetic(2, "second", "CREATE TABLE second_step (x INTEGER);"), + synthetic(3, "third", "CREATE TABLE third_step (y INTEGER);"), + ) + if _, err := migrateWith(ctx, db, migrations, t.TempDir()); err != nil { + t.Fatalf("migrateWith: %v", err) + } + if _, err := db.Exec(`DELETE FROM schema_migration WHERE version = 2`); err != nil { + t.Fatalf("punching a hole in the ledger: %v", err) + } + if _, err := migrateWith(ctx, db, migrations, t.TempDir()); !errors.Is(err, ErrMigrationLedger) { + t.Fatalf("Migrate accepted a ledger with a gap, got %v", err) + } +} + +// --------------------------------------------------------------------------- +// Forward-only means the snapshot is mandatory +// --------------------------------------------------------------------------- + +func TestMigrateRefusesToUpgradeWithoutASnapshotDirectory(t *testing.T) { + db, _ := openOnDisk(t) + ctx := context.Background() + if _, err := Migrate(ctx, db, ""); err != nil { + t.Fatalf("initial Migrate: %v", err) + } + + migrations := withSynthetic(t, synthetic(2, "second", "CREATE TABLE second_step (x INTEGER);")) + _, err := migrateWith(ctx, db, migrations, "") + if !errors.Is(err, ErrSnapshotRequired) { + t.Fatalf("Migrate upgraded a populated database with no snapshot directory, got %v", err) + } + if !strings.Contains(err.Error(), "forward-only") { + t.Errorf("error does not explain why the snapshot is mandatory: %v", err) + } + if v := mustSchemaVersion(t, db); v != 1 { + t.Fatalf("a refused upgrade changed user_version to %d", v) + } +} + +// TestSnapshotIsTakenBeforeAnUpgrade proves the replacement for down +// migrations actually exists on disk and is a readable database at the +// pre-upgrade version — an unopenable file would be a rollback path in name +// only. +func TestSnapshotIsTakenBeforeAnUpgrade(t *testing.T) { + db, dir := openOnDisk(t) + ctx := context.Background() + if _, err := Migrate(ctx, db, ""); err != nil { + t.Fatalf("initial Migrate: %v", err) + } + + snapDir := filepath.Join(dir, "snapshots") + migrations := withSynthetic(t, synthetic(2, "second", "CREATE TABLE second_step (x INTEGER);")) + if _, err := migrateWith(ctx, db, migrations, snapDir); err != nil { + t.Fatalf("migrateWith: %v", err) + } + + snap := filepath.Join(snapDir, "anvil-pre-v1.db") + if _, err := os.Stat(snap); err != nil { + t.Fatalf("no pre-migration snapshot at %s: %v", snap, err) + } + + restored, err := sql.Open("sqlite", snap) + if err != nil { + t.Fatalf("reopening the snapshot: %v", err) + } + defer func() { _ = restored.Close() }() + restored.SetMaxOpenConns(1) + + if v := mustSchemaVersion(t, restored); v != 1 { + t.Fatalf("snapshot is at user_version %d, want the pre-upgrade version 1", v) + } + var n int + if err := restored.QueryRow(`SELECT count(*) FROM sqlite_schema WHERE name = 'second_step'`).Scan(&n); err != nil { + t.Fatalf("querying the snapshot: %v", err) + } + if n != 0 { + t.Fatal("the snapshot contains the migration it was supposed to precede") + } +} + +func TestSnapshotRefusesToOverwriteAnExistingFile(t *testing.T) { + db, dir := openOnDisk(t) + ctx := context.Background() + if _, err := Migrate(ctx, db, ""); err != nil { + t.Fatalf("Migrate: %v", err) + } + dest := filepath.Join(dir, "snap.db") + if err := Snapshot(ctx, db, dest); err != nil { + t.Fatalf("Snapshot: %v", err) + } + if err := Snapshot(ctx, db, dest); err == nil { + t.Fatal("Snapshot overwrote an existing snapshot") + } + + // snapshotPath must therefore never hand back a name that is already + // taken, or a retried upgrade would fail on its own previous snapshot. + first, err := snapshotPath(dir, 1) + if err != nil { + t.Fatalf("snapshotPath: %v", err) + } + if err := os.WriteFile(first, []byte("occupied"), 0o600); err != nil { + t.Fatalf("occupying the snapshot name: %v", err) + } + second, err := snapshotPath(dir, 1) + if err != nil { + t.Fatalf("snapshotPath: %v", err) + } + if second == first { + t.Fatalf("snapshotPath returned the occupied name %q twice", first) + } +} + +func TestSnapshotRejectsEmptyArguments(t *testing.T) { + db := openMemory(t) + if err := Snapshot(context.Background(), db, " "); err == nil { + t.Fatal("Snapshot accepted an empty destination") + } + if err := Snapshot(context.Background(), nil, "x.db"); err == nil { + t.Fatal("Snapshot accepted a nil database") + } +} + +func TestMigrateRejectsNilDatabase(t *testing.T) { + if _, err := Migrate(context.Background(), nil, ""); err == nil { + t.Fatal("Migrate accepted a nil database handle") + } +} + +// --------------------------------------------------------------------------- +// Synthetic migrations +// --------------------------------------------------------------------------- + +func synthetic(version int, name, body string) Migration { + sum := sha256.Sum256([]byte(body)) + return Migration{ + Version: version, + Name: name, + Filename: fmt.Sprintf("%04d_%s.sql", version, name), + SQL: body, + Checksum: hex.EncodeToString(sum[:]), + } +} + +// withSynthetic appends test-only migrations after the real embedded set. +func withSynthetic(t *testing.T, extra ...Migration) []Migration { + t.Helper() + migrations, err := Migrations() + if err != nil { + t.Fatalf("Migrations: %v", err) + } + return append(migrations, extra...) +} diff --git a/internal/store/migrations/0001_init.sql b/internal/store/migrations/0001_init.sql new file mode 100644 index 0000000..80353ef --- /dev/null +++ b/internal/store/migrations/0001_init.sql @@ -0,0 +1,33 @@ +-- 0001_init — the initial Anvil schema. +-- +-- This migration deliberately contains no DDL of its own. Its body is R.4's +-- internal/store/schema.sql, included verbatim by the directive below and +-- expanded by migrate.go at load time from the string ddl.go already embeds. +-- +-- WHY AN INCLUDE AND NOT A COPY. plan/40-record-and-storage.md declares +-- schema.sql a frozen interface and R.4 built ddl.go explicitly as the "Go +-- wrapper exposing the DDL as an embedded string for R.5's migrations". A +-- second, byte-identical copy of 26 KB of DDL in this directory would be a +-- second definition of that interface: the two would drift the first time +-- someone edited one of them, and the drift would be invisible until a fresh +-- install and an upgraded install disagreed about the shape of a table. There +-- is exactly one copy of the schema in this repository, and this is a pointer +-- to it. +-- +-- CHECKSUM SENSITIVITY. migrate.go checksums the EXPANDED text — this file's +-- bytes with the directive line replaced by schema.sql's bytes. So editing +-- either file changes migration 0001's checksum, and a database that has +-- already applied the old text refuses to start rather than silently running +-- on a schema nobody applied. That is the whole point of the ledger. +-- +-- FORWARD-ONLY. There is no 0001_init.down.sql and there never will be, per +-- research/07-database-design.md §7: the rollback story is the +-- `VACUUM INTO 'anvil-pre-v{N}.db'` snapshot migrate.go takes before touching +-- an existing database, not reversible DDL nobody ever tests. +-- +-- NO PRAGMA HERE. migrate.go runs every migration inside BEGIN ... COMMIT and +-- `PRAGMA journal_mode = WAL` cannot run in a transaction. Connection pragmas +-- are ddl.go's ConnectionPragmas(); `PRAGMA user_version` is set by migrate.go +-- inside the same transaction as this file's DDL. + +-- @anvil:include schema.sql diff --git a/internal/store/schema.sql b/internal/store/schema.sql new file mode 100644 index 0000000..69b147d --- /dev/null +++ b/internal/store/schema.sql @@ -0,0 +1,459 @@ +-- Anvil store of record — complete DDL for schema version 1 (step R.4). +-- +-- plan/00-SPINE.md S1 collapses the originally-specified "8-hour buffer file" +-- into ONE SQLite database plus a `handoff` table plus a regenerable tmpfs +-- packet. There is no second durable buffer file and no second durable table +-- carrying finding dispositions: plan/IMPLEMENTATION-PLAN.md §6 rulings G9 and +-- G10 make THIS FILE the only definition of `handoff` anywhere in Anvil. +-- Area 70's O.3 migration and area 60's `anvil_ledger` are both deleted and +-- folded in here. +-- +-- SOURCES, in precedence order: +-- 1. plan/40-record-and-storage.md, "Store Schema" — authoritative for +-- `scan_run`, `audit_record`, `finding`, `finding_occurrence`, `handoff`. +-- 2. plan/IMPLEMENTATION-PLAN.md §6 (G9, G10) — supersedes (1) where they +-- disagree: `handoff.state` carries all thirteen dispositions and +-- `handoff.consumption_class` is present. +-- 3. research/07-database-design.md §2 — carried forward unchanged for every +-- other table. No research/07 table is dropped. +-- +-- WHAT IS DELIBERATELY NOT IN THIS FILE: +-- +-- * PRAGMA statements. `PRAGMA journal_mode = WAL` cannot run inside a +-- transaction, and R.5 applies this file inside `BEGIN ... COMMIT`. The +-- connection pragmas plan/40-record-and-storage.md specifies are exposed +-- by ddl.go as ConnectionPragmas() and are applied per connection, before +-- any other store operation. `foreign_keys` in particular is per +-- connection and OFF by default; this schema depends on it being ON. +-- * `PRAGMA user_version` and the migration ledger's contents. R.5 owns +-- both; the `schema_migration` table itself is created here. +-- +-- CHECK-CONSTRAINT POLICY. A CHECK constraint that names literals freezes a +-- vocabulary, and plan/IMPLEMENTATION-PLAN.md §6 documents what happens when +-- two areas freeze the same vocabulary differently: one area writes a literal +-- another area's NOT NULL column cannot accept. So enum CHECKs appear here +-- ONLY where internal/record/contract.go owns the enum, and ddl_test.go +-- asserts every such constraint agrees literal-for-literal with the Go values. +-- Columns whose comment names a vocabulary that contract.go does NOT own +-- (severity, ecosystem, suppression classification, ...) are left +-- unconstrained on purpose; constraining them here would be area 40 inventing +-- vocabulary for another area, which is the defect §6 exists to stop. +-- +-- Every CHECK is NAMED. research/07 Risk #15: batch-recreate tooling silently +-- drops unnamed CHECK constraints, which on a security tool's schema is an +-- integrity regression with no error message. + +-- ============ ADVISORY DOMAIN (feeds owned by area 20) ============ +CREATE TABLE advisory ( + advisory_id TEXT PRIMARY KEY, -- 'CVE-2026-1234' | 'GHSA-xxxx-yyyy-zzzz' + source TEXT NOT NULL, -- 'osv' | 'ghsa' | 'nvd' | ... + published_at TEXT NOT NULL, -- ISO-8601 UTC + modified_at TEXT NOT NULL, + severity TEXT, -- 'critical'|'high'|'medium'|'low'|'none' + cvss_vector TEXT, + cvss_score REAL, + summary TEXT NOT NULL, + details TEXT, + raw_json BLOB, -- zstd-compressed original + content_hash TEXT NOT NULL, -- sha256 of canonical form; drives delta ingest + ingested_at TEXT NOT NULL +); + +CREATE TABLE advisory_alias ( -- CVE <-> GHSA <-> OSV + advisory_id TEXT NOT NULL REFERENCES advisory(advisory_id) ON DELETE CASCADE, + alias_id TEXT NOT NULL, + PRIMARY KEY (advisory_id, alias_id) +); +CREATE INDEX idx_alias_lookup ON advisory_alias(alias_id); + +-- BM25 over advisory text. External-content table: no duplicate storage of +-- summary/details. Query: +-- ... WHERE advisory_fts MATCH ? ORDER BY bm25(advisory_fts, 4.0, 1.0, 8.0) +-- with `aliases` weighted highest so a literal 'CVE-2026-1234' beats prose. +-- +-- This is also the schema's built-in FTS5 availability probe: if the driver +-- lacks FTS5, applying this file fails loudly here rather than at first query. +-- R.5's CheckFTS5 guard still runs independently at every process start. +-- +-- KNOWN DEFECT, CARRIED FORWARD DELIBERATELY AND REPORTED, NOT SILENTLY +-- PATCHED. `content='advisory'` makes FTS5 read column values back from +-- `advisory`, but `advisory` has no `aliases` column — aliases are a +-- one-to-many in `advisory_alias`. So `INSERT INTO advisory_fts(advisory_fts) +-- VALUES('rebuild')`, any `SELECT FROM advisory_fts`, and any +-- snippet()/highlight() call fail with: +-- SQL logic error: no such column: T.aliases +-- Verified empirically on modernc.org/sqlite v1.56.0. Writes into +-- advisory_fts and rowid-only MATCH queries work; ddl_test.go exercises +-- exactly those. R.4 does not invent a fix because the two candidate repairs +-- (point `content=` at a view that projects the aliases, or drop `aliases` +-- from the FTS columns and re-derive the bm25 weights) both change an +-- interface area 20's ingestion owns. Flagged to the orchestrator. +CREATE VIRTUAL TABLE advisory_fts USING fts5( + summary, details, aliases, + content='advisory', content_rowid='rowid', + tokenize='porter unicode61' +); + +-- ============ COMPONENT DOMAIN ============ +CREATE TABLE component ( + component_id INTEGER PRIMARY KEY, + ecosystem TEXT NOT NULL, -- 'npm'|'pypi'|'cargo'|'go'|'deb'|'rpm'|'maven' + name TEXT NOT NULL, + purl_base TEXT NOT NULL, -- 'pkg:pypi/requests' (no version) + UNIQUE (ecosystem, name) +); + +CREATE TABLE advisory_affects ( + advisory_id TEXT NOT NULL REFERENCES advisory(advisory_id) ON DELETE CASCADE, + component_id INTEGER NOT NULL REFERENCES component(component_id), + introduced TEXT, -- version range endpoints, ecosystem-native + fixed TEXT, + range_kind TEXT NOT NULL, -- 'semver'|'ecosystem'|'git' + PRIMARY KEY (advisory_id, component_id, introduced, fixed) +); +CREATE INDEX idx_affects_component ON advisory_affects(component_id, advisory_id); + +-- ============ TARGET + TRIGGER (never hard-coded) ============ +CREATE TABLE target ( + target_id INTEGER PRIMARY KEY, + kind TEXT NOT NULL, -- 'repo'|'host' + locator TEXT NOT NULL UNIQUE, -- clone URL or hostname + config_json TEXT NOT NULL DEFAULT '{}', + CONSTRAINT ck_target_config_json CHECK (json_valid(config_json)) +); + +CREATE TABLE trigger_policy ( -- HARD CONSTRAINT: policy lives in data, not code + policy_id INTEGER PRIMARY KEY, + target_id INTEGER NOT NULL REFERENCES target(target_id) ON DELETE CASCADE, + kind TEXT NOT NULL, -- 'cron'|'github_action'|'webhook'|'manual' + spec TEXT NOT NULL, -- cron expr | tag glob 'v*.0.0' | event name + scan_depth TEXT NOT NULL, -- 'full'|'incremental'|'sca_only' + enabled INTEGER NOT NULL DEFAULT 1, + config_json TEXT NOT NULL DEFAULT '{}', + CONSTRAINT ck_trigger_policy_enabled_bool CHECK (enabled IN (0, 1)), + CONSTRAINT ck_trigger_policy_config_json CHECK (json_valid(config_json)) +); +CREATE INDEX idx_policy_enabled ON trigger_policy(target_id, kind) WHERE enabled = 1; + +CREATE TABLE ingest_watermark ( -- delta scraping cursors (feeds = area 20) + source TEXT PRIMARY KEY, + cursor TEXT, + etag TEXT, + last_success_at TEXT +); + +-- ============ SCAN RUN — unchanged shape from research/07 ============ +CREATE TABLE scan_run ( + scan_run_id INTEGER PRIMARY KEY, + target_id INTEGER NOT NULL REFERENCES target(target_id), + policy_id INTEGER REFERENCES trigger_policy(policy_id), + trigger_ref TEXT, -- tag / commit / cron fire id + commit_sha TEXT, + started_at TEXT NOT NULL, -- anvil/deadline.deadlineAt is computed from THIS, never last write + finished_at TEXT, + status TEXT NOT NULL, -- scan_run.status, owned by internal/record + sast_engine_ver TEXT, + dast_engine_ver TEXT, + ruleset_version TEXT NOT NULL, + advisory_snapshot TEXT, -- max(advisory.modified_at) at scan time + rollup_hash TEXT, -- sha256 over sorted finding fingerprints + CONSTRAINT ck_scan_run_status CHECK (status IN ('running', 'ok', 'failed', 'partial')) +); +CREATE INDEX idx_scan_target_time ON scan_run(target_id, started_at DESC); +CREATE INDEX idx_scan_running ON scan_run(target_id) WHERE status = 'running'; + +-- ============ AUDIT RECORD — the collapsed store (S1: no second durable buffer file) ============ +CREATE TABLE audit_record ( + audit_record_id INTEGER PRIMARY KEY, + scan_run_id INTEGER NOT NULL UNIQUE REFERENCES scan_run(scan_run_id), + schema_version TEXT NOT NULL, -- anvil/schemaVersion + audit_version INTEGER NOT NULL DEFAULT 1, -- anvil/version; a bump triggers R.11's queue re-cut + state TEXT NOT NULL, -- anvil/state + sast_status TEXT, -- per-half status (anvil/status) + sast_sealed_at TEXT, + dast_status TEXT NOT NULL DEFAULT 'not_run', -- S6: never NULL, never silently 'completed_clean' + dast_sealed_at TEXT, + dast_coverage_json TEXT, -- S6: {probedCount, inventoryUnionCount, inventoryProvenanceMix} + target_provenance TEXT NOT NULL, -- S6: a target that failed to boot must be distinguishable from scanned clean + deadline_at TEXT NOT NULL, -- = scan_run.started_at + claim_timeout_seconds. NEVER recomputed. + claim_timeout_seconds INTEGER NOT NULL DEFAULT 28800, -- 8h default, configurable. A CLAIM timeout, not a deletion policy. + dast_deadline_seconds INTEGER, -- S6: configurable, independent clock from claim_timeout_seconds + payload BLOB, -- zstd(canonical SARIF JSON); NULLed by the reaper at deadline_at + payload_sha256 TEXT NOT NULL, -- survives payload deletion: proof of what was handed over + created_at TEXT NOT NULL, + consumed_at TEXT, + purged_at TEXT, + CONSTRAINT ck_audit_record_state CHECK ( + state IN ('collecting', 'sast_sealed', 'dast_sealed', 'both_sealed', 'consumed', 'expired')), + CONSTRAINT ck_audit_record_sast_status CHECK ( + sast_status IS NULL OR sast_status IN ('running', 'sealed', 'failed', 'timed_out', 'skipped')), + CONSTRAINT ck_audit_record_dast_status CHECK ( + dast_status IN ('not_run', 'skipped_no_manifest', 'running', 'completed_clean', + 'completed_findings', 'completed_partial', 'target_boot_failed', + 'target_unreachable', 'timed_out')), + CONSTRAINT ck_audit_record_target_provenance CHECK ( + target_provenance IN ('booted_clean', 'boot_failed', 'build_failed', + 'no_target_declared', 'unreachable_at_scan_time')), + CONSTRAINT ck_audit_record_payload_sha256_hex CHECK ( + length(payload_sha256) = 64 AND payload_sha256 NOT GLOB '*[^0-9a-f]*'), + CONSTRAINT ck_audit_record_claim_timeout_positive CHECK (claim_timeout_seconds > 0), + CONSTRAINT ck_audit_record_dast_deadline_positive CHECK ( + dast_deadline_seconds IS NULL OR dast_deadline_seconds > 0), + CONSTRAINT ck_audit_record_audit_version_positive CHECK (audit_version >= 1), + CONSTRAINT ck_audit_record_dast_coverage_json CHECK ( + dast_coverage_json IS NULL OR json_valid(dast_coverage_json)) +); +CREATE INDEX idx_audit_deadline ON audit_record(deadline_at) WHERE purged_at IS NULL; +CREATE INDEX idx_audit_state ON audit_record(state); + +-- ============ CODE LOCATION ============ +CREATE TABLE code_location ( + location_id INTEGER PRIMARY KEY, + repo_relpath TEXT NOT NULL, -- POSIX separators, repo-root-relative, never absolute + start_line INTEGER, + end_line INTEGER, + start_col INTEGER, + end_col INTEGER, + symbol TEXT, -- 'pkg/mod.py::ClassA.method_b' + symbol_kind TEXT, -- 'function'|'method'|'class'|'module' + blob_sha TEXT, -- git blob hash of the file at scan time + snippet_hash TEXT -- sha256 of NORMALISED snippet; never the raw snippet +); +CREATE INDEX idx_loc_path ON code_location(repo_relpath, start_line); + +-- ============ FINDING: the stable identity ============ +-- Extended from research/07 with the S6/S24 fields (evidence_class, verdict, +-- remediable_by_agent). `finding_id` is the permanent key; `fingerprint` is a +-- lookup key that may be re-derived, because no vendor guarantees fingerprint +-- stability (research/07 Risk #1). +CREATE TABLE finding ( + finding_id INTEGER PRIMARY KEY, + target_id INTEGER NOT NULL REFERENCES target(target_id), + fingerprint TEXT NOT NULL, -- anvil-fp/v1, full 64-hex SHA-256, never truncated + fingerprint_alg TEXT NOT NULL DEFAULT 'anvil-fp/v1', + detector TEXT NOT NULL, -- finding.detector + evidence_class TEXT NOT NULL, -- anvil/evidenceClass + rule_id TEXT NOT NULL, -- versioned: 'anvil.py.sqli/v3' + verdict TEXT NOT NULL DEFAULT 'true_positive', -- S6: anvil/verdict + remediable_by_agent INTEGER NOT NULL, -- S6: 0/1; host findings are ALWAYS 0 (S7 read-only host agent) + advisory_id TEXT REFERENCES advisory(advisory_id), + component_id INTEGER REFERENCES component(component_id), + severity TEXT NOT NULL, + title TEXT NOT NULL, + state TEXT NOT NULL, -- finding.state + first_seen_scan INTEGER NOT NULL REFERENCES scan_run(scan_run_id), + first_seen_at TEXT NOT NULL, + last_seen_scan INTEGER REFERENCES scan_run(scan_run_id), + last_seen_at TEXT, + resolved_at TEXT, + resolved_by_fix INTEGER, -- -> fix_attempt(fix_attempt_id); intentionally not an FK, see note below + UNIQUE (target_id, fingerprint), -- THE regression-check index + CONSTRAINT ck_finding_detector CHECK (detector IN ('sast', 'dast', 'sca', 'host')), + CONSTRAINT ck_finding_evidence_class CHECK ( + evidence_class IN ('dast_confirmed', 'sast_reachable', 'sast_static_only', 'sca', 'host')), + CONSTRAINT ck_finding_verdict CHECK ( + verdict IN ('true_positive', 'false_positive', 'insufficient_context')), + CONSTRAINT ck_finding_state CHECK (state IN ('open', 'resolved', 'suppressed', 'regressed')), + CONSTRAINT ck_finding_remediable_bool CHECK (remediable_by_agent IN (0, 1)), + CONSTRAINT ck_finding_host_not_remediable CHECK (detector != 'host' OR remediable_by_agent = 0), + CONSTRAINT ck_finding_fingerprint_hex CHECK ( + length(fingerprint) = 64 AND fingerprint NOT GLOB '*[^0-9a-f]*') +); +CREATE INDEX idx_finding_state ON finding(target_id, state, severity); +CREATE INDEX idx_finding_evclass ON finding(target_id, evidence_class); +CREATE INDEX idx_finding_verdict ON finding(verdict) WHERE verdict != 'true_positive'; +CREATE INDEX idx_finding_adv ON finding(advisory_id) WHERE advisory_id IS NOT NULL; +CREATE INDEX idx_finding_comp ON finding(component_id) WHERE component_id IS NOT NULL; +-- NOTE on `resolved_by_fix`: research/07 §2 and plan/40's Store Schema both +-- declare it a bare INTEGER, not a REFERENCES clause, because `fix_attempt` +-- also references `finding` and a mutual FK pair cannot be satisfied by either +-- insert order without deferred constraints. Carried forward as specified. + +-- Multi-fingerprint side table -> fuzzy fallback + algorithm migration +-- without data loss. +CREATE TABLE finding_fingerprint ( + finding_id INTEGER NOT NULL REFERENCES finding(finding_id) ON DELETE CASCADE, + kind TEXT NOT NULL, -- 'primary'|'line_hash'|'symbol_hash'|'purl_advisory'|'dast_route' + alg TEXT NOT NULL, -- 'anvil-fp/v1' | 'lineHash/v1' | ... + value TEXT NOT NULL, + PRIMARY KEY (finding_id, kind, alg) +); +CREATE INDEX idx_fp_lookup ON finding_fingerprint(kind, value); + +-- ============ OCCURRENCE: one row per (finding, scan) ============ +-- Extended with the S6 advisory-staleness fields. +CREATE TABLE finding_occurrence ( + occurrence_id INTEGER PRIMARY KEY, + finding_id INTEGER NOT NULL REFERENCES finding(finding_id) ON DELETE CASCADE, + scan_run_id INTEGER NOT NULL REFERENCES scan_run(scan_run_id) ON DELETE CASCADE, + location_id INTEGER REFERENCES code_location(location_id), + confidence REAL, + message TEXT, -- hashes/pointers only; oversized inserts rejected by trigger + evidence_ref TEXT, -- pointer INTO the sealed payload; never the raw request/response + advisory_as_of TEXT, -- S6: as_of + advisory_staleness_seconds INTEGER, -- S6: staleness_seconds + advisory_parse_degraded INTEGER NOT NULL DEFAULT 0, -- S6: parse_degraded + UNIQUE (finding_id, scan_run_id), + CONSTRAINT ck_occurrence_parse_degraded_bool CHECK (advisory_parse_degraded IN (0, 1)) +); +CREATE INDEX idx_occ_scan ON finding_occurrence(scan_run_id, finding_id); -- drives the delta query + +-- research/07 Risk #13, verbatim: "The 8-hour rule is defeatable by careless +-- denormalisation. If raw snippets or DAST request/response bodies get copied +-- into finding_occurrence.message, they outlive the purge forever." These two +-- triggers are that risk's named mitigation. The cap is deliberately far below +-- internal/record's smallest inline body cap (8 KiB) so that no request or +-- response body can be smuggled into a durable column even at its minimum +-- size, while remaining ample for a rule title plus a pointer. +CREATE TRIGGER trg_occurrence_durable_text_cap_insert +BEFORE INSERT ON finding_occurrence +FOR EACH ROW WHEN + length(CAST(COALESCE(NEW.message, '') AS BLOB)) > 2048 + OR length(CAST(COALESCE(NEW.evidence_ref, '') AS BLOB)) > 2048 +BEGIN + SELECT RAISE(ABORT, + 'finding_occurrence: message/evidence_ref exceed the durable-text cap; durable tables store hashes and pointers only (research/07 Risk #13)'); +END; + +CREATE TRIGGER trg_occurrence_durable_text_cap_update +BEFORE UPDATE ON finding_occurrence +FOR EACH ROW WHEN + length(CAST(COALESCE(NEW.message, '') AS BLOB)) > 2048 + OR length(CAST(COALESCE(NEW.evidence_ref, '') AS BLOB)) > 2048 +BEGIN + SELECT RAISE(ABORT, + 'finding_occurrence: message/evidence_ref exceed the durable-text cap; durable tables store hashes and pointers only (research/07 Risk #13)'); +END; + +-- ============ APPEND-ONLY STATE HISTORY ============ +CREATE TABLE finding_state_event ( + event_id INTEGER PRIMARY KEY, + finding_id INTEGER NOT NULL REFERENCES finding(finding_id) ON DELETE CASCADE, + scan_run_id INTEGER REFERENCES scan_run(scan_run_id), + from_state TEXT, + to_state TEXT NOT NULL, + cause TEXT NOT NULL, -- 'first_seen'|'absent_in_scan'|'fix_verified'|'regression'|'suppressed'|'expired' + at TEXT NOT NULL +); +CREATE INDEX idx_state_hist ON finding_state_event(finding_id, at DESC); + +-- ============ FIX + VERIFICATION ============ +CREATE TABLE fix_attempt ( + fix_attempt_id INTEGER PRIMARY KEY, + finding_id INTEGER NOT NULL REFERENCES finding(finding_id), + audit_record_id INTEGER REFERENCES audit_record(audit_record_id), + agent_model_id TEXT NOT NULL, + started_at TEXT NOT NULL, + finished_at TEXT, + status TEXT NOT NULL, -- 'proposed'|'applied'|'rejected'|'failed' + patch_ref TEXT, -- git ref / blob sha, not the diff text + branch_name TEXT, + pr_url TEXT +); +CREATE INDEX idx_fix_finding ON fix_attempt(finding_id, started_at DESC); + +CREATE TABLE verification ( + verification_id INTEGER PRIMARY KEY, + fix_attempt_id INTEGER NOT NULL REFERENCES fix_attempt(fix_attempt_id) ON DELETE CASCADE, + kind TEXT NOT NULL, -- 'rescan_sast'|'rescan_dast'|'build'|'unit_tests' + scan_run_id INTEGER REFERENCES scan_run(scan_run_id), + result TEXT NOT NULL, -- 'pass'|'fail'|'inconclusive' + details_json TEXT, + verified_at TEXT NOT NULL, + CONSTRAINT ck_verification_details_json CHECK (details_json IS NULL OR json_valid(details_json)) +); +CREATE INDEX idx_verif_fix ON verification(fix_attempt_id); + +-- ============ SUPPRESSION / FALSE POSITIVE ============ +CREATE TABLE suppression ( + suppression_id INTEGER PRIMARY KEY, + target_id INTEGER NOT NULL REFERENCES target(target_id) ON DELETE CASCADE, + match_kind TEXT NOT NULL, -- 'fingerprint'|'rule'|'path_glob'|'advisory'|'component' + match_value TEXT NOT NULL, + classification TEXT NOT NULL, -- 'false_positive'|'accepted_risk'|'not_exploitable'|'wont_fix' + justification TEXT NOT NULL, + created_by TEXT NOT NULL, + created_at TEXT NOT NULL, + expires_at TEXT, -- expiring suppressions prevent permanent blindness + active INTEGER NOT NULL DEFAULT 1, + CONSTRAINT ck_suppression_active_bool CHECK (active IN (0, 1)) +); +CREATE INDEX idx_supp_match ON suppression(target_id, match_kind, match_value) WHERE active = 1; + +-- ============ INCREMENTAL-SCAN CACHE (this is where compute is actually saved) ============ +CREATE TABLE file_state ( + target_id INTEGER NOT NULL REFERENCES target(target_id) ON DELETE CASCADE, + repo_relpath TEXT NOT NULL, + blob_sha TEXT NOT NULL, + ruleset_version TEXT NOT NULL, + last_scan_id INTEGER NOT NULL REFERENCES scan_run(scan_run_id), + PRIMARY KEY (target_id, repo_relpath) +); + +-- ============ HANDOFF — the collapsed buffer replacement (S1: state/lease/attempts/expiry) ============ +-- +-- ONE TABLE, ONE STATE COLUMN. plan/IMPLEMENTATION-PLAN.md §6 G10 traced the +-- concrete bug that a second table produces: area X wrote `skipped_budget` to +-- its own `anvil_ledger` while area 40's ready-set index still saw the finding +-- as 'ready', so the finding was re-leased forever. `anvil_ledger` is deleted; +-- its four extra dispositions (fixed_incidentally, split_required, withdrawn, +-- superseded) are values of `state` here. +-- +-- `consumption_class` arrives from area 70's O.3 (§6 G9). Nothing else in this +-- schema can express the gate research/21 §5 requires: `static_only` findings +-- are claimable once the SAST half is sealed, `requires_dynamic_confirmation` +-- findings must wait on the DAST half. It is NOT NULL with NO DEFAULT on +-- purpose — a default would silently grant every row the permissive value, and +-- plan/00-SPINE.md S7 says only a DAST reproduction earns "verified fixed". +-- +-- TWO INDEPENDENT CLOCKS, NEVER CONFLATED (S1): +-- handoff.lease_expires_at 15-30 min, heartbeat-renewed, governs ONE +-- coding-agent attempt. +-- audit_record.claim_timeout_seconds default 8h, governs how long an +-- UNCLAIMED finding stays eligible. At its +-- expiry the reaper drops only the tmpfs +-- packet and NULLs audit_record.payload; +-- this row, `finding`, and +-- `finding_state_event` are never deleted. +CREATE TABLE handoff ( + handoff_id INTEGER PRIMARY KEY, + finding_id INTEGER NOT NULL REFERENCES finding(finding_id) ON DELETE CASCADE, + audit_record_id INTEGER NOT NULL REFERENCES audit_record(audit_record_id), + fingerprint TEXT NOT NULL, -- denormalised for the reaper's WHERE clause + group_id TEXT, -- fix-group id, assigned by the coding-agent consumption pipeline + state TEXT NOT NULL DEFAULT 'ready', + consumption_class TEXT NOT NULL, -- from O.3 per §6 G9; no default, see note above + claimed_by TEXT, -- worker_id (O.3's lease_owner) + lease_expires_at TEXT, -- claim lease. NOT audit_record.claim_timeout_seconds. + attempts INTEGER NOT NULL DEFAULT 0, + max_attempts INTEGER NOT NULL DEFAULT 2, + idempotency_key TEXT UNIQUE, -- sha256(audit_id || finding_fingerprint || base_commit_sha); mirrors the git trailer + created_at TEXT NOT NULL, + updated_at TEXT NOT NULL, + UNIQUE (finding_id, audit_record_id), + CONSTRAINT ck_handoff_state CHECK ( + state IN ('ready', 'leased', 'validated', 'failed_validation', 'failed_format', + 'skipped_budget', 'false_positive', 'regression_introduced', + 'fixed_incidentally', 'split_required', 'withdrawn', 'superseded', + 'expired')), + CONSTRAINT ck_handoff_consumption_class CHECK ( + consumption_class IN ('static_only', 'requires_dynamic_confirmation')), + CONSTRAINT ck_handoff_fingerprint_hex CHECK ( + length(fingerprint) = 64 AND fingerprint NOT GLOB '*[^0-9a-f]*'), + CONSTRAINT ck_handoff_attempts_nonneg CHECK (attempts >= 0 AND max_attempts >= 0), + CONSTRAINT ck_handoff_lease_requires_holder CHECK ( + state != 'leased' OR (claimed_by IS NOT NULL AND lease_expires_at IS NOT NULL)) +); +CREATE INDEX idx_handoff_ready ON handoff(state) WHERE state = 'ready'; +CREATE INDEX idx_handoff_lease ON handoff(lease_expires_at) WHERE state = 'leased'; +CREATE INDEX idx_handoff_fp ON handoff(fingerprint); + +-- ============ MIGRATIONS ============ +-- Rows are written by R.5's migrate.go, in the same transaction that applies +-- the migration and bumps PRAGMA user_version. +CREATE TABLE schema_migration ( + version INTEGER PRIMARY KEY, + name TEXT NOT NULL, + checksum TEXT NOT NULL, + applied_at TEXT NOT NULL +); diff --git a/testdata/fingerprint_corpus/dast-01-sqli-error-based-body.json b/testdata/fingerprint_corpus/dast-01-sqli-error-based-body.json new file mode 100644 index 0000000..25258bc --- /dev/null +++ b/testdata/fingerprint_corpus/dast-01-sqli-error-based-body.json @@ -0,0 +1,148 @@ +{ + "id": "dast-01-sqli-error-based-body", + "tier": "dast", + "description": "A dynamically-confirmed SQL injection: payload injected into a JSON body parameter, observed as a database error string in the response. The baseline DAST fixture.", + "input": { + "target_id": "t-0003", + "rule_id_versioned": "nuclei:sqli-error-based@2026.07.1", + "http_method": "POST", + "route_template": "/api/v1/users/{id}/orders", + "injection_point": "body", + "param_name": "sortBy", + "evidence_signal": "dbErrorString" + }, + "not_hashed": { + "scheme": "https", + "host": "staging-7f3a.ephemeral.internal", + "port": 8443, + "concrete_url": "https://staging-7f3a.ephemeral.internal:8443/api/v1/users/4192/orders", + "payload": "' OR 1=1--", + "response_time_ms": 1204, + "observed_at": "2026-08-08T12:04:11Z", + "comment": "plan/40-record-and-storage.md: 'NEVER hashed: host, port, scheme, the concrete payload string, any timestamp.' A redeployed container, a rotated ephemeral port, or a staging-to-prod move must not fork identity, and a fuzzer varying its payload must not mint N findings for one bug." + }, + "hashed_fields": [ + "t-0003", + "dast", + "nuclei:sqli-error-based@2026.07.1", + "POST", + "/api/v1/users//orders", + "body", + "sortBy", + "dbErrorString" + ], + "expected_digest": "199c3b5fa4615b4caa74c05394d08e78cc84d2d2c40ffd3178f268c02bf7db90", + "mutations": [ + { + "name": "http_method_lowercased", + "description": "A producer emitting 'post' must not fork identity from one emitting 'POST'; the method is uppercased before hashing.", + "input": { + "target_id": "t-0003", + "rule_id_versioned": "nuclei:sqli-error-based@2026.07.1", + "http_method": "post", + "route_template": "/api/v1/users/{id}/orders", + "injection_point": "body", + "param_name": "sortBy", + "evidence_signal": "dbErrorString" + } + }, + { + "name": "http_method_padded", + "description": "Whitespace from a parsed request line is trimmed.", + "input": { + "target_id": "t-0003", + "rule_id_versioned": "nuclei:sqli-error-based@2026.07.1", + "http_method": " POST ", + "route_template": "/api/v1/users/{id}/orders", + "injection_point": "body", + "param_name": "sortBy", + "evidence_signal": "dbErrorString" + } + }, + { + "name": "route_template_carrying_the_payload_in_a_query_string", + "description": "The single most important DAST canonicalisation. A producer that smuggles the concrete payload into the 'template' would otherwise mint one finding per fuzzer iteration; everything from the first '?' is dropped.", + "input": { + "target_id": "t-0003", + "rule_id_versioned": "nuclei:sqli-error-based@2026.07.1", + "http_method": "POST", + "route_template": "/api/v1/users/{id}/orders?debug=1&sortBy=%27%20OR%201%3D1--", + "injection_point": "body", + "param_name": "sortBy", + "evidence_signal": "dbErrorString" + } + }, + { + "name": "route_template_carrying_a_fragment", + "description": "Everything from the first '#' is dropped for the same reason.", + "input": { + "target_id": "t-0003", + "rule_id_versioned": "nuclei:sqli-error-based@2026.07.1", + "http_method": "POST", + "route_template": "/api/v1/users/{id}/orders#results", + "injection_point": "body", + "param_name": "sortBy", + "evidence_signal": "dbErrorString" + } + }, + { + "name": "route_template_with_a_trailing_slash", + "description": "A trailing slash is removed (except on the root path).", + "input": { + "target_id": "t-0003", + "rule_id_versioned": "nuclei:sqli-error-based@2026.07.1", + "http_method": "POST", + "route_template": "/api/v1/users/{id}/orders/", + "injection_point": "body", + "param_name": "sortBy", + "evidence_signal": "dbErrorString" + } + }, + { + "name": "route_template_with_duplicate_slashes_and_no_leading_slash", + "description": "Slash runs collapse and a leading slash is added, so a crawler joining path segments naively agrees with one that does not.", + "input": { + "target_id": "t-0003", + "rule_id_versioned": "nuclei:sqli-error-based@2026.07.1", + "http_method": "POST", + "route_template": "api//v1/users/{id}//orders", + "injection_point": "body", + "param_name": "sortBy", + "evidence_signal": "dbErrorString" + } + }, + { + "name": "route_template_carrying_the_concrete_numeric_id_this_scan_requested", + "description": "The route this fixture's not_hashed.concrete_url actually carried. CanonicalRouteTemplate replaces the all-digit segment '4192' with , so the producer does not have to template the route itself. Before the R.3 blocker-2 ruling this input produced a THIRD digest for the same defect.", + "input": { + "target_id": "t-0003", + "rule_id_versioned": "nuclei:sqli-error-based@2026.07.1", + "http_method": "POST", + "route_template": "/api/v1/users/4192/orders", + "injection_point": "body", + "param_name": "sortBy", + "evidence_signal": "dbErrorString" + } + }, + { + "name": "route_template_in_express_placeholder_syntax", + "description": "A producer reading the repository's own Express route table emits ':id'; a producer reading its OpenAPI document emits '{id}'. Both normalise to , so the two producers agree.", + "input": { + "target_id": "t-0003", + "rule_id_versioned": "nuclei:sqli-error-based@2026.07.1", + "http_method": "POST", + "route_template": "/api/v1/users/:id/orders", + "injection_point": "body", + "param_name": "sortBy", + "evidence_signal": "dbErrorString" + } + } + ], + "notes": [ + "Field order is target_id, \"dast\", rule_id_versioned, http_method, route_template, injection_point, param_name, evidence_class_detail.", + "DIGEST CHANGED 2026-08-08 under the R.3 blocker-2 ruling, from ca801b8d64fdabf43aad112e3b62c53211cf369a66683c12352081357eb9d125. The hashed route_template moved from '/api/v1/users/{id}/orders' to '/api/v1/users//orders' because CanonicalRouteTemplate now DERIVES the template, as the specification always said it should ('route_template := numeric/UUID/hash path segments replaced with a placeholder token'). Nothing was stored under the old digest - anvil-fp/v1 has not shipped - so this is a correction, not a v2 event. See internal/record/FINGERPRINT-SPEC.md section 6.", + "injection_point (WHERE the payload went in) and evidence_class_detail (HOW the defect was observed) are independent hashed fields. research/07 had only the former and research/18 only the latter; the resolution hashes both, because an SQL injection proved by a database error string and one proved by a timing side channel on the same parameter are different findings with different remediation evidence.", + "param_name is a parameter NAME, never a parameter VALUE.", + "Both literals are lowercase snake_case-free tokens frozen in R.1's contract.go (InjectionPoint, EvidenceSignal); an unfrozen or SCREAMING_CASE literal is rejected rather than hashed." + ] +} diff --git a/testdata/fingerprint_corpus/dast-02-xss-reflected-query.json b/testdata/fingerprint_corpus/dast-02-xss-reflected-query.json new file mode 100644 index 0000000..79c2fd8 --- /dev/null +++ b/testdata/fingerprint_corpus/dast-02-xss-reflected-query.json @@ -0,0 +1,78 @@ +{ + "id": "dast-02-xss-reflected-query", + "tier": "dast", + "description": "A reflected cross-site scripting finding: payload injected into a query parameter, observed by seeing the payload reflected in the response. Differs from dast-01 in every discriminating field, including both the injection point and the evidence signal.", + "input": { + "target_id": "t-0003", + "rule_id_versioned": "nuclei:xss-reflected@2026.07.1", + "http_method": "GET", + "route_template": "/search", + "injection_point": "query", + "param_name": "q", + "evidence_signal": "reflectedPayload" + }, + "not_hashed": { + "scheme": "http", + "host": "127.0.0.1", + "port": 41337, + "payload": "", + "response_status": 200, + "observed_at": "2026-08-08T12:06:02Z" + }, + "hashed_fields": [ + "t-0003", + "dast", + "nuclei:xss-reflected@2026.07.1", + "GET", + "/search", + "query", + "q", + "reflectedPayload" + ], + "expected_digest": "bbe1a200328dfd7415d4a209384515a7ae01cb0494a896e69ba2bf565299c489", + "mutations": [ + { + "name": "http_method_lowercased", + "description": "Method case does not fork identity.", + "input": { + "target_id": "t-0003", + "rule_id_versioned": "nuclei:xss-reflected@2026.07.1", + "http_method": "get", + "route_template": "/search", + "injection_point": "query", + "param_name": "q", + "evidence_signal": "reflectedPayload" + } + }, + { + "name": "route_template_carrying_the_injected_query_parameter", + "description": "The query string is dropped: injection_point and param_name already record which query parameter was targeted, so keeping it would hash the payload value as well.", + "input": { + "target_id": "t-0003", + "rule_id_versioned": "nuclei:xss-reflected@2026.07.1", + "http_method": "GET", + "route_template": "/search?q=%3Cscript%3Ealert(1)%3C%2Fscript%3E", + "injection_point": "query", + "param_name": "q", + "evidence_signal": "reflectedPayload" + } + }, + { + "name": "route_template_with_a_trailing_slash", + "description": "'/search/' and '/search' are one route.", + "input": { + "target_id": "t-0003", + "rule_id_versioned": "nuclei:xss-reflected@2026.07.1", + "http_method": "GET", + "route_template": "/search/", + "injection_point": "query", + "param_name": "q", + "evidence_signal": "reflectedPayload" + } + } + ], + "notes": [ + "Paired with dast-01 to demonstrate that two DAST findings on one target stay distinct. Sharing a target_id with dast-01 and dast-03 is deliberate: the target is not what separates them.", + "Route templates are case-sensitive and are NOT folded: URL paths are case-sensitive, and '/Search' is a different route from '/search' on most servers." + ] +} diff --git a/testdata/fingerprint_corpus/dast-03-path-traversal-no-param-name.json b/testdata/fingerprint_corpus/dast-03-path-traversal-no-param-name.json new file mode 100644 index 0000000..1ba82e1 --- /dev/null +++ b/testdata/fingerprint_corpus/dast-03-path-traversal-no-param-name.json @@ -0,0 +1,105 @@ +{ + "id": "dast-03-path-traversal-no-param-name", + "tier": "dast", + "description": "A path-traversal finding injected into a templated path segment. There is no named parameter, so this fixture pins the EMPTY param_name case, and it observes the defect by a stack trace in the response body rather than by reflection.", + "input": { + "target_id": "t-0003", + "rule_id_versioned": "nuclei:path-traversal@2026.07.1", + "http_method": "GET", + "route_template": "/files/{path}", + "injection_point": "path", + "param_name": "", + "evidence_signal": "responseStackTrace" + }, + "not_hashed": { + "scheme": "http", + "host": "127.0.0.1", + "port": 41337, + "payload": "../../../../etc/passwd", + "response_body_sha256": "referenced, never inlined - plan/00-SPINE.md S7 names the DAST response body the highest-risk field", + "observed_at": "2026-08-08T12:07:44Z" + }, + "hashed_fields": [ + "t-0003", + "dast", + "nuclei:path-traversal@2026.07.1", + "GET", + "/files/", + "path", + "", + "responseStackTrace" + ], + "expected_digest": "5fc15c556531a9a7b6d6fe4a56c36a5383bbc2a0555c4e6ba60e686c0dd21550", + "mutations": [ + { + "name": "http_method_lowercased", + "description": "Method case does not fork identity.", + "input": { + "target_id": "t-0003", + "rule_id_versioned": "nuclei:path-traversal@2026.07.1", + "http_method": "get", + "route_template": "/files/{path}", + "injection_point": "path", + "param_name": "", + "evidence_signal": "responseStackTrace" + } + }, + { + "name": "route_template_with_duplicate_leading_slashes", + "description": "Slash runs collapse to one.", + "input": { + "target_id": "t-0003", + "rule_id_versioned": "nuclei:path-traversal@2026.07.1", + "http_method": "GET", + "route_template": "//files/{path}", + "injection_point": "path", + "param_name": "", + "evidence_signal": "responseStackTrace" + } + }, + { + "name": "route_template_carrying_the_traversal_payload_as_a_query_string", + "description": "The payload smuggled into the template is dropped with the rest of the query string.", + "input": { + "target_id": "t-0003", + "rule_id_versioned": "nuclei:path-traversal@2026.07.1", + "http_method": "GET", + "route_template": "/files/{path}?f=..%2F..%2F..%2Fetc%2Fpasswd", + "injection_point": "path", + "param_name": "", + "evidence_signal": "responseStackTrace" + } + }, + { + "name": "route_template_in_flask_placeholder_syntax", + "description": "Flask and Werkzeug spell a variable segment ''. It normalises to the same as '{path}' and ':path'.", + "input": { + "target_id": "t-0003", + "rule_id_versioned": "nuclei:path-traversal@2026.07.1", + "http_method": "GET", + "route_template": "/files/", + "injection_point": "path", + "param_name": "", + "evidence_signal": "responseStackTrace" + } + }, + { + "name": "route_template_already_carrying_the_canonical_placeholder_token", + "description": "Idempotence: feeding a canonicalised route back through CanonicalRouteTemplate must be a fixed point, or a record round-tripped through the store would fork its own identity.", + "input": { + "target_id": "t-0003", + "rule_id_versioned": "nuclei:path-traversal@2026.07.1", + "http_method": "GET", + "route_template": "/files/", + "injection_point": "path", + "param_name": "", + "evidence_signal": "responseStackTrace" + } + } + ], + "notes": [ + "An empty param_name is legal and is hashed as an empty field, keeping the DAST field count at exactly eight. A whole-body, raw-request, or path-segment injection has no single named parameter, and rejecting those would make them unfingerprintable.", + "DIGEST CHANGED 2026-08-08 under the R.3 blocker-2 ruling, from 84fe311debbc87f852e1f4e17a3242d071169448c5852a00b1b32316fb30b036. The hashed route_template moved from '/files/{path}' to '/files/' because CanonicalRouteTemplate now normalises an already-templated segment onto the one frozen placeholder token rather than trusting the producer's choice of syntax. Nothing was stored under the old digest - anvil-fp/v1 has not shipped. See internal/record/FINGERPRINT-SPEC.md section 6.", + "The input's '{path}' was the ONLY normalization the old corpus exercised on this field, and it exercised it by arriving pre-templated - which is exactly why CRITIQUE-01 could not detect that CanonicalRouteTemplate did no templating at all. dast-04 is the fixture that closes that hole, with a concrete numeric id and a concrete UUID." + ] +} diff --git a/testdata/fingerprint_corpus/dast-04-idor-concrete-numeric-and-uuid-segments.json b/testdata/fingerprint_corpus/dast-04-idor-concrete-numeric-and-uuid-segments.json new file mode 100644 index 0000000..96208a8 --- /dev/null +++ b/testdata/fingerprint_corpus/dast-04-idor-concrete-numeric-and-uuid-segments.json @@ -0,0 +1,200 @@ +{ + "id": "dast-04-idor-concrete-numeric-and-uuid-segments", + "tier": "dast", + "description": "An insecure-direct-object-reference finding whose observed route carries BOTH a concrete numeric id and a concrete UUID. This is the fixture CRITIQUE-01 said the corpus was missing: dast-01 through dast-03 all arrived pre-templated, so no fixture could tell whether CanonicalRouteTemplate templated anything at all. Every mutation below is a different concrete instance of the SAME defect and must produce the SAME digest.", + "input": { + "target_id": "t-0003", + "rule_id_versioned": "nuclei:idor-object-reference@2026.07.1", + "http_method": "GET", + "route_template": "/api/v1/users/12345/orders/3f2504e0-4f89-11d3-9a0c-0305e82c3301", + "injection_point": "path", + "param_name": "", + "evidence_signal": "statusCodeFlip" + }, + "not_hashed": { + "scheme": "https", + "host": "staging-7f3a.ephemeral.internal", + "port": 8443, + "concrete_url": "https://staging-7f3a.ephemeral.internal:8443/api/v1/users/12345/orders/3f2504e0-4f89-11d3-9a0c-0305e82c3301", + "authenticated_as": "user 4192", + "observed_status_flip": "403 -> 200", + "observed_at": "2026-08-08T16:41:09Z", + "comment": "The numeric id and the UUID are volatile in the strongest sense: the next scan against a freshly seeded ephemeral target will observe different ones for the same defect. If they were hashed, every scan would open a new finding and 'verified fixed' could never be proved - which is precisely why plan/00-SPINE.md S7's fix-verification gate depends on this tier." + }, + "hashed_fields": [ + "t-0003", + "dast", + "nuclei:idor-object-reference@2026.07.1", + "GET", + "/api/v1/users//orders/", + "path", + "", + "statusCodeFlip" + ], + "expected_digest": "84f96a6be8790384b52dda96908acc0ca9c493ada224d996f065d9eb3be4605b", + "mutations": [ + { + "name": "different_numeric_id", + "description": "THE mutation the R.3 ruling demands. A second scan lands on user 98765 instead of user 12345; it is the same defect and must keep the same identity. Without route templating this input produced a different digest and the finding was reported resolved and re-opened as new.", + "input": { + "target_id": "t-0003", + "rule_id_versioned": "nuclei:idor-object-reference@2026.07.1", + "http_method": "GET", + "route_template": "/api/v1/users/98765/orders/3f2504e0-4f89-11d3-9a0c-0305e82c3301", + "injection_point": "path", + "param_name": "", + "evidence_signal": "statusCodeFlip" + } + }, + { + "name": "different_uuid", + "description": "The order id is a freshly minted UUID on every seeded run.", + "input": { + "target_id": "t-0003", + "rule_id_versioned": "nuclei:idor-object-reference@2026.07.1", + "http_method": "GET", + "route_template": "/api/v1/users/12345/orders/9e107d9d-372b-b682-6bd8-1d3542a419d6", + "injection_point": "path", + "param_name": "", + "evidence_signal": "statusCodeFlip" + } + }, + { + "name": "both_ids_different_at_once", + "description": "The realistic re-scan: nothing about the concrete instance survives, and the identity still holds.", + "input": { + "target_id": "t-0003", + "rule_id_versioned": "nuclei:idor-object-reference@2026.07.1", + "http_method": "GET", + "route_template": "/api/v1/users/1/orders/00000000-0000-0000-0000-000000000000", + "injection_point": "path", + "param_name": "", + "evidence_signal": "statusCodeFlip" + } + }, + { + "name": "uuid_uppercased", + "description": "The UUID predicate is case-insensitive: the same identifier rendered in upper hex is the same identifier.", + "input": { + "target_id": "t-0003", + "rule_id_versioned": "nuclei:idor-object-reference@2026.07.1", + "http_method": "GET", + "route_template": "/api/v1/users/12345/orders/3F2504E0-4F89-11D3-9A0C-0305E82C3301", + "injection_point": "path", + "param_name": "", + "evidence_signal": "statusCodeFlip" + } + }, + { + "name": "uuid_without_dashes", + "description": "A 32-character dash-free UUID is caught by the long-hex rule (>= 16 hex characters) rather than the UUID rule. Both land on the same token, so a service that renders ids with dashes and one that renders them without do not fork identity.", + "input": { + "target_id": "t-0003", + "rule_id_versioned": "nuclei:idor-object-reference@2026.07.1", + "http_method": "GET", + "route_template": "/api/v1/users/12345/orders/3f2504e04f8911d39a0c0305e82c3301", + "injection_point": "path", + "param_name": "", + "evidence_signal": "statusCodeFlip" + } + }, + { + "name": "sha256_hex_object_id", + "description": "A content-addressed order id: 64 hex characters, also caught by the long-hex rule.", + "input": { + "target_id": "t-0003", + "rule_id_versioned": "nuclei:idor-object-reference@2026.07.1", + "http_method": "GET", + "route_template": "/api/v1/users/12345/orders/e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", + "injection_point": "path", + "param_name": "", + "evidence_signal": "statusCodeFlip" + } + }, + { + "name": "opaque_base64ish_order_token", + "description": "A 22-character alphanumeric token carrying both letters and digits, caught by the long-opaque rule (>= 20 characters). Note it is alphanumeric only: a base64url token containing '-' or '_' is deliberately NOT templated, because hyphenated slugs are route structure and over-templating merges distinct routes.", + "input": { + "target_id": "t-0003", + "rule_id_versioned": "nuclei:idor-object-reference@2026.07.1", + "http_method": "GET", + "route_template": "/api/v1/users/12345/orders/dXNlcjEyMzQ1Njc4OTAxMg", + "injection_point": "path", + "param_name": "", + "evidence_signal": "statusCodeFlip" + } + }, + { + "name": "openapi_curly_placeholders", + "description": "The repository's own OpenAPI document spells both segments '{userId}' and '{orderId}'. Different placeholder NAMES must not fork identity either - the token replaces the whole segment, name included.", + "input": { + "target_id": "t-0003", + "rule_id_versioned": "nuclei:idor-object-reference@2026.07.1", + "http_method": "GET", + "route_template": "/api/v1/users/{userId}/orders/{orderId}", + "injection_point": "path", + "param_name": "", + "evidence_signal": "statusCodeFlip" + } + }, + { + "name": "express_colon_placeholders", + "description": "Express, Rails and Sinatra spell the same route ':userId'/':orderId'.", + "input": { + "target_id": "t-0003", + "rule_id_versioned": "nuclei:idor-object-reference@2026.07.1", + "http_method": "GET", + "route_template": "/api/v1/users/:userId/orders/:orderId", + "injection_point": "path", + "param_name": "", + "evidence_signal": "statusCodeFlip" + } + }, + { + "name": "flask_angle_placeholders_with_converters", + "description": "Flask and Werkzeug spell it ''/'', converter prefix and all.", + "input": { + "target_id": "t-0003", + "rule_id_versioned": "nuclei:idor-object-reference@2026.07.1", + "http_method": "GET", + "route_template": "/api/v1/users//orders/", + "injection_point": "path", + "param_name": "", + "evidence_signal": "statusCodeFlip" + } + }, + { + "name": "already_carrying_the_canonical_placeholder_token", + "description": "Idempotence. A canonicalised route fed back through CanonicalRouteTemplate is a fixed point, so a record read out of the store and re-fingerprinted keeps its identity.", + "input": { + "target_id": "t-0003", + "rule_id_versioned": "nuclei:idor-object-reference@2026.07.1", + "http_method": "GET", + "route_template": "/api/v1/users//orders/", + "injection_point": "path", + "param_name": "", + "evidence_signal": "statusCodeFlip" + } + }, + { + "name": "concrete_ids_with_query_string_trailing_slash_and_duplicate_slashes", + "description": "Templating composes with the pre-existing canonicalisation: query string dropped, slash runs collapsed, trailing slash removed, leading slash added, and only then are the segments templated.", + "input": { + "target_id": "t-0003", + "rule_id_versioned": "nuclei:idor-object-reference@2026.07.1", + "http_method": "get", + "route_template": "api//v1/users/12345//orders/3f2504e0-4f89-11d3-9a0c-0305e82c3301/?debug=1&as=4192", + "injection_point": "path", + "param_name": "", + "evidence_signal": "statusCodeFlip" + } + } + ], + "notes": [ + "This fixture exists because of CRITIQUE-01 blocker 2: 'All three DAST fixtures arrive already templated, so the corpus cannot detect this. There is no test anywhere that feeds a concrete numeric or UUID segment.'", + "route_template is a DERIVED field. Area 40 owns the fingerprint, so area 40 canonicalises: area D emits whatever route it observed and CanonicalRouteTemplate templates it. Pushing templating to the producer would give the field two owners and let two producers emit three digests for one defect.", + "The placeholder token is , frozen. Angle brackets cannot appear unencoded in a URI path (RFC 3986), so no literal segment can collide with it, and is itself recognised as an already-templated segment, which makes the function idempotent.", + "The templating rules are conservative on purpose. Over-templating merges two distinct routes into one identity and loses a finding on upsert; under-templating only leaves a volatile route un-merged, which a producer can still fix by emitting '{id}' itself. The exact predicates and their length thresholds are in internal/record/FINGERPRINT-SPEC.md section 6.", + "Note the deliberate NON-mutations: '/api/v1/users/me/orders/...' and '/api/v1/users/12345/invoices/...' are DIFFERENT routes and must produce different digests. TestCanonicalRouteTemplateDoesNotOverTemplate in fingerprint_test.go pins the segments that must survive untouched." + ] +} diff --git a/testdata/fingerprint_corpus/host-01-openssl-debian.json b/testdata/fingerprint_corpus/host-01-openssl-debian.json new file mode 100644 index 0000000..03964af --- /dev/null +++ b/testdata/fingerprint_corpus/host-01-openssl-debian.json @@ -0,0 +1,88 @@ +{ + "id": "host-01-openssl-debian", + "tier": "host", + "description": "An operating-system package on the scanned host that matched a vulnerable version range. Host findings are read-only for the agent (plan/00-SPINE.md S7, remediable_by_agent=false) but still need a stable identity so they can be tracked, suppressed, and reported as resolved.", + "input": { + "target_id": "t-0002", + "advisory_id": "CVE-2022-0778", + "purl": "pkg:deb/debian/openssl@1.1.1n-0+deb11u3", + "package_manager": "apt", + "host_identifier": "openssl:amd64" + }, + "not_hashed": { + "installed_version": "1.1.1n-0+deb11u3", + "fixed_version": "1.1.1n-0+deb11u4", + "hostname": "build-runner-04", + "scanned_at": "2026-08-08T12:00:00Z" + }, + "hashed_fields": [ + "t-0002", + "host", + "CVE-2022-0778", + "pkg:deb/debian/openssl", + "apt:openssl:amd64" + ], + "expected_digest": "c6d2c1a505849bf987d9a03a5a3ce8ab6053a9ae5e0d154d332b464bb69e5953", + "mutations": [ + { + "name": "version_bumped", + "description": "A Debian point release. The version lives only in the purl and is stripped before hashing.", + "input": { + "target_id": "t-0002", + "advisory_id": "CVE-2022-0778", + "purl": "pkg:deb/debian/openssl@1.1.1w-0+deb11u1", + "package_manager": "apt", + "host_identifier": "openssl:amd64" + } + }, + { + "name": "purl_carrying_an_arch_qualifier", + "description": "Qualifiers are stripped with the version.", + "input": { + "target_id": "t-0002", + "advisory_id": "CVE-2022-0778", + "purl": "pkg:deb/debian/openssl@1.1.1n-0+deb11u3?arch=amd64", + "package_manager": "apt", + "host_identifier": "openssl:amd64" + } + }, + { + "name": "package_manager_uppercased", + "description": "'APT' and 'apt' are the same manager. The manager segment of the locator is lowercased, because a case difference between two scanner versions would otherwise fork every host finding at once.", + "input": { + "target_id": "t-0002", + "advisory_id": "CVE-2022-0778", + "purl": "pkg:deb/debian/openssl@1.1.1n-0+deb11u3", + "package_manager": "APT", + "host_identifier": "openssl:amd64" + } + }, + { + "name": "package_manager_and_identifier_padded", + "description": "Surrounding whitespace from a parsed CLI table is trimmed.", + "input": { + "target_id": "t-0002", + "advisory_id": "CVE-2022-0778", + "purl": "pkg:deb/debian/openssl@1.1.1n-0+deb11u3", + "package_manager": " apt ", + "host_identifier": " openssl:amd64 " + } + }, + { + "name": "purl_already_version_free", + "description": "PurlBase is idempotent, so a producer that already stripped the version agrees with one that did not.", + "input": { + "target_id": "t-0002", + "advisory_id": "CVE-2022-0778", + "purl": "pkg:deb/debian/openssl", + "package_manager": "apt", + "host_identifier": "openssl:amd64" + } + } + ], + "notes": [ + "The host tier shares the SCA formula, parameterised by detector_kind. Its locator is ':'.", + "host_identifier keeps its architecture suffix verbatim: 'openssl:amd64' and 'openssl:i386' are two installed packages the manager treats separately, so they are two findings.", + "The locator therefore contains more than one ':'. That is fine - it is a composed field, not a parsed one - but package_manager itself is rejected if it contains ':', because that would make the composition ambiguous." + ] +} diff --git a/testdata/fingerprint_corpus/sast-01-go-sql-string-concat.json b/testdata/fingerprint_corpus/sast-01-go-sql-string-concat.json new file mode 100644 index 0000000..a5504c4 --- /dev/null +++ b/testdata/fingerprint_corpus/sast-01-go-sql-string-concat.json @@ -0,0 +1,103 @@ +{ + "id": "sast-01-go-sql-string-concat", + "tier": "sast", + "description": "Go SQL injection by string concatenation, inside a named method. The baseline SAST fixture: a non-empty enclosing symbol, ordinal 0, two string literals and one abstracted local flowing into a preserved callee name.", + "input": { + "target_id": "t-0001", + "rule_id_versioned": "opengrep.go.lang.security.audit.sqli-string-concat@2026.07.1", + "repo_rel_path": "internal/api/store.go", + "enclosing_symbol_path": "internal/api/store.go::Store.FindByName", + "snippet": "rows, err := db.Query(\"SELECT * FROM users WHERE name = '\" + name + \"'\")", + "ordinal": 0 + }, + "not_hashed": { + "line": 128, + "column": 9, + "end_line": 128, + "advisory_id": "CWE-89", + "evidence_class": "sast_static_only", + "scanned_at": "2026-08-08T12:00:00Z", + "comment": "Every value in this object was available to the producer and MUST NOT appear in the hash input. plan/40-record-and-storage.md: 'NEVER hashed: line number, column number, the literal (non-normalized) snippet text, advisory_id.'" + }, + "hashed_fields": [ + "t-0001", + "sast", + "opengrep.go.lang.security.audit.sqli-string-concat@2026.07.1", + "internal/api/store.go", + "internal/api/store.go::Store.FindByName", + "$1, $2 := $3.Query( + $4 + )", + "0" + ], + "expected_digest": "13c60ccf8ec84530e075db4190005b03bc87050210065398fd48088683bb1aa6", + "mutations": [ + { + "name": "shifted_down_three_lines_and_reindented", + "description": "The match moved down the file and was reindented. This is the line-number-only mutation R.2's validation requires: SastInput carries no line field, so the strongest available form is to change everything a line move implies.", + "input": { + "target_id": "t-0001", + "rule_id_versioned": "opengrep.go.lang.security.audit.sqli-string-concat@2026.07.1", + "repo_rel_path": "internal/api/store.go", + "enclosing_symbol_path": "internal/api/store.go::Store.FindByName", + "snippet": "\n\n\n rows, err := db.Query(\"SELECT * FROM users WHERE name = '\" + name + \"'\")\n", + "ordinal": 0 + }, + "not_hashed": { + "line": 131, + "column": 9 + } + }, + { + "name": "crlf_endings_and_trailing_comment_naming_the_old_line", + "description": "A Windows checkout plus a reviewer comment that literally records the old line number. Comments are stripped and CRLF is folded to LF before hashing.", + "input": { + "target_id": "t-0001", + "rule_id_versioned": "opengrep.go.lang.security.audit.sqli-string-concat@2026.07.1", + "repo_rel_path": "internal/api/store.go", + "enclosing_symbol_path": "internal/api/store.go::Store.FindByName", + "snippet": "rows, err := db.Query(\"SELECT * FROM users WHERE name = '\" + name + \"'\") // was line 128\r\n", + "ordinal": 0 + } + }, + { + "name": "leading_comment_line_added", + "description": "A triage annotation added above the match. Comments carry no identity.", + "input": { + "target_id": "t-0001", + "rule_id_versioned": "opengrep.go.lang.security.audit.sqli-string-concat@2026.07.1", + "repo_rel_path": "internal/api/store.go", + "enclosing_symbol_path": "internal/api/store.go::Store.FindByName", + "snippet": "// SECURITY: reviewed 2026-01-14, tracked as ANV-412\nrows, err := db.Query(\"SELECT * FROM users WHERE name = '\" + name + \"'\")", + "ordinal": 0 + } + }, + { + "name": "locals_renamed_and_literal_text_changed", + "description": "A pure rename refactor plus a different SQL string. This is the metavariable-abstraction property the specification adopts from Semgrep's match_based_id: local identifiers become positional $1..$N and string literals become , so neither a rename nor a literal edit forks the finding's identity.", + "input": { + "target_id": "t-0001", + "rule_id_versioned": "opengrep.go.lang.security.audit.sqli-string-concat@2026.07.1", + "repo_rel_path": "internal/api/store.go", + "enclosing_symbol_path": "internal/api/store.go::Store.FindByName", + "snippet": "records, e := conn.Query(\"SELECT * FROM accounts WHERE label = '\" + userName + \"'\")", + "ordinal": 0 + } + }, + { + "name": "repo_rel_path_in_windows_form", + "description": "The same file reported by a Windows producer. repo_relpath is canonicalised to POSIX form before hashing, so two producers scanning one repository agree.", + "input": { + "target_id": "t-0001", + "rule_id_versioned": "opengrep.go.lang.security.audit.sqli-string-concat@2026.07.1", + "repo_rel_path": ".\\internal\\api\\store.go", + "enclosing_symbol_path": "internal/api/store.go::Store.FindByName", + "snippet": "rows, err := db.Query(\"SELECT * FROM users WHERE name = '\" + name + \"'\")", + "ordinal": 0 + } + } + ], + "notes": [ + "Field order is target_id, \"sast\", rule_id_versioned, repo_relpath, enclosing_symbol_path, normalized_match, ordinal.", + "The tier token in position 2 is the literal \"sast\" for both sast_reachable and sast_static_only findings, so promoting a static-only finding to reachable does not change its identity.", + "Normalization of the snippet: `db` is a receiver bound to a local, so it is abstracted to $3; `Query` follows a '.' selector so it is preserved as API surface." + ] +} diff --git a/testdata/fingerprint_corpus/sast-02-python-shell-command.json b/testdata/fingerprint_corpus/sast-02-python-shell-command.json new file mode 100644 index 0000000..b0f0631 --- /dev/null +++ b/testdata/fingerprint_corpus/sast-02-python-shell-command.json @@ -0,0 +1,89 @@ +{ + "id": "sast-02-python-shell-command", + "tier": "sast", + "description": "Python command injection at module level: the SECOND of two byte-identical call sites in one file, so this fixture exercises ordinal 1 and an EMPTY enclosing_symbol_path (top-level module code has no enclosing symbol).", + "input": { + "target_id": "t-0001", + "rule_id_versioned": "opengrep.python.lang.security.audit.dangerous-system-call@2026.07.1", + "repo_rel_path": "scripts/deploy.py", + "enclosing_symbol_path": "", + "snippet": "os.system(\"rm -rf \" + path)", + "ordinal": 1 + }, + "not_hashed": { + "line": 41, + "column": 5, + "advisory_id": "CWE-78", + "evidence_class": "sast_reachable", + "sibling_call_site_line": 12, + "comment": "The identical call site at line 12 gets ordinal 0 and this one gets ordinal 1. Without the ordinal field the two would share a digest and one finding would be lost on upsert against UNIQUE (target_id, fingerprint)." + }, + "hashed_fields": [ + "t-0001", + "sast", + "opengrep.python.lang.security.audit.dangerous-system-call@2026.07.1", + "scripts/deploy.py", + "", + "$1.system( + $2)", + "1" + ], + "expected_digest": "d2239ee40c3f9da1cc9db3417962e3794b0c5baca0409015ff331a0aea095242", + "mutations": [ + { + "name": "shifted_down_and_reindented_into_a_block", + "description": "The statement moved and gained indentation. Neither line position nor indentation is hashed.", + "input": { + "target_id": "t-0001", + "rule_id_versioned": "opengrep.python.lang.security.audit.dangerous-system-call@2026.07.1", + "repo_rel_path": "scripts/deploy.py", + "enclosing_symbol_path": "", + "snippet": "\n\n os.system(\"rm -rf \" + path)\n", + "ordinal": 1 + }, + "not_hashed": { + "line": 58, + "column": 9 + } + }, + { + "name": "hash_line_comment_appended", + "description": "A '#' comment is stripped like any other comment, which is why the normalizer treats '#' as a line comment across languages.", + "input": { + "target_id": "t-0001", + "rule_id_versioned": "opengrep.python.lang.security.audit.dangerous-system-call@2026.07.1", + "repo_rel_path": "scripts/deploy.py", + "enclosing_symbol_path": "", + "snippet": "os.system(\"rm -rf \" + path) # noqa: was line 41", + "ordinal": 1 + } + }, + { + "name": "local_renamed_and_literal_changed", + "description": "`path` renamed and the shell string edited; both are abstracted, so identity holds.", + "input": { + "target_id": "t-0001", + "rule_id_versioned": "opengrep.python.lang.security.audit.dangerous-system-call@2026.07.1", + "repo_rel_path": "scripts/deploy.py", + "enclosing_symbol_path": "", + "snippet": "os.system(\"rm -rf --one-file-system \" + target_dir)", + "ordinal": 1 + } + }, + { + "name": "repo_rel_path_with_dot_slash_prefix", + "description": "'./scripts/deploy.py' and 'scripts/deploy.py' are one path after canonicalisation.", + "input": { + "target_id": "t-0001", + "rule_id_versioned": "opengrep.python.lang.security.audit.dangerous-system-call@2026.07.1", + "repo_rel_path": "./scripts/deploy.py", + "enclosing_symbol_path": "", + "snippet": "os.system(\"rm -rf \" + path)", + "ordinal": 1 + } + } + ], + "notes": [ + "An empty enclosing_symbol_path is hashed as an empty field rather than dropped, so the SAST tier always joins exactly seven fields. Dropping it instead would let a match with no symbol collide with a shorter field list.", + "`os` is abstracted to $1: a lexer cannot distinguish a module qualifier from a receiver variable, and guessing wrong in the other direction would fork the digest of unchanged code." + ] +} diff --git a/testdata/fingerprint_corpus/sca-01-log4shell-maven.json b/testdata/fingerprint_corpus/sca-01-log4shell-maven.json new file mode 100644 index 0000000..47aef2d --- /dev/null +++ b/testdata/fingerprint_corpus/sca-01-log4shell-maven.json @@ -0,0 +1,102 @@ +{ + "id": "sca-01-log4shell-maven", + "tier": "sca", + "description": "A repository dependency (Log4Shell, declared in a monorepo service's pom.xml) that matched a vulnerable version range. This is the fixture that pins the single most load-bearing exclusion in the specification: the version string is never hashed.", + "input": { + "target_id": "t-0001", + "advisory_id": "GHSA-jfh8-c2jp-5v3q", + "purl": "pkg:maven/org.apache.logging.log4j/log4j-core@2.14.1", + "manifest_rel_path": "services/api/pom.xml" + }, + "not_hashed": { + "resolved_version": "2.14.1", + "fixed_version": "2.15.0", + "first_seen_at": "2026-08-08T12:00:00Z", + "comment": "plan/40-record-and-storage.md: 'NEVER hashed: the version string - bumping 1.2.3->1.2.4 while still inside the vulnerable range must not mint a new finding; resolution is proved by re-evaluating advisory_affects, not by identity change.'" + }, + "hashed_fields": [ + "t-0001", + "sca", + "GHSA-jfh8-c2jp-5v3q", + "pkg:maven/org.apache.logging.log4j/log4j-core", + "services/api/pom.xml" + ], + "expected_digest": "c3208f00ab0f426a1c2cbb2ccf0033881bb4bf6cb506763dea0840f03dd24cd8", + "mutations": [ + { + "name": "version_bumped_inside_the_vulnerable_range", + "description": "2.14.1 -> 2.14.0. Still vulnerable, still the same finding. If this changed the digest, first_seen_at would reset and any fingerprint-keyed suppression would silently stop applying.", + "input": { + "target_id": "t-0001", + "advisory_id": "GHSA-jfh8-c2jp-5v3q", + "purl": "pkg:maven/org.apache.logging.log4j/log4j-core@2.14.0", + "manifest_rel_path": "services/api/pom.xml" + } + }, + { + "name": "version_bumped_out_of_the_vulnerable_range", + "description": "2.14.1 -> 2.17.1. The identity is still the same; resolution is proved by re-evaluating advisory_affects, never by the fingerprint changing.", + "input": { + "target_id": "t-0001", + "advisory_id": "GHSA-jfh8-c2jp-5v3q", + "purl": "pkg:maven/org.apache.logging.log4j/log4j-core@2.17.1", + "manifest_rel_path": "services/api/pom.xml" + } + }, + { + "name": "purl_carrying_qualifiers", + "description": "Qualifiers are stripped by PurlBase along with the version.", + "input": { + "target_id": "t-0001", + "advisory_id": "GHSA-jfh8-c2jp-5v3q", + "purl": "pkg:maven/org.apache.logging.log4j/log4j-core@2.14.1?type=jar&classifier=sources", + "manifest_rel_path": "services/api/pom.xml" + } + }, + { + "name": "purl_carrying_qualifiers_and_a_subpath", + "description": "Subpath ('#') is stripped as well.", + "input": { + "target_id": "t-0001", + "advisory_id": "GHSA-jfh8-c2jp-5v3q", + "purl": "pkg:maven/org.apache.logging.log4j/log4j-core@2.14.1?type=jar#META-INF/MANIFEST.MF", + "manifest_rel_path": "services/api/pom.xml" + } + }, + { + "name": "purl_scheme_and_type_uppercased", + "description": "The purl specification defines the scheme and type as case-insensitive with a lowercase canonical form, so 'PKG:MAVEN/...' and 'pkg:maven/...' are one package. The namespace and name are NOT folded, because their case-sensitivity is type-dependent.", + "input": { + "target_id": "t-0001", + "advisory_id": "GHSA-jfh8-c2jp-5v3q", + "purl": "PKG:MAVEN/org.apache.logging.log4j/log4j-core@2.14.1", + "manifest_rel_path": "services/api/pom.xml" + } + }, + { + "name": "purl_already_version_free", + "description": "A producer that strips the version itself gets the same digest as one that does not; PurlBase is idempotent.", + "input": { + "target_id": "t-0001", + "advisory_id": "GHSA-jfh8-c2jp-5v3q", + "purl": "pkg:maven/org.apache.logging.log4j/log4j-core", + "manifest_rel_path": "services/api/pom.xml" + } + }, + { + "name": "manifest_path_in_windows_form", + "description": "The locator is canonicalised to POSIX form like any other repo-relative path.", + "input": { + "target_id": "t-0001", + "advisory_id": "GHSA-jfh8-c2jp-5v3q", + "purl": "pkg:maven/org.apache.logging.log4j/log4j-core@2.14.1", + "manifest_rel_path": "services\\api\\pom.xml" + } + } + ], + "notes": [ + "Field order is target_id, detector_kind, advisory_id, purl_base, locator. For the SCA tier the locator is the manifest or lockfile path: the same vulnerable package pulled in by two manifests in a monorepo is two findings, with two owners and two fixes.", + "advisory_id is hashed verbatim, not case-folded. GHSA identifiers mix case meaningfully and folding them would fork identity against the advisory table.", + "detector_kind is what keeps this fixture from colliding with host-01 when a repository dependency and a host package share an advisory." + ] +}