Skip to content

Read the bytes git carries in the PHI sweep, as a union with the walk - #72

Merged
NSchatz merged 1 commit into
mainfrom
phi-scan-index-corpus
Aug 11, 2026
Merged

Read the bytes git carries in the PHI sweep, as a union with the walk#72
NSchatz merged 1 commit into
mainfrom
phi-scan-index-corpus

Conversation

@NSchatz

@NSchatz NSchatz commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

What was wrong

All mode reconciled the paths it walked against the paths git tracks. A comparison of path sets is satisfied completely by a working tree carrying different content at those paths, so [phi-scan] OK: no hits at exit 0 was reachable over a corpus the sweep never opened.

Four states reproduced on 26daeda, each exiting 0 with OK: no hits over a synthetic stream carrying a patient name, a mother's maiden name, a birthdate and a dashed SSN:

state why neither route saw it
decoy content at a tracked path the walk read the clean file on disk; the reconciliation only asked whether a file was there
a tracked path outside every walk root the roots are three names and the reconciliation only looks within them
a tracked symlink or gitlink outside every walk root the walk classifies entries inside a root; git carries a link's target path, itself a PHI surface
an empty index every clause is satisfied for free by having nothing to reconcile against

All four now report or refuse.

The remedy

All mode reads the blob behind every index entry, as a union with the walk and never as a replacement. No root was narrowed, no clause dropped, and a blob whose bytes the walk provably already scanned is skipped by byte comparison, so nothing is reported twice. The skip is deliberately not a stat, an mtime or a hash (those are exactly what a decoy defeats), and line endings are deliberately not normalized before it: that compares a derived form, and content differing only in what the normalizer erases would then be skipped.

The mechanism is written down in exactly one place, buildTargetsForIndex; every other surface states only what a green means.

The port is not a copy

The five non-transferable axes, re-derived against this repo rather than inherited:

  • Exit codes. 0/1/2 unchanged; every new refusal is an InvocationError returning 2. A top-level backstop was added because an uncaught throw exits 1, and 1 is the one code a caller reads as evidence.
  • --staged scope. Deliberately unchanged, and not for the reason the class note gives: that rationale rests on a mode-blind violator exemption this repo does not have. It is a hook decision, because it changes what a commit is blocked on.
  • EOL normalization. Does not fire: no .gitattributes, core.autocrlf and core.eol both unset, Linux CI. Measured, not assumed.
  • Gitlinks. None tracked, but the index is not uniformly 100644 (three 100755 entries), so narrowing the readable-mode set would red-lock the repo.
  • Roots and exclusions. Three roots; .md excluded on both routes and applied last, so a NAME exemption cannot excuse an entry whose bytes the route cannot read; gitignore not consulted on the index route.

And one state a sibling's list names that was already closed here, so this route is not credited with it: a tracked file absent from the working tree under a declared root.

What it found

18 tracked non-markdown files sit outside every walk root and no route had ever opened one. One carried this package's own published contact address, already public in every release's registry metadata and not patient data. It is declared with the cost written beside it: an allowed email domain is global and route-blind.

The positive control

A green over an unopened corpus is indistinguishable from a green over a corpus that was read, so a case showing the scanner passing proves nothing. The control puts this package's own manifest, byte for byte, at the same out-of-root path in a throwaway tree and then strikes the declaration: the same corpus reds at exit 1 naming the file. A third case shows the superseded scanner exiting 0 on the undeclared address, because no route reached the path at all.

17 cases, 10 red under the mutation the suite itself defines. The other seven are green on both by design and the notes say which and why.

Gates

scripts/verify.sh astm green (run after staging, so the corpus-scanning gates saw the new files). conformance-refuter NOT REFUTED x3, passes 1, 2 and 3 of a cap of 4.

  • Pass 1 found two claim defects and a drifting figure. Both code-affecting fixes are in: the report() justification no longer asserts a universal about the input space, and the empty-index refusal now runs after the walk is scanned, because refused before it the run was strictly worse than the superseded scanner's for one input (exit 1 naming every locus before, exit 2 naming nothing after).
  • Pass 2 found a miscount in the remedy that closed the first finding, and an ordering the new control did not pin.
  • Pass 3 found no introduced defect.

Disclosed, not closed

  • Working-tree bytes at a path outside every walk root are read by neither route, whether or not git tracks the path. Base-identical; now written down.
  • .md is exempt on both routes, so patient data committed in tracked markdown is opened by no route. Base-identical and disclosed as the one exclusion.
  • One assertion added after pass 3 (pinning print order on the sibling refusal path) is ungraded; it was verified red against a refusal-first scanner and green on head.

… (PHI-SCAN)

All mode reconciled the paths it walked against the paths git tracks, and a
comparison of path SETS is satisfied completely by a working tree carrying
different content at those paths. Four states were reproduced on 26daeda, each
printing `[phi-scan] OK: no hits` at exit 0 over a synthetic stream carrying a
patient name, a mother's maiden name, a birthdate and a dashed SSN: decoy
content at a tracked path; a tracked path outside every walk root; a tracked
symlink or gitlink outside every walk root; and an empty index, against which
every check passes vacuously. All four now report or refuse.

It is a UNION, never a replacement: no root narrowed, no clause dropped, and a
blob whose bytes the walk provably already scanned is skipped by byte
comparison, so nothing is reported twice. The skip is deliberately not a stat,
an mtime or a hash, because those are what a decoy defeats, and line endings are
deliberately not normalized before it: that compares a derived form, and a decoy
differing only in what the normalizer erases would then be skipped.

THE PORT IS NOT A COPY. The five axes are re-derived against this repo:

- Exit codes 0/1/2 unchanged, plus a top-level backstop, because an uncaught
  throw exits 1 and 1 is the one code a caller reads as evidence.
- `--staged` deliberately unchanged. The sibling's red-lock rationale does not
  apply here (this repo has no mode-blind violator exemption to unscope); it is
  a HOOK decision, because it changes what a commit is blocked on.
- EOL normalization does not fire: no `.gitattributes`, `core.autocrlf` and
  `core.eol` both unset, Linux CI. Measured, not assumed.
- No gitlinks tracked, but the index is not uniformly 100644 (three 100755
  entries), so narrowing REGULAR_BLOB_MODES would red-lock the repo.
- Three roots, `.md` excluded on both routes and applied LAST so a NAME
  exemption cannot excuse a link whose target path is itself a PHI surface,
  gitignore not consulted on the index route.

And one state a sibling's list names was ALREADY closed here, so this route is
not credited with it: a tracked file absent from the working tree under a
declared root, which `refuseUnobserved`'s second clause has refused since
2026-08-07.

The sweep reported 18 tracked non-markdown files outside every walk root that no
route had opened. One carried this package's own published contact address,
already public in every release's registry metadata and not patient data,
declared with the cost written beside it: an allowed email domain is global and
route-blind. A positive control puts this package's own manifest at the same
out-of-root path in a throwaway tree and strikes the declaration, so the same
corpus reds at exit 1: the green is earned by the declaration rather than by the
file never being opened.

17 cases, 10 red under the mutation the suite defines. `trackedUnder` no longer
shells out per root, so the reconciliation and the index route read one
`git ls-files -s -z` rather than two answers that can disagree.

conformance-refuter: NOT REFUTED x3 (passes 1, 2 and 3 of a cap of 4). Pass 1
found two claim defects and a drifting figure, pass 2 found a miscount in the
remedy for the first, pass 3 found none. Both code-affecting fixes are in: the
`report()` justification no longer asserts a universal about the input space,
and the empty-index refusal now runs AFTER the walk is scanned, because refused
before it the run was strictly worse than the superseded scanner's for one input
(exit 1 naming every locus before, exit 2 naming nothing after).
@NSchatz
NSchatz merged commit c75328a into main Aug 11, 2026
8 checks passed
@NSchatz
NSchatz deleted the phi-scan-index-corpus branch August 11, 2026 04:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant