fix(hl7): make policy safe-harbor actually Safe Harbor for the encounter loci - #49
Merged
Conversation
…esidual
Parked by the coordinator after every worker in the fleet died on an API auth
error (Login expired) mid-slice. This commit exists ONLY so the work is not
lost. It MUST NOT be treated as landed:
- scripts/verify.sh NEVER RAN on it
- the conformance-refuter NEVER GRADED it
- it changes what leaves the box in a de-identification toolkit, which is
the highest-consequence surface in this tree
Intended scope was the founder decision of 2026-08-11: make policy safe-harbor
actually Safe Harbor by dropping PV1-19, PV1-44, PV1-45, OBR-7 and DG1-5 from
retention under SAFE_HARBOR_PROFILE while keeping them under
LIMITED_DATA_SET_PROFILE, plus seeding the leak corpus that is structurally
blind to the class and correcting docs-content/limitations.md.
Whether this tree does any of that is UNKNOWN and unmeasured. src/retention.ts
and test/fixtures/hl7/adt-a03.hl7 are new and ungraded; probe.local.mts is
worker scratch. Re-measure from scratch before building on it. Do not
cherry-pick it green.
119 lines the worker flushed to test/corpus/leak-corpus.test.ts after the first park commit. Same status as everything else on this branch: verify never ran, no refuter graded it. Worth keeping specifically because seeding this corpus was the DEEPER half of the slice. The gate was structurally blind to the class its own roadmap says it must catch -- no HL7 fixture carried PV1-19/44/45 -- so a fix without a fixture would leave nothing to catch a regression.
…loci (DEID-SAFE-HARBOR-DATES) Founder decision, 2026-08-11, on a measured finding: under `policy: "safe-harbor"` seven identifying loci survived BYTE-IDENTICAL with no manifest entry at all, because RETAIN_SEGMENTS retains a segment whole and nothing carved its identifying fields back out: PV1-19 visit number, PV1-44 admit, PV1-45 discharge, OBR-7 observation date, DG1-5 diagnosis date, and the placer/filler order numbers (OBR-2/3, ORC-2/3). The manifest read only `PID-3[0]:transformed PID-5:removed PID-7:transformed`. HIPAA Safe Harbor 164.514(b)(2)(i)(C) requires removal of all elements of dates except year directly related to an individual, and names admission and discharge dates in the regulation text itself; the visit and order numbers are unique identifying codes the (R) catch-all reaches. A policy NAMED safe-harbor that returns them is a compliance trap for anyone who trusts the name. WHAT CHANGES - New format-agnostic retention model (src/retention.ts): two named classes, `encounter-dates` and `encounter-identifiers`, each grounded in a DIFFERENT regulation. 164.514(e)(2)'s limited-data-set exclusion list enumerates sixteen direct identifiers and contains no date and NO CATCH-ALL, which is exactly why these two classes may stay in a limited data set and may not stay under Safe Harbor. - SAFE_HARBOR_PROFILE retains neither class; LIMITED_DATA_SET_PROFILE retains both. - New HL7 carve-out table (RETAINED_LOCUS_RULES): retaining a SEGMENT no longer means retaining every field in it. Under safe-harbor the four dates generalize to their year and the five identifier loci are blocked as (R). - Anything still retained is RECORDED. A kept locus emits a manifest entry with disposition `retained`, transform `retain`, code DEID_RESIDUAL_RETAINED, so it reaches the Expert-Determination support report's residual inventory instead of being invisible twice over. DeidManifestEntry.disposition and ReportDisposition gain "retained"; DispositionSummary gains a `retained` count. - widen-never-narrow now covers retention, and it reads the OPPOSITE way round from a transform override: DROPPING a class removes more (allowed), ADDING one keeps more (fatal DEID_PROFILE_INVALID). It is a subset test, not a rank comparison. - Fail closed by default: retention rides on DeidOptions and only profileOptions() puts it there, so a bare options bag, or one built by hand from a profile's `policy`, keeps nothing. The engine's retention branch also runs AFTER the clinical / free-text / unrecognized-structure guards, so the flag can never keep prose or unknown structure. - New applier edit kind `none`: a retained locus is not written back at all, so it survives byte-identical rather than being flattened into a single-component field. THE LEAK CORPUS WAS STRUCTURALLY BLIND TO THIS CLASS, WHICH IS THE DEEPER DEFECT No HL7 fixture carried PV1-19/44/45 at all, so the headline gate could not have caught a regression: a sweep only reports on sentinels a fixture actually seeds. test/fixtures/hl7/adt-a03.hl7 seeds all seven plus the usual demographics, and is wired into test/corpus/leak-corpus.test.ts. Proven RED on the pre-fix source: the new fixture + tests over f8b7f56's src/ fail 8 tests, including the headline leak gate reporting all seven encounter sentinels surviving. POSITIVE CONTROL IN BOTH DIRECTIONS, because a detector that reports zero can be a gap rather than a clearance: safe-harbor removes all seven, limited-data-set still carries all seven byte-identical (composites intact, assigning authority and all) while still removing the patient identifiers 164.514(e)(2) does name. DOCS docs-content/limitations.md's "loci absent from the parser models fail closed" row read as the opposite of the truth for a retained segment, so it is DELETED rather than caveated. Its place is taken by an accurate section that names what is retained AND names what is still passed through and recorded nowhere: SPM-17, PV1-7/8, OBR-16. The support report and its guide no longer redirect a determiner to "each format's retained-segment notes" for something the report can now inventory.
…angeset `retain` joins the fail-closed arm of applyTransform explicitly rather than falling into `default`, so the exhaustiveness rule is satisfied by naming it and the comment records WHY reaching that arm must still block: it means a policy asked for retention per-category, which is not how retention is decided. verify.sh is green on this tree: typecheck, lint, format:check, check, phi-scan, check:no-internal-refs, test:coverage, build, attw, smoke.
…er pass 1
Pass 1 returned VERDICT: REFUTED with three INTRODUCED findings inside the bar.
All three are addressed here; the remedy diff is what pass 2 grades.
F2 (blocker, criteria 2 and 3) -- the 164.514(e)(2) ground was ASSERTED, not
checked. PV1-19 was hard-coded to the (R) catch-all without ever reading CX-5,
although the adapter already ships categoryForIdentifierType() and uses it for
PID-3. 164.514(e)(2)(vii) NAMES medical record numbers and (ix) NAMES account
numbers, so for a routine `MR`-typed visit number the slice's own justification
was false -- and under limited-data-set the pass republished in the clear the
very MRN it had just pseudonymized at PID-3, in the same message, while the
determiner's inventory labelled it OTHER_UNIQUE_ID.
- PV1-19 now carries routeByTypeCode, resolving its category per repetition
from CX-5 exactly as PID-3 does. An MR-typed visit number is pseudonymized
to the SAME surrogate as PID-3; AN/SS/MA are transformed likewise; only a
VN-typed or untyped value is the encounter identifier and retainable.
- LIMITED_DATA_SET_DIRECT_IDENTIFIERS encodes the sixteen direct identifiers
as data, so the citation is enforced rather than restated in prose. Exactly
two of the eighteen categories are retainable: DATES and (R).
- The refusal is defence in depth: the extractor will not mark such a locus,
AND the engine refuses it anyway via isRetainableCategory().
F3 (major, criterion 7) -- the engine retained on the locus marker alone and
never read options.retainedLoci, so DeidOptions.retainedLoci's own JSDoc ("absent
or empty retains nothing") was false at the layer it was declared on, and the
class contract was unenforceable because the marker was a class-less boolean.
GenericLocus.retainedByPolicy?: boolean becomes retention?: RetainedLocusClass,
and handleLocus now needs THREE keys to line up: an adapter proposal, the
configured options naming that class, and a retainable category.
F4 (major) -- a hand-built bag pairing the reserved `safe-harbor` label with a
retention set returned all seven loci while the report printed
`Policy: safe-harbor`. assertRetentionContract() makes that a fatal
DEID_POLICY_INVALID, the retention analogue of the existing date-shift label
guard, on the one route no profile-level check can see.
F1 (major, criterion 8) -- the corrected honesty page replaced a true broad
limitation with a false narrow enumeration: it named SPM-17 and PV1-7/8/OBR-16
as though complete, when a routine message carries 15 more unrecorded
full-precision timestamps (EVN, PV2, PR1, RXA, RXD, FT1, TXA, SPM). Criterion 8
tests exactly this, and a new false claim is worse than the old one. The
enumeration is DELETED and replaced by the true class statement in
limitations.md, README.md, guides-hl7.md, src/hl7/index.ts, src/hl7/retain.ts
and the changeset.
F5 (minor) -- the inventory printed a kept year and a kept 14-digit timestamp
identically. RetainedQuasiIdentifier now carries its transform, and the
rendering says "coarse residual" or "whole value kept".
F6 (minor) -- that the retention classes are HL7-only is now stated in the
retention module and the changeset rather than left to be discovered.
F7 and F8 are PRE-EXISTING (they reproduce on f8b7f56) and are reported as
backlog lines, not fixed here.
verify.sh green: 621 tests, typecheck, lint, format:check, check, phi-scan,
check-no-internal-refs, coverage, build, attw, smoke.
…cked conformance-refuter pass 2 returned VERDICT: NOT REFUTED, with one INTRODUCED minor (N2) it flagged as worth fixing before merge: four published lines still said PV1-19 is "blocked as the (R) catch-all" and none mentioned the type-code routing the remedy added. After that remedy an MR/AN/SS-typed PV1-19 is PSEUDONYMIZED or REDACTED as the identifier it really is, under BOTH profiles, and the limited-data-set line "keeps the visit number unchanged" lacked the caveat that this is only ever true of a VN-typed or untyped one. The defect under-states protection rather than over-stating it, so it is not the class of claim defect that gated pass 1 -- but a published surface that describes a mechanism the code no longer has is a claim defect either way. Corrected in docs-content/limitations.md, docs-content/guides-hl7.md, README.md and src/hl7/index.ts. The changeset already described the routing in full and is unchanged. Not fixed here, held as backlog lines because they reproduce on f8b7f56: - N1: the id-number write-back clears CX.1 only, so a fully-populated PV1-19 keeps CX.6 assigning facility and the CX.7/CX.8 effective/expiration dates while the manifest calls the row `blocked`. Same shape as the existing PID-3 behaviour for an SS-typed repetition. - F2 residual: an UNTYPED PV1-19 carrying an MRN is still retainable under limited-data-set. The wire gives no signal, and no surface claims otherwise. - F7: ~15 unrecorded full-precision timestamps across EVN, PV2, PR1, RXA, RXD, FT1, TXA, SPM, plus a DOB riding through as OBX-5 with a DTM type. - F8: under limited-data-set, PID-7 is date-shifted while the retained encounter dates are not, so intervals are not preserved across that line. verify.sh green. Refuter passes spent: 2 of 4.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The defect, reproduced by probe
Under
policy: "safe-harbor", seven identifying loci survived byte-identical with no manifest entry at all, becauseRETAIN_SEGMENTSretains a segment whole and nothing carved its identifying fields back out:PV1-19visit number,PV1-44admit,PV1-45discharge,OBR-7observation date,DG1-5diagnosis date, and the placer/filler order numbers (OBR-2/3,ORC-2/3).The manifest read only
PID-3[0]:transformed PID-5:removed PID-7:transformed. Reproduced onf8b7f56before any change, and again as a red test.HIPAA Safe Harbor 45 CFR 164.514(b)(2)(i)(C) requires removal of all elements of dates except year that are directly related to an individual, and names admission and discharge dates in the regulation text itself; a visit or order number is a unique identifying code the (R) catch-all reaches. A policy named
safe-harborthat returns them is a compliance trap for anyone who trusts the name. Founder decision, 2026-08-11; breaking on purpose while the package is pre-alpha, since0.1.0would freeze the guarantee.What changes
SAFE_HARBOR_PROFILEretains nothing. The four dates generalize to their year; the five identifier loci are removed as category (R).LIMITED_DATA_SET_PROFILEkeeps all seven, grounded in 164.514(e)(2), whose limited-data-set exclusion list enumerates sixteen direct identifiers, contains no date, and has no catch-all. Encoded as data (LIMITED_DATA_SET_DIRECT_IDENTIFIERS,isRetainableCategory()) so the citation is enforced rather than restated: exactly two of the eighteen categories are retainable.retainedLocikeeps nothing.PV1-19is routed by its CX-5 identifier-type code, exactly asPID-3is. This is load bearing, not cosmetic: a visit-number field routinely carries a medical record or account number, both named by 164.514(e)(2). AnMR-typed value now gets the same keyed surrogate as the matching PID-3 entry instead of being republished in the clear beside it.safe-harbor-labelled policy may not retain at all, whatever the options bag says: fatalDEID_POLICY_INVALID, the retention analogue of the existing date-shift label guard.retained, transformretain, codeDEID_RESIDUAL_RETAINED, so it reaches the Expert-Determination support report's residual inventory instead of being invisible in both artifacts. Each inventory row now carries its transform, so a kept year is distinguishable from a kept full-precision timestamp.defineDeidProfile()'s widen-never-narrow contract now covers retention, and it reads the opposite way round from a transform override: dropping a retained class removes more (allowed); adding one keeps more (fatal). A subset test, not a rank comparison.The leak corpus was structurally blind to this class, which is the deeper defect
No HL7 fixture carried
PV1-19/44/45at all, so the headline gate could not have caught a regression: a sweep only reports on sentinels a fixture actually seeds.test/fixtures/hl7/adt-a03.hl7seeds all seven plus the usual demographics and is wired intotest/corpus/leak-corpus.test.ts.Proven red on the pre-fix source. The new fixture and tests run against
f8b7f56'ssrc/fail 8 tests, including the headline leak gate reporting all seven encounter sentinels surviving. The refuter independently reproduced this.Positive control in both directions, because a detector that reports zero can be a gap rather than a clearance:
safe-harborremoves all seven;limited-data-setstill carries all seven byte-identical, composites and assigning authorities intact, while still removing the patient identifiers 164.514(e)(2) does name.Corrected claims
docs-content/limitations.mdis the honesty page, and its "loci absent from the parser models fail closed" row read as the opposite of the truth for a retained segment. That claim is deleted, not caveated. In its place the page states the true class: every field of a retained segment the carve-out does not name is still passed through and recorded nowhere, still including full-precision timestamps in EVN, PV2, PR1, RXA, RXD, FT1, TXA and SPM and the provider names in PV1-7/8 and OBR-16. The carve-out narrows that class; it does not close it.Gate
scripts/verify.sh deidgreen: 621 tests, typecheck, lint, format:check, check, phi-scan, check-no-internal-refs, coverage, build, attw, smoke.conformance-refuterpass 1 returned REFUTED with three INTRODUCED findings inside the bar, all addressed ina1851a1:MR-typedPV1-19was retained underlimited-data-set, republishing in the clear the MRN pseudonymized at PID-3 in the same message.options.retainedLoci, making the documented fail-closed default false at the layer it was declared on.Two findings were tagged PRE-EXISTING and are deliberately not fixed here; they are going out as their own backlog lines.
Note on history
Two commits in this branch's history are titled
park: UNVERIFIED, UNGRADED .... They were written by a coordinator to preserve work in progress after the worker fleet died mid-slice on an auth error. Everything in them has since been re-measured from scratch, verified and graded. Squash-merging collapses them.