fix(hl7): make policy safe-harbor actually Safe Harbor for the encounter loci

.changeset/safe-harbor-encounter-loci.md

@@ -0,0 +1,19 @@
+---
+"@cosyte/deid": patch
+---
+
+The `safe-harbor` policy now actually removes the encounter loci Safe Harbor requires it to remove. Seven identifying values previously survived a `safe-harbor` pass byte-identical, with no manifest entry at all: the visit number (PV1-19), the admit and discharge dates (PV1-44/45), the observation date (OBR-7), the diagnosis date (DG1-5), and the placer and filler order numbers (OBR-2/3, ORC-2/3). Retaining an HL7 segment retained every field inside it, and nothing carved these back out.
+
+45 CFR 164.514(b)(2)(i)(C) requires removal of all elements of dates except year that are directly related to an individual, and names admission and discharge dates in the regulation text itself; a visit or order number is a unique identifying code the (R) catch-all reaches. A policy named `safe-harbor` that returned them was a trap for anyone who trusted the name, so this is a deliberate breaking change while the package is pre-alpha.
+
+**What changes.** Under `SAFE_HARBOR_PROFILE` the four dates now generalize to their year and the five identifier loci are removed as category (R). Under `LIMITED_DATA_SET_PROFILE` all seven are kept unchanged, because 164.514(e)(2)'s limited-data-set exclusion list enumerates sixteen direct identifiers, contains no date, and has no catch-all. The split is expressed by two named retention classes, `encounter-dates` and `encounter-identifiers`, on the new `retainedLoci` field of a profile.
+
+**Retention takes three independent keys, and a missing one always means the transform runs.** The adapter must propose a class for the locus; the configured options must list that class, so an adapter can never retain anything by itself and an options bag that omits `retainedLoci` keeps nothing; and the resolved category must be one a limited data set may carry at all. That last key is the one that matters most in practice: `PV1-19` is a CX list, and a visit-number field routinely carries a medical record or account number typed as such by the standard's own CX-5 identifier-type code. Both are named by 164.514(e)(2), so both are now routed through the identifier-type code and transformed, and an `MR`-typed visit number gets the _same_ keyed surrogate as the matching PID-3 entry rather than being republished in the clear beside it. `LIMITED_DATA_SET_DIRECT_IDENTIFIERS` and `isRetainableCategory()` are exported so the rule is inspectable: exactly two of the eighteen categories are retainable.
+
+**A policy carrying the reserved `safe-harbor` label may not retain at all**, whatever the options bag says. That is a fatal `DEID_POLICY_INVALID`, the retention analogue of the guard that stops a date-shifting policy wearing the same label, and it closes the hand-built-options route no profile-level check can see.
+
+**Anything still retained is now recorded.** A kept locus emits a manifest entry with disposition `retained`, transform `retain`, and code `DEID_RESIDUAL_RETAINED`, so it reaches the Expert-Determination support report's residual inventory rather than being invisible in both artifacts. `DeidManifestEntry["disposition"]` and `ReportDisposition` gain `"retained"`, `DispositionSummary` gains a `retained` count, and each inventory row now carries its `transform` so a kept year is distinguishable from a kept full-precision timestamp.
+
+**`defineDeidProfile()`'s widen-never-narrow contract now covers retention, and it reads the opposite way round from a transform override:** dropping a retained class removes more and is allowed; adding one keeps more and is a fatal `DEID_PROFILE_INVALID`. It is a subset test, not a rank comparison.
+
+**Corrected claims.** The published limitations page claimed that loci absent from the parser models fail closed, which read as the opposite of the truth for a retained segment; that claim is deleted. In its place the page states the true class: every field of a retained segment that the carve-out does not name is still passed through and recorded nowhere, still including full-precision timestamps in EVN, PV2, PR1, RXA, RXD, FT1, TXA and SPM and the provider names in PV1-7/8 and OBR-16. The carve-out narrows that class; it does not close it. Note also that the retention classes are read by the HL7 v2 adapter only: passing a retention set to the other five adapters changes nothing there.

README.md

@@ -100,6 +100,7 @@ const { document, manifest } = deidentifyHl7(parseHL7(rawMessage), { context });
 document.toString(); // spec-clean, de-identified HL7 wire
 // PID-5 (name), NK1/GT1/IN1/IN2 relatives, SSN, phone → removed; MRN/account → consistent surrogate;
 // DOB → year; address → safe 3-digit ZIP. OBX-5/NTE free text and Z-segments fail closed (blocked).
+// Admit/discharge/observation/diagnosis dates → year; visit and order numbers blocked as (R).
 // Structured clinical OBX values, units, codes, and statuses survive untouched.
 ```
 
@@ -112,11 +113,19 @@ so a known patient-identity segment absent from the map (e.g. **MRG** prior name
 coded / date); narrative (`TX`/`FT`), ambiguous String (`ST`), and any empty/unknown OBX-2 fail closed,
 as do **NTE-3** comments. Structured clinical values, units, codes, and statuses survive untouched.
 
+**Inside a retained segment**, the identifying loci are carved back out: under a Safe-Harbor-labelled
+policy the admit (PV1-44), discharge (PV1-45), observation (OBR-7) and diagnosis (DG1-5) dates keep only
+their **year**, and the visit number (PV1-19) with the placer and filler order numbers (OBR-2/3, ORC-2/3)
+are **removed**. A profile that names their retention class, as the limited-data-set preset does, keeps
+them **unchanged and recorded**. PV1-19 is a CX list routed by its CX-5 identifier-type code, like PID-3:
+only a `VN`-typed or untyped visit number is the encounter identifier, while an `MR`/`AN`/`SS`-typed one
+is transformed as the medical record / account / social security number it is, under **both** profiles.
+
 **Known limitations (this release).** Free text is block-by-default (no built-in scrub; opt-in BYO
-redaction: see [Free text](#free-text-block-by-default--byo-redaction)); within **retained** clinical /
-visit segments, patient-related _dates_ (OBR/DG1/PV1 timestamps), _visit identifiers_ (PV1-19), and
-_provider_ names (PV1-7/8, OBR-16) are **not** de-identified; the address generalization keeps only the
-Safe Harbor 3-digit ZIP.
+redaction: see [Free text](#free-text-block-by-default--byo-redaction)); **every** field of a retained
+segment that the carve-out does not name is **not** de-identified and is recorded nowhere, which still
+includes full-precision timestamps in EVN, PV2, PR1, RXA, RXD, FT1, TXA and SPM and the _provider_ names
+in PV1-7/8 and OBR-16, among others; the address generalization keeps only the Safe Harbor 3-digit ZIP.
 
 ## De-identify a C-CDA document
 
@@ -371,7 +380,7 @@ Determination** (§164.514(b)(1), a qualified statistician's risk judgment). `@c
 the latter and **never renders** it. `buildExpertDeterminationSupportReport(manifest)` structures the
 value-free manifest into what an expert reasons about: per-locus dispositions, coverage across all 18
 categories, and the **retained-quasi-identifier inventory** (year-only dates, safe 3-digit ZIP prefixes,
-exact ages ≤ 89), then hands it over.
+exact ages ≤ 89, and any whole value a profile's retention set kept), then hands it over.
 
 ```ts
 import { buildExpertDeterminationSupportReport } from "@cosyte/deid";

docs-content/guides-expert-determination.md

@@ -80,12 +80,17 @@ report.totals.categoriesActedOn; // => 2
 report.retainedQuasiIdentifiers[0].locus; // => "PID-7"
 ```
 
-The **retained-quasi-identifier inventory** is the residual an expert cares about most: the coarse
-identifying elements the pass kept for utility and **recorded** as `DEID_RESIDUAL_RETAINED`, a year-only
-date, a safe 3-digit ZIP prefix, an exact age ≤ 89. These are the §164.514(b)(2)(ii) actual-knowledge
-considerations. (Clinical values retained untouched by the over-scrub guard are not identifiers and are
-not enumerated in the value-free manifest; consult each format's retained-segment notes for residual
-dates in retained clinical segments.)
+The **retained-quasi-identifier inventory** is the residual an expert cares about most: the identifying
+elements the pass kept for utility and **recorded** as `DEID_RESIDUAL_RETAINED`. Two kinds land there: a
+coarse residual left by a generalization (a year-only date, a safe 3-digit ZIP prefix, an exact age
+≤ 89), and a **whole unreduced value** a profile's retention set kept, such as the admission, discharge
+and service dates and the encounter and order numbers a limited-data-set preset carries. The second kind
+is the stronger residual, and it is inventoried here rather than left to a footnote. These are the
+§164.514(b)(2)(ii) actual-knowledge considerations.
+
+Clinical values retained untouched by the over-scrub guard are not identifiers and are not enumerated.
+What is enumerated **nowhere** is a field inside a retained structure that no locus map reaches; those
+are named per format in the published limitations, and this report cannot see them.
 
 ## The optional k-anonymity indicator: caller-supplied, descriptive only
 

docs-content/guides-hl7.md

@@ -30,8 +30,8 @@ import { createDeidContext } from "@cosyte/deid";
 const context = createDeidContext({ key: process.env.DEID_KEY! });
 const { document, manifest } = deidentifyHl7(parseHL7(rawMessage), { context });
 
-document.toString();  // spec-clean, de-identified HL7 wire
-manifest;             // value-free audit: category + locus + disposition, never a value
+document.toString(); // spec-clean, de-identified HL7 wire
+manifest; // value-free audit: category + locus + disposition, never a value
 ```
 
 A keyed transform (MRN / account / beneficiary pseudonymization) requires a `context`; calling without
@@ -40,14 +40,16 @@ surrogate.
 
 ## What is located, and how it is transformed
 
-| Segment | Loci | Transform |
-|---|---|---|
-| **PID** | name (5/6/9), DOB (7/29), address (11), SSN (19), phone (13/14), driver's licence (20), MRN/account/mother-id (2/3/4/18/21), county (12), birth place (23) | names/phone/SSN/licence **removed**; MRN/account → consistent **surrogate** (keyed HMAC); DOB → **year**; ZIP → safe **3-digit** (or `000`); county/birth place fail closed |
-| **NK1 / GT1 / IN1 / IN2** | relatives / guarantor / insured names, addresses, phones, SSNs, DOBs, member/policy/Medicare/Medicaid ids | same category transforms: Safe Harbor removes identifiers of **relatives, employers, and household members**, not only the patient |
-| **OBX-5, NTE-3** | narrative / ambiguous free text (OBX-5 unless OBX-2 types it structured) | **fail closed**: blocked, never regex-scrubbed |
-| **MRG / ACC / FAM / PEO / PDA** | known patient-identity / relative / geographic segments absent from the map | **fail closed**: blocked (e.g. a merge message's prior name + MRN) |
-| **Z-segments / unknown structure** | every populated field | **fail closed**: blocked |
-| Retained clinical/administrative segments (an explicit allow-list: OBR, ORC, AL1, DG1, PV1, RX*, …) | n/a | **retained untouched** (the over-scrub guard) |
+| Segment                                                                                              | Loci                                                                                                                                                       | Transform                                                                                                                                                                   |
+| ---------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| **PID**                                                                                              | name (5/6/9), DOB (7/29), address (11), SSN (19), phone (13/14), driver's licence (20), MRN/account/mother-id (2/3/4/18/21), county (12), birth place (23) | names/phone/SSN/licence **removed**; MRN/account → consistent **surrogate** (keyed HMAC); DOB → **year**; ZIP → safe **3-digit** (or `000`); county/birth place fail closed |
+| **NK1 / GT1 / IN1 / IN2**                                                                            | relatives / guarantor / insured names, addresses, phones, SSNs, DOBs, member/policy/Medicare/Medicaid ids                                                  | same category transforms: Safe Harbor removes identifiers of **relatives, employers, and household members**, not only the patient                                          |
+| **OBX-5, NTE-3**                                                                                     | narrative / ambiguous free text (OBX-5 unless OBX-2 types it structured)                                                                                   | **fail closed**: blocked, never regex-scrubbed                                                                                                                              |
+| **MRG / ACC / FAM / PEO / PDA**                                                                      | known patient-identity / relative / geographic segments absent from the map                                                                                | **fail closed**: blocked (e.g. a merge message's prior name + MRN)                                                                                                          |
+| **Z-segments / unknown structure**                                                                   | every populated field                                                                                                                                      | **fail closed**: blocked                                                                                                                                                    |
+| **PV1-19, OBR-2/3, ORC-2/3** | visit number, placer + filler order numbers, inside retained segments | **removed**. PV1-19 is routed by its CX-5 type code like PID-3: `VN`/untyped is the encounter identifier, removed as (R) and retainable under a profile that names the class; `MR`/`AN`/`SS` is transformed as that identifier under **both** profiles, never retained |
+| **PV1-44/45, OBR-7, DG1-5**                                                                          | admit, discharge, observation and diagnosis dates, inside retained segments                                                                                | → **year** (§164.514(b)(2)(i)(C) names admission and discharge); kept whole, and recorded, only under a profile that names the class                                        |
+| Retained clinical/administrative segments (an explicit allow-list: OBR, ORC, AL1, DG1, PV1, RX\*, …) | every field except the two rows above                                                                                                                      | **retained untouched** (the over-scrub guard)                                                                                                                               |
 
 A recognized segment is retained **only** if it is on the explicit retain-list; anything else fails
 closed. OBX-5 is retained only when OBX-2 positively types it as a structured clinical value (numeric,
@@ -59,18 +61,21 @@ handled differently, structurally, from the parser's typing.
 
 ## The two guarantees
 
-- **No leak.** Every seeded PHI sentinel across PID/NK1/GT1/IN1/IN2, the free-text loci, and Z-segments
-  is gone from the serialized output. An unmapped locus that could carry PHI is blocked, never passed
-  through in the clear.
+- **No leak.** Every seeded PHI sentinel across PID/NK1/GT1/IN1/IN2, the encounter dates and order
+  identifiers, the free-text loci, and Z-segments is gone from the serialized output under the Safe
+  Harbor profile. An unmapped locus that could carry PHI is blocked, never passed through in the clear.
 - **No over-scrub.** Structured clinical OBX values, units, LOINC/coded observation identifiers,
   reference ranges, and result statuses are retained byte-identical: the de-identifier never degenerates
   into a blanket-blanking "safe but useless" scrubber.
 
 ## Known limitations (this release)
 
 - Free text is **block-only**: there is no built-in NLP scrub.
-- Within **retained** clinical / visit segments, patient-related **dates** (OBR / DG1 / PV1 timestamps),
-  **visit identifiers** (PV1-19), and **provider** names (PV1-7/8, OBR-16) are **not** de-identified.
+- Within **retained** clinical / visit segments, **every** field the carve-out above does not name is
+  **not** de-identified and is **not recorded**. That still includes full-precision timestamps in EVN,
+  PV2, PR1, RXA, RXD, FT1, TXA and SPM, and the **provider** names in PV1-7/8 and OBR-16, among others.
+  Retaining a segment is not auditing every field in it, and the carve-out narrows this class rather
+  than closing it.
 - The address generalization keeps only the Safe Harbor 3-digit ZIP (the permitted state is also
   dropped, conservative, never a leak).