From 169d517da5a9c34ae2c5e5f2502783cd48f8368a Mon Sep 17 00:00:00 2001 From: "Abdulaziz M. Shehri" Date: Sat, 15 Aug 2026 03:25:43 +0300 Subject: [PATCH 01/82] docs(cf-10): define public real-IG delta corpus --- specs/010-cf-10-real-ig-delta-corpus/spec.md | 281 +++++++++++++++++++ 1 file changed, 281 insertions(+) create mode 100644 specs/010-cf-10-real-ig-delta-corpus/spec.md diff --git a/specs/010-cf-10-real-ig-delta-corpus/spec.md b/specs/010-cf-10-real-ig-delta-corpus/spec.md new file mode 100644 index 00000000..23e2608e --- /dev/null +++ b/specs/010-cf-10-real-ig-delta-corpus/spec.md @@ -0,0 +1,281 @@ +# CF-10 — Public Real-IG Delta Corpus + +Status: planned / selection rules frozen before digest discovery + +## Purpose + +CF-10 establishes a small, reproducible, public corpus of **real published FHIR R4 Implementation Guide version deltas** for commandF evaluation. + +The corpus is benchmark evidence, not a new compatibility authority. It reuses canonical commandF capabilities to measure real upstream change: + +- CF-01 package resolution / digest verification; +- CF-03 deterministic structural diff; +- CF-04 compatibility classification; +- CF-07 terminology evidence; +- CF-06 pinned HL7 oracle evidence where comparable StructureDefinitions changed. + +CF-10 MUST NOT change the semantics of those slices to improve corpus results. + +## Anti-cherry-picking rule + +Selection criteria are frozen **before** package digests or commandF results are collected. + +A case is eligible only when all of the following are true: + +1. the IG has at least two stable, published, permanent versions; +2. both selected package versions are FHIR R4 / 4.0.1 compatible with the current commandF surface; +3. the upstream publication provides an explicit package identity and public version/change evidence; +4. the selected versions are ordinary published releases, not CI builds, nightly builds, ballots, snapshots, or local forks; +5. the package can be resolved from the public FHIR package ecosystem by commandF's existing resolver; +6. the case contains no PHI and requires no private dataset access; +7. commandF results are **not** used to decide whether the case remains in the corpus; +8. divergence, unsupported content, oracle disagreement, or zero findings are evidence and MUST NOT cause silent case removal. + +Once a case passes this gate, later unfavorable output cannot remove it from corpus v1 except for a documented upstream-rights/access failure or a proven mistaken eligibility fact. + +## Corpus v1 families + +The initial frozen candidate set deliberately spans three distinct interoperability contexts: + +### C001 — US Core annual national-core delta + +```text +package: hl7.fhir.us.core +before: 8.0.1 +before publication: https://hl7.org/fhir/us/core/STU8.0.1/ +after: 9.0.0 +after publication: https://hl7.org/fhir/us/core/STU9/ +FHIR: 4.0.1 +realm/context: US national/base core +``` + +Rationale: US Core is an annually revised base IG and publishes explicit cross-version/change guidance. The pair is adjacent published annual releases and is not chosen from commandF output. + +### C002 — International Patient Summary major-generation delta + +```text +package: hl7.fhir.uv.ips +before: 1.1.0 +before publication: https://hl7.org/fhir/uv/ips/STU1.1/ +after: 2.0.1 +after publication: https://hl7.org/fhir/uv/ips/en/ +FHIR: 4.0.1 +realm/context: international patient summary +``` + +Rationale: IPS 2 documents explicit non-compatible and compatible substantive changes relative to the previous generation. This gives the corpus an international, terminology-rich document/profile family rather than another US-only base guide. + +### C003 — mCODE specialty-oncology major delta + +```text +package: hl7.fhir.us.mcode +before: 3.0.0 +before publication: https://hl7.org/fhir/us/mcode/STU3/ +after: 4.0.0 +after publication: https://hl7.org/fhir/us/mcode/STU4/ +FHIR: 4.0.1 +realm/context: oncology specialty IG +``` + +Rationale: mCODE publishes stable R4 generations and has explicit version-differential/change material. It exercises specialty profiles and terminology rather than general core/document-only content. + +## Rights and redistribution boundary + +CF-10 is **metadata-only in the repository**. + +The repository MUST NOT vendor, commit, mirror, repackage, or redistribute the selected IG NPM archives, examples, terminology expansions, SNOMED CT content, RxNorm content, ISO content, or other upstream copyrighted/licensed payloads. + +The corpus manifest stores only: + +- package/version identity; +- commandF-observed content digest after public resolution; +- publication/provenance URLs; +- upstream rights/IP evidence URLs and a conservative rights note; +- deterministic commandF result metadata/hashes that contain no redistributed source package bytes. + +Package bytes are ephemeral runtime inputs acquired through the existing public package resolver and verified against the locked digest. Mixed upstream terminology/IP statements remain upstream obligations; commandF's repository license MUST NOT be interpreted as relicensing them. + +## Digest discovery and freeze rule + +The package SHA-256 values are not copied from web pages or manually invented. + +For each frozen case, an isolated discovery gate MUST: + +1. resolve the exact package version through commandF CF-01; +2. record the resolver-produced content SHA-256 and byte size; +3. independently resolve the same exact version into a second clean cache; +4. require identical digest and archive bytes between the two resolutions; +5. verify the package through CF-01 cache verification; +6. persist only the digest/size metadata into the corpus manifest. + +After the digest is frozen, ordinary corpus runs MUST fail closed if a public resolution produces different bytes for the same package/version. + +## Manifest contract + +Canonical v1 manifest path: + +```text +corpus/real-ig/v1/corpus.json +``` + +Schema-v1 logical shape: + +```text +schema +selection_policy +cases[] + id + package + before + version + archive_sha256 + archive_bytes + publication_url + after + version + archive_sha256 + archive_bytes + publication_url + fhir_version + publisher + change_evidence_url + rights_evidence_url + rights_mode + oracle_mode +``` + +Rules: + +- case ids are unique and stable; +- package names/versions are exact, never ranges/wildcards; +- archive digests are lowercase SHA-256 hex; +- archive sizes are positive and bounded; +- `fhir_version` is exactly `4.0.1` in CF-10 v1; +- publication/change/rights evidence is explicit per case; +- `rights_mode` for v1 is `metadata_only_no_redistribution`; +- `oracle_mode` is `changed_structure_definitions_only`; +- unknown schema versions fail closed; +- malformed/duplicate/unsorted cases fail closed. + +## Runner contract + +CF-10 MAY introduce a thin deterministic corpus orchestrator, but it MUST call existing commandF authorities rather than reimplement their semantics. + +A corpus run for one case performs: + +1. exact before/after package resolution into isolated caches; +2. digest and archive-size attestation against the frozen manifest; +3. CF-01 cache verification; +4. CF-03 structural diff; +5. CF-04 classification; +6. CF-07 terminology diff; +7. CF-06 oracle comparison for changed comparable StructureDefinition pairs using the pinned local adapter; +8. deterministic case-summary emission. + +A case failure is represented explicitly. The runner MUST NOT skip a selected case because of unsupported content, a commandF/oracle disagreement, a network/package failure, or an unexpected finding count. + +## Result contract + +CF-10 produces commandF-owned deterministic summary evidence only. It does not store upstream IG artifacts. + +Per-case result must identify at least: + +```text +case_id +package +before_version / before_sha256 +after_version / after_sha256 +structural_change_count +compatibility_finding_count +compatibility severity/direction aggregate counts +terminology aggregate counts +oracle compared/agreement/divergence aggregate counts +case_status +``` + +`case_status` is operational/evidence state only and MUST NOT rewrite CF-04/05 compatibility truth. + +V1 summary output MUST be deterministic for the same manifest, pinned packages, commandF build, and pinned oracle. + +## No golden-answer fabrication + +Before the first real corpus execution, CF-10 MUST NOT hard-code expected structural counts, compatibility severities, terminology counts, oracle agreement counts, or a required "good" outcome. + +After the first independently verified run, exact result hashes/counts MAY be frozen as regression evidence only if: + +- the raw commandF outputs are preserved in CI artifacts for review; +- a second clean run is byte-identical; +- the expected values are derived from the observed deterministic run, not edited to make tests pass; +- future differences fail visibly and require explicit corpus-version reconciliation. + +## Failure semantics + +CF-10 fails closed on: + +- manifest schema/shape violations; +- duplicate or non-canonical case ordering; +- package resolution failure; +- package/version/digest/size mismatch; +- cache verification failure; +- unsupported non-R4 case in v1; +- missing rights/change/publication evidence metadata; +- malformed commandF sub-report; +- oracle operational failure for a case configured to run it; +- non-deterministic repeated result bytes. + +Failures are never silently converted to corpus exclusion or pass. + +## Determinism and reproducibility + +The corpus must not serialize: + +- timestamps; +- host-absolute paths; +- random ids; +- network timing; +- temporary cache paths; +- unordered map iteration. + +Case order is lexicographic by stable case id. Aggregate maps are canonically ordered. + +## Security boundary + +Corpus metadata is untrusted input. The implementation must bound manifest/result sizes and counts, reject path-like package/version tricks where a package identity is expected, use explicit subprocess paths for the oracle boundary, and inherit CF-01/CF-06 process/cache hardening rather than bypass it. + +No corpus field becomes shell code. CI uses quoted argv and no `eval`. + +## Acceptance gates + +A converged CF-10 candidate requires: + +- frozen selection methodology before result discovery; +- exact package digest discovery using two independent clean resolutions; +- provenance/rights evidence for every case; +- schema and failure-path tests; +- deterministic two-run corpus equality; +- `cargo fmt --all -- --check`; +- locked workspace Clippy with `-D warnings`; +- full workspace tests; +- real public corpus CI on all frozen v1 cases; +- existing CF-08/CF-09 security regressions remain green; +- existing real FHIR / terminology gates remain green; +- `cf06-oracle` remains green; +- CodeRabbit/Qodo/other reviewer findings verified and dispositioned without inventing unavailable PASS states. + +## Explicit deferrals + +CF-10 does not add: + +- new compatibility rules or policy thresholds; +- a new terminology semantic authority; +- a replacement HL7 oracle; +- private/credentialed datasets; +- PHI; +- MIMIC-derived data; +- ballot/nightly IG stratum; +- cross-model FHIR↔openEHR/OMOP benchmark; +- server behavior benchmark; +- ecosystem blast-radius graph; +- baselines/suppressions; +- AutoFix; +- AI/agent semantic authority. From 1407b9c6c5fcf629e12641519bae9d56d5a5fcee Mon Sep 17 00:00:00 2001 From: "Abdulaziz M. Shehri" Date: Sat, 15 Aug 2026 03:26:23 +0300 Subject: [PATCH 02/82] docs(cf-10): plan corpus discovery and execution --- specs/010-cf-10-real-ig-delta-corpus/plan.md | 271 +++++++++++++++++++ 1 file changed, 271 insertions(+) create mode 100644 specs/010-cf-10-real-ig-delta-corpus/plan.md diff --git a/specs/010-cf-10-real-ig-delta-corpus/plan.md b/specs/010-cf-10-real-ig-delta-corpus/plan.md new file mode 100644 index 00000000..b205c89b --- /dev/null +++ b/specs/010-cf-10-real-ig-delta-corpus/plan.md @@ -0,0 +1,271 @@ +# CF-10 Implementation Plan — Public Real-IG Delta Corpus + +Status: planned / selection rules frozen + +## Architecture decision + +CF-10 is a benchmark/evidence layer above existing commandF authorities. It does not add a new semantic engine. + +The implementation is split into four boundaries: + +1. **Corpus manifest model** — validates frozen case metadata, digests, rights/provenance evidence, ordering, and v1 R4 constraints. +2. **Acquisition/attestation** — uses CF-01 resolver/cache verification in isolated state; package bytes are ephemeral and never committed. +3. **Case evaluation** — invokes the existing CF-03/04/07 evidence stack and CF-06 oracle adapter without modifying their rules. +4. **Deterministic summary** — emits compact commandF-owned benchmark evidence and explicit operational failure states. + +The first executable gate is digest discovery. Result counts/severities are deliberately unknown until after the cases and package identities are frozen. + +## Phase A — Freeze selection and provenance + +Canonical v1 cases are fixed before result discovery: + +```text +C001 hl7.fhir.us.core 8.0.1 -> 9.0.0 +C002 hl7.fhir.uv.ips 1.1.0 -> 2.0.1 +C003 hl7.fhir.us.mcode 3.0.0 -> 4.0.0 +``` + +Selection is based on published/stable R4 status, public package identity, explicit version/change evidence, and domain diversity. commandF output is not a selection input. + +The donor/provenance record must distinguish: + +- publication metadata rights; +- package/runtime access; +- terminology/IP statements inside the IG; +- repository redistribution rights. + +CF-10 repository mode is metadata-only. No upstream NPM tarball is checked in. + +## Phase B — Digest discovery + +Add a one-shot/guarded GitHub Actions discovery workflow or equivalent isolated execution tool that: + +1. checks out the exact CF-10 planning head; +2. resolves every selected exact package version into cache A; +3. verifies cache A; +4. resolves the same package/version into independent cache B; +5. verifies cache B; +6. requires archive bytes and SHA-256 to match A == B; +7. records byte size and digest in a machine-readable artifact; +8. fails if any selected version is unavailable or non-reproducible. + +The discovery artifact is reviewed before digest metadata is copied into the canonical corpus manifest. + +No expected commandF semantic result is generated or frozen in this phase. + +## Phase C — Manifest model + +Preferred ownership: `commandf-pkg` typed model and validation, not ad-hoc shell parsing. + +Proposed modules: + +```text +crates/commandf-pkg/src/corpus_model.rs +crates/commandf-pkg/src/corpus_error.rs +crates/commandf-pkg/src/corpus.rs +``` + +Public model concepts: + +```text +RealIgCorpus +CorpusSelectionPolicy +RealIgCase +CorpusPackageState +CorpusRightsEvidence +CorpusOracleMode +``` + +Validation requirements: + +- schema exactly 1; +- bounded input bytes and case count; +- stable unique case ids; +- canonical lexicographic case ordering; +- exact package names/versions; +- R4 `4.0.1` only in v1; +- SHA-256 lowercase/64-hex; +- positive bounded archive size; +- before != after version; +- publication/change/rights evidence URLs required; +- `metadata_only_no_redistribution` only in v1; +- `changed_structure_definitions_only` oracle mode only in v1; +- unknown enum/schema values fail closed. + +The model does not fetch the network. + +## Phase D — Deterministic evaluator + +Add a thin evaluator that consumes an already validated manifest and explicit package state. + +Do not introduce another matcher/classifier/terminology engine/oracle implementation. + +For each case: + +1. require attested before/after archive bytes matching manifest digest/size; +2. call `diff_package_archives`; +3. call canonical compatibility classification; +4. call canonical terminology diff against the same verified states; +5. call the CF-06 oracle path only for changed matched StructureDefinitions; +6. reduce those reports into deterministic aggregate counts without losing the underlying report identities/hashes. + +Raw detailed reports remain CI artifacts; the committed/public corpus summary remains compact commandF-owned evidence. + +## Phase E — User-visible execution surface + +Prefer one narrow CLI rather than benchmark shell glue: + +```text +commandf corpus run \ + --manifest corpus/real-ig/v1/corpus.json \ + --work-root \ + --oracle-adapter \ + --oracle-java \ + --format json +``` + +The command may perform acquisition because corpus execution is explicitly an integration/benchmark operation, but acquisition and evaluation remain internally separated so digest verification happens before semantic analysis. + +Alternative if implementation pressure reveals the CLI is unnecessary: keep the typed evaluator public in `commandf-pkg` and use a repository-owned executable harness. This decision must be made before implementation and reflected in `spec.md`; do not silently create both. + +## Phase F — Summary schema + +Proposed deterministic v1 summary: + +```text +schema +manifest_sha256 +cases[] + case_id + package + before { version, sha256 } + after { version, sha256 } + structural + changes + compatibility + findings + breaking + risky + additive + producer + consumer + both + terminology + code_system_changes + value_set_changes + binding_refinements + oracle + compared + agreement + commandf_only + authority_only + both_changed + uncomparable + status +``` + +Exact field vocabulary must reuse existing public enum names where possible rather than invent parallel terms. + +The summary does not claim clinical safety, semantic equivalence, or universal benchmark coverage. + +## Phase G — First-result freeze + +After implementation and digest lock: + +1. run the entire v1 corpus from a clean environment; +2. preserve raw reports as CI artifact; +3. repeat from another clean cache; +4. require byte-identical deterministic summary; +5. review disagreements/unsupported evidence as findings, not failures to hide; +6. only then freeze exact summary digest/counts as a regression baseline in corpus v1 metadata or a separate expected-results file. + +Any future upstream-package byte change at the same version is a hard attestation failure, not an automatic baseline update. + +## CI layout + +Preserve current `ci` and `cf06-oracle` workflows. + +Add a dedicated corpus job/workflow with explicit resource/time bounds because three public IG pairs plus the Java oracle are heavier than ordinary unit tests. + +Recommended stages: + +```text +manifest-validation +package-attestation +structural-classification-terminology +oracle-evidence +repeat-determinism +summary-verification +``` + +CI must upload raw reports and the deterministic summary as short-retention review artifacts. Artifacts are evidence, not repository-vendored benchmark data. + +## Test strategy + +### Manifest tests + +- wrong schema; +- empty corpus; +- too many cases; +- duplicate/out-of-order ids; +- malformed package/version; +- same before/after version; +- non-R4 FHIR version; +- malformed digest/size; +- missing publication/change/rights metadata; +- unsupported rights/oracle mode; +- deterministic JSON round trip. + +### Attestation tests + +Use synthetic/local package fixtures for unit tests: + +- matching digest/size passes; +- digest mismatch fails; +- size mismatch fails; +- corrupted cache fails; +- before/after state cannot alias unexpectedly. + +### Evaluation tests + +Use small synthetic packages to verify aggregation equals canonical underlying reports. Do not make ordinary unit tests depend on public network or Java/Maven. + +### Real integration tests + +The dedicated CF-10 job uses the three frozen public cases and the pinned CF-06 adapter. + +## Rights controls + +No package bytes enter git history. + +The corpus manifest records upstream legal/IP evidence but does not collapse mixed terminology rights into a single permissive license label. In particular, IPS contains terminology with separate upstream licensing statements; metadata-only benchmarking avoids redistribution and does not grant downstream terminology rights. + +Any future proposal to publish/download-bundle the source packages as a commandF benchmark dataset is a separate founder/legal authorization gate and is not CF-10. + +## Reviewer priorities + +1. no benchmark cherry-picking after result discovery; +2. no semantic leakage from benchmark expectations into CF-03/04/07/06; +3. no package-content redistribution; +4. exact digest attestation before analysis; +5. no silent skipped cases; +6. deterministic result bytes; +7. bounded untrusted manifest/package metadata; +8. rights evidence remains conservative and source-specific; +9. real integration failures remain visible evidence; +10. no golden expected result authored before first verified run. + +## Convergence condition + +CF-10 converges only after: + +- selection policy and case set remain unchanged from the pre-result spec unless an eligibility/right fact is proven false; +- exact digests/sizes are independently discovered and frozen; +- all three cases resolve and attest; +- typed manifest/evaluator tests pass; +- real corpus run completes without silent case removal; +- two clean runs produce byte-identical summary; +- current `ci` and `cf06-oracle` remain green; +- dedicated corpus workflow is green on the exact final head; +- substantive reviewer findings are fixed or explicitly rejected with evidence; +- `spec.md`, `plan.md`, `tasks.md`, donor record, corpus manifest, and `convergence.md` agree on the final truth. From 0b66579fed9d821b03682cd6f2465fd51cc1a23c Mon Sep 17 00:00:00 2001 From: "Abdulaziz M. Shehri" Date: Sat, 15 Aug 2026 03:26:41 +0300 Subject: [PATCH 03/82] docs(cf-10): define corpus execution tasks --- specs/010-cf-10-real-ig-delta-corpus/tasks.md | 76 +++++++++++++++++++ 1 file changed, 76 insertions(+) create mode 100644 specs/010-cf-10-real-ig-delta-corpus/tasks.md diff --git a/specs/010-cf-10-real-ig-delta-corpus/tasks.md b/specs/010-cf-10-real-ig-delta-corpus/tasks.md new file mode 100644 index 00000000..0a3fb6fb --- /dev/null +++ b/specs/010-cf-10-real-ig-delta-corpus/tasks.md @@ -0,0 +1,76 @@ +# CF-10 Tasks — Public Real-IG Delta Corpus + +Status: planned / selection rules frozen + +## Selection and provenance + +- [x] T001 Freeze anti-cherry-picking eligibility rules before result discovery. +- [x] T002 Freeze corpus v1 families: US Core, IPS, mCODE. +- [x] T003 Freeze exact candidate version pairs: US Core 8.0.1→9.0.0, IPS 1.1.0→2.0.1, mCODE 3.0.0→4.0.0. +- [ ] T004 Record publication/change/rights evidence for every case in donor/provenance metadata. +- [ ] T005 Confirm repository mode is metadata-only and no package/terminology payload is committed. + +## Digest discovery + +- [ ] T006 Add guarded exact-version digest-discovery workflow/tool. +- [ ] T007 Resolve every selected package/version into independent cache A and cache B. +- [ ] T008 Verify both caches with CF-01. +- [ ] T009 Require byte/digest equality between independent resolutions. +- [ ] T010 Record package archive SHA-256 and byte size in discovery artifact. +- [ ] T011 Review discovery artifact and freeze digests/sizes in canonical corpus manifest. + +## Manifest implementation + +- [ ] T012 Add typed corpus schema-v1 model. +- [ ] T013 Add bounded pre-decode manifest input validation. +- [ ] T014 Validate unique lexicographically ordered case ids. +- [ ] T015 Validate exact package/version, R4 4.0.1, SHA-256, positive archive size, rights and publication evidence. +- [ ] T016 Reject unsupported schema/rights/oracle modes and malformed evidence. +- [ ] T017 Add deterministic JSON round-trip tests. + +## Package attestation + +- [ ] T018 Add reusable before/after archive attestation against manifest digest and size. +- [ ] T019 Add digest mismatch and size mismatch fail-closed tests. +- [ ] T020 Preserve CF-01 cache verification as mandatory authority. + +## Corpus evaluator + +- [ ] T021 Reuse CF-03 structural diff for every case. +- [ ] T022 Reuse CF-04 compatibility classification for every case. +- [ ] T023 Reuse CF-07 terminology evidence for every case. +- [ ] T024 Reuse CF-06 oracle only for changed matched StructureDefinitions. +- [ ] T025 Add deterministic per-case aggregate summary without changing underlying semantic rules. +- [ ] T026 Represent operational/unsupported/divergence states explicitly; never silently skip a selected case. + +## Execution surface + +- [ ] T027 Decide and document one execution surface: narrow `commandf corpus run` CLI or repository-owned typed harness. +- [ ] T028 Implement only the selected surface. +- [ ] T029 Keep acquisition and semantic evaluation internally separated by digest verification. + +## Real corpus evidence + +- [ ] T030 Run all three frozen cases from a clean environment. +- [ ] T031 Upload raw structural/compatibility/terminology/oracle reports as short-retention CI artifacts. +- [ ] T032 Run the full corpus a second time from a clean cache. +- [ ] T033 Require byte-identical deterministic summary between the two runs. +- [ ] T034 Freeze exact result digest/count regression evidence only after the two-run proof. + +## CI and security + +- [ ] T035 Preserve current `ci` workflow gates. +- [ ] T036 Preserve current `cf06-oracle` workflow gates. +- [ ] T037 Add bounded dedicated real-corpus workflow/job. +- [ ] T038 Keep quoted argv/no `eval`; corpus metadata must never become shell code. +- [ ] T039 Bound manifest bytes/case count/result bytes and fail closed on malformed input. +- [ ] T040 Prove no upstream IG NPM tarball or terminology payload enters git history. + +## Review and convergence + +- [ ] T041 Request CodeRabbit review when available and disposition substantive findings. +- [ ] T042 Request Qodo review when available and disposition substantive findings. +- [ ] T043 Request configured independent/Codex review when available; no unavailable PASS may be invented. +- [ ] T044 Reconcile `spec.md`, `plan.md`, `tasks.md`, donor record, manifest, and implementation truth. +- [ ] T045 Add `convergence.md` with exact final-head CI/reviewer evidence rule. +- [ ] T046 Keep PR open/unmerged until exact final head passes ordinary CI, oracle CI, and dedicated corpus CI. From 209a7ac9e80b0d685bda9120f6845ef8bf06e273 Mon Sep 17 00:00:00 2001 From: "Abdulaziz M. Shehri" Date: Sat, 15 Aug 2026 03:27:12 +0300 Subject: [PATCH 04/82] docs(cf-10): record real-IG corpus provenance boundary --- donors/cf-10-real-ig-delta-corpus.yaml | 95 ++++++++++++++++++++++++++ 1 file changed, 95 insertions(+) create mode 100644 donors/cf-10-real-ig-delta-corpus.yaml diff --git a/donors/cf-10-real-ig-delta-corpus.yaml b/donors/cf-10-real-ig-delta-corpus.yaml new file mode 100644 index 00000000..8f617b40 --- /dev/null +++ b/donors/cf-10-real-ig-delta-corpus.yaml @@ -0,0 +1,95 @@ +schema: commandf.donor-manifest/v1 +updated: 2026-08-15 + +sources: + - id: us-core-real-delta + project: HL7 US Core Implementation Guide + package: hl7.fhir.us.core + before_version: 8.0.1 + after_version: 9.0.0 + before_publication: https://hl7.org/fhir/us/core/STU8.0.1/ + after_publication: https://hl7.org/fhir/us/core/STU9/ + change_evidence: https://hl7.org/fhir/us/core/STU9/changes.html + rights_evidence: + - https://hl7.org/fhir/us/core/STU8.0.1/ImplementationGuide-hl7.fhir.us.core.html + - https://hl7.org/fhir/us/core/STU9/ImplementationGuide-hl7.fhir.us.core.html + fhir_version: 4.0.1 + mode: [PUBLIC_RUNTIME_INPUT, BENCHMARK_METADATA] + redistribution: NONE + rights_note: >- + Publication metadata identifies HL7 copyright/legal Creative Commons terms. CF-10 does not + vendor or redistribute the NPM package or its terminology/content; package bytes are ephemeral + public runtime inputs and retain all upstream rights/terms. + adopted_patterns: + - published exact package/version identity + - public change-history evidence + - runtime digest attestation through commandF + exclusions: + - no NPM archive copied into the repository + - no examples or terminology expansions copied into the repository + - no ballot/CI build selected for corpus v1 + + - id: ips-real-delta + project: HL7 International Patient Summary Implementation Guide + package: hl7.fhir.uv.ips + before_version: 1.1.0 + after_version: 2.0.1 + before_publication: https://hl7.org/fhir/uv/ips/STU1.1/ + after_publication: https://hl7.org/fhir/uv/ips/en/ + change_evidence: https://hl7.org/fhir/uv/ips/en/changes.html + rights_evidence: + - https://hl7.org/fhir/uv/ips/STU1.1/terminology.html + - https://hl7.org/fhir/uv/ips/en/terminology.html + fhir_version: 4.0.1 + mode: [PUBLIC_RUNTIME_INPUT, BENCHMARK_METADATA] + redistribution: NONE + rights_note: >- + IPS includes mixed upstream IP/terminology statements, including SNOMED CT and other code-system + rights. CF-10 intentionally records metadata only and does not redistribute package content, + terminology expansions, examples, or imply that commandF's repository license grants rights to + those upstream materials. + adopted_patterns: + - published exact package/version identity + - explicit upstream non-compatible/compatible change evidence + - runtime digest attestation through commandF + exclusions: + - no NPM archive copied into the repository + - no SNOMED CT/RxNorm/ISO/EDQM payload copied into the repository + - no ballot/continuous-build package selected for corpus v1 + + - id: mcode-real-delta + project: HL7 minimal Common Oncology Data Elements (mCODE) Implementation Guide + package: hl7.fhir.us.mcode + before_version: 3.0.0 + after_version: 4.0.0 + before_publication: https://hl7.org/fhir/us/mcode/STU3/ + after_publication: https://hl7.org/fhir/us/mcode/STU4/ + change_evidence: https://hl7.org/fhir/us/mcode/STU4/changes.html + rights_evidence: + - https://hl7.org/fhir/us/mcode/STU3/downloads.html + - https://hl7.org/fhir/us/mcode/STU4/downloads.html + - https://hl7.org/fhir/us/mcode/STU4/terminology.html + fhir_version: 4.0.1 + mode: [PUBLIC_RUNTIME_INPUT, BENCHMARK_METADATA] + redistribution: NONE + rights_note: >- + The publication records HL7/Creative Commons-publication terms while individual terminology + artifacts may carry separate upstream terms such as SNOMED CT. CF-10 does not redistribute + package or terminology content and retains those upstream obligations. + adopted_patterns: + - published exact package/version identity + - specialty-oncology version-delta evidence + - runtime digest attestation through commandF + exclusions: + - no NPM archive copied into the repository + - no terminology expansion copied into the repository + - no license simplification from mixed artifact-level IP statements + +rules: + - corpus v1 selection criteria and version pairs are frozen before commandF result discovery + - selected cases cannot be removed because commandF reports an unfavorable or divergent result + - exact package digests and byte sizes must come from two independent clean commandF resolutions + - digest metadata is adopted only after byte-for-byte equality and CF-01 verification + - package bytes remain ephemeral and never enter git history + - repository license must not be presented as relicensing upstream IG or terminology content + - a rights/access failure may block a case but must be documented rather than silently substituted From 8742f5f980353866faefd96067e3865b6abf69c5 Mon Sep 17 00:00:00 2001 From: "Abdulaziz M. Shehri" Date: Sat, 15 Aug 2026 03:27:43 +0300 Subject: [PATCH 05/82] ci(cf-10): discover reproducible public package digests --- .github/workflows/cf10-digest-discovery.yml | 126 ++++++++++++++++++++ 1 file changed, 126 insertions(+) create mode 100644 .github/workflows/cf10-digest-discovery.yml diff --git a/.github/workflows/cf10-digest-discovery.yml b/.github/workflows/cf10-digest-discovery.yml new file mode 100644 index 00000000..982c24ca --- /dev/null +++ b/.github/workflows/cf10-digest-discovery.yml @@ -0,0 +1,126 @@ +name: cf10-digest-discovery + +on: + push: + branches: + - feat/cf-10-real-ig-delta-corpus + paths: + - .github/workflows/cf10-digest-discovery.yml + - specs/010-cf-10-real-ig-delta-corpus/** + - donors/cf-10-real-ig-delta-corpus.yaml + workflow_dispatch: + +permissions: + contents: read + +jobs: + discover: + runs-on: ubuntu-latest + timeout-minutes: 30 + steps: + - uses: actions/checkout@v4 + + - uses: dtolnay/rust-toolchain@1.97.1 + + - name: Build commandF resolver + run: cargo build --locked -p commandf + + - name: Resolve every frozen package twice and attest identical bytes + run: | + set -euo pipefail + rm -rf /tmp/cf10-digest-discovery + mkdir -p /tmp/cf10-digest-discovery + + cat > /tmp/cf10-cases.tsv <<'CASES' + C001-us-core-before hl7.fhir.us.core 8.0.1 + C001-us-core-after hl7.fhir.us.core 9.0.0 + C002-ips-before hl7.fhir.uv.ips 1.1.0 + C002-ips-after hl7.fhir.uv.ips 2.0.1 + C003-mcode-before hl7.fhir.us.mcode 3.0.0 + C003-mcode-after hl7.fhir.us.mcode 4.0.0 + CASES + + while IFS=$'\t' read -r id package version; do + test -n "$id" + for pass in a b; do + root="/tmp/cf10-digest-discovery/$id/$pass" + mkdir -p "$root" + cargo run --locked --quiet -p commandf -- \ + pkg resolve "$package@$version" \ + --cache "$root/cache" \ + --lock "$root/commandf.lock" + cargo run --locked --quiet -p commandf -- \ + pkg verify \ + --cache "$root/cache" \ + --lock "$root/commandf.lock" + done + done < /tmp/cf10-cases.tsv + + python - <<'PY' + import hashlib + import json + from pathlib import Path + + root = Path('/tmp/cf10-digest-discovery') + rows = [] + for raw in Path('/tmp/cf10-cases.tsv').read_text().splitlines(): + case_state, package, version = raw.split('\t') + observed = [] + for run in ('a', 'b'): + state = root / case_state / run + lock = json.loads((state / 'commandf.lock').read_text()) + matches = [ + item for item in lock['packages'] + if item['name'] == package and item['version'] == version + ] + assert len(matches) == 1, (case_state, run, matches) + locked = matches[0] + archive = state / 'cache' / 'sha256' / f"{locked['sha256']}.tgz" + data = archive.read_bytes() + actual = hashlib.sha256(data).hexdigest() + assert actual == locked['sha256'], (case_state, run, actual, locked['sha256']) + observed.append({ + 'sha256': actual, + 'bytes': len(data), + 'data': data, + }) + assert observed[0]['sha256'] == observed[1]['sha256'], case_state + assert observed[0]['bytes'] == observed[1]['bytes'], case_state + assert observed[0]['data'] == observed[1]['data'], case_state + rows.append({ + 'state_id': case_state, + 'package': package, + 'version': version, + 'sha256': observed[0]['sha256'], + 'archive_bytes': observed[0]['bytes'], + 'independent_resolutions': 2, + 'byte_identical': True, + }) + + rows.sort(key=lambda item: item['state_id']) + output = { + 'schema': 1, + 'selection_frozen_before_results': True, + 'states': rows, + } + out = Path('/tmp/cf10-digest-discovery/cf10-package-attestation.json') + out.write_text(json.dumps(output, indent=2, sort_keys=True) + '\n') + print(out.read_text()) + PY + + - name: Assert no upstream package archives are staged in the repository + run: | + set -euo pipefail + test -z "$(git status --porcelain)" + if git ls-files | grep -E '\.(tgz|tar\.gz)$'; then + echo 'unexpected archive tracked in repository' >&2 + exit 1 + fi + + - name: Upload digest discovery evidence + uses: actions/upload-artifact@v4 + with: + name: cf10-package-attestation + path: /tmp/cf10-digest-discovery/cf10-package-attestation.json + if-no-files-found: error + retention-days: 3 From fdd0395250e9ad8178159fc1ae45c11dd7a87c59 Mon Sep 17 00:00:00 2001 From: "Abdulaziz M. Shehri" Date: Sat, 15 Aug 2026 04:25:29 +0300 Subject: [PATCH 06/82] ci(cf-10): pin digest discovery action revisions --- .github/workflows/cf10-digest-discovery.yml | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/.github/workflows/cf10-digest-discovery.yml b/.github/workflows/cf10-digest-discovery.yml index 982c24ca..dfcf2bf8 100644 --- a/.github/workflows/cf10-digest-discovery.yml +++ b/.github/workflows/cf10-digest-discovery.yml @@ -18,9 +18,11 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 30 steps: - - uses: actions/checkout@v4 + - uses: actions/checkout@fbc6f3992d24b796d5a048ff273f7fcc4a7b6c09 # v5 / Node 24 + with: + persist-credentials: false - - uses: dtolnay/rust-toolchain@1.97.1 + - uses: dtolnay/rust-toolchain@032958afbdc797a9164d3bc0b56325c1308924a5 # 1.97.1 - name: Build commandF resolver run: cargo build --locked -p commandf @@ -118,7 +120,7 @@ jobs: fi - name: Upload digest discovery evidence - uses: actions/upload-artifact@v4 + uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4 with: name: cf10-package-attestation path: /tmp/cf10-digest-discovery/cf10-package-attestation.json From 0a6fa339c5e4ef18fd58b6cd2ed658b414cfc30b Mon Sep 17 00:00:00 2001 From: "Abdulaziz M. Shehri" Date: Sat, 15 Aug 2026 04:26:05 +0300 Subject: [PATCH 07/82] ci(cf-10): run digest discovery on pull requests --- .github/workflows/cf10-digest-discovery.yml | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/.github/workflows/cf10-digest-discovery.yml b/.github/workflows/cf10-digest-discovery.yml index dfcf2bf8..4f0c43eb 100644 --- a/.github/workflows/cf10-digest-discovery.yml +++ b/.github/workflows/cf10-digest-discovery.yml @@ -1,6 +1,11 @@ name: cf10-digest-discovery on: + pull_request: + paths: + - .github/workflows/cf10-digest-discovery.yml + - specs/010-cf-10-real-ig-delta-corpus/** + - donors/cf-10-real-ig-delta-corpus.yaml push: branches: - feat/cf-10-real-ig-delta-corpus From 1443141f77e8364fe8bf5b0cc3f724a69998ee02 Mon Sep 17 00:00:00 2001 From: "Abdulaziz M. Shehri" Date: Sat, 15 Aug 2026 04:28:09 +0300 Subject: [PATCH 08/82] ci(cf-10): report all frozen eligibility failures --- .github/workflows/cf10-digest-discovery.yml | 74 +++++++++++++++++++-- 1 file changed, 69 insertions(+), 5 deletions(-) diff --git a/.github/workflows/cf10-digest-discovery.yml b/.github/workflows/cf10-digest-discovery.yml index 4f0c43eb..8c2e2bee 100644 --- a/.github/workflows/cf10-digest-discovery.yml +++ b/.github/workflows/cf10-digest-discovery.yml @@ -37,6 +37,7 @@ jobs: set -euo pipefail rm -rf /tmp/cf10-digest-discovery mkdir -p /tmp/cf10-digest-discovery + : > /tmp/cf10-digest-discovery/failures.tsv cat > /tmp/cf10-cases.tsv <<'CASES' C001-us-core-before hl7.fhir.us.core 8.0.1 @@ -52,14 +53,27 @@ jobs: for pass in a b; do root="/tmp/cf10-digest-discovery/$id/$pass" mkdir -p "$root" - cargo run --locked --quiet -p commandf -- \ + if ! cargo run --locked --quiet -p commandf -- \ pkg resolve "$package@$version" \ --cache "$root/cache" \ - --lock "$root/commandf.lock" - cargo run --locked --quiet -p commandf -- \ + --lock "$root/commandf.lock" \ + 2>"$root/resolve.err"; then + error="$(tr '\r\n\t' ' ' < "$root/resolve.err" | head -c 1000)" + printf '%s\t%s\t%s\t%s\tresolve\t%s\n' \ + "$id" "$package" "$version" "$pass" "$error" \ + >> /tmp/cf10-digest-discovery/failures.tsv + continue + fi + if ! cargo run --locked --quiet -p commandf -- \ pkg verify \ --cache "$root/cache" \ - --lock "$root/commandf.lock" + --lock "$root/commandf.lock" \ + 2>"$root/verify.err"; then + error="$(tr '\r\n\t' ' ' < "$root/verify.err" | head -c 1000)" + printf '%s\t%s\t%s\t%s\tverify\t%s\n' \ + "$id" "$package" "$version" "$pass" "$error" \ + >> /tmp/cf10-digest-discovery/failures.tsv + fi done done < /tmp/cf10-cases.tsv @@ -69,9 +83,33 @@ jobs: from pathlib import Path root = Path('/tmp/cf10-digest-discovery') + failures = {} + failure_path = root / 'failures.tsv' + for raw in failure_path.read_text().splitlines(): + state_id, package, version, run, phase, error = raw.split('\t', 5) + failures.setdefault(state_id, []).append({ + 'run': run, + 'phase': phase, + 'error': error, + }) + rows = [] for raw in Path('/tmp/cf10-cases.tsv').read_text().splitlines(): case_state, package, version = raw.split('\t') + state_failures = sorted( + failures.get(case_state, []), + key=lambda item: (item['run'], item['phase'], item['error']), + ) + if state_failures: + rows.append({ + 'state_id': case_state, + 'package': package, + 'version': version, + 'status': 'ineligible_resolution_failure', + 'failures': state_failures, + }) + continue + observed = [] for run in ('a', 'b'): state = root / case_state / run @@ -98,6 +136,7 @@ jobs: 'state_id': case_state, 'package': package, 'version': version, + 'status': 'attested', 'sha256': observed[0]['sha256'], 'archive_bytes': observed[0]['bytes'], 'independent_resolutions': 2, @@ -108,9 +147,10 @@ jobs: output = { 'schema': 1, 'selection_frozen_before_results': True, + 'eligibility_sweep_complete': True, 'states': rows, } - out = Path('/tmp/cf10-digest-discovery/cf10-package-attestation.json') + out = root / 'cf10-package-attestation.json' out.write_text(json.dumps(output, indent=2, sort_keys=True) + '\n') print(out.read_text()) PY @@ -131,3 +171,27 @@ jobs: path: /tmp/cf10-digest-discovery/cf10-package-attestation.json if-no-files-found: error retention-days: 3 + + - name: Fail if any frozen package state is ineligible + run: | + python - <<'PY' + import json + from pathlib import Path + + evidence = json.loads( + Path('/tmp/cf10-digest-discovery/cf10-package-attestation.json').read_text() + ) + failed = [state for state in evidence['states'] if state['status'] != 'attested'] + if failed: + for state in failed: + print( + f"ineligible frozen state: {state['state_id']} " + f"{state['package']}@{state['version']}" + ) + for failure in state['failures']: + print( + f" run={failure['run']} phase={failure['phase']} " + f"error={failure['error']}" + ) + raise SystemExit(1) + PY From d8cb88ba2bb3696f44322eb302957c8c50e1c8f4 Mon Sep 17 00:00:00 2001 From: "Abdulaziz M. Shehri" Date: Sat, 15 Aug 2026 17:28:38 +0300 Subject: [PATCH 09/82] docs(cf10): authorize frozen eligibility rerun --- specs/010-cf-10-real-ig-delta-corpus/tasks.md | 15 +++++++++++++-- 1 file changed, 13 insertions(+), 2 deletions(-) diff --git a/specs/010-cf-10-real-ig-delta-corpus/tasks.md b/specs/010-cf-10-real-ig-delta-corpus/tasks.md index 0a3fb6fb..3acebe48 100644 --- a/specs/010-cf-10-real-ig-delta-corpus/tasks.md +++ b/specs/010-cf-10-real-ig-delta-corpus/tasks.md @@ -1,6 +1,17 @@ # CF-10 Tasks — Public Real-IG Delta Corpus -Status: planned / selection rules frozen +Status: canonical CF-11 foundation reconciled / frozen six-state eligibility rerun authorized + +## Foundation reconciliation gate + +- [x] CF-11 multi-version package graph is canonical on `main` at merge commit `5cb1a4c3445c0ebd86654cfb467a5e008e801c3e`. +- [x] Canonical CF-11 was merged into this frozen CF-10 branch by reconciliation merge `5ec463f0ae53b76f9c2c151335d98598b53e5abc`. +- [x] The five pre-existing CF-10 paths remained byte-identical across reconciliation before this authorization update. +- [x] Frozen selection remains exactly: US Core 8.0.1→9.0.0, IPS 1.1.0→2.0.1, mCODE 3.0.0→4.0.0. +- [ ] Rerun the exact same six package states through the existing digest-discovery workflow on the reconciled foundation. +- [ ] Review the resulting artifact before authorizing any semantic diff/classify/check/terminology/oracle corpus execution. + +No semantic CF-10 execution is authorized by this task update. An eligibility failure must remain visible and must not cause case replacement. ## Selection and provenance @@ -12,7 +23,7 @@ Status: planned / selection rules frozen ## Digest discovery -- [ ] T006 Add guarded exact-version digest-discovery workflow/tool. +- [x] T006 Add guarded exact-version digest-discovery workflow/tool. - [ ] T007 Resolve every selected package/version into independent cache A and cache B. - [ ] T008 Verify both caches with CF-01. - [ ] T009 Require byte/digest equality between independent resolutions. From 4f691458f8b05918bdddd6c7dc696d7817edf857 Mon Sep 17 00:00:00 2001 From: "Abdulaziz M. Shehri" Date: Sat, 15 Aug 2026 17:35:01 +0300 Subject: [PATCH 10/82] docs(cf10): freeze real IG package attestations --- donors/cf-10-real-ig-delta-corpus.yaml | 36 ++++++++++++++++++++++++++ 1 file changed, 36 insertions(+) diff --git a/donors/cf-10-real-ig-delta-corpus.yaml b/donors/cf-10-real-ig-delta-corpus.yaml index 8f617b40..566b46f4 100644 --- a/donors/cf-10-real-ig-delta-corpus.yaml +++ b/donors/cf-10-real-ig-delta-corpus.yaml @@ -1,12 +1,32 @@ schema: commandf.donor-manifest/v1 updated: 2026-08-15 +discovery_evidence: + head: d8cb88ba2bb3696f44322eb302957c8c50e1c8f4 + workflow: cf10-digest-discovery + run: 31890014888 + artifact_id: 9248341586 + artifact_digest: sha256:02f08bced4c30665a40c7967611057753bf56330a5e7e10e85081619b433dd8c + independent_resolutions_per_state: 2 + selection_frozen_before_results: true + eligibility_sweep_complete: true + eligible_states: 6 + selected_states: 6 + sources: - id: us-core-real-delta project: HL7 US Core Implementation Guide package: hl7.fhir.us.core before_version: 8.0.1 after_version: 9.0.0 + before_attestation: + sha256: 3c02eef48ef10617021bee95e58cbc66d596ceda8cada24b72000d33ad67c464 + archive_bytes: 2713046 + byte_identical_across_independent_resolutions: true + after_attestation: + sha256: d7b54d2ec2a48cea94ffea5d939ad67a681f80b94d69594a08cebac36da9e059 + archive_bytes: 2749959 + byte_identical_across_independent_resolutions: true before_publication: https://hl7.org/fhir/us/core/STU8.0.1/ after_publication: https://hl7.org/fhir/us/core/STU9/ change_evidence: https://hl7.org/fhir/us/core/STU9/changes.html @@ -34,6 +54,14 @@ sources: package: hl7.fhir.uv.ips before_version: 1.1.0 after_version: 2.0.1 + before_attestation: + sha256: 403c4141101810e924f2928287985084819d8a5cc3a62e2b3840a557129840ef + archive_bytes: 1065103 + byte_identical_across_independent_resolutions: true + after_attestation: + sha256: 7183242b70fb2a9058aa3701fb607517a3c2fd0e3100d1d8c538d744c2adf799 + archive_bytes: 725312 + byte_identical_across_independent_resolutions: true before_publication: https://hl7.org/fhir/uv/ips/STU1.1/ after_publication: https://hl7.org/fhir/uv/ips/en/ change_evidence: https://hl7.org/fhir/uv/ips/en/changes.html @@ -62,6 +90,14 @@ sources: package: hl7.fhir.us.mcode before_version: 3.0.0 after_version: 4.0.0 + before_attestation: + sha256: c94c91971747efeae760aa037d168e4df992cefb6dacece08217c464b9d39214 + archive_bytes: 1014084 + byte_identical_across_independent_resolutions: true + after_attestation: + sha256: e603283bafa508a3705ad022bce95bba1fbd0b8b3b87b978e7412813b7bc1778 + archive_bytes: 1003918 + byte_identical_across_independent_resolutions: true before_publication: https://hl7.org/fhir/us/mcode/STU3/ after_publication: https://hl7.org/fhir/us/mcode/STU4/ change_evidence: https://hl7.org/fhir/us/mcode/STU4/changes.html From 81ba5ab1c7273d77926b684f40de38f4c6a97646 Mon Sep 17 00:00:00 2001 From: "Abdulaziz M. Shehri" Date: Sat, 15 Aug 2026 17:35:22 +0300 Subject: [PATCH 11/82] feat(cf10): freeze corpus v1 package manifest --- corpus/real-ig/v1/corpus.json | 72 +++++++++++++++++++++++++++++++++++ 1 file changed, 72 insertions(+) create mode 100644 corpus/real-ig/v1/corpus.json diff --git a/corpus/real-ig/v1/corpus.json b/corpus/real-ig/v1/corpus.json new file mode 100644 index 00000000..655949a8 --- /dev/null +++ b/corpus/real-ig/v1/corpus.json @@ -0,0 +1,72 @@ +{ + "schema": 1, + "selection_policy": "frozen_pre_result_v1", + "cases": [ + { + "id": "C001", + "package": "hl7.fhir.us.core", + "before": { + "version": "8.0.1", + "archive_sha256": "3c02eef48ef10617021bee95e58cbc66d596ceda8cada24b72000d33ad67c464", + "archive_bytes": 2713046, + "publication_url": "https://hl7.org/fhir/us/core/STU8.0.1/" + }, + "after": { + "version": "9.0.0", + "archive_sha256": "d7b54d2ec2a48cea94ffea5d939ad67a681f80b94d69594a08cebac36da9e059", + "archive_bytes": 2749959, + "publication_url": "https://hl7.org/fhir/us/core/STU9/" + }, + "fhir_version": "4.0.1", + "publisher": "HL7 International", + "change_evidence_url": "https://hl7.org/fhir/us/core/STU9/changes.html", + "rights_evidence_url": "https://hl7.org/fhir/us/core/STU9/ImplementationGuide-hl7.fhir.us.core.html", + "rights_mode": "metadata_only_no_redistribution", + "oracle_mode": "changed_structure_definitions_only" + }, + { + "id": "C002", + "package": "hl7.fhir.uv.ips", + "before": { + "version": "1.1.0", + "archive_sha256": "403c4141101810e924f2928287985084819d8a5cc3a62e2b3840a557129840ef", + "archive_bytes": 1065103, + "publication_url": "https://hl7.org/fhir/uv/ips/STU1.1/" + }, + "after": { + "version": "2.0.1", + "archive_sha256": "7183242b70fb2a9058aa3701fb607517a3c2fd0e3100d1d8c538d744c2adf799", + "archive_bytes": 725312, + "publication_url": "https://hl7.org/fhir/uv/ips/en/" + }, + "fhir_version": "4.0.1", + "publisher": "HL7 International", + "change_evidence_url": "https://hl7.org/fhir/uv/ips/en/changes.html", + "rights_evidence_url": "https://hl7.org/fhir/uv/ips/en/terminology.html", + "rights_mode": "metadata_only_no_redistribution", + "oracle_mode": "changed_structure_definitions_only" + }, + { + "id": "C003", + "package": "hl7.fhir.us.mcode", + "before": { + "version": "3.0.0", + "archive_sha256": "c94c91971747efeae760aa037d168e4df992cefb6dacece08217c464b9d39214", + "archive_bytes": 1014084, + "publication_url": "https://hl7.org/fhir/us/mcode/STU3/" + }, + "after": { + "version": "4.0.0", + "archive_sha256": "e603283bafa508a3705ad022bce95bba1fbd0b8b3b87b978e7412813b7bc1778", + "archive_bytes": 1003918, + "publication_url": "https://hl7.org/fhir/us/mcode/STU4/" + }, + "fhir_version": "4.0.1", + "publisher": "HL7 International", + "change_evidence_url": "https://hl7.org/fhir/us/mcode/STU4/changes.html", + "rights_evidence_url": "https://hl7.org/fhir/us/mcode/STU4/terminology.html", + "rights_mode": "metadata_only_no_redistribution", + "oracle_mode": "changed_structure_definitions_only" + } + ] +} From 3f87506c040eab7389cf9584d40c944db61a1a1c Mon Sep 17 00:00:00 2001 From: "Abdulaziz M. Shehri" Date: Sat, 15 Aug 2026 17:36:34 +0300 Subject: [PATCH 12/82] feat(cf10): add typed corpus manifest model --- crates/commandf-pkg/src/corpus_model.rs | 51 +++++++++++++++++++++++++ 1 file changed, 51 insertions(+) create mode 100644 crates/commandf-pkg/src/corpus_model.rs diff --git a/crates/commandf-pkg/src/corpus_model.rs b/crates/commandf-pkg/src/corpus_model.rs new file mode 100644 index 00000000..cb45872a --- /dev/null +++ b/crates/commandf-pkg/src/corpus_model.rs @@ -0,0 +1,51 @@ +use serde::{Deserialize, Serialize}; + +#[derive(Clone, Debug, Eq, PartialEq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct RealIgCorpus { + pub schema: u64, + pub selection_policy: CorpusSelectionPolicy, + pub cases: Vec, +} + +#[derive(Clone, Copy, Debug, Eq, PartialEq, Serialize, Deserialize)] +#[serde(rename_all = "snake_case")] +pub enum CorpusSelectionPolicy { + FrozenPreResultV1, +} + +#[derive(Clone, Debug, Eq, PartialEq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct RealIgCase { + pub id: String, + pub package: String, + pub before: CorpusPackageState, + pub after: CorpusPackageState, + pub fhir_version: String, + pub publisher: String, + pub change_evidence_url: String, + pub rights_evidence_url: String, + pub rights_mode: CorpusRightsMode, + pub oracle_mode: CorpusOracleMode, +} + +#[derive(Clone, Debug, Eq, PartialEq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct CorpusPackageState { + pub version: String, + pub archive_sha256: String, + pub archive_bytes: u64, + pub publication_url: String, +} + +#[derive(Clone, Copy, Debug, Eq, PartialEq, Serialize, Deserialize)] +#[serde(rename_all = "snake_case")] +pub enum CorpusRightsMode { + MetadataOnlyNoRedistribution, +} + +#[derive(Clone, Copy, Debug, Eq, PartialEq, Serialize, Deserialize)] +#[serde(rename_all = "snake_case")] +pub enum CorpusOracleMode { + ChangedStructureDefinitionsOnly, +} From edd24369421adb62ca8bf476274a5f0ebd65ad6f Mon Sep 17 00:00:00 2001 From: "Abdulaziz M. Shehri" Date: Sat, 15 Aug 2026 17:36:46 +0300 Subject: [PATCH 13/82] feat(cf10): add corpus manifest errors --- crates/commandf-pkg/src/corpus_error.rs | 50 +++++++++++++++++++++++++ 1 file changed, 50 insertions(+) create mode 100644 crates/commandf-pkg/src/corpus_error.rs diff --git a/crates/commandf-pkg/src/corpus_error.rs b/crates/commandf-pkg/src/corpus_error.rs new file mode 100644 index 00000000..e6928ad5 --- /dev/null +++ b/crates/commandf-pkg/src/corpus_error.rs @@ -0,0 +1,50 @@ +use thiserror::Error; + +#[derive(Clone, Debug, Eq, Error, PartialEq)] +pub enum CorpusError { + #[error("corpus manifest is {actual} bytes; maximum is {maximum}")] + ManifestTooLarge { actual: usize, maximum: usize }, + #[error("corpus manifest JSON is invalid: {0}")] + InvalidJson(String), + #[error("unsupported corpus schema {0}; expected schema 1")] + UnsupportedSchema(u64), + #[error("corpus must contain at least one case")] + EmptyCorpus, + #[error("corpus contains {actual} cases; maximum is {maximum}")] + TooManyCases { actual: usize, maximum: usize }, + #[error("duplicate corpus case id {0}")] + DuplicateCaseId(String), + #[error("corpus cases are not in canonical lexicographic order: {previous} before {current}")] + NonCanonicalCaseOrder { previous: String, current: String }, + #[error("invalid corpus case id {0}")] + InvalidCaseId(String), + #[error("case {case_id} has invalid package name {package}")] + InvalidPackageName { case_id: String, package: String }, + #[error("case {case_id} has invalid {side} version {version}")] + InvalidVersion { + case_id: String, + side: &'static str, + version: String, + }, + #[error("case {0} uses the same before and after version")] + SameVersion(String), + #[error("case {case_id} uses unsupported FHIR version {version}; CF-10 v1 requires 4.0.1")] + UnsupportedFhirVersion { case_id: String, version: String }, + #[error("case {case_id} has invalid {side} SHA-256 {sha256}")] + InvalidArchiveSha256 { + case_id: String, + side: &'static str, + sha256: String, + }, + #[error("case {case_id} has invalid {side} archive size {bytes}; maximum is {maximum}")] + InvalidArchiveSize { + case_id: String, + side: &'static str, + bytes: u64, + maximum: u64, + }, + #[error("case {case_id} has invalid or missing {field}")] + InvalidEvidence { case_id: String, field: &'static str }, + #[error("canonical corpus serialization failed: {0}")] + Serialization(String), +} From a2f3174231393703f45a179374fc3c7e0ea4e760 Mon Sep 17 00:00:00 2001 From: "Abdulaziz M. Shehri" Date: Sat, 15 Aug 2026 17:37:10 +0300 Subject: [PATCH 14/82] feat(cf10): validate corpus manifests fail closed --- crates/commandf-pkg/src/corpus.rs | 186 ++++++++++++++++++++++++++++++ 1 file changed, 186 insertions(+) create mode 100644 crates/commandf-pkg/src/corpus.rs diff --git a/crates/commandf-pkg/src/corpus.rs b/crates/commandf-pkg/src/corpus.rs new file mode 100644 index 00000000..e0464b35 --- /dev/null +++ b/crates/commandf-pkg/src/corpus.rs @@ -0,0 +1,186 @@ +use std::collections::BTreeSet; + +use semver::Version; + +use crate::corpus_error::CorpusError; +use crate::corpus_model::{CorpusPackageState, RealIgCase, RealIgCorpus}; +use crate::PackageName; + +pub const MAX_CORPUS_MANIFEST_BYTES: usize = 256 * 1024; +pub const MAX_CORPUS_CASES: usize = 64; +pub const MAX_CORPUS_ARCHIVE_BYTES: u64 = 128 * 1024 * 1024; +const MAX_EVIDENCE_URL_BYTES: usize = 4096; +const MAX_PUBLISHER_BYTES: usize = 256; + +pub fn parse_corpus_manifest(bytes: &[u8]) -> Result { + if bytes.len() > MAX_CORPUS_MANIFEST_BYTES { + return Err(CorpusError::ManifestTooLarge { + actual: bytes.len(), + maximum: MAX_CORPUS_MANIFEST_BYTES, + }); + } + + let corpus: RealIgCorpus = + serde_json::from_slice(bytes).map_err(|error| CorpusError::InvalidJson(error.to_string()))?; + validate_corpus_manifest(&corpus)?; + Ok(corpus) +} + +pub fn validate_corpus_manifest(corpus: &RealIgCorpus) -> Result<(), CorpusError> { + if corpus.schema != 1 { + return Err(CorpusError::UnsupportedSchema(corpus.schema)); + } + if corpus.cases.is_empty() { + return Err(CorpusError::EmptyCorpus); + } + if corpus.cases.len() > MAX_CORPUS_CASES { + return Err(CorpusError::TooManyCases { + actual: corpus.cases.len(), + maximum: MAX_CORPUS_CASES, + }); + } + + let mut ids = BTreeSet::new(); + let mut previous_id: Option<&str> = None; + for case in &corpus.cases { + validate_case_id(&case.id)?; + if !ids.insert(case.id.as_str()) { + return Err(CorpusError::DuplicateCaseId(case.id.clone())); + } + if let Some(previous) = previous_id { + if previous > case.id.as_str() { + return Err(CorpusError::NonCanonicalCaseOrder { + previous: previous.to_owned(), + current: case.id.clone(), + }); + } + } + previous_id = Some(case.id.as_str()); + validate_case(case)?; + } + + Ok(()) +} + +pub fn canonical_corpus_manifest_bytes(corpus: &RealIgCorpus) -> Result, CorpusError> { + validate_corpus_manifest(corpus)?; + let mut bytes = serde_json::to_vec_pretty(corpus) + .map_err(|error| CorpusError::Serialization(error.to_string()))?; + bytes.push(b'\n'); + Ok(bytes) +} + +fn validate_case(case: &RealIgCase) -> Result<(), CorpusError> { + if PackageName::parse(case.package.clone()).is_err() { + return Err(CorpusError::InvalidPackageName { + case_id: case.id.clone(), + package: case.package.clone(), + }); + } + + let before = validate_version(&case.id, "before", &case.before.version)?; + let after = validate_version(&case.id, "after", &case.after.version)?; + if before == after { + return Err(CorpusError::SameVersion(case.id.clone())); + } + + if case.fhir_version != "4.0.1" { + return Err(CorpusError::UnsupportedFhirVersion { + case_id: case.id.clone(), + version: case.fhir_version.clone(), + }); + } + + validate_state(&case.id, "before", &case.before)?; + validate_state(&case.id, "after", &case.after)?; + validate_text(&case.id, "publisher", &case.publisher, MAX_PUBLISHER_BYTES)?; + validate_https_url(&case.id, "change_evidence_url", &case.change_evidence_url)?; + validate_https_url(&case.id, "rights_evidence_url", &case.rights_evidence_url)?; + + Ok(()) +} + +fn validate_case_id(id: &str) -> Result<(), CorpusError> { + let bytes = id.as_bytes(); + if bytes.len() != 4 + || bytes[0] != b'C' + || !bytes[1..].iter().all(u8::is_ascii_digit) + { + return Err(CorpusError::InvalidCaseId(id.to_owned())); + } + Ok(()) +} + +fn validate_version( + case_id: &str, + side: &'static str, + raw: &str, +) -> Result { + Version::parse(raw).map_err(|_| CorpusError::InvalidVersion { + case_id: case_id.to_owned(), + side, + version: raw.to_owned(), + }) +} + +fn validate_state( + case_id: &str, + side: &'static str, + state: &CorpusPackageState, +) -> Result<(), CorpusError> { + if !is_lower_sha256(&state.archive_sha256) { + return Err(CorpusError::InvalidArchiveSha256 { + case_id: case_id.to_owned(), + side, + sha256: state.archive_sha256.clone(), + }); + } + if state.archive_bytes == 0 || state.archive_bytes > MAX_CORPUS_ARCHIVE_BYTES { + return Err(CorpusError::InvalidArchiveSize { + case_id: case_id.to_owned(), + side, + bytes: state.archive_bytes, + maximum: MAX_CORPUS_ARCHIVE_BYTES, + }); + } + validate_https_url(case_id, "publication_url", &state.publication_url) +} + +fn is_lower_sha256(value: &str) -> bool { + value.len() == 64 + && value + .bytes() + .all(|byte| byte.is_ascii_digit() || (b'a'..=b'f').contains(&byte)) +} + +fn validate_https_url(case_id: &str, field: &'static str, value: &str) -> Result<(), CorpusError> { + if value.len() > MAX_EVIDENCE_URL_BYTES + || !value.starts_with("https://") + || value.len() <= "https://".len() + || value.chars().any(|ch| ch.is_whitespace() || ch.is_control()) + { + return Err(CorpusError::InvalidEvidence { + case_id: case_id.to_owned(), + field, + }); + } + Ok(()) +} + +fn validate_text( + case_id: &str, + field: &'static str, + value: &str, + maximum: usize, +) -> Result<(), CorpusError> { + if value.trim().is_empty() + || value.len() > maximum + || value.chars().any(char::is_control) + { + return Err(CorpusError::InvalidEvidence { + case_id: case_id.to_owned(), + field, + }); + } + Ok(()) +} From 7ebc1d7dea0fadcd5bf88b2eb1dde12ea4a4e9b2 Mon Sep 17 00:00:00 2001 From: "Abdulaziz M. Shehri" Date: Sat, 15 Aug 2026 17:37:35 +0300 Subject: [PATCH 15/82] feat(cf10): expose typed corpus manifest boundary --- crates/commandf-pkg/src/lib.rs | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/crates/commandf-pkg/src/lib.rs b/crates/commandf-pkg/src/lib.rs index 3f98df9c..d5b6f5cd 100644 --- a/crates/commandf-pkg/src/lib.rs +++ b/crates/commandf-pkg/src/lib.rs @@ -19,6 +19,9 @@ mod compatibility; mod compatibility_error; mod compatibility_model; mod compatibility_validate; +mod corpus; +mod corpus_error; +mod corpus_model; mod error; mod lock; mod model; @@ -63,6 +66,15 @@ pub use compatibility_model::{ CompatibilityDirection, CompatibilityFinding, CompatibilityReport, CompatibilitySeverity, }; pub use compatibility_validate::classify_structural_diff; +pub use corpus::{ + canonical_corpus_manifest_bytes, parse_corpus_manifest, validate_corpus_manifest, + MAX_CORPUS_ARCHIVE_BYTES, MAX_CORPUS_CASES, MAX_CORPUS_MANIFEST_BYTES, +}; +pub use corpus_error::CorpusError; +pub use corpus_model::{ + CorpusOracleMode, CorpusPackageState, CorpusRightsMode, CorpusSelectionPolicy, RealIgCase, + RealIgCorpus, +}; pub use error::PackageError; pub use lock::{LockedPackage, Lockfile}; pub use model::{PackageName, PackageRequest, VersionConstraint}; From b9206edd3f25e5cffb0ad48138e3487c20c42d23 Mon Sep 17 00:00:00 2001 From: "Abdulaziz M. Shehri" Date: Sat, 15 Aug 2026 17:38:09 +0300 Subject: [PATCH 16/82] test(cf10): cover corpus manifest contract --- crates/commandf-pkg/tests/corpus_manifest.rs | 245 +++++++++++++++++++ 1 file changed, 245 insertions(+) create mode 100644 crates/commandf-pkg/tests/corpus_manifest.rs diff --git a/crates/commandf-pkg/tests/corpus_manifest.rs b/crates/commandf-pkg/tests/corpus_manifest.rs new file mode 100644 index 00000000..ab097db5 --- /dev/null +++ b/crates/commandf-pkg/tests/corpus_manifest.rs @@ -0,0 +1,245 @@ +use std::fs; +use std::path::PathBuf; + +use commandf_pkg::{ + canonical_corpus_manifest_bytes, parse_corpus_manifest, CorpusError, MAX_CORPUS_ARCHIVE_BYTES, + MAX_CORPUS_CASES, MAX_CORPUS_MANIFEST_BYTES, +}; +use serde_json::{json, Value}; + +fn canonical_manifest_bytes() -> Vec { + let path = PathBuf::from(env!("CARGO_MANIFEST_DIR")) + .join("../../corpus/real-ig/v1/corpus.json"); + fs::read(path).expect("canonical CF-10 manifest should be readable") +} + +fn canonical_value() -> Value { + serde_json::from_slice(&canonical_manifest_bytes()).expect("canonical manifest should be JSON") +} + +fn parse_value(value: &Value) -> Result { + parse_corpus_manifest(&serde_json::to_vec(value).expect("test JSON should serialize")) +} + +#[test] +fn canonical_manifest_matches_frozen_discovery_evidence() { + let corpus = parse_corpus_manifest(&canonical_manifest_bytes()).expect("manifest should validate"); + assert_eq!(corpus.cases.len(), 3); + + assert_eq!(corpus.cases[0].id, "C001"); + assert_eq!(corpus.cases[0].package, "hl7.fhir.us.core"); + assert_eq!(corpus.cases[0].before.version, "8.0.1"); + assert_eq!( + corpus.cases[0].before.archive_sha256, + "3c02eef48ef10617021bee95e58cbc66d596ceda8cada24b72000d33ad67c464" + ); + assert_eq!(corpus.cases[0].before.archive_bytes, 2_713_046); + assert_eq!(corpus.cases[0].after.version, "9.0.0"); + assert_eq!( + corpus.cases[0].after.archive_sha256, + "d7b54d2ec2a48cea94ffea5d939ad67a681f80b94d69594a08cebac36da9e059" + ); + assert_eq!(corpus.cases[0].after.archive_bytes, 2_749_959); + + assert_eq!(corpus.cases[1].id, "C002"); + assert_eq!(corpus.cases[1].package, "hl7.fhir.uv.ips"); + assert_eq!(corpus.cases[1].before.version, "1.1.0"); + assert_eq!( + corpus.cases[1].before.archive_sha256, + "403c4141101810e924f2928287985084819d8a5cc3a62e2b3840a557129840ef" + ); + assert_eq!(corpus.cases[1].before.archive_bytes, 1_065_103); + assert_eq!(corpus.cases[1].after.version, "2.0.1"); + assert_eq!( + corpus.cases[1].after.archive_sha256, + "7183242b70fb2a9058aa3701fb607517a3c2fd0e3100d1d8c538d744c2adf799" + ); + assert_eq!(corpus.cases[1].after.archive_bytes, 725_312); + + assert_eq!(corpus.cases[2].id, "C003"); + assert_eq!(corpus.cases[2].package, "hl7.fhir.us.mcode"); + assert_eq!(corpus.cases[2].before.version, "3.0.0"); + assert_eq!( + corpus.cases[2].before.archive_sha256, + "c94c91971747efeae760aa037d168e4df992cefb6dacece08217c464b9d39214" + ); + assert_eq!(corpus.cases[2].before.archive_bytes, 1_014_084); + assert_eq!(corpus.cases[2].after.version, "4.0.0"); + assert_eq!( + corpus.cases[2].after.archive_sha256, + "e603283bafa508a3705ad022bce95bba1fbd0b8b3b87b978e7412813b7bc1778" + ); + assert_eq!(corpus.cases[2].after.archive_bytes, 1_003_918); +} + +#[test] +fn canonical_round_trip_is_deterministic() { + let corpus = parse_corpus_manifest(&canonical_manifest_bytes()).expect("manifest should validate"); + let first = canonical_corpus_manifest_bytes(&corpus).expect("serialization should succeed"); + let reparsed = parse_corpus_manifest(&first).expect("canonical bytes should validate"); + let second = canonical_corpus_manifest_bytes(&reparsed).expect("serialization should succeed"); + assert_eq!(first, second); + assert_eq!(corpus, reparsed); +} + +#[test] +fn oversized_manifest_fails_before_json_decode() { + let bytes = vec![b' '; MAX_CORPUS_MANIFEST_BYTES + 1]; + assert!(matches!( + parse_corpus_manifest(&bytes), + Err(CorpusError::ManifestTooLarge { .. }) + )); +} + +#[test] +fn wrong_schema_fails_closed() { + let mut value = canonical_value(); + value["schema"] = json!(2); + assert_eq!(parse_value(&value), Err(CorpusError::UnsupportedSchema(2))); +} + +#[test] +fn empty_and_oversized_case_sets_fail_closed() { + let mut empty = canonical_value(); + empty["cases"] = json!([]); + assert_eq!(parse_value(&empty), Err(CorpusError::EmptyCorpus)); + + let mut oversized = canonical_value(); + let case = oversized["cases"][0].clone(); + oversized["cases"] = Value::Array(vec![case; MAX_CORPUS_CASES + 1]); + assert!(matches!( + parse_value(&oversized), + Err(CorpusError::TooManyCases { .. }) + )); +} + +#[test] +fn duplicate_and_out_of_order_case_ids_fail_closed() { + let mut duplicate = canonical_value(); + duplicate["cases"][1]["id"] = json!("C001"); + assert_eq!( + parse_value(&duplicate), + Err(CorpusError::DuplicateCaseId("C001".to_owned())) + ); + + let mut out_of_order = canonical_value(); + out_of_order["cases"].as_array_mut().unwrap().swap(0, 1); + assert!(matches!( + parse_value(&out_of_order), + Err(CorpusError::NonCanonicalCaseOrder { .. }) + )); +} + +#[test] +fn malformed_identity_and_versions_fail_closed() { + let mut bad_id = canonical_value(); + bad_id["cases"][0]["id"] = json!("../1"); + assert!(matches!( + parse_value(&bad_id), + Err(CorpusError::InvalidCaseId(_)) + )); + + let mut bad_package = canonical_value(); + bad_package["cases"][0]["package"] = json!("../../etc/passwd"); + assert!(matches!( + parse_value(&bad_package), + Err(CorpusError::InvalidPackageName { .. }) + )); + + let mut bad_version = canonical_value(); + bad_version["cases"][0]["before"]["version"] = json!("8.x"); + assert!(matches!( + parse_value(&bad_version), + Err(CorpusError::InvalidVersion { .. }) + )); + + let mut same_version = canonical_value(); + same_version["cases"][0]["after"]["version"] = json!("8.0.1"); + assert_eq!( + parse_value(&same_version), + Err(CorpusError::SameVersion("C001".to_owned())) + ); +} + +#[test] +fn non_r4_digest_and_size_fail_closed() { + let mut non_r4 = canonical_value(); + non_r4["cases"][0]["fhir_version"] = json!("5.0.0"); + assert!(matches!( + parse_value(&non_r4), + Err(CorpusError::UnsupportedFhirVersion { .. }) + )); + + let mut bad_digest = canonical_value(); + bad_digest["cases"][0]["before"]["archive_sha256"] = json!("ABCDEF"); + assert!(matches!( + parse_value(&bad_digest), + Err(CorpusError::InvalidArchiveSha256 { .. }) + )); + + let mut zero_size = canonical_value(); + zero_size["cases"][0]["before"]["archive_bytes"] = json!(0); + assert!(matches!( + parse_value(&zero_size), + Err(CorpusError::InvalidArchiveSize { .. }) + )); + + let mut huge_size = canonical_value(); + huge_size["cases"][0]["before"]["archive_bytes"] = json!(MAX_CORPUS_ARCHIVE_BYTES + 1); + assert!(matches!( + parse_value(&huge_size), + Err(CorpusError::InvalidArchiveSize { .. }) + )); +} + +#[test] +fn evidence_urls_and_publisher_fail_closed() { + let mut http_publication = canonical_value(); + http_publication["cases"][0]["before"]["publication_url"] = + json!("http://example.invalid/ig"); + assert!(matches!( + parse_value(&http_publication), + Err(CorpusError::InvalidEvidence { .. }) + )); + + let mut empty_publisher = canonical_value(); + empty_publisher["cases"][0]["publisher"] = json!(" "); + assert!(matches!( + parse_value(&empty_publisher), + Err(CorpusError::InvalidEvidence { .. }) + )); +} + +#[test] +fn unknown_fields_missing_fields_and_unknown_enums_fail_closed() { + let mut unknown_field = canonical_value(); + unknown_field["cases"][0]["unexpected"] = json!(true); + assert!(matches!( + parse_value(&unknown_field), + Err(CorpusError::InvalidJson(_)) + )); + + let mut missing_field = canonical_value(); + missing_field["cases"][0] + .as_object_mut() + .unwrap() + .remove("change_evidence_url"); + assert!(matches!( + parse_value(&missing_field), + Err(CorpusError::InvalidJson(_)) + )); + + let mut unknown_rights = canonical_value(); + unknown_rights["cases"][0]["rights_mode"] = json!("redistribute_everything"); + assert!(matches!( + parse_value(&unknown_rights), + Err(CorpusError::InvalidJson(_)) + )); + + let mut unknown_oracle = canonical_value(); + unknown_oracle["cases"][0]["oracle_mode"] = json!("always"); + assert!(matches!( + parse_value(&unknown_oracle), + Err(CorpusError::InvalidJson(_)) + )); +} From 1ecb4dc3a5cc5367ebdb3c38e73ddb49a98b4edd Mon Sep 17 00:00:00 2001 From: "Abdulaziz M. Shehri" Date: Sat, 15 Aug 2026 17:39:17 +0300 Subject: [PATCH 17/82] style(cf10): apply rustfmt to corpus boundary --- crates/commandf-pkg/src/corpus.rs | 24 ++++++++---------------- 1 file changed, 8 insertions(+), 16 deletions(-) diff --git a/crates/commandf-pkg/src/corpus.rs b/crates/commandf-pkg/src/corpus.rs index e0464b35..7e3e372e 100644 --- a/crates/commandf-pkg/src/corpus.rs +++ b/crates/commandf-pkg/src/corpus.rs @@ -20,8 +20,8 @@ pub fn parse_corpus_manifest(bytes: &[u8]) -> Result }); } - let corpus: RealIgCorpus = - serde_json::from_slice(bytes).map_err(|error| CorpusError::InvalidJson(error.to_string()))?; + let corpus: RealIgCorpus = serde_json::from_slice(bytes) + .map_err(|error| CorpusError::InvalidJson(error.to_string()))?; validate_corpus_manifest(&corpus)?; Ok(corpus) } @@ -102,20 +102,13 @@ fn validate_case(case: &RealIgCase) -> Result<(), CorpusError> { fn validate_case_id(id: &str) -> Result<(), CorpusError> { let bytes = id.as_bytes(); - if bytes.len() != 4 - || bytes[0] != b'C' - || !bytes[1..].iter().all(u8::is_ascii_digit) - { + if bytes.len() != 4 || bytes[0] != b'C' || !bytes[1..].iter().all(u8::is_ascii_digit) { return Err(CorpusError::InvalidCaseId(id.to_owned())); } Ok(()) } -fn validate_version( - case_id: &str, - side: &'static str, - raw: &str, -) -> Result { +fn validate_version(case_id: &str, side: &'static str, raw: &str) -> Result { Version::parse(raw).map_err(|_| CorpusError::InvalidVersion { case_id: case_id.to_owned(), side, @@ -157,7 +150,9 @@ fn validate_https_url(case_id: &str, field: &'static str, value: &str) -> Result if value.len() > MAX_EVIDENCE_URL_BYTES || !value.starts_with("https://") || value.len() <= "https://".len() - || value.chars().any(|ch| ch.is_whitespace() || ch.is_control()) + || value + .chars() + .any(|ch| ch.is_whitespace() || ch.is_control()) { return Err(CorpusError::InvalidEvidence { case_id: case_id.to_owned(), @@ -173,10 +168,7 @@ fn validate_text( value: &str, maximum: usize, ) -> Result<(), CorpusError> { - if value.trim().is_empty() - || value.len() > maximum - || value.chars().any(char::is_control) - { + if value.trim().is_empty() || value.len() > maximum || value.chars().any(char::is_control) { return Err(CorpusError::InvalidEvidence { case_id: case_id.to_owned(), field, From 6f07ab57410f47132db33903f04b4c26f8d92ec5 Mon Sep 17 00:00:00 2001 From: "Abdulaziz M. Shehri" Date: Sat, 15 Aug 2026 17:39:41 +0300 Subject: [PATCH 18/82] style(cf10): format corpus errors --- crates/commandf-pkg/src/corpus_error.rs | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/crates/commandf-pkg/src/corpus_error.rs b/crates/commandf-pkg/src/corpus_error.rs index e6928ad5..f209d256 100644 --- a/crates/commandf-pkg/src/corpus_error.rs +++ b/crates/commandf-pkg/src/corpus_error.rs @@ -44,7 +44,10 @@ pub enum CorpusError { maximum: u64, }, #[error("case {case_id} has invalid or missing {field}")] - InvalidEvidence { case_id: String, field: &'static str }, + InvalidEvidence { + case_id: String, + field: &'static str, + }, #[error("canonical corpus serialization failed: {0}")] Serialization(String), } From 7712894453155481649091def55a804a853124fc Mon Sep 17 00:00:00 2001 From: "Abdulaziz M. Shehri" Date: Sat, 15 Aug 2026 17:40:07 +0300 Subject: [PATCH 19/82] style(cf10): format corpus manifest tests --- crates/commandf-pkg/tests/corpus_manifest.rs | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/crates/commandf-pkg/tests/corpus_manifest.rs b/crates/commandf-pkg/tests/corpus_manifest.rs index ab097db5..5509c177 100644 --- a/crates/commandf-pkg/tests/corpus_manifest.rs +++ b/crates/commandf-pkg/tests/corpus_manifest.rs @@ -8,8 +8,8 @@ use commandf_pkg::{ use serde_json::{json, Value}; fn canonical_manifest_bytes() -> Vec { - let path = PathBuf::from(env!("CARGO_MANIFEST_DIR")) - .join("../../corpus/real-ig/v1/corpus.json"); + let path = + PathBuf::from(env!("CARGO_MANIFEST_DIR")).join("../../corpus/real-ig/v1/corpus.json"); fs::read(path).expect("canonical CF-10 manifest should be readable") } @@ -23,7 +23,8 @@ fn parse_value(value: &Value) -> Result #[test] fn canonical_manifest_matches_frozen_discovery_evidence() { - let corpus = parse_corpus_manifest(&canonical_manifest_bytes()).expect("manifest should validate"); + let corpus = + parse_corpus_manifest(&canonical_manifest_bytes()).expect("manifest should validate"); assert_eq!(corpus.cases.len(), 3); assert_eq!(corpus.cases[0].id, "C001"); @@ -74,7 +75,8 @@ fn canonical_manifest_matches_frozen_discovery_evidence() { #[test] fn canonical_round_trip_is_deterministic() { - let corpus = parse_corpus_manifest(&canonical_manifest_bytes()).expect("manifest should validate"); + let corpus = + parse_corpus_manifest(&canonical_manifest_bytes()).expect("manifest should validate"); let first = canonical_corpus_manifest_bytes(&corpus).expect("serialization should succeed"); let reparsed = parse_corpus_manifest(&first).expect("canonical bytes should validate"); let second = canonical_corpus_manifest_bytes(&reparsed).expect("serialization should succeed"); @@ -195,8 +197,7 @@ fn non_r4_digest_and_size_fail_closed() { #[test] fn evidence_urls_and_publisher_fail_closed() { let mut http_publication = canonical_value(); - http_publication["cases"][0]["before"]["publication_url"] = - json!("http://example.invalid/ig"); + http_publication["cases"][0]["before"]["publication_url"] = json!("http://example.invalid/ig"); assert!(matches!( parse_value(&http_publication), Err(CorpusError::InvalidEvidence { .. }) From 0241e6e60e87a8e46cc9e71b4f46ba23e6aeb77b Mon Sep 17 00:00:00 2001 From: "Abdulaziz M. Shehri" Date: Sat, 15 Aug 2026 17:42:39 +0300 Subject: [PATCH 20/82] feat(cf10): model package attestation evidence --- crates/commandf-pkg/src/corpus_model.rs | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/crates/commandf-pkg/src/corpus_model.rs b/crates/commandf-pkg/src/corpus_model.rs index cb45872a..f5cf87ac 100644 --- a/crates/commandf-pkg/src/corpus_model.rs +++ b/crates/commandf-pkg/src/corpus_model.rs @@ -49,3 +49,21 @@ pub enum CorpusRightsMode { pub enum CorpusOracleMode { ChangedStructureDefinitionsOnly, } + +#[derive(Clone, Copy, Debug, Eq, PartialEq, Serialize, Deserialize)] +#[serde(rename_all = "snake_case")] +pub enum CorpusPackageSide { + Before, + After, +} + +#[derive(Clone, Debug, Eq, PartialEq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct CorpusPackageAttestation { + pub case_id: String, + pub package: String, + pub side: CorpusPackageSide, + pub version: String, + pub sha256: String, + pub archive_bytes: u64, +} From 40b6ae5b90800403c0f8de2941c36cd5ea6eb597 Mon Sep 17 00:00:00 2001 From: "Abdulaziz M. Shehri" Date: Sat, 15 Aug 2026 17:43:55 +0300 Subject: [PATCH 21/82] feat(cf10): expose verified cache bytes internally --- crates/commandf-pkg/src/cache.rs | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/crates/commandf-pkg/src/cache.rs b/crates/commandf-pkg/src/cache.rs index 57f09791..b098b18c 100644 --- a/crates/commandf-pkg/src/cache.rs +++ b/crates/commandf-pkg/src/cache.rs @@ -80,6 +80,18 @@ impl PackageCache { } Ok(()) } + + pub(crate) fn read_verified(&self, digest: &str) -> Result, PackageError> { + self.verify(digest)?; + let path = self.object_path(digest); + fs::read(&path).map_err(|error| { + if error.kind() == std::io::ErrorKind::NotFound { + PackageError::CacheMissing(digest.to_owned()) + } else { + PackageError::Io(error) + } + }) + } } fn validate_digest(digest: &str) -> Result<(), PackageError> { From 6e6df8ba8ab01631c6d8cda25ca3ee47b60eb6cc Mon Sep 17 00:00:00 2001 From: "Abdulaziz M. Shehri" Date: Sat, 15 Aug 2026 17:44:10 +0300 Subject: [PATCH 22/82] feat(cf10): add package attestation failures --- crates/commandf-pkg/src/corpus_error.rs | 36 +++++++++++++++++++++++++ 1 file changed, 36 insertions(+) diff --git a/crates/commandf-pkg/src/corpus_error.rs b/crates/commandf-pkg/src/corpus_error.rs index f209d256..fcc09406 100644 --- a/crates/commandf-pkg/src/corpus_error.rs +++ b/crates/commandf-pkg/src/corpus_error.rs @@ -48,6 +48,42 @@ pub enum CorpusError { case_id: String, field: &'static str, }, + #[error("case {case_id} failed mandatory CF-01 cache verification: {message}")] + CacheVerification { case_id: String, message: String }, + #[error("case {case_id} lock is missing exact package {package}@{version}")] + LockedPackageMissing { + case_id: String, + package: String, + version: String, + }, + #[error("case {case_id} lock contains duplicate exact package {package}@{version}")] + LockedPackageAmbiguous { + case_id: String, + package: String, + version: String, + }, + #[error("case {case_id} lock digest mismatch for {package}@{version}: expected {expected}, found {found}")] + LockedPackageDigestMismatch { + case_id: String, + package: String, + version: String, + expected: String, + found: String, + }, + #[error("case {case_id} {side} archive size mismatch: expected {expected}, found {found}")] + ArchiveSizeMismatch { + case_id: String, + side: &'static str, + expected: u64, + found: u64, + }, + #[error("case {case_id} {side} archive digest mismatch: expected {expected}, found {found}")] + ArchiveDigestMismatch { + case_id: String, + side: &'static str, + expected: String, + found: String, + }, #[error("canonical corpus serialization failed: {0}")] Serialization(String), } From eb625bbd02b0c836462d8dca18f127cf12e6582a Mon Sep 17 00:00:00 2001 From: "Abdulaziz M. Shehri" Date: Sat, 15 Aug 2026 17:44:37 +0300 Subject: [PATCH 23/82] feat(cf10): require verified package attestation --- crates/commandf-pkg/src/corpus.rs | 96 ++++++++++++++++++++++++++++++- 1 file changed, 94 insertions(+), 2 deletions(-) diff --git a/crates/commandf-pkg/src/corpus.rs b/crates/commandf-pkg/src/corpus.rs index 7e3e372e..8ebae31b 100644 --- a/crates/commandf-pkg/src/corpus.rs +++ b/crates/commandf-pkg/src/corpus.rs @@ -3,8 +3,10 @@ use std::collections::BTreeSet; use semver::Version; use crate::corpus_error::CorpusError; -use crate::corpus_model::{CorpusPackageState, RealIgCase, RealIgCorpus}; -use crate::PackageName; +use crate::corpus_model::{ + CorpusPackageAttestation, CorpusPackageSide, CorpusPackageState, RealIgCase, RealIgCorpus, +}; +use crate::{Lockfile, PackageCache, PackageName}; pub const MAX_CORPUS_MANIFEST_BYTES: usize = 256 * 1024; pub const MAX_CORPUS_CASES: usize = 64; @@ -70,6 +72,96 @@ pub fn canonical_corpus_manifest_bytes(corpus: &RealIgCorpus) -> Result, Ok(bytes) } +pub fn attest_corpus_package_state( + case: &RealIgCase, + side: CorpusPackageSide, + lockfile: &Lockfile, + cache: &PackageCache, +) -> Result { + validate_case(case)?; + lockfile + .verify_cache(cache) + .map_err(|error| CorpusError::CacheVerification { + case_id: case.id.clone(), + message: error.to_string(), + })?; + + let (side_name, expected) = expected_state(case, side); + let mut matches = lockfile + .packages + .iter() + .filter(|package| package.name == case.package && package.version == expected.version); + let locked = matches + .next() + .ok_or_else(|| CorpusError::LockedPackageMissing { + case_id: case.id.clone(), + package: case.package.clone(), + version: expected.version.clone(), + })?; + if matches.next().is_some() { + return Err(CorpusError::LockedPackageAmbiguous { + case_id: case.id.clone(), + package: case.package.clone(), + version: expected.version.clone(), + }); + } + + if locked.sha256 != expected.archive_sha256 { + return Err(CorpusError::LockedPackageDigestMismatch { + case_id: case.id.clone(), + package: case.package.clone(), + version: expected.version.clone(), + expected: expected.archive_sha256.clone(), + found: locked.sha256.clone(), + }); + } + + let bytes = cache + .read_verified(&locked.sha256) + .map_err(|error| CorpusError::CacheVerification { + case_id: case.id.clone(), + message: error.to_string(), + })?; + let actual_bytes = u64::try_from(bytes.len()).unwrap_or(u64::MAX); + if actual_bytes != expected.archive_bytes { + return Err(CorpusError::ArchiveSizeMismatch { + case_id: case.id.clone(), + side: side_name, + expected: expected.archive_bytes, + found: actual_bytes, + }); + } + + let actual_sha256 = PackageCache::digest(&bytes); + if actual_sha256 != expected.archive_sha256 { + return Err(CorpusError::ArchiveDigestMismatch { + case_id: case.id.clone(), + side: side_name, + expected: expected.archive_sha256.clone(), + found: actual_sha256, + }); + } + + Ok(CorpusPackageAttestation { + case_id: case.id.clone(), + package: case.package.clone(), + side, + version: expected.version.clone(), + sha256: expected.archive_sha256.clone(), + archive_bytes: expected.archive_bytes, + }) +} + +fn expected_state( + case: &RealIgCase, + side: CorpusPackageSide, +) -> (&'static str, &CorpusPackageState) { + match side { + CorpusPackageSide::Before => ("before", &case.before), + CorpusPackageSide::After => ("after", &case.after), + } +} + fn validate_case(case: &RealIgCase) -> Result<(), CorpusError> { if PackageName::parse(case.package.clone()).is_err() { return Err(CorpusError::InvalidPackageName { From d1975f8f0532d118c1c35358a63a015404e19014 Mon Sep 17 00:00:00 2001 From: "Abdulaziz M. Shehri" Date: Sat, 15 Aug 2026 17:45:03 +0300 Subject: [PATCH 24/82] feat(cf10): expose verified corpus attestation --- crates/commandf-pkg/src/lib.rs | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/crates/commandf-pkg/src/lib.rs b/crates/commandf-pkg/src/lib.rs index d5b6f5cd..a953939f 100644 --- a/crates/commandf-pkg/src/lib.rs +++ b/crates/commandf-pkg/src/lib.rs @@ -67,13 +67,14 @@ pub use compatibility_model::{ }; pub use compatibility_validate::classify_structural_diff; pub use corpus::{ - canonical_corpus_manifest_bytes, parse_corpus_manifest, validate_corpus_manifest, - MAX_CORPUS_ARCHIVE_BYTES, MAX_CORPUS_CASES, MAX_CORPUS_MANIFEST_BYTES, + attest_corpus_package_state, canonical_corpus_manifest_bytes, parse_corpus_manifest, + validate_corpus_manifest, MAX_CORPUS_ARCHIVE_BYTES, MAX_CORPUS_CASES, + MAX_CORPUS_MANIFEST_BYTES, }; pub use corpus_error::CorpusError; pub use corpus_model::{ - CorpusOracleMode, CorpusPackageState, CorpusRightsMode, CorpusSelectionPolicy, RealIgCase, - RealIgCorpus, + CorpusOracleMode, CorpusPackageAttestation, CorpusPackageSide, CorpusPackageState, + CorpusRightsMode, CorpusSelectionPolicy, RealIgCase, RealIgCorpus, }; pub use error::PackageError; pub use lock::{LockedPackage, Lockfile}; From 43dbc2eb89a42950eff3213238d587f4172a6c5c Mon Sep 17 00:00:00 2001 From: "Abdulaziz M. Shehri" Date: Sat, 15 Aug 2026 17:45:29 +0300 Subject: [PATCH 25/82] test(cf10): enforce verified package attestation --- .../commandf-pkg/tests/corpus_attestation.rs | 216 ++++++++++++++++++ 1 file changed, 216 insertions(+) create mode 100644 crates/commandf-pkg/tests/corpus_attestation.rs diff --git a/crates/commandf-pkg/tests/corpus_attestation.rs b/crates/commandf-pkg/tests/corpus_attestation.rs new file mode 100644 index 00000000..a92d2b9c --- /dev/null +++ b/crates/commandf-pkg/tests/corpus_attestation.rs @@ -0,0 +1,216 @@ +use std::collections::BTreeMap; +use std::fs; + +use commandf_pkg::{ + attest_corpus_package_state, CorpusError, CorpusOracleMode, CorpusPackageSide, + CorpusPackageState, CorpusRightsMode, LockedPackage, Lockfile, PackageCache, RealIgCase, +}; +use tempfile::TempDir; + +fn package_state(version: &str, digest: String, bytes: u64, label: &str) -> CorpusPackageState { + CorpusPackageState { + version: version.to_owned(), + archive_sha256: digest, + archive_bytes: bytes, + publication_url: format!("https://example.org/{label}"), + } +} + +fn case_with_states(before: CorpusPackageState, after: CorpusPackageState) -> RealIgCase { + RealIgCase { + id: "C001".to_owned(), + package: "acme.root".to_owned(), + before, + after, + fhir_version: "4.0.1".to_owned(), + publisher: "Example Publisher".to_owned(), + change_evidence_url: "https://example.org/changes".to_owned(), + rights_evidence_url: "https://example.org/rights".to_owned(), + rights_mode: CorpusRightsMode::MetadataOnlyNoRedistribution, + oracle_mode: CorpusOracleMode::ChangedStructureDefinitionsOnly, + } +} + +fn locked(name: &str, version: &str, digest: &str) -> LockedPackage { + LockedPackage { + name: name.to_owned(), + version: version.to_owned(), + sha256: digest.to_owned(), + source: format!("https://registry.example.org/{name}/{version}"), + dependencies: BTreeMap::new(), + } +} + +fn fixture() -> (TempDir, PackageCache, RealIgCase, Lockfile) { + let temp = TempDir::new().expect("tempdir"); + let cache = PackageCache::new(temp.path()); + let before_bytes = b"before-package-bytes"; + let after_bytes = b"after-package-bytes"; + let before_digest = cache.put(before_bytes).expect("cache before"); + let after_digest = cache.put(after_bytes).expect("cache after"); + let case = case_with_states( + package_state( + "1.0.0", + before_digest.clone(), + before_bytes.len() as u64, + "before", + ), + package_state( + "2.0.0", + after_digest.clone(), + after_bytes.len() as u64, + "after", + ), + ); + let lockfile = Lockfile::new( + vec!["acme.root@1.0.0".to_owned(), "acme.root@2.0.0".to_owned()], + vec![ + locked("acme.root", "1.0.0", &before_digest), + locked("acme.root", "2.0.0", &after_digest), + ], + ); + (temp, cache, case, lockfile) +} + +#[test] +fn matching_before_and_after_states_attest() { + let (_temp, cache, case, lockfile) = fixture(); + + let before = attest_corpus_package_state( + &case, + CorpusPackageSide::Before, + &lockfile, + &cache, + ) + .expect("before should attest"); + assert_eq!(before.case_id, "C001"); + assert_eq!(before.package, "acme.root"); + assert_eq!(before.side, CorpusPackageSide::Before); + assert_eq!(before.version, "1.0.0"); + assert_eq!(before.sha256, case.before.archive_sha256); + assert_eq!(before.archive_bytes, case.before.archive_bytes); + + let after = attest_corpus_package_state( + &case, + CorpusPackageSide::After, + &lockfile, + &cache, + ) + .expect("after should attest"); + assert_eq!(after.side, CorpusPackageSide::After); + assert_eq!(after.version, "2.0.0"); + assert_eq!(after.sha256, case.after.archive_sha256); + assert_eq!(after.archive_bytes, case.after.archive_bytes); +} + +#[test] +fn manifest_digest_mismatch_fails_closed() { + let (_temp, cache, mut case, lockfile) = fixture(); + case.before.archive_sha256 = "0".repeat(64); + + assert!(matches!( + attest_corpus_package_state( + &case, + CorpusPackageSide::Before, + &lockfile, + &cache + ), + Err(CorpusError::LockedPackageDigestMismatch { .. }) + )); +} + +#[test] +fn manifest_size_mismatch_fails_closed() { + let (_temp, cache, mut case, lockfile) = fixture(); + case.before.archive_bytes += 1; + + assert!(matches!( + attest_corpus_package_state( + &case, + CorpusPackageSide::Before, + &lockfile, + &cache + ), + Err(CorpusError::ArchiveSizeMismatch { .. }) + )); +} + +#[test] +fn corrupted_target_cache_fails_during_mandatory_cf01_verification() { + let (temp, cache, case, lockfile) = fixture(); + let path = temp + .path() + .join("sha256") + .join(format!("{}.tgz", case.before.archive_sha256)); + fs::write(path, b"corrupted").expect("corrupt cache object"); + + assert!(matches!( + attest_corpus_package_state( + &case, + CorpusPackageSide::Before, + &lockfile, + &cache + ), + Err(CorpusError::CacheVerification { .. }) + )); +} + +#[test] +fn missing_exact_locked_state_fails_closed() { + let (_temp, cache, case, mut lockfile) = fixture(); + lockfile + .packages + .retain(|package| package.version != case.before.version); + + assert!(matches!( + attest_corpus_package_state( + &case, + CorpusPackageSide::Before, + &lockfile, + &cache + ), + Err(CorpusError::LockedPackageMissing { .. }) + )); +} + +#[test] +fn duplicate_exact_locked_state_fails_closed() { + let (_temp, cache, case, mut lockfile) = fixture(); + let duplicate = lockfile + .packages + .iter() + .find(|package| package.version == case.before.version) + .expect("before locked package") + .clone(); + lockfile.packages.push(duplicate); + + assert!(matches!( + attest_corpus_package_state( + &case, + CorpusPackageSide::Before, + &lockfile, + &cache + ), + Err(CorpusError::LockedPackageAmbiguous { .. }) + )); +} + +#[test] +fn unrelated_unverified_lock_entry_blocks_attestation() { + let (_temp, cache, case, mut lockfile) = fixture(); + lockfile.packages.push(locked( + "acme.unrelated", + "1.0.0", + &"f".repeat(64), + )); + + assert!(matches!( + attest_corpus_package_state( + &case, + CorpusPackageSide::Before, + &lockfile, + &cache + ), + Err(CorpusError::CacheVerification { .. }) + )); +} From e1004426389d2d7e0d0b1a9a2a2d6bcd5635a990 Mon Sep 17 00:00:00 2001 From: "Abdulaziz M. Shehri" Date: Sat, 15 Aug 2026 17:46:25 +0300 Subject: [PATCH 26/82] style(cf10): format package attestation boundary --- crates/commandf-pkg/src/corpus.rs | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/crates/commandf-pkg/src/corpus.rs b/crates/commandf-pkg/src/corpus.rs index 8ebae31b..09d32232 100644 --- a/crates/commandf-pkg/src/corpus.rs +++ b/crates/commandf-pkg/src/corpus.rs @@ -116,12 +116,13 @@ pub fn attest_corpus_package_state( }); } - let bytes = cache - .read_verified(&locked.sha256) - .map_err(|error| CorpusError::CacheVerification { - case_id: case.id.clone(), - message: error.to_string(), - })?; + let bytes = + cache + .read_verified(&locked.sha256) + .map_err(|error| CorpusError::CacheVerification { + case_id: case.id.clone(), + message: error.to_string(), + })?; let actual_bytes = u64::try_from(bytes.len()).unwrap_or(u64::MAX); if actual_bytes != expected.archive_bytes { return Err(CorpusError::ArchiveSizeMismatch { From 971bf85fe3f9c67cd6a2f730c1e90a3a9491bc82 Mon Sep 17 00:00:00 2001 From: "Abdulaziz M. Shehri" Date: Sat, 15 Aug 2026 17:46:44 +0300 Subject: [PATCH 27/82] style(cf10): format package attestation tests --- .../commandf-pkg/tests/corpus_attestation.rs | 68 ++++--------------- 1 file changed, 13 insertions(+), 55 deletions(-) diff --git a/crates/commandf-pkg/tests/corpus_attestation.rs b/crates/commandf-pkg/tests/corpus_attestation.rs index a92d2b9c..ca601928 100644 --- a/crates/commandf-pkg/tests/corpus_attestation.rs +++ b/crates/commandf-pkg/tests/corpus_attestation.rs @@ -76,13 +76,8 @@ fn fixture() -> (TempDir, PackageCache, RealIgCase, Lockfile) { fn matching_before_and_after_states_attest() { let (_temp, cache, case, lockfile) = fixture(); - let before = attest_corpus_package_state( - &case, - CorpusPackageSide::Before, - &lockfile, - &cache, - ) - .expect("before should attest"); + let before = attest_corpus_package_state(&case, CorpusPackageSide::Before, &lockfile, &cache) + .expect("before should attest"); assert_eq!(before.case_id, "C001"); assert_eq!(before.package, "acme.root"); assert_eq!(before.side, CorpusPackageSide::Before); @@ -90,13 +85,8 @@ fn matching_before_and_after_states_attest() { assert_eq!(before.sha256, case.before.archive_sha256); assert_eq!(before.archive_bytes, case.before.archive_bytes); - let after = attest_corpus_package_state( - &case, - CorpusPackageSide::After, - &lockfile, - &cache, - ) - .expect("after should attest"); + let after = attest_corpus_package_state(&case, CorpusPackageSide::After, &lockfile, &cache) + .expect("after should attest"); assert_eq!(after.side, CorpusPackageSide::After); assert_eq!(after.version, "2.0.0"); assert_eq!(after.sha256, case.after.archive_sha256); @@ -109,12 +99,7 @@ fn manifest_digest_mismatch_fails_closed() { case.before.archive_sha256 = "0".repeat(64); assert!(matches!( - attest_corpus_package_state( - &case, - CorpusPackageSide::Before, - &lockfile, - &cache - ), + attest_corpus_package_state(&case, CorpusPackageSide::Before, &lockfile, &cache), Err(CorpusError::LockedPackageDigestMismatch { .. }) )); } @@ -125,12 +110,7 @@ fn manifest_size_mismatch_fails_closed() { case.before.archive_bytes += 1; assert!(matches!( - attest_corpus_package_state( - &case, - CorpusPackageSide::Before, - &lockfile, - &cache - ), + attest_corpus_package_state(&case, CorpusPackageSide::Before, &lockfile, &cache), Err(CorpusError::ArchiveSizeMismatch { .. }) )); } @@ -145,12 +125,7 @@ fn corrupted_target_cache_fails_during_mandatory_cf01_verification() { fs::write(path, b"corrupted").expect("corrupt cache object"); assert!(matches!( - attest_corpus_package_state( - &case, - CorpusPackageSide::Before, - &lockfile, - &cache - ), + attest_corpus_package_state(&case, CorpusPackageSide::Before, &lockfile, &cache), Err(CorpusError::CacheVerification { .. }) )); } @@ -163,12 +138,7 @@ fn missing_exact_locked_state_fails_closed() { .retain(|package| package.version != case.before.version); assert!(matches!( - attest_corpus_package_state( - &case, - CorpusPackageSide::Before, - &lockfile, - &cache - ), + attest_corpus_package_state(&case, CorpusPackageSide::Before, &lockfile, &cache), Err(CorpusError::LockedPackageMissing { .. }) )); } @@ -185,12 +155,7 @@ fn duplicate_exact_locked_state_fails_closed() { lockfile.packages.push(duplicate); assert!(matches!( - attest_corpus_package_state( - &case, - CorpusPackageSide::Before, - &lockfile, - &cache - ), + attest_corpus_package_state(&case, CorpusPackageSide::Before, &lockfile, &cache), Err(CorpusError::LockedPackageAmbiguous { .. }) )); } @@ -198,19 +163,12 @@ fn duplicate_exact_locked_state_fails_closed() { #[test] fn unrelated_unverified_lock_entry_blocks_attestation() { let (_temp, cache, case, mut lockfile) = fixture(); - lockfile.packages.push(locked( - "acme.unrelated", - "1.0.0", - &"f".repeat(64), - )); + lockfile + .packages + .push(locked("acme.unrelated", "1.0.0", &"f".repeat(64))); assert!(matches!( - attest_corpus_package_state( - &case, - CorpusPackageSide::Before, - &lockfile, - &cache - ), + attest_corpus_package_state(&case, CorpusPackageSide::Before, &lockfile, &cache), Err(CorpusError::CacheVerification { .. }) )); } From 64f603d3e6d481aebf2d0f09ef3c945acce4abbe Mon Sep 17 00:00:00 2001 From: "Abdulaziz M. Shehri" Date: Sat, 15 Aug 2026 18:20:09 +0300 Subject: [PATCH 28/82] docs(cf10): authorize deterministic corpus execution surface --- specs/010-cf-10-real-ig-delta-corpus/spec.md | 134 ++++++++++++------- 1 file changed, 89 insertions(+), 45 deletions(-) diff --git a/specs/010-cf-10-real-ig-delta-corpus/spec.md b/specs/010-cf-10-real-ig-delta-corpus/spec.md index 23e2608e..d7a80c7e 100644 --- a/specs/010-cf-10-real-ig-delta-corpus/spec.md +++ b/specs/010-cf-10-real-ig-delta-corpus/spec.md @@ -1,6 +1,6 @@ # CF-10 — Public Real-IG Delta Corpus -Status: planned / selection rules frozen before digest discovery +Status: implementation authorized / six frozen package states eligible and digest-frozen ## Purpose @@ -35,7 +35,7 @@ Once a case passes this gate, later unfavorable output cannot remove it from cor ## Corpus v1 families -The initial frozen candidate set deliberately spans three distinct interoperability contexts: +The frozen corpus remains exactly: ### C001 — US Core annual national-core delta @@ -49,8 +49,6 @@ FHIR: 4.0.1 realm/context: US national/base core ``` -Rationale: US Core is an annually revised base IG and publishes explicit cross-version/change guidance. The pair is adjacent published annual releases and is not chosen from commandF output. - ### C002 — International Patient Summary major-generation delta ```text @@ -63,8 +61,6 @@ FHIR: 4.0.1 realm/context: international patient summary ``` -Rationale: IPS 2 documents explicit non-compatible and compatible substantive changes relative to the previous generation. This gives the corpus an international, terminology-rich document/profile family rather than another US-only base guide. - ### C003 — mCODE specialty-oncology major delta ```text @@ -77,7 +73,27 @@ FHIR: 4.0.1 realm/context: oncology specialty IG ``` -Rationale: mCODE publishes stable R4 generations and has explicit version-differential/change material. It exercises specialty profiles and terminology rather than general core/document-only content. +No case or version was replaced after discovery. + +## Foundation reconciliation and eligibility truth + +CF-11 multi-version package-graph support is canonical on `main` at merge commit: + +```text +5cb1a4c3445c0ebd86654cfb467a5e008e801c3e +``` + +It was reconciled into the frozen CF-10 branch by merge commit: + +```text +5ec463f0ae53b76f9c2c151335d98598b53e5abc +``` + +The original frozen CF-10 files remained byte-identical across that reconciliation. + +The same six package states were then rerun without changing the corpus. `cf10-digest-discovery` run `31890014888` succeeded and produced reviewed artifact `9248341586`; a later exact-head rerun `31890859039` also succeeded. Every frozen state resolved twice into independent clean caches, passed CF-01 verification, and produced identical archive bytes/digests across the two resolutions. + +The canonical manifest freezes only those observed digest/size facts. No semantic result counts were used to select or replace cases. ## Rights and redistribution boundary @@ -90,7 +106,7 @@ The corpus manifest stores only: - package/version identity; - commandF-observed content digest after public resolution; - publication/provenance URLs; -- upstream rights/IP evidence URLs and a conservative rights note; +- upstream rights/IP evidence URLs and conservative source-specific metadata; - deterministic commandF result metadata/hashes that contain no redistributed source package bytes. Package bytes are ephemeral runtime inputs acquired through the existing public package resolver and verified against the locked digest. Mixed upstream terminology/IP statements remain upstream obligations; commandF's repository license MUST NOT be interpreted as relicensing them. @@ -99,16 +115,16 @@ Package bytes are ephemeral runtime inputs acquired through the existing public The package SHA-256 values are not copied from web pages or manually invented. -For each frozen case, an isolated discovery gate MUST: +For each frozen state, discovery MUST: 1. resolve the exact package version through commandF CF-01; 2. record the resolver-produced content SHA-256 and byte size; 3. independently resolve the same exact version into a second clean cache; 4. require identical digest and archive bytes between the two resolutions; -5. verify the package through CF-01 cache verification; -6. persist only the digest/size metadata into the corpus manifest. +5. verify both package graphs through CF-01 cache verification; +6. persist only reviewed digest/size metadata into the corpus manifest. -After the digest is frozen, ordinary corpus runs MUST fail closed if a public resolution produces different bytes for the same package/version. +After a digest is frozen, ordinary corpus runs MUST fail closed if resolution produces different bytes for the same package/version. ## Manifest contract @@ -146,33 +162,73 @@ cases[] Rules: -- case ids are unique and stable; -- package names/versions are exact, never ranges/wildcards; +- schema is exactly `1`; +- manifest bytes and case count are bounded before/after decode as applicable; +- case ids are unique, stable, and lexicographically ordered; +- package names/versions are exact, never ranges/wildcards or path-like identities; - archive digests are lowercase SHA-256 hex; - archive sizes are positive and bounded; - `fhir_version` is exactly `4.0.1` in CF-10 v1; - publication/change/rights evidence is explicit per case; - `rights_mode` for v1 is `metadata_only_no_redistribution`; - `oracle_mode` is `changed_structure_definitions_only`; -- unknown schema versions fail closed; -- malformed/duplicate/unsorted cases fail closed. +- unknown schema, fields, or enum values fail closed; +- malformed/duplicate/unsorted cases fail closed; +- canonical JSON round-trip bytes are deterministic. + +The manifest parser does not fetch the network. + +## Package attestation contract + +Before any semantic analysis of a case side, commandF MUST: + +1. verify the complete supplied lockfile cache through the canonical CF-01 `Lockfile::verify_cache` authority; +2. require exactly one locked package matching the manifest `(package, version)` identity; +3. require the locked digest to equal the manifest digest; +4. read the target archive only through verified cache access; +5. require the observed archive byte length and SHA-256 to equal the manifest state. + +A corrupt or missing dependency anywhere in the supplied lock graph blocks attestation even if the root archive itself is present. + +## Authorized execution surface + +CF-10 v1 authorizes exactly one user-visible execution surface: + +```text +commandf corpus run \ + --manifest corpus/real-ig/v1/corpus.json \ + --work-root \ + --oracle-adapter \ + --oracle-java \ + --format json +``` + +There is no second repository-owned public corpus harness in v1. + +The command is an integration/benchmark operation and MAY acquire exact packages from the public FHIR package ecosystem, but it MUST preserve a hard internal boundary: + +```text +acquire -> CF-01 verify -> manifest digest/size attest -> semantic evaluation +``` + +Semantic evaluation MUST NOT begin for a case until both before/after states attest successfully. ## Runner contract -CF-10 MAY introduce a thin deterministic corpus orchestrator, but it MUST call existing commandF authorities rather than reimplement their semantics. +A corpus run processes every frozen case in canonical case-id order. For one case it performs: -A corpus run for one case performs: +1. exact before/after package resolution into isolated state roots; +2. full CF-01 cache verification for each state; +3. manifest digest and archive-size attestation for both roots; +4. CF-03 structural diff using the attested root bytes; +5. CF-04 compatibility classification of that exact structural report; +6. CF-07 terminology evidence using those same verified package graphs; +7. CF-06 oracle comparison only for changed matched StructureDefinitions and only through the pinned adapter boundary; +8. deterministic case-summary emission plus raw sub-report evidence. -1. exact before/after package resolution into isolated caches; -2. digest and archive-size attestation against the frozen manifest; -3. CF-01 cache verification; -4. CF-03 structural diff; -5. CF-04 classification; -6. CF-07 terminology diff; -7. CF-06 oracle comparison for changed comparable StructureDefinition pairs using the pinned local adapter; -8. deterministic case-summary emission. +The runner MUST reuse those canonical authorities rather than reimplementing their semantics. -A case failure is represented explicitly. The runner MUST NOT skip a selected case because of unsupported content, a commandF/oracle disagreement, a network/package failure, or an unexpected finding count. +A case failure is represented explicitly. The runner MUST NOT silently skip a selected case because of unsupported content, a commandF/oracle disagreement, network/package failure, attestation failure, or unexpected finding count. ## Result contract @@ -195,7 +251,7 @@ case_status `case_status` is operational/evidence state only and MUST NOT rewrite CF-04/05 compatibility truth. -V1 summary output MUST be deterministic for the same manifest, pinned packages, commandF build, and pinned oracle. +V1 summary output MUST be deterministic for the same canonical manifest, pinned packages, commandF build, and pinned oracle. It MUST NOT serialize timestamps, host-absolute paths, random ids, network timing, temporary cache paths, or unordered-map iteration. ## No golden-answer fabrication @@ -203,9 +259,9 @@ Before the first real corpus execution, CF-10 MUST NOT hard-code expected struct After the first independently verified run, exact result hashes/counts MAY be frozen as regression evidence only if: -- the raw commandF outputs are preserved in CI artifacts for review; -- a second clean run is byte-identical; -- the expected values are derived from the observed deterministic run, not edited to make tests pass; +- raw commandF sub-reports are preserved in CI artifacts for review; +- a second clean run is byte-identical at the deterministic summary layer; +- expected values are derived from the observed deterministic run, not edited to make tests pass; - future differences fail visibly and require explicit corpus-version reconciliation. ## Failure semantics @@ -225,19 +281,6 @@ CF-10 fails closed on: Failures are never silently converted to corpus exclusion or pass. -## Determinism and reproducibility - -The corpus must not serialize: - -- timestamps; -- host-absolute paths; -- random ids; -- network timing; -- temporary cache paths; -- unordered map iteration. - -Case order is lexicographic by stable case id. Aggregate maps are canonically ordered. - ## Security boundary Corpus metadata is untrusted input. The implementation must bound manifest/result sizes and counts, reject path-like package/version tricks where a package identity is expected, use explicit subprocess paths for the oracle boundary, and inherit CF-01/CF-06 process/cache hardening rather than bypass it. @@ -251,7 +294,8 @@ A converged CF-10 candidate requires: - frozen selection methodology before result discovery; - exact package digest discovery using two independent clean resolutions; - provenance/rights evidence for every case; -- schema and failure-path tests; +- typed manifest and failure-path tests; +- mandatory package attestation before semantic execution; - deterministic two-run corpus equality; - `cargo fmt --all -- --check`; - locked workspace Clippy with `-D warnings`; From 39b9db7653826681b5d1a1a2ccb0cf7c0dd2e66f Mon Sep 17 00:00:00 2001 From: "Abdulaziz M. Shehri" Date: Sat, 15 Aug 2026 18:20:39 +0300 Subject: [PATCH 29/82] docs(cf10): lock corpus execution plan --- specs/010-cf-10-real-ig-delta-corpus/plan.md | 269 ++++++++----------- 1 file changed, 110 insertions(+), 159 deletions(-) diff --git a/specs/010-cf-10-real-ig-delta-corpus/plan.md b/specs/010-cf-10-real-ig-delta-corpus/plan.md index b205c89b..5823f2ac 100644 --- a/specs/010-cf-10-real-ig-delta-corpus/plan.md +++ b/specs/010-cf-10-real-ig-delta-corpus/plan.md @@ -1,23 +1,21 @@ # CF-10 Implementation Plan — Public Real-IG Delta Corpus -Status: planned / selection rules frozen +Status: implementation in progress / eligibility and digest freeze complete ## Architecture decision CF-10 is a benchmark/evidence layer above existing commandF authorities. It does not add a new semantic engine. -The implementation is split into four boundaries: +The implementation is split into four hard boundaries: 1. **Corpus manifest model** — validates frozen case metadata, digests, rights/provenance evidence, ordering, and v1 R4 constraints. -2. **Acquisition/attestation** — uses CF-01 resolver/cache verification in isolated state; package bytes are ephemeral and never committed. -3. **Case evaluation** — invokes the existing CF-03/04/07 evidence stack and CF-06 oracle adapter without modifying their rules. +2. **Acquisition/attestation** — uses CF-01 resolver/cache verification in isolated state and refuses semantic work until digest/size attestation succeeds. +3. **Case evaluation** — invokes existing CF-03/04/07 evidence and CF-06 oracle behavior without changing their rules. 4. **Deterministic summary** — emits compact commandF-owned benchmark evidence and explicit operational failure states. -The first executable gate is digest discovery. Result counts/severities are deliberately unknown until after the cases and package identities are frozen. +## Phase A — Frozen selection and provenance — COMPLETE -## Phase A — Freeze selection and provenance - -Canonical v1 cases are fixed before result discovery: +Canonical v1 cases remain fixed: ```text C001 hl7.fhir.us.core 8.0.1 -> 9.0.0 @@ -25,95 +23,94 @@ C002 hl7.fhir.uv.ips 1.1.0 -> 2.0.1 C003 hl7.fhir.us.mcode 3.0.0 -> 4.0.0 ``` -Selection is based on published/stable R4 status, public package identity, explicit version/change evidence, and domain diversity. commandF output is not a selection input. - -The donor/provenance record must distinguish: +Selection was frozen before commandF semantic result discovery. No case or version has been replaced. -- publication metadata rights; -- package/runtime access; -- terminology/IP statements inside the IG; -- repository redistribution rights. +The donor/provenance record distinguishes publication/change/rights evidence and keeps repository mode metadata-only. No upstream package payload is committed. -CF-10 repository mode is metadata-only. No upstream NPM tarball is checked in. +## Phase B — Foundation reconciliation and digest discovery — COMPLETE -## Phase B — Digest discovery +CF-11 multi-version graph support is canonical at merge commit: -Add a one-shot/guarded GitHub Actions discovery workflow or equivalent isolated execution tool that: +```text +5cb1a4c3445c0ebd86654cfb467a5e008e801c3e +``` -1. checks out the exact CF-10 planning head; -2. resolves every selected exact package version into cache A; -3. verifies cache A; -4. resolves the same package/version into independent cache B; -5. verifies cache B; -6. requires archive bytes and SHA-256 to match A == B; -7. records byte size and digest in a machine-readable artifact; -8. fails if any selected version is unavailable or non-reproducible. +It was reconciled into the frozen CF-10 branch by: -The discovery artifact is reviewed before digest metadata is copied into the canonical corpus manifest. +```text +5ec463f0ae53b76f9c2c151335d98598b53e5abc +``` -No expected commandF semantic result is generated or frozen in this phase. +The same six frozen package states were then resolved twice from independent clean caches and verified through CF-01. `cf10-digest-discovery` run `31890014888` succeeded and artifact `9248341586` was reviewed before digest/size metadata was frozen in `corpus/real-ig/v1/corpus.json`. Later exact-head rerun `31890859039` also succeeded. -## Phase C — Manifest model +No expected semantic result was generated or frozen in this phase. -Preferred ownership: `commandf-pkg` typed model and validation, not ad-hoc shell parsing. +## Phase C — Manifest model — COMPLETE -Proposed modules: +Ownership is `commandf-pkg` typed code: ```text crates/commandf-pkg/src/corpus_model.rs crates/commandf-pkg/src/corpus_error.rs crates/commandf-pkg/src/corpus.rs +crates/commandf-pkg/tests/corpus_manifest.rs ``` -Public model concepts: - -```text -RealIgCorpus -CorpusSelectionPolicy -RealIgCase -CorpusPackageState -CorpusRightsEvidence -CorpusOracleMode -``` - -Validation requirements: +Implemented validation includes: - schema exactly 1; -- bounded input bytes and case count; -- stable unique case ids; -- canonical lexicographic case ordering; -- exact package names/versions; +- bounded pre-decode input bytes and bounded case count; +- stable unique lexicographically ordered case ids; +- exact package names and semver versions; - R4 `4.0.1` only in v1; -- SHA-256 lowercase/64-hex; +- lowercase 64-hex SHA-256; - positive bounded archive size; - before != after version; -- publication/change/rights evidence URLs required; -- `metadata_only_no_redistribution` only in v1; -- `changed_structure_definitions_only` oracle mode only in v1; -- unknown enum/schema values fail closed. +- publication/change/rights evidence required; +- closed rights/oracle enums; +- unknown fields fail closed; +- deterministic canonical JSON round trip; +- canonical manifest assertions against the reviewed frozen digests/sizes. + +The manifest parser performs no network access. + +## Phase D — Package attestation — COMPLETE -The model does not fetch the network. +Reusable package-state attestation is owned by `commandf-pkg` and enforces this order: -## Phase D — Deterministic evaluator +```text +Lockfile::verify_cache(whole graph) +-> exact (package, version) selection +-> locked digest == manifest digest +-> verified target archive read +-> archive bytes and SHA-256 == manifest +``` -Add a thin evaluator that consumes an already validated manifest and explicit package state. +Regression tests cover matching states, manifest digest mismatch, size mismatch, corrupted target cache, missing/ambiguous exact lock identity, and an unrelated unverified lock entry blocking the target attestation. -Do not introduce another matcher/classifier/terminology engine/oracle implementation. +This makes CF-01 verification a mandatory authority rather than a caller convention. + +## Phase E — Deterministic evaluator — NEXT + +Add a thin evaluator that consumes a validated `RealIgCase` plus already resolved/attested package states. For each case: -1. require attested before/after archive bytes matching manifest digest/size; -2. call `diff_package_archives`; -3. call canonical compatibility classification; -4. call canonical terminology diff against the same verified states; -5. call the CF-06 oracle path only for changed matched StructureDefinitions; -6. reduce those reports into deterministic aggregate counts without losing the underlying report identities/hashes. +1. obtain attested before/after root bytes from verified state; +2. call `diff_package_archives` (CF-03); +3. call canonical compatibility classification (CF-04); +4. call canonical terminology evidence against the same verified lock/cache states (CF-07); +5. call the CF-06 oracle boundary only for changed matched StructureDefinitions; +6. reduce canonical sub-reports into deterministic aggregate counts without changing their semantic rules; +7. retain raw sub-report bytes/hashes as evidence for CI artifacts. -Raw detailed reports remain CI artifacts; the committed/public corpus summary remains compact commandF-owned evidence. +Do not introduce another matcher, classifier, terminology engine, or oracle implementation. -## Phase E — User-visible execution surface +Unit tests use synthetic/local package fixtures. Ordinary tests must not require public network, Java, or Maven. -Prefer one narrow CLI rather than benchmark shell glue: +## Phase F — Execution surface — DECIDED + +CF-10 v1 exposes exactly one corpus execution surface: ```text commandf corpus run \ @@ -124,13 +121,21 @@ commandf corpus run \ --format json ``` -The command may perform acquisition because corpus execution is explicitly an integration/benchmark operation, but acquisition and evaluation remain internally separated so digest verification happens before semantic analysis. +Do not add a second public repository harness. + +The CLI may acquire packages because this is an explicit integration/benchmark operation, but acquisition and semantic evaluation remain separated internally: + +```text +acquire -> CF-01 verify -> manifest attest -> evaluate +``` + +A selected case must never be silently skipped. Operational failure is represented explicitly and causes the real-corpus gate to fail while preserving evidence. -Alternative if implementation pressure reveals the CLI is unnecessary: keep the typed evaluator public in `commandf-pkg` and use a repository-owned executable harness. This decision must be made before implementation and reflected in `spec.md`; do not silently create both. +## Phase G — Deterministic summary -## Phase F — Summary schema +V1 summary reuses existing vocabulary where possible and contains no timestamps/absolute paths/random ids/network timing/temp paths. -Proposed deterministic v1 summary: +Logical shape: ```text schema @@ -140,132 +145,78 @@ cases[] package before { version, sha256 } after { version, sha256 } - structural - changes - compatibility - findings - breaking - risky - additive - producer - consumer - both - terminology - code_system_changes - value_set_changes - binding_refinements - oracle - compared - agreement - commandf_only - authority_only - both_changed - uncomparable + structural { changes } + compatibility { findings, breaking, risky, additive, producer, consumer, both } + terminology { code_system_changes, value_set_changes, binding_refinements } + oracle { compared, agreement, commandf_only, authority_only, both_changed, uncomparable } status ``` -Exact field vocabulary must reuse existing public enum names where possible rather than invent parallel terms. - -The summary does not claim clinical safety, semantic equivalence, or universal benchmark coverage. - -## Phase G — First-result freeze - -After implementation and digest lock: - -1. run the entire v1 corpus from a clean environment; -2. preserve raw reports as CI artifact; -3. repeat from another clean cache; -4. require byte-identical deterministic summary; -5. review disagreements/unsupported evidence as findings, not failures to hide; -6. only then freeze exact summary digest/counts as a regression baseline in corpus v1 metadata or a separate expected-results file. - -Any future upstream-package byte change at the same version is a hard attestation failure, not an automatic baseline update. +Exact counting rules must be direct reductions over canonical report fields, not new compatibility policy. -## CI layout +## Phase H — Real corpus workflow and first-result freeze -Preserve current `ci` and `cf06-oracle` workflows. - -Add a dedicated corpus job/workflow with explicit resource/time bounds because three public IG pairs plus the Java oracle are heavier than ordinary unit tests. - -Recommended stages: +Add one bounded dedicated workflow for the three frozen cases. Required stages: ```text manifest-validation -package-attestation +acquisition-and-attestation structural-classification-terminology oracle-evidence -repeat-determinism -summary-verification +repeat-clean-run +deterministic-summary-equality +artifact-upload +repository-payload-scan ``` -CI must upload raw reports and the deterministic summary as short-retention review artifacts. Artifacts are evidence, not repository-vendored benchmark data. - -## Test strategy - -### Manifest tests +The workflow must: -- wrong schema; -- empty corpus; -- too many cases; -- duplicate/out-of-order ids; -- malformed package/version; -- same before/after version; -- non-R4 FHIR version; -- malformed digest/size; -- missing publication/change/rights metadata; -- unsupported rights/oracle mode; -- deterministic JSON round trip. +1. run all three cases from a clean work root; +2. upload raw structural/compatibility/terminology/oracle reports plus deterministic summary as short-retention artifacts; +3. run the full corpus a second time from another clean work root/cache; +4. require byte-identical deterministic summary output; +5. preserve failures/divergences as evidence rather than removing cases; +6. verify no upstream IG tarball/terminology payload enters git history; +7. only after two-run equality, freeze observed summary hash/count regression evidence. -### Attestation tests +Any upstream byte change at the same package/version is a hard attestation failure, not an automatic baseline update. -Use synthetic/local package fixtures for unit tests: - -- matching digest/size passes; -- digest mismatch fails; -- size mismatch fails; -- corrupted cache fails; -- before/after state cannot alias unexpectedly. - -### Evaluation tests +## Test strategy -Use small synthetic packages to verify aggregation equals canonical underlying reports. Do not make ordinary unit tests depend on public network or Java/Maven. +Manifest tests and attestation tests are already implemented as described above. -### Real integration tests +Evaluator tests must use small synthetic packages and prove aggregation equals canonical underlying reports. Add explicit failure-path tests for malformed/unsupported sub-reports and no silent case removal. -The dedicated CF-10 job uses the three frozen public cases and the pinned CF-06 adapter. +Real integration testing belongs only in the dedicated CF-10 workflow. ## Rights controls No package bytes enter git history. -The corpus manifest records upstream legal/IP evidence but does not collapse mixed terminology rights into a single permissive license label. In particular, IPS contains terminology with separate upstream licensing statements; metadata-only benchmarking avoids redistribution and does not grant downstream terminology rights. - -Any future proposal to publish/download-bundle the source packages as a commandF benchmark dataset is a separate founder/legal authorization gate and is not CF-10. +The corpus manifest records upstream legal/IP evidence conservatively and does not collapse mixed terminology rights into one permissive license label. Publishing or bundling source packages as a dataset is outside CF-10 and requires a separate founder/legal authorization gate. ## Reviewer priorities 1. no benchmark cherry-picking after result discovery; -2. no semantic leakage from benchmark expectations into CF-03/04/07/06; +2. no semantic leakage into CF-03/04/07/06; 3. no package-content redistribution; 4. exact digest attestation before analysis; 5. no silent skipped cases; 6. deterministic result bytes; 7. bounded untrusted manifest/package metadata; -8. rights evidence remains conservative and source-specific; +8. source-specific rights evidence; 9. real integration failures remain visible evidence; -10. no golden expected result authored before first verified run. +10. no golden expected result authored before first verified two-run proof. ## Convergence condition CF-10 converges only after: -- selection policy and case set remain unchanged from the pre-result spec unless an eligibility/right fact is proven false; -- exact digests/sizes are independently discovered and frozen; -- all three cases resolve and attest; -- typed manifest/evaluator tests pass; -- real corpus run completes without silent case removal; -- two clean runs produce byte-identical summary; -- current `ci` and `cf06-oracle` remain green; -- dedicated corpus workflow is green on the exact final head; +- frozen selection remains unchanged; +- exact digests/sizes remain frozen from reviewed independent discovery; +- typed manifest/attestation/evaluator tests pass; +- all three real cases execute without silent removal; +- two clean runs produce byte-identical deterministic summary; +- `ci`, `cf06-oracle`, CF-11 proof, and dedicated corpus workflow are green on the exact final head; - substantive reviewer findings are fixed or explicitly rejected with evidence; -- `spec.md`, `plan.md`, `tasks.md`, donor record, corpus manifest, and `convergence.md` agree on the final truth. +- `spec.md`, `plan.md`, `tasks.md`, donor record, corpus manifest, implementation, and `convergence.md` agree on final truth. From 4bf5aebbd8c17f1a1ef63eb021419ab51246ea28 Mon Sep 17 00:00:00 2001 From: "Abdulaziz M. Shehri" Date: Sat, 15 Aug 2026 18:21:00 +0300 Subject: [PATCH 30/82] docs(cf10): reconcile completed foundation and attestation tasks --- specs/010-cf-10-real-ig-delta-corpus/tasks.md | 50 ++++++++++--------- 1 file changed, 26 insertions(+), 24 deletions(-) diff --git a/specs/010-cf-10-real-ig-delta-corpus/tasks.md b/specs/010-cf-10-real-ig-delta-corpus/tasks.md index 3acebe48..d1a8d7b2 100644 --- a/specs/010-cf-10-real-ig-delta-corpus/tasks.md +++ b/specs/010-cf-10-real-ig-delta-corpus/tasks.md @@ -1,49 +1,51 @@ # CF-10 Tasks — Public Real-IG Delta Corpus -Status: canonical CF-11 foundation reconciled / frozen six-state eligibility rerun authorized +Status: implementation in progress / foundation, eligibility, manifest, and attestation gates complete ## Foundation reconciliation gate - [x] CF-11 multi-version package graph is canonical on `main` at merge commit `5cb1a4c3445c0ebd86654cfb467a5e008e801c3e`. - [x] Canonical CF-11 was merged into this frozen CF-10 branch by reconciliation merge `5ec463f0ae53b76f9c2c151335d98598b53e5abc`. -- [x] The five pre-existing CF-10 paths remained byte-identical across reconciliation before this authorization update. +- [x] The five pre-existing CF-10 paths remained byte-identical across reconciliation before authorization updates. - [x] Frozen selection remains exactly: US Core 8.0.1→9.0.0, IPS 1.1.0→2.0.1, mCODE 3.0.0→4.0.0. -- [ ] Rerun the exact same six package states through the existing digest-discovery workflow on the reconciled foundation. -- [ ] Review the resulting artifact before authorizing any semantic diff/classify/check/terminology/oracle corpus execution. +- [x] Rerun the exact same six package states through the existing digest-discovery workflow on the reconciled foundation. +- [x] Review the resulting artifact before authorizing semantic corpus execution. -No semantic CF-10 execution is authorized by this task update. An eligibility failure must remain visible and must not cause case replacement. +Eligibility evidence: `cf10-digest-discovery` run `31890014888`, reviewed artifact `9248341586`, with all six states independently A/B-attested. Exact-head rerun `31890859039` also passed. ## Selection and provenance - [x] T001 Freeze anti-cherry-picking eligibility rules before result discovery. - [x] T002 Freeze corpus v1 families: US Core, IPS, mCODE. - [x] T003 Freeze exact candidate version pairs: US Core 8.0.1→9.0.0, IPS 1.1.0→2.0.1, mCODE 3.0.0→4.0.0. -- [ ] T004 Record publication/change/rights evidence for every case in donor/provenance metadata. +- [x] T004 Record publication/change/rights evidence for every case in donor/provenance metadata. - [ ] T005 Confirm repository mode is metadata-only and no package/terminology payload is committed. ## Digest discovery - [x] T006 Add guarded exact-version digest-discovery workflow/tool. -- [ ] T007 Resolve every selected package/version into independent cache A and cache B. -- [ ] T008 Verify both caches with CF-01. -- [ ] T009 Require byte/digest equality between independent resolutions. -- [ ] T010 Record package archive SHA-256 and byte size in discovery artifact. -- [ ] T011 Review discovery artifact and freeze digests/sizes in canonical corpus manifest. +- [x] T007 Resolve every selected package/version into independent cache A and cache B. +- [x] T008 Verify both caches with CF-01. +- [x] T009 Require byte/digest equality between independent resolutions. +- [x] T010 Record package archive SHA-256 and byte size in discovery artifact. +- [x] T011 Review discovery artifact and freeze digests/sizes in canonical corpus manifest. ## Manifest implementation -- [ ] T012 Add typed corpus schema-v1 model. -- [ ] T013 Add bounded pre-decode manifest input validation. -- [ ] T014 Validate unique lexicographically ordered case ids. -- [ ] T015 Validate exact package/version, R4 4.0.1, SHA-256, positive archive size, rights and publication evidence. -- [ ] T016 Reject unsupported schema/rights/oracle modes and malformed evidence. -- [ ] T017 Add deterministic JSON round-trip tests. +- [x] T012 Add typed corpus schema-v1 model. +- [x] T013 Add bounded pre-decode manifest input validation. +- [x] T014 Validate unique lexicographically ordered case ids. +- [x] T015 Validate exact package/version, R4 4.0.1, SHA-256, positive archive size, rights and publication evidence. +- [x] T016 Reject unsupported schema/rights/oracle modes and malformed evidence. +- [x] T017 Add deterministic JSON round-trip tests. ## Package attestation -- [ ] T018 Add reusable before/after archive attestation against manifest digest and size. -- [ ] T019 Add digest mismatch and size mismatch fail-closed tests. -- [ ] T020 Preserve CF-01 cache verification as mandatory authority. +- [x] T018 Add reusable before/after archive attestation against manifest digest and size. +- [x] T019 Add digest mismatch and size mismatch fail-closed tests. +- [x] T020 Preserve CF-01 cache verification as mandatory authority. + +T020 is enforced in code: attestation verifies the complete supplied lock graph before selecting/reading the target state; a corrupt or missing unrelated lock entry blocks attestation. ## Corpus evaluator @@ -56,7 +58,7 @@ No semantic CF-10 execution is authorized by this task update. An eligibility fa ## Execution surface -- [ ] T027 Decide and document one execution surface: narrow `commandf corpus run` CLI or repository-owned typed harness. +- [x] T027 Decide and document one execution surface: narrow `commandf corpus run` CLI only. - [ ] T028 Implement only the selected surface. - [ ] T029 Keep acquisition and semantic evaluation internally separated by digest verification. @@ -70,8 +72,8 @@ No semantic CF-10 execution is authorized by this task update. An eligibility fa ## CI and security -- [ ] T035 Preserve current `ci` workflow gates. -- [ ] T036 Preserve current `cf06-oracle` workflow gates. +- [ ] T035 Preserve current `ci` workflow gates through the exact final head. +- [ ] T036 Preserve current `cf06-oracle` workflow gates through the exact final head. - [ ] T037 Add bounded dedicated real-corpus workflow/job. - [ ] T038 Keep quoted argv/no `eval`; corpus metadata must never become shell code. - [ ] T039 Bound manifest bytes/case count/result bytes and fail closed on malformed input. @@ -84,4 +86,4 @@ No semantic CF-10 execution is authorized by this task update. An eligibility fa - [ ] T043 Request configured independent/Codex review when available; no unavailable PASS may be invented. - [ ] T044 Reconcile `spec.md`, `plan.md`, `tasks.md`, donor record, manifest, and implementation truth. - [ ] T045 Add `convergence.md` with exact final-head CI/reviewer evidence rule. -- [ ] T046 Keep PR open/unmerged until exact final head passes ordinary CI, oracle CI, and dedicated corpus CI. +- [ ] T046 Keep PR open/unmerged until exact final head passes ordinary CI, oracle CI, multi-version proof, and dedicated corpus CI. From 3333ddff721b83a456bbef023d6026daf0c52f30 Mon Sep 17 00:00:00 2001 From: "Abdulaziz M. Shehri" Date: Sat, 15 Aug 2026 18:22:59 +0300 Subject: [PATCH 31/82] feat(cf10): add deterministic corpus summary model --- crates/commandf-pkg/src/corpus_model.rs | 95 +++++++++++++++++++++++++ 1 file changed, 95 insertions(+) diff --git a/crates/commandf-pkg/src/corpus_model.rs b/crates/commandf-pkg/src/corpus_model.rs index f5cf87ac..8ec96665 100644 --- a/crates/commandf-pkg/src/corpus_model.rs +++ b/crates/commandf-pkg/src/corpus_model.rs @@ -67,3 +67,98 @@ pub struct CorpusPackageAttestation { pub sha256: String, pub archive_bytes: u64, } + +#[derive(Clone, Debug, Eq, PartialEq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct CorpusSummaryPackageState { + pub version: String, + pub sha256: String, +} + +#[derive(Clone, Copy, Debug, Eq, PartialEq, Serialize, Deserialize)] +#[serde(rename_all = "snake_case")] +pub enum CorpusCaseStatus { + Complete, + AcquisitionFailed, + AttestationFailed, + StructuralFailed, + CompatibilityFailed, + TerminologyFailed, + OracleFailed, +} + +#[derive(Clone, Debug, Eq, PartialEq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct CorpusStructuralSummary { + pub changes: usize, + pub report_sha256: String, +} + +#[derive(Clone, Debug, Eq, PartialEq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct CorpusCompatibilitySummary { + pub findings: usize, + pub breaking: usize, + pub risky: usize, + pub additive: usize, + pub producer: usize, + pub consumer: usize, + pub report_sha256: String, +} + +#[derive(Clone, Debug, Eq, PartialEq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct CorpusTerminologySummary { + pub code_system_changes: usize, + pub value_set_changes: usize, + pub binding_refinements: usize, + pub report_sha256: String, +} + +#[derive(Clone, Debug, Eq, PartialEq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct CorpusOracleSummary { + pub compared: usize, + pub agreement: usize, + pub commandf_only: usize, + pub authority_only: usize, + pub both_changed: usize, + pub uncomparable: usize, + pub report_sha256: String, +} + +#[derive(Clone, Debug, Eq, PartialEq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct CorpusCaseSummary { + pub case_id: String, + pub package: String, + pub before: CorpusSummaryPackageState, + pub after: CorpusSummaryPackageState, + pub status: CorpusCaseStatus, + #[serde(skip_serializing_if = "Option::is_none")] + pub structural: Option, + #[serde(skip_serializing_if = "Option::is_none")] + pub compatibility: Option, + #[serde(skip_serializing_if = "Option::is_none")] + pub terminology: Option, + #[serde(skip_serializing_if = "Option::is_none")] + pub oracle: Option, +} + +#[derive(Clone, Debug, Eq, PartialEq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct CorpusRunSummary { + pub schema: u32, + pub manifest_sha256: String, + pub cases: Vec, +} + +impl CorpusRunSummary { + pub const SCHEMA_V1: u32 = 1; + + pub fn to_json_bytes(&self) -> Result, serde_json::Error> { + let mut bytes = serde_json::to_vec_pretty(self)?; + bytes.push(b'\n'); + Ok(bytes) + } +} From 83c5c2f25e3a9f67aa756172eaaa71690db5c90c Mon Sep 17 00:00:00 2001 From: "Abdulaziz M. Shehri" Date: Sat, 15 Aug 2026 18:23:14 +0300 Subject: [PATCH 32/82] feat(cf10): add corpus evaluation failures --- crates/commandf-pkg/src/corpus_error.rs | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/crates/commandf-pkg/src/corpus_error.rs b/crates/commandf-pkg/src/corpus_error.rs index fcc09406..1ee49f81 100644 --- a/crates/commandf-pkg/src/corpus_error.rs +++ b/crates/commandf-pkg/src/corpus_error.rs @@ -84,6 +84,22 @@ pub enum CorpusError { expected: String, found: String, }, + #[error("case {case_id} {stage} evaluation failed: {message}")] + Evaluation { + case_id: String, + stage: &'static str, + message: String, + }, + #[error("case {case_id} {report} report identity does not match the corpus case")] + ReportIdentityMismatch { + case_id: String, + report: &'static str, + }, + #[error("case {case_id} {report} report uses unsupported schema/ruleset")] + UnsupportedReport { + case_id: String, + report: &'static str, + }, #[error("canonical corpus serialization failed: {0}")] Serialization(String), } From 16c6f890797cb8d9ab805f3684c1c1663e61dff4 Mon Sep 17 00:00:00 2001 From: "Abdulaziz M. Shehri" Date: Sat, 15 Aug 2026 18:23:47 +0300 Subject: [PATCH 33/82] feat(cf10): add canonical corpus evaluator --- crates/commandf-pkg/src/corpus_evaluate.rs | 288 +++++++++++++++++++++ 1 file changed, 288 insertions(+) create mode 100644 crates/commandf-pkg/src/corpus_evaluate.rs diff --git a/crates/commandf-pkg/src/corpus_evaluate.rs b/crates/commandf-pkg/src/corpus_evaluate.rs new file mode 100644 index 00000000..0154e48d --- /dev/null +++ b/crates/commandf-pkg/src/corpus_evaluate.rs @@ -0,0 +1,288 @@ +use crate::{ + attest_corpus_package_state, build_terminology_diff_report, classify_structural_diff, + diff_package_archives, CompatibilityDirection, CompatibilityReport, CompatibilitySeverity, + CorpusCaseStatus, CorpusCaseSummary, CorpusCompatibilitySummary, CorpusError, + CorpusOracleSummary, CorpusPackageSide, CorpusStructuralSummary, CorpusSummaryPackageState, + CorpusTerminologySummary, Hl7OracleReport, Lockfile, OracleDivergenceReport, OracleIdentity, + OracleResourceStatus, PackageCache, RealIgCase, StructuralDiffReport, TerminologyDiffReport, + TerminologyPackageState, +}; + +pub struct CorpusPackageStateInput<'a> { + pub lockfile: &'a Lockfile, + pub cache: &'a PackageCache, +} + +#[derive(Clone, Debug, Eq, PartialEq)] +pub struct CorpusCaseReports { + pub structural: StructuralDiffReport, + pub compatibility: CompatibilityReport, + pub terminology: TerminologyDiffReport, +} + +pub fn evaluate_corpus_case( + case: &RealIgCase, + before: CorpusPackageStateInput<'_>, + after: CorpusPackageStateInput<'_>, +) -> Result { + attest_corpus_package_state(case, CorpusPackageSide::Before, before.lockfile, before.cache)?; + attest_corpus_package_state(case, CorpusPackageSide::After, after.lockfile, after.cache)?; + + let before_bytes = before + .cache + .read_verified(&case.before.archive_sha256) + .map_err(|error| CorpusError::CacheVerification { + case_id: case.id.clone(), + message: error.to_string(), + })?; + let after_bytes = after + .cache + .read_verified(&case.after.archive_sha256) + .map_err(|error| CorpusError::CacheVerification { + case_id: case.id.clone(), + message: error.to_string(), + })?; + + let structural = diff_package_archives( + case.package.clone(), + case.before.version.clone(), + case.before.archive_sha256.clone(), + &before_bytes, + case.after.version.clone(), + case.after.archive_sha256.clone(), + &after_bytes, + ) + .map_err(|error| CorpusError::Evaluation { + case_id: case.id.clone(), + stage: "structural", + message: error.to_string(), + })?; + + let compatibility = + classify_structural_diff(&structural).map_err(|error| CorpusError::Evaluation { + case_id: case.id.clone(), + stage: "compatibility", + message: error.to_string(), + })?; + + let terminology = build_terminology_diff_report( + TerminologyPackageState { + lockfile: before.lockfile, + cache: before.cache, + root_bytes: &before_bytes, + }, + TerminologyPackageState { + lockfile: after.lockfile, + cache: after.cache, + root_bytes: &after_bytes, + }, + &structural, + &compatibility, + ) + .map_err(|error| CorpusError::Evaluation { + case_id: case.id.clone(), + stage: "terminology", + message: error.to_string(), + })?; + + Ok(CorpusCaseReports { + structural, + compatibility, + terminology, + }) +} + +pub fn summarize_corpus_case( + case: &RealIgCase, + reports: &CorpusCaseReports, + oracle: &OracleDivergenceReport, +) -> Result { + validate_report_identity(case, reports, oracle)?; + + let structural_bytes = reports + .structural + .to_json_bytes() + .map_err(|error| serialization_error(case, "structural", error))?; + let compatibility_bytes = reports + .compatibility + .to_json_bytes() + .map_err(|error| serialization_error(case, "compatibility", error))?; + let terminology_bytes = reports + .terminology + .to_json_bytes() + .map_err(|error| serialization_error(case, "terminology", error))?; + let oracle_bytes = oracle + .to_json_bytes() + .map_err(|error| serialization_error(case, "oracle", error))?; + + let mut breaking = 0usize; + let mut risky = 0usize; + let mut additive = 0usize; + let mut producer = 0usize; + let mut consumer = 0usize; + for finding in &reports.compatibility.findings { + match finding.severity { + CompatibilitySeverity::Breaking => breaking += 1, + CompatibilitySeverity::Risky => risky += 1, + CompatibilitySeverity::Additive => additive += 1, + } + match finding.direction { + CompatibilityDirection::Producer => producer += 1, + CompatibilityDirection::Consumer => consumer += 1, + } + } + + let mut agreement = 0usize; + let mut commandf_only = 0usize; + let mut authority_only = 0usize; + let mut both_changed = 0usize; + let mut uncomparable = 0usize; + for resource in &oracle.resources { + match resource.status { + OracleResourceStatus::Agreement => agreement += 1, + OracleResourceStatus::CommandfOnly => commandf_only += 1, + OracleResourceStatus::AuthorityOnly => authority_only += 1, + OracleResourceStatus::BothChanged => both_changed += 1, + OracleResourceStatus::Uncomparable => uncomparable += 1, + } + } + + Ok(CorpusCaseSummary { + case_id: case.id.clone(), + package: case.package.clone(), + before: summary_state(&case.before.version, &case.before.archive_sha256), + after: summary_state(&case.after.version, &case.after.archive_sha256), + status: CorpusCaseStatus::Complete, + structural: Some(CorpusStructuralSummary { + changes: reports.structural.changes.len(), + report_sha256: PackageCache::digest(&structural_bytes), + }), + compatibility: Some(CorpusCompatibilitySummary { + findings: reports.compatibility.findings.len(), + breaking, + risky, + additive, + producer, + consumer, + report_sha256: PackageCache::digest(&compatibility_bytes), + }), + terminology: Some(CorpusTerminologySummary { + code_system_changes: reports.terminology.code_systems.len(), + value_set_changes: reports.terminology.value_sets.len(), + binding_refinements: reports.terminology.binding_refinements.len(), + report_sha256: PackageCache::digest(&terminology_bytes), + }), + oracle: Some(CorpusOracleSummary { + compared: oracle.resources.len(), + agreement, + commandf_only, + authority_only, + both_changed, + uncomparable, + report_sha256: PackageCache::digest(&oracle_bytes), + }), + }) +} + +pub fn failed_corpus_case_summary(case: &RealIgCase, status: CorpusCaseStatus) -> CorpusCaseSummary { + CorpusCaseSummary { + case_id: case.id.clone(), + package: case.package.clone(), + before: summary_state(&case.before.version, &case.before.archive_sha256), + after: summary_state(&case.after.version, &case.after.archive_sha256), + status, + structural: None, + compatibility: None, + terminology: None, + oracle: None, + } +} + +fn validate_report_identity( + case: &RealIgCase, + reports: &CorpusCaseReports, + oracle: &OracleDivergenceReport, +) -> Result<(), CorpusError> { + if reports.structural.schema != StructuralDiffReport::SCHEMA_V1 { + return Err(unsupported(case, "structural")); + } + if reports.structural.package_name != case.package + || reports.structural.before.version != case.before.version + || reports.structural.before.archive_sha256 != case.before.archive_sha256 + || reports.structural.after.version != case.after.version + || reports.structural.after.archive_sha256 != case.after.archive_sha256 + { + return Err(identity(case, "structural")); + } + + if reports.compatibility.schema != CompatibilityReport::SCHEMA_V1 + || reports.compatibility.ruleset != CompatibilityReport::RULESET_V1 + { + return Err(unsupported(case, "compatibility")); + } + if reports.compatibility.package_name != case.package + || reports.compatibility.before != reports.structural.before + || reports.compatibility.after != reports.structural.after + { + return Err(identity(case, "compatibility")); + } + + if reports.terminology.schema != TerminologyDiffReport::SCHEMA_V1 + || reports.terminology.ruleset != TerminologyDiffReport::RULESET_V1 + { + return Err(unsupported(case, "terminology")); + } + if reports.terminology.package_name != case.package + || reports.terminology.before != reports.structural.before + || reports.terminology.after != reports.structural.after + || reports.terminology.compatibility != reports.compatibility + { + return Err(identity(case, "terminology")); + } + + if oracle.schema != OracleDivergenceReport::SCHEMA_V1 || oracle.oracle != OracleIdentity::pinned_hl7() + { + return Err(unsupported(case, "oracle")); + } + if oracle.package_name != case.package || oracle.structural_diff != reports.structural { + return Err(identity(case, "oracle")); + } + + Ok(()) +} + +fn summary_state(version: &str, sha256: &str) -> CorpusSummaryPackageState { + CorpusSummaryPackageState { + version: version.to_owned(), + sha256: sha256.to_owned(), + } +} + +fn identity(case: &RealIgCase, report: &'static str) -> CorpusError { + CorpusError::ReportIdentityMismatch { + case_id: case.id.clone(), + report, + } +} + +fn unsupported(case: &RealIgCase, report: &'static str) -> CorpusError { + CorpusError::UnsupportedReport { + case_id: case.id.clone(), + report, + } +} + +fn serialization_error( + case: &RealIgCase, + stage: &'static str, + error: serde_json::Error, +) -> CorpusError { + CorpusError::Evaluation { + case_id: case.id.clone(), + stage, + message: error.to_string(), + } +} + +#[allow(dead_code)] +fn _assert_oracle_report_type(_: &Hl7OracleReport) {} From a6ca271e1c82b764fd9c6237f93a442624ebef14 Mon Sep 17 00:00:00 2001 From: "Abdulaziz M. Shehri" Date: Sat, 15 Aug 2026 18:24:26 +0300 Subject: [PATCH 34/82] refactor(cf10): clean corpus evaluator imports --- crates/commandf-pkg/src/corpus_evaluate.rs | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/crates/commandf-pkg/src/corpus_evaluate.rs b/crates/commandf-pkg/src/corpus_evaluate.rs index 0154e48d..3f42d667 100644 --- a/crates/commandf-pkg/src/corpus_evaluate.rs +++ b/crates/commandf-pkg/src/corpus_evaluate.rs @@ -3,7 +3,7 @@ use crate::{ diff_package_archives, CompatibilityDirection, CompatibilityReport, CompatibilitySeverity, CorpusCaseStatus, CorpusCaseSummary, CorpusCompatibilitySummary, CorpusError, CorpusOracleSummary, CorpusPackageSide, CorpusStructuralSummary, CorpusSummaryPackageState, - CorpusTerminologySummary, Hl7OracleReport, Lockfile, OracleDivergenceReport, OracleIdentity, + CorpusTerminologySummary, Lockfile, OracleDivergenceReport, OracleIdentity, OracleResourceStatus, PackageCache, RealIgCase, StructuralDiffReport, TerminologyDiffReport, TerminologyPackageState, }; @@ -240,7 +240,8 @@ fn validate_report_identity( return Err(identity(case, "terminology")); } - if oracle.schema != OracleDivergenceReport::SCHEMA_V1 || oracle.oracle != OracleIdentity::pinned_hl7() + if oracle.schema != OracleDivergenceReport::SCHEMA_V1 + || oracle.oracle != OracleIdentity::pinned_hl7() { return Err(unsupported(case, "oracle")); } @@ -283,6 +284,3 @@ fn serialization_error( message: error.to_string(), } } - -#[allow(dead_code)] -fn _assert_oracle_report_type(_: &Hl7OracleReport) {} From aa944bed643941f5d1c269955373190bbd84b702 Mon Sep 17 00:00:00 2001 From: "Abdulaziz M. Shehri" Date: Sat, 15 Aug 2026 18:24:41 +0300 Subject: [PATCH 35/82] feat(cf10): export corpus evaluator APIs --- crates/commandf-pkg/src/lib.rs | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/crates/commandf-pkg/src/lib.rs b/crates/commandf-pkg/src/lib.rs index a953939f..33879194 100644 --- a/crates/commandf-pkg/src/lib.rs +++ b/crates/commandf-pkg/src/lib.rs @@ -21,6 +21,7 @@ mod compatibility_model; mod compatibility_validate; mod corpus; mod corpus_error; +mod corpus_evaluate; mod corpus_model; mod error; mod lock; @@ -72,9 +73,15 @@ pub use corpus::{ MAX_CORPUS_MANIFEST_BYTES, }; pub use corpus_error::CorpusError; +pub use corpus_evaluate::{ + evaluate_corpus_case, failed_corpus_case_summary, summarize_corpus_case, CorpusCaseReports, + CorpusPackageStateInput, +}; pub use corpus_model::{ - CorpusOracleMode, CorpusPackageAttestation, CorpusPackageSide, CorpusPackageState, - CorpusRightsMode, CorpusSelectionPolicy, RealIgCase, RealIgCorpus, + CorpusCaseStatus, CorpusCaseSummary, CorpusCompatibilitySummary, CorpusOracleMode, + CorpusOracleSummary, CorpusPackageAttestation, CorpusPackageSide, CorpusPackageState, + CorpusRightsMode, CorpusRunSummary, CorpusSelectionPolicy, CorpusStructuralSummary, + CorpusSummaryPackageState, CorpusTerminologySummary, RealIgCase, RealIgCorpus, }; pub use error::PackageError; pub use lock::{LockedPackage, Lockfile}; From dd5542ad32eecfe3d21eb0ac7fd4602c46dab7b3 Mon Sep 17 00:00:00 2001 From: "Abdulaziz M. Shehri" Date: Sat, 15 Aug 2026 18:25:21 +0300 Subject: [PATCH 36/82] test(cf10): cover canonical corpus evaluation and summary --- crates/commandf-pkg/tests/corpus_evaluator.rs | 232 ++++++++++++++++++ 1 file changed, 232 insertions(+) create mode 100644 crates/commandf-pkg/tests/corpus_evaluator.rs diff --git a/crates/commandf-pkg/tests/corpus_evaluator.rs b/crates/commandf-pkg/tests/corpus_evaluator.rs new file mode 100644 index 00000000..66cf13b8 --- /dev/null +++ b/crates/commandf-pkg/tests/corpus_evaluator.rs @@ -0,0 +1,232 @@ +use std::collections::BTreeMap; +use std::io::Cursor; + +use commandf_pkg::{ + evaluate_corpus_case, failed_corpus_case_summary, summarize_corpus_case, CorpusCaseStatus, + CorpusError, CorpusOracleMode, CorpusPackageState, CorpusPackageStateInput, CorpusRightsMode, + CorpusRunSummary, LockedPackage, Lockfile, OracleDivergenceReport, OracleIdentity, PackageCache, + RealIgCase, +}; +use flate2::write::GzEncoder; +use flate2::Compression; +use tar::{Builder, Header}; +use tempfile::TempDir; + +fn archive(name: &str, version: &str, patient: &[u8]) -> Vec { + let manifest = format!( + r#"{{"name":"{name}","version":"{version}","dependencies":{{}}}}"# + ); + let entries = [ + ("package/package.json", manifest.as_bytes()), + ("package/Patient-example.json", patient), + ]; + + let mut encoder = GzEncoder::new(Vec::new(), Compression::default()); + { + let mut builder = Builder::new(&mut encoder); + for (path, body) in entries { + let mut header = Header::new_gnu(); + header.set_path(path).unwrap(); + header.set_size(body.len() as u64); + header.set_mode(0o644); + header.set_cksum(); + builder.append(&header, Cursor::new(body)).unwrap(); + } + builder.finish().unwrap(); + } + encoder.finish().unwrap() +} + +fn locked(name: &str, version: &str, sha256: &str) -> LockedPackage { + LockedPackage { + name: name.to_owned(), + version: version.to_owned(), + sha256: sha256.to_owned(), + source: format!("https://packages.example.org/{name}/{version}"), + dependencies: BTreeMap::new(), + } +} + +struct Fixture { + _before_temp: TempDir, + _after_temp: TempDir, + before_cache: PackageCache, + after_cache: PackageCache, + before_lock: Lockfile, + after_lock: Lockfile, + case: RealIgCase, +} + +fn fixture() -> Fixture { + let before_temp = TempDir::new().unwrap(); + let after_temp = TempDir::new().unwrap(); + let before_cache = PackageCache::new(before_temp.path()); + let after_cache = PackageCache::new(after_temp.path()); + + let before_bytes = archive( + "example.pkg", + "1.0.0", + br#"{"resourceType":"Patient","id":"example","active":true}"#, + ); + let after_bytes = archive( + "example.pkg", + "2.0.0", + br#"{"resourceType":"Patient","id":"example","active":false}"#, + ); + let before_sha = before_cache.put(&before_bytes).unwrap(); + let after_sha = after_cache.put(&after_bytes).unwrap(); + + let before_lock = Lockfile::new( + vec!["example.pkg@1.0.0".to_owned()], + vec![locked("example.pkg", "1.0.0", &before_sha)], + ); + let after_lock = Lockfile::new( + vec!["example.pkg@2.0.0".to_owned()], + vec![locked("example.pkg", "2.0.0", &after_sha)], + ); + + let case = RealIgCase { + id: "C001".to_owned(), + package: "example.pkg".to_owned(), + before: CorpusPackageState { + version: "1.0.0".to_owned(), + archive_sha256: before_sha, + archive_bytes: before_bytes.len() as u64, + publication_url: "https://example.org/before".to_owned(), + }, + after: CorpusPackageState { + version: "2.0.0".to_owned(), + archive_sha256: after_sha, + archive_bytes: after_bytes.len() as u64, + publication_url: "https://example.org/after".to_owned(), + }, + fhir_version: "4.0.1".to_owned(), + publisher: "Example Publisher".to_owned(), + change_evidence_url: "https://example.org/changes".to_owned(), + rights_evidence_url: "https://example.org/rights".to_owned(), + rights_mode: CorpusRightsMode::MetadataOnlyNoRedistribution, + oracle_mode: CorpusOracleMode::ChangedStructureDefinitionsOnly, + }; + + Fixture { + _before_temp: before_temp, + _after_temp: after_temp, + before_cache, + after_cache, + before_lock, + after_lock, + case, + } +} + +#[test] +fn evaluator_reuses_canonical_reports_and_summary_hashes_them() { + let fixture = fixture(); + let reports = evaluate_corpus_case( + &fixture.case, + CorpusPackageStateInput { + lockfile: &fixture.before_lock, + cache: &fixture.before_cache, + }, + CorpusPackageStateInput { + lockfile: &fixture.after_lock, + cache: &fixture.after_cache, + }, + ) + .unwrap(); + + assert!(!reports.structural.changes.is_empty()); + assert_eq!(reports.compatibility.package_name, fixture.case.package); + assert_eq!(reports.terminology.package_name, fixture.case.package); + assert!(reports.terminology.code_systems.is_empty()); + assert!(reports.terminology.value_sets.is_empty()); + assert!(reports.terminology.binding_refinements.is_empty()); + + let oracle = OracleDivergenceReport { + schema: OracleDivergenceReport::SCHEMA_V1, + oracle: OracleIdentity::pinned_hl7(), + package_name: fixture.case.package.clone(), + structural_diff: reports.structural.clone(), + resources: Vec::new(), + }; + let summary = summarize_corpus_case(&fixture.case, &reports, &oracle).unwrap(); + + assert_eq!(summary.status, CorpusCaseStatus::Complete); + assert_eq!( + summary.structural.as_ref().unwrap().changes, + reports.structural.changes.len() + ); + assert_eq!( + summary.structural.as_ref().unwrap().report_sha256, + PackageCache::digest(&reports.structural.to_json_bytes().unwrap()) + ); + assert_eq!( + summary.compatibility.as_ref().unwrap().report_sha256, + PackageCache::digest(&reports.compatibility.to_json_bytes().unwrap()) + ); + assert_eq!( + summary.terminology.as_ref().unwrap().report_sha256, + PackageCache::digest(&reports.terminology.to_json_bytes().unwrap()) + ); + assert_eq!( + summary.oracle.as_ref().unwrap().report_sha256, + PackageCache::digest(&oracle.to_json_bytes().unwrap()) + ); + assert_eq!(summary.oracle.as_ref().unwrap().compared, 0); +} + +#[test] +fn summary_rejects_cross_case_oracle_identity() { + let fixture = fixture(); + let reports = evaluate_corpus_case( + &fixture.case, + CorpusPackageStateInput { + lockfile: &fixture.before_lock, + cache: &fixture.before_cache, + }, + CorpusPackageStateInput { + lockfile: &fixture.after_lock, + cache: &fixture.after_cache, + }, + ) + .unwrap(); + let oracle = OracleDivergenceReport { + schema: OracleDivergenceReport::SCHEMA_V1, + oracle: OracleIdentity::pinned_hl7(), + package_name: "other.pkg".to_owned(), + structural_diff: reports.structural.clone(), + resources: Vec::new(), + }; + + assert_eq!( + summarize_corpus_case(&fixture.case, &reports, &oracle), + Err(CorpusError::ReportIdentityMismatch { + case_id: "C001".to_owned(), + report: "oracle", + }) + ); +} + +#[test] +fn deterministic_run_summary_has_no_failure_detail_or_paths() { + let fixture = fixture(); + let failed = failed_corpus_case_summary(&fixture.case, CorpusCaseStatus::OracleFailed); + assert!(failed.structural.is_none()); + assert!(failed.compatibility.is_none()); + assert!(failed.terminology.is_none()); + assert!(failed.oracle.is_none()); + + let report = CorpusRunSummary { + schema: CorpusRunSummary::SCHEMA_V1, + manifest_sha256: "a".repeat(64), + cases: vec![failed], + }; + let first = report.to_json_bytes().unwrap(); + let second = report.to_json_bytes().unwrap(); + assert_eq!(first, second); + + let text = String::from_utf8(first).unwrap(); + assert!(text.contains("oracle_failed")); + assert!(!text.contains("/tmp")); + assert!(!text.contains("\\\\")); +} From 6e5c08c6b0453e7b027ee7147f5a2330640b02c8 Mon Sep 17 00:00:00 2001 From: "Abdulaziz M. Shehri" Date: Sat, 15 Aug 2026 18:26:27 +0300 Subject: [PATCH 37/82] style(cf10): format corpus evaluator --- crates/commandf-pkg/src/corpus_evaluate.rs | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/crates/commandf-pkg/src/corpus_evaluate.rs b/crates/commandf-pkg/src/corpus_evaluate.rs index 3f42d667..7da923c6 100644 --- a/crates/commandf-pkg/src/corpus_evaluate.rs +++ b/crates/commandf-pkg/src/corpus_evaluate.rs @@ -25,7 +25,12 @@ pub fn evaluate_corpus_case( before: CorpusPackageStateInput<'_>, after: CorpusPackageStateInput<'_>, ) -> Result { - attest_corpus_package_state(case, CorpusPackageSide::Before, before.lockfile, before.cache)?; + attest_corpus_package_state( + case, + CorpusPackageSide::Before, + before.lockfile, + before.cache, + )?; attest_corpus_package_state(case, CorpusPackageSide::After, after.lockfile, after.cache)?; let before_bytes = before @@ -184,7 +189,10 @@ pub fn summarize_corpus_case( }) } -pub fn failed_corpus_case_summary(case: &RealIgCase, status: CorpusCaseStatus) -> CorpusCaseSummary { +pub fn failed_corpus_case_summary( + case: &RealIgCase, + status: CorpusCaseStatus, +) -> CorpusCaseSummary { CorpusCaseSummary { case_id: case.id.clone(), package: case.package.clone(), From 36f5731997750ac57bfe184d18a46ba6457fd3b6 Mon Sep 17 00:00:00 2001 From: "Abdulaziz M. Shehri" Date: Sat, 15 Aug 2026 18:26:49 +0300 Subject: [PATCH 38/82] style(cf10): format corpus evaluator tests --- crates/commandf-pkg/tests/corpus_evaluator.rs | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/crates/commandf-pkg/tests/corpus_evaluator.rs b/crates/commandf-pkg/tests/corpus_evaluator.rs index 66cf13b8..8b3cfbb5 100644 --- a/crates/commandf-pkg/tests/corpus_evaluator.rs +++ b/crates/commandf-pkg/tests/corpus_evaluator.rs @@ -4,8 +4,8 @@ use std::io::Cursor; use commandf_pkg::{ evaluate_corpus_case, failed_corpus_case_summary, summarize_corpus_case, CorpusCaseStatus, CorpusError, CorpusOracleMode, CorpusPackageState, CorpusPackageStateInput, CorpusRightsMode, - CorpusRunSummary, LockedPackage, Lockfile, OracleDivergenceReport, OracleIdentity, PackageCache, - RealIgCase, + CorpusRunSummary, LockedPackage, Lockfile, OracleDivergenceReport, OracleIdentity, + PackageCache, RealIgCase, }; use flate2::write::GzEncoder; use flate2::Compression; @@ -13,9 +13,7 @@ use tar::{Builder, Header}; use tempfile::TempDir; fn archive(name: &str, version: &str, patient: &[u8]) -> Vec { - let manifest = format!( - r#"{{"name":"{name}","version":"{version}","dependencies":{{}}}}"# - ); + let manifest = format!(r#"{{"name":"{name}","version":"{version}","dependencies":{{}}}}"#); let entries = [ ("package/package.json", manifest.as_bytes()), ("package/Patient-example.json", patient), From f5292e31e733ca8175f86c6a0b1cab45cd7507f5 Mon Sep 17 00:00:00 2001 From: "Abdulaziz M. Shehri" Date: Sat, 15 Aug 2026 18:29:22 +0300 Subject: [PATCH 39/82] refactor(cf10): expose canonical oracle report --- crates/commandf-cli/src/oracle.rs | 27 ++++++++++++++++++++++++--- 1 file changed, 24 insertions(+), 3 deletions(-) diff --git a/crates/commandf-cli/src/oracle.rs b/crates/commandf-cli/src/oracle.rs index 791f88fc..a90d7db1 100644 --- a/crates/commandf-cli/src/oracle.rs +++ b/crates/commandf-cli/src/oracle.rs @@ -6,7 +6,8 @@ use std::time::Duration; use commandf_pkg::{ diff_package_archives, matched_structure_definition_pairs, reconcile_hl7_oracle, run_hl7_oracle_adapter, validate_hl7_oracle_adapter, Hl7OracleInvocation, LockedPackage, - Lockfile, PackageCache, PackageName, ResourceKey, ResourceKeyKind, DEFAULT_ORACLE_TIMEOUT_SECS, + Lockfile, OracleDivergenceReport, PackageCache, PackageName, ResourceKey, ResourceKeyKind, + DEFAULT_ORACLE_TIMEOUT_SECS, }; const ORACLE_CORE_PACKAGE: &str = "hl7.fhir.r4.core"; @@ -21,6 +22,27 @@ pub fn run( oracle_adapter: PathBuf, oracle_java: Option, ) -> Result, Box> { + Ok(run_report( + package, + before_lock, + before_cache, + after_lock, + after_cache, + oracle_adapter, + oracle_java, + )? + .to_json_bytes()?) +} + +pub fn run_report( + package: String, + before_lock: PathBuf, + before_cache: PathBuf, + after_lock: PathBuf, + after_cache: PathBuf, + oracle_adapter: PathBuf, + oracle_java: Option, +) -> Result> { let package_name = PackageName::parse(package)?; validate_hl7_oracle_adapter(&oracle_adapter, oracle_java.as_deref())?; @@ -101,8 +123,7 @@ pub fn run( } } - let report = reconcile_hl7_oracle(structural_diff, observations)?; - Ok(report.to_json_bytes()?) + Ok(reconcile_hl7_oracle(structural_diff, observations)?) } fn select_oracle_core(lockfile: &Lockfile) -> Result<&LockedPackage, io::Error> { From 18dfc62d987d072301f6fa15f8ea187acdbf506a Mon Sep 17 00:00:00 2001 From: "Abdulaziz M. Shehri" Date: Sat, 15 Aug 2026 18:31:30 +0300 Subject: [PATCH 40/82] feat(cf10): add changed-only oracle execution mode --- crates/commandf-cli/src/oracle.rs | 52 +++++++++++++++++++++++++++++++ 1 file changed, 52 insertions(+) diff --git a/crates/commandf-cli/src/oracle.rs b/crates/commandf-cli/src/oracle.rs index a90d7db1..ce563276 100644 --- a/crates/commandf-cli/src/oracle.rs +++ b/crates/commandf-cli/src/oracle.rs @@ -42,6 +42,50 @@ pub fn run_report( after_cache: PathBuf, oracle_adapter: PathBuf, oracle_java: Option, +) -> Result> { + run_report_inner( + package, + before_lock, + before_cache, + after_lock, + after_cache, + oracle_adapter, + oracle_java, + false, + ) +} + +pub fn run_changed_report( + package: String, + before_lock: PathBuf, + before_cache: PathBuf, + after_lock: PathBuf, + after_cache: PathBuf, + oracle_adapter: PathBuf, + oracle_java: Option, +) -> Result> { + run_report_inner( + package, + before_lock, + before_cache, + after_lock, + after_cache, + oracle_adapter, + oracle_java, + true, + ) +} + +#[allow(clippy::too_many_arguments)] +fn run_report_inner( + package: String, + before_lock: PathBuf, + before_cache: PathBuf, + after_lock: PathBuf, + after_cache: PathBuf, + oracle_adapter: PathBuf, + oracle_java: Option, + changed_only: bool, ) -> Result> { let package_name = PackageName::parse(package)?; validate_hl7_oracle_adapter(&oracle_adapter, oracle_java.as_deref())?; @@ -103,6 +147,14 @@ pub fn run_report( if pair.resource.kind != ResourceKeyKind::Canonical { continue; } + if changed_only + && !structural_diff + .changes + .iter() + .any(|change| change.resource == pair.resource) + { + continue; + } let (url, version) = canonical_parts(&pair.resource)?; let invocation = Hl7OracleInvocation { core_package: &core_archive, From f659482f866cea171879d84315efbaeb0115f101 Mon Sep 17 00:00:00 2001 From: "Abdulaziz M. Shehri" Date: Sat, 15 Aug 2026 18:32:12 +0300 Subject: [PATCH 41/82] feat(cf10): add bounded corpus execution orchestrator --- crates/commandf-cli/src/corpus.rs | 259 ++++++++++++++++++++++++++++++ 1 file changed, 259 insertions(+) create mode 100644 crates/commandf-cli/src/corpus.rs diff --git a/crates/commandf-cli/src/corpus.rs b/crates/commandf-cli/src/corpus.rs new file mode 100644 index 00000000..a197cc6e --- /dev/null +++ b/crates/commandf-cli/src/corpus.rs @@ -0,0 +1,259 @@ +use std::fs; +use std::io::{self, Read}; +use std::path::{Path, PathBuf}; + +use commandf_pkg::{ + evaluate_corpus_case, failed_corpus_case_summary, parse_corpus_manifest, + summarize_corpus_case, CorpusCaseStatus, CorpusError, CorpusPackageStateInput, + CorpusRunSummary, FhirRegistrySource, Lockfile, PackageCache, PackageRequest, RealIgCase, + Resolver, MAX_CORPUS_MANIFEST_BYTES, +}; + +use crate::oracle; + +const MAX_FAILURE_DIAGNOSTIC_CHARS: usize = 16_384; + +pub struct CorpusExecution { + pub summary: CorpusRunSummary, + pub failed: bool, +} + +struct ResolvedState { + cache_path: PathBuf, + lock_path: PathBuf, + cache: PackageCache, + lockfile: Lockfile, +} + +pub fn run( + manifest_path: PathBuf, + work_root: PathBuf, + oracle_adapter: PathBuf, + oracle_java: Option, +) -> Result> { + let manifest_bytes = read_bounded_file(&manifest_path, MAX_CORPUS_MANIFEST_BYTES as u64)?; + let corpus = parse_corpus_manifest(&manifest_bytes)?; + let manifest_sha256 = PackageCache::digest(&manifest_bytes); + + prepare_fresh_work_root(&work_root)?; + let evidence_root = work_root.join("evidence"); + fs::create_dir_all(&evidence_root)?; + + let mut summaries = Vec::with_capacity(corpus.cases.len()); + let mut failed = false; + + for case in &corpus.cases { + let evidence_dir = evidence_root.join(&case.id); + fs::create_dir_all(&evidence_dir)?; + + let before = resolve_state(case, true, &work_root); + let after = resolve_state(case, false, &work_root); + let (before, after) = match (before, after) { + (Ok(before), Ok(after)) => (before, after), + (before, after) => { + failed = true; + write_resolution_failures(&evidence_dir, before.err(), after.err())?; + summaries.push(failed_corpus_case_summary( + case, + CorpusCaseStatus::AcquisitionFailed, + )); + continue; + } + }; + + let reports = match evaluate_corpus_case( + case, + CorpusPackageStateInput { + lockfile: &before.lockfile, + cache: &before.cache, + }, + CorpusPackageStateInput { + lockfile: &after.lockfile, + cache: &after.cache, + }, + ) { + Ok(reports) => reports, + Err(error) => { + failed = true; + let status = evaluation_status(&error); + write_failure(&evidence_dir, "evaluation-failure.txt", &error)?; + summaries.push(failed_corpus_case_summary(case, status)); + continue; + } + }; + + fs::write( + evidence_dir.join("structural.json"), + reports.structural.to_json_bytes()?, + )?; + fs::write( + evidence_dir.join("compatibility.json"), + reports.compatibility.to_json_bytes()?, + )?; + fs::write( + evidence_dir.join("terminology.json"), + reports.terminology.to_json_bytes()?, + )?; + + let oracle_report = match oracle::run_changed_report( + case.package.clone(), + before.lock_path.clone(), + before.cache_path.clone(), + after.lock_path.clone(), + after.cache_path.clone(), + oracle_adapter.clone(), + oracle_java.clone(), + ) { + Ok(report) => report, + Err(error) => { + failed = true; + write_failure(&evidence_dir, "oracle-failure.txt", error.as_ref())?; + summaries.push(failed_corpus_case_summary( + case, + CorpusCaseStatus::OracleFailed, + )); + continue; + } + }; + fs::write( + evidence_dir.join("oracle.json"), + oracle_report.to_json_bytes()?, + )?; + + match summarize_corpus_case(case, &reports, &oracle_report) { + Ok(summary) => summaries.push(summary), + Err(error) => { + failed = true; + write_failure(&evidence_dir, "summary-failure.txt", &error)?; + summaries.push(failed_corpus_case_summary( + case, + CorpusCaseStatus::OracleFailed, + )); + } + } + } + + let summary = CorpusRunSummary { + schema: CorpusRunSummary::SCHEMA_V1, + manifest_sha256, + cases: summaries, + }; + fs::write(work_root.join("summary.json"), summary.to_json_bytes()?)?; + + Ok(CorpusExecution { summary, failed }) +} + +fn resolve_state( + case: &RealIgCase, + before: bool, + work_root: &Path, +) -> Result> { + let (side, state) = if before { + ("before", &case.before) + } else { + ("after", &case.after) + }; + let state_root = work_root.join("states").join(&case.id).join(side); + let cache_path = state_root.join("cache"); + let lock_path = state_root.join("commandf.lock"); + fs::create_dir_all(&state_root)?; + + let request = PackageRequest::parse(&format!("{}@{}", case.package, state.version))?; + let cache = PackageCache::new(&cache_path); + let lockfile = Resolver::new(&FhirRegistrySource::new(), &cache).resolve(vec![request])?; + lockfile.verify_cache(&cache)?; + fs::write(&lock_path, lockfile.to_bytes()?)?; + + Ok(ResolvedState { + cache_path, + lock_path, + cache, + lockfile, + }) +} + +fn prepare_fresh_work_root(work_root: &Path) -> io::Result<()> { + if work_root.exists() { + return Err(io::Error::new( + io::ErrorKind::AlreadyExists, + format!( + "corpus work root already exists; refusing to reuse or delete it: {}", + work_root.display() + ), + )); + } + fs::create_dir_all(work_root) +} + +fn read_bounded_file(path: &Path, max_bytes: u64) -> io::Result> { + let file = fs::File::open(path)?; + let mut bytes = Vec::new(); + file.take(max_bytes + 1).read_to_end(&mut bytes)?; + if bytes.len() as u64 > max_bytes { + return Err(io::Error::new( + io::ErrorKind::InvalidData, + format!("input exceeds {max_bytes} byte limit: {}", path.display()), + )); + } + Ok(bytes) +} + +fn evaluation_status(error: &CorpusError) -> CorpusCaseStatus { + match error { + CorpusError::Evaluation { stage, .. } => match *stage { + "structural" => CorpusCaseStatus::StructuralFailed, + "compatibility" => CorpusCaseStatus::CompatibilityFailed, + "terminology" => CorpusCaseStatus::TerminologyFailed, + _ => CorpusCaseStatus::TerminologyFailed, + }, + CorpusError::CacheVerification { .. } + | CorpusError::LockedPackageMissing { .. } + | CorpusError::LockedPackageAmbiguous { .. } + | CorpusError::LockedPackageDigestMismatch { .. } + | CorpusError::ArchiveSizeMismatch { .. } + | CorpusError::ArchiveDigestMismatch { .. } => CorpusCaseStatus::AttestationFailed, + _ => CorpusCaseStatus::AttestationFailed, + } +} + +fn write_resolution_failures( + evidence_dir: &Path, + before: Option>, + after: Option>, +) -> io::Result<()> { + let mut text = String::new(); + if let Some(error) = before { + text.push_str("before: "); + text.push_str(&bounded_diagnostic(error.as_ref())); + text.push('\n'); + } + if let Some(error) = after { + text.push_str("after: "); + text.push_str(&bounded_diagnostic(error.as_ref())); + text.push('\n'); + } + fs::write(evidence_dir.join("acquisition-failure.txt"), text) +} + +fn write_failure( + evidence_dir: &Path, + file_name: &str, + error: &dyn std::fmt::Display, +) -> io::Result<()> { + let mut text = bounded_diagnostic(error); + text.push('\n'); + fs::write(evidence_dir.join(file_name), text) +} + +fn bounded_diagnostic(error: &dyn std::fmt::Display) -> String { + let text = error.to_string(); + let mut chars = text.chars(); + let mut output = chars + .by_ref() + .take(MAX_FAILURE_DIAGNOSTIC_CHARS) + .collect::(); + if chars.next().is_some() { + output.push_str("… [diagnostic truncated]"); + } + output +} From 629c6b61885257836b9db8034da6a5e278f30355 Mon Sep 17 00:00:00 2001 From: "Abdulaziz M. Shehri" Date: Sat, 15 Aug 2026 18:34:16 +0300 Subject: [PATCH 42/82] feat(cf10): wire corpus run CLI --- crates/commandf-cli/src/main.rs | 40 +++++++++++++++++++++++++++++++++ 1 file changed, 40 insertions(+) diff --git a/crates/commandf-cli/src/main.rs b/crates/commandf-cli/src/main.rs index daa8244f..1452651f 100644 --- a/crates/commandf-cli/src/main.rs +++ b/crates/commandf-cli/src/main.rs @@ -1,3 +1,4 @@ +mod corpus; mod oracle; use std::ffi::OsStr; @@ -39,6 +40,10 @@ enum Command { #[command(subcommand)] command: PkgCommand, }, + Corpus { + #[command(subcommand)] + command: CorpusCommand, + }, Inspect { package: String, #[arg(long, default_value = ".commandf/cache")] @@ -214,6 +219,22 @@ enum PkgCommand { }, } +#[derive(Subcommand)] +enum CorpusCommand { + Run { + #[arg(long)] + manifest: PathBuf, + #[arg(long)] + work_root: PathBuf, + #[arg(long)] + oracle_adapter: PathBuf, + #[arg(long)] + oracle_java: Option, + #[arg(long, value_enum, default_value = "json")] + format: OutputFormat, + }, +} + fn main() -> ExitCode { let is_check = std::env::args_os().nth(1).as_deref() == Some(OsStr::new("check")); let cli = match Cli::try_parse() { @@ -273,6 +294,25 @@ fn run(cli: Cli) -> Result> { println!("verified {} packages", lockfile.packages.len()); } }, + Command::Corpus { command } => match command { + CorpusCommand::Run { + manifest, + work_root, + oracle_adapter, + oracle_java, + format, + } => { + let execution = + corpus::run(manifest, work_root, oracle_adapter, oracle_java)?; + let bytes = execution.summary.to_json_bytes()?; + match format { + OutputFormat::Json => io::stdout().write_all(&bytes)?, + } + if execution.failed { + return Ok(ExitCode::from(1)); + } + } + }, Command::Inspect { package, cache, From 00e3bfe3cc28cd277310d792798bbaaaf204b0e4 Mon Sep 17 00:00:00 2001 From: "Abdulaziz M. Shehri" Date: Sat, 15 Aug 2026 18:35:22 +0300 Subject: [PATCH 43/82] style(cf10): format corpus orchestrator --- crates/commandf-cli/src/corpus.rs | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/crates/commandf-cli/src/corpus.rs b/crates/commandf-cli/src/corpus.rs index a197cc6e..cc362787 100644 --- a/crates/commandf-cli/src/corpus.rs +++ b/crates/commandf-cli/src/corpus.rs @@ -3,10 +3,9 @@ use std::io::{self, Read}; use std::path::{Path, PathBuf}; use commandf_pkg::{ - evaluate_corpus_case, failed_corpus_case_summary, parse_corpus_manifest, - summarize_corpus_case, CorpusCaseStatus, CorpusError, CorpusPackageStateInput, - CorpusRunSummary, FhirRegistrySource, Lockfile, PackageCache, PackageRequest, RealIgCase, - Resolver, MAX_CORPUS_MANIFEST_BYTES, + evaluate_corpus_case, failed_corpus_case_summary, parse_corpus_manifest, summarize_corpus_case, + CorpusCaseStatus, CorpusError, CorpusPackageStateInput, CorpusRunSummary, FhirRegistrySource, + Lockfile, PackageCache, PackageRequest, RealIgCase, Resolver, MAX_CORPUS_MANIFEST_BYTES, }; use crate::oracle; From 9f7a16986bf3df4f13b015a333d8f850df65ceb8 Mon Sep 17 00:00:00 2001 From: "Abdulaziz M. Shehri" Date: Sat, 15 Aug 2026 18:36:10 +0300 Subject: [PATCH 44/82] style(cf10): format corpus CLI wiring --- crates/commandf-cli/src/main.rs | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/crates/commandf-cli/src/main.rs b/crates/commandf-cli/src/main.rs index 1452651f..a74730cf 100644 --- a/crates/commandf-cli/src/main.rs +++ b/crates/commandf-cli/src/main.rs @@ -302,8 +302,7 @@ fn run(cli: Cli) -> Result> { oracle_java, format, } => { - let execution = - corpus::run(manifest, work_root, oracle_adapter, oracle_java)?; + let execution = corpus::run(manifest, work_root, oracle_adapter, oracle_java)?; let bytes = execution.summary.to_json_bytes()?; match format { OutputFormat::Json => io::stdout().write_all(&bytes)?, From c27a465d2d21329f2cf3c2405143123c416fd5ab Mon Sep 17 00:00:00 2001 From: "Abdulaziz M. Shehri" Date: Sat, 15 Aug 2026 18:38:03 +0300 Subject: [PATCH 45/82] test(cf10): cover corpus CLI fail-closed boundaries --- crates/commandf-cli/tests/corpus_behavior.rs | 135 +++++++++++++++++++ 1 file changed, 135 insertions(+) create mode 100644 crates/commandf-cli/tests/corpus_behavior.rs diff --git a/crates/commandf-cli/tests/corpus_behavior.rs b/crates/commandf-cli/tests/corpus_behavior.rs new file mode 100644 index 00000000..13977db3 --- /dev/null +++ b/crates/commandf-cli/tests/corpus_behavior.rs @@ -0,0 +1,135 @@ +use std::fs; +use std::path::PathBuf; +use std::process::Command; +use std::time::{SystemTime, UNIX_EPOCH}; + +use commandf_pkg::MAX_CORPUS_MANIFEST_BYTES; + +fn commandf() -> Command { + Command::new(env!("CARGO_BIN_EXE_commandf")) +} + +fn unique_temp_dir(label: &str) -> PathBuf { + let nonce = SystemTime::now() + .duration_since(UNIX_EPOCH) + .expect("system clock must be after the Unix epoch") + .as_nanos(); + std::env::temp_dir().join(format!("commandf-{label}-{}-{nonce}", std::process::id())) +} + +fn valid_manifest() -> &'static [u8] { + br#"{ + "schema": 1, + "selection_policy": "frozen_pre_result_v1", + "cases": [ + { + "id": "C001", + "package": "example.package", + "before": { + "version": "1.0.0", + "archive_sha256": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "archive_bytes": 1, + "publication_url": "https://example.org/before" + }, + "after": { + "version": "2.0.0", + "archive_sha256": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + "archive_bytes": 1, + "publication_url": "https://example.org/after" + }, + "fhir_version": "4.0.1", + "publisher": "Example Publisher", + "change_evidence_url": "https://example.org/changes", + "rights_evidence_url": "https://example.org/rights", + "rights_mode": "metadata_only_no_redistribution", + "oracle_mode": "changed_structure_definitions_only" + } + ] +} +"# +} + +#[test] +fn corpus_run_help_is_available() { + let output = commandf() + .args(["corpus", "run", "--help"]) + .output() + .expect("commandf corpus run --help must execute"); + assert!(output.status.success()); + let stdout = String::from_utf8_lossy(&output.stdout); + assert!(stdout.contains("--manifest")); + assert!(stdout.contains("--work-root")); + assert!(stdout.contains("--oracle-adapter")); +} + +#[test] +fn oversized_manifest_fails_before_work_root_creation() { + let root = unique_temp_dir("corpus-oversized"); + fs::create_dir_all(&root).expect("create test root"); + let manifest = root.join("oversized.json"); + let work_root = root.join("work"); + fs::write(&manifest, vec![b' '; MAX_CORPUS_MANIFEST_BYTES + 1]).expect("write oversized manifest"); + + let output = commandf() + .args([ + "corpus", + "run", + "--manifest", + manifest.to_str().expect("UTF-8 path"), + "--work-root", + work_root.to_str().expect("UTF-8 path"), + "--oracle-adapter", + root.join("missing-oracle.jar").to_str().expect("UTF-8 path"), + "--format", + "json", + ]) + .env("HTTP_PROXY", "http://127.0.0.1:9") + .env("HTTPS_PROXY", "http://127.0.0.1:9") + .env("NO_PROXY", "") + .output() + .expect("commandf corpus run must execute"); + + assert_eq!(output.status.code(), Some(1)); + assert!(!work_root.exists()); + assert!(output.stdout.is_empty()); + let _ = fs::remove_dir_all(root); +} + +#[test] +fn existing_work_root_fails_before_acquisition() { + let root = unique_temp_dir("corpus-existing-root"); + let work_root = root.join("work"); + fs::create_dir_all(&work_root).expect("create existing work root"); + let marker = work_root.join("do-not-delete.txt"); + fs::write(&marker, b"preserve").expect("write marker"); + let manifest = root.join("corpus.json"); + fs::write(&manifest, valid_manifest()).expect("write manifest"); + let oracle = root.join("missing-oracle.jar"); + + let output = commandf() + .args([ + "corpus", + "run", + "--manifest", + manifest.to_str().expect("UTF-8 path"), + "--work-root", + work_root.to_str().expect("UTF-8 path"), + "--oracle-adapter", + oracle.to_str().expect("UTF-8 path"), + "--format", + "json", + ]) + .env("HTTP_PROXY", "http://127.0.0.1:9") + .env("HTTPS_PROXY", "http://127.0.0.1:9") + .env("NO_PROXY", "") + .output() + .expect("commandf corpus run must execute"); + + assert_eq!(output.status.code(), Some(1)); + assert!(marker.exists(), "existing work root content must be preserved"); + assert!(!oracle.exists(), "oracle must not be touched before work-root gate"); + assert!(output.stdout.is_empty()); + let stderr = String::from_utf8_lossy(&output.stderr); + assert!(stderr.contains("work root already exists")); + let _ = fs::remove_dir_all(root); +} From 6b66a28edc9670a343d33a05a85d8432943abe55 Mon Sep 17 00:00:00 2001 From: "Abdulaziz M. Shehri" Date: Sat, 15 Aug 2026 18:38:38 +0300 Subject: [PATCH 46/82] test(cf10): harden corpus CLI early-failure fixtures --- crates/commandf-cli/tests/corpus_behavior.rs | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/crates/commandf-cli/tests/corpus_behavior.rs b/crates/commandf-cli/tests/corpus_behavior.rs index 13977db3..ea9d220f 100644 --- a/crates/commandf-cli/tests/corpus_behavior.rs +++ b/crates/commandf-cli/tests/corpus_behavior.rs @@ -68,7 +68,9 @@ fn oversized_manifest_fails_before_work_root_creation() { fs::create_dir_all(&root).expect("create test root"); let manifest = root.join("oversized.json"); let work_root = root.join("work"); - fs::write(&manifest, vec![b' '; MAX_CORPUS_MANIFEST_BYTES + 1]).expect("write oversized manifest"); + let oracle = root.join("missing-oracle.jar"); + fs::write(&manifest, vec![b' '; MAX_CORPUS_MANIFEST_BYTES + 1]) + .expect("write oversized manifest"); let output = commandf() .args([ @@ -79,7 +81,7 @@ fn oversized_manifest_fails_before_work_root_creation() { "--work-root", work_root.to_str().expect("UTF-8 path"), "--oracle-adapter", - root.join("missing-oracle.jar").to_str().expect("UTF-8 path"), + oracle.to_str().expect("UTF-8 path"), "--format", "json", ]) From e3b89c6558532a1736da90ad4a9d19f04b2d528e Mon Sep 17 00:00:00 2001 From: "Abdulaziz M. Shehri" Date: Sat, 15 Aug 2026 18:39:31 +0300 Subject: [PATCH 47/82] fix(cf10): bound corpus result evidence bytes --- crates/commandf-cli/src/corpus.rs | 55 +++++++++++++++++++++++-------- 1 file changed, 42 insertions(+), 13 deletions(-) diff --git a/crates/commandf-cli/src/corpus.rs b/crates/commandf-cli/src/corpus.rs index cc362787..750c16e2 100644 --- a/crates/commandf-cli/src/corpus.rs +++ b/crates/commandf-cli/src/corpus.rs @@ -11,6 +11,8 @@ use commandf_pkg::{ use crate::oracle; const MAX_FAILURE_DIAGNOSTIC_CHARS: usize = 16_384; +const MAX_CORPUS_RAW_REPORT_BYTES: usize = 64 * 1024 * 1024; +const MAX_CORPUS_SUMMARY_BYTES: usize = 1024 * 1024; pub struct CorpusExecution { pub summary: CorpusRunSummary, @@ -81,17 +83,17 @@ pub fn run( } }; - fs::write( - evidence_dir.join("structural.json"), - reports.structural.to_json_bytes()?, + write_bounded_report( + &evidence_dir.join("structural.json"), + &reports.structural.to_json_bytes()?, )?; - fs::write( - evidence_dir.join("compatibility.json"), - reports.compatibility.to_json_bytes()?, + write_bounded_report( + &evidence_dir.join("compatibility.json"), + &reports.compatibility.to_json_bytes()?, )?; - fs::write( - evidence_dir.join("terminology.json"), - reports.terminology.to_json_bytes()?, + write_bounded_report( + &evidence_dir.join("terminology.json"), + &reports.terminology.to_json_bytes()?, )?; let oracle_report = match oracle::run_changed_report( @@ -114,9 +116,9 @@ pub fn run( continue; } }; - fs::write( - evidence_dir.join("oracle.json"), - oracle_report.to_json_bytes()?, + write_bounded_report( + &evidence_dir.join("oracle.json"), + &oracle_report.to_json_bytes()?, )?; match summarize_corpus_case(case, &reports, &oracle_report) { @@ -137,7 +139,19 @@ pub fn run( manifest_sha256, cases: summaries, }; - fs::write(work_root.join("summary.json"), summary.to_json_bytes()?)?; + let summary_bytes = summary.to_json_bytes()?; + if summary_bytes.len() > MAX_CORPUS_SUMMARY_BYTES { + return Err(io::Error::new( + io::ErrorKind::InvalidData, + format!( + "corpus summary is {} bytes; maximum is {}", + summary_bytes.len(), + MAX_CORPUS_SUMMARY_BYTES + ), + ) + .into()); + } + fs::write(work_root.join("summary.json"), summary_bytes)?; Ok(CorpusExecution { summary, failed }) } @@ -197,6 +211,21 @@ fn read_bounded_file(path: &Path, max_bytes: u64) -> io::Result> { Ok(bytes) } +fn write_bounded_report(path: &Path, bytes: &[u8]) -> io::Result<()> { + if bytes.len() > MAX_CORPUS_RAW_REPORT_BYTES { + return Err(io::Error::new( + io::ErrorKind::InvalidData, + format!( + "corpus report is {} bytes; maximum is {}: {}", + bytes.len(), + MAX_CORPUS_RAW_REPORT_BYTES, + path.display() + ), + )); + } + fs::write(path, bytes) +} + fn evaluation_status(error: &CorpusError) -> CorpusCaseStatus { match error { CorpusError::Evaluation { stage, .. } => match *stage { From 52721aa1b06c9f45a970eff698e47b34734c86a9 Mon Sep 17 00:00:00 2001 From: "Abdulaziz M. Shehri" Date: Sat, 15 Aug 2026 18:39:57 +0300 Subject: [PATCH 48/82] ci(cf10): add deterministic real corpus proof --- .github/workflows/cf10-real-corpus.yml | 120 +++++++++++++++++++++++++ 1 file changed, 120 insertions(+) create mode 100644 .github/workflows/cf10-real-corpus.yml diff --git a/.github/workflows/cf10-real-corpus.yml b/.github/workflows/cf10-real-corpus.yml new file mode 100644 index 00000000..aa1c9b55 --- /dev/null +++ b/.github/workflows/cf10-real-corpus.yml @@ -0,0 +1,120 @@ +name: cf10-real-corpus + +on: + pull_request: + paths: + - .github/workflows/cf10-real-corpus.yml + - corpus/real-ig/v1/** + - donors/cf-10-real-ig-delta-corpus.yaml + - specs/010-cf-10-real-ig-delta-corpus/** + - crates/commandf-pkg/** + - crates/commandf-cli/** + - tools/hl7-oracle/** + - Cargo.toml + - Cargo.lock + push: + branches: + - feat/cf-10-real-ig-delta-corpus + paths: + - .github/workflows/cf10-real-corpus.yml + - corpus/real-ig/v1/** + - donors/cf-10-real-ig-delta-corpus.yaml + - specs/010-cf-10-real-ig-delta-corpus/** + - crates/commandf-pkg/** + - crates/commandf-cli/** + - tools/hl7-oracle/** + - Cargo.toml + - Cargo.lock + workflow_dispatch: + +permissions: + contents: read + +jobs: + real-corpus: + runs-on: ubuntu-latest + timeout-minutes: 120 + steps: + - uses: actions/checkout@fbc6f3992d24b796d5a048ff273f7fcc4a7b6c09 # v5 / Node 24 + with: + persist-credentials: false + + - uses: actions/setup-java@v5.7.0 + with: + distribution: temurin + java-version: '17' + + - uses: dtolnay/rust-toolchain@032958afbdc797a9164d3bc0b56325c1308924a5 # 1.97.1 + + - name: Build pinned HL7 oracle adapter + run: mvn -B -ntp -f tools/hl7-oracle/pom.xml package + + - name: Build commandF corpus runner + run: cargo build --locked -p commandf + + - name: Run frozen corpus from clean state A + run: | + set -euo pipefail + test ! -e /tmp/cf10-real-a + ./target/debug/commandf corpus run \ + --manifest corpus/real-ig/v1/corpus.json \ + --work-root /tmp/cf10-real-a \ + --oracle-adapter tools/hl7-oracle/target/commandf-hl7-oracle.jar \ + --oracle-java "$JAVA_HOME/bin/java" \ + --format json \ + > /tmp/cf10-summary-a.json + cmp /tmp/cf10-summary-a.json /tmp/cf10-real-a/summary.json + + - name: Run frozen corpus from independent clean state B + run: | + set -euo pipefail + test ! -e /tmp/cf10-real-b + ./target/debug/commandf corpus run \ + --manifest corpus/real-ig/v1/corpus.json \ + --work-root /tmp/cf10-real-b \ + --oracle-adapter tools/hl7-oracle/target/commandf-hl7-oracle.jar \ + --oracle-java "$JAVA_HOME/bin/java" \ + --format json \ + > /tmp/cf10-summary-b.json + cmp /tmp/cf10-summary-b.json /tmp/cf10-real-b/summary.json + + - name: Require byte-identical deterministic summaries + run: | + set -euo pipefail + cmp /tmp/cf10-summary-a.json /tmp/cf10-summary-b.json + python - <<'PY' + import json + from pathlib import Path + + summary = json.loads(Path('/tmp/cf10-summary-a.json').read_text()) + assert summary['schema'] == 1 + assert [case['case_id'] for case in summary['cases']] == ['C001', 'C002', 'C003'] + assert all(case['status'] == 'complete' for case in summary['cases']) + assert len(summary['manifest_sha256']) == 64 + print(Path('/tmp/cf10-summary-a.json').read_text()) + PY + + - name: Assert metadata-only repository boundary + run: | + set -euo pipefail + test -z "$(git status --porcelain)" + if git ls-files | grep -E '\.(tgz|tar\.gz)$'; then + echo 'unexpected upstream archive tracked in repository' >&2 + exit 1 + fi + test -z "$(find corpus donors specs -type f \( -name '*.tgz' -o -name '*.tar.gz' \) -print)" + + - name: Upload short-retention corpus evidence + if: always() + uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4 + with: + name: cf10-real-corpus-evidence + path: | + /tmp/cf10-summary-a.json + /tmp/cf10-summary-b.json + /tmp/cf10-real-a/summary.json + /tmp/cf10-real-b/summary.json + /tmp/cf10-real-a/evidence + /tmp/cf10-real-b/evidence + if-no-files-found: warn + retention-days: 3 From b3daf4d38f2cedc29a7ce45cce99f7c2d5da9646 Mon Sep 17 00:00:00 2001 From: "Abdulaziz M. Shehri" Date: Sat, 15 Aug 2026 18:40:46 +0300 Subject: [PATCH 49/82] style(cf10): format corpus CLI boundary tests --- crates/commandf-cli/tests/corpus_behavior.rs | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/crates/commandf-cli/tests/corpus_behavior.rs b/crates/commandf-cli/tests/corpus_behavior.rs index ea9d220f..6e80cdf5 100644 --- a/crates/commandf-cli/tests/corpus_behavior.rs +++ b/crates/commandf-cli/tests/corpus_behavior.rs @@ -128,8 +128,14 @@ fn existing_work_root_fails_before_acquisition() { .expect("commandf corpus run must execute"); assert_eq!(output.status.code(), Some(1)); - assert!(marker.exists(), "existing work root content must be preserved"); - assert!(!oracle.exists(), "oracle must not be touched before work-root gate"); + assert!( + marker.exists(), + "existing work root content must be preserved" + ); + assert!( + !oracle.exists(), + "oracle must not be touched before work-root gate" + ); assert!(output.stdout.is_empty()); let stderr = String::from_utf8_lossy(&output.stderr); assert!(stderr.contains("work root already exists")); From c97bb4b1ed4680c042b81b0f0239e66fa38aa83a Mon Sep 17 00:00:00 2001 From: "Abdulaziz M. Shehri" Date: Sat, 15 Aug 2026 18:43:00 +0300 Subject: [PATCH 50/82] ci(cf10): pin Java setup action --- .github/workflows/cf10-real-corpus.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/cf10-real-corpus.yml b/.github/workflows/cf10-real-corpus.yml index aa1c9b55..202c8123 100644 --- a/.github/workflows/cf10-real-corpus.yml +++ b/.github/workflows/cf10-real-corpus.yml @@ -39,7 +39,7 @@ jobs: with: persist-credentials: false - - uses: actions/setup-java@v5.7.0 + - uses: actions/setup-java@b6effb05e454b25005698d916606bdc6ffcbf961 # v5.7.0 with: distribution: temurin java-version: '17' From e5111169017098ce28cbf699b1be4dbc0db2d97d Mon Sep 17 00:00:00 2001 From: "Abdulaziz M. Shehri" Date: Sat, 15 Aug 2026 18:44:37 +0300 Subject: [PATCH 51/82] fix(cf10): count actual oracle comparisons --- crates/commandf-pkg/src/corpus_evaluate.rs | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/crates/commandf-pkg/src/corpus_evaluate.rs b/crates/commandf-pkg/src/corpus_evaluate.rs index 7da923c6..8555434d 100644 --- a/crates/commandf-pkg/src/corpus_evaluate.rs +++ b/crates/commandf-pkg/src/corpus_evaluate.rs @@ -137,6 +137,11 @@ pub fn summarize_corpus_case( } } + let compared = oracle + .resources + .iter() + .filter(|resource| resource.oracle.is_some()) + .count(); let mut agreement = 0usize; let mut commandf_only = 0usize; let mut authority_only = 0usize; @@ -178,7 +183,7 @@ pub fn summarize_corpus_case( report_sha256: PackageCache::digest(&terminology_bytes), }), oracle: Some(CorpusOracleSummary { - compared: oracle.resources.len(), + compared, agreement, commandf_only, authority_only, From d4813db0492e01854c5a74a5103eea7b1bcd1a1b Mon Sep 17 00:00:00 2001 From: "Abdulaziz M. Shehri" Date: Sat, 15 Aug 2026 18:45:14 +0300 Subject: [PATCH 52/82] test(cf10): distinguish compared from uncomparable oracle evidence --- crates/commandf-pkg/tests/corpus_evaluator.rs | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/crates/commandf-pkg/tests/corpus_evaluator.rs b/crates/commandf-pkg/tests/corpus_evaluator.rs index 8b3cfbb5..65500988 100644 --- a/crates/commandf-pkg/tests/corpus_evaluator.rs +++ b/crates/commandf-pkg/tests/corpus_evaluator.rs @@ -5,7 +5,8 @@ use commandf_pkg::{ evaluate_corpus_case, failed_corpus_case_summary, summarize_corpus_case, CorpusCaseStatus, CorpusError, CorpusOracleMode, CorpusPackageState, CorpusPackageStateInput, CorpusRightsMode, CorpusRunSummary, LockedPackage, Lockfile, OracleDivergenceReport, OracleIdentity, - PackageCache, RealIgCase, + OracleResourceResult, OracleResourceStatus, PackageCache, RealIgCase, ResourceKey, + ResourceKeyKind, }; use flate2::write::GzEncoder; use flate2::Compression; @@ -145,7 +146,15 @@ fn evaluator_reuses_canonical_reports_and_summary_hashes_them() { oracle: OracleIdentity::pinned_hl7(), package_name: fixture.case.package.clone(), structural_diff: reports.structural.clone(), - resources: Vec::new(), + resources: vec![OracleResourceResult { + resource: ResourceKey { + kind: ResourceKeyKind::ResourceId, + value: "Patient/example".to_owned(), + }, + status: OracleResourceStatus::Uncomparable, + oracle: None, + commandf_change_kinds: Vec::new(), + }], }; let summary = summarize_corpus_case(&fixture.case, &reports, &oracle).unwrap(); @@ -171,6 +180,7 @@ fn evaluator_reuses_canonical_reports_and_summary_hashes_them() { PackageCache::digest(&oracle.to_json_bytes().unwrap()) ); assert_eq!(summary.oracle.as_ref().unwrap().compared, 0); + assert_eq!(summary.oracle.as_ref().unwrap().uncomparable, 1); } #[test] From 6efea1a09e33679e7575b594c7576b5980b917a7 Mon Sep 17 00:00:00 2001 From: "Abdulaziz M. Shehri" Date: Sat, 15 Aug 2026 18:47:33 +0300 Subject: [PATCH 53/82] fix(cf07): scope terminology closure canonical indexing --- crates/commandf-pkg/src/terminology_index.rs | 189 ++++++++++++++++--- 1 file changed, 158 insertions(+), 31 deletions(-) diff --git a/crates/commandf-pkg/src/terminology_index.rs b/crates/commandf-pkg/src/terminology_index.rs index fb5dc694..c1a19e74 100644 --- a/crates/commandf-pkg/src/terminology_index.rs +++ b/crates/commandf-pkg/src/terminology_index.rs @@ -1,11 +1,11 @@ -use std::collections::BTreeMap; +use std::collections::{BTreeMap, BTreeSet}; use std::fs; -use serde_json::Value; +use serde_json::{Map, Value}; use crate::{ - archive::read_manifest, artifact_scan::scan_package_resources, inspect_package, Lockfile, - PackageCache, PackageError, TerminologyError, + archive::read_manifest, artifact_scan::scan_package_resources, Lockfile, PackageCache, + PackageError, TerminologyError, }; const TERMINOLOGY_TYPES: [&str; 2] = ["CodeSystem", "ValueSet"]; @@ -53,48 +53,45 @@ impl TerminologyClosure { }); } - let inspection = - inspect_package(&package.name, &package.version, &package.sha256, &bytes)?; - let mut raw = BTreeMap::new(); - for resource in scan_package_resources(&bytes)? { - let filename = resource.filename; - let value = serde_json::from_slice(&resource.bytes).map_err(|source| { - TerminologyError::Json { - file: filename.clone(), - source, - } - })?; - if raw.insert(filename.clone(), value).is_some() { + let mut seen_filenames = BTreeSet::new(); + for scanned in scan_package_resources(&bytes)? { + let filename = scanned.filename; + if !seen_filenames.insert(filename.clone()) { return Err(TerminologyError::InvalidField { resource: format!("{}@{}", package.name, package.version), field: filename, message: "duplicate package resource filename".to_owned(), }); } - } - for resource in inspection.resources { - if !TERMINOLOGY_TYPES.contains(&resource.resource_type.as_str()) { + let value: Value = serde_json::from_slice(&scanned.bytes).map_err(|source| { + TerminologyError::Json { + file: filename.clone(), + source, + } + })?; + let object = value.as_object().ok_or_else(|| TerminologyError::InvalidField { + resource: filename.clone(), + field: "resourceType".to_owned(), + message: "FHIR package resource must be a JSON object".to_owned(), + })?; + let resource_type = required_string(object, "resourceType", &filename)?; + + if !TERMINOLOGY_TYPES.contains(&resource_type.as_str()) { continue; } - let Some(url) = resource.canonical_url else { + + let Some(url) = optional_string(object, "url", &filename)? else { continue; }; - let value = raw.get(&resource.filename).cloned().ok_or_else(|| { - TerminologyError::InvalidField { - resource: resource.filename.clone(), - field: "resource".to_owned(), - message: "inspected terminology resource is missing from scanned archive" - .to_owned(), - } - })?; + let version = optional_string(object, "version", &filename)?; closure.insert(TerminologyResource { package_name: package.name.clone(), package_version: package.version.clone(), - filename: resource.filename, - resource_type: resource.resource_type, + filename, + resource_type, url, - version: resource.canonical_version, + version, value, })?; } @@ -160,6 +157,38 @@ impl TerminologyClosure { } } +fn required_string( + object: &Map, + field: &str, + filename: &str, +) -> Result { + object + .get(field) + .and_then(Value::as_str) + .map(str::to_owned) + .ok_or_else(|| TerminologyError::InvalidField { + resource: filename.to_owned(), + field: field.to_owned(), + message: "must be a string".to_owned(), + }) +} + +fn optional_string( + object: &Map, + field: &str, + filename: &str, +) -> Result, TerminologyError> { + match object.get(field) { + None => Ok(None), + Some(Value::String(value)) => Ok(Some(value.clone())), + Some(_) => Err(TerminologyError::InvalidField { + resource: filename.to_owned(), + field: field.to_owned(), + message: "must be a string when present".to_owned(), + }), + } +} + fn exact_identity(url: &str, version: Option<&str>) -> String { match version { Some(version) => format!("{url}|{version}"), @@ -196,7 +225,61 @@ fn location(resource: &TerminologyResource) -> String { #[cfg(test)] mod tests { + use std::collections::BTreeMap; + use std::io::Cursor; + + use flate2::write::GzEncoder; + use flate2::Compression; + use tar::{Builder, Header}; + use tempfile::TempDir; + use super::*; + use crate::LockedPackage; + + fn package_archive(resources: &[(&str, &str)]) -> Vec { + let manifest = br#"{"name":"example.pkg","version":"1.0.0","dependencies":{}}"#; + let mut encoder = GzEncoder::new(Vec::new(), Compression::default()); + { + let mut builder = Builder::new(&mut encoder); + append(&mut builder, "package/package.json", manifest); + for (filename, body) in resources { + append( + &mut builder, + &format!("package/{filename}"), + body.as_bytes(), + ); + } + builder.finish().unwrap(); + } + encoder.finish().unwrap() + } + + fn append(builder: &mut Builder<&mut GzEncoder>>, path: &str, body: &[u8]) { + let mut header = Header::new_gnu(); + header.set_path(path).unwrap(); + header.set_size(body.len() as u64); + header.set_mode(0o644); + header.set_cksum(); + builder.append(&header, Cursor::new(body)).unwrap(); + } + + fn closure_for(resources: &[(&str, &str)]) -> Result { + let temp = TempDir::new().unwrap(); + let cache = PackageCache::new(temp.path()); + let bytes = package_archive(resources); + let sha256 = cache.put(&bytes).unwrap(); + let lockfile = Lockfile::new( + vec!["example.pkg@1.0.0".to_owned()], + vec![LockedPackage { + name: "example.pkg".to_owned(), + version: "1.0.0".to_owned(), + sha256, + source: "https://packages.example.org/example.pkg/1.0.0".to_owned(), + dependencies: BTreeMap::new(), + }], + ); + TerminologyClosure::load(&lockfile, &cache) + } #[test] fn canonical_reference_parser_is_exact_and_fail_closed() { @@ -215,4 +298,48 @@ mod tests { )); } } + + #[test] + fn unrelated_duplicate_canonicals_do_not_block_terminology_closure() { + let closure = closure_for(&[ + ( + "CapabilityStatement-example.json", + r#"{"resourceType":"CapabilityStatement","url":"urn:uuid:shared","version":"1"}"#, + ), + ( + "TerminologyCapabilities-example.json", + r#"{"resourceType":"TerminologyCapabilities","url":"urn:uuid:shared","version":"1"}"#, + ), + ( + "ValueSet-test.json", + r#"{"resourceType":"ValueSet","url":"http://example.org/ValueSet/test","version":"1"}"#, + ), + ]) + .unwrap(); + + let resolved = closure + .resolve_value_set("http://example.org/ValueSet/test|1") + .unwrap() + .expect("ValueSet should resolve"); + assert_eq!(resolved.filename, "ValueSet-test.json"); + } + + #[test] + fn duplicate_terminology_canonical_still_fails_closed() { + let result = closure_for(&[ + ( + "ValueSet-a.json", + r#"{"resourceType":"ValueSet","url":"http://example.org/ValueSet/test","version":"1"}"#, + ), + ( + "ValueSet-b.json", + r#"{"resourceType":"ValueSet","url":"http://example.org/ValueSet/test","version":"1"}"#, + ), + ]); + + assert!(matches!( + result, + Err(TerminologyError::DuplicateCanonical { .. }) + )); + } } From 1fb9f8f5362e53f27c5fce776b40db7d383dd1f0 Mon Sep 17 00:00:00 2001 From: "Abdulaziz M. Shehri" Date: Sat, 15 Aug 2026 18:48:39 +0300 Subject: [PATCH 54/82] style(cf07): format scoped terminology closure --- crates/commandf-pkg/src/terminology_index.rs | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/crates/commandf-pkg/src/terminology_index.rs b/crates/commandf-pkg/src/terminology_index.rs index c1a19e74..08366a03 100644 --- a/crates/commandf-pkg/src/terminology_index.rs +++ b/crates/commandf-pkg/src/terminology_index.rs @@ -70,11 +70,13 @@ impl TerminologyClosure { source, } })?; - let object = value.as_object().ok_or_else(|| TerminologyError::InvalidField { - resource: filename.clone(), - field: "resourceType".to_owned(), - message: "FHIR package resource must be a JSON object".to_owned(), - })?; + let object = value + .as_object() + .ok_or_else(|| TerminologyError::InvalidField { + resource: filename.clone(), + field: "resourceType".to_owned(), + message: "FHIR package resource must be a JSON object".to_owned(), + })?; let resource_type = required_string(object, "resourceType", &filename)?; if !TERMINOLOGY_TYPES.contains(&resource_type.as_str()) { From a8d30fcc2cb668e51947d52216d45d56b414a452 Mon Sep 17 00:00:00 2001 From: "Abdulaziz M. Shehri" Date: Sat, 15 Aug 2026 18:53:48 +0300 Subject: [PATCH 55/82] fix(cf07): scope closure resolution to ValueSets --- crates/commandf-pkg/src/terminology_index.rs | 20 +++++++++++++++----- 1 file changed, 15 insertions(+), 5 deletions(-) diff --git a/crates/commandf-pkg/src/terminology_index.rs b/crates/commandf-pkg/src/terminology_index.rs index 08366a03..b9bfe076 100644 --- a/crates/commandf-pkg/src/terminology_index.rs +++ b/crates/commandf-pkg/src/terminology_index.rs @@ -8,8 +8,6 @@ use crate::{ PackageError, TerminologyError, }; -const TERMINOLOGY_TYPES: [&str; 2] = ["CodeSystem", "ValueSet"]; - #[derive(Clone, Debug)] pub(crate) struct TerminologyResource { pub package_name: String, @@ -79,7 +77,11 @@ impl TerminologyClosure { })?; let resource_type = required_string(object, "resourceType", &filename)?; - if !TERMINOLOGY_TYPES.contains(&resource_type.as_str()) { + // The lock-closure index exists only to resolve StructureDefinition binding + // references. Direct root CodeSystem/ValueSet deltas are handled by CF-03 + // matched-resource authority, so dependency CodeSystem canonicals must not create + // unrelated ambiguity here. + if resource_type != "ValueSet" { continue; } @@ -302,7 +304,7 @@ mod tests { } #[test] - fn unrelated_duplicate_canonicals_do_not_block_terminology_closure() { + fn non_binding_canonical_collisions_do_not_block_value_set_closure() { let closure = closure_for(&[ ( "CapabilityStatement-example.json", @@ -312,6 +314,14 @@ mod tests { "TerminologyCapabilities-example.json", r#"{"resourceType":"TerminologyCapabilities","url":"urn:uuid:shared","version":"1"}"#, ), + ( + "CodeSystem-a.json", + r#"{"resourceType":"CodeSystem","url":"http://example.org/CodeSystem/shared","version":"1"}"#, + ), + ( + "CodeSystem-b.json", + r#"{"resourceType":"CodeSystem","url":"http://example.org/CodeSystem/shared","version":"1"}"#, + ), ( "ValueSet-test.json", r#"{"resourceType":"ValueSet","url":"http://example.org/ValueSet/test","version":"1"}"#, @@ -327,7 +337,7 @@ mod tests { } #[test] - fn duplicate_terminology_canonical_still_fails_closed() { + fn duplicate_value_set_canonical_still_fails_closed() { let result = closure_for(&[ ( "ValueSet-a.json", From fed73a4fbbafc401450fdc3dc461a0f55d545675 Mon Sep 17 00:00:00 2001 From: "Abdulaziz M. Shehri" Date: Sat, 15 Aug 2026 18:59:26 +0300 Subject: [PATCH 56/82] fix(cf10): distinguish evidence-stage failures --- crates/commandf-pkg/src/corpus_model.rs | 1 + 1 file changed, 1 insertion(+) diff --git a/crates/commandf-pkg/src/corpus_model.rs b/crates/commandf-pkg/src/corpus_model.rs index 8ec96665..69666793 100644 --- a/crates/commandf-pkg/src/corpus_model.rs +++ b/crates/commandf-pkg/src/corpus_model.rs @@ -85,6 +85,7 @@ pub enum CorpusCaseStatus { CompatibilityFailed, TerminologyFailed, OracleFailed, + EvidenceFailed, } #[derive(Clone, Debug, Eq, PartialEq, Serialize, Deserialize)] From b57368d449a7f1266b82f315115bd35d4fdbee49 Mon Sep 17 00:00:00 2001 From: "Abdulaziz M. Shehri" Date: Sat, 15 Aug 2026 18:59:56 +0300 Subject: [PATCH 57/82] fix(cf10): keep evidence-bound failures case-local --- crates/commandf-cli/src/corpus.rs | 118 ++++++++++++++++++++++-------- 1 file changed, 89 insertions(+), 29 deletions(-) diff --git a/crates/commandf-cli/src/corpus.rs b/crates/commandf-cli/src/corpus.rs index 750c16e2..5a754ba5 100644 --- a/crates/commandf-cli/src/corpus.rs +++ b/crates/commandf-cli/src/corpus.rs @@ -83,18 +83,60 @@ pub fn run( } }; - write_bounded_report( - &evidence_dir.join("structural.json"), - &reports.structural.to_json_bytes()?, - )?; - write_bounded_report( - &evidence_dir.join("compatibility.json"), - &reports.compatibility.to_json_bytes()?, - )?; - write_bounded_report( - &evidence_dir.join("terminology.json"), - &reports.terminology.to_json_bytes()?, - )?; + let structural_bytes = match bounded_report_bytes( + "structural", + reports.structural.to_json_bytes().map_err(|error| error.to_string()), + ) { + Ok(bytes) => bytes, + Err(message) => { + failed = true; + write_failure_message(&evidence_dir, "evidence-failure.txt", &message)?; + summaries.push(failed_corpus_case_summary( + case, + CorpusCaseStatus::EvidenceFailed, + )); + continue; + } + }; + let compatibility_bytes = match bounded_report_bytes( + "compatibility", + reports + .compatibility + .to_json_bytes() + .map_err(|error| error.to_string()), + ) { + Ok(bytes) => bytes, + Err(message) => { + failed = true; + write_failure_message(&evidence_dir, "evidence-failure.txt", &message)?; + summaries.push(failed_corpus_case_summary( + case, + CorpusCaseStatus::EvidenceFailed, + )); + continue; + } + }; + let terminology_bytes = match bounded_report_bytes( + "terminology", + reports + .terminology + .to_json_bytes() + .map_err(|error| error.to_string()), + ) { + Ok(bytes) => bytes, + Err(message) => { + failed = true; + write_failure_message(&evidence_dir, "evidence-failure.txt", &message)?; + summaries.push(failed_corpus_case_summary( + case, + CorpusCaseStatus::EvidenceFailed, + )); + continue; + } + }; + fs::write(evidence_dir.join("structural.json"), structural_bytes)?; + fs::write(evidence_dir.join("compatibility.json"), compatibility_bytes)?; + fs::write(evidence_dir.join("terminology.json"), terminology_bytes)?; let oracle_report = match oracle::run_changed_report( case.package.clone(), @@ -116,10 +158,24 @@ pub fn run( continue; } }; - write_bounded_report( - &evidence_dir.join("oracle.json"), - &oracle_report.to_json_bytes()?, - )?; + let oracle_bytes = match bounded_report_bytes( + "oracle", + oracle_report + .to_json_bytes() + .map_err(|error| error.to_string()), + ) { + Ok(bytes) => bytes, + Err(message) => { + failed = true; + write_failure_message(&evidence_dir, "evidence-failure.txt", &message)?; + summaries.push(failed_corpus_case_summary( + case, + CorpusCaseStatus::EvidenceFailed, + )); + continue; + } + }; + fs::write(evidence_dir.join("oracle.json"), oracle_bytes)?; match summarize_corpus_case(case, &reports, &oracle_report) { Ok(summary) => summaries.push(summary), @@ -128,7 +184,7 @@ pub fn run( write_failure(&evidence_dir, "summary-failure.txt", &error)?; summaries.push(failed_corpus_case_summary( case, - CorpusCaseStatus::OracleFailed, + CorpusCaseStatus::EvidenceFailed, )); } } @@ -211,19 +267,16 @@ fn read_bounded_file(path: &Path, max_bytes: u64) -> io::Result> { Ok(bytes) } -fn write_bounded_report(path: &Path, bytes: &[u8]) -> io::Result<()> { +fn bounded_report_bytes(label: &str, result: Result, String>) -> Result, String> { + let bytes = result.map_err(|message| format!("{label} report serialization failed: {message}"))?; if bytes.len() > MAX_CORPUS_RAW_REPORT_BYTES { - return Err(io::Error::new( - io::ErrorKind::InvalidData, - format!( - "corpus report is {} bytes; maximum is {}: {}", - bytes.len(), - MAX_CORPUS_RAW_REPORT_BYTES, - path.display() - ), + return Err(format!( + "{label} report is {} bytes; maximum is {}", + bytes.len(), + MAX_CORPUS_RAW_REPORT_BYTES )); } - fs::write(path, bytes) + Ok(bytes) } fn evaluation_status(error: &CorpusError) -> CorpusCaseStatus { @@ -268,13 +321,20 @@ fn write_failure( file_name: &str, error: &dyn std::fmt::Display, ) -> io::Result<()> { - let mut text = bounded_diagnostic(error); + write_failure_message(evidence_dir, file_name, &error.to_string()) +} + +fn write_failure_message(evidence_dir: &Path, file_name: &str, message: &str) -> io::Result<()> { + let mut text = bounded_text(message); text.push('\n'); fs::write(evidence_dir.join(file_name), text) } fn bounded_diagnostic(error: &dyn std::fmt::Display) -> String { - let text = error.to_string(); + bounded_text(&error.to_string()) +} + +fn bounded_text(text: &str) -> String { let mut chars = text.chars(); let mut output = chars .by_ref() From fd129d1dbe15b6ce2652255c35512a958438a1b5 Mon Sep 17 00:00:00 2001 From: "Abdulaziz M. Shehri" Date: Sat, 15 Aug 2026 19:00:25 +0300 Subject: [PATCH 58/82] ci(cf10): avoid duplicate feature-branch corpus runs --- .github/workflows/cf10-real-corpus.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/cf10-real-corpus.yml b/.github/workflows/cf10-real-corpus.yml index 202c8123..318dffc6 100644 --- a/.github/workflows/cf10-real-corpus.yml +++ b/.github/workflows/cf10-real-corpus.yml @@ -14,7 +14,7 @@ on: - Cargo.lock push: branches: - - feat/cf-10-real-ig-delta-corpus + - main paths: - .github/workflows/cf10-real-corpus.yml - corpus/real-ig/v1/** From b038cf68d7354fe5ab121ae9cb4a45bbbbddfa6b Mon Sep 17 00:00:00 2001 From: "Abdulaziz M. Shehri" Date: Sat, 15 Aug 2026 19:03:43 +0300 Subject: [PATCH 59/82] fix(cf07): dedupe identical ValueSet canonicals --- crates/commandf-pkg/src/terminology_index.rs | 25 +++++++++++++++++--- 1 file changed, 22 insertions(+), 3 deletions(-) diff --git a/crates/commandf-pkg/src/terminology_index.rs b/crates/commandf-pkg/src/terminology_index.rs index b9bfe076..fe66f45c 100644 --- a/crates/commandf-pkg/src/terminology_index.rs +++ b/crates/commandf-pkg/src/terminology_index.rs @@ -116,6 +116,13 @@ impl TerminologyClosure { fn insert(&mut self, resource: TerminologyResource) -> Result<(), TerminologyError> { let exact = exact_identity(&resource.url, resource.version.as_deref()); if let Some(first) = self.exact.get(&exact) { + // Multi-version package graphs and companion packages can contain byte-distinct + // archives that repeat the exact same ValueSet resource. serde_json::Value equality + // is semantic with respect to object-key order, so identical FHIR JSON carries no + // binding-resolution ambiguity. Conflicting duplicates remain fail-closed. + if first.value == resource.value { + return Ok(()); + } return Err(TerminologyError::DuplicateCanonical { canonical: exact, first: location(first), @@ -337,15 +344,27 @@ mod tests { } #[test] - fn duplicate_value_set_canonical_still_fails_closed() { + fn identical_value_set_canonical_is_deduplicated() { + let body = r#"{"resourceType":"ValueSet","url":"http://example.org/ValueSet/test","version":"1","status":"active"}"#; + let closure = closure_for(&[("ValueSet-a.json", body), ("ValueSet-b.json", body)]).unwrap(); + + let resolved = closure + .resolve_value_set("http://example.org/ValueSet/test|1") + .unwrap() + .expect("identical ValueSet should resolve once"); + assert_eq!(resolved.url, "http://example.org/ValueSet/test"); + } + + #[test] + fn conflicting_value_set_canonical_still_fails_closed() { let result = closure_for(&[ ( "ValueSet-a.json", - r#"{"resourceType":"ValueSet","url":"http://example.org/ValueSet/test","version":"1"}"#, + r#"{"resourceType":"ValueSet","url":"http://example.org/ValueSet/test","version":"1","status":"active"}"#, ), ( "ValueSet-b.json", - r#"{"resourceType":"ValueSet","url":"http://example.org/ValueSet/test","version":"1"}"#, + r#"{"resourceType":"ValueSet","url":"http://example.org/ValueSet/test","version":"1","status":"draft"}"#, ), ]); From 1f6f4c57ddd6cb5812cb92cdfdc43b62b186940c Mon Sep 17 00:00:00 2001 From: "Abdulaziz M. Shehri" Date: Sat, 15 Aug 2026 19:05:04 +0300 Subject: [PATCH 60/82] style(cf10): format evidence failure handling --- crates/commandf-cli/src/corpus.rs | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/crates/commandf-cli/src/corpus.rs b/crates/commandf-cli/src/corpus.rs index 5a754ba5..3a0f8ad3 100644 --- a/crates/commandf-cli/src/corpus.rs +++ b/crates/commandf-cli/src/corpus.rs @@ -85,7 +85,10 @@ pub fn run( let structural_bytes = match bounded_report_bytes( "structural", - reports.structural.to_json_bytes().map_err(|error| error.to_string()), + reports + .structural + .to_json_bytes() + .map_err(|error| error.to_string()), ) { Ok(bytes) => bytes, Err(message) => { @@ -268,7 +271,8 @@ fn read_bounded_file(path: &Path, max_bytes: u64) -> io::Result> { } fn bounded_report_bytes(label: &str, result: Result, String>) -> Result, String> { - let bytes = result.map_err(|message| format!("{label} report serialization failed: {message}"))?; + let bytes = + result.map_err(|message| format!("{label} report serialization failed: {message}"))?; if bytes.len() > MAX_CORPUS_RAW_REPORT_BYTES { return Err(format!( "{label} report is {} bytes; maximum is {}", From dbfcdb1e0f752d23ebbd58f67b4528e0529318ec Mon Sep 17 00:00:00 2001 From: "Abdulaziz M. Shehri" Date: Sat, 15 Aug 2026 19:07:34 +0300 Subject: [PATCH 61/82] ci(cf10): harden digest discovery triggers and diagnostics --- .github/workflows/cf10-digest-discovery.yml | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-) diff --git a/.github/workflows/cf10-digest-discovery.yml b/.github/workflows/cf10-digest-discovery.yml index 8c2e2bee..9bcbdfa4 100644 --- a/.github/workflows/cf10-digest-discovery.yml +++ b/.github/workflows/cf10-digest-discovery.yml @@ -4,15 +4,25 @@ on: pull_request: paths: - .github/workflows/cf10-digest-discovery.yml + - corpus/real-ig/v1/** - specs/010-cf-10-real-ig-delta-corpus/** - donors/cf-10-real-ig-delta-corpus.yaml + - crates/commandf-pkg/** + - crates/commandf-cli/** + - Cargo.toml + - Cargo.lock push: branches: - - feat/cf-10-real-ig-delta-corpus + - main paths: - .github/workflows/cf10-digest-discovery.yml + - corpus/real-ig/v1/** - specs/010-cf-10-real-ig-delta-corpus/** - donors/cf-10-real-ig-delta-corpus.yaml + - crates/commandf-pkg/** + - crates/commandf-cli/** + - Cargo.toml + - Cargo.lock workflow_dispatch: permissions: @@ -58,7 +68,7 @@ jobs: --cache "$root/cache" \ --lock "$root/commandf.lock" \ 2>"$root/resolve.err"; then - error="$(tr '\r\n\t' ' ' < "$root/resolve.err" | head -c 1000)" + error="$(head -c 1000 "$root/resolve.err" | tr '\r\n\t' ' ')" printf '%s\t%s\t%s\t%s\tresolve\t%s\n' \ "$id" "$package" "$version" "$pass" "$error" \ >> /tmp/cf10-digest-discovery/failures.tsv @@ -69,7 +79,7 @@ jobs: --cache "$root/cache" \ --lock "$root/commandf.lock" \ 2>"$root/verify.err"; then - error="$(tr '\r\n\t' ' ' < "$root/verify.err" | head -c 1000)" + error="$(head -c 1000 "$root/verify.err" | tr '\r\n\t' ' ')" printf '%s\t%s\t%s\t%s\tverify\t%s\n' \ "$id" "$package" "$version" "$pass" "$error" \ >> /tmp/cf10-digest-discovery/failures.tsv From 61c4ea4aeb5fd2b34c2a0bf0d8fba1632c4b5fbb Mon Sep 17 00:00:00 2001 From: "Abdulaziz M. Shehri" Date: Sat, 15 Aug 2026 19:08:10 +0300 Subject: [PATCH 62/82] ci(cf10): preserve both clean corpus runs on failure --- .github/workflows/cf10-real-corpus.yml | 92 +++++++++++++++++++++----- 1 file changed, 76 insertions(+), 16 deletions(-) diff --git a/.github/workflows/cf10-real-corpus.yml b/.github/workflows/cf10-real-corpus.yml index 318dffc6..8df8365f 100644 --- a/.github/workflows/cf10-real-corpus.yml +++ b/.github/workflows/cf10-real-corpus.yml @@ -54,47 +54,81 @@ jobs: - name: Run frozen corpus from clean state A run: | - set -euo pipefail + set -u test ! -e /tmp/cf10-real-a + runner_status=0 ./target/debug/commandf corpus run \ --manifest corpus/real-ig/v1/corpus.json \ --work-root /tmp/cf10-real-a \ --oracle-adapter tools/hl7-oracle/target/commandf-hl7-oracle.jar \ --oracle-java "$JAVA_HOME/bin/java" \ --format json \ - > /tmp/cf10-summary-a.json - cmp /tmp/cf10-summary-a.json /tmp/cf10-real-a/summary.json + > /tmp/cf10-summary-a.json \ + || runner_status=$? + + internal_match=0 + if [[ -f /tmp/cf10-real-a/summary.json ]]; then + cmp /tmp/cf10-summary-a.json /tmp/cf10-real-a/summary.json || internal_match=$? + else + internal_match=1 + fi + printf '%s\n' "$runner_status" > /tmp/cf10-runner-status-a + printf '%s\n' "$internal_match" > /tmp/cf10-internal-match-a - name: Run frozen corpus from independent clean state B run: | - set -euo pipefail + set -u test ! -e /tmp/cf10-real-b + runner_status=0 ./target/debug/commandf corpus run \ --manifest corpus/real-ig/v1/corpus.json \ --work-root /tmp/cf10-real-b \ --oracle-adapter tools/hl7-oracle/target/commandf-hl7-oracle.jar \ --oracle-java "$JAVA_HOME/bin/java" \ --format json \ - > /tmp/cf10-summary-b.json - cmp /tmp/cf10-summary-b.json /tmp/cf10-real-b/summary.json + > /tmp/cf10-summary-b.json \ + || runner_status=$? + + internal_match=0 + if [[ -f /tmp/cf10-real-b/summary.json ]]; then + cmp /tmp/cf10-summary-b.json /tmp/cf10-real-b/summary.json || internal_match=$? + else + internal_match=1 + fi + printf '%s\n' "$runner_status" > /tmp/cf10-runner-status-b + printf '%s\n' "$internal_match" > /tmp/cf10-internal-match-b - - name: Require byte-identical deterministic summaries + - name: Record deterministic summary comparison + if: always() run: | - set -euo pipefail - cmp /tmp/cf10-summary-a.json /tmp/cf10-summary-b.json - python - <<'PY' + set -u + comparison_status=0 + if [[ -f /tmp/cf10-summary-a.json && -f /tmp/cf10-summary-b.json ]]; then + cmp /tmp/cf10-summary-a.json /tmp/cf10-summary-b.json || comparison_status=$? + else + comparison_status=1 + fi + printf '%s\n' "$comparison_status" > /tmp/cf10-summary-comparison-status + + parse_status=0 + python - <<'PY' || parse_status=$? import json from pathlib import Path - summary = json.loads(Path('/tmp/cf10-summary-a.json').read_text()) - assert summary['schema'] == 1 - assert [case['case_id'] for case in summary['cases']] == ['C001', 'C002', 'C003'] - assert all(case['status'] == 'complete' for case in summary['cases']) - assert len(summary['manifest_sha256']) == 64 - print(Path('/tmp/cf10-summary-a.json').read_text()) + a = json.loads(Path('/tmp/cf10-summary-a.json').read_text()) + b = json.loads(Path('/tmp/cf10-summary-b.json').read_text()) + for summary in (a, b): + assert summary['schema'] == 1 + assert [case['case_id'] for case in summary['cases']] == ['C001', 'C002', 'C003'] + assert len(summary['manifest_sha256']) == 64 + assert all(case['status'] == 'complete' for case in a['cases']) + assert all(case['status'] == 'complete' for case in b['cases']) + print(json.dumps(a, indent=2, sort_keys=True)) PY + printf '%s\n' "$parse_status" > /tmp/cf10-summary-parse-status - name: Assert metadata-only repository boundary + if: always() run: | set -euo pipefail test -z "$(git status --porcelain)" @@ -112,9 +146,35 @@ jobs: path: | /tmp/cf10-summary-a.json /tmp/cf10-summary-b.json + /tmp/cf10-runner-status-a + /tmp/cf10-runner-status-b + /tmp/cf10-internal-match-a + /tmp/cf10-internal-match-b + /tmp/cf10-summary-comparison-status + /tmp/cf10-summary-parse-status /tmp/cf10-real-a/summary.json /tmp/cf10-real-b/summary.json /tmp/cf10-real-a/evidence /tmp/cf10-real-b/evidence if-no-files-found: warn retention-days: 3 + + - name: Enforce corpus proof outcome + if: always() + run: | + set -euo pipefail + status_a="$(cat /tmp/cf10-runner-status-a 2>/dev/null || printf '1')" + status_b="$(cat /tmp/cf10-runner-status-b 2>/dev/null || printf '1')" + internal_a="$(cat /tmp/cf10-internal-match-a 2>/dev/null || printf '1')" + internal_b="$(cat /tmp/cf10-internal-match-b 2>/dev/null || printf '1')" + comparison="$(cat /tmp/cf10-summary-comparison-status 2>/dev/null || printf '1')" + parsed="$(cat /tmp/cf10-summary-parse-status 2>/dev/null || printf '1')" + + printf 'runner_a=%s runner_b=%s internal_a=%s internal_b=%s comparison=%s parsed=%s\n' \ + "$status_a" "$status_b" "$internal_a" "$internal_b" "$comparison" "$parsed" + + if [[ "$status_a" != 0 || "$status_b" != 0 || \ + "$internal_a" != 0 || "$internal_b" != 0 || \ + "$comparison" != 0 || "$parsed" != 0 ]]; then + exit 1 + fi From aad9ca58911737ca2849b8dc3761cade8c01b899 Mon Sep 17 00:00:00 2001 From: "Abdulaziz M. Shehri" Date: Sat, 15 Aug 2026 19:09:03 +0300 Subject: [PATCH 63/82] test(cf10): cover oracle failure handling --- crates/commandf-cli/src/corpus.rs | 84 ++++++++++++++++++++++++++++--- 1 file changed, 78 insertions(+), 6 deletions(-) diff --git a/crates/commandf-cli/src/corpus.rs b/crates/commandf-cli/src/corpus.rs index 3a0f8ad3..f2f7d994 100644 --- a/crates/commandf-cli/src/corpus.rs +++ b/crates/commandf-cli/src/corpus.rs @@ -4,8 +4,9 @@ use std::path::{Path, PathBuf}; use commandf_pkg::{ evaluate_corpus_case, failed_corpus_case_summary, parse_corpus_manifest, summarize_corpus_case, - CorpusCaseStatus, CorpusError, CorpusPackageStateInput, CorpusRunSummary, FhirRegistrySource, - Lockfile, PackageCache, PackageRequest, RealIgCase, Resolver, MAX_CORPUS_MANIFEST_BYTES, + CorpusCaseStatus, CorpusCaseSummary, CorpusError, CorpusPackageStateInput, CorpusRunSummary, + FhirRegistrySource, Lockfile, PackageCache, PackageRequest, RealIgCase, Resolver, + MAX_CORPUS_MANIFEST_BYTES, }; use crate::oracle; @@ -153,11 +154,11 @@ pub fn run( Ok(report) => report, Err(error) => { failed = true; - write_failure(&evidence_dir, "oracle-failure.txt", error.as_ref())?; - summaries.push(failed_corpus_case_summary( + summaries.push(record_oracle_failure( case, - CorpusCaseStatus::OracleFailed, - )); + &evidence_dir, + error.as_ref(), + )?); continue; } }; @@ -301,6 +302,18 @@ fn evaluation_status(error: &CorpusError) -> CorpusCaseStatus { } } +fn record_oracle_failure( + case: &RealIgCase, + evidence_dir: &Path, + error: &dyn std::fmt::Display, +) -> io::Result { + write_failure(evidence_dir, "oracle-failure.txt", error)?; + Ok(failed_corpus_case_summary( + case, + CorpusCaseStatus::OracleFailed, + )) +} + fn write_resolution_failures( evidence_dir: &Path, before: Option>, @@ -349,3 +362,62 @@ fn bounded_text(text: &str) -> String { } output } + +#[cfg(test)] +mod tests { + use std::time::{SystemTime, UNIX_EPOCH}; + + use commandf_pkg::{CorpusOracleMode, CorpusPackageState, CorpusRightsMode}; + + use super::*; + + fn test_case() -> RealIgCase { + RealIgCase { + id: "C001".to_owned(), + package: "example.package".to_owned(), + before: CorpusPackageState { + version: "1.0.0".to_owned(), + archive_sha256: "a".repeat(64), + archive_bytes: 1, + publication_url: "https://example.org/before".to_owned(), + }, + after: CorpusPackageState { + version: "2.0.0".to_owned(), + archive_sha256: "b".repeat(64), + archive_bytes: 1, + publication_url: "https://example.org/after".to_owned(), + }, + fhir_version: "4.0.1".to_owned(), + publisher: "Example Publisher".to_owned(), + change_evidence_url: "https://example.org/changes".to_owned(), + rights_evidence_url: "https://example.org/rights".to_owned(), + rights_mode: CorpusRightsMode::MetadataOnlyNoRedistribution, + oracle_mode: CorpusOracleMode::ChangedStructureDefinitionsOnly, + } + } + + #[test] + fn oracle_failure_records_typed_status_and_bounded_evidence() { + let nonce = SystemTime::now() + .duration_since(UNIX_EPOCH) + .expect("clock after epoch") + .as_nanos(); + let evidence_dir = std::env::temp_dir().join(format!( + "commandf-corpus-oracle-failure-{}-{nonce}", + std::process::id() + )); + fs::create_dir_all(&evidence_dir).unwrap(); + + let error = io::Error::other("deterministic oracle failure"); + let summary = record_oracle_failure(&test_case(), &evidence_dir, &error).unwrap(); + assert_eq!(summary.status, CorpusCaseStatus::OracleFailed); + assert!(summary.structural.is_none()); + assert!(summary.compatibility.is_none()); + assert!(summary.terminology.is_none()); + assert!(summary.oracle.is_none()); + + let evidence = fs::read_to_string(evidence_dir.join("oracle-failure.txt")).unwrap(); + assert_eq!(evidence, "deterministic oracle failure\n"); + fs::remove_dir_all(evidence_dir).unwrap(); + } +} From 340fa5e9fcf8f8033c73b9739ba9752e98cce356 Mon Sep 17 00:00:00 2001 From: "Abdulaziz M. Shehri" Date: Sat, 15 Aug 2026 19:10:15 +0300 Subject: [PATCH 64/82] style(cf10): format oracle failure handler --- crates/commandf-cli/src/corpus.rs | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/crates/commandf-cli/src/corpus.rs b/crates/commandf-cli/src/corpus.rs index f2f7d994..b6d84a43 100644 --- a/crates/commandf-cli/src/corpus.rs +++ b/crates/commandf-cli/src/corpus.rs @@ -154,11 +154,7 @@ pub fn run( Ok(report) => report, Err(error) => { failed = true; - summaries.push(record_oracle_failure( - case, - &evidence_dir, - error.as_ref(), - )?); + summaries.push(record_oracle_failure(case, &evidence_dir, error.as_ref())?); continue; } }; From d3aae3e85e6cfd97600d4a8ab2ccc33cc56fc300 Mon Sep 17 00:00:00 2001 From: "Abdulaziz M. Shehri" Date: Sat, 15 Aug 2026 20:57:58 +0300 Subject: [PATCH 65/82] fix(cf07): dedupe equivalent binding evidence --- crates/commandf-pkg/src/terminology_index.rs | 113 +++++++++++++++++-- 1 file changed, 103 insertions(+), 10 deletions(-) diff --git a/crates/commandf-pkg/src/terminology_index.rs b/crates/commandf-pkg/src/terminology_index.rs index fe66f45c..2387d9ad 100644 --- a/crates/commandf-pkg/src/terminology_index.rs +++ b/crates/commandf-pkg/src/terminology_index.rs @@ -4,8 +4,9 @@ use std::fs; use serde_json::{Map, Value}; use crate::{ - archive::read_manifest, artifact_scan::scan_package_resources, Lockfile, PackageCache, - PackageError, TerminologyError, + archive::read_manifest, artifact_scan::scan_package_resources, compare_value_set_expansions, + Lockfile, PackageCache, PackageError, ResourceKey, ResourceKeyKind, TerminologyError, + TerminologyProofMode, TerminologyRelation, }; #[derive(Clone, Debug)] @@ -116,11 +117,12 @@ impl TerminologyClosure { fn insert(&mut self, resource: TerminologyResource) -> Result<(), TerminologyError> { let exact = exact_identity(&resource.url, resource.version.as_deref()); if let Some(first) = self.exact.get(&exact) { - // Multi-version package graphs and companion packages can contain byte-distinct - // archives that repeat the exact same ValueSet resource. serde_json::Value equality - // is semantic with respect to object-key order, so identical FHIR JSON carries no - // binding-resolution ambiguity. Conflicting duplicates remain fail-closed. - if first.value == resource.value { + // Multi-version package graphs and companion packages may repeat the same canonical + // ValueSet with byte/JSON differences in metadata that CF-07 never uses for binding + // proof. Reuse CF-07's own normalized ValueSet-expansion comparison as the authority: + // only identical binding evidence is safely deduplicated. Conflicting evidence remains + // fail-closed. + if value_set_binding_evidence_equivalent(first, &resource)? { return Ok(()); } return Err(TerminologyError::DuplicateCanonical { @@ -168,6 +170,38 @@ impl TerminologyClosure { } } +fn value_set_binding_evidence_equivalent( + first: &TerminologyResource, + second: &TerminologyResource, +) -> Result { + let resource = ResourceKey { + kind: ResourceKeyKind::Canonical, + value: exact_identity(&first.url, first.version.as_deref()), + }; + let first_self = compare_value_set_expansions( + resource.clone(), + &first.value, + &first.value, + )?; + let second_self = compare_value_set_expansions( + resource.clone(), + &second.value, + &second.value, + )?; + if first_self != second_self { + return Ok(false); + } + + let cross = compare_value_set_expansions(resource, &first.value, &second.value)?; + match cross.proof_mode { + Some(TerminologyProofMode::ValueSetExpansion) => { + Ok(cross.relation == TerminologyRelation::Equal) + } + Some(_) => Ok(false), + None => Ok(cross.relation == TerminologyRelation::Indeterminate), + } +} + fn required_string( object: &Map, field: &str, @@ -356,15 +390,74 @@ mod tests { } #[test] - fn conflicting_value_set_canonical_still_fails_closed() { + fn equivalent_indeterminate_binding_evidence_is_deduplicated() { + let closure = closure_for(&[ + ( + "ValueSet-a.json", + r#"{"resourceType":"ValueSet","url":"http://example.org/ValueSet/test","version":"1","status":"active","name":"First"}"#, + ), + ( + "ValueSet-b.json", + r#"{"resourceType":"ValueSet","url":"http://example.org/ValueSet/test","version":"1","status":"draft","name":"Second"}"#, + ), + ]) + .unwrap(); + + assert!(closure + .resolve_value_set("http://example.org/ValueSet/test|1") + .unwrap() + .is_some()); + } + + #[test] + fn equivalent_finite_binding_evidence_is_deduplicated() { + let closure = closure_for(&[ + ( + "ValueSet-a.json", + r#"{"resourceType":"ValueSet","url":"http://example.org/ValueSet/test","version":"1","status":"active","expansion":{"total":1,"parameter":[{"name":"includeDesignations","valueBoolean":false}],"contains":[{"system":"http://example.org/system","code":"A"}]}}"#, + ), + ( + "ValueSet-b.json", + r#"{"resourceType":"ValueSet","url":"http://example.org/ValueSet/test","version":"1","status":"draft","title":"Different metadata","expansion":{"contains":[{"code":"A","system":"http://example.org/system"}],"parameter":[{"valueBoolean":false,"name":"includeDesignations"}],"total":1}}"#, + ), + ]) + .unwrap(); + + assert!(closure + .resolve_value_set("http://example.org/ValueSet/test|1") + .unwrap() + .is_some()); + } + + #[test] + fn conflicting_value_set_binding_evidence_still_fails_closed() { + let result = closure_for(&[ + ( + "ValueSet-a.json", + r#"{"resourceType":"ValueSet","url":"http://example.org/ValueSet/test","version":"1","expansion":{"total":1,"contains":[{"system":"http://example.org/system","code":"A"}]}}"#, + ), + ( + "ValueSet-b.json", + r#"{"resourceType":"ValueSet","url":"http://example.org/ValueSet/test","version":"1","expansion":{"total":1,"contains":[{"system":"http://example.org/system","code":"B"}]}}"#, + ), + ]); + + assert!(matches!( + result, + Err(TerminologyError::DuplicateCanonical { .. }) + )); + } + + #[test] + fn different_indeterminate_binding_evidence_still_fails_closed() { let result = closure_for(&[ ( "ValueSet-a.json", - r#"{"resourceType":"ValueSet","url":"http://example.org/ValueSet/test","version":"1","status":"active"}"#, + r#"{"resourceType":"ValueSet","url":"http://example.org/ValueSet/test","version":"1"}"#, ), ( "ValueSet-b.json", - r#"{"resourceType":"ValueSet","url":"http://example.org/ValueSet/test","version":"1","status":"draft"}"#, + r#"{"resourceType":"ValueSet","url":"http://example.org/ValueSet/test","version":"1","expansion":{"offset":1,"total":0}}"#, ), ]); From 0b915dd158d21da156fed88125ff52f77ea4fe1a Mon Sep 17 00:00:00 2001 From: "Abdulaziz M. Shehri" Date: Sat, 15 Aug 2026 20:59:01 +0300 Subject: [PATCH 66/82] style(cf07): format binding evidence helper --- crates/commandf-pkg/src/terminology_index.rs | 12 ++---------- 1 file changed, 2 insertions(+), 10 deletions(-) diff --git a/crates/commandf-pkg/src/terminology_index.rs b/crates/commandf-pkg/src/terminology_index.rs index 2387d9ad..392ba9d1 100644 --- a/crates/commandf-pkg/src/terminology_index.rs +++ b/crates/commandf-pkg/src/terminology_index.rs @@ -178,16 +178,8 @@ fn value_set_binding_evidence_equivalent( kind: ResourceKeyKind::Canonical, value: exact_identity(&first.url, first.version.as_deref()), }; - let first_self = compare_value_set_expansions( - resource.clone(), - &first.value, - &first.value, - )?; - let second_self = compare_value_set_expansions( - resource.clone(), - &second.value, - &second.value, - )?; + let first_self = compare_value_set_expansions(resource.clone(), &first.value, &first.value)?; + let second_self = compare_value_set_expansions(resource.clone(), &second.value, &second.value)?; if first_self != second_self { return Ok(false); } From b91792f6c4737c6e1991d705e3d48d243e72cd9f Mon Sep 17 00:00:00 2001 From: "Abdulaziz M. Shehri" Date: Sat, 15 Aug 2026 21:02:43 +0300 Subject: [PATCH 67/82] fix(cf07): keep expansion context ambiguity fail closed --- crates/commandf-pkg/src/terminology_index.rs | 23 +++++++++++++++++++- 1 file changed, 22 insertions(+), 1 deletion(-) diff --git a/crates/commandf-pkg/src/terminology_index.rs b/crates/commandf-pkg/src/terminology_index.rs index 392ba9d1..4703b325 100644 --- a/crates/commandf-pkg/src/terminology_index.rs +++ b/crates/commandf-pkg/src/terminology_index.rs @@ -190,7 +190,9 @@ fn value_set_binding_evidence_equivalent( Ok(cross.relation == TerminologyRelation::Equal) } Some(_) => Ok(false), - None => Ok(cross.relation == TerminologyRelation::Indeterminate), + None => Ok(first_self.proof_mode.is_none() + && cross.relation == TerminologyRelation::Indeterminate + && cross.reason == first_self.reason), } } @@ -440,6 +442,25 @@ mod tests { )); } + #[test] + fn conflicting_expansion_context_still_fails_closed() { + let result = closure_for(&[ + ( + "ValueSet-a.json", + r#"{"resourceType":"ValueSet","url":"http://example.org/ValueSet/test","version":"1","expansion":{"total":1,"parameter":[{"name":"includeDesignations","valueBoolean":false}],"contains":[{"system":"http://example.org/system","code":"A"}]}}"#, + ), + ( + "ValueSet-b.json", + r#"{"resourceType":"ValueSet","url":"http://example.org/ValueSet/test","version":"1","expansion":{"total":1,"parameter":[{"name":"includeDesignations","valueBoolean":true}],"contains":[{"system":"http://example.org/system","code":"A"}]}}"#, + ), + ]); + + assert!(matches!( + result, + Err(TerminologyError::DuplicateCanonical { .. }) + )); + } + #[test] fn different_indeterminate_binding_evidence_still_fails_closed() { let result = closure_for(&[ From f974c191e33900a2cb52168831ed9ed35ab57289 Mon Sep 17 00:00:00 2001 From: "Abdulaziz M. Shehri" Date: Sat, 15 Aug 2026 21:09:09 +0300 Subject: [PATCH 68/82] fix(cf10): ignore expected build outputs in payload gate --- .github/workflows/cf10-real-corpus.yml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/workflows/cf10-real-corpus.yml b/.github/workflows/cf10-real-corpus.yml index 8df8365f..8f4eb0a1 100644 --- a/.github/workflows/cf10-real-corpus.yml +++ b/.github/workflows/cf10-real-corpus.yml @@ -131,7 +131,7 @@ jobs: if: always() run: | set -euo pipefail - test -z "$(git status --porcelain)" + test -z "$(git status --porcelain --untracked-files=no)" if git ls-files | grep -E '\.(tgz|tar\.gz)$'; then echo 'unexpected upstream archive tracked in repository' >&2 exit 1 @@ -177,4 +177,4 @@ jobs: "$internal_a" != 0 || "$internal_b" != 0 || \ "$comparison" != 0 || "$parsed" != 0 ]]; then exit 1 - fi + fi \ No newline at end of file From 70992c9e539da29c7279e977ba837304cda59183 Mon Sep 17 00:00:00 2001 From: "Abdulaziz M. Shehri" Date: Sat, 15 Aug 2026 21:22:02 +0300 Subject: [PATCH 69/82] fix(cf07): preserve ambiguous binding evidence --- crates/commandf-pkg/src/terminology.rs | 51 ++++++++++++++++++++------ 1 file changed, 39 insertions(+), 12 deletions(-) diff --git a/crates/commandf-pkg/src/terminology.rs b/crates/commandf-pkg/src/terminology.rs index b603ecbd..a9e6f83a 100644 --- a/crates/commandf-pkg/src/terminology.rs +++ b/crates/commandf-pkg/src/terminology.rs @@ -220,8 +220,36 @@ fn build_binding_refinements( }; let references_changed = before_reference != after_reference; - let before_resource = before_closure.resolve_value_set(before_reference)?; - let after_resource = after_closure.resolve_value_set(after_reference)?; + let before_resource = match before_closure.resolve_value_set(before_reference) { + Ok(resource) => resource, + Err(TerminologyError::AmbiguousCanonical { .. }) => { + output.push(indeterminate_refinement( + &pair.key, + binding.view, + &binding.element_id, + before_value_set, + after_value_set, + TerminologyIndeterminateReason::AmbiguousCanonical, + )); + continue; + } + Err(error) => return Err(error), + }; + let after_resource = match after_closure.resolve_value_set(after_reference) { + Ok(resource) => resource, + Err(TerminologyError::AmbiguousCanonical { .. }) => { + output.push(indeterminate_refinement( + &pair.key, + binding.view, + &binding.element_id, + before_value_set, + after_value_set, + TerminologyIndeterminateReason::AmbiguousCanonical, + )); + continue; + } + Err(error) => return Err(error), + }; let (before_resource, after_resource) = match (before_resource, after_resource) { (None, None) if !references_changed => continue, (Some(before_resource), Some(after_resource)) => (before_resource, after_resource), @@ -272,16 +300,15 @@ fn compare_binding_value_sets( after: &TerminologyResource, ) -> Result { let mut after_value = after.value.clone(); - let before_url = - before - .value - .get("url") - .cloned() - .ok_or_else(|| TerminologyError::InvalidField { - resource: before.filename.clone(), - field: "url".to_owned(), - message: "resolved ValueSet is missing its canonical URL".to_owned(), - })?; + let before_url = before + .value + .get("url") + .cloned() + .ok_or_else(|| TerminologyError::InvalidField { + resource: before.filename.clone(), + field: "url".to_owned(), + message: "resolved ValueSet is missing its canonical URL".to_owned(), + })?; let after_object = after_value .as_object_mut() From 61c8dc29c300b32c7a86196610d6b151bc6b83ba Mon Sep 17 00:00:00 2001 From: "Abdulaziz M. Shehri" Date: Sat, 15 Aug 2026 21:22:35 +0300 Subject: [PATCH 70/82] test(cf07): preserve ambiguous bare binding evidence --- .../commandf-pkg/tests/terminology_binding.rs | 103 +++++++++++++++++- 1 file changed, 102 insertions(+), 1 deletion(-) diff --git a/crates/commandf-pkg/tests/terminology_binding.rs b/crates/commandf-pkg/tests/terminology_binding.rs index 86bf2f55..dfc819c8 100644 --- a/crates/commandf-pkg/tests/terminology_binding.rs +++ b/crates/commandf-pkg/tests/terminology_binding.rs @@ -4,7 +4,7 @@ use std::io::Cursor; use commandf_pkg::{ build_terminology_diff_report, classify_structural_diff, diff_package_archives, CompatibilityDirection, CompatibilitySeverity, LockedPackage, Lockfile, PackageCache, - TerminologyPackageState, TerminologyRelation, + TerminologyIndeterminateReason, TerminologyPackageState, TerminologyRelation, }; use flate2::write::GzEncoder; use flate2::Compression; @@ -219,6 +219,107 @@ fn unchanged_binding_detects_narrowed_dependency_value_set() { ); } +#[test] +fn bare_binding_ambiguity_is_explicit_indeterminate_evidence() { + let before_dir = tempdir().unwrap(); + let after_dir = tempdir().unwrap(); + let before_cache = PackageCache::new(before_dir.path()); + let after_cache = PackageCache::new(after_dir.path()); + + let root = archive( + "example.root", + "1.0.0", + json!({"example.term": "1.x"}), + &[("StructureDefinition-test-profile.json", profile())], + ); + let term_v1 = archive( + "example.term", + "1.0.0", + json!({}), + &[("ValueSet-gender.json", value_set("1", &["a"]))], + ); + let term_v2 = archive( + "example.term", + "2.0.0", + json!({}), + &[("ValueSet-gender.json", value_set("2", &["a", "b"]))], + ); + + let before_root_digest = before_cache.put(&root).unwrap(); + let after_root_digest = after_cache.put(&root).unwrap(); + let before_v1_digest = before_cache.put(&term_v1).unwrap(); + let before_v2_digest = before_cache.put(&term_v2).unwrap(); + let after_v1_digest = after_cache.put(&term_v1).unwrap(); + let after_v2_digest = after_cache.put(&term_v2).unwrap(); + + let before_lock = Lockfile::new( + vec!["example.root@1.0.0".to_owned()], + vec![ + locked( + "example.root", + "1.0.0", + before_root_digest.clone(), + BTreeMap::from([("example.term".to_owned(), "1.x".to_owned())]), + ), + locked("example.term", "1.0.0", before_v1_digest, BTreeMap::new()), + locked("example.term", "2.0.0", before_v2_digest, BTreeMap::new()), + ], + ); + let after_lock = Lockfile::new( + vec!["example.root@1.0.0".to_owned()], + vec![ + locked( + "example.root", + "1.0.0", + after_root_digest.clone(), + BTreeMap::from([("example.term".to_owned(), "1.x".to_owned())]), + ), + locked("example.term", "1.0.0", after_v1_digest, BTreeMap::new()), + locked("example.term", "2.0.0", after_v2_digest, BTreeMap::new()), + ], + ); + + let structural = diff_package_archives( + "example.root", + "1.0.0", + &before_root_digest, + &root, + "1.0.0", + &after_root_digest, + &root, + ) + .unwrap(); + let compatibility = classify_structural_diff(&structural).unwrap(); + let report = build_terminology_diff_report( + TerminologyPackageState { + lockfile: &before_lock, + cache: &before_cache, + root_bytes: &root, + }, + TerminologyPackageState { + lockfile: &after_lock, + cache: &after_cache, + root_bytes: &root, + }, + &structural, + &compatibility, + ) + .unwrap(); + + assert_eq!(report.binding_refinements.len(), 1); + let refinement = &report.binding_refinements[0]; + assert_eq!(refinement.relation, TerminologyRelation::Indeterminate); + assert_eq!( + refinement.reason, + Some(TerminologyIndeterminateReason::AmbiguousCanonical) + ); + assert!(!refinement.binding_proof_eligible); + assert!(refinement.proof_mode.is_none()); + assert!(refinement.rule_id.is_none()); + assert!(refinement.severity.is_none()); + assert!(refinement.direction.is_none()); +} + #[test] fn corrupted_dependency_cache_fails_before_binding_proof() { let before_dir = tempdir().unwrap(); From e0ee918e6ee6928077ec81be3b9c8d6c1e89c904 Mon Sep 17 00:00:00 2001 From: "Abdulaziz M. Shehri" Date: Sat, 15 Aug 2026 21:22:52 +0300 Subject: [PATCH 71/82] refactor(cf07): parameterize archive scan budget --- crates/commandf-pkg/src/artifact_scan.rs | 44 +++++++++++++++++++++--- 1 file changed, 39 insertions(+), 5 deletions(-) diff --git a/crates/commandf-pkg/src/artifact_scan.rs b/crates/commandf-pkg/src/artifact_scan.rs index 9fe9a671..426fcb55 100644 --- a/crates/commandf-pkg/src/artifact_scan.rs +++ b/crates/commandf-pkg/src/artifact_scan.rs @@ -19,11 +19,16 @@ pub(crate) struct ScannedResource { struct BoundedReader { inner: R, read: u64, + max_bytes: u64, } impl BoundedReader { - fn new(inner: R) -> Self { - Self { inner, read: 0 } + fn new(inner: R, max_bytes: u64) -> Self { + Self { + inner, + read: 0, + max_bytes, + } } } @@ -32,7 +37,7 @@ impl Read for BoundedReader { if buf.is_empty() { return Ok(0); } - if self.read == MAX_ARCHIVE_BYTES { + if self.read == self.max_bytes { let mut probe = [0_u8; 1]; return match self.inner.read(&mut probe)? { 0 => Ok(0), @@ -42,7 +47,7 @@ impl Read for BoundedReader { )), }; } - let remaining = (MAX_ARCHIVE_BYTES - self.read) as usize; + let remaining = (self.max_bytes - self.read) as usize; let allowed = buf.len().min(remaining); let count = self.inner.read(&mut buf[..allowed])?; self.read += count as u64; @@ -52,9 +57,16 @@ impl Read for BoundedReader { pub(crate) fn scan_package_resources( archive_bytes: &[u8], +) -> Result, ArtifactError> { + scan_package_resources_with_limit(archive_bytes, MAX_ARCHIVE_BYTES) +} + +pub(crate) fn scan_package_resources_with_limit( + archive_bytes: &[u8], + max_archive_bytes: u64, ) -> Result, ArtifactError> { let decoder = GzDecoder::new(Cursor::new(archive_bytes)); - let mut archive = Archive::new(BoundedReader::new(decoder)); + let mut archive = Archive::new(BoundedReader::new(decoder, max_archive_bytes)); let mut resources = Vec::new(); let mut entry_count = 0_usize; @@ -95,3 +107,25 @@ pub(crate) fn scan_package_resources( resources.sort_by(|left, right| left.filename.cmp(&right.filename)); Ok(resources) } + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn bounded_reader_accepts_exact_limit() { + let mut reader = BoundedReader::new(Cursor::new(b"abcd"), 4); + let mut output = Vec::new(); + reader.read_to_end(&mut output).unwrap(); + assert_eq!(output, b"abcd"); + } + + #[test] + fn bounded_reader_rejects_byte_beyond_limit() { + let mut reader = BoundedReader::new(Cursor::new(b"abcde"), 4); + let mut output = Vec::new(); + let error = reader.read_to_end(&mut output).unwrap_err(); + assert_eq!(error.kind(), io::ErrorKind::InvalidData); + assert!(error.to_string().contains("decompressed limit")); + } +} From cedae0a4f0d89975be6ffd36880ebd7b25816558 Mon Sep 17 00:00:00 2001 From: "Abdulaziz M. Shehri" Date: Sat, 15 Aug 2026 21:23:41 +0300 Subject: [PATCH 72/82] fix(cf07): bound terminology closure scan separately --- crates/commandf-pkg/src/terminology_index.rs | 15 +++++++++++---- 1 file changed, 11 insertions(+), 4 deletions(-) diff --git a/crates/commandf-pkg/src/terminology_index.rs b/crates/commandf-pkg/src/terminology_index.rs index 4703b325..04c31e5c 100644 --- a/crates/commandf-pkg/src/terminology_index.rs +++ b/crates/commandf-pkg/src/terminology_index.rs @@ -4,11 +4,16 @@ use std::fs; use serde_json::{Map, Value}; use crate::{ - archive::read_manifest, artifact_scan::scan_package_resources, compare_value_set_expansions, - Lockfile, PackageCache, PackageError, ResourceKey, ResourceKeyKind, TerminologyError, - TerminologyProofMode, TerminologyRelation, + archive::read_manifest, artifact_scan::scan_package_resources_with_limit, + compare_value_set_expansions, Lockfile, PackageCache, PackageError, ResourceKey, + ResourceKeyKind, TerminologyError, TerminologyProofMode, TerminologyRelation, }; +// CF-03 keeps its 512 MiB decompressed root-package scan limit unchanged. Binding resolution has +// to inspect the verified dependency closure, where valid public terminology packages can exceed +// that root-diff envelope. Keep CF-07 independently bounded rather than relaxing CF-03 globally. +const MAX_TERMINOLOGY_ARCHIVE_BYTES: u64 = 1024 * 1024 * 1024; + #[derive(Clone, Debug)] pub(crate) struct TerminologyResource { pub package_name: String, @@ -53,7 +58,9 @@ impl TerminologyClosure { } let mut seen_filenames = BTreeSet::new(); - for scanned in scan_package_resources(&bytes)? { + for scanned in + scan_package_resources_with_limit(&bytes, MAX_TERMINOLOGY_ARCHIVE_BYTES)? + { let filename = scanned.filename; if !seen_filenames.insert(filename.clone()) { return Err(TerminologyError::InvalidField { From 72c4355be8fe2bb751a65a5551197851b4ea407f Mon Sep 17 00:00:00 2001 From: "Abdulaziz M. Shehri" Date: Sat, 15 Aug 2026 21:25:40 +0300 Subject: [PATCH 73/82] style(cf07): format ambiguous binding correction --- crates/commandf-pkg/src/terminology.rs | 19 ++++++++++--------- 1 file changed, 10 insertions(+), 9 deletions(-) diff --git a/crates/commandf-pkg/src/terminology.rs b/crates/commandf-pkg/src/terminology.rs index a9e6f83a..c12b20c6 100644 --- a/crates/commandf-pkg/src/terminology.rs +++ b/crates/commandf-pkg/src/terminology.rs @@ -300,15 +300,16 @@ fn compare_binding_value_sets( after: &TerminologyResource, ) -> Result { let mut after_value = after.value.clone(); - let before_url = before - .value - .get("url") - .cloned() - .ok_or_else(|| TerminologyError::InvalidField { - resource: before.filename.clone(), - field: "url".to_owned(), - message: "resolved ValueSet is missing its canonical URL".to_owned(), - })?; + let before_url = + before + .value + .get("url") + .cloned() + .ok_or_else(|| TerminologyError::InvalidField { + resource: before.filename.clone(), + field: "url".to_owned(), + message: "resolved ValueSet is missing its canonical URL".to_owned(), + })?; let after_object = after_value .as_object_mut() From f6f7f24c2826e79a16c79e45d2b5868b049969b5 Mon Sep 17 00:00:00 2001 From: "Abdulaziz M. Shehri" Date: Sat, 15 Aug 2026 21:26:17 +0300 Subject: [PATCH 74/82] style(cf07): format terminology scan budget --- crates/commandf-pkg/src/terminology_index.rs | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/crates/commandf-pkg/src/terminology_index.rs b/crates/commandf-pkg/src/terminology_index.rs index 04c31e5c..f603eded 100644 --- a/crates/commandf-pkg/src/terminology_index.rs +++ b/crates/commandf-pkg/src/terminology_index.rs @@ -58,8 +58,7 @@ impl TerminologyClosure { } let mut seen_filenames = BTreeSet::new(); - for scanned in - scan_package_resources_with_limit(&bytes, MAX_TERMINOLOGY_ARCHIVE_BYTES)? + for scanned in scan_package_resources_with_limit(&bytes, MAX_TERMINOLOGY_ARCHIVE_BYTES)? { let filename = scanned.filename; if !seen_filenames.insert(filename.clone()) { From 9c03b2366c62df494fd1e2ade8cd3ba429d9bd39 Mon Sep 17 00:00:00 2001 From: "Abdulaziz M. Shehri" Date: Sat, 15 Aug 2026 21:28:33 +0300 Subject: [PATCH 75/82] test(cf07): reconcile ambiguous binding contract --- .../tests/terminology_binding_matrix.rs | 19 +++++++++++++++---- 1 file changed, 15 insertions(+), 4 deletions(-) diff --git a/crates/commandf-pkg/tests/terminology_binding_matrix.rs b/crates/commandf-pkg/tests/terminology_binding_matrix.rs index 38ffefd4..73107577 100644 --- a/crates/commandf-pkg/tests/terminology_binding_matrix.rs +++ b/crates/commandf-pkg/tests/terminology_binding_matrix.rs @@ -354,13 +354,13 @@ fn changed_unresolved_reference_is_indeterminate() { } #[test] -fn ambiguous_bare_value_set_reference_fails_closed() { +fn ambiguous_bare_value_set_reference_is_indeterminate_not_hard_proof() { let url = "http://example.org/ValueSet/gender"; let before_one = value_set("gender-1", url, "1", &["a"]); let before_two = value_set("gender-2", url, "2", &["a"]); let after_one = value_set("gender-1", url, "1", &["a"]); let after_two = value_set("gender-2", url, "2", &["a"]); - let error = report_case( + let report = report_case( "required", url, &[ @@ -374,6 +374,17 @@ fn ambiguous_bare_value_set_reference_fails_closed() { ("ValueSet-gender-2.json", after_two), ], ) - .unwrap_err(); - assert!(error.contains("ambiguous terminology canonical"), "{error}"); + .unwrap(); + assert_eq!(report.binding_refinements.len(), 1); + let finding = &report.binding_refinements[0]; + assert_eq!(finding.relation, TerminologyRelation::Indeterminate); + assert_eq!( + finding.reason, + Some(TerminologyIndeterminateReason::AmbiguousCanonical) + ); + assert!(!finding.binding_proof_eligible); + assert!(finding.proof_mode.is_none()); + assert!(finding.rule_id.is_none()); + assert!(finding.severity.is_none()); + assert!(finding.direction.is_none()); } From e5391d8ce968d001e1f9e3434b9e310acce248f8 Mon Sep 17 00:00:00 2001 From: "Abdulaziz M. Shehri" Date: Sat, 15 Aug 2026 22:13:24 +0300 Subject: [PATCH 76/82] fix(corpus): bind closure and oracle dependency context --- crates/commandf-cli/src/corpus.rs | 227 +++++++++++++--- crates/commandf-cli/src/oracle.rs | 242 ++++++++++++++++-- crates/commandf-pkg/src/corpus_evaluate.rs | 212 +++++++++++---- crates/commandf-pkg/src/corpus_model.rs | 15 ++ crates/commandf-pkg/src/lib.rs | 13 +- crates/commandf-pkg/src/oracle_process.rs | 15 +- crates/commandf-pkg/tests/corpus_evaluator.rs | 69 ++++- crates/commandf-pkg/tests/oracle_process.rs | 68 +++++ .../main/java/dev/commandf/oracle/Main.java | 30 ++- 9 files changed, 770 insertions(+), 121 deletions(-) diff --git a/crates/commandf-cli/src/corpus.rs b/crates/commandf-cli/src/corpus.rs index b6d84a43..0e18aaf1 100644 --- a/crates/commandf-cli/src/corpus.rs +++ b/crates/commandf-cli/src/corpus.rs @@ -3,16 +3,18 @@ use std::io::{self, Read}; use std::path::{Path, PathBuf}; use commandf_pkg::{ - evaluate_corpus_case, failed_corpus_case_summary, parse_corpus_manifest, summarize_corpus_case, - CorpusCaseStatus, CorpusCaseSummary, CorpusError, CorpusPackageStateInput, CorpusRunSummary, - FhirRegistrySource, Lockfile, PackageCache, PackageRequest, RealIgCase, Resolver, - MAX_CORPUS_MANIFEST_BYTES, + evaluate_corpus_compatibility, evaluate_corpus_structural, evaluate_corpus_terminology, + failed_corpus_case_summary, failed_corpus_case_summary_with_closure, parse_corpus_manifest, + summarize_corpus_case, CorpusCaseReports, CorpusCaseStatus, CorpusCaseSummary, CorpusError, + CorpusPackageStateInput, CorpusRunSummary, FhirRegistrySource, Lockfile, PackageCache, + PackageRequest, RealIgCase, Resolver, MAX_CORPUS_MANIFEST_BYTES, }; use crate::oracle; const MAX_FAILURE_DIAGNOSTIC_CHARS: usize = 16_384; const MAX_CORPUS_RAW_REPORT_BYTES: usize = 64 * 1024 * 1024; +const MAX_CORPUS_LOCK_BYTES: usize = 4 * 1024 * 1024; const MAX_CORPUS_SUMMARY_BYTES: usize = 1024 * 1024; pub struct CorpusExecution { @@ -63,31 +65,73 @@ pub fn run( } }; - let reports = match evaluate_corpus_case( - case, - CorpusPackageStateInput { - lockfile: &before.lockfile, - cache: &before.cache, - }, - CorpusPackageStateInput { - lockfile: &after.lockfile, - cache: &after.cache, - }, + let before_lock_bytes = match bounded_lock_bytes( + "before", + before + .lockfile + .to_bytes() + .map_err(|error| error.to_string()), ) { - Ok(reports) => reports, + Ok(bytes) => bytes, + Err(message) => { + failed = true; + write_failure_message(&evidence_dir, "evidence-failure.txt", &message)?; + summaries.push(failed_summary_with_closure( + case, + CorpusCaseStatus::EvidenceFailed, + &before.lockfile, + &after.lockfile, + )); + continue; + } + }; + let after_lock_bytes = match bounded_lock_bytes( + "after", + after.lockfile.to_bytes().map_err(|error| error.to_string()), + ) { + Ok(bytes) => bytes, + Err(message) => { + failed = true; + write_failure_message(&evidence_dir, "evidence-failure.txt", &message)?; + summaries.push(failed_summary_with_closure( + case, + CorpusCaseStatus::EvidenceFailed, + &before.lockfile, + &after.lockfile, + )); + continue; + } + }; + fs::write(evidence_dir.join("before.commandf.lock"), before_lock_bytes)?; + fs::write(evidence_dir.join("after.commandf.lock"), after_lock_bytes)?; + + let before_input = CorpusPackageStateInput { + lockfile: &before.lockfile, + cache: &before.cache, + }; + let after_input = CorpusPackageStateInput { + lockfile: &after.lockfile, + cache: &after.cache, + }; + + let structural = match evaluate_corpus_structural(case, before_input, after_input) { + Ok(report) => report, Err(error) => { failed = true; let status = evaluation_status(&error); write_failure(&evidence_dir, "evaluation-failure.txt", &error)?; - summaries.push(failed_corpus_case_summary(case, status)); + summaries.push(failed_summary_with_closure( + case, + status, + &before.lockfile, + &after.lockfile, + )); continue; } }; - let structural_bytes = match bounded_report_bytes( "structural", - reports - .structural + structural .to_json_bytes() .map_err(|error| error.to_string()), ) { @@ -95,17 +139,35 @@ pub fn run( Err(message) => { failed = true; write_failure_message(&evidence_dir, "evidence-failure.txt", &message)?; - summaries.push(failed_corpus_case_summary( + summaries.push(failed_summary_with_closure( case, CorpusCaseStatus::EvidenceFailed, + &before.lockfile, + &after.lockfile, + )); + continue; + } + }; + fs::write(evidence_dir.join("structural.json"), structural_bytes)?; + + let compatibility = match evaluate_corpus_compatibility(case, &structural) { + Ok(report) => report, + Err(error) => { + failed = true; + let status = evaluation_status(&error); + write_failure(&evidence_dir, "evaluation-failure.txt", &error)?; + summaries.push(failed_summary_with_closure( + case, + status, + &before.lockfile, + &after.lockfile, )); continue; } }; let compatibility_bytes = match bounded_report_bytes( "compatibility", - reports - .compatibility + compatibility .to_json_bytes() .map_err(|error| error.to_string()), ) { @@ -113,17 +175,41 @@ pub fn run( Err(message) => { failed = true; write_failure_message(&evidence_dir, "evidence-failure.txt", &message)?; - summaries.push(failed_corpus_case_summary( + summaries.push(failed_summary_with_closure( case, CorpusCaseStatus::EvidenceFailed, + &before.lockfile, + &after.lockfile, + )); + continue; + } + }; + fs::write(evidence_dir.join("compatibility.json"), compatibility_bytes)?; + + let terminology = match evaluate_corpus_terminology( + case, + before_input, + after_input, + &structural, + &compatibility, + ) { + Ok(report) => report, + Err(error) => { + failed = true; + let status = evaluation_status(&error); + write_failure(&evidence_dir, "evaluation-failure.txt", &error)?; + summaries.push(failed_summary_with_closure( + case, + status, + &before.lockfile, + &after.lockfile, )); continue; } }; let terminology_bytes = match bounded_report_bytes( "terminology", - reports - .terminology + terminology .to_json_bytes() .map_err(|error| error.to_string()), ) { @@ -131,17 +217,23 @@ pub fn run( Err(message) => { failed = true; write_failure_message(&evidence_dir, "evidence-failure.txt", &message)?; - summaries.push(failed_corpus_case_summary( + summaries.push(failed_summary_with_closure( case, CorpusCaseStatus::EvidenceFailed, + &before.lockfile, + &after.lockfile, )); continue; } }; - fs::write(evidence_dir.join("structural.json"), structural_bytes)?; - fs::write(evidence_dir.join("compatibility.json"), compatibility_bytes)?; fs::write(evidence_dir.join("terminology.json"), terminology_bytes)?; + let reports = CorpusCaseReports { + structural, + compatibility, + terminology, + }; + let oracle_report = match oracle::run_changed_report( case.package.clone(), before.lock_path.clone(), @@ -154,7 +246,13 @@ pub fn run( Ok(report) => report, Err(error) => { failed = true; - summaries.push(record_oracle_failure(case, &evidence_dir, error.as_ref())?); + summaries.push(record_oracle_failure( + case, + &evidence_dir, + error.as_ref(), + &before.lockfile, + &after.lockfile, + )?); continue; } }; @@ -168,23 +266,33 @@ pub fn run( Err(message) => { failed = true; write_failure_message(&evidence_dir, "evidence-failure.txt", &message)?; - summaries.push(failed_corpus_case_summary( + summaries.push(failed_summary_with_closure( case, CorpusCaseStatus::EvidenceFailed, + &before.lockfile, + &after.lockfile, )); continue; } }; fs::write(evidence_dir.join("oracle.json"), oracle_bytes)?; - match summarize_corpus_case(case, &reports, &oracle_report) { + match summarize_corpus_case( + case, + &reports, + &oracle_report, + &before.lockfile, + &after.lockfile, + ) { Ok(summary) => summaries.push(summary), Err(error) => { failed = true; write_failure(&evidence_dir, "summary-failure.txt", &error)?; - summaries.push(failed_corpus_case_summary( + summaries.push(failed_summary_with_closure( case, CorpusCaseStatus::EvidenceFailed, + &before.lockfile, + &after.lockfile, )); } } @@ -267,6 +375,19 @@ fn read_bounded_file(path: &Path, max_bytes: u64) -> io::Result> { Ok(bytes) } +fn bounded_lock_bytes(label: &str, result: Result, String>) -> Result, String> { + let bytes = + result.map_err(|message| format!("{label} lock serialization failed: {message}"))?; + if bytes.len() > MAX_CORPUS_LOCK_BYTES { + return Err(format!( + "{label} lock is {} bytes; maximum is {}", + bytes.len(), + MAX_CORPUS_LOCK_BYTES + )); + } + Ok(bytes) +} + fn bounded_report_bytes(label: &str, result: Result, String>) -> Result, String> { let bytes = result.map_err(|message| format!("{label} report serialization failed: {message}"))?; @@ -298,15 +419,29 @@ fn evaluation_status(error: &CorpusError) -> CorpusCaseStatus { } } +fn failed_summary_with_closure( + case: &RealIgCase, + status: CorpusCaseStatus, + before_lockfile: &Lockfile, + after_lockfile: &Lockfile, +) -> CorpusCaseSummary { + failed_corpus_case_summary_with_closure(case, status, before_lockfile, after_lockfile) + .unwrap_or_else(|_| failed_corpus_case_summary(case, CorpusCaseStatus::EvidenceFailed)) +} + fn record_oracle_failure( case: &RealIgCase, evidence_dir: &Path, error: &dyn std::fmt::Display, + before_lockfile: &Lockfile, + after_lockfile: &Lockfile, ) -> io::Result { write_failure(evidence_dir, "oracle-failure.txt", error)?; - Ok(failed_corpus_case_summary( + Ok(failed_summary_with_closure( case, CorpusCaseStatus::OracleFailed, + before_lockfile, + after_lockfile, )) } @@ -404,13 +539,37 @@ mod tests { )); fs::create_dir_all(&evidence_dir).unwrap(); + let case = test_case(); + let before_lock = Lockfile::new( + vec!["example.package@1.0.0".to_owned()], + vec![commandf_pkg::LockedPackage { + name: "example.package".to_owned(), + version: "1.0.0".to_owned(), + sha256: case.before.archive_sha256.clone(), + source: "https://example.org/before.tgz".to_owned(), + dependencies: std::collections::BTreeMap::new(), + }], + ); + let after_lock = Lockfile::new( + vec!["example.package@2.0.0".to_owned()], + vec![commandf_pkg::LockedPackage { + name: "example.package".to_owned(), + version: "2.0.0".to_owned(), + sha256: case.after.archive_sha256.clone(), + source: "https://example.org/after.tgz".to_owned(), + dependencies: std::collections::BTreeMap::new(), + }], + ); let error = io::Error::other("deterministic oracle failure"); - let summary = record_oracle_failure(&test_case(), &evidence_dir, &error).unwrap(); + let summary = + record_oracle_failure(&case, &evidence_dir, &error, &before_lock, &after_lock).unwrap(); assert_eq!(summary.status, CorpusCaseStatus::OracleFailed); assert!(summary.structural.is_none()); assert!(summary.compatibility.is_none()); assert!(summary.terminology.is_none()); assert!(summary.oracle.is_none()); + assert!(summary.before.closure.is_some()); + assert!(summary.after.closure.is_some()); let evidence = fs::read_to_string(evidence_dir.join("oracle-failure.txt")).unwrap(); assert_eq!(evidence, "deterministic oracle failure\n"); diff --git a/crates/commandf-cli/src/oracle.rs b/crates/commandf-cli/src/oracle.rs index ce563276..894faddd 100644 --- a/crates/commandf-cli/src/oracle.rs +++ b/crates/commandf-cli/src/oracle.rs @@ -1,3 +1,4 @@ +use std::collections::BTreeSet; use std::fs; use std::io; use std::path::PathBuf; @@ -6,8 +7,8 @@ use std::time::Duration; use commandf_pkg::{ diff_package_archives, matched_structure_definition_pairs, reconcile_hl7_oracle, run_hl7_oracle_adapter, validate_hl7_oracle_adapter, Hl7OracleInvocation, LockedPackage, - Lockfile, OracleDivergenceReport, PackageCache, PackageName, ResourceKey, ResourceKeyKind, - DEFAULT_ORACLE_TIMEOUT_SECS, + Lockfile, OracleDivergenceReport, PackageCache, PackageName, PackageRequest, ResourceKey, + ResourceKeyKind, VersionConstraint, DEFAULT_ORACLE_TIMEOUT_SECS, }; const ORACLE_CORE_PACKAGE: &str = "hl7.fhir.r4.core"; @@ -92,8 +93,8 @@ fn run_report_inner( let before_lockfile = Lockfile::from_slice(&fs::read(&before_lock)?)?; let after_lockfile = Lockfile::from_slice(&fs::read(&after_lock)?)?; - let before_locked = select_locked_package(&before_lockfile, package_name.as_str())?; - let after_locked = select_locked_package(&after_lockfile, package_name.as_str())?; + let before_locked = select_root_locked_package(&before_lockfile, package_name.as_str())?; + let after_locked = select_root_locked_package(&after_lockfile, package_name.as_str())?; let before_core = select_oracle_core(&before_lockfile)?; let after_core = select_oracle_core(&after_lockfile)?; @@ -110,14 +111,16 @@ fn run_report_inner( let before_cache = PackageCache::new(before_cache); let after_cache = PackageCache::new(after_cache); - before_cache.verify(&before_locked.sha256)?; - after_cache.verify(&after_locked.sha256)?; - before_cache.verify(&before_core.sha256)?; - after_cache.verify(&after_core.sha256)?; + before_lockfile.verify_cache(&before_cache)?; + after_lockfile.verify_cache(&after_cache)?; let before_archive = archive_path(&before_cache, before_locked); let after_archive = archive_path(&after_cache, after_locked); let core_archive = archive_path(&before_cache, before_core); + let before_context_archives = + dependency_context_archives(&before_lockfile, &before_cache, before_locked)?; + let after_context_archives = + dependency_context_archives(&after_lockfile, &after_cache, after_locked)?; let before_bytes = fs::read(&before_archive)?; let after_bytes = fs::read(&after_archive)?; @@ -160,6 +163,8 @@ fn run_report_inner( core_package: &core_archive, left_package: &before_archive, right_package: &after_archive, + left_context_packages: &before_context_archives, + right_context_packages: &after_context_archives, left_url: url, left_version: version, right_url: url, @@ -179,40 +184,139 @@ fn run_report_inner( } fn select_oracle_core(lockfile: &Lockfile) -> Result<&LockedPackage, io::Error> { - let core = select_locked_package(lockfile, ORACLE_CORE_PACKAGE)?; - if core.version != ORACLE_CORE_VERSION { - return Err(io::Error::new( - io::ErrorKind::InvalidData, - format!( - "CF-06 requires {ORACLE_CORE_PACKAGE}@{ORACLE_CORE_VERSION}, found {}@{}", - core.name, core.version - ), - )); - } - Ok(core) + let request = parse_package_request(&format!("{ORACLE_CORE_PACKAGE}@{ORACLE_CORE_VERSION}"))?; + select_matching_locked_package(lockfile, &request, "oracle core") } -fn select_locked_package<'a>( +fn select_root_locked_package<'a>( lockfile: &'a Lockfile, package_name: &str, ) -> Result<&'a LockedPackage, io::Error> { - let mut matches = lockfile - .packages + let mut requests = lockfile + .roots .iter() - .filter(|candidate| candidate.name == package_name); - let selected = matches.next().ok_or_else(|| { + .map(|root| parse_package_request(root)) + .collect::, _>>()? + .into_iter() + .filter(|request| request.name.as_str() == package_name); + let request = requests.next().ok_or_else(|| { io::Error::new( io::ErrorKind::NotFound, - format!("package {package_name} is not present in the lockfile"), + format!("root package {package_name} is not present in the lockfile roots"), ) })?; - if matches.next().is_some() { + if requests.next().is_some() { return Err(io::Error::new( io::ErrorKind::InvalidData, - format!("package {package_name} appears more than once in the lockfile"), + format!("root package {package_name} appears more than once in lockfile roots"), )); } - Ok(selected) + select_matching_locked_package(lockfile, &request, "root") +} + +fn dependency_context_archives( + lockfile: &Lockfile, + cache: &PackageCache, + root: &LockedPackage, +) -> Result, io::Error> { + dependency_context_packages(lockfile, root).map(|packages| { + packages + .into_iter() + .map(|package| archive_path(cache, package)) + .collect() + }) +} + +fn dependency_context_packages<'a>( + lockfile: &'a Lockfile, + root: &'a LockedPackage, +) -> Result, io::Error> { + let mut visited = BTreeSet::new(); + visited.insert(package_identity(root)); + let mut packages = Vec::new(); + collect_dependency_context_packages(lockfile, root, &mut visited, &mut packages)?; + Ok(packages) +} + +fn collect_dependency_context_packages<'a>( + lockfile: &'a Lockfile, + parent: &'a LockedPackage, + visited: &mut BTreeSet, + output: &mut Vec<&'a LockedPackage>, +) -> Result<(), io::Error> { + for (dependency_name, constraint) in &parent.dependencies { + let request = parse_package_request(&format!("{dependency_name}@{constraint}"))?; + let dependency = select_matching_locked_package(lockfile, &request, "dependency")?; + let identity = package_identity(dependency); + if !visited.insert(identity) { + continue; + } + if dependency.name == ORACLE_CORE_PACKAGE && dependency.version == ORACLE_CORE_VERSION { + continue; + } + collect_dependency_context_packages(lockfile, dependency, visited, output)?; + output.push(dependency); + } + Ok(()) +} + +fn select_matching_locked_package<'a>( + lockfile: &'a Lockfile, + request: &PackageRequest, + role: &str, +) -> Result<&'a LockedPackage, io::Error> { + let mut matches = Vec::new(); + for candidate in &lockfile.packages { + if candidate.name != request.name.as_str() { + continue; + } + let candidate_request = + parse_package_request(&format!("{}@{}", candidate.name, candidate.version))?; + let VersionConstraint::Exact(candidate_version) = candidate_request.constraint else { + return Err(io::Error::new( + io::ErrorKind::InvalidData, + format!( + "locked package version is not exact: {}@{}", + candidate.name, candidate.version + ), + )); + }; + if request.constraint.matches(&candidate_version) { + matches.push(candidate); + } + } + + match matches.as_slice() { + [single] => Ok(*single), + [] => Err(io::Error::new( + io::ErrorKind::NotFound, + format!( + "{role} request {} has no matching locked package", + request.display() + ), + )), + _ => Err(io::Error::new( + io::ErrorKind::InvalidData, + format!( + "{role} request {} matches {} locked packages", + request.display(), + matches.len() + ), + )), + } +} + +fn parse_package_request(raw: &str) -> Result { + PackageRequest::parse(raw).map_err(|error| { + io::Error::new( + io::ErrorKind::InvalidData, + format!("invalid lock package request {raw}: {error}"), + ) + }) +} + +fn package_identity(package: &LockedPackage) -> String { + format!("{}@{}", package.name, package.version) } fn archive_path(cache: &PackageCache, package: &LockedPackage) -> PathBuf { @@ -244,3 +348,85 @@ fn canonical_parts(resource: &ResourceKey) -> Result<(&str, Option<&str>), io::E Ok((resource.value.as_str(), None)) } } + +#[cfg(test)] +mod tests { + use std::collections::BTreeMap; + + use super::*; + + fn locked(name: &str, version: &str, dependencies: &[(&str, &str)]) -> LockedPackage { + LockedPackage { + name: name.to_owned(), + version: version.to_owned(), + sha256: format!("{name}-{version}"), + source: format!("https://example.test/{name}/{version}"), + dependencies: dependencies + .iter() + .map(|(name, version)| ((*name).to_owned(), (*version).to_owned())) + .collect::>(), + } + } + + #[test] + fn root_selection_uses_root_request_not_name_only() { + let lockfile = Lockfile::new( + vec!["example.root@2.0.0".to_owned()], + vec![ + locked("example.root", "1.0.0", &[]), + locked("example.root", "2.0.0", &[]), + ], + ); + let selected = select_root_locked_package(&lockfile, "example.root").unwrap(); + assert_eq!(selected.version, "2.0.0"); + } + + #[test] + fn dependency_context_is_deterministic_leaf_first_and_excludes_core() { + let root = locked( + "example.root", + "1.0.0", + &[ + ("example.a", "1.0.0"), + (ORACLE_CORE_PACKAGE, ORACLE_CORE_VERSION), + ], + ); + let lockfile = Lockfile::new( + vec!["example.root@1.0.0".to_owned()], + vec![ + root, + locked("example.a", "1.0.0", &[("example.b", "2.0.x")]), + locked("example.b", "2.0.3", &[]), + locked(ORACLE_CORE_PACKAGE, ORACLE_CORE_VERSION, &[]), + ], + ); + let root = select_root_locked_package(&lockfile, "example.root").unwrap(); + let contexts = dependency_context_packages(&lockfile, root).unwrap(); + assert_eq!( + contexts + .iter() + .map(|package| package_identity(package)) + .collect::>(), + vec!["example.b@2.0.3", "example.a@1.0.0"] + ); + } + + #[test] + fn ambiguous_dependency_constraint_fails_closed() { + let root = locked("example.root", "1.0.0", &[("example.dep", "1.0.x")]); + let lockfile = Lockfile::new( + vec!["example.root@1.0.0".to_owned()], + vec![ + root, + locked("example.dep", "1.0.1", &[]), + locked("example.dep", "1.0.2", &[]), + ], + ); + let root = select_root_locked_package(&lockfile, "example.root").unwrap(); + let error = dependency_context_packages(&lockfile, root).unwrap_err(); + assert!( + error.to_string().contains("matches 2 locked packages"), + "{error}" + ); + } +} diff --git a/crates/commandf-pkg/src/corpus_evaluate.rs b/crates/commandf-pkg/src/corpus_evaluate.rs index 8555434d..cc5263c4 100644 --- a/crates/commandf-pkg/src/corpus_evaluate.rs +++ b/crates/commandf-pkg/src/corpus_evaluate.rs @@ -1,13 +1,14 @@ use crate::{ attest_corpus_package_state, build_terminology_diff_report, classify_structural_diff, diff_package_archives, CompatibilityDirection, CompatibilityReport, CompatibilitySeverity, - CorpusCaseStatus, CorpusCaseSummary, CorpusCompatibilitySummary, CorpusError, - CorpusOracleSummary, CorpusPackageSide, CorpusStructuralSummary, CorpusSummaryPackageState, - CorpusTerminologySummary, Lockfile, OracleDivergenceReport, OracleIdentity, - OracleResourceStatus, PackageCache, RealIgCase, StructuralDiffReport, TerminologyDiffReport, - TerminologyPackageState, + CorpusCaseStatus, CorpusCaseSummary, CorpusClosurePackage, CorpusCompatibilitySummary, + CorpusError, CorpusOracleSummary, CorpusPackageSide, CorpusStructuralSummary, + CorpusSummaryPackageState, CorpusTerminologySummary, Lockfile, OracleDivergenceReport, + OracleIdentity, OracleResourceStatus, PackageCache, RealIgCase, StructuralDiffReport, + TerminologyDiffReport, TerminologyPackageState, }; +#[derive(Clone, Copy)] pub struct CorpusPackageStateInput<'a> { pub lockfile: &'a Lockfile, pub cache: &'a PackageCache, @@ -20,35 +21,15 @@ pub struct CorpusCaseReports { pub terminology: TerminologyDiffReport, } -pub fn evaluate_corpus_case( +pub fn evaluate_corpus_structural( case: &RealIgCase, before: CorpusPackageStateInput<'_>, after: CorpusPackageStateInput<'_>, -) -> Result { - attest_corpus_package_state( - case, - CorpusPackageSide::Before, - before.lockfile, - before.cache, - )?; - attest_corpus_package_state(case, CorpusPackageSide::After, after.lockfile, after.cache)?; +) -> Result { + let before_bytes = attested_root_bytes(case, CorpusPackageSide::Before, before)?; + let after_bytes = attested_root_bytes(case, CorpusPackageSide::After, after)?; - let before_bytes = before - .cache - .read_verified(&case.before.archive_sha256) - .map_err(|error| CorpusError::CacheVerification { - case_id: case.id.clone(), - message: error.to_string(), - })?; - let after_bytes = after - .cache - .read_verified(&case.after.archive_sha256) - .map_err(|error| CorpusError::CacheVerification { - case_id: case.id.clone(), - message: error.to_string(), - })?; - - let structural = diff_package_archives( + diff_package_archives( case.package.clone(), case.before.version.clone(), case.before.archive_sha256.clone(), @@ -61,16 +42,31 @@ pub fn evaluate_corpus_case( case_id: case.id.clone(), stage: "structural", message: error.to_string(), - })?; + }) +} - let compatibility = - classify_structural_diff(&structural).map_err(|error| CorpusError::Evaluation { - case_id: case.id.clone(), - stage: "compatibility", - message: error.to_string(), - })?; +pub fn evaluate_corpus_compatibility( + case: &RealIgCase, + structural: &StructuralDiffReport, +) -> Result { + classify_structural_diff(structural).map_err(|error| CorpusError::Evaluation { + case_id: case.id.clone(), + stage: "compatibility", + message: error.to_string(), + }) +} - let terminology = build_terminology_diff_report( +pub fn evaluate_corpus_terminology( + case: &RealIgCase, + before: CorpusPackageStateInput<'_>, + after: CorpusPackageStateInput<'_>, + structural: &StructuralDiffReport, + compatibility: &CompatibilityReport, +) -> Result { + let before_bytes = attested_root_bytes(case, CorpusPackageSide::Before, before)?; + let after_bytes = attested_root_bytes(case, CorpusPackageSide::After, after)?; + + build_terminology_diff_report( TerminologyPackageState { lockfile: before.lockfile, cache: before.cache, @@ -81,14 +77,25 @@ pub fn evaluate_corpus_case( cache: after.cache, root_bytes: &after_bytes, }, - &structural, - &compatibility, + structural, + compatibility, ) .map_err(|error| CorpusError::Evaluation { case_id: case.id.clone(), stage: "terminology", message: error.to_string(), - })?; + }) +} + +pub fn evaluate_corpus_case( + case: &RealIgCase, + before: CorpusPackageStateInput<'_>, + after: CorpusPackageStateInput<'_>, +) -> Result { + let structural = evaluate_corpus_structural(case, before, after)?; + let compatibility = evaluate_corpus_compatibility(case, &structural)?; + let terminology = + evaluate_corpus_terminology(case, before, after, &structural, &compatibility)?; Ok(CorpusCaseReports { structural, @@ -101,6 +108,8 @@ pub fn summarize_corpus_case( case: &RealIgCase, reports: &CorpusCaseReports, oracle: &OracleDivergenceReport, + before_lockfile: &Lockfile, + after_lockfile: &Lockfile, ) -> Result { validate_report_identity(case, reports, oracle)?; @@ -160,8 +169,20 @@ pub fn summarize_corpus_case( Ok(CorpusCaseSummary { case_id: case.id.clone(), package: case.package.clone(), - before: summary_state(&case.before.version, &case.before.archive_sha256), - after: summary_state(&case.after.version, &case.after.archive_sha256), + before: summary_state_with_closure( + case, + "before_closure", + &case.before.version, + &case.before.archive_sha256, + before_lockfile, + )?, + after: summary_state_with_closure( + case, + "after_closure", + &case.after.version, + &case.after.archive_sha256, + after_lockfile, + )?, status: CorpusCaseStatus::Complete, structural: Some(CorpusStructuralSummary { changes: reports.structural.changes.len(), @@ -201,8 +222,8 @@ pub fn failed_corpus_case_summary( CorpusCaseSummary { case_id: case.id.clone(), package: case.package.clone(), - before: summary_state(&case.before.version, &case.before.archive_sha256), - after: summary_state(&case.after.version, &case.after.archive_sha256), + before: summary_state_without_closure(&case.before.version, &case.before.archive_sha256), + after: summary_state_without_closure(&case.after.version, &case.after.archive_sha256), status, structural: None, compatibility: None, @@ -211,6 +232,56 @@ pub fn failed_corpus_case_summary( } } +pub fn failed_corpus_case_summary_with_closure( + case: &RealIgCase, + status: CorpusCaseStatus, + before_lockfile: &Lockfile, + after_lockfile: &Lockfile, +) -> Result { + Ok(CorpusCaseSummary { + case_id: case.id.clone(), + package: case.package.clone(), + before: summary_state_with_closure( + case, + "before_closure", + &case.before.version, + &case.before.archive_sha256, + before_lockfile, + )?, + after: summary_state_with_closure( + case, + "after_closure", + &case.after.version, + &case.after.archive_sha256, + after_lockfile, + )?, + status, + structural: None, + compatibility: None, + terminology: None, + oracle: None, + }) +} + +fn attested_root_bytes( + case: &RealIgCase, + side: CorpusPackageSide, + state: CorpusPackageStateInput<'_>, +) -> Result, CorpusError> { + let manifest_state = match side { + CorpusPackageSide::Before => &case.before, + CorpusPackageSide::After => &case.after, + }; + attest_corpus_package_state(case, side, state.lockfile, state.cache)?; + state + .cache + .read_verified(&manifest_state.archive_sha256) + .map_err(|error| CorpusError::CacheVerification { + case_id: case.id.clone(), + message: error.to_string(), + }) +} + fn validate_report_identity( case: &RealIgCase, reports: &CorpusCaseReports, @@ -265,10 +336,59 @@ fn validate_report_identity( Ok(()) } -fn summary_state(version: &str, sha256: &str) -> CorpusSummaryPackageState { +fn summary_state_with_closure( + case: &RealIgCase, + report: &'static str, + version: &str, + sha256: &str, + lockfile: &Lockfile, +) -> Result { + let matches = lockfile + .packages + .iter() + .filter(|package| { + package.name == case.package && package.version == version && package.sha256 == sha256 + }) + .count(); + let expected_root = format!("{}@{}", case.package, version); + if matches != 1 || !lockfile.roots.iter().any(|root| root == &expected_root) { + return Err(identity(case, report)); + } + + let mut closure = lockfile + .packages + .iter() + .map(|package| CorpusClosurePackage { + name: package.name.clone(), + version: package.version.clone(), + sha256: package.sha256.clone(), + dependencies: package.dependencies.clone(), + }) + .collect::>(); + closure.sort_by(|left, right| { + left.name + .cmp(&right.name) + .then_with(|| left.version.cmp(&right.version)) + .then_with(|| left.sha256.cmp(&right.sha256)) + .then_with(|| left.dependencies.cmp(&right.dependencies)) + }); + let closure_bytes = + serde_json::to_vec(&closure).map_err(|error| serialization_error(case, report, error))?; + + Ok(CorpusSummaryPackageState { + version: version.to_owned(), + sha256: sha256.to_owned(), + closure_sha256: Some(PackageCache::digest(&closure_bytes)), + closure: Some(closure), + }) +} + +fn summary_state_without_closure(version: &str, sha256: &str) -> CorpusSummaryPackageState { CorpusSummaryPackageState { version: version.to_owned(), sha256: sha256.to_owned(), + closure_sha256: None, + closure: None, } } diff --git a/crates/commandf-pkg/src/corpus_model.rs b/crates/commandf-pkg/src/corpus_model.rs index 69666793..d04a50c3 100644 --- a/crates/commandf-pkg/src/corpus_model.rs +++ b/crates/commandf-pkg/src/corpus_model.rs @@ -1,3 +1,5 @@ +use std::collections::BTreeMap; + use serde::{Deserialize, Serialize}; #[derive(Clone, Debug, Eq, PartialEq, Serialize, Deserialize)] @@ -68,11 +70,24 @@ pub struct CorpusPackageAttestation { pub archive_bytes: u64, } +#[derive(Clone, Debug, Eq, PartialEq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct CorpusClosurePackage { + pub name: String, + pub version: String, + pub sha256: String, + pub dependencies: BTreeMap, +} + #[derive(Clone, Debug, Eq, PartialEq, Serialize, Deserialize)] #[serde(deny_unknown_fields)] pub struct CorpusSummaryPackageState { pub version: String, pub sha256: String, + #[serde(skip_serializing_if = "Option::is_none")] + pub closure_sha256: Option, + #[serde(skip_serializing_if = "Option::is_none")] + pub closure: Option>, } #[derive(Clone, Copy, Debug, Eq, PartialEq, Serialize, Deserialize)] diff --git a/crates/commandf-pkg/src/lib.rs b/crates/commandf-pkg/src/lib.rs index 33879194..c64e48a8 100644 --- a/crates/commandf-pkg/src/lib.rs +++ b/crates/commandf-pkg/src/lib.rs @@ -74,14 +74,17 @@ pub use corpus::{ }; pub use corpus_error::CorpusError; pub use corpus_evaluate::{ - evaluate_corpus_case, failed_corpus_case_summary, summarize_corpus_case, CorpusCaseReports, + evaluate_corpus_case, evaluate_corpus_compatibility, evaluate_corpus_structural, + evaluate_corpus_terminology, failed_corpus_case_summary, + failed_corpus_case_summary_with_closure, summarize_corpus_case, CorpusCaseReports, CorpusPackageStateInput, }; pub use corpus_model::{ - CorpusCaseStatus, CorpusCaseSummary, CorpusCompatibilitySummary, CorpusOracleMode, - CorpusOracleSummary, CorpusPackageAttestation, CorpusPackageSide, CorpusPackageState, - CorpusRightsMode, CorpusRunSummary, CorpusSelectionPolicy, CorpusStructuralSummary, - CorpusSummaryPackageState, CorpusTerminologySummary, RealIgCase, RealIgCorpus, + CorpusCaseStatus, CorpusCaseSummary, CorpusClosurePackage, CorpusCompatibilitySummary, + CorpusOracleMode, CorpusOracleSummary, CorpusPackageAttestation, CorpusPackageSide, + CorpusPackageState, CorpusRightsMode, CorpusRunSummary, CorpusSelectionPolicy, + CorpusStructuralSummary, CorpusSummaryPackageState, CorpusTerminologySummary, RealIgCase, + RealIgCorpus, }; pub use error::PackageError; pub use lock::{LockedPackage, Lockfile}; diff --git a/crates/commandf-pkg/src/oracle_process.rs b/crates/commandf-pkg/src/oracle_process.rs index bd5281bb..35ee79a9 100644 --- a/crates/commandf-pkg/src/oracle_process.rs +++ b/crates/commandf-pkg/src/oracle_process.rs @@ -1,5 +1,5 @@ use std::io::{self, Read}; -use std::path::Path; +use std::path::{Path, PathBuf}; use std::process::{Child, Command, Stdio}; use std::thread; use std::time::{Duration, Instant}; @@ -14,6 +14,8 @@ pub struct Hl7OracleInvocation<'a> { pub core_package: &'a Path, pub left_package: &'a Path, pub right_package: &'a Path, + pub left_context_packages: &'a [PathBuf], + pub right_context_packages: &'a [PathBuf], pub left_url: &'a str, pub left_version: Option<&'a str>, pub right_url: &'a str, @@ -60,7 +62,16 @@ pub fn run_hl7_oracle_adapter( .arg("--left-package") .arg(invocation.left_package) .arg("--right-package") - .arg(invocation.right_package) + .arg(invocation.right_package); + + for context_package in invocation.left_context_packages { + command.arg("--left-context-package").arg(context_package); + } + for context_package in invocation.right_context_packages { + command.arg("--right-context-package").arg(context_package); + } + + command .arg("--left-url") .arg(invocation.left_url) .arg("--right-url") diff --git a/crates/commandf-pkg/tests/corpus_evaluator.rs b/crates/commandf-pkg/tests/corpus_evaluator.rs index 65500988..eca1e258 100644 --- a/crates/commandf-pkg/tests/corpus_evaluator.rs +++ b/crates/commandf-pkg/tests/corpus_evaluator.rs @@ -156,7 +156,14 @@ fn evaluator_reuses_canonical_reports_and_summary_hashes_them() { commandf_change_kinds: Vec::new(), }], }; - let summary = summarize_corpus_case(&fixture.case, &reports, &oracle).unwrap(); + let summary = summarize_corpus_case( + &fixture.case, + &reports, + &oracle, + &fixture.before_lock, + &fixture.after_lock, + ) + .unwrap(); assert_eq!(summary.status, CorpusCaseStatus::Complete); assert_eq!( @@ -181,6 +188,56 @@ fn evaluator_reuses_canonical_reports_and_summary_hashes_them() { ); assert_eq!(summary.oracle.as_ref().unwrap().compared, 0); assert_eq!(summary.oracle.as_ref().unwrap().uncomparable, 1); + let before_closure = summary + .before + .closure + .as_ref() + .expect("before closure evidence"); + assert_eq!(before_closure.len(), fixture.before_lock.packages.len()); + assert_eq!(before_closure[0].name, "example.pkg"); + assert_eq!(before_closure[0].sha256, fixture.case.before.archive_sha256); + assert_eq!( + summary.before.closure_sha256.as_deref().map(str::len), + Some(64) + ); + + let mut transport_changed = fixture.before_lock.clone(); + transport_changed.packages[0].source = + "https://fallback.example.org/example.pkg/1.0.0".to_owned(); + let transport_summary = summarize_corpus_case( + &fixture.case, + &reports, + &oracle, + &transport_changed, + &fixture.after_lock, + ) + .unwrap(); + assert_eq!( + transport_summary.before.closure_sha256, + summary.before.closure_sha256 + ); + + let mut dependency_digest_changed = fixture.before_lock.clone(); + dependency_digest_changed + .packages + .push(locked("example.dep", "1.0.0", &"c".repeat(64))); + dependency_digest_changed.packages.sort_by(|left, right| { + left.name + .cmp(&right.name) + .then_with(|| left.version.cmp(&right.version)) + }); + let changed_summary = summarize_corpus_case( + &fixture.case, + &reports, + &oracle, + &dependency_digest_changed, + &fixture.after_lock, + ) + .unwrap(); + assert_ne!( + changed_summary.before.closure_sha256, + summary.before.closure_sha256 + ); } #[test] @@ -207,7 +264,13 @@ fn summary_rejects_cross_case_oracle_identity() { }; assert_eq!( - summarize_corpus_case(&fixture.case, &reports, &oracle), + summarize_corpus_case( + &fixture.case, + &reports, + &oracle, + &fixture.before_lock, + &fixture.after_lock, + ), Err(CorpusError::ReportIdentityMismatch { case_id: "C001".to_owned(), report: "oracle", @@ -223,6 +286,8 @@ fn deterministic_run_summary_has_no_failure_detail_or_paths() { assert!(failed.compatibility.is_none()); assert!(failed.terminology.is_none()); assert!(failed.oracle.is_none()); + assert!(failed.before.closure.is_none()); + assert!(failed.after.closure.is_none()); let report = CorpusRunSummary { schema: CorpusRunSummary::SCHEMA_V1, diff --git a/crates/commandf-pkg/tests/oracle_process.rs b/crates/commandf-pkg/tests/oracle_process.rs index ac2611cf..d9a1e1a2 100644 --- a/crates/commandf-pkg/tests/oracle_process.rs +++ b/crates/commandf-pkg/tests/oracle_process.rs @@ -45,10 +45,13 @@ fn invoke( right: &Path, timeout: Duration, ) -> Result { + let context_packages = Vec::::new(); let invocation = Hl7OracleInvocation { core_package: core, left_package: left, right_package: right, + left_context_packages: &context_packages, + right_context_packages: &context_packages, left_url: "http://example.org/StructureDefinition/test", left_version: None, right_url: "http://example.org/StructureDefinition/test", @@ -72,6 +75,71 @@ fn executable_adapter_accepts_valid_pinned_json() { let _ = fs::remove_dir_all(root); } +#[test] +fn context_package_arguments_are_forwarded_in_input_order() { + let root = unique_temp_dir("contexts"); + fs::create_dir_all(&root).expect("create temp dir"); + let adapter = root.join("adapter.sh"); + let capture = root.join("args.txt"); + write_executable( + &adapter, + &format!( + "printf '%s\n' \"$@\" > '{}'; printf '%s\n' '{}'", + capture.display(), + GOOD_REPORT + ), + ); + let (core, left, right) = package_inputs(&root); + let left_contexts = vec![root.join("left-a.tgz"), root.join("left-b.tgz")]; + let right_contexts = vec![root.join("right-a.tgz")]; + for path in left_contexts.iter().chain(right_contexts.iter()) { + fs::write(path, b"fixture").expect("write context fixture"); + } + + let invocation = Hl7OracleInvocation { + core_package: &core, + left_package: &left, + right_package: &right, + left_context_packages: &left_contexts, + right_context_packages: &right_contexts, + left_url: "http://example.org/StructureDefinition/test", + left_version: None, + right_url: "http://example.org/StructureDefinition/test", + right_version: None, + }; + run_hl7_oracle_adapter(&adapter, None, &invocation, Duration::from_secs(1)) + .expect("context argv adapter report"); + + let args = fs::read_to_string(&capture) + .expect("read captured args") + .lines() + .map(str::to_owned) + .collect::>(); + let context_args = args + .windows(2) + .filter(|pair| pair[0] == "--left-context-package" || pair[0] == "--right-context-package") + .map(|pair| (pair[0].clone(), pair[1].clone())) + .collect::>(); + assert_eq!( + context_args, + vec![ + ( + "--left-context-package".to_owned(), + left_contexts[0].display().to_string(), + ), + ( + "--left-context-package".to_owned(), + left_contexts[1].display().to_string(), + ), + ( + "--right-context-package".to_owned(), + right_contexts[0].display().to_string(), + ), + ] + ); + let _ = fs::remove_dir_all(root); +} + #[test] fn jar_adapter_requires_explicit_java_path() { let root = unique_temp_dir("java-required"); diff --git a/tools/hl7-oracle/src/main/java/dev/commandf/oracle/Main.java b/tools/hl7-oracle/src/main/java/dev/commandf/oracle/Main.java index da11c843..5c935e04 100644 --- a/tools/hl7-oracle/src/main/java/dev/commandf/oracle/Main.java +++ b/tools/hl7-oracle/src/main/java/dev/commandf/oracle/Main.java @@ -59,8 +59,10 @@ public static void main(String[] args) { } static Hl7OracleReport compare(Arguments args) throws Exception { - ContextAndPackage left = loadContext(args.corePackage(), args.leftPackage()); - ContextAndPackage right = loadContext(args.corePackage(), args.rightPackage()); + ContextAndPackage left = loadContext( + args.corePackage(), args.leftContextPackages(), args.leftPackage()); + ContextAndPackage right = loadContext( + args.corePackage(), args.rightContextPackages(), args.rightPackage()); StructureDefinition leftResource = fetchStructureDefinition( left.context(), args.leftUrl(), args.leftVersion(), "left"); @@ -115,7 +117,8 @@ static Hl7OracleReport compare(Arguments args) throws Exception { List.copyOf(normalizedMessages)); } - private static ContextAndPackage loadContext(Path corePath, Path sidePath) throws Exception { + private static ContextAndPackage loadContext( + Path corePath, List contextPaths, Path sidePath) throws Exception { NpmPackage core = loadPackage(corePath); requirePackage(core, CORE_PACKAGE_NAME, CORE_PACKAGE_VERSION, "core"); @@ -127,6 +130,14 @@ private static ContextAndPackage loadContext(Path corePath, Path sidePath) throw context.setCanRunWithoutTerminology(true); NpmPackage side = loadPackage(sidePath); + for (Path contextPath : contextPaths) { + NpmPackage dependency = loadPackage(contextPath); + if (samePackage(core, dependency) || samePackage(side, dependency)) { + continue; + } + IContextResourceLoader dependencyLoader = ValidatorUtils.loaderForVersion(dependency.fhirVersion()); + context.loadFromPackage(dependency, dependencyLoader, false); + } if (!samePackage(core, side)) { IContextResourceLoader sideLoader = ValidatorUtils.loaderForVersion(side.fhirVersion()); context.loadFromPackage(side, sideLoader, false); @@ -247,6 +258,8 @@ record Arguments( Path corePackage, Path leftPackage, Path rightPackage, + List leftContextPackages, + List rightContextPackages, String leftUrl, String leftVersion, String rightUrl, @@ -254,6 +267,8 @@ record Arguments( static Arguments parse(String[] args) { Map values = new LinkedHashMap<>(); + List leftContextPackages = new ArrayList<>(); + List rightContextPackages = new ArrayList<>(); for (int index = 0; index < args.length; index += 2) { if (index + 1 >= args.length) { throw new IllegalArgumentException("missing value for " + args[index]); @@ -262,7 +277,12 @@ static Arguments parse(String[] args) { if (!key.startsWith("--")) { throw new IllegalArgumentException("unexpected positional argument: " + key); } - if (values.put(key, args[index + 1]) != null) { + String value = args[index + 1]; + if (key.equals("--left-context-package")) { + leftContextPackages.add(Path.of(value)); + } else if (key.equals("--right-context-package")) { + rightContextPackages.add(Path.of(value)); + } else if (values.put(key, value) != null) { throw new IllegalArgumentException("duplicate argument: " + key); } } @@ -289,6 +309,8 @@ static Arguments parse(String[] args) { Path.of(required(values, "--core-package")), Path.of(required(values, "--left-package")), Path.of(required(values, "--right-package")), + List.copyOf(leftContextPackages), + List.copyOf(rightContextPackages), required(values, "--left-url"), values.get("--left-version"), required(values, "--right-url"), From 60347c5ab847d39bde1b08af7be3596cc4ea797a Mon Sep 17 00:00:00 2001 From: commandF evidence bot Date: Sat, 15 Aug 2026 20:06:23 +0000 Subject: [PATCH 77/82] fix(corpus): scope terminology and oracle contexts --- crates/commandf-cli/src/oracle.rs | 29 +-- crates/commandf-pkg/src/terminology.rs | 45 ++-- crates/commandf-pkg/src/terminology_index.rs | 212 +++++++++++++++++- .../main/java/dev/commandf/oracle/Main.java | 2 + 4 files changed, 239 insertions(+), 49 deletions(-) diff --git a/crates/commandf-cli/src/oracle.rs b/crates/commandf-cli/src/oracle.rs index 894faddd..8b00d2b6 100644 --- a/crates/commandf-cli/src/oracle.rs +++ b/crates/commandf-cli/src/oracle.rs @@ -1,4 +1,3 @@ -use std::collections::BTreeSet; use std::fs; use std::io; use std::path::PathBuf; @@ -231,33 +230,16 @@ fn dependency_context_packages<'a>( lockfile: &'a Lockfile, root: &'a LockedPackage, ) -> Result, io::Error> { - let mut visited = BTreeSet::new(); - visited.insert(package_identity(root)); let mut packages = Vec::new(); - collect_dependency_context_packages(lockfile, root, &mut visited, &mut packages)?; - Ok(packages) -} - -fn collect_dependency_context_packages<'a>( - lockfile: &'a Lockfile, - parent: &'a LockedPackage, - visited: &mut BTreeSet, - output: &mut Vec<&'a LockedPackage>, -) -> Result<(), io::Error> { - for (dependency_name, constraint) in &parent.dependencies { + for (dependency_name, constraint) in &root.dependencies { let request = parse_package_request(&format!("{dependency_name}@{constraint}"))?; let dependency = select_matching_locked_package(lockfile, &request, "dependency")?; - let identity = package_identity(dependency); - if !visited.insert(identity) { - continue; - } if dependency.name == ORACLE_CORE_PACKAGE && dependency.version == ORACLE_CORE_VERSION { continue; } - collect_dependency_context_packages(lockfile, dependency, visited, output)?; - output.push(dependency); + packages.push(dependency); } - Ok(()) + Ok(packages) } fn select_matching_locked_package<'a>( @@ -315,6 +297,7 @@ fn parse_package_request(raw: &str) -> Result { }) } +#[cfg(test)] fn package_identity(package: &LockedPackage) -> String { format!("{}@{}", package.name, package.version) } @@ -382,7 +365,7 @@ mod tests { } #[test] - fn dependency_context_is_deterministic_leaf_first_and_excludes_core() { + fn dependency_context_is_direct_deterministic_and_excludes_core() { let root = locked( "example.root", "1.0.0", @@ -407,7 +390,7 @@ mod tests { .iter() .map(|package| package_identity(package)) .collect::>(), - vec!["example.b@2.0.3", "example.a@1.0.0"] + vec!["example.a@1.0.0"] ); } diff --git a/crates/commandf-pkg/src/terminology.rs b/crates/commandf-pkg/src/terminology.rs index c12b20c6..afc54d96 100644 --- a/crates/commandf-pkg/src/terminology.rs +++ b/crates/commandf-pkg/src/terminology.rs @@ -8,9 +8,9 @@ use crate::{ compare_complete_code_systems, compare_value_set_expansions, terminology_index::{TerminologyClosure, TerminologyResource}, BindingRefinement, CompatibilityDirection, CompatibilityReport, CompatibilitySeverity, - ElementView, Lockfile, PackageCache, PackageEvidence, ResourceKeyKind, StructuralDiffReport, - TerminologyDiffReport, TerminologyError, TerminologyIndeterminateReason, TerminologyRelation, - TerminologySetDelta, + ElementView, LockedPackage, Lockfile, PackageCache, PackageEvidence, ResourceKeyKind, + StructuralDiffReport, TerminologyDiffReport, TerminologyError, TerminologyIndeterminateReason, + TerminologyRelation, TerminologySetDelta, }; pub struct TerminologyPackageState<'a> { @@ -26,15 +26,17 @@ pub fn build_terminology_diff_report( compatibility: &CompatibilityReport, ) -> Result { validate_report_contract(structural, compatibility)?; - validate_root_evidence( + let before_root = validate_root_evidence( before.lockfile, &structural.package_name, &structural.before, )?; - validate_root_evidence(after.lockfile, &structural.package_name, &structural.after)?; + let after_root = + validate_root_evidence(after.lockfile, &structural.package_name, &structural.after)?; - let before_closure = TerminologyClosure::load(before.lockfile, before.cache)?; - let after_closure = TerminologyClosure::load(after.lockfile, after.cache)?; + let before_closure = + TerminologyClosure::load_for_root(before.lockfile, before.cache, before_root)?; + let after_closure = TerminologyClosure::load_for_root(after.lockfile, after.cache, after_root)?; let pairs = matched_resource_pairs( &structural.package_name, @@ -119,37 +121,32 @@ fn validate_report_contract( Ok(()) } -fn validate_root_evidence( - lockfile: &Lockfile, +fn validate_root_evidence<'a>( + lockfile: &'a Lockfile, package_name: &str, evidence: &PackageEvidence, -) -> Result<(), TerminologyError> { - let mut matches = lockfile - .packages - .iter() - .filter(|package| package.name == package_name); +) -> Result<&'a LockedPackage, TerminologyError> { + let mut matches = lockfile.packages.iter().filter(|package| { + package.name == package_name + && package.version == evidence.version + && package.sha256 == evidence.archive_sha256 + }); let Some(package) = matches.next() else { return Err(TerminologyError::InvalidField { resource: package_name.to_owned(), field: "lockfile".to_owned(), - message: "root package is not present in the lockfile".to_owned(), + message: "exact root package identity is not present in the lockfile".to_owned(), }); }; if matches.next().is_some() { return Err(TerminologyError::InvalidField { resource: package_name.to_owned(), field: "lockfile".to_owned(), - message: "root package appears more than once in the lockfile".to_owned(), - }); - } - if package.version != evidence.version || package.sha256 != evidence.archive_sha256 { - return Err(TerminologyError::InvalidField { - resource: package_name.to_owned(), - field: "lockfile".to_owned(), - message: "root lock identity does not match CF-03 package evidence".to_owned(), + message: "exact root package identity appears more than once in the lockfile" + .to_owned(), }); } - Ok(()) + Ok(package) } #[derive(Clone, Debug, Eq, PartialEq, Ord, PartialOrd)] diff --git a/crates/commandf-pkg/src/terminology_index.rs b/crates/commandf-pkg/src/terminology_index.rs index f603eded..0e25123b 100644 --- a/crates/commandf-pkg/src/terminology_index.rs +++ b/crates/commandf-pkg/src/terminology_index.rs @@ -5,8 +5,9 @@ use serde_json::{Map, Value}; use crate::{ archive::read_manifest, artifact_scan::scan_package_resources_with_limit, - compare_value_set_expansions, Lockfile, PackageCache, PackageError, ResourceKey, - ResourceKeyKind, TerminologyError, TerminologyProofMode, TerminologyRelation, + compare_value_set_expansions, LockedPackage, Lockfile, PackageCache, PackageError, + PackageRequest, ResourceKey, ResourceKeyKind, TerminologyError, TerminologyProofMode, + TerminologyRelation, VersionConstraint, }; // CF-03 keeps its 512 MiB decompressed root-package scan limit unchanged. Binding resolution has @@ -32,14 +33,37 @@ pub(crate) struct TerminologyClosure { } impl TerminologyClosure { + pub(crate) fn load_for_root( + lockfile: &Lockfile, + cache: &PackageCache, + root: &LockedPackage, + ) -> Result { + let target_core = root_core_family(root)?; + Self::load_scoped(lockfile, cache, target_core) + } + + #[cfg(test)] pub(crate) fn load( lockfile: &Lockfile, cache: &PackageCache, + ) -> Result { + Self::load_scoped(lockfile, cache, None) + } + + fn load_scoped( + lockfile: &Lockfile, + cache: &PackageCache, + target_core: Option<&str>, ) -> Result { lockfile.verify_cache(cache)?; let mut closure = Self::default(); for package in &lockfile.packages { + if let Some(target_core) = target_core { + if !package_matches_core_family(lockfile, package, target_core)? { + continue; + } + } let path = cache .root() .join("sha256") @@ -202,6 +226,140 @@ fn value_set_binding_evidence_equivalent( } } +fn root_core_family(root: &LockedPackage) -> Result, TerminologyError> { + let cores = root + .dependencies + .keys() + .filter(|name| is_fhir_core_package(name)) + .map(String::as_str) + .collect::>(); + match cores.as_slice() { + [single] => Ok(Some(*single)), + [] => Ok(None), + _ => Err(lock_graph_error( + root, + "root package declares more than one FHIR core dependency", + )), + } +} + +fn package_matches_core_family( + lockfile: &Lockfile, + package: &LockedPackage, + target_core: &str, +) -> Result { + let mut visiting = BTreeSet::new(); + package_matches_core_family_inner(lockfile, package, target_core, &mut visiting) +} + +fn package_matches_core_family_inner( + lockfile: &Lockfile, + package: &LockedPackage, + target_core: &str, + visiting: &mut BTreeSet, +) -> Result { + if package.name == target_core { + return Ok(true); + } + if is_fhir_core_package(&package.name) { + return Ok(false); + } + if package.dependencies.contains_key(target_core) { + return Ok(true); + } + if package + .dependencies + .keys() + .any(|name| is_fhir_core_package(name)) + { + return Ok(false); + } + + let identity = format!("{}@{}", package.name, package.version); + if !visiting.insert(identity.clone()) { + return Err(lock_graph_error( + package, + "dependency cycle while scoping FHIR core family", + )); + } + + for (dependency_name, constraint) in &package.dependencies { + let dependency = select_locked_dependency(lockfile, package, dependency_name, constraint)?; + if package_matches_core_family_inner(lockfile, dependency, target_core, visiting)? { + visiting.remove(&identity); + return Ok(true); + } + } + visiting.remove(&identity); + Ok(false) +} + +fn select_locked_dependency<'a>( + lockfile: &'a Lockfile, + parent: &LockedPackage, + dependency_name: &str, + constraint: &str, +) -> Result<&'a LockedPackage, TerminologyError> { + let raw = format!("{dependency_name}@{constraint}"); + let request = PackageRequest::parse(&raw).map_err(|error| TerminologyError::InvalidField { + resource: format!("{}@{}", parent.name, parent.version), + field: "lockfile".to_owned(), + message: format!("invalid dependency request {raw}: {error}"), + })?; + + let mut matches = Vec::new(); + for candidate in &lockfile.packages { + if candidate.name != request.name.as_str() { + continue; + } + let candidate_raw = format!("{}@{}", candidate.name, candidate.version); + let candidate_request = PackageRequest::parse(&candidate_raw).map_err(|error| { + TerminologyError::InvalidField { + resource: candidate_raw.clone(), + field: "lockfile".to_owned(), + message: format!("invalid locked package identity: {error}"), + } + })?; + let VersionConstraint::Exact(candidate_version) = candidate_request.constraint else { + return Err(TerminologyError::InvalidField { + resource: candidate_raw, + field: "lockfile".to_owned(), + message: "locked package version is not exact".to_owned(), + }); + }; + if request.constraint.matches(&candidate_version) { + matches.push(candidate); + } + } + + match matches.as_slice() { + [single] => Ok(*single), + [] => Err(lock_graph_error( + parent, + &format!("dependency request {raw} has no matching locked package"), + )), + _ => Err(lock_graph_error( + parent, + &format!( + "dependency request {raw} matches {} locked packages", + matches.len() + ), + )), + } +} + +fn is_fhir_core_package(name: &str) -> bool { + name.starts_with("hl7.fhir.r") && name.ends_with(".core") +} + +fn lock_graph_error(package: &LockedPackage, message: &str) -> TerminologyError { + TerminologyError::InvalidField { + resource: format!("{}@{}", package.name, package.version), + field: "lockfile".to_owned(), + message: message.to_owned(), + } +} + fn required_string( object: &Map, field: &str, @@ -326,6 +484,56 @@ mod tests { TerminologyClosure::load(&lockfile, &cache) } + fn locked(name: &str, version: &str, dependencies: &[(&str, &str)]) -> LockedPackage { + LockedPackage { + name: name.to_owned(), + version: version.to_owned(), + sha256: format!("{name}-{version}"), + source: format!("https://example.test/{name}/{version}"), + dependencies: dependencies + .iter() + .map(|(name, version)| ((*name).to_owned(), (*version).to_owned())) + .collect(), + } + } + + #[test] + fn fhir_core_family_scope_excludes_cross_version_dependency_branch() { + let root = locked( + "example.root", + "1.0.0", + &[("hl7.fhir.r4.core", "4.0.1"), ("example.mixed", "1.0.0")], + ); + let mixed = locked( + "example.mixed", + "1.0.0", + &[("hl7.fhir.r4.core", "4.0.1"), ("example.r5", "1.0.0")], + ); + let r5 = locked("example.r5", "1.0.0", &[("hl7.fhir.r5.core", "5.0.0")]); + let lockfile = Lockfile::new( + vec!["example.root@1.0.0".to_owned()], + vec![ + root.clone(), + mixed.clone(), + r5.clone(), + locked("hl7.fhir.r4.core", "4.0.1", &[]), + locked("hl7.fhir.r5.core", "5.0.0", &[]), + ], + ); + + assert_eq!(root_core_family(&root).unwrap(), Some("hl7.fhir.r4.core")); + let synthetic = locked("example.synthetic", "1.0.0", &[]); + assert_eq!(root_core_family(&synthetic).unwrap(), None); + assert!(package_matches_core_family(&lockfile, &mixed, "hl7.fhir.r4.core").unwrap()); + assert!(!package_matches_core_family(&lockfile, &r5, "hl7.fhir.r4.core").unwrap()); + let r5_core = lockfile + .packages + .iter() + .find(|package| package.name == "hl7.fhir.r5.core") + .unwrap(); + assert!(!package_matches_core_family(&lockfile, r5_core, "hl7.fhir.r4.core").unwrap()); + } + #[test] fn canonical_reference_parser_is_exact_and_fail_closed() { assert_eq!( diff --git a/tools/hl7-oracle/src/main/java/dev/commandf/oracle/Main.java b/tools/hl7-oracle/src/main/java/dev/commandf/oracle/Main.java index 5c935e04..3e6bc320 100644 --- a/tools/hl7-oracle/src/main/java/dev/commandf/oracle/Main.java +++ b/tools/hl7-oracle/src/main/java/dev/commandf/oracle/Main.java @@ -28,6 +28,7 @@ import java.util.Locale; import java.util.Map; import java.util.Objects; +import java.util.Set; import java.util.TreeSet; public final class Main { @@ -136,6 +137,7 @@ private static ContextAndPackage loadContext( continue; } IContextResourceLoader dependencyLoader = ValidatorUtils.loaderForVersion(dependency.fhirVersion()); + dependencyLoader.getTypes().retainAll(Set.of("StructureDefinition")); context.loadFromPackage(dependency, dependencyLoader, false); } if (!samePackage(core, side)) { From 60e773434ed72136377bd239572e5d289846669b Mon Sep 17 00:00:00 2001 From: "Abdulaziz M. Shehri" Date: Sat, 15 Aug 2026 23:07:01 +0300 Subject: [PATCH 78/82] chore(corpus): re-anchor validated context scope From 7da0ea6ac007d321455d1a0ab959c89ddf12f947 Mon Sep 17 00:00:00 2001 From: commandF evidence bot Date: Sat, 15 Aug 2026 20:30:20 +0000 Subject: [PATCH 79/82] fix(oracle): allow versioned dependency profile coexistence --- .../main/java/dev/commandf/oracle/Main.java | 19 ++++++++++++------- 1 file changed, 12 insertions(+), 7 deletions(-) diff --git a/tools/hl7-oracle/src/main/java/dev/commandf/oracle/Main.java b/tools/hl7-oracle/src/main/java/dev/commandf/oracle/Main.java index 3e6bc320..c709b6aa 100644 --- a/tools/hl7-oracle/src/main/java/dev/commandf/oracle/Main.java +++ b/tools/hl7-oracle/src/main/java/dev/commandf/oracle/Main.java @@ -131,14 +131,19 @@ private static ContextAndPackage loadContext( context.setCanRunWithoutTerminology(true); NpmPackage side = loadPackage(sidePath); - for (Path contextPath : contextPaths) { - NpmPackage dependency = loadPackage(contextPath); - if (samePackage(core, dependency) || samePackage(side, dependency)) { - continue; + context.setAllowLoadingDuplicates(true); + try { + for (Path contextPath : contextPaths) { + NpmPackage dependency = loadPackage(contextPath); + if (samePackage(core, dependency) || samePackage(side, dependency)) { + continue; + } + IContextResourceLoader dependencyLoader = ValidatorUtils.loaderForVersion(dependency.fhirVersion()); + dependencyLoader.getTypes().retainAll(Set.of("StructureDefinition")); + context.loadFromPackage(dependency, dependencyLoader, false); } - IContextResourceLoader dependencyLoader = ValidatorUtils.loaderForVersion(dependency.fhirVersion()); - dependencyLoader.getTypes().retainAll(Set.of("StructureDefinition")); - context.loadFromPackage(dependency, dependencyLoader, false); + } finally { + context.setAllowLoadingDuplicates(false); } if (!samePackage(core, side)) { IContextResourceLoader sideLoader = ValidatorUtils.loaderForVersion(side.fhirVersion()); From edad98f8becb81cc5be043cdbdd4347e7c913eb9 Mon Sep 17 00:00:00 2001 From: "Abdulaziz M. Shehri" Date: Sat, 15 Aug 2026 23:30:51 +0300 Subject: [PATCH 80/82] chore(oracle): re-anchor validated dependency coexistence From 381327f45f2d3aec5080167c837f0a6ea64b0b3a Mon Sep 17 00:00:00 2001 From: Abdulaziz Date: Sun, 16 Aug 2026 02:05:57 +0300 Subject: [PATCH 81/82] test(oracle): add deterministic CF-10 qualification probe --- .../QualificationProbe.java | 384 +++++ .../qualify_cf10_oracle.py | 1360 +++++++++++++++++ .../test_qualify_cf10_oracle.py | 457 ++++++ .../workflows/cf10-oracle-qualification.yml | 174 +++ 4 files changed, 2375 insertions(+) create mode 100644 .github/scripts/cf10-oracle-qualification/QualificationProbe.java create mode 100644 .github/scripts/cf10-oracle-qualification/qualify_cf10_oracle.py create mode 100644 .github/scripts/cf10-oracle-qualification/test_qualify_cf10_oracle.py create mode 100644 .github/workflows/cf10-oracle-qualification.yml diff --git a/.github/scripts/cf10-oracle-qualification/QualificationProbe.java b/.github/scripts/cf10-oracle-qualification/QualificationProbe.java new file mode 100644 index 00000000..d75fe467 --- /dev/null +++ b/.github/scripts/cf10-oracle-qualification/QualificationProbe.java @@ -0,0 +1,384 @@ +package dev.commandf.oracle.qualification; + +import com.fasterxml.jackson.databind.MapperFeature; +import com.fasterxml.jackson.databind.ObjectMapper; +import com.fasterxml.jackson.databind.SerializationFeature; +import org.hl7.fhir.r5.comparison.ComparisonSession; +import org.hl7.fhir.r5.comparison.ResourceComparer; +import org.hl7.fhir.r5.comparison.ResourceComparer.ResourceComparison; +import org.hl7.fhir.r5.comparison.StructureDefinitionComparer.ProfileComparison; +import org.hl7.fhir.r5.context.IContextResourceLoader; +import org.hl7.fhir.r5.context.SimpleWorkerContext; +import org.hl7.fhir.r5.model.StructureDefinition; +import org.hl7.fhir.utilities.i18n.RenderingI18nContext; +import org.hl7.fhir.utilities.npm.NpmPackage; +import org.hl7.fhir.validation.ValidatorUtils; + +import java.io.IOException; +import java.io.InputStream; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.ArrayList; +import java.util.Comparator; +import java.util.IdentityHashMap; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; +import java.util.Objects; +import java.util.Set; + +/** + * Isolated diagnostic for the CF-10 pinned-HL7 self/cross comparison matrix. + * + *

The context construction is intentionally identical to the production adapter at the + * qualified repository head. Unlike the adapter, this diagnostic preserves the exception stored + * in {@link ResourceComparer.PlaceHolderComparison}, emits a bounded normalized stack, and exits + * non-zero. It never reconciles the exception into agreement or uncomparable.

+ */ +public final class QualificationProbe { + static final int SCHEMA = 1; + static final String ORACLE_PROJECT = "hapifhir/org.hl7.fhir.core"; + static final String ORACLE_RELEASE = "6.10.2"; + static final String ORACLE_SOURCE_COMMIT = + "d06577dbc5c62c74a2a8823fbc4830a3024d5b0b"; + static final String CORE_PACKAGE_NAME = "hl7.fhir.r4.core"; + static final String CORE_PACKAGE_VERSION = "4.0.1"; + static final int MAX_STACK_FRAMES = 32; + static final int MAX_TEXT_CODE_POINTS = 4096; + + private static final ObjectMapper JSON = new ObjectMapper() + .enable(SerializationFeature.INDENT_OUTPUT) + .enable(SerializationFeature.ORDER_MAP_ENTRIES_BY_KEYS) + .enable(MapperFeature.SORT_PROPERTIES_ALPHABETICALLY); + + private QualificationProbe() { + } + + public static void main(String[] args) throws Exception { + ProbeResult result; + try { + result = qualify(Arguments.parse(args)); + } catch (Throwable error) { + result = failure("argument_parse", null, null, null, error); + } + System.out.write(JSON.writeValueAsBytes(result)); + System.out.write('\n'); + if (!result.status().equals("completed")) { + System.exit(2); + } + } + + static ProbeResult qualify(Arguments args) { + ContextAndPackage left; + ContextAndPackage right; + try { + left = loadContext(args.corePackage(), args.leftContextPackages(), args.leftPackage()); + right = loadContext(args.corePackage(), args.rightContextPackages(), args.rightPackage()); + } catch (Throwable error) { + return failure("context_load", null, null, null, error); + } + + StructureDefinition leftResource; + StructureDefinition rightResource; + try { + leftResource = fetchStructureDefinition( + left.context(), args.leftUrl(), args.leftVersion(), "left"); + rightResource = fetchStructureDefinition( + right.context(), args.rightUrl(), args.rightVersion(), "right"); + } catch (Throwable error) { + return failure("resource_resolution", null, null, null, error); + } + + ComparisonSession session = new ComparisonSession( + new RenderingI18nContext(), + left.context(), + right.context(), + "commandF HL7 oracle qualification", + null, + null); + session.setAnnotate(false); + + try { + ResourceComparison comparison = session.compare(leftResource, rightResource); + String comparisonClass = comparison == null ? "null" : comparison.getClass().getName(); + if (comparison instanceof ProfileComparison) { + return completed(comparisonClass, leftResource, rightResource); + } + if (comparison instanceof ResourceComparer.PlaceHolderComparison placeholder + && placeholder.getE() != null) { + return failure( + "comparison", comparisonClass, leftResource, rightResource, placeholder.getE()); + } + return failure( + "comparison", + comparisonClass, + leftResource, + rightResource, + new IllegalStateException( + "HL7 comparison did not return ProfileComparison: " + comparisonClass)); + } catch (Throwable error) { + return failure("comparison", null, leftResource, rightResource, error); + } + } + + private static ContextAndPackage loadContext( + Path corePath, List contextPaths, Path sidePath) throws Exception { + NpmPackage core = loadPackage(corePath); + requirePackage(core, CORE_PACKAGE_NAME, CORE_PACKAGE_VERSION, "core"); + + IContextResourceLoader coreLoader = ValidatorUtils.loaderForVersion(core.fhirVersion()); + SimpleWorkerContext context = new SimpleWorkerContext.SimpleWorkerContextBuilder() + .withAllowLoadingDuplicates(true) + .fromPackage(core, coreLoader, false); + context.setAllowLoadingDuplicates(false); + context.setCanRunWithoutTerminology(true); + + NpmPackage side = loadPackage(sidePath); + context.setAllowLoadingDuplicates(true); + try { + for (Path contextPath : contextPaths) { + NpmPackage dependency = loadPackage(contextPath); + if (samePackage(core, dependency) || samePackage(side, dependency)) { + continue; + } + IContextResourceLoader dependencyLoader = + ValidatorUtils.loaderForVersion(dependency.fhirVersion()); + dependencyLoader.getTypes().retainAll(Set.of("StructureDefinition")); + context.loadFromPackage(dependency, dependencyLoader, false); + } + } finally { + context.setAllowLoadingDuplicates(false); + } + if (!samePackage(core, side)) { + IContextResourceLoader sideLoader = ValidatorUtils.loaderForVersion(side.fhirVersion()); + context.loadFromPackage(side, sideLoader, false); + } + return new ContextAndPackage(context, side.name(), side.version()); + } + + private static NpmPackage loadPackage(Path path) throws IOException { + if (!Files.isRegularFile(path)) { + throw new IOException("package path is not a regular file: " + path); + } + try (InputStream input = Files.newInputStream(path)) { + return NpmPackage.fromPackage(input, path.getFileName().toString(), false); + } + } + + private static boolean samePackage(NpmPackage left, NpmPackage right) { + return Objects.equals(left.name(), right.name()) + && Objects.equals(left.version(), right.version()); + } + + private static void requirePackage(NpmPackage npm, String name, String version, String role) { + if (!Objects.equals(npm.name(), name) || !Objects.equals(npm.version(), version)) { + throw new IllegalArgumentException( + role + " package must be " + name + "#" + version + + " but was " + npm.name() + "#" + npm.version()); + } + } + + private static StructureDefinition fetchStructureDefinition( + SimpleWorkerContext context, + String url, + String version, + String side) { + String canonical = version == null || version.isBlank() ? url : url + "|" + version; + StructureDefinition resource = context.fetchResource(StructureDefinition.class, canonical); + if (resource == null && version != null && !version.isBlank()) { + resource = context.fetchResource(StructureDefinition.class, url); + } + if (resource == null) { + throw new IllegalArgumentException( + "unable to resolve " + side + " StructureDefinition " + canonical); + } + return resource; + } + + private static ProbeResult completed( + String comparisonClass, + StructureDefinition leftResource, + StructureDefinition rightResource) { + return new ProbeResult( + SCHEMA, + pinnedOracle(), + "completed", + "comparison", + comparisonClass, + resourceIdentity(leftResource), + resourceIdentity(rightResource), + null, + null, + List.of()); + } + + private static ProbeResult failure( + String phase, + String comparisonClass, + StructureDefinition leftResource, + StructureDefinition rightResource, + Throwable error) { + Throwable root = rootCause(error); + return new ProbeResult( + SCHEMA, + pinnedOracle(), + "exception", + phase, + comparisonClass, + resourceIdentity(leftResource), + resourceIdentity(rightResource), + root.getClass().getName(), + boundedText(root.getMessage() == null ? root.getClass().getSimpleName() : root.getMessage()), + boundedStack(root)); + } + + private static ProbeOracle pinnedOracle() { + return new ProbeOracle(ORACLE_PROJECT, ORACLE_RELEASE, ORACLE_SOURCE_COMMIT); + } + + private static ProbeResource resourceIdentity(StructureDefinition resource) { + if (resource == null) { + return null; + } + return new ProbeResource( + emptyToNull(resource.getUrl()), + emptyToNull(resource.getVersion()), + emptyToNull(resource.getId()), + emptyToNull(resource.getType())); + } + + private static String emptyToNull(String value) { + return value == null || value.isBlank() ? null : value; + } + + private static Throwable rootCause(Throwable error) { + Set seen = java.util.Collections.newSetFromMap(new IdentityHashMap<>()); + Throwable current = error; + seen.add(current); + while (current.getCause() != null && seen.add(current.getCause())) { + current = current.getCause(); + } + return current; + } + + private static List boundedStack(Throwable error) { + List stack = new ArrayList<>(); + StackTraceElement[] frames = error.getStackTrace(); + int limit = Math.min(frames.length, MAX_STACK_FRAMES); + for (int index = 0; index < limit; index++) { + StackTraceElement frame = frames[index]; + String file = frame.getFileName() == null ? "unknown" : frame.getFileName(); + stack.add(boundedText( + frame.getClassName() + "#" + frame.getMethodName() + + "(" + file + ":" + frame.getLineNumber() + ")")); + } + if (frames.length > MAX_STACK_FRAMES) { + stack.add("... [stack trace truncated]"); + } + return List.copyOf(stack); + } + + private static String boundedText(String value) { + int codePoints = value.codePointCount(0, value.length()); + if (codePoints <= MAX_TEXT_CODE_POINTS) { + return value; + } + int end = value.offsetByCodePoints(0, MAX_TEXT_CODE_POINTS); + return value.substring(0, end) + "... [text truncated]"; + } + + record ContextAndPackage(SimpleWorkerContext context, String packageName, String packageVersion) { + } + + record ProbeOracle(String project, String release, String source_commit) { + } + + record ProbeResource(String url, String version, String id, String type) { + } + + record ProbeResult( + int schema, + ProbeOracle oracle, + String status, + String phase, + String comparison_class, + ProbeResource left_resource, + ProbeResource right_resource, + String exception_class, + String exception_message, + List stack_trace) { + } + + record Arguments( + Path corePackage, + Path leftPackage, + Path rightPackage, + List leftContextPackages, + List rightContextPackages, + String leftUrl, + String leftVersion, + String rightUrl, + String rightVersion) { + + static Arguments parse(String[] args) { + Map values = new LinkedHashMap<>(); + List leftContextPackages = new ArrayList<>(); + List rightContextPackages = new ArrayList<>(); + for (int index = 0; index < args.length; index += 2) { + if (index + 1 >= args.length) { + throw new IllegalArgumentException("missing value for " + args[index]); + } + String key = args[index]; + if (!key.startsWith("--")) { + throw new IllegalArgumentException("unexpected positional argument: " + key); + } + String value = args[index + 1]; + if (key.equals("--left-context-package")) { + leftContextPackages.add(Path.of(value)); + } else if (key.equals("--right-context-package")) { + rightContextPackages.add(Path.of(value)); + } else if (values.put(key, value) != null) { + throw new IllegalArgumentException("duplicate argument: " + key); + } + } + + List allowed = List.of( + "--core-package", + "--left-package", + "--right-package", + "--left-url", + "--left-version", + "--right-url", + "--right-version"); + List unknown = new ArrayList<>(); + for (String key : values.keySet()) { + if (!allowed.contains(key)) { + unknown.add(key); + } + } + if (!unknown.isEmpty()) { + unknown.sort(Comparator.naturalOrder()); + throw new IllegalArgumentException("unknown arguments: " + String.join(", ", unknown)); + } + + return new Arguments( + Path.of(required(values, "--core-package")), + Path.of(required(values, "--left-package")), + Path.of(required(values, "--right-package")), + List.copyOf(leftContextPackages), + List.copyOf(rightContextPackages), + required(values, "--left-url"), + values.get("--left-version"), + required(values, "--right-url"), + values.get("--right-version")); + } + + private static String required(Map values, String key) { + String value = values.get(key); + if (value == null || value.isBlank()) { + throw new IllegalArgumentException("missing required argument " + key); + } + return value; + } + } +} diff --git a/.github/scripts/cf10-oracle-qualification/qualify_cf10_oracle.py b/.github/scripts/cf10-oracle-qualification/qualify_cf10_oracle.py new file mode 100644 index 00000000..cfdea805 --- /dev/null +++ b/.github/scripts/cf10-oracle-qualification/qualify_cf10_oracle.py @@ -0,0 +1,1360 @@ +#!/usr/bin/env python3 +"""Deterministic CF-10 qualification for pinned HL7 comparison failures. + +Acquisition is intentionally separate from qualification. The ``acquire`` command uses commandF's +own resolver and cache verifier. The ``qualify`` command consumes only those verified local bytes, +so callers can remove network access for the comparison matrix. +""" + +from __future__ import annotations + +import argparse +import hashlib +import json +import os +import re +import signal +import subprocess +import tarfile +import threading +from dataclasses import dataclass +from pathlib import Path, PurePosixPath +from typing import Any, BinaryIO, Iterable + + +SCHEMA = 1 +ORACLE = { + "project": "hapifhir/org.hl7.fhir.core", + "release": "6.10.2", + "source_commit": "d06577dbc5c62c74a2a8823fbc4830a3024d5b0b", +} +CORE_NAME = "hl7.fhir.r4.core" +CORE_VERSION = "4.0.1" +MAX_LOCK_BYTES = 4 * 1024 * 1024 +MAX_JSON_MEMBER_BYTES = 64 * 1024 * 1024 +MAX_JSON_MEMBERS = 100_000 +MAX_STDOUT_BYTES = 8 * 1024 * 1024 +MAX_STDERR_BYTES = 1024 * 1024 +MAX_DIAGNOSTIC_CHARS = 8192 +PROCESS_TIMEOUT_SECONDS = 180 + +ROOT_CAUSE_CONTEXT = "CF10_ORACLE_ROOT_CAUSE_COMMAND_F_CONTEXT_DEFECT" +ROOT_CAUSE_PINNED = "CF10_ORACLE_ROOT_CAUSE_PINNED_HL7_COMPARATOR_LIMITATION" +ROOT_CAUSE_MIXED = "CF10_ORACLE_ROOT_CAUSE_MIXED" +ROOT_CAUSE_NOT_PROVEN = "CF10_ORACLE_ROOT_CAUSE_NOT_PROVEN" + + +@dataclass(frozen=True) +class StateSpec: + case_id: str + side: str + package: str + version: str + archive_sha256: str + archive_bytes: int + lock_sha256: str + + +STATE_SPECS = ( + StateSpec( + "C001", + "before", + "hl7.fhir.us.core", + "8.0.1", + "3c02eef48ef10617021bee95e58cbc66d596ceda8cada24b72000d33ad67c464", + 2_713_046, + "2ba6240dc7ffc3c63d1fdaa6597775f083ae1937aa5ec21d34126487632f45ee", + ), + StateSpec( + "C001", + "after", + "hl7.fhir.us.core", + "9.0.0", + "d7b54d2ec2a48cea94ffea5d939ad67a681f80b94d69594a08cebac36da9e059", + 2_749_959, + "a88d94cce6743624829bbac5a64464a6d276620961cfb1f072d5518c71236558", + ), + StateSpec( + "C002", + "before", + "hl7.fhir.uv.ips", + "1.1.0", + "403c4141101810e924f2928287985084819d8a5cc3a62e2b3840a557129840ef", + 1_065_103, + "1bd1ca2c3c690f9de59c8403a0ea505832a5beecea46d789254a1b7a11f4b3fd", + ), + StateSpec( + "C002", + "after", + "hl7.fhir.uv.ips", + "2.0.1", + "7183242b70fb2a9058aa3701fb607517a3c2fd0e3100d1d8c538d744c2adf799", + 725_312, + "b2485f1caa7872aa3d95bcc3254addd9204fe62a59769265de7f271ed14b3953", + ), +) + +CASE_HYPOTHESES = { + "C001": { + "canonical_url": ( + "http://hl7.org/fhir/us/core/StructureDefinition/" + "head-occipital-frontal-circumference-percentile" + ), + "failing_path": "Observation.category", + }, + "C002": { + "canonical_url": "http://hl7.org/fhir/uv/ips/StructureDefinition/Composition-uv-ips", + "failing_path": "Composition.section", + }, +} + + +@dataclass(frozen=True) +class VerifiedState: + spec: StateSpec + root: Path + lock: dict[str, Any] + root_package: dict[str, Any] + core_package: dict[str, Any] + + def archive(self, package: dict[str, Any]) -> Path: + return self.root / "cache" / "sha256" / f"{package['sha256']}.tgz" + + +@dataclass(frozen=True) +class ProfileResource: + filename: str + value: dict[str, Any] + raw_sha256: str | None = None + + +@dataclass(frozen=True) +class MatchedProfilePair: + resource_key: str + canonical_url: str + lookup_version: str | None + before: ProfileResource + after: ProfileResource + + +@dataclass(frozen=True) +class ProcessResult: + returncode: int + stdout: bytes + stderr: bytes + + +def sha256_bytes(value: bytes) -> str: + return hashlib.sha256(value).hexdigest() + + +def sha256_file(path: Path) -> str: + digest = hashlib.sha256() + with path.open("rb") as stream: + while chunk := stream.read(1024 * 1024): + digest.update(chunk) + return digest.hexdigest() + + +def bounded_text(value: str, limit: int = MAX_DIAGNOSTIC_CHARS) -> str: + if len(value) <= limit: + return value + return value[:limit] + "... [diagnostic truncated]" + + +def canonical_json_bytes(value: Any) -> bytes: + return (json.dumps(value, indent=2, sort_keys=True, ensure_ascii=False) + "\n").encode( + "utf-8" + ) + + +def read_bounded(path: Path, limit: int) -> bytes: + with path.open("rb") as stream: + value = stream.read(limit + 1) + if len(value) > limit: + raise ValueError(f"{path.name} exceeds {limit} bytes") + return value + + +def _drain(stream: BinaryIO, limit: int, retained: bytearray, total: list[int]) -> None: + while True: + chunk = stream.read(64 * 1024) + if not chunk: + return + total[0] += len(chunk) + remaining = limit + 1 - len(retained) + if remaining > 0: + retained.extend(chunk[:remaining]) + + +def run_bounded(argv: list[str], timeout: int = PROCESS_TIMEOUT_SECONDS) -> ProcessResult: + process = subprocess.Popen( + argv, + stdin=subprocess.DEVNULL, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + start_new_session=True, + ) + assert process.stdout is not None + assert process.stderr is not None + stdout = bytearray() + stderr = bytearray() + stdout_total = [0] + stderr_total = [0] + threads = [ + threading.Thread( + target=_drain, + args=(process.stdout, MAX_STDOUT_BYTES, stdout, stdout_total), + daemon=True, + ), + threading.Thread( + target=_drain, + args=(process.stderr, MAX_STDERR_BYTES, stderr, stderr_total), + daemon=True, + ), + ] + for thread in threads: + thread.start() + try: + returncode = process.wait(timeout=timeout) + except subprocess.TimeoutExpired as error: + try: + os.killpg(process.pid, signal.SIGKILL) + except (AttributeError, OSError): + process.kill() + process.wait() + for thread in threads: + thread.join() + raise RuntimeError(f"process timed out after {timeout} seconds") from error + for thread in threads: + thread.join() + if stdout_total[0] > MAX_STDOUT_BYTES: + raise RuntimeError( + f"process stdout exceeded {MAX_STDOUT_BYTES} bytes: {stdout_total[0]}" + ) + if stderr_total[0] > MAX_STDERR_BYTES: + raise RuntimeError( + f"process stderr exceeded {MAX_STDERR_BYTES} bytes: {stderr_total[0]}" + ) + return ProcessResult(returncode, bytes(stdout), bytes(stderr)) + + +def state_path(work_root: Path, spec: StateSpec) -> Path: + return work_root / "states" / spec.case_id / spec.side + + +def identity(package: dict[str, Any]) -> dict[str, str]: + return { + "name": required_string(package, "name"), + "version": required_string(package, "version"), + "sha256": required_digest(package, "sha256"), + } + + +def required_string(value: dict[str, Any], key: str) -> str: + result = value.get(key) + if not isinstance(result, str) or not result: + raise ValueError(f"{key} must be a non-empty string") + return result + + +def required_digest(value: dict[str, Any], key: str) -> str: + result = required_string(value, key) + if len(result) != 64 or result != result.lower() or any( + char not in "0123456789abcdef" for char in result + ): + raise ValueError(f"{key} must be lowercase SHA-256") + return result + + +def package_identity(package: dict[str, Any]) -> tuple[str, str]: + return required_string(package, "name"), required_string(package, "version") + + +def select_exact( + lock: dict[str, Any], name: str, version: str, role: str +) -> dict[str, Any]: + packages = lock.get("packages") + if not isinstance(packages, list): + raise ValueError("lock packages must be an array") + matches = [ + package + for package in packages + if isinstance(package, dict) + and package.get("name") == name + and package.get("version") == version + ] + if len(matches) != 1: + raise ValueError(f"{role} {name}@{version} matched {len(matches)} locked packages") + return matches[0] + + +def select_dependency( + lock: dict[str, Any], parent: dict[str, Any], name: str, constraint: str +) -> dict[str, Any]: + # The retained CF-10 locks use exact dependency constraints. Refuse to invent semver behavior + # in this diagnostic rather than accidentally selecting a different concrete identity. + if not isinstance(constraint, str) or not constraint: + raise ValueError("dependency constraint must be a non-empty string") + return select_exact( + lock, + name, + constraint, + f"dependency of {parent['name']}@{parent['version']}", + ) + + +def verify_state(work_root: Path, spec: StateSpec) -> VerifiedState: + root = state_path(work_root, spec) + lock_path = root / "commandf.lock" + lock_bytes = read_bounded(lock_path, MAX_LOCK_BYTES) + actual_lock_sha = sha256_bytes(lock_bytes) + if actual_lock_sha != spec.lock_sha256: + raise ValueError( + f"{spec.case_id} {spec.side} lock digest mismatch: " + f"{actual_lock_sha} != {spec.lock_sha256}" + ) + lock = json.loads(lock_bytes) + if lock.get("schema") != 1: + raise ValueError("lock schema must be 1") + if lock.get("roots") != [f"{spec.package}@{spec.version}"]: + raise ValueError(f"{spec.case_id} {spec.side} root request mismatch") + root_package = select_exact(lock, spec.package, spec.version, "root") + core_package = select_exact(lock, CORE_NAME, CORE_VERSION, "core") + if required_digest(root_package, "sha256") != spec.archive_sha256: + raise ValueError(f"{spec.case_id} {spec.side} root archive digest mismatch") + + packages = lock.get("packages") + assert isinstance(packages, list) + seen: dict[tuple[str, str], str] = {} + for package in packages: + if not isinstance(package, dict): + raise ValueError("locked package must be an object") + identity_key = package_identity(package) + package_digest = required_digest(package, "sha256") + if identity_key in seen: + raise ValueError( + f"duplicate locked package identity {identity_key[0]}@{identity_key[1]}: " + f"{seen[identity_key]} and {package_digest}" + ) + seen[identity_key] = package_digest + archive = root / "cache" / "sha256" / f"{package_digest}.tgz" + if not archive.is_file(): + raise ValueError( + f"missing verified archive for {identity_key[0]}@{identity_key[1]}" + ) + actual_digest = sha256_file(archive) + if actual_digest != package_digest: + raise ValueError( + f"archive digest mismatch for {identity_key[0]}@{identity_key[1]}" + ) + root_archive = root / "cache" / "sha256" / f"{spec.archive_sha256}.tgz" + if root_archive.stat().st_size != spec.archive_bytes: + raise ValueError(f"{spec.case_id} {spec.side} root archive size mismatch") + return VerifiedState(spec, root, lock, root_package, core_package) + + +def is_fhir_core(name: str) -> bool: + return name.startswith("hl7.fhir.r") and name.endswith(".core") + + +def root_core_family(state: VerifiedState) -> str: + dependencies = state.root_package.get("dependencies") + if not isinstance(dependencies, dict): + raise ValueError("root dependencies must be an object") + cores = sorted(name for name in dependencies if is_fhir_core(name)) + if cores != [CORE_NAME]: + raise ValueError(f"root must declare exactly {CORE_NAME}: {cores}") + return cores[0] + + +def package_core_families( + lock: dict[str, Any], package: dict[str, Any] +) -> set[str]: + memo: dict[tuple[str, str], frozenset[str]] = {} + visiting: set[tuple[str, str]] = set() + + def visit(current: dict[str, Any]) -> frozenset[str]: + key = package_identity(current) + if key in memo: + return memo[key] + if key in visiting: + raise ValueError( + f"dependency cycle while scoping core family at {key[0]}@{key[1]}" + ) + if is_fhir_core(key[0]): + result = frozenset((key[0],)) + memo[key] = result + return result + dependencies = current.get("dependencies") + if not isinstance(dependencies, dict): + raise ValueError(f"dependencies for {key[0]}@{key[1]} must be an object") + visiting.add(key) + try: + families: set[str] = set() + for dependency_name in sorted(dependencies): + dependency = select_dependency( + lock, current, dependency_name, dependencies[dependency_name] + ) + families.update(visit(dependency)) + result = frozenset(families) + memo[key] = result + return result + finally: + visiting.remove(key) + + return set(visit(package)) + + +def direct_context_packages(state: VerifiedState) -> list[dict[str, Any]]: + dependencies = state.root_package.get("dependencies") + if not isinstance(dependencies, dict): + raise ValueError("root dependencies must be an object") + output = [] + for name in sorted(dependencies): + package = select_dependency( + state.lock, state.root_package, name, dependencies[name] + ) + if package_identity(package) == (CORE_NAME, CORE_VERSION): + continue + output.append(package) + return output + + +def full_context_packages(state: VerifiedState) -> list[dict[str, Any]]: + target_core = root_core_family(state) + selected: dict[tuple[str, str], dict[str, Any]] = {} + traversed: set[tuple[str, str]] = set() + + def include_from_r4_branch(package: dict[str, Any]) -> None: + key = package_identity(package) + if key in traversed: + return + traversed.add(key) + if key == (CORE_NAME, CORE_VERSION): + return + if is_fhir_core(key[0]): + return + dependencies = package.get("dependencies") + if not isinstance(dependencies, dict): + raise ValueError(f"dependencies for {key[0]}@{key[1]} must be an object") + direct_cores = {name for name in dependencies if is_fhir_core(name)} + if target_core in direct_cores: + other_cores = direct_cores - {target_core} + if other_cores: + raise ValueError( + f"mixed direct FHIR core families at {key[0]}@{key[1]}: " + f"{sorted(direct_cores)}" + ) + else: + families = package_core_families(state.lock, package) + if target_core in families and len(families) > 1: + raise ValueError( + f"mixed transitive FHIR core families at {key[0]}@{key[1]}: " + f"{sorted(families)}" + ) + if families and families != {target_core}: + return + if key in selected: + raise ValueError(f"duplicate locked package identity {key[0]}@{key[1]}") + selected[key] = package + for dependency_name in sorted(dependencies): + dependency = select_dependency( + state.lock, package, dependency_name, dependencies[dependency_name] + ) + include_from_r4_branch(dependency) + + root_dependencies = state.root_package.get("dependencies") + if not isinstance(root_dependencies, dict): + raise ValueError("root dependencies must be an object") + for dependency_name in sorted(root_dependencies): + dependency = select_dependency( + state.lock, + state.root_package, + dependency_name, + root_dependencies[dependency_name], + ) + include_from_r4_branch(dependency) + + ordered: list[dict[str, Any]] = [] + visiting: set[tuple[str, str]] = set() + visited: set[tuple[str, str]] = set() + + def visit(package: dict[str, Any]) -> None: + key = package_identity(package) + if key in visited: + return + if key in visiting: + raise ValueError(f"dependency cycle while ordering context at {key[0]}@{key[1]}") + visiting.add(key) + dependencies = package.get("dependencies") + if not isinstance(dependencies, dict): + raise ValueError(f"dependencies for {key[0]}@{key[1]} must be an object") + for dependency_name in sorted(dependencies): + dependency = select_dependency( + state.lock, package, dependency_name, dependencies[dependency_name] + ) + if package_identity(dependency) in selected: + visit(dependency) + visiting.remove(key) + visited.add(key) + ordered.append(package) + + for key in sorted(selected): + visit(selected[key]) + return ordered + + +def context_packages(state: VerifiedState, mode: str) -> list[dict[str, Any]]: + if mode == "DIRECT": + return direct_context_packages(state) + if mode == "FULL_CLOSURE": + return full_context_packages(state) + raise ValueError(f"unknown context mode {mode}") + + +def iter_json_resources( + archive: Path, +) -> Iterable[tuple[str, dict[str, Any], str]]: + count = 0 + with tarfile.open(archive, mode="r:gz") as package: + for member in package: + if not member.isfile() or not member.name.startswith("package/"): + continue + if not member.name.endswith(".json") or member.name == "package/package.json": + continue + count += 1 + if count > MAX_JSON_MEMBERS: + raise ValueError(f"archive contains more than {MAX_JSON_MEMBERS} JSON resources") + if member.size > MAX_JSON_MEMBER_BYTES: + raise ValueError(f"JSON member {member.name} exceeds size limit") + stream = package.extractfile(member) + if stream is None: + raise ValueError(f"unable to read {member.name}") + raw = stream.read(MAX_JSON_MEMBER_BYTES + 1) + if len(raw) > MAX_JSON_MEMBER_BYTES: + raise ValueError(f"JSON member {member.name} exceeds size limit") + value = json.loads(raw) + if not isinstance(value, dict): + raise ValueError(f"FHIR resource {member.name} must be an object") + yield PurePosixPath(member.name).name, value, sha256_bytes(raw) + + +def find_profile( + state: VerifiedState, canonical_url: str, lookup_version: str | None = None +) -> ProfileResource: + matches = [] + for filename, value, raw_sha256 in iter_json_resources( + state.archive(state.root_package) + ): + if ( + value.get("resourceType") == "StructureDefinition" + and value.get("url") == canonical_url + and (lookup_version is None or value.get("version") == lookup_version) + ): + matches.append(ProfileResource(filename, value, raw_sha256)) + if len(matches) != 1: + identity_text = canonical_url + if lookup_version is not None: + identity_text += f"|{lookup_version}" + raise ValueError( + f"{state.spec.case_id} {state.spec.side} canonical " + f"{identity_text} matched {len(matches)} profiles" + ) + return matches[0] + + +def root_canonical_inventory( + state: VerifiedState, +) -> tuple[dict[str, int], list[ProfileResource]]: + counts: dict[str, int] = {} + profiles = [] + for filename, value, raw_sha256 in iter_json_resources( + state.archive(state.root_package) + ): + url = value.get("url") + if isinstance(url, str) and url: + counts[url] = counts.get(url, 0) + 1 + if value.get("resourceType") == "StructureDefinition" and isinstance(url, str) and url: + profiles.append(ProfileResource(filename, value, raw_sha256)) + return counts, profiles + + +def profile_resource_key( + profile: ProfileResource, + before_counts: dict[str, int], + after_counts: dict[str, int], +) -> tuple[str, str, str | None]: + url = profile.value.get("url") + if not isinstance(url, str) or not url: + raise ValueError(f"{profile.filename} has no canonical URL") + if before_counts.get(url, 0) <= 1 and after_counts.get(url, 0) <= 1: + return url, url, None + version = profile.value.get("version") + if not isinstance(version, str) or not version.strip(): + raise ValueError( + f"canonical multiplicity requires a usable version in {profile.filename}" + ) + return f"{url}|{version}", url, version + + +def structural_changed_canonical_keys(report: dict[str, Any]) -> set[str]: + if report.get("schema") != 1: + raise ValueError("structural report schema must be 1") + changes = report.get("changes") + if not isinstance(changes, list): + raise ValueError("structural report changes must be an array") + keys: set[str] = set() + for change in changes: + if not isinstance(change, dict): + raise ValueError("structural change must be an object") + resource = change.get("resource") + if not isinstance(resource, dict): + raise ValueError("structural change resource must be an object") + if resource.get("kind") != "canonical": + continue + value = resource.get("value") + if not isinstance(value, str) or not value: + raise ValueError("canonical structural resource key must be non-empty") + keys.add(value) + return keys + + +def matched_changed_profiles( + before: VerifiedState, + after: VerifiedState, + structural_report: dict[str, Any], +) -> list[MatchedProfilePair]: + before_counts, before_profiles = root_canonical_inventory(before) + after_counts, after_profiles = root_canonical_inventory(after) + + def build_index(profiles: list[ProfileResource]) -> dict[str, tuple[str, str | None, ProfileResource]]: + index: dict[str, tuple[str, str | None, ProfileResource]] = {} + for profile in profiles: + key, url, version = profile_resource_key( + profile, before_counts, after_counts + ) + if key in index: + raise ValueError(f"duplicate StructureDefinition resource key {key}") + index[key] = (url, version, profile) + return index + + before_index = build_index(before_profiles) + after_index = build_index(after_profiles) + changed = structural_changed_canonical_keys(structural_report) + output = [] + for key in sorted(before_index.keys() & after_index.keys() & changed): + before_url, before_version, before_profile = before_index[key] + after_url, after_version, after_profile = after_index[key] + if (before_url, before_version) != (after_url, after_version): + raise ValueError(f"matched StructureDefinition identity differs for {key}") + output.append( + MatchedProfilePair( + key, + before_url, + before_version, + before_profile, + after_profile, + ) + ) + return output + + +def element_view_shape(value: dict[str, Any], view: str, failing_path: str) -> dict[str, Any]: + container = value.get(view) + elements = container.get("element", []) if isinstance(container, dict) else [] + if not isinstance(elements, list): + raise ValueError(f"{view}.element must be an array") + + exact_indices = [ + index + for index, element in enumerate(elements) + if isinstance(element, dict) and element.get("path") == failing_path + ] + window_indices: set[int] = set() + for index in exact_indices: + window_indices.update( + candidate + for candidate in (index - 1, index, index + 1) + if 0 <= candidate < len(elements) + ) + + def element_evidence(index: int) -> dict[str, Any]: + element = elements[index] + assert isinstance(element, dict) + return { + "index": index, + "path": element.get("path"), + "id": element.get("id"), + "slice_name": element.get("sliceName"), + "has_slicing": "slicing" in element, + "slicing": element.get("slicing"), + "raw_element": element, + } + + exact = [element_evidence(index) for index in exact_indices] + named_indices = [ + item["index"] for item in exact if isinstance(item.get("slice_name"), str) + ] + slicing_indices = [item["index"] for item in exact if item["has_slicing"]] + first_named = min(named_indices) if named_indices else None + local_slicing_before_named = ( + first_named is not None + and any(index < first_named for index in slicing_indices) + ) + return { + "exists": isinstance(container, dict), + "element_count": len(elements), + "path_elements": exact, + "window": [element_evidence(index) for index in sorted(window_indices)], + "first_named_slice_index": first_named, + "slicing_indices": slicing_indices, + "slicing_before_first_named_slice": local_slicing_before_named, + "named_slice_precedes_slicing": first_named is not None + and not local_slicing_before_named, + } + + +def differential_shape(profile: ProfileResource, failing_path: str) -> dict[str, Any]: + value = profile.value + return { + "url": value.get("url"), + "version": value.get("version"), + "resource_sha256": profile.raw_sha256, + "base_definition": value.get("baseDefinition"), + "derivation": value.get("derivation"), + "snapshot_exists": isinstance(value.get("snapshot"), dict), + "differential_exists": isinstance(value.get("differential"), dict), + "failing_path": failing_path, + "differential": element_view_shape(value, "differential", failing_path), + "snapshot": element_view_shape(value, "snapshot", failing_path), + } + + +def context_evidence( + state: VerifiedState, packages: list[dict[str, Any]] +) -> dict[str, Any]: + core = identity(state.core_package) + subject = identity(state.root_package) + additional = [identity(package) for package in packages] + return { + "core": core, + "additional_packages": additional, + "subject": subject, + "load_order": [core, *additional, subject], + } + + +def profile_identity(profile: ProfileResource) -> dict[str, str | None]: + def optional_string(key: str) -> str | None: + value = profile.value.get(key) + return value if isinstance(value, str) and value.strip() else None + + return { + "url": optional_string("url"), + "version": optional_string("version"), + "id": optional_string("id"), + "type": optional_string("type"), + } + + +def probe_argv( + java: Path, + probe_classes: Path, + oracle_jar: Path, + pair: MatchedProfilePair, + left: VerifiedState, + right: VerifiedState, + left_profile: ProfileResource, + right_profile: ProfileResource, + left_context: list[dict[str, Any]], + right_context: list[dict[str, Any]], +) -> list[str]: + if required_digest(left.core_package, "sha256") != required_digest( + right.core_package, "sha256" + ): + raise ValueError("left/right R4 core package digests differ") + argv = [ + str(java), + "-cp", + os.pathsep.join((str(probe_classes), str(oracle_jar))), + "dev.commandf.oracle.qualification.QualificationProbe", + "--core-package", + str(left.archive(left.core_package)), + "--left-package", + str(left.archive(left.root_package)), + "--right-package", + str(right.archive(right.root_package)), + ] + for package in left_context: + argv.extend(["--left-context-package", str(left.archive(package))]) + for package in right_context: + argv.extend(["--right-context-package", str(right.archive(package))]) + argv.extend(["--left-url", pair.canonical_url]) + if pair.lookup_version is not None: + argv.extend(["--left-version", pair.lookup_version]) + argv.extend(["--right-url", pair.canonical_url]) + if pair.lookup_version is not None: + argv.extend(["--right-version", pair.lookup_version]) + return argv + + +def invocation_evidence( + label: str, + mode: str, + pair: MatchedProfilePair, + left: VerifiedState, + right: VerifiedState, + left_profile: ProfileResource, + right_profile: ProfileResource, + left_context: list[dict[str, Any]], + right_context: list[dict[str, Any]], +) -> dict[str, Any]: + return { + "case_id": left.spec.case_id, + "probe": label, + "context_mode": mode, + "package": left.spec.package, + "resource_key": pair.resource_key, + "canonical_url": pair.canonical_url, + "lookup_version": pair.lookup_version, + "left": { + "package": identity(left.root_package), + "canonical_version": left_profile.value.get("version"), + "filename": left_profile.filename, + "resource": profile_identity(left_profile), + "context": context_evidence(left, left_context), + }, + "right": { + "package": identity(right.root_package), + "canonical_version": right_profile.value.get("version"), + "filename": right_profile.filename, + "resource": profile_identity(right_profile), + "context": context_evidence(right, right_context), + }, + } + + +def run_probe( + work_root: Path, + invocation: dict[str, Any], + argv: list[str], +) -> dict[str, Any]: + event = {"event": "probe_start", **invocation} + print(json.dumps(event, sort_keys=True, separators=(",", ":")), flush=True) + process = run_bounded(argv) + stdout = process.stdout.decode("utf-8") + stderr = process.stderr.decode("utf-8") + forbidden_paths = {str(work_root.resolve())} + for index, argument in enumerate(argv): + candidates = ( + argument.split(os.pathsep) + if index > 0 and argv[index - 1] in {"-cp", "--class-path"} + else (argument,) + ) + for value in candidates: + candidate = Path(value) + if candidate.is_absolute(): + forbidden_paths.add(str(candidate)) + if any(path in stdout or path in stderr for path in forbidden_paths): + raise ValueError("probe output contains a host-absolute path") + result = json.loads(stdout) + if result.get("schema") != 1 or result.get("oracle") != ORACLE: + raise ValueError("qualification probe schema/oracle identity mismatch") + status = result.get("status") + if status in {"agreement", "uncomparable"}: + raise ValueError("qualification exception was coerced into an oracle evidence status") + if status not in {"completed", "exception"}: + raise ValueError(f"unexpected qualification status {status}") + if status == "completed" and process.returncode != 0: + raise ValueError("completed comparison exited non-zero") + if status == "exception" and process.returncode == 0: + raise ValueError("comparison exception exited zero") + if result.get("phase") == "comparison": + if result.get("left_resource") != invocation["left"]["resource"]: + raise ValueError("resolved left StructureDefinition identity mismatch") + if result.get("right_resource") != invocation["right"]["resource"]: + raise ValueError("resolved right StructureDefinition identity mismatch") + return { + **invocation, + "process_exit_code": process.returncode, + "process_stderr": bounded_text(stderr.replace("\r\n", "\n").rstrip()), + "result": result, + } + + +def is_slice_definition_failure(result: dict[str, Any]) -> bool: + return ( + result.get("status") == "exception" + and result.get("phase") == "comparison" + and result.get("exception_class") + == "org.hl7.fhir.exceptions.DefinitionException" + and exception_path(result) is not None + ) + + +def failure_signature(result: dict[str, Any]) -> tuple[Any, ...]: + return ( + result.get("status"), + result.get("phase"), + result.get("exception_class"), + result.get("exception_message"), + exception_path(result), + ) + + +def case_classification( + case_id: str, + probes: list[dict[str, Any]], + discovered_result: dict[str, Any] | None, +) -> str: + if discovered_result is None or not is_slice_definition_failure(discovered_result): + return ROOT_CAUSE_NOT_PROVEN + case_probes = [probe for probe in probes if probe["case_id"] == case_id] + by_key = { + (probe["probe"], probe["context_mode"]): probe["result"] + for probe in case_probes + } + required = { + (label, mode) + for label in ("self_before", "self_after", "cross") + for mode in ("DIRECT", "FULL_CLOSURE") + } + if not required.issubset(by_key): + return ROOT_CAUSE_NOT_PROVEN + discovered_signature = failure_signature(discovered_result) + cross_direct = by_key[("cross", "DIRECT")] + cross_full = by_key[("cross", "FULL_CLOSURE")] + if failure_signature(cross_direct) != discovered_signature: + return ROOT_CAUSE_NOT_PROVEN + context_fixed = ( + cross_full.get("status") == "completed" + and cross_full.get("phase") == "comparison" + ) + pinned_self_failure = False + for label in ("self_before", "self_after"): + direct = by_key[(label, "DIRECT")] + full = by_key[(label, "FULL_CLOSURE")] + if ( + failure_signature(direct) + == failure_signature(full) + == discovered_signature + ): + pinned_self_failure = True + pinned_cross_failure = failure_signature(cross_full) == discovered_signature + if pinned_self_failure and context_fixed: + return ROOT_CAUSE_NOT_PROVEN + if pinned_self_failure and pinned_cross_failure: + return ROOT_CAUSE_PINNED + if context_fixed: + return ROOT_CAUSE_CONTEXT + return ROOT_CAUSE_NOT_PROVEN + + +def primary_classification(case_classes: dict[str, str]) -> str: + values = set(case_classes.values()) + if values == {ROOT_CAUSE_PINNED}: + return ROOT_CAUSE_PINNED + if values == {ROOT_CAUSE_CONTEXT}: + return ROOT_CAUSE_CONTEXT + if values == {ROOT_CAUSE_PINNED, ROOT_CAUSE_CONTEXT}: + return ROOT_CAUSE_MIXED + return ROOT_CAUSE_NOT_PROVEN + + +def structural_report_path(work_root: Path, case_id: str) -> Path: + return work_root / "cases" / case_id / "structural.json" + + +def validate_structural_report( + report: dict[str, Any], before: VerifiedState, after: VerifiedState +) -> None: + if report.get("schema") != 1: + raise ValueError("structural report schema must be 1") + if report.get("package_name") != before.spec.package: + raise ValueError(f"{before.spec.case_id} structural package identity mismatch") + expected_before = { + "version": before.spec.version, + "archive_sha256": before.spec.archive_sha256, + } + expected_after = { + "version": after.spec.version, + "archive_sha256": after.spec.archive_sha256, + } + if report.get("before") != expected_before or report.get("after") != expected_after: + raise ValueError(f"{before.spec.case_id} structural state evidence mismatch") + structural_changed_canonical_keys(report) + + +def load_structural_report( + work_root: Path, before: VerifiedState, after: VerifiedState +) -> dict[str, Any]: + report = json.loads( + read_bounded(structural_report_path(work_root, before.spec.case_id), MAX_STDOUT_BYTES) + ) + if not isinstance(report, dict): + raise ValueError("structural report must be an object") + validate_structural_report(report, before, after) + return report + + +def acquire(commandf: Path, work_root: Path, output: Path) -> None: + if work_root.exists(): + raise ValueError(f"qualification work root already exists: {work_root.name}") + for spec in STATE_SPECS: + root = state_path(work_root, spec) + root.mkdir(parents=True, exist_ok=False) + resolve = run_bounded( + [ + str(commandf), + "pkg", + "resolve", + f"{spec.package}@{spec.version}", + "--cache", + str(root / "cache"), + "--lock", + str(root / "commandf.lock"), + ], + timeout=600, + ) + if resolve.returncode != 0: + raise RuntimeError( + f"resolver failed for {spec.case_id} {spec.side}: " + + bounded_text(resolve.stderr.decode("utf-8", errors="replace")) + ) + verify = run_bounded( + [ + str(commandf), + "pkg", + "verify", + "--cache", + str(root / "cache"), + "--lock", + str(root / "commandf.lock"), + ] + ) + if verify.returncode != 0: + raise RuntimeError(f"cache verification failed for {spec.case_id} {spec.side}") + verify_state(work_root, spec) + states = [verify_state(work_root, spec) for spec in STATE_SPECS] + states_by_key = { + (state.spec.case_id, state.spec.side): state for state in states + } + structural_evidence = [] + for case_id in ("C001", "C002"): + before = states_by_key[(case_id, "before")] + after = states_by_key[(case_id, "after")] + process = run_bounded( + [ + str(commandf), + "diff", + before.spec.package, + "--before-lock", + str(before.root / "commandf.lock"), + "--before-cache", + str(before.root / "cache"), + "--after-lock", + str(after.root / "commandf.lock"), + "--after-cache", + str(after.root / "cache"), + ] + ) + if process.returncode != 0: + raise RuntimeError( + f"structural diff failed for {case_id}: " + + bounded_text(process.stderr.decode("utf-8", errors="replace")) + ) + report = json.loads(process.stdout) + if not isinstance(report, dict): + raise ValueError(f"{case_id} structural report must be an object") + validate_structural_report(report, before, after) + report_bytes = canonical_json_bytes(report) + report_path = structural_report_path(work_root, case_id) + report_path.parent.mkdir(parents=True, exist_ok=False) + report_path.write_bytes(report_bytes) + structural_evidence.append( + { + "case_id": case_id, + "sha256": sha256_bytes(report_bytes), + "changed_canonical_count": len( + structural_changed_canonical_keys(report) + ), + } + ) + evidence = { + "schema": SCHEMA, + "states": [ + { + "case_id": state.spec.case_id, + "side": state.spec.side, + "root": identity(state.root_package), + "lock_sha256": state.spec.lock_sha256, + "closure": [identity(package) for package in state.lock["packages"]], + } + for state in states + ], + "structural_reports": structural_evidence, + } + output.write_bytes(canonical_json_bytes(evidence)) + + +SLICE_FAILURE = re.compile( + r"Found a slice at '([^']+)'\s+but\s+there was no definition for the slicing" +) + + +def exception_path(result: dict[str, Any]) -> str | None: + message = result.get("exception_message") + if not isinstance(message, str): + return None + match = SLICE_FAILURE.search(message.replace("\r", " ").replace("\n", " ")) + return match.group(1) if match else None + + +def execute_pair_probe( + java: Path, + probe_classes: Path, + oracle_jar: Path, + work_root: Path, + pair: MatchedProfilePair, + label: str, + mode: str, + left: VerifiedState, + right: VerifiedState, + left_profile: ProfileResource, + right_profile: ProfileResource, + extra_evidence: dict[str, Any] | None = None, +) -> dict[str, Any]: + left_context = context_packages(left, mode) + right_context = context_packages(right, mode) + invocation = invocation_evidence( + label, + mode, + pair, + left, + right, + left_profile, + right_profile, + left_context, + right_context, + ) + if extra_evidence: + invocation.update(extra_evidence) + argv = probe_argv( + java, + probe_classes, + oracle_jar, + pair, + left, + right, + left_profile, + right_profile, + left_context, + right_context, + ) + return run_probe(work_root, invocation, argv) + + +def context_mode_comparisons(probes: list[dict[str, Any]]) -> list[dict[str, Any]]: + output = [] + for case_id in ("C001", "C002"): + for label in ("self_before", "self_after", "cross"): + selected = { + probe["context_mode"]: probe["result"] + for probe in probes + if probe["case_id"] == case_id and probe["probe"] == label + } + if set(selected) != {"DIRECT", "FULL_CLOSURE"}: + continue + direct = selected["DIRECT"] + full = selected["FULL_CLOSURE"] + direct_path = exception_path(direct) + full_path = exception_path(full) + output.append( + { + "case_id": case_id, + "probe": label, + "direct_status": direct.get("status"), + "full_closure_status": full.get("status"), + "exception_class_changed": direct.get("exception_class") + != full.get("exception_class"), + "exception_path_changed": direct_path != full_path, + "direct_exception_path": direct_path, + "full_closure_exception_path": full_path, + } + ) + return output + + +def qualify( + java: Path, + probe_classes: Path, + oracle_jar: Path, + work_root: Path, + output: Path, +) -> None: + states = { + (spec.case_id, spec.side): verify_state(work_root, spec) for spec in STATE_SPECS + } + probes = [] + discovery_probes = [] + discoveries = [] + shapes = [] + discovered_results: dict[str, dict[str, Any] | None] = {} + + for case_id in ("C001", "C002"): + before = states[(case_id, "before")] + after = states[(case_id, "after")] + structural = load_structural_report(work_root, before, after) + candidates = matched_changed_profiles(before, after, structural) + selected: MatchedProfilePair | None = None + selected_result: dict[str, Any] | None = None + selected_index: int | None = None + terminal_result: dict[str, Any] | None = None + for index, pair in enumerate(candidates): + probe = execute_pair_probe( + java, + probe_classes, + oracle_jar, + work_root, + pair, + "discovery_cross", + "DIRECT", + before, + after, + pair.before, + pair.after, + { + "candidate_index": index, + "candidate_count": len(candidates), + }, + ) + discovery_probes.append(probe) + if probe["result"].get("status") == "exception": + terminal_result = probe["result"] + if is_slice_definition_failure(terminal_result): + selected = pair + selected_result = terminal_result + selected_index = index + break + + observed_path = exception_path(selected_result or {}) + discovered_results[case_id] = selected_result + hypothesis = CASE_HYPOTHESES[case_id] + discovery = { + "case_id": case_id, + "changed_matched_structure_definition_count": len(candidates), + "selected_candidate_index": selected_index, + "selected_resource_key": selected.resource_key if selected else None, + "selected_canonical_url": selected.canonical_url if selected else None, + "selected_lookup_version": selected.lookup_version if selected else None, + "observed_exception_path": observed_path, + "terminal_exception_phase": ( + terminal_result.get("phase") if terminal_result else None + ), + "terminal_exception_class": ( + terminal_result.get("exception_class") if terminal_result else None + ), + "terminal_exception_message": ( + terminal_result.get("exception_message") if terminal_result else None + ), + "hypothesis": hypothesis, + "hypothesis_canonical_confirmed": selected is not None + and selected.canonical_url == hypothesis["canonical_url"], + "hypothesis_path_confirmed": observed_path == hypothesis["failing_path"], + } + discoveries.append(discovery) + if selected is None: + continue + + if observed_path is not None: + for side, profile in (("before", selected.before), ("after", selected.after)): + shapes.append( + { + "case_id": case_id, + "side": side, + "filename": profile.filename, + **differential_shape(profile, observed_path), + } + ) + + combinations = ( + ("self_before", before, before, selected.before, selected.before), + ("self_after", after, after, selected.after, selected.after), + ("cross", before, after, selected.before, selected.after), + ) + for label, left, right, left_profile, right_profile in combinations: + for mode in ("DIRECT", "FULL_CLOSURE"): + probes.append( + execute_pair_probe( + java, + probe_classes, + oracle_jar, + work_root, + selected, + label, + mode, + left, + right, + left_profile, + right_profile, + ) + ) + + case_classes = { + case_id: case_classification( + case_id, probes, discovered_results.get(case_id) + ) + for case_id in ("C001", "C002") + } + report = { + "schema": SCHEMA, + "oracle": ORACLE, + "source_path": { + "snapshot_then_differential": "StructureDefinitionComparer.compare", + "differential_navigation": "DefinitionNavigator(diff=true)", + "local_slice_failure": "DefinitionNavigator.loadChildren/DN_SLICE_NO_DEFINITION", + }, + "discoveries": discoveries, + "discovery_probes": discovery_probes, + "differential_shapes": shapes, + "probes": probes, + "context_mode_comparisons": context_mode_comparisons(probes), + "case_classifications": case_classes, + "root_cause_class": primary_classification(case_classes), + } + output.write_bytes(canonical_json_bytes(report)) + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser() + subparsers = parser.add_subparsers(dest="command", required=True) + acquire_parser = subparsers.add_parser("acquire") + acquire_parser.add_argument("--commandf", type=Path, required=True) + acquire_parser.add_argument("--work-root", type=Path, required=True) + acquire_parser.add_argument("--output", type=Path, required=True) + qualify_parser = subparsers.add_parser("qualify") + qualify_parser.add_argument("--java", type=Path, required=True) + qualify_parser.add_argument("--probe-classes", type=Path, required=True) + qualify_parser.add_argument("--oracle-jar", type=Path, required=True) + qualify_parser.add_argument("--work-root", type=Path, required=True) + qualify_parser.add_argument("--output", type=Path, required=True) + return parser.parse_args() + + +def main() -> None: + args = parse_args() + if args.command == "acquire": + acquire(args.commandf, args.work_root, args.output) + else: + qualify( + args.java, + args.probe_classes, + args.oracle_jar, + args.work_root, + args.output, + ) + + +if __name__ == "__main__": + main() diff --git a/.github/scripts/cf10-oracle-qualification/test_qualify_cf10_oracle.py b/.github/scripts/cf10-oracle-qualification/test_qualify_cf10_oracle.py new file mode 100644 index 00000000..24914c6c --- /dev/null +++ b/.github/scripts/cf10-oracle-qualification/test_qualify_cf10_oracle.py @@ -0,0 +1,457 @@ +from __future__ import annotations + +import importlib.util +import io +import json +import os +import sys +import tarfile +import tempfile +import unittest +from pathlib import Path +from unittest import mock + + +SCRIPT = Path(__file__).with_name("qualify_cf10_oracle.py") +SPEC = importlib.util.spec_from_file_location("qualify_cf10_oracle", SCRIPT) +assert SPEC is not None and SPEC.loader is not None +qualification = importlib.util.module_from_spec(SPEC) +sys.modules[SPEC.name] = qualification +SPEC.loader.exec_module(qualification) + + +def package(name: str, version: str, dependencies: dict[str, str] | None = None) -> dict: + return { + "name": name, + "version": version, + "sha256": (name + version).encode().hex()[:64].ljust(64, "0"), + "source": f"https://example.test/{name}/{version}", + "dependencies": dependencies or {}, + } + + +def state_for(lock: dict, root: dict, core: dict) -> qualification.VerifiedState: + spec = qualification.StateSpec( + "T001", + "before", + root["name"], + root["version"], + root["sha256"], + 1, + "a" * 64, + ) + return qualification.VerifiedState(spec, Path("."), lock, root, core) + + +def probe_result(status: str, exception: str | None = None, message: str | None = None) -> dict: + return { + "status": status, + "phase": "comparison", + "exception_class": exception, + "exception_message": message, + } + + +def state_with_resources( + parent: Path, + side: str, + version: str, + resources: list[tuple[str, dict]], +) -> qualification.VerifiedState: + root = package("example.root", version) + core = package("hl7.fhir.r4.core", "4.0.1") + state_root = parent / side + archive = state_root / "cache" / "sha256" / f"{root['sha256']}.tgz" + archive.parent.mkdir(parents=True) + with tarfile.open(archive, mode="w:gz") as package_file: + for filename, value in resources: + raw = qualification.canonical_json_bytes(value) + info = tarfile.TarInfo(f"package/{filename}") + info.size = len(raw) + package_file.addfile(info, io.BytesIO(raw)) + spec = qualification.StateSpec( + "T001", + side, + root["name"], + root["version"], + root["sha256"], + archive.stat().st_size, + "a" * 64, + ) + return qualification.VerifiedState( + spec, state_root, {"packages": [root, core]}, root, core + ) + + +def structural_report(*keys: str) -> dict: + return { + "schema": 1, + "changes": [ + {"resource": {"kind": "canonical", "value": key}} for key in keys + ], + } + + +class QualificationTests(unittest.TestCase): + def test_unique_canonical_version_change_uses_unversioned_production_key(self) -> None: + url = "https://example.test/StructureDefinition/unique" + with tempfile.TemporaryDirectory() as directory: + root = Path(directory) + before = state_with_resources( + root, + "before", + "1.0.0", + [("before.json", {"resourceType": "StructureDefinition", "url": url, "version": "1"})], + ) + after = state_with_resources( + root, + "after", + "2.0.0", + [("after.json", {"resourceType": "StructureDefinition", "url": url, "version": "2"})], + ) + pairs = qualification.matched_changed_profiles( + before, after, structural_report(url) + ) + self.assertEqual([pair.resource_key for pair in pairs], [url]) + self.assertIsNone(pairs[0].lookup_version) + + def test_multi_version_canonical_identities_are_not_collapsed(self) -> None: + url = "https://example.test/StructureDefinition/versioned" + before_resources = [ + ( + f"before-{version}.json", + { + "resourceType": "StructureDefinition", + "url": url, + "version": version, + "status": "draft", + }, + ) + for version in ("1", "2") + ] + after_resources = [ + ( + f"after-{version}.json", + { + "resourceType": "StructureDefinition", + "url": url, + "version": version, + "status": "active", + }, + ) + for version in ("1", "2") + ] + with tempfile.TemporaryDirectory() as directory: + root = Path(directory) + before = state_with_resources(root, "before", "1.0.0", before_resources) + after = state_with_resources(root, "after", "2.0.0", after_resources) + pairs = qualification.matched_changed_profiles( + before, + after, + structural_report(f"{url}|1", f"{url}|2"), + ) + self.assertEqual( + [(pair.resource_key, pair.lookup_version) for pair in pairs], + [(f"{url}|1", "1"), (f"{url}|2", "2")], + ) + + def test_full_closure_preserves_versions_excludes_r5_and_orders_dependencies_first(self) -> None: + core = package("hl7.fhir.r4.core", "4.0.1") + r5_core = package("hl7.fhir.r5.core", "5.0.0") + extension_v1 = package( + "hl7.fhir.uv.extensions.r4", "1.0.0", {core["name"]: core["version"]} + ) + extension_v2 = package( + "hl7.fhir.uv.extensions.r4", "5.2.0", {core["name"]: core["version"]} + ) + neutral_leaf = package("hl7.fhir.r4.examples", "4.0.1") + r5_branch = package( + "hl7.fhir.uv.extensions", "5.1.0", {r5_core["name"]: r5_core["version"]} + ) + bridge = package( + "example.bridge", + "1.0.0", + { + core["name"]: core["version"], + extension_v1["name"]: extension_v1["version"], + neutral_leaf["name"]: neutral_leaf["version"], + r5_branch["name"]: r5_branch["version"], + }, + ) + root = package( + "example.root", + "1.0.0", + { + core["name"]: core["version"], + bridge["name"]: bridge["version"], + extension_v2["name"]: extension_v2["version"], + }, + ) + lock = { + "schema": 1, + "roots": ["example.root@1.0.0"], + "packages": [ + bridge, + core, + extension_v1, + extension_v2, + neutral_leaf, + r5_branch, + r5_core, + root, + ], + } + contexts = qualification.full_context_packages(state_for(lock, root, core)) + identities = [qualification.package_identity(item) for item in contexts] + self.assertIn((extension_v1["name"], extension_v1["version"]), identities) + self.assertIn((extension_v2["name"], extension_v2["version"]), identities) + self.assertIn((neutral_leaf["name"], neutral_leaf["version"]), identities) + self.assertNotIn((r5_branch["name"], r5_branch["version"]), identities) + self.assertLess( + identities.index((extension_v1["name"], extension_v1["version"])), + identities.index((bridge["name"], bridge["version"])), + ) + self.assertLess( + identities.index((neutral_leaf["name"], neutral_leaf["version"])), + identities.index((bridge["name"], bridge["version"])), + ) + + def test_duplicate_dependency_identity_fails_closed(self) -> None: + root = package("example.root", "1.0.0", {"example.dep": "1.0.0"}) + first = package("example.dep", "1.0.0") + second = {**first, "sha256": "f" * 64} + lock = {"packages": [root, first, second]} + with self.assertRaisesRegex(ValueError, "matched 2 locked packages"): + qualification.select_dependency(lock, root, "example.dep", "1.0.0") + + def test_named_slice_before_local_slicing_is_preserved(self) -> None: + profile = qualification.ProfileResource( + "StructureDefinition-profile.json", + { + "url": "https://example.test/StructureDefinition/profile", + "version": "1.0.0", + "baseDefinition": "http://hl7.org/fhir/StructureDefinition/Observation", + "derivation": "constraint", + "snapshot": { + "element": [ + {"id": "Observation", "path": "Observation"}, + { + "id": "Observation.category", + "path": "Observation.category", + "slicing": {"rules": "open"}, + }, + { + "id": "Observation.category:lab", + "path": "Observation.category", + "sliceName": "lab", + }, + ] + }, + "differential": { + "element": [ + {"id": "Observation", "path": "Observation"}, + { + "id": "Observation.category:lab", + "path": "Observation.category", + "sliceName": "lab", + }, + { + "id": "Observation.category", + "path": "Observation.category", + "slicing": {"rules": "open"}, + }, + ] + }, + }, + ) + shape = qualification.differential_shape(profile, "Observation.category") + differential = shape["differential"] + snapshot = shape["snapshot"] + self.assertTrue(differential["named_slice_precedes_slicing"]) + self.assertFalse(differential["slicing_before_first_named_slice"]) + self.assertEqual(differential["first_named_slice_index"], 1) + self.assertEqual(differential["slicing_indices"], [2]) + self.assertTrue(snapshot["slicing_before_first_named_slice"]) + self.assertEqual(snapshot["slicing_indices"], [1]) + + def test_self_definition_exception_in_both_modes_classifies_pinned_limit(self) -> None: + probes = [] + failure = probe_result( + "exception", + "org.hl7.fhir.exceptions.DefinitionException", + "Found a slice at 'Observation.category' but there was no definition for the slicing", + ) + for label in ("self_before", "self_after", "cross"): + for mode in ("DIRECT", "FULL_CLOSURE"): + result = probe_result("completed") + if label in {"self_before", "cross"}: + result = failure + probes.append( + {"case_id": "C001", "probe": label, "context_mode": mode, "result": result} + ) + self.assertEqual( + qualification.case_classification("C001", probes, failure), + qualification.ROOT_CAUSE_PINNED, + ) + + def test_full_closure_completion_after_direct_failure_classifies_context(self) -> None: + probes = [] + failure = probe_result( + "exception", + "org.hl7.fhir.exceptions.DefinitionException", + "Found a slice at 'Observation.category' but there was no definition for the slicing", + ) + for label in ("self_before", "self_after", "cross"): + for mode in ("DIRECT", "FULL_CLOSURE"): + result = probe_result("completed") + if label == "cross" and mode == "DIRECT": + result = failure + probes.append( + {"case_id": "C001", "probe": label, "context_mode": mode, "result": result} + ) + self.assertEqual( + qualification.case_classification("C001", probes, failure), + qualification.ROOT_CAUSE_CONTEXT, + ) + + def test_exception_cannot_be_coerced_to_oracle_status(self) -> None: + payload = { + "schema": 1, + "oracle": qualification.ORACLE, + "status": "uncomparable", + } + completed = qualification.ProcessResult(0, qualification.canonical_json_bytes(payload), b"") + with tempfile.TemporaryDirectory() as directory: + with mock.patch.object(qualification, "run_bounded", return_value=completed), mock.patch( + "builtins.print" + ): + with self.assertRaisesRegex(ValueError, "coerced"): + qualification.run_probe( + Path(directory), + {"case_id": "C001"}, + ["java"], + ) + + def test_classpath_component_cannot_leak_into_evidence(self) -> None: + payload = { + "schema": 1, + "oracle": qualification.ORACLE, + "status": "exception", + "phase": "context_load", + } + with tempfile.TemporaryDirectory() as directory: + root = Path(directory) + classes = root.parent / "probe-classes" + process = qualification.ProcessResult( + 2, + qualification.canonical_json_bytes(payload), + str(classes).encode(), + ) + classpath = os.pathsep.join((str(classes), str(root.parent / "oracle.jar"))) + with mock.patch.object( + qualification, "run_bounded", return_value=process + ), mock.patch("builtins.print"): + with self.assertRaisesRegex(ValueError, "host-absolute path"): + qualification.run_probe( + root, + {"case_id": "C001"}, + ["java", "-cp", classpath], + ) + + def test_resolved_resource_identity_mismatch_fails_closed(self) -> None: + expected = { + "url": "https://example.test/StructureDefinition/expected", + "version": "1", + "id": "expected", + "type": "Observation", + } + actual = {**expected, "id": "different"} + payload = { + "schema": 1, + "oracle": qualification.ORACLE, + "status": "completed", + "phase": "comparison", + "left_resource": actual, + "right_resource": expected, + } + process = qualification.ProcessResult( + 0, qualification.canonical_json_bytes(payload), b"" + ) + invocation = { + "case_id": "C001", + "left": {"resource": expected}, + "right": {"resource": expected}, + } + with tempfile.TemporaryDirectory() as directory: + with mock.patch.object( + qualification, "run_bounded", return_value=process + ), mock.patch("builtins.print"): + with self.assertRaisesRegex(ValueError, "resolved left"): + qualification.run_probe(Path(directory), invocation, ["java"]) + + def test_unrelated_discovery_exception_cannot_classify_root_cause(self) -> None: + probes = [] + for label in ("self_before", "self_after", "cross"): + for mode in ("DIRECT", "FULL_CLOSURE"): + probes.append( + { + "case_id": "C001", + "probe": label, + "context_mode": mode, + "result": probe_result("completed"), + } + ) + unrelated = probe_result("exception", "example.ContextException", "missing") + self.assertEqual( + qualification.case_classification("C001", probes, unrelated), + qualification.ROOT_CAUSE_NOT_PROVEN, + ) + + def test_duplicate_canonical_profile_fails_closed(self) -> None: + root = package("example.root", "1.0.0") + core = package("hl7.fhir.r4.core", "4.0.1") + with tempfile.TemporaryDirectory() as directory: + state_root = Path(directory) + archive = state_root / "cache" / "sha256" / f"{root['sha256']}.tgz" + archive.parent.mkdir(parents=True) + resource = json.dumps( + { + "resourceType": "StructureDefinition", + "url": "https://example.test/StructureDefinition/profile", + } + ).encode() + with tarfile.open(archive, mode="w:gz") as package_file: + for name in ("package/first.json", "package/second.json"): + info = tarfile.TarInfo(name) + info.size = len(resource) + package_file.addfile(info, io.BytesIO(resource)) + spec = qualification.StateSpec( + "T001", + "before", + root["name"], + root["version"], + root["sha256"], + archive.stat().st_size, + "a" * 64, + ) + state = qualification.VerifiedState(spec, state_root, {"packages": [root]}, root, core) + with self.assertRaisesRegex(ValueError, "matched 2 profiles"): + qualification.find_profile( + state, "https://example.test/StructureDefinition/profile" + ) + + def test_output_and_bounding_are_deterministic(self) -> None: + value = {"b": 2, "a": [3, 1]} + self.assertEqual( + qualification.canonical_json_bytes(value), qualification.canonical_json_bytes(value) + ) + bounded = qualification.bounded_text("x" * (qualification.MAX_DIAGNOSTIC_CHARS + 10)) + self.assertTrue(bounded.endswith("... [diagnostic truncated]")) + self.assertLessEqual( + len(bounded), + qualification.MAX_DIAGNOSTIC_CHARS + len("... [diagnostic truncated]"), + ) + + +if __name__ == "__main__": + unittest.main() diff --git a/.github/workflows/cf10-oracle-qualification.yml b/.github/workflows/cf10-oracle-qualification.yml new file mode 100644 index 00000000..7c91ec2f --- /dev/null +++ b/.github/workflows/cf10-oracle-qualification.yml @@ -0,0 +1,174 @@ +name: cf10-oracle-qualification + +on: + pull_request: + paths: + - .github/scripts/cf10-oracle-qualification/** + - .github/workflows/cf10-oracle-qualification.yml + workflow_dispatch: + +permissions: + contents: read + +jobs: + qualify-pinned-hl7: + runs-on: ubuntu-latest + timeout-minutes: 90 + steps: + - uses: actions/checkout@fbc6f3992d24b796d5a048ff273f7fcc4a7b6c09 # v5 / Node 24 + with: + persist-credentials: false + + - uses: actions/setup-java@b6effb05e454b25005698d916606bdc6ffcbf961 # v5.7.0 + with: + distribution: temurin + java-version: '17' + + - uses: dtolnay/rust-toolchain@032958afbdc797a9164d3bc0b56325c1308924a5 # 1.97.1 + + - name: Test deterministic qualification harness + run: >- + python3 -m unittest discover + -s .github/scripts/cf10-oracle-qualification + -p 'test_*.py' + + - name: Build exact commandF and pinned HL7 adapter + run: | + cargo build --locked -p commandf + mvn -B -ntp -f tools/hl7-oracle/pom.xml package + mkdir -p /tmp/cf10-probe-classes + javac \ + -cp tools/hl7-oracle/target/commandf-hl7-oracle.jar \ + -d /tmp/cf10-probe-classes \ + .github/scripts/cf10-oracle-qualification/QualificationProbe.java + + - name: Acquire only C001 and C002 exact states A + run: | + test ! -e /tmp/cf10-oracle-qualification-a + python3 .github/scripts/cf10-oracle-qualification/qualify_cf10_oracle.py \ + acquire \ + --commandf ./target/debug/commandf \ + --work-root /tmp/cf10-oracle-qualification-a \ + --output /tmp/cf10-acquisition-a.json + + - name: Acquire only C001 and C002 exact states B + run: | + test ! -e /tmp/cf10-oracle-qualification-b + python3 .github/scripts/cf10-oracle-qualification/qualify_cf10_oracle.py \ + acquire \ + --commandf ./target/debug/commandf \ + --work-root /tmp/cf10-oracle-qualification-b \ + --output /tmp/cf10-acquisition-b.json + + - name: Prove independent acquisition identity + run: | + cmp /tmp/cf10-acquisition-a.json /tmp/cf10-acquisition-b.json + for case in C001 C002; do + for side in before after; do + cmp \ + "/tmp/cf10-oracle-qualification-a/states/$case/$side/commandf.lock" \ + "/tmp/cf10-oracle-qualification-b/states/$case/$side/commandf.lock" + done + cmp \ + "/tmp/cf10-oracle-qualification-a/cases/$case/structural.json" \ + "/tmp/cf10-oracle-qualification-b/cases/$case/structural.json" + done + + - name: Run pinned self/cross DIRECT and FULL_CLOSURE matrix A without network + run: | + sudo unshare --net -- \ + python3 .github/scripts/cf10-oracle-qualification/qualify_cf10_oracle.py \ + qualify \ + --java "$JAVA_HOME/bin/java" \ + --probe-classes /tmp/cf10-probe-classes \ + --oracle-jar tools/hl7-oracle/target/commandf-hl7-oracle.jar \ + --work-root /tmp/cf10-oracle-qualification-a \ + --output /tmp/cf10-qualification-a.json + + - name: Run pinned self/cross DIRECT and FULL_CLOSURE matrix B without network + run: | + sudo unshare --net -- \ + python3 .github/scripts/cf10-oracle-qualification/qualify_cf10_oracle.py \ + qualify \ + --java "$JAVA_HOME/bin/java" \ + --probe-classes /tmp/cf10-probe-classes \ + --oracle-jar tools/hl7-oracle/target/commandf-hl7-oracle.jar \ + --work-root /tmp/cf10-oracle-qualification-b \ + --output /tmp/cf10-qualification-b.json + + - name: Enforce deterministic complete evidence + run: | + cmp /tmp/cf10-qualification-a.json /tmp/cf10-qualification-b.json + python3 - <<'PY' + import json + from pathlib import Path + + report = json.loads(Path('/tmp/cf10-qualification-a.json').read_text()) + assert report['schema'] == 1 + assert report['oracle'] == { + 'project': 'hapifhir/org.hl7.fhir.core', + 'release': '6.10.2', + 'source_commit': 'd06577dbc5c62c74a2a8823fbc4830a3024d5b0b', + } + assert len(report['probes']) == 12 + assert len(report['discoveries']) == 2 + assert len(report['discovery_probes']) >= 2 + assert len(report['differential_shapes']) == 4 + allowed = { + 'CF10_ORACLE_ROOT_CAUSE_COMMAND_F_CONTEXT_DEFECT', + 'CF10_ORACLE_ROOT_CAUSE_PINNED_HL7_COMPARATOR_LIMITATION', + 'CF10_ORACLE_ROOT_CAUSE_MIXED', + 'CF10_ORACLE_ROOT_CAUSE_NOT_PROVEN', + } + assert report['root_cause_class'] in allowed + assert set(report['case_classifications']) == {'C001', 'C002'} + for discovery in report['discoveries']: + assert discovery['selected_canonical_url'] + assert discovery['observed_exception_path'] + for probe in report['discovery_probes'] + report['probes']: + assert probe['result']['status'] in {'completed', 'exception'} + print(report['root_cause_class']) + for discovery in report['discoveries']: + print( + discovery['case_id'], + discovery['selected_canonical_url'], + discovery['observed_exception_path'], + discovery['hypothesis_canonical_confirmed'], + discovery['hypothesis_path_confirmed'], + ) + for probe in report['probes']: + result = probe['result'] + print( + probe['case_id'], + probe['probe'], + probe['context_mode'], + result['status'], + result.get('exception_class'), + result.get('exception_message'), + ) + PY + + - name: Upload qualification evidence without package archives + if: always() + uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4 + with: + name: cf10-oracle-qualification-evidence + path: | + /tmp/cf10-acquisition-a.json + /tmp/cf10-acquisition-b.json + /tmp/cf10-qualification-a.json + /tmp/cf10-qualification-b.json + /tmp/cf10-oracle-qualification-a/states/C001/before/commandf.lock + /tmp/cf10-oracle-qualification-a/states/C001/after/commandf.lock + /tmp/cf10-oracle-qualification-a/states/C002/before/commandf.lock + /tmp/cf10-oracle-qualification-a/states/C002/after/commandf.lock + /tmp/cf10-oracle-qualification-a/cases/C001/structural.json + /tmp/cf10-oracle-qualification-a/cases/C002/structural.json + /tmp/cf10-oracle-qualification-b/states/C001/before/commandf.lock + /tmp/cf10-oracle-qualification-b/states/C001/after/commandf.lock + /tmp/cf10-oracle-qualification-b/states/C002/before/commandf.lock + /tmp/cf10-oracle-qualification-b/states/C002/after/commandf.lock + /tmp/cf10-oracle-qualification-b/cases/C001/structural.json + /tmp/cf10-oracle-qualification-b/cases/C002/structural.json + if-no-files-found: warn + retention-days: 7 From 5fe10d9859407272acf6649fc3e868d3eb2fbd12 Mon Sep 17 00:00:00 2001 From: "Abdulaziz M. Shehri" Date: Sun, 16 Aug 2026 02:59:25 +0300 Subject: [PATCH 82/82] fix(corpus): bind retained closure evidence --- .../test_verify_closure_binding.py | 186 +++++++++++++ .../verify_closure_binding.py | 246 ++++++++++++++++++ .github/workflows/cf10-real-corpus.yml | 35 ++- 3 files changed, 463 insertions(+), 4 deletions(-) create mode 100644 .github/scripts/cf10-closure-evidence/test_verify_closure_binding.py create mode 100644 .github/scripts/cf10-closure-evidence/verify_closure_binding.py diff --git a/.github/scripts/cf10-closure-evidence/test_verify_closure_binding.py b/.github/scripts/cf10-closure-evidence/test_verify_closure_binding.py new file mode 100644 index 00000000..fb65dfda --- /dev/null +++ b/.github/scripts/cf10-closure-evidence/test_verify_closure_binding.py @@ -0,0 +1,186 @@ +import importlib.util +import json +from pathlib import Path +import sys +import tempfile +import unittest + + +MODULE_PATH = Path(__file__).with_name("verify_closure_binding.py") +SPEC = importlib.util.spec_from_file_location("verify_closure_binding", MODULE_PATH) +assert SPEC is not None and SPEC.loader is not None +MODULE = importlib.util.module_from_spec(SPEC) +sys.modules[SPEC.name] = MODULE +SPEC.loader.exec_module(MODULE) + + +def package(name, version, sha, dependencies=None, source=None): + value = { + "name": name, + "version": version, + "sha256": sha, + "source": source or f"https://packages.example/{name}/{version}", + "dependencies": dependencies or {}, + } + return value + + +def normalized(packages): + values = [ + { + "name": p["name"], + "version": p["version"], + "sha256": p["sha256"], + "dependencies": { + key: p["dependencies"][key] for key in sorted(p["dependencies"]) + }, + } + for p in packages + ] + return sorted( + values, + key=lambda p: ( + p["name"], + p["version"], + p["sha256"], + tuple(p["dependencies"].items()), + ), + ) + + +def summary_for(before_packages, after_packages): + before = normalized(before_packages) + after = normalized(after_packages) + return { + "schema": 1, + "manifest_sha256": "f" * 64, + "cases": [ + { + "case_id": "C001", + "package": "root.pkg", + "before": { + "version": "1.0.0", + "sha256": "a" * 64, + "closure_sha256": MODULE.closure_sha256(before), + "closure": before, + }, + "after": { + "version": "2.0.0", + "sha256": "b" * 64, + "closure_sha256": MODULE.closure_sha256(after), + "closure": after, + }, + "status": "oracle_failed", + "structural": None, + "compatibility": None, + "terminology": None, + "oracle": None, + } + ], + } + + +class BindingTests(unittest.TestCase): + def setUp(self): + self.temp = tempfile.TemporaryDirectory() + self.root = Path(self.temp.name) + self.evidence = self.root / "evidence" / "C001" + self.evidence.mkdir(parents=True) + + self.before = [ + package( + "root.pkg", + "1.0.0", + "a" * 64, + {"same.dep": "1.0.0", "leaf.dep": "3.0.0"}, + ), + package("same.dep", "1.0.0", "c" * 64), + package("same.dep", "2.0.0", "d" * 64), + package("leaf.dep", "3.0.0", "e" * 64), + ] + self.after = [ + package("root.pkg", "2.0.0", "b" * 64, {"same.dep": "2.0.0"}), + package("same.dep", "2.0.0", "d" * 64), + ] + self.summary = summary_for(self.before, self.after) + self._write_all(self.before, self.after) + + def tearDown(self): + self.temp.cleanup() + + def _write_lock(self, side, root_version, packages): + value = { + "schema": 1, + "roots": [f"root.pkg@{root_version}"], + "packages": packages, + } + (self.evidence / f"{side}.commandf.lock").write_text( + json.dumps(value, indent=2) + "\n", encoding="utf-8" + ) + + def _write_summary(self, summary=None): + path = self.root / "summary.json" + path.write_text(json.dumps(summary or self.summary), encoding="utf-8") + return path + + def _write_all(self, before, after): + self._write_lock("before", "1.0.0", before) + self._write_lock("after", "2.0.0", after) + + def _verify(self, summary=None): + MODULE.verify_summary(self._write_summary(summary), self.root / "evidence") + + def test_exact_binding_with_same_name_multi_version_passes(self): + self._verify() + + def test_source_only_change_is_excluded_from_closure_contract(self): + changed = [dict(value) for value in self.before] + changed[1] = dict( + changed[1], source="https://mirror.example/same.dep/1.0.0" + ) + self._write_lock("before", "1.0.0", changed) + self._verify() + + def test_dependency_map_only_tampering_fails(self): + changed = [dict(value) for value in self.before] + changed[0] = dict( + changed[0], + dependencies={"same.dep": "2.0.0", "leaf.dep": "3.0.0"}, + ) + self._write_lock("before", "1.0.0", changed) + with self.assertRaisesRegex(MODULE.BindingError, "does not match summary closure"): + self._verify() + + def test_missing_package_partial_binding_fails(self): + self._write_lock("before", "1.0.0", self.before[:-1]) + with self.assertRaisesRegex(MODULE.BindingError, "does not match summary closure"): + self._verify() + + def test_digest_mismatch_fails_even_when_closure_matches(self): + changed_summary = json.loads(json.dumps(self.summary)) + changed_summary["cases"][0]["before"]["closure_sha256"] = "0" * 64 + with self.assertRaisesRegex(MODULE.BindingError, "closure digest mismatch"): + self._verify(changed_summary) + + def test_root_identity_mismatch_fails(self): + value = { + "schema": 1, + "roots": ["other.pkg@1.0.0"], + "packages": self.before, + } + (self.evidence / "before.commandf.lock").write_text( + json.dumps(value), encoding="utf-8" + ) + with self.assertRaisesRegex(MODULE.BindingError, "roots must contain exactly one"): + self._verify() + + def test_summary_closure_order_tampering_fails(self): + changed_summary = json.loads(json.dumps(self.summary)) + closure = changed_summary["cases"][0]["before"]["closure"] + closure[0], closure[1] = closure[1], closure[0] + with self.assertRaisesRegex(MODULE.BindingError, "not canonically sorted"): + self._verify(changed_summary) + + +if __name__ == "__main__": + unittest.main() diff --git a/.github/scripts/cf10-closure-evidence/verify_closure_binding.py b/.github/scripts/cf10-closure-evidence/verify_closure_binding.py new file mode 100644 index 00000000..3ac49a06 --- /dev/null +++ b/.github/scripts/cf10-closure-evidence/verify_closure_binding.py @@ -0,0 +1,246 @@ +#!/usr/bin/env python3 +"""Verify CF-10 durable lockfile evidence binds exactly to summary closure evidence.""" + +from __future__ import annotations + +import argparse +import hashlib +import json +import sys +from pathlib import Path +from typing import Any + +SCHEMA = 1 +SIDES = ("before", "after") + + +class BindingError(ValueError): + pass + + +def _require_dict(value: Any, label: str) -> dict[str, Any]: + if not isinstance(value, dict): + raise BindingError(f"{label} must be an object") + return value + + +def _require_list(value: Any, label: str) -> list[Any]: + if not isinstance(value, list): + raise BindingError(f"{label} must be an array") + return value + + +def _require_string(value: Any, label: str) -> str: + if not isinstance(value, str) or not value: + raise BindingError(f"{label} must be a non-empty string") + return value + + +def _require_sha256(value: Any, label: str) -> str: + text = _require_string(value, label) + if len(text) != 64 or any(char not in "0123456789abcdef" for char in text): + raise BindingError(f"{label} must be a lowercase SHA-256 hex digest") + return text + + +def _load_json(path: Path, label: str) -> dict[str, Any]: + try: + raw = path.read_bytes() + except OSError as error: + raise BindingError(f"{label} is unreadable: {path}: {error}") from error + try: + value = json.loads(raw) + except json.JSONDecodeError as error: + raise BindingError(f"{label} is invalid JSON: {path}: {error}") from error + return _require_dict(value, label) + + +def _normalize_dependencies(value: Any, label: str) -> dict[str, str]: + dependencies = _require_dict(value, label) + normalized: dict[str, str] = {} + for name in sorted(dependencies): + dep_name = _require_string(name, f"{label} key") + dep_version = _require_string(dependencies[name], f"{label}[{dep_name!r}]") + normalized[dep_name] = dep_version + return normalized + + +def _normalize_package(value: Any, label: str) -> dict[str, Any]: + package = _require_dict(value, label) + return { + "name": _require_string(package.get("name"), f"{label}.name"), + "version": _require_string(package.get("version"), f"{label}.version"), + "sha256": _require_sha256(package.get("sha256"), f"{label}.sha256"), + "dependencies": _normalize_dependencies( + package.get("dependencies"), f"{label}.dependencies" + ), + } + + +def _package_sort_key(package: dict[str, Any]) -> tuple[Any, ...]: + return ( + package["name"], + package["version"], + package["sha256"], + tuple(package["dependencies"].items()), + ) + + +def normalize_lock_closure(lockfile: dict[str, Any], label: str) -> list[dict[str, Any]]: + if lockfile.get("schema") != SCHEMA: + raise BindingError(f"{label}.schema must be {SCHEMA}") + packages = _require_list(lockfile.get("packages"), f"{label}.packages") + closure = [ + _normalize_package(value, f"{label}.packages[{index}]") + for index, value in enumerate(packages) + ] + closure.sort(key=_package_sort_key) + return closure + + +def closure_bytes(closure: list[dict[str, Any]]) -> bytes: + # Mirrors serde_json::to_vec(Vec): struct field insertion + # order is name, version, sha256, dependencies; BTreeMap dependency keys are sorted. + return json.dumps( + closure, + ensure_ascii=False, + separators=(",", ":"), + ).encode("utf-8") + + +def closure_sha256(closure: list[dict[str, Any]]) -> str: + return hashlib.sha256(closure_bytes(closure)).hexdigest() + + +def _assert_root_identity( + lockfile: dict[str, Any], + *, + package_name: str, + version: str, + sha256: str, + label: str, +) -> None: + roots = _require_list(lockfile.get("roots"), f"{label}.roots") + expected_root = f"{package_name}@{version}" + if roots.count(expected_root) != 1: + raise BindingError( + f"{label}.roots must contain exactly one {expected_root!r}" + ) + + closure = normalize_lock_closure(lockfile, label) + matches = [ + package + for package in closure + if package["name"] == package_name + and package["version"] == version + and package["sha256"] == sha256 + ] + if len(matches) != 1: + raise BindingError( + f"{label} must contain exactly one root package identity " + f"{package_name}@{version}#{sha256}" + ) + + +def verify_side( + *, + summary_case: dict[str, Any], + side: str, + lock_path: Path, +) -> None: + case_id = _require_string(summary_case.get("case_id"), "summary case_id") + package_name = _require_string( + summary_case.get("package"), f"{case_id}.package" + ) + summary_state = _require_dict( + summary_case.get(side), f"{case_id}.{side}" + ) + version = _require_string( + summary_state.get("version"), f"{case_id}.{side}.version" + ) + sha256 = _require_sha256( + summary_state.get("sha256"), f"{case_id}.{side}.sha256" + ) + expected_digest = _require_sha256( + summary_state.get("closure_sha256"), + f"{case_id}.{side}.closure_sha256", + ) + summary_closure_raw = _require_list( + summary_state.get("closure"), f"{case_id}.{side}.closure" + ) + summary_closure = [ + _normalize_package(value, f"{case_id}.{side}.closure[{index}]") + for index, value in enumerate(summary_closure_raw) + ] + if summary_closure != sorted(summary_closure, key=_package_sort_key): + raise BindingError(f"{case_id}.{side}.closure is not canonically sorted") + + lockfile = _load_json(lock_path, f"{case_id}.{side} retained lockfile") + _assert_root_identity( + lockfile, + package_name=package_name, + version=version, + sha256=sha256, + label=f"{case_id}.{side} retained lockfile", + ) + retained_closure = normalize_lock_closure( + lockfile, f"{case_id}.{side} retained lockfile" + ) + + if retained_closure != summary_closure: + raise BindingError( + f"{case_id}.{side} retained lockfile closure does not match summary closure" + ) + + retained_digest = closure_sha256(retained_closure) + if retained_digest != expected_digest: + raise BindingError( + f"{case_id}.{side} closure digest mismatch: " + f"retained={retained_digest} summary={expected_digest}" + ) + + +def verify_summary(summary_path: Path, evidence_root: Path) -> None: + summary = _load_json(summary_path, "summary") + if summary.get("schema") != SCHEMA: + raise BindingError(f"summary.schema must be {SCHEMA}") + cases = _require_list(summary.get("cases"), "summary.cases") + if not cases: + raise BindingError("summary.cases must not be empty") + + seen_case_ids: set[str] = set() + for index, raw_case in enumerate(cases): + summary_case = _require_dict(raw_case, f"summary.cases[{index}]") + case_id = _require_string( + summary_case.get("case_id"), f"summary.cases[{index}].case_id" + ) + if case_id in seen_case_ids: + raise BindingError(f"duplicate summary case_id {case_id!r}") + seen_case_ids.add(case_id) + + for side in SIDES: + verify_side( + summary_case=summary_case, + side=side, + lock_path=evidence_root / case_id / f"{side}.commandf.lock", + ) + + +def main(argv: list[str] | None = None) -> int: + parser = argparse.ArgumentParser() + parser.add_argument("--summary", type=Path, required=True) + parser.add_argument("--evidence-root", type=Path, required=True) + args = parser.parse_args(argv) + + try: + verify_summary(args.summary, args.evidence_root) + except BindingError as error: + print(f"CF10_CLOSURE_BINDING_FAILED: {error}", file=sys.stderr) + return 1 + + print("CF10_CLOSURE_BINDING_OK") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/.github/workflows/cf10-real-corpus.yml b/.github/workflows/cf10-real-corpus.yml index 8f4eb0a1..c7bb7b22 100644 --- a/.github/workflows/cf10-real-corpus.yml +++ b/.github/workflows/cf10-real-corpus.yml @@ -4,6 +4,7 @@ on: pull_request: paths: - .github/workflows/cf10-real-corpus.yml + - .github/scripts/cf10-closure-evidence/** - corpus/real-ig/v1/** - donors/cf-10-real-ig-delta-corpus.yaml - specs/010-cf-10-real-ig-delta-corpus/** @@ -17,6 +18,7 @@ on: - main paths: - .github/workflows/cf10-real-corpus.yml + - .github/scripts/cf10-closure-evidence/** - corpus/real-ig/v1/** - donors/cf-10-real-ig-delta-corpus.yaml - specs/010-cf-10-real-ig-delta-corpus/** @@ -46,6 +48,9 @@ jobs: - uses: dtolnay/rust-toolchain@032958afbdc797a9164d3bc0b56325c1308924a5 # 1.97.1 + - name: Test closure evidence verifier + run: python -B -m unittest discover -s .github/scripts/cf10-closure-evidence -p 'test_*.py' + - name: Build pinned HL7 oracle adapter run: mvn -B -ntp -f tools/hl7-oracle/pom.xml package @@ -127,6 +132,23 @@ jobs: PY printf '%s\n' "$parse_status" > /tmp/cf10-summary-parse-status + - name: Bind retained lockfiles to summary closure evidence + if: always() + run: | + set -u + closure_a=0 + closure_b=0 + python -B .github/scripts/cf10-closure-evidence/verify_closure_binding.py \ + --summary /tmp/cf10-summary-a.json \ + --evidence-root /tmp/cf10-real-a/evidence \ + || closure_a=$? + python -B .github/scripts/cf10-closure-evidence/verify_closure_binding.py \ + --summary /tmp/cf10-summary-b.json \ + --evidence-root /tmp/cf10-real-b/evidence \ + || closure_b=$? + printf '%s\n' "$closure_a" > /tmp/cf10-closure-binding-status-a + printf '%s\n' "$closure_b" > /tmp/cf10-closure-binding-status-b + - name: Assert metadata-only repository boundary if: always() run: | @@ -152,6 +174,8 @@ jobs: /tmp/cf10-internal-match-b /tmp/cf10-summary-comparison-status /tmp/cf10-summary-parse-status + /tmp/cf10-closure-binding-status-a + /tmp/cf10-closure-binding-status-b /tmp/cf10-real-a/summary.json /tmp/cf10-real-b/summary.json /tmp/cf10-real-a/evidence @@ -169,12 +193,15 @@ jobs: internal_b="$(cat /tmp/cf10-internal-match-b 2>/dev/null || printf '1')" comparison="$(cat /tmp/cf10-summary-comparison-status 2>/dev/null || printf '1')" parsed="$(cat /tmp/cf10-summary-parse-status 2>/dev/null || printf '1')" + closure_a="$(cat /tmp/cf10-closure-binding-status-a 2>/dev/null || printf '1')" + closure_b="$(cat /tmp/cf10-closure-binding-status-b 2>/dev/null || printf '1')" - printf 'runner_a=%s runner_b=%s internal_a=%s internal_b=%s comparison=%s parsed=%s\n' \ - "$status_a" "$status_b" "$internal_a" "$internal_b" "$comparison" "$parsed" + printf 'runner_a=%s runner_b=%s internal_a=%s internal_b=%s comparison=%s parsed=%s closure_a=%s closure_b=%s\n' \ + "$status_a" "$status_b" "$internal_a" "$internal_b" "$comparison" "$parsed" "$closure_a" "$closure_b" if [[ "$status_a" != 0 || "$status_b" != 0 || \ "$internal_a" != 0 || "$internal_b" != 0 || \ - "$comparison" != 0 || "$parsed" != 0 ]]; then + "$comparison" != 0 || "$parsed" != 0 || \ + "$closure_a" != 0 || "$closure_b" != 0 ]]; then exit 1 - fi \ No newline at end of file + fi