From c46e1cb2d2f5126ce10d241dd1ad178df3e3b607 Mon Sep 17 00:00:00 2001 From: Nick Ruest <127058086+nicholas-ruest@users.noreply.github.com> Date: Mon, 27 Jul 2026 04:59:49 +0000 Subject: [PATCH 1/3] Scope README compliance claims to implemented controls Removes the SOC 2 Type II, HIPAA, and GDPR assertions from README.md and replaces them with design-stage language, since no audit artifact exists and the compliance service has never run against a real schema. Deletes "BAA management" outright: a Business Associate Agreement is a signed legal contract, not a software feature. Retains the encryption claim, which is genuinely implemented, with a pointer to the Terraform resources. Marks soc2-evidence-collection.yaml as a design specification and flags its 10 unresolvable evidence paths with TODO. Adds two CI checks so this cannot regress: check_compliance_claims.py fails on a compliance claim not backed by a resolvable link or a design-stage qualifier, and check_evidence_paths.py fails on an unmarked evidence path that does not exist. Implements copilot-agent/docs/adr/ADR-0001-compliance-claims-remediation.md --- README.md | 41 +-- .../compliance/soc2-evidence-collection.yaml | 23 +- .../ADR-0001-compliance-claims-remediation.md | 315 ++++++++++++++++++ docs/compliance-claims-baseline.txt | 23 ++ scripts/check_compliance_claims.py | 134 ++++++++ scripts/check_evidence_paths.py | 42 +++ 6 files changed, 549 insertions(+), 29 deletions(-) create mode 100644 docs/adr/ADR-0001-compliance-claims-remediation.md create mode 100644 docs/compliance-claims-baseline.txt create mode 100755 scripts/check_compliance_claims.py create mode 100755 scripts/check_evidence_paths.py diff --git a/README.md b/README.md index bef3578..d4a9a89 100644 --- a/README.md +++ b/README.md @@ -30,10 +30,10 @@ LLM-CoPilot-Agent is a comprehensive AI platform that serves as both a conversat - **Tool/Function Calling** - Extensible tool framework with validation and execution - **RAG Pipeline** - Document ingestion, chunking strategies, vector search, and generation -### Compliance & Governance (Phase 5) -- **SOC 2 Type II** - Control management, audits, findings, evidence collection -- **HIPAA Compliance** - PHI access logging, BAA management, breach reporting -- **Data Residency** - Policy enforcement, regional data controls, transfer workflows +### Compliance & Governance (Design Stage) +- **SOC 2 control mapping and evidence-collection design (no audit performed)** - see [`deployment/compliance/soc2-evidence-collection.yaml`](deployment/compliance/soc2-evidence-collection.yaml) +- **HIPAA-oriented control catalog covering §164.308/310/312 (design stage, not operational)** +- **Data residency policy model (design stage)** - **Content Filtering** - Safety filters, PII detection, moderation integration - **Usage Policies** - Policy evaluation, enforcement modes, violation tracking - **Audit Trail** - Comprehensive event logging, anomaly detection, retention @@ -205,22 +205,24 @@ Enterprise AI/ML platform providing model management, agent orchestration, and R ### Compliance Service (Port 3009) -Enterprise compliance management for SOC 2, HIPAA, and data residency requirements. +Design-stage compliance service for SOC 2 control mapping, HIPAA safeguard cataloguing, and data residency policy modelling. Not deployed; see [`docs/adr/ADR-0001-compliance-claims-remediation.md`](docs/adr/ADR-0001-compliance-claims-remediation.md). -| Feature | Description | -|---------|-------------| -| **Control Management** | SOC 2 controls with status tracking and evidence | -| **Audit Management** | Schedule audits, track findings, generate reports | -| **HIPAA Compliance** | PHI access logging, BAA management, breach reporting | -| **Data Residency** | Regional policies, asset tracking, transfer workflows | -| **Compliance Reports** | Gap analysis, risk assessment, audit readiness | +| Feature | Description | Status | +|---------|-------------|--------| +| **Control Management** | SOC 2 controls with status tracking and evidence | Design | +| **Audit Management** | Schedule audits, track findings, generate reports | Design | +| **HIPAA Compliance** | PHI access logging, breach reporting | Design | +| **Data Residency** | Regional policies, asset tracking, transfer workflows | Design | +| **Compliance Reports** | Gap analysis, risk assessment, audit readiness | Design | + +> **Note:** The compliance service is not currently deployed in any environment. It appears in no `docker-compose.yml`, Kubernetes, or Helm manifest, and the database tables it queries are not yet created by any migration. **Endpoints:** - `GET/POST /api/v1/compliance/controls` - Control management - `GET/POST /api/v1/compliance/audits` - Audit management - `GET/POST /api/v1/compliance/findings` - Finding tracking - `POST /api/v1/hipaa/phi-access` - PHI access logging -- `GET/POST /api/v1/hipaa/baa` - BAA management +- `GET/POST /api/v1/hipaa/baa` - BAA registry records, design stage (tracking only; does not execute agreements) - `GET/POST /api/v1/data-residency/policies` - Residency policies - `POST /api/v1/data-residency/transfers` - Data transfers @@ -467,17 +469,18 @@ CORS_ORIGIN=http://localhost:3000,https://your-domain.com | Phase 2 | Complete | Multi-tenancy & Auth | | Phase 3 | Complete | Advanced Features | | Phase 4 | Complete | Enterprise Operations | -| Phase 5 | Complete | AI/ML Platform & Compliance | +| Phase 5 | Complete | AI/ML Platform | +| Phase 5 | Design Complete, Implementation In Progress | Compliance | | Phase 6 | Planned | Scale & Advanced Analytics | -## Security & Compliance +## Security -- **SOC 2 Type II** - Comprehensive control framework -- **HIPAA** - PHI protection and access logging -- **GDPR** - Data residency and privacy controls - **Content Safety** - Multi-layer content filtering - **Audit Trail** - Complete activity logging -- **Encryption** - At-rest and in-transit encryption +- **Encryption** - At-rest and in-transit encryption (see `deployment/terraform/main.tf:243,298-299,340-344`) + +For the project's compliance posture — including why SOC 2 Type II, HIPAA, and GDPR are +not claimed here — see [`docs/adr/ADR-0001-compliance-claims-remediation.md`](docs/adr/ADR-0001-compliance-claims-remediation.md). ## Contributing diff --git a/deployment/compliance/soc2-evidence-collection.yaml b/deployment/compliance/soc2-evidence-collection.yaml index 712df9a..43c8297 100644 --- a/deployment/compliance/soc2-evidence-collection.yaml +++ b/deployment/compliance/soc2-evidence-collection.yaml @@ -1,5 +1,8 @@ # SOC 2 Type II Evidence Collection Procedures # Automated evidence gathering for compliance audit +# +# Status: Design Specification -- no audit performed. +# See docs/adr/ADR-0001-compliance-claims-remediation.md evidenceCollection: @@ -27,7 +30,7 @@ evidenceCollection: manual: - type: "Authentication Policy Document" - path: "docs/policies/authentication-policy.md" + path: "docs/policies/authentication-policy.md" # TODO: artifact does not yet exist reviewFrequency: "Quarterly" - id: "CC6.1.2" @@ -40,7 +43,7 @@ evidenceCollection: - type: "Session Timeout Settings" source: "Application code" - path: "services/auth/session.ts" + path: "services/auth/session.ts" # TODO: artifact does not yet exist # CC6.2 - New User Provisioning CC6_2: @@ -50,10 +53,10 @@ evidenceCollection: evidence: manual: - type: "Provisioning Procedure" - path: "docs/procedures/user-provisioning.md" + path: "docs/procedures/user-provisioning.md" # TODO: artifact does not yet exist - type: "Background Check Records" - path: "hr/background-checks/" + path: "hr/background-checks/" # TODO: artifact does not yet exist storage: "Secure HR system" automated: @@ -71,7 +74,7 @@ evidenceCollection: automated: - type: "Password Policy Configuration" source: "Application code" - path: "services/auth/password-policy.ts" + path: "services/auth/password-policy.ts" # TODO: artifact does not yet exist - type: "Password Change Audit" source: "Audit logs" @@ -89,7 +92,7 @@ evidenceCollection: manual: - type: "Access Review Approval" - path: "compliance/access-reviews/" + path: "compliance/access-reviews/" # TODO: artifact does not yet exist format: "PDF with manager signatures" # CC6.6 - Encryption @@ -169,7 +172,7 @@ evidenceCollection: - type: "Backup Storage Verification" source: "AWS S3" - path: "s3://llm-copilot-backups/" + path: "s3://llm-copilot-backups/" # TODO: artifact does not yet exist schedule: "Daily" - id: "CC7.3.2" @@ -177,7 +180,7 @@ evidenceCollection: evidence: manual: - type: "DR Test Results" - path: "compliance/dr-tests/" + path: "compliance/dr-tests/" # TODO: artifact does not yet exist schedule: "Quarterly" requiredSignoff: "CTO" @@ -213,7 +216,7 @@ evidenceCollection: - type: "Test Coverage Reports" source: "Jest/Coverage" - path: "coverage/" + path: "coverage/" # TODO: artifact does not yet exist schedule: "Per commit" # CC9.2 - Business Continuity @@ -236,7 +239,7 @@ evidenceCollection: evidence: manual: - type: "Failover Test Report" - path: "compliance/failover-tests/" + path: "compliance/failover-tests/" # TODO: artifact does not yet exist schedule: "Quarterly" # Automated Evidence Collection Scripts diff --git a/docs/adr/ADR-0001-compliance-claims-remediation.md b/docs/adr/ADR-0001-compliance-claims-remediation.md new file mode 100644 index 0000000..70cdc0e --- /dev/null +++ b/docs/adr/ADR-0001-compliance-claims-remediation.md @@ -0,0 +1,315 @@ +# ADR-0001: Scope Compliance Claims to Implemented Controls + +**Status:** Proposed +**Date:** 2026-07-27 + +## Context + +`README.md` presents SOC 2 Type II, HIPAA, and data residency as delivered, completed +capabilities. `README.md:470` marks Phase 5 ("AI/ML Platform & Compliance") as +**Complete**, and the "Security & Compliance" section states them as properties of the +system: + +| Location | Claim | +|---|---| +| `README.md:34` | "**SOC 2 Type II** - Control management, audits, findings, evidence collection" | +| `README.md:35` | "**HIPAA Compliance** - PHI access logging, BAA management, breach reporting" | +| `README.md:36` | "**Data Residency** - Policy enforcement, regional data controls, transfer workflows" | +| `README.md:208` | "Enterprise compliance management for SOC 2, HIPAA, and data residency requirements." | +| `README.md:214` | "**HIPAA Compliance** \| PHI access logging, BAA management, breach reporting" | +| `README.md:470` | "\| Phase 5 \| Complete \| AI/ML Platform & Compliance \|" | +| `README.md:475-477` | "**SOC 2 Type II** - Comprehensive control framework" / "**HIPAA** - PHI protection and access logging" / "**GDPR** - Data residency and privacy controls" | +| `README.md:480` | "**Encryption** - At-rest and in-transit encryption" | + +These are not pure marketing prose — `services/compliance/` contains 4,115 lines of real +TypeScript (`hipaaService.ts` 815, `complianceService.ts` 930, `dataResidencyService.ts` +831, plus routes and models) using parameterized SQL against Postgres and Redis. The +HIPAA control catalog in `hipaaService.ts:401-583` correctly enumerates 45 CFR §164.308, +§164.310, and §164.312 safeguards with accurate citations. This is competent work. + +The problem is that the code does not run, is not reachable, and is not evidence. + +### Finding 1 — Every table the HIPAA service writes to is undefined + +The only migration is `migrations/V001_initial_schema.sql`, which creates `users`, +`sessions`, `conversations`, `messages`, `workflows`, `workflow_executions`, `incidents`, +`runbooks`, and `audit_logs` (+ partitions). It does **not** create `phi_access_logs` +(written at `hipaaService.ts:84`), `business_associate_agreements` (`:257`), +`hipaa_breaches` (`:712`), `compliance_controls` (read at `:614`), `security_alerts` +(`:767`), or `scheduled_tasks` (`:383`). No `.sql` file anywhere in the repo defines them. +Every PHI logging call fails at runtime with `relation "phi_access_logs" does not exist`. +PHI access logging is currently a 100% failure path. + +### Finding 2 — Nothing calls the PHI logger + +`logPHIAccess` is invoked from exactly one place: its own HTTP handler at +`routes/hipaa.ts:22`. No other service, middleware, or data-access path in the repo calls +it. PHI access logging is therefore opt-in self-reporting by whoever chooses to POST. +§164.312(b) Audit Controls requires the system to *record* access to ePHI; a system that +depends on callers voluntarily reporting their own access does not implement that control. + +### Finding 3 — The compliance API has no authentication or authorization + +`index.ts:183-185` mounts all three routers behind only `executionContextMiddleware`, +which is span/trace plumbing, not auth. Consequences: + +- `routes/hipaa.ts:22` passes `req.body` unvalidated into `logPHIAccess`, so any caller + can forge the `userId` attributed in the audit record — the audit trail is repudiable. +- `GET /phi-access` (`routes/hipaa.ts:32`) lets any party who can reach port 3009 read the + entire PHI access log, including `patientId`, `ipAddress`, and `purpose`. + +This contradicts §164.312(a)(1) Access Control and §164.312(d) Person or Entity +Authentication — two controls the repo's own catalog marks `required: true` +(`hipaaService.ts:511-517`, `:558-565`). + +### Finding 4 — `pseudonymize()` is non-deterministic, breaking patient-level accounting + +`hipaaService.ts:748-754` generates a fresh random IV on every call: + +```ts +private pseudonymize(value: string): string { + const iv = crypto.randomBytes(16); + const cipher = crypto.createCipheriv('aes-256-cbc', this.encryptionKey, iv); + ... +} +``` + +The same `patientId` therefore produces different ciphertext each time. The equality +filter at `hipaaService.ts:122-123` (`AND patient_id = $n` on a freshly-pseudonymized +value) can never match a stored row, and the `uniquePatients` count at `:172` counts one +distinct patient per access event. Accounting of disclosures under §164.528 is +structurally impossible with this function. A keyed HMAC is required, not encryption. + +### Finding 5 — Hardcoded default key material + +`hipaaService.ts:51-52` falls back to the literal string +`'default-key-change-in-production'` with a hardcoded salt `'salt'` when +`HIPAA_ENCRYPTION_KEY` is unset. There is no production fail-closed check, so a +misconfigured deploy silently pseudonymizes PHI identifiers under a publicly known key. + +### Finding 6 — The service is neither deployed nor tested + +`compliance` appears nowhere in `docker-compose.yml`, `deploy/`, +`deployment/kubernetes/`, or `deployment/helm/`. No `*.test.ts` or `*.spec.ts` matches +`compliance|hipaa|phi|residency`. The service is unshipped and unverified. + +### Finding 7 — The SOC 2 evidence manifest points mostly at nothing + +`deployment/compliance/soc2-evidence-collection.yaml` is a well-structured plan mapping +CC6.x controls to evidence sources, but 10 of its 14 referenced paths do not exist: +`docs/policies/authentication-policy.md`, `docs/procedures/user-provisioning.md`, +`services/auth/session.ts`, `services/auth/password-policy.ts`, `hr/background-checks/`, +`compliance/access-reviews/`, `compliance/dr-tests/`, `compliance/failover-tests/`, +`coverage/`, and `s3://llm-copilot-backups/`. The four that resolve are Kubernetes and +Terraform config files. + +### Finding 8 — No audit artifact exists, and none could yet + +There is no SOC 2 Type II report, auditor name, observation window, or bridge letter in +the repo. Type II is definitionally an *independent CPA firm's opinion on control +operating effectiveness across an observation period* (typically 3-12 months). No +quantity of code in this repository can produce it. Likewise, a Business Associate +Agreement is a signed contract between a covered entity and a business associate under +§164.502(e); `business_associate_agreements` is a registry schema for tracking such +contracts, not a mechanism that creates them. "BAA management" in `README.md:35` reads to +a healthcare buyer as "Agentics will sign a BAA" — a legal commitment, not a feature. + +### Finding 9 — The security architecture docs already say they are designs + +`security-architecture/03-data-protection-architecture.md:4` and +`security-architecture/05-audit-compliance-architecture.md:4` both carry +`**Status:** Design Specification`. The authors labelled these accurately. The README +promotes them to shipped compliance; the gap is in the README, not in those documents. + +### Finding 10 — One claim is genuinely substantiated + +Encryption at rest and in transit (`README.md:480`) is real and configured: +`deployment/terraform/main.tf:243` (`storage_encrypted = true` on RDS), `:298-299` +(ElastiCache `at_rest_encryption_enabled` and `transit_encryption_enabled`), `:340-344` +(S3 SSE-KMS), `:356` (customer-managed KMS key), and +`deployment/terraform/modules/eks/main.tf:64` (encrypted EBS volumes). This claim stands +and should not be weakened. + +### Summary + +This is not a documentation-only problem, so "just document the evidence" (option b) is +not available: the controls are absent, not undocumented. Nor is it vaporware — the +design work is substantially done and the control catalog is accurate. The accurate +description of today's state is **a designed compliance subsystem that has never been +wired up, deployed, or audited**, plus **genuinely implemented infrastructure +encryption**. + +## Decision + +**Scope the README claims down to what is true today, and gate every future compliance +claim on a linked evidence artifact.** + +Specifically: + +1. **SOC 2 Type II → removed as a claim.** Replaced with a roadmap entry. The repo may + describe *"a SOC 2 control-mapping framework and evidence-collection design"* — that is + true and is what `soc2-evidence-collection.yaml` is. It may not assert SOC 2 Type II + until an independent auditor issues a report. +2. **HIPAA Compliance → "HIPAA-oriented architecture (design stage, not operational)."** + The control catalog is accurate and worth citing; the enforcement is not present. +3. **BAA management → removed from the feature list entirely.** It is a legal commitment, + not a software feature, and Agentics is not currently positioned to sign one. It may + later return as "BAA lifecycle tracking" — a registry — clearly distinct from executing + agreements. +4. **Data Residency / GDPR → "design specification."** Same posture as HIPAA; the service + is undeployed and untested. +5. **Encryption at rest and in transit → retained as-is**, with a pointer to the Terraform + resources that implement it. +6. **Phase 5 status → "Compliance: Design Complete, Implementation In Progress."** + Splitting the row is more honest than marking the whole phase Complete. + +We choose narrowing over deletion because the underlying engineering is sound and +recoverable, and over "document what exists" because Findings 1-6 show the controls do not +operate. We treat this as urgent rather than cosmetic: representing unaudited software as +SOC 2 Type II certified to a customer is a misrepresentation with contractual and +regulatory exposure independent of the code quality, and HIPAA claims specifically invite +healthcare customers to route PHI into a system whose PHI logging currently throws on +every call. + +## Consequences + +**Positive** + +- The README stops asserting an audit opinion that does not exist, removing the primary + contractual and misrepresentation exposure. +- Healthcare prospects are not induced to send PHI into a system with no operational PHI + audit trail. +- Findings 1-5 are concrete, fixable engineering defects; the ADR converts a vague + "compliance risk" into a tracked backlog. +- The evidence-gate (below) prevents recurrence rather than fixing this one instance. + +**Negative** + +- The product narrows in sales positioning. Deals gated on SOC 2 Type II or a signed BAA + become explicitly out of scope until the work lands. This cost is already real and + merely unrecognized; the claims do not survive a customer security review or auditor + walkthrough today. +- Anyone who relied on the current README — internally or externally — may need to be + re-briefed. If these claims have already been made to a customer, correcting them is a + disclosure conversation, not just a docs edit. + +**Neutral** + +- The `services/compliance/` code is retained unchanged. This ADR reclassifies it as + in-progress; it does not deprecate it. + +## Implementation Plan + +Phase 1 is documentation-only and should ship immediately, independent of Phases 2-4. + +**Phase 1 — Correct the claims (immediate)** + +1. `README.md:33-36`: retitle the section "Compliance & Governance (Phase 5)" to + "Compliance & Governance (Design Stage)". Reword `:34` to "SOC 2 control mapping and + evidence-collection design (no audit performed)"; `:35` to "HIPAA-oriented control + catalog covering §164.308/310/312 (design stage, not operational)"; `:36` to "Data + residency policy model (design stage)". Delete "BAA management" from `:35`. +2. `README.md:208`: reword to "Design-stage compliance service for SOC 2 control mapping, + HIPAA safeguard cataloguing, and data residency policy modelling. Not deployed; see + `docs/adr/ADR-0001-compliance-claims-remediation.md`." +3. `README.md:212-216`: add a `Status` column to the feature table; mark every row + `Design` except where a linked artifact exists. Delete "BAA management" from `:214`. +4. `README.md:219-223`: add a note above the endpoint list stating the compliance service + is not currently deployed in any environment. +5. `README.md:470`: split the Phase 5 row into "AI/ML Platform — Complete" and + "Compliance — Design Complete, Implementation In Progress". +6. `README.md:473-480`: retitle "Security & Compliance" to "Security". Delete the SOC 2 + Type II (`:475`), HIPAA (`:476`), and GDPR (`:477`) bullets. Retain Encryption (`:480`) + and append "(see `deployment/terraform/main.tf:243,298-299,340-344`)". Add a line + pointing to this ADR for compliance posture. +7. Add `**Status:** Design Specification — no audit performed` to the top of + `deployment/compliance/soc2-evidence-collection.yaml`, and mark the 10 unresolved + `path:` entries from Finding 7 with `# TODO: artifact does not yet exist`. + +**Phase 2 — Make the HIPAA service actually function** + +8. Write `migrations/V002_compliance_schema.sql` creating `phi_access_logs`, + `business_associate_agreements`, `hipaa_breaches`, `compliance_controls`, + `security_alerts`, and `scheduled_tasks`, matching the column lists in + `hipaaService.ts:84-93`, `:257-267`, and `:712-721`. Index `phi_access_logs` on + `(patient_id)`, `(user_id)`, and `(timestamp)`. +9. Replace `pseudonymize()` (`hipaaService.ts:748-754`) with a deterministic keyed HMAC + (`crypto.createHmac('sha256', key)`), so equality lookups at `:122-123` work and + `uniquePatients` at `:172` is correct. Add a unit test asserting the same input yields + the same output across calls. +10. Remove the `'default-key-change-in-production'` fallback at `hipaaService.ts:51` and + the hardcoded `'salt'` at `:52`. Throw on startup when `HIPAA_ENCRYPTION_KEY` is unset + and `NODE_ENV === 'production'`; source the salt from config. +11. Add authentication and role-based authorization middleware to all three routers at + `index.ts:183-185`. Derive `userId` in `routes/hipaa.ts:22` from the authenticated + principal, never from `req.body`. Restrict `GET /phi-access` to a compliance-auditor + role. +12. Add schema validation (the repo already uses Zod elsewhere) on every compliance route + body and query. + +**Phase 3 — Make PHI logging an enforced control, not an opt-in call** + +13. Identify every code path that reads or writes ePHI-classified fields and route it + through a data-access layer that emits a PHI access record as a side effect of the + read, so logging cannot be skipped by omission. +14. Make `phi_access_logs` append-only: revoke `UPDATE`/`DELETE` from the application role + and add a hash-chain or equivalent tamper-evidence mechanism, per the design already + written in `security-architecture/05-audit-compliance-architecture.md`. +15. Add integration tests proving (a) a PHI read with no accompanying log entry fails the + test suite, (b) an unauthenticated request to any `/api/v1/hipaa/*` route is rejected, + (c) patient-scoped lookup returns that patient's records. +16. Add the compliance service to `docker-compose.yml` and the Kubernetes and Helm + manifests, with the encryption key sourced from a secret store. + +**Phase 4 — Earn the stronger claims** + +17. Author the missing policy and procedure artifacts enumerated in Finding 7. +18. Implement automated evidence collection against the CC6.x mappings in + `soc2-evidence-collection.yaml` and land the artifacts in a real `compliance/` + directory. +19. Engage an independent CPA firm. Complete a Type I readiness assessment, then a Type II + observation window (3-12 months). Only on receipt of the Type II report may + `README.md` state SOC 2 Type II, and it must then cite the auditor, report date, and + observation window. +20. For HIPAA: obtain a third-party §164.308(a)(1)(ii)(A) risk analysis, and have counsel + produce a BAA template plus an execution process before any BAA language returns to + the README. + +**Controls that must exist before the stronger claims are accurate** + +*Before "HIPAA-compliant":* enforced PHI access logging (13), tamper-evident append-only +audit storage (14), authentication and authorization on all PHI paths (11), deterministic +patient-scoped accounting of disclosures (9), managed key material with rotation (10), +a documented breach notification runbook exercised at least once, a third-party risk +analysis, and an executed BAA process. + +*Before "SOC 2 Type II":* all of the above, plus operating evidence for each CC6.x control +across the full observation window, documented access reviews, DR and failover test +records, and an unqualified independent auditor opinion. + +## Verification + +1. **No unbacked compliance claim ships.** Every compliance assertion in `README.md` must + link to either (a) a file path in this repo implementing the control, or (b) a dated + third-party audit artifact. A claim with neither is a release blocker. +2. **Automated check.** Add a CI job that greps `README.md` and `docs/` for + `SOC 2|SOC2|HIPAA compliant|BAA|GDPR compliant|certified` and fails unless each hit is + within 3 lines of a resolvable link or an explicit `(design stage)` qualifier. This + catches regressions mechanically rather than relying on review vigilance. +3. **Evidence-path integrity.** Add a CI job that resolves every `path:` entry in + `deployment/compliance/soc2-evidence-collection.yaml` and fails on any path that does + not exist and is not explicitly marked `# TODO`. Finding 7 would have been caught at + authoring time by this check. +4. **Phase 2 acceptance.** `migrations/V002_compliance_schema.sql` applies cleanly and an + integration test performs a full write-then-read of `POST /api/v1/hipaa/phi-access` + followed by `GET /api/v1/hipaa/phi-access?patientId=...` returning the same record — + proving Findings 1 and 4 are closed. +5. **Phase 3 acceptance.** A deliberately introduced ePHI read that bypasses the logging + layer causes a test failure. An unauthenticated request to any `/api/v1/hipaa/*` route + returns 401. Attempting `UPDATE` or `DELETE` on `phi_access_logs` as the application + role is rejected by the database. +6. **Claim restoration is gated on artifacts, not on code completion.** SOC 2 Type II + language returns to `README.md` only in a PR that also adds the auditor's report + reference. HIPAA and BAA language returns only in a PR that also adds the third-party + risk analysis and counsel-approved BAA template. Reviewers reject such PRs on a missing + artifact regardless of implementation state. diff --git a/docs/compliance-claims-baseline.txt b/docs/compliance-claims-baseline.txt new file mode 100644 index 0000000..a1914c3 --- /dev/null +++ b/docs/compliance-claims-baseline.txt @@ -0,0 +1,23 @@ +# Pre-existing unbacked compliance claims, recorded by ADR-0001. +# +# These predate the ADR and live in documents its Phase 1 did not scope. They are +# recorded here so scripts/check_compliance_claims.py can block NEW unbacked claims +# without failing CI on the existing backlog. Entries are matched on file + exact +# text, so editing a line surfaces it again as new. +# +# This file is a backlog, not an exemption list. Do NOT add entries by hand to +# silence the check -- fix the claim instead. Draining it is ADR-0001 Phase 4 work. +README.md::"framework": "soc2_type2", +README.md::│ │ │ - Models │ │ - SOC 2 │ │ - Filters │ │ │ +docs/README.md::- Compliance (GDPR, SOC 2) +docs/README.md::- [SOC 2 Compliance](https://www.aicpa.org/soc) +docs/SPECIFICATION.md::- Ensure data privacy and compliance (SOC2, GDPR, HIPAA) +docs/architecture/DATA_STORAGE_DELIVERABLES.md::- SOC 2 Type II certification +docs/architecture/DATA_STORAGE_DELIVERABLES.md::Compliance: GDPR, SOC 2, HIPAA compatible +docs/architecture/DELIVERABLES.md::- Compliance certification (SOC 2) +docs/architecture/DELIVERABLES.md::- SOC 2 Type II: Certified +docs/architecture/DELIVERABLES.md::- [SOC 2 Compliance Guide](https://www.aicpa.org/soc) +docs/specification-summary.md::- SOC 2 Type II: Certified +docs/specification-summary.md::16. Compliance certification (SOC 2) +docs/specification.md::- SOC 2 Type II compliance +docs/specification.md::soc2: Type II certified diff --git a/scripts/check_compliance_claims.py b/scripts/check_compliance_claims.py new file mode 100755 index 0000000..560040d --- /dev/null +++ b/scripts/check_compliance_claims.py @@ -0,0 +1,134 @@ +#!/usr/bin/env python3 +"""ADR-0001 Verification 2: no unbacked compliance claim ships. + +Greps README.md and docs/ for compliance claim keywords and fails unless each hit +sits within 3 lines of a resolvable repo path or an explicit "design stage" qualifier. + +docs/adr/ is exempt: an ADR is the record of a decision about these claims (and +quotes the offending text as evidence), not an assertion of them. +""" +import os +import re +import subprocess +import sys + +REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) +BASELINE = os.path.join(REPO, "docs", "compliance-claims-baseline.txt") +CLAIM = re.compile(r"SOC 2|SOC2|HIPAA compliant|BAA|GDPR compliant|certified", re.IGNORECASE) +QUALIFIER = re.compile( + r"design stage|design specification|no audit performed|\|\s*Design\b", re.IGNORECASE +) +LINK = re.compile(r"\[[^\]]*\]\(([^)]+)\)|`([^`]+)`") +# Bare paths in directory-tree listings, e.g. "├── hipaaService.ts # PHI logging" +BARE = re.compile(r"[\w./-]+\.(?:ts|tf|sql|ya?ml|md|json)\b") +CONTEXT = 3 + +BASELINE_HEADER = """\ +# Pre-existing unbacked compliance claims, recorded by ADR-0001. +# +# These predate the ADR and live in documents its Phase 1 did not scope. They are +# recorded here so scripts/check_compliance_claims.py can block NEW unbacked claims +# without failing CI on the existing backlog. Entries are matched on file + exact +# text, so editing a line surfaces it again as new. +# +# This file is a backlog, not an exemption list. Do NOT add entries by hand to +# silence the check -- fix the claim instead. Draining it is ADR-0001 Phase 4 work. +""" + + +def tracked_basenames(): + """Basenames of every git-tracked file, for resolving directory-tree listings.""" + if not hasattr(tracked_basenames, "cache"): + out = subprocess.run( + ["git", "-C", REPO, "ls-files"], capture_output=True, text=True, check=True + ).stdout + tracked_basenames.cache = {os.path.basename(p) for p in out.split("\n") if p} + return tracked_basenames.cache + + +def resolves(candidate): + candidate = candidate.split("#")[0].strip() + if not candidate or candidate.startswith(("http://", "https://", "s3://")): + return False + # Strip a trailing :line or :line,line citation, e.g. main.tf:243,298-299 + candidate = re.sub(r":[\d,\-]+$", "", candidate) + return os.path.exists(os.path.join(REPO, candidate)) + + +def backed(lines, idx): + lo, hi = max(0, idx - CONTEXT), min(len(lines), idx + CONTEXT + 1) + for line in lines[lo:hi]: + if QUALIFIER.search(line): + return True + for m in LINK.finditer(line): + if resolves(m.group(1) or m.group(2) or ""): + return True + for m in BARE.finditer(line): + if resolves(m.group(0)) or m.group(0) in tracked_basenames(): + return True + return False + + +def targets(): + yield os.path.join(REPO, "README.md") + for root, dirs, files in os.walk(os.path.join(REPO, "docs")): + if os.path.join("docs", "adr") in root: + dirs[:] = [] + continue + for f in files: + if f.endswith(".md"): + yield os.path.join(root, f) + + +def load_baseline(): + """Pre-existing unbacked claims, keyed by text rather than line number so the + baseline does not silently absorb a new claim when a file is edited.""" + if not os.path.exists(BASELINE): + return set() + entries = set() + for line in open(BASELINE, encoding="utf-8"): + line = line.strip() + if line and not line.startswith("#"): + entries.add(line) + return entries + + +def main(): + found = [] + for path in targets(): + if not os.path.exists(path): + continue + lines = open(path, encoding="utf-8").read().split("\n") + for i, line in enumerate(lines): + if CLAIM.search(line) and not backed(lines, i): + rel = os.path.relpath(path, REPO) + found.append((f"{rel}::{line.strip()}", f"{rel}:{i + 1}: {line.strip()}")) + + if "--update-baseline" in sys.argv: + with open(BASELINE, "w", encoding="utf-8") as fh: + fh.write(BASELINE_HEADER) + for key, _ in sorted(found): + fh.write(key + "\n") + print(f"Wrote {len(found)} entries to {os.path.relpath(BASELINE, REPO)}") + return 0 + + baseline = load_baseline() + new = [display for key, display in found if key not in baseline] + + if new: + print("New unbacked compliance claims (need a resolvable link or a 'design stage'") + print("qualifier within 3 lines) -- see docs/adr/ADR-0001-compliance-claims-remediation.md:\n") + for v in sorted(new): + print(f" {v}") + print("\nIf the claim is genuinely backed, add the link. Do not add it to the baseline.") + return 1 + + stale = len(baseline) - (len(found) - len(new)) + print(f"check_compliance_claims: OK -- no new unbacked claims ({len(baseline)} baselined).") + if stale > 0: + print(f" {stale} baseline entries no longer match; run --update-baseline to prune.") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/scripts/check_evidence_paths.py b/scripts/check_evidence_paths.py new file mode 100755 index 0000000..a89e913 --- /dev/null +++ b/scripts/check_evidence_paths.py @@ -0,0 +1,42 @@ +#!/usr/bin/env python3 +"""ADR-0001 Verification 3: evidence-path integrity. + +Resolves every `path:` entry in deployment/compliance/soc2-evidence-collection.yaml +and fails on any that does not exist and is not explicitly marked `# TODO`. +""" +import os +import re +import sys + +REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) +MANIFEST = os.path.join(REPO, "deployment", "compliance", "soc2-evidence-collection.yaml") +ENTRY = re.compile(r'^\s*path:\s*"([^"]+)"\s*(#.*)?$') + + +def main(): + violations = [] + checked = 0 + for i, line in enumerate(open(MANIFEST, encoding="utf-8").read().split("\n"), 1): + m = ENTRY.match(line) + if not m: + continue + checked += 1 + path, comment = m.group(1), m.group(2) or "" + if "TODO" in comment: + continue + # External URIs cannot be resolved from the repo and must be marked TODO. + if os.path.exists(os.path.join(REPO, path)): + continue + violations.append(f"soc2-evidence-collection.yaml:{i}: {path}") + + if violations: + print("Evidence paths that do not resolve and are not marked '# TODO':\n") + for v in violations: + print(f" {v}") + return 1 + print(f"check_evidence_paths: OK -- {checked} path entries, all resolvable or marked TODO.") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) From c6a000279f329c50d81773fe5f906c5052950105 Mon Sep 17 00:00:00 2001 From: Nick Ruest <127058086+nicholas-ruest@users.noreply.github.com> Date: Mon, 27 Jul 2026 14:28:35 +0000 Subject: [PATCH 2/3] Run the compliance claims checks in CI Adds the compliance-claims job the ADR's Verification section calls for. It was written and validated with the original change but could not be pushed then: the token lacked workflow scope. Uses needs['compliance-claims'].result rather than dot access in the ci-success result check -- a hyphenated job id is parsed as subtraction in a GitHub expression, so the dot form would not have evaluated correctly. Implements copilot-agent/docs/adr/ADR-0001-compliance-claims-remediation.md --- .github/workflows/ci.yml | 20 ++++++++++++++++++-- 1 file changed, 18 insertions(+), 2 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index f6bdd86..5118b2a 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -304,11 +304,26 @@ jobs: name: documentation path: target/doc/ + # ADR-0001: every compliance claim must be backed by an artifact. + # Pure-Python and dependency-free, so it needs no toolchain setup. + compliance-claims: + name: Compliance Claims Check + runs-on: ubuntu-latest + steps: + - name: Checkout code + uses: actions/checkout@v4 + + - name: Check compliance claims are backed + run: python3 scripts/check_compliance_claims.py + + - name: Check SOC 2 evidence paths resolve + run: python3 scripts/check_evidence_paths.py + # Summary job ci-success: name: CI Success runs-on: ubuntu-latest - needs: [format, lint, security, test, integration, coverage] + needs: [format, lint, security, test, integration, coverage, compliance-claims] if: always() steps: - name: Check job results @@ -318,7 +333,8 @@ jobs: [ "${{ needs.security.result }}" != "success" ] || \ [ "${{ needs.test.result }}" != "success" ] || \ [ "${{ needs.integration.result }}" != "success" ] || \ - [ "${{ needs.coverage.result }}" != "success" ]; then + [ "${{ needs.coverage.result }}" != "success" ] || \ + [ "${{ needs['compliance-claims'].result }}" != "success" ]; then echo "One or more required jobs failed" exit 1 fi From 7dbae85694e19b494594d4202b5f03b25ed0c649 Mon Sep 17 00:00:00 2001 From: Nick Ruest <127058086+nicholas-ruest@users.noreply.github.com> Date: Mon, 27 Jul 2026 14:40:06 +0000 Subject: [PATCH 3/3] Report the compliance claims check independently of ci-success Removes compliance-claims from the ci-success needs list. That umbrella job is currently red on main for unrelated reasons -- format, lint, security, test and coverage all fail there -- so nesting a passing documentation check inside it masked a meaningful signal behind debt the check cannot influence. The job still runs on every push and pull request and reports its own status. If it should block merges, make it a required check in branch protection, which gates on it directly rather than through an aggregate. This also makes the ci.yml change purely additive: the ci-success block is now byte-identical to main. Implements copilot-agent/docs/adr/ADR-0001-compliance-claims-remediation.md --- .github/workflows/ci.yml | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 5118b2a..913a004 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -306,6 +306,12 @@ jobs: # ADR-0001: every compliance claim must be backed by an artifact. # Pure-Python and dependency-free, so it needs no toolchain setup. + # + # Deliberately NOT in the ci-success needs list. This check is about documentation + # honesty and is independent of the Rust build; folding it into that umbrella would + # bury a green signal inside a job that is currently red for unrelated reasons it + # cannot influence. Read this job's own status instead, and make it a required check + # in branch protection if it should block merges. compliance-claims: name: Compliance Claims Check runs-on: ubuntu-latest @@ -323,7 +329,7 @@ jobs: ci-success: name: CI Success runs-on: ubuntu-latest - needs: [format, lint, security, test, integration, coverage, compliance-claims] + needs: [format, lint, security, test, integration, coverage] if: always() steps: - name: Check job results @@ -333,8 +339,7 @@ jobs: [ "${{ needs.security.result }}" != "success" ] || \ [ "${{ needs.test.result }}" != "success" ] || \ [ "${{ needs.integration.result }}" != "success" ] || \ - [ "${{ needs.coverage.result }}" != "success" ] || \ - [ "${{ needs['compliance-claims'].result }}" != "success" ]; then + [ "${{ needs.coverage.result }}" != "success" ]; then echo "One or more required jobs failed" exit 1 fi