GxP Equipment Recovery Intelligence Copilot

A source-grounded engineering decision-support tool for GMP biotech / cell-therapy facilities. When equipment breaks down, the copilot assembles the evidence - manuals, vendor identity, similar prior failures, calibration impact - into one reviewable package, with every decision gated by a transparent source-status policy and recorded in an append-only audit log.

Synthetic data only - not a validated GMP system. Does not approve repair or return-to-service. Does not replace QA, Validation, or OEM instructions. See corporate_boundary/NON_PRODUCTION_BOUNDARY.md for the full boundary statement.

Screenshots

Screen	What you see
	Start Center. Live asset count, audit-log card, governance disclaimer.
	Asset list. Filter by family / room grade / GMP criticality; URL-synced.
	Asset Object Page. Identity / History / Manuals tabs; vendor-resolution evidence trail.
	Maximo-style Failure Reporting. Class → Symptom → Cause → Remedy chain, with red flagging on missing steps.
	Recovery Wizard. Submit a breakdown, get the evidence package + gate decision + review-gate checklist.
	Audit Log explorer. Every recovery run, every gate decision, every cited evidence ID - drillable.

Why this exists

Facility engineering teams in regulated environments spend a disproportionate share of an unplanned-downtime incident gathering context: which manual is the right one, who's the OEM (vs. the distributor on the asset register), what did the team do last time this happened, what does the calibration record say. The copilot collapses that hunting into a structured, source-grounded retrieval pass - labeled clearly with official / similar / reference-only / insufficient - and forces every conclusion through a gate that can say "not enough evidence; escalate."

It is engineering decision support, not automation. The deterministic template fallback (no LLM required) is the default behaviour for a reason: every claim must cite a source, every source must carry a status, and the audit log records who did what.

What you get

• HTTP API (FastAPI). Typed Pydantic schema → auto-generated OpenAPI → typed TS client → Tanstack Query hooks. No hand-maintained DTOs.
• Frontend (Next.js 15 + @ui5/webcomponents-react - the real SAP Fiori design system). Asset / Manual / Work-Order list reports
  • Object Pages + a Maximo-style Failure Reporting view.
• Recovery wizard that drives the whole pipeline: intake → manual matcher → vendor resolver → similar-WO matcher → serviceability risk → RTS readiness → checklist → RAG orchestrator (template or OpenAI) → audit log.
• Source-status policy centralised in governance/source_policy.py. Visual badges in the UI mirror the gating rules.
• Audit log explorer with per-run timeline, evidence-ID drillthrough, and filters.
• Auth + RBAC (opt-in). NextAuth with Credentials and SMTP magic-link providers; HS256 JWTs verified by the Python backend; four-role hierarchy gated per-endpoint.
• CSV importers for IBM Maximo and SAP PM exports - bring your own data and the copilot runs against it.
• One-command Docker stack: docker-compose up.
• Test pyramid: ~210 backend tests, ~120 frontend unit tests, 18-route smoke script, 22 Playwright e2e tests. CI runs all of it.

Quickstart with Docker (recommended)

docker-compose up --build       # cold first build: ~3-5 minutes
# open http://localhost:3000

Use your own data:

docker-compose exec api python -m gxp_copilot.main import maximo \
  --type asset --file /app/data/sample_imports/maximo/assets.csv

docker-compose exec api python -m gxp_copilot.main import sap \
  --type asset --file /app/data/sample_imports/sap/equipment.csv

docker-compose down -v stops everything and wipes the data volume.

Quickstart from source

# 1. Create a virtualenv and install the package + deps
python3 -m venv .venv
source .venv/bin/activate
pip install -e .
pip install -r requirements.txt

# 2. Initialize the SQLite DB and load synthetic seed data
python -m gxp_copilot.main seed-db

# 3. Run the full recovery pipeline against a demo scenario
python -m gxp_copilot.main recover GTHX-001 \
  --issue "Pump will not start after setup"

# 4. Run the evaluation suite (writes outputs/evaluation_report.{json,md})
python -m gxp_copilot.main evaluate

# 5. Run the test suite
python -m pytest tests/ -q

# 6. Launch the Streamlit demo
bash scripts/run_streamlit.sh

To override settings, point --config at a YAML file (e.g. --config config/default.yaml); without --config the CLI uses the built-in defaults.

Demo scenarios

Three polished scenarios stress different parts of the pipeline:

ID	Asset	Failure	What's interesting
S1	GTHX-001 (GatheRex cell harvest pump)	Pump will not start	Listed manufacturer is a distributor (ABC Lab Supply); the OEM (Wilson Wolf) is recoverable from PO + manual evidence. Resolver surfaces the contradiction and recommends the right escalation path.
S2	INC-001 (CO2 incubator)	CO2 deviation alarm	Official OEM manual exists; calibration impact is in scope; checklist must include alarm_trend_attached + calibration_impact_assessed + validation_assessment.
S3	PC-001 (particle counter)	Data transfer failed	Data-integrity flavor; checklist must include data_evidence_attached.

Run any of them via:

python -m gxp_copilot.main recover INC-001 \
  --issue "CO2 deviation alarm during routine check" \
  --alarm "CO2 high deviation" \
  --failure-category co2_deviation \
  --product-material no

Bring your own data - Maximo / SAP PM importers

Two CSV importers ship out of the box:

gxp-copilot import maximo --type asset --file <path>.csv
gxp-copilot import maximo --type work-order --file <path>.csv

gxp-copilot import sap --type asset --file <path>.csv
gxp-copilot import sap --type work-order --file <path>.csv

Both report rows-seen / rows-imported / rows-rejected with per-row error detail. Re-running an import is idempotent (INSERT OR REPLACE against natural keys).

Sample CSVs and column-mapping references live in data/sample_imports/{maximo,sap}/. To support a new CMMS, copy one of src/gxp_copilot/data/importers/{maximo,sap}.py and adjust the FieldMapping tuple - see docs/adr/0012-importer-architecture.md.

HTTP API

A FastAPI layer wraps the same AssetRecoveryService the CLI and Streamlit use. This is the boundary the Next.js / Fiori frontend consumes.

python -m gxp_copilot.main api               # http://127.0.0.1:8000
# or:
gxp-copilot api --host 0.0.0.0 --port 8000

Then open http://127.0.0.1:8000/docs for auto-generated OpenAPI docs.

Endpoints:

Method	Path	Description
GET	/health	Liveness + LLM-mode flag
POST	/recover	Run the full recovery pipeline (returns RecoveryPackage)
GET	/assets · /assets/{id}	Asset list / detail
GET	/vendors · /vendors/{id}	Vendor list / detail
GET	/manuals · /manuals/{id}	Manual list / detail
GET	/work-orders · /work-orders/{id} · /work-orders/by-asset/{asset_id}	Work-order list, detail, history
GET	/audit?limit=N · /audit/{run_id}	Audit-log queries
GET	/clusters?threshold=N	Equipment clusters + documentation gap findings

Default CORS allows http://localhost:3000 (Next.js dev) and http://localhost:8501 (Streamlit). Override with GXP_API_CORS_ORIGINS.

Authentication (optional)

Auth is off by default - fresh clones and docker-compose up boot without any sign-in friction. To enable RBAC:

# 1. Generate a shared HMAC secret (≥32 bytes).
openssl rand -hex 32   # paste the output as AUTH_SECRET

# 2. In .env (or your shell):
GXP_AUTH_ENABLED=1
NEXT_PUBLIC_GXP_AUTH_ENABLED=1
AUTH_SECRET=<the secret>
GXP_ADMIN_EMAIL=you@example.com   # auto-seeded as ADMIN on first start

# 3. (Re)start.
docker-compose up --build
# open http://localhost:3000 → click "Sign in" → enter your email

Roles in ascending order: engineer < supervisor < qa < admin. Sprint 6 ships two gates: POST /recover requires ENGINEER+ and the audit log endpoints require SUPERVISOR+. Adding a new gate is a one-line Depends(require_role(Role.X)) on the route.

When GXP_AUTH_ENABLED=0 (the default), the backend attaches a synthetic anonymous admin to every request; nothing visibly changes versus pre-Sprint-6 behavior. See docs/adr/0013-auth-rbac.md for the full design rationale.

LLM mode

The RagOrchestrator accepts an injectable llm_call. When unset (the default) it produces a deterministic template response - useful for CI, reproducible evaluation, and offline demos.

To enable OpenAI, copy .env.example to .env and fill it in. Entry points (gxp-copilot api, gxp-copilot recover, streamlit run …) auto-load .env at startup, so you never have to export again:

cp .env.example .env
# Edit .env: set OPENAI_API_KEY=sk-... and GXP_USE_LLM=1
python -m gxp_copilot.main api

Shell-exported vars still win - .env is the defaults layer, not an override (see docs/adr/0006-dotenv-at-entry-points.md). Tests do NOT auto-load .env, so CI stays deterministic.

Default model is gpt-4.1-mini - see docs/adr/0002-openai-default-model.md for why. Hard 10s timeout; on any failure (timeout, network, parse error, hallucinated citations) the orchestrator silently falls back to the deterministic template. Tests run with the LLM path off; no real API call is made in CI.

Frontend (Next.js + Fiori)

A Next.js 15 frontend in apps/web/ consumes the FastAPI HTTP API and renders a SAP Fiori-styled UI via @ui5/webcomponents-react. It is the target frontend for the long-term project; the Streamlit demo is kept as an in-process ops UI.

# 1. JS prerequisites
corepack enable                    # ships with Node ≥ 20
pnpm install                       # workspace root

# 2. Generate (or refresh) the typed API client from the FastAPI schema
python scripts/dump_openapi.py     # writes apps/web/src/api/openapi.json
pnpm codegen                       # writes apps/web/src/api/schema.d.ts

# 3. Boot the API and the dev server (in two terminals)
python -m gxp_copilot.main api     # http://127.0.0.1:8000
pnpm dev:web                       # http://127.0.0.1:3000

Routes shipped:

Route	Description
/	Start Center - live asset count + audit-log card
/assets	List Report - filters (family / grade / GMP) + AnalyticalTable
/assets/{asset_id}	Object Page - header + Identity / History / Manuals tabs
/manuals	List Report - filters (family / status / source type)
/manuals/{manual_id}	Object Page - Identity / Source / Related
/work-orders	List Report - filters (category / cal impact / RTS)
/work-orders/{wo_id}	Maximo-style Failure Reporting + Identity / Review status
/recovery	Recovery Wizard - submit a breakdown report and assemble the evidence package; supports ?asset={id} deep-link prefill
/audit	Audit Log explorer - filter by actor / action / gate; click any row to drill into the run timeline
/audit/{run_id}	Per-run timeline of every audit event for one recovery
/clusters	Equipment-cluster explorer: documentation gap findings (orphans, duplicates, manufacturer mismatches) over a two-layer asset clustering

Smoke test the whole stack (boots API + frontend, hits every route, verifies the synthetic-data disclaimer, tears down):

bash scripts/smoke_web.sh

End-to-end browser tests (Playwright, Chromium):

pnpm --filter @gxp-copilot/web e2e:install   # one-time Chromium download (~92 MB)
bash scripts/run_e2e.sh                      # boots API + runs full suite

The full architectural rationale is in docs/adr/0004-pragmatic-monorepo.md, docs/adr/0005-no-tailwind.md, docs/adr/0007-openapi-typescript.md, docs/adr/0008-pnpm.md, and docs/adr/0009-defer-playwright.md.

Streamlit demo

Launch the UI locally:

bash scripts/run_streamlit.sh

The app exposes five pages, one per pipeline stage:

#	Page	What it shows
1	Intake	Asset ID + issue description; immediate flags
2	Asset identity	Vendor resolution, listed-vendor vs OEM contradictions, escalation path
3	Manual finder	Ranked manual evidence with source-status labels (official / similar / reference / insufficient)
4	Similar failures	Historical work-order analogues with transparent component scores and reasons
5	RTS readiness	Return-to-service gap, review gates, recommended next steps

To add screenshots, save PNGs to docs/screenshots/ and replace the placeholders below with ![Page](docs/screenshots/<file>.png):

• 01-intake.png - Intake page
• 02-asset-identity.png - Asset-identity / vendor resolution
• 03-manual-finder.png - Manual finder with source-status badges
• 04-similar-failures.png - Similar work-order list with reasons
• 05-rts-readiness.png - RTS readiness gap

Architecture

┌─────────────────────────────────────────────────────────────────┐
│ Streamlit UI  ·  CLI  ·  Evaluation harness                      │
└──────────────────────┬──────────────────────────────────────────┘
                       │  AssetRecoveryService.run(RecoveryRequest)
                       ▼
   IntakeService → ManualMatcher → VendorResolver → WorkOrderMatcher
                       │
                       ├─→ ServiceabilityRisk
                       ├─→ RtsGap
                       ├─→ ChecklistEngine
                       └─→ RagOrchestrator (gate → template / LLM, audit)
                       ▼
                 RecoveryPackage  →  EvidencePackageService (MD/JSON)

• Data layer (src/gxp_copilot/data/) - SQLite via stdlib sqlite3, CSV-driven seed loader, typed repositories returning Pydantic schemas.
• Retrieval (src/gxp_copilot/retrieval/) - BM25 + deterministic hashing-based vector index + metadata reranker, all wrapped by a HybridRetriever.
• Matching (src/gxp_copilot/matching/) - ManualMatcher, VendorResolver, WorkOrderMatcher with weighted, transparent scoring.
• Risk + Checklist (src/gxp_copilot/risk/, src/gxp_copilot/checklist/) - additive 0-100 serviceability risk; gate-based RTS readiness; data-driven checklist engine.
• Governance + LLM (src/gxp_copilot/governance/, src/gxp_copilot/llm/) - append-only JSONL audit log, source-status policy, source-gating, RAG orchestrator with deterministic template fallback and prohibited-claim post-validation.
• Services (src/gxp_copilot/services/) - orchestration layer used by all entry points (CLI, UI, evaluation).

Hard guardrails

The prototype never:

• Approves repair or return-to-service.
• Generates uncontrolled repair instructions (prohibited_claims_check enforced inside RagOrchestrator).
• Labels a similar or reference-only manual as if it were official (centralized in governance/source_policy.py).
• Connects to live CMMS, QMS, EMS, LIMS, ERP, or document-management systems.
• Uses non-synthetic data.

When evidence is insufficient, the gate forces an INSUFFICIENT_EVIDENCE or ESCALATE decision and the system declines to generate a summary.

Repository layout

config/                   default.yaml lives at the repo root
corporate_boundary/       Notes defining non-production and non-GMP boundaries
data/synthetic/           Seed CSVs (assets, vendors, manuals, WOs, calibrations,
                          service reports, POs, checklist rules)
data/manuals/             Synthetic manual text files used by the chunker
docs/                     Research goal, architecture, code plan, tests, evaluation
scripts/                  setup_db.py · seed_demo_data.py · run_evaluation.py ·
                          run_streamlit.sh
src/gxp_copilot/
  app.py                  AppContext wiring
  main.py                 CLI entrypoint (seed-db / recover / evaluate / readiness)
  schemas.py              Pydantic v2 DTOs and enums
  settings.py             YAML/env settings
  data/                   db, repositories, seed loader, validators
  ingestion/              chunker, document_loader, metadata_extractor
  retrieval/              bm25, vector, hybrid, reranker
  matching/               manual_matcher, vendor_resolver, work_order_matcher
  risk/                   serviceability_risk, rts_gap
  checklist/              checklist_engine
  governance/             audit_log, source_policy, validation_readiness
  llm/                    prompts, source_gating, rag_orchestrator
  services/               intake, asset_recovery, evidence_package
  ui/                     streamlit_app + 5 pages
  evaluation/             test_cases, metrics, run_evaluation
tests/
  unit/                   normalization, matchers, risk, checklist, audit, gating, policy
  integration/            asset_recovery_flow, evidence_package, rag_orchestrator
  evaluation/             metrics, regression scenarios

Evaluation results (synthetic)

n_cases:                  8
manual_recall_at_3:       1.00
manual_mrr:               1.00
wo_precision_at_3:        0.92
wo_recall_at_5:           1.00
checklist_coverage:       0.94
oem_accuracy:             1.00
vendor_role_accuracy:     1.00
gate_decision_accuracy:   1.00
unsupported_answer_rate:  0.00

Retrieval metrics (manual_recall_at_3, manual_mrr, wo_precision_at_3, wo_recall_at_5) are averaged only over cases with a non-empty expected list - cases with no labeled ground truth are excluded so they cannot inflate or deflate the headline.

These are computed by python -m gxp_copilot.main evaluate against the 8 labeled cases in src/gxp_copilot/evaluation/test_cases.py.

Baselines and ablations (Sprint 12)

The same evaluation run produces a head-to-head comparison of the full hybrid manual matcher against three baselines (BM25-only, fuzzy-metadata- only, embedding-only) and seven single-component ablations of the full matcher. Output lands in outputs/evaluation_report.md under the "Baselines and ablations" section.

Representative result on the labeled-cases subset:

Matcher                Recall@1  Recall@3  MRR
full_hybrid            -         1.00      1.00
bm25_only              0.75      1.00      0.83
fuzzy_metadata_only    1.00      1.00      1.00
embedding_only         0.75      0.75      0.80

Embedding-only is clearly weakest on this seed; BM25-only finds the right manual eventually but misranks the top hit on one case. The ablations all return identical scores on the current 4-labeled-cases subset - an honest signal that the labeled set is too small to discriminate between weight configurations and the path to a stronger research claim is more labeled cases, not more model tuning.

Equipment clustering and gap analysis (Sprint 11)

python -m gxp_copilot.main cluster-equipment

A two-layer asset clusterer (deterministic family key, then embedding merge of singletons) groups assets into manual families and emits four classes of actionable finding:

• singleton_no_manual - asset has no cluster siblings AND no manual
• duplicate_manuals - one cluster carries multiple manuals
• manufacturer_mismatch - listed manufacturer disagrees with cluster's dominant OEM
• orphan_asset - cluster siblings have a manual but this asset doesn't

Same data is exposed at GET /clusters and rendered at /clusters in the frontend.

Boundary statement

Every UI surface, every Markdown export, and every JSON evidence package is labeled synthetic data only. Any real deployment in a regulated environment would require formal computer software assurance, data governance, validation strategy, security review, and controlled-system integration review. The non-production checklist in governance/validation_readiness.py enumerates the gates a real deployment would have to clear.