Skip to content

docs: sharpen PetroBrain product and safety positioning #25

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Lingz450 merged 1 commit into from
Jul 21, 2026
Merged

docs: sharpen PetroBrain product and safety positioning #25

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
263 changes: 99 additions & 164 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,195 +1,130 @@
# PetroBrain - Phase-1 Repository (Tier A spine + two specialist modules)

[![CI](https://github.com/Lingz450/PetroBrain/actions/workflows/ci.yml/badge.svg)](https://github.com/Lingz450/PetroBrain/actions/workflows/ci.yml)

A working Phase-1 scaffold for a domain-locked oil & gas AI. This repo contains:

1. **The shared spine** - FastAPI services, the orchestrator/agent runtime, the
LLM-provider abstraction (Tier A hosted / Tier B self-hosted), the safety guardrail
layer, and the RAG pipeline (clause-aware chunking → embeddings → pgvector hybrid
search → rerank).
2. **The deterministic calculation engine** - unit-safe (`pint`) engineering calcs.
Numbers come from here, never from the LLM's head.
3. **Specialist module A - Well Control / Kill Sheet** - fully built: KMW, ICP, FCP,
strokes, Wait-and-Weight pressure schedule, influx analysis, MAASP, live-event
routing, and the decision-support safety banner.
4. **Specialist module B - NUPRC Tier-3 MRV** - emissions engine (flaring carbon
balance, venting, fugitive Tier 2 *and* Tier 3, combustion), CO2e with configurable
IPCC GWP, and a GHGEMP report generator with tier-readiness gaps and an audit hash.
5. **The eval / safety harness** - golden engineering set + a red-team safety set that
must pass with **zero failures** before any deploy.

All engineering math is validated by tests (`python tests/test_calculations.py` →
16/16 pass; `python tests/eval_harness.py` → 0 failures).

---

## Why it's built this way (the load-bearing decisions)

- **Two tiers, one codebase.** `PB_LLM_PROVIDER=anthropic` runs Tier A (cloud knowledge
tier, hosted frontier model). `PB_LLM_PROVIDER=self_hosted` runs Tier B (on-prem behind
the OT DMZ, open-weights model, no outbound calls). Same orchestrator, same tools.
- **The LLM never does arithmetic.** Every number is produced by the deterministic calc
engine / specialist modules and called as a *tool*. The orchestrator executes the tool
and feeds the result back. The post-guardrail flags any unverified number.
- **Safety is structural, not just prompted.** Guardrails refuse safety-system-bypass
requests, route live events to immediate-action guidance first, and enforce the
verification banner on safety-critical output. The red-team eval gates deploys.
- **Citations are first-class.** Chunks carry document/revision/clause metadata; the
retriever returns citation-grade hits; the post-guardrail rejects fabricated clause
references.
- **Tenant isolation is mandatory.** Every retrieval is tenant-filtered (and should also
use Postgres RLS). The retriever can't query without a tenant id.

---

## Layout
# PetroBrain

```
Safety-focused oil and gas AI decision support with cited retrieval, deterministic engineering calculations, governed research, and auditable operational workflows.

> **Status:** Active development. PetroBrain is decision-support software, not an autonomous control system and not a substitute for qualified engineering review, approved procedures, or regulatory verification.

[![CI](https://github.com/Trust-Code-System/PetroBrain/actions/workflows/ci.yml/badge.svg)](https://github.com/Trust-Code-System/PetroBrain/actions/workflows/ci.yml)

## What the system contains

- A FastAPI service layer and provider-abstracted orchestration runtime.
- Tenant-scoped retrieval with clause-aware ingestion, embeddings, pgvector search, reranking, and citation metadata.
- Deterministic, unit-aware engineering calculations that remain separate from language-model generation.
- Well-control and kill-sheet decision-support workflows.
- Emissions and MRV workflows with audit-oriented outputs.
- Governed research that requires explicit plan approval and records a source ledger.
- Background document ingestion, object storage, structured logs, metrics, and operational health endpoints.

## Safety model

PetroBrain is structured around four boundaries:

1. **Models do not perform authoritative arithmetic.** Engineering values come from reviewed calculation modules and are returned to the orchestrator as tool output.
2. **Sources remain visible.** Retrieval results preserve document, revision, clause, and tenant context for citation-grade responses.
3. **Critical output is guarded.** Pre- and post-generation checks handle unsafe requests, unsupported references, and unverified numerical claims.
4. **Humans retain responsibility.** Operational decisions, filings, procedures, and live-event actions require qualified review and current source material.

## Architecture

```text
app/
main.py FastAPI entrypoint (Tier-A spine)
config.py env-driven settings (tier, providers, stores)
api/ routes: /chat, /well-control/kill-sheet, /emissions/inventory
core/
prompts.py base prompt + module preambles + runtime context assembly
llm_service.py hosted (Anthropic) + self-hosted (vLLM/TGI) abstraction
guardrails.py pre/post safety checks
orchestrator.py the agent runtime (guardrail→retrieve→prompt→LLM→tools→guardrail)
rag/
chunking.py clause-aware splitter
embeddings.py embedding provider abstraction
vectorstore.py pgvector hybrid search + reciprocal-rank fusion
retriever.py embed→hybrid→rerank→citation-grade hits
ingest.py classify→extract→chunk→embed→index
calc/
units.py pint registry + named oilfield constants
drilling.py hydrostatic, ECD, kill mud weight, MAASP
production.py Vogel IPR, Arps decline
modules/
well_control/ SPECIALIST MODULE A (kill_sheet.py + agent.py)
emissions_mrv/ SPECIALIST MODULE B (engine.py + factors.py + ghgemp_template.py)
models/schemas.py API schemas
tests/
test_calculations.py 16 validated engineering tests
eval_harness.py golden + red-team safety gate
├── api/ Versioned HTTP routes and dependencies
├── core/ Orchestration, providers, prompts, and guardrails
├── rag/ Ingestion, embeddings, retrieval, and vector storage
├── calc/ Deterministic engineering calculations
├── modules/ Well-control and emissions specialist workflows
├── workers/ Asynchronous document processing
├── db/ Persistence and migrations
└── models/ API and domain schemas
frontend/ Web, field, admin, and shared frontend packages
infra/ Deployment, security, and infrastructure guidance
tests/ Unit, integration, safety, and evaluation coverage
```

---
## Stack

## Run it
| Layer | Technology |
| --- | --- |
| API | Python, FastAPI, Pydantic, Uvicorn |
| Data | PostgreSQL, asyncpg, psycopg, pgvector, Redis |
| AI | Anthropic/OpenAI provider abstraction, embeddings, RAG |
| Documents | Celery, S3-compatible storage, pdfplumber, python-docx |
| Security | JWT, bcrypt, TOTP, tenant context, guarded outputs |
| Observability | structlog, OpenTelemetry, Prometheus |
| Delivery | Docker Compose, Render configuration, GitHub Actions |

```bash
# 1. infra (Postgres+pgvector, Redis, MinIO)
docker compose up -d db redis minio
## Local setup

# 2. python env
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
### Prerequisites

# 3. validate the engineering math and the safety gate (no infra needed)
python tests/test_calculations.py
python tests/eval_harness.py
- Python 3.11 or newer
- Docker with Compose
- PostgreSQL with pgvector, Redis, and S3-compatible storage

# 4. configure + run the API
cp .env.example .env # add ANTHROPIC_API_KEY / OPENAI_API_KEY
uvicorn app.main:app --reload
# POST /chat, /well-control/kill-sheet, /emissions/inventory, /research/plan

# 5. async document ingestion (A5) - run the Celery worker
celery -A app.workers.celery_app worker --loglevel=info -Q petrobrain.ingest
# then POST a SOP as multipart:
# curl -X POST http://localhost:8000/admin/documents \
# -H "Authorization: Bearer <admin token>" \
# -F "file=@sop.pdf" \
# -F 'metadata={"document_id":"SOP-1","title":"Kick SOP","asset":"Asset-A"}'
# poll GET /admin/documents/{ingest_id} for status: queued|extracting|embedding|done|failed
### Start infrastructure and API

```bash
docker compose up -d db redis minio
python -m venv .venv
```

For local development, mint a JWT with the same `PB_JWT_SECRET`, issuer, and audience
as `.env` and paste it as `Authorization: Bearer <token>`:
Activate the virtual environment for your shell, then:

```bash
python - <<'PY'
from datetime import datetime, timedelta, timezone
import jwt

now = datetime.now(timezone.utc)
print(jwt.encode({
"sub": "u1",
"user_id": "u1",
"tenant_id": "demo",
"role": "engineer",
"allowed_assets": ["*"],
"iss": "petrobrain",
"aud": "petrobrain-api",
"iat": now,
"exp": now + timedelta(hours=8),
}, "dev-secret-change-me-32-bytes-minimum", algorithm="HS256"))
PY
pip install -r requirements.txt
cp .env.example .env
uvicorn app.main:app --reload
```

Initialize the vector schema once from `app/rag/vectorstore.py::SCHEMA`.

---
Use [`.env.example`](./.env.example) as the variable-name reference. Real provider credentials, JWT secrets, database URLs, storage keys, and tenant data must never be committed.

## Governed Research Mode
## Validation

Research Mode is a tenant-scoped, read-only analyst workflow for oil and gas
questions. It requires explicit plan approval before execution, searches only
enabled tenant documents and approved public domains, streams progress with
server-sent events, and persists the source ledger, report, evidence pack, and
audit metadata.
Run the checks relevant to your change. The repository CI is the source of truth for the complete gate.

The lifecycle is:
```bash
pytest
python tests/eval_harness.py
ruff check .
mypy app
```

1. `POST /research/plan`
2. `POST /research/{id}/approve-plan`
3. `POST /research/run?stream=true`
4. `GET /research/{id}` or `GET /research/{id}/events`
5. `POST /research/{id}/export`
For calculation changes, add or update deterministic fixtures and record the governing formula/source. For retrieval changes, verify tenant isolation, citation integrity, refusal behavior, and evaluation results.

Set `PB_TAVILY_API_KEY` to enable public web research. Without it, runs continue
against tenant documents and record `web_search_disabled`. External connectors
are intentionally rejected even if requested; `PB_RESEARCH_CONNECTORS_ENABLED`
is reserved for a future governed connector registry and does not bypass that
boundary.
## Governed research lifecycle

Local JSON persistence uses `PB_RESEARCH_STORE_PATH`. With
`PB_PERSISTENCE_BACKEND=postgres`, apply migration
`app/db/migrations/015_research_runs.sql`; tenant row-level security is enforced
through the same connection context as the other repositories.
```text
create plan → review plan → approve → execute → inspect sources → export
```

The analyst workspace is available at `/research` in the web app. Roles are
limited to `platform_admin`, `admin`, `engineer`, and `hse`.
Research is tenant-scoped and read-only. Without a configured public-search provider, runs continue against approved tenant material and explicitly record that web search was unavailable.

---
## Deployment

## The three deep-dives, mapped to files
Deployment requires more than a successful container build:

**(a) Phase-1 spine + RAG + calc engine** → `app/` (core, rag, calc) + `app/main.py`.
- rotate and store secrets in the deployment platform;
- apply database migrations and tenant-level policies;
- configure immutable or off-host audit retention;
- configure alerts for provider, ingestion, audit, and storage failures;
- validate backup and restore procedures;
- run safety and retrieval evaluations against the release candidate;
- confirm current engineering and regulatory source material.

**(b) Well-control kill-sheet specialist** → `app/modules/well_control/`. `kill_sheet.py`
is the full worked engine (validated: KMW 10.37 ppg, ICP 1200 psi, FCP 864 psi, MAASP
1144 psi, gas influx inferred). `agent.py` shows the module pattern (base prompt + module
preamble + tool schema) and the live-event routing.
See [`infra/README.md`](./infra/README.md), [`infra/SECURITY.md`](./infra/SECURITY.md), and [`docs/`](./docs/) for implementation-specific guidance.

**(c) NUPRC Tier-3 MRV product** → `app/modules/emissions_mrv/`. `engine.py` computes the
inventory; the *same* engine serves Tier 2 (factor-based) and Tier 3 (measurement-based) -
the difference is recorded per line, which is exactly the Q3-2026→Jan-2027 transition.
`ghgemp_template.py` emits the audit-ready report with tier-readiness gaps.
## Security and data handling

---
- Do not ingest confidential operational material into an unapproved environment.
- Keep service and provider credentials server-side and rotate any exposed value immediately.
- Enforce tenant filtering at application and database layers.
- Avoid logging prompts, documents, secrets, or personally identifiable information by default.
- Treat model output as untrusted until source, calculation, and policy checks pass.
- Review dependency, container, and workflow findings before each production release.

## What this scaffold deliberately leaves as plug-points (Phase 2+)
## Contribution context and licence

- P&ID / scanned-document OCR extractors (slot into `rag/ingest.py::extract`).
- Cross-encoder reranker (slot into `rag/retriever.py`).
- Trained classifiers replacing the regex guardrail baselines.
- Tier-B historian/SCADA read-only connectors behind the DMZ.
- The knowledge-graph asset hierarchy service.
- Auth: real JWT/SSO + Postgres row-level security in `api/deps.py`.
PetroBrain is maintained collaboratively under TrustCode System Limited. Use commit and pull-request history when describing individual contributions.

> Compliance note: GWP values and emission factors in `emissions_mrv/factors.py` are
> reference values. Before any NUPRC filing, set them to the current gazetted NUPRC
> guidance and the operator's applicable IPCC tier, and record the source in the audit
> trail. PetroBrain is decision support - submissions remain the operator's responsibility.
No open-source licence is currently granted. Public visibility does not by itself permit reuse, modification, or redistribution.
Loading