medgate

A governed LLM gateway for clinical workloads — PHI guardrails, multi-provider routing with fallback, and per-call cost accounting, exposed behind an OpenAI-compatible API. Built on LiteLLM.

A gateway is where governance belongs. Instead of every service calling an LLM directly, medgate sits in front: it scrubs or blocks PHI before a prompt ever leaves the building, routes each request across providers with automatic fallback, and records cost and latency for every call. LiteLLM does the hard work of talking to 100+ providers; medgate is the clinical control plane around it.

Built on BerriAI/litellm (MIT core). LiteLLM is the multi-provider engine; medgate is the guardrail + routing + accounting layer. Ships a deterministic mock provider so the whole gateway runs and tests offline — no keys, no network.

Architecture

flowchart LR
  C[Client] -->|OpenAI-compatible<br/>/v1/chat/completions| G[medgate]
  G --> P[PHI guardrail<br/>block or redact]
  P --> R[Router<br/>priority + fallback]
  R --> L1[Provider A]
  R -. on failure .-> L2[Provider B]
  L1 --> M[Cost + latency<br/>accounting]
  M --> U[(Usage log)]
  M --> C

Loading

What it adds over calling LiteLLM directly

• PHI guardrail — every prompt is scanned for identifiers (names, emails, phones, MRNs, DOBs, Medicare numbers). In redact mode they're masked before the call; in block mode the request is refused with a 4xx. PHI never reaches a third-party model.
• Routing with fallback — a model name maps to an ordered list of providers; if the first errors or is unavailable, the next is tried automatically. One endpoint, many providers, no client changes.
• Cost accounting — token counts × a versioned price book give a per-call cost, logged with latency and provider. Feeds a FinOps pipeline like llm-ledger.
• OpenAI-compatible — drop-in /v1/chat/completions, so existing SDKs and tools work unchanged.

Quickstart

Prerequisites: Python 3.11+. Runs fully offline with the mock provider — no API key, no network.

git clone https://github.com/skalaliya/medgate && cd medgate
pip install -e ".[dev]"          # or: uv sync

uvicorn medgate.api:app --reload    # http://localhost:8000/docs

Call it like OpenAI — PHI is scrubbed automatically:

curl -X POST localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{
  "model": "clinical-default",
  "messages": [{"role": "user", "content": "Summarise notes for John Doe, DOB 1980-04-12, email a@b.com"}]
}'
# the prompt is redacted to "...for [NAME], DOB [DATE], email [EMAIL]" before routing

curl localhost:8000/usage    # per-call cost, latency, provider, PHI-redaction count

Configuration

Routing and guardrail mode are declared in medgate/config.py:

ROUTES = {
    "clinical-default": ["mock-fast", "mock-backup"],  # try mock-fast, fall back to mock-backup
}
GUARDRAIL_MODE = "redact"   # or "block"

Point a route at real providers (OpenAI, Anthropic, a local vLLM) by adding them to the provider registry with LiteLLM model strings and setting LITELLM_ENABLED=1.

Stack

Python 3.12 · FastAPI · LiteLLM (optional real backend) · Pydantic · pytest · ruff · GitHub Actions

Design notes

• Guardrail before routing — redaction happens first, so no downstream provider (mock or real) ever sees raw PHI.
• Provider protocol — MockProvider (deterministic, offline) and LiteLLMProvider (real) implement the same interface; routing and accounting don't care which is in use.
• Versioned price book — cost is computed from a seed table, so pricing changes are a one-line diff, not a code change.
• Accounting is first-class — every call, success or fallback, is logged with cost, latency, provider, and redaction count.

Roadmap

• Output guardrails (block responses that leak identifiers or unsafe content)
• Per-tenant rate limits and budgets
• Presidio-based PHI detection alongside the regex gate
• Streaming responses

License

MIT