Groundedness Engine

A reusable TypeScript library for deterministic structural verification of AI-generated claims against cited source documents.

Motivation

LLMs can generate convincing claims with citations and quoted text. Before a downstream system trusts those claims, it needs a fast way to verify that:

• The cited section exists in the source document
• The model actually saw that section (retrieval trace)
• The quoted evidence appears verbatim in that section

Groundedness Engine performs those checks deterministically. It intentionally focuses on structural verification and leaves semantic reasoning to higher-level evaluation systems.

How It Works

flowchart TD
    A[Source Document] --> B[Section Parser]
    B --> C[Structured Sections]
    C --> D[AI Generated Memo]
    D --> E[Claims with Citations]
    E --> F[Groundedness Engine]
    F --> G{4 Ordered Checks}
    G --> G1[Citation Present?]
    G1 -->|No| H1[claim_unsupported]
    G1 -->|Yes| G2[Section Exists?]
    G2 -->|No| H2[section_not_found]
    G2 -->|Yes| G3[Section Retrieved?]
    G3 -->|No| H3[section_not_retrieved]
    G3 -->|Yes| G4[Quote Matched?]
    G4 -->|No| H4[quote_mismatch]
    G4 -->|Yes| H5[verified]
    H1 --> I[Verification Report]
    H2 --> I
    H3 --> I
    H4 --> I
    H5 --> I

Loading

Checks run in strict order. The first failure determines the verdict.

Example

Claim:

"Criminal penalties include imprisonment of up to 5 years."

Actual text in Section 302:

"...shall be subject to imprisonment for not more than 3 years..."

Result:

verdict: quote_mismatch

checks:
  citationPresent:   true
  sectionExists:     true
  sectionRetrieved:  true
  quoteMatched:      false

The claim cited the correct section and was retrieved, but the quoted excerpt doesn't appear in the text. The AI changed "3 years" to "5 years" — a hallucination the engine caught deterministically.

Verification Report

Claim: "EPA must respond within 180 days."
  ✓ Citation Present
  ✓ Section Exists
  ✓ Section Retrieved
  ✓ Quote Matched
  → verified

Claim: "Violations result in $25,000 per day."
  ✓ Citation Present
  ✓ Section Exists
  ✓ Section Retrieved
  ✓ Quote Matched
  → verified

Claim: "Criminal penalties up to 5 years."
  ✓ Citation Present
  ✓ Section Exists
  ✓ Section Retrieved
  ✗ Quote Matched
  → quote_mismatch

Groundedness Score: 83% (5/6 claims verified)

Benchmark

Check	What it catches	Example
citation_present	Empty citation fields	LLM says "X is true" with no source
section_not_found	Citation to nonexistent section	LLM cites "Section 999" that doesn't exist
section_not_retrieved	Citation to unseen section	LLM cites a section that wasn't in its context
quote_mismatch	Hallucinated or altered quotes	LLM changes "3 years" to "5 years"

Features

• Zero LLM calls — The verifier never calls an LLM. All checks are string-based.
• Deterministic — Same inputs always produce the same output.
• Retrieval tracing — Detects when an AI cites a section it never saw.
• Flexible section matching — Handles "Section 302", "SEC. 302.", "§ 302", etc.
• Whitespace normalization — Tolerates formatting differences without fuzzy matching.
• Clean API — Two functions: verifyClaim() and verifyClaims().

Running the Project

npm install
npm test

To run the demo server:

npm run dev --prefix apps/legislative-demo

Then open http://localhost:3000.

Project Structure

packages/verifier/    Core verification engine (reusable npm package)
cli/                  Standalone CLI for verifying claims
apps/legislative-demo Demo application (Express + Federal Register API)

CLI Usage

npx tsx cli/verify-claim.ts \
  --doc packages/verifier/fixtures/sample-bill.txt \
  --section 301 \
  --quote "The Administrator shall act on all permit applications within 180 days of receipt."

Output:

{
  "verdict": "verified",
  "checks": {
    "citationPresent": true,
    "sectionExists": true,
    "sectionRetrieved": true,
    "quoteMatched": true
  }
}

Known Limitations

This engine verifies structural groundedness, not semantic correctness.

It answers: "Did the AI actually quote this text from this section?"

It does not answer: "Is this quote taken out of context?" or "Does this claim logically follow from the source?"

Semantic evaluation is intentionally outside the scope of this project. The engine is designed to be the first layer of verification — fast, cheap, and deterministic. Semantic analysis can be layered on top.

Other limitations:

• Section parsing handles common legislative formats (SEC., §, Section, TITLE) but not all document types.
• The engine requires retrievedSectionIds to detect retrieval failures. If you don't track what was sent to the LLM, that check is skipped.

Why This Architecture

The verifier is a standalone npm package (@groundedness-engine/verifier), not coupled to the demo application.

This means you can import it into any AI pipeline — legal, medical, financial, technical — without pulling in Express, Federal Register APIs, or DeepSeek. The demo is just one way to use it.

Future Work

• Hybrid semantic evaluation for context and logical consistency
• Support for additional document formats (markdown, JSON, XML)
• Batch verification with parallel processing
• Integration hooks for popular LLM frameworks