no-fabricate-gate

A read-only fact-honesty gate for AI-assisted writing. Every statistic in your published copy must trace to a cited source. Anything that doesn't fails the build.

Sister gate to de-ai-gate: de-ai-gate catches machine-generated style; no-fabricate-gate catches machine-generated facts. Together they are an honesty suite for content that ships from an AI pipeline.

The problem

LLM-drafted content invents numbers. Not maliciously: a model asked to write a persuasive paragraph will produce "studies show 73% of teams..." because that is what persuasive paragraphs look like. Once published, a fabricated statistic is a liability you can't see, because it reads exactly like a real one.

The fix is mechanical, not editorial: maintain a source-pack manifest (the cited facts your article is allowed to state) and gate the build on it. A number with no source in the manifest does not ship.

Why this one is safe to install

• Read-only: scans files, writes nothing, exits with a code.
• No auto-run: invoked explicitly; no SessionStart hook, no side effects.
• No credentials: zero stored keys, zero cookie/token access, no auth.
• No network: pure local regex over local files. Nothing leaves your machine.
• Zero dependencies: Node >= 18 or Bun.

MIT licensed. Author: Bryan Benner.

Usage

node cli.mjs --manifest sources.json ./public/articles ./public/guides

Scans .html, .htm, .md, .txt. Exit 1 if any data-claim in the content has no matching value in the manifest (use --warn-only for a non-blocking baseline run). Exit 2 on usage/manifest errors: a malformed manifest fails closed, never open.

The manifest

{
  "sources": [
    { "claim": "share of enterprises with AI in production",
      "value": "41%",
      "url": "https://example.org/2026-enterprise-ai-study" }
  ],
  "ignore": ["\\$99(/mo)?\\b"]
}

• sources[].value: the exact figure as it may appear in copy. Matching is case-, comma-, and whitespace-insensitive, so 41% in the manifest matches <strong>41</strong>% in served HTML and 1,200 matches 1200.
• sources[].claim / sources[].url: what the figure asserts and where it comes from. The gate requires them so the manifest stays a citation record, not a bare allowlist.
• ignore: optional regex list for numbers that are yours, not claims about the world: your own prices, your own version strings. These never need a citation.

What counts as a data-claim

Five statistic shapes: percentages (41%), money ($12/mo, $1,200k), multipliers (3.5x), ratios (7 out of 10), and scale words (2.4 million).

Structural numbers are exempt automatically: years (2026), ordinals (2nd), listicle counts (7 reasons), section references (section 3).

Wiring it into a pipeline

The intended shape (proven in production on a fully autonomous content engine):

1. Research produces the source-pack: real figures with real URLs.
2. The writing step receives only the manifest as its numeric universe; the prompt says "use no number that is not a value below".
3. This gate runs pre-publish (pre-commit hook or CI) and blocks anything the writing step invented anyway.

Step 3 is what makes steps 1 and 2 honest. A prompt rule without a gate is a suggestion.

# pre-commit / CI
node /path/to/no-fabricate-gate/cli.mjs --manifest content/sources.json public/ || exit 1

Programmatic use

import { checkNoFabricate } from './engine.js';
const { ok, unsourced, claims } = checkNoFabricate(html, manifest);

Test

bun test

Live examples

The gate in production, on pages that pass it:

• What the no-fabricate gate does with a live dogfood run
• Why AI writing invents numbers
• Source packs: the manifest workflow
• Provably safe to install
• All honesty tools