An evidence-backed auditor for completed AI coding-agent sessions — it extracts every claim an agent made, re-runs the evidence in an isolated sandbox, and hands you a verdict you can actually trust: VERIFIED, REFUTED, or UNVERIFIABLE, never a guess.
Demo Video · Quick Start · Demo Walkthrough · Architecture · Security
▶ Watch RECEIPTS in action (3 min)
A live audit: an agent claims "all tests pass," RECEIPTS re-runs the evidence in a locked-down sandbox and returns a verdict backed by executed proof — including a green test suite refuted by an AST tamper scan.
Every session with a coding agent ends the same way — a confident final message:
"Done. All tests pass." "Fixed the race condition." "Implementation complete, no other changes needed."
These statements are the agent grading its own homework. Nothing forces them to be true, and in practice they frequently aren't:
- Stale success — the agent ran the suite once, made one more edit, and never re-ran it. The "all tests pass" you're reading describes a repo state that no longer exists.
- Removed tests, not fixed bugs — the fastest way to turn a red suite green is to delete the failing test. It works. It is also a lie.
- Phantom files and scope creep — the agent claims it touched three files; the diff touches eleven, or the file it claims to have created was never written.
- Hallucinated completion — a claim with no corresponding action anywhere in the transcript.
- Context bloat and prompt drift — by turn 40 the agent is working from a version of your instructions that quietly diverged from what you actually asked for.
- Instruction conflicts —
AGENTS.mdsays one thing,CLAUDE.mdsays another, and the agent picked whichever it saw last.
None of this is exotic. It's the default failure mode of trusting a model's summary of its own work. And today, catching it means a human re-reading the diff, re-running the suite, and re-deriving the same evidence the agent already claimed to have — every single time. That's the tax RECEIPTS removes.
| Tool category | What it does | What it doesn't do |
|---|---|---|
| CI/CD | Runs a fixed pipeline on push | Doesn't audit an agent's claims against the transcript, and won't catch a suite that passes only because coverage was quietly deleted |
| Code review / GitHub PRs | Human or bot reads a diff | Reviews what changed, not whether the agent's narration about the session matches what actually happened |
| IDE extensions & coding agents | Write and edit code | Grade their own work; the claim and the verifier are the same untrusted party |
| Agent observability platforms | Log tokens, latency, tool calls | Tell you what happened, not whether what the agent said happened is true |
There is no category of tool today whose job is to sit between a finished agent session and your trust in it, re-deriving evidence independently. That's the open lane RECEIPTS occupies: a post-hoc, evidence-required auditor — not a CI gate, not a code reviewer, not another agent. It never edits your code and never blocks a merge. It answers one narrow, load-bearing question: did the agent's claims survive a fresh check of reality?
RECEIPTS turns a finished agent session into an audit, end to end, with no manual re-verification:
AI agent finishes its session
│
▼
You open RECEIPTS
│
▼
RECEIPTS discovers the completed session (Codex, Claude Code, OpenHands, OpenCode,
│ Gemini CLI, Copilot CLI, Cline, Amazon Q)
▼
Extracts every claim, explicit or implied
│
▼
Plans deterministic verification per claim
│
▼
Runs it in a locked-down, no-network Docker sandbox
│
▼
Builds the Evidence Graph (every verdict traces to a node)
│
▼
Adjudicates: VERIFIED / REFUTED / UNVERIFIABLE — evidence required
│
▼
Computes a reproducible Trust Score
│
▼
You review the claim, the verdict, and the exact evidence behind it
No claim is ever taken on faith, and no claim is ever refuted on faith either — REFUTED requires an attached, freshly executed command, diff, or file observation. If nothing can be checked deterministically, the claim is honestly UNVERIFIABLE, never silently passed.
Verification Core — the foundation everything else builds on. Normalizes any supported agent's transcript into one provider-agnostic Session Package, extracts claims with a strict schema (no free-text parsing), plans a verifier per claim type, executes it in the sandbox, and adjudicates the result. This is where INV-1 lives: a claim may be marked REFUTED only when contradicting executed evidence is attached.
Evidence Graph — every verdict, score, and insight is a node connected to the evidence that produced it (SQLite adjacency tables + recursive CTE traversal + full-text search). If a conclusion can't be traced back through the graph, RECEIPTS refuses to display it. This is what makes "why did it say REFUTED?" a one-click answer instead of a re-investigation.
Trust Score — six independent, deterministic sub-scores (evidence coverage, test validation, build status, contradiction penalty, missing evidence, instruction consistency) combined with fixed, versioned weights. A single REFUTED claim caps the whole score — you can't average away a lie. Identical input always reproduces the identical score.
Prompt Studio — analyzes a prompt before you send it: exact token counts, near-duplicate/redundant instruction detection, instruction-file conflicts — all deterministic, before any model is consulted. Rewrites always ship with a diff and a measured (never estimated) token-savings number.
Ask RECEIPTS — a scope-restricted assistant for the audit you're looking at, not a general chatbot. It labels every statement Verified / Inferred / Unknown, cites the evidence node behind each claim, and refuses questions outside the selected audit.
Dashboard — Trust Score, session/repository/workflow/instruction health, and verification summary in one cached view; every card drills down into its backing evidence subgraph.
Provider Management — auto-discovers LLM provider keys from the environment or OS keychain, routes through LiteLLM with built-in rotation/cooldown/fallback, and is honest in the UI about what multiple keys actually buy you (real resilience across providers — not the throughput myth of keys sharing one org's rate limit).
Plugin SDK — third parties add session adapters, intelligence analyzers, or exporters via receipts_sdk (entry-point discovery + a declared permission manifest) with zero changes to core. A sample plugin ships in examples/ and runs in CI to prove the contract holds.
Exports — Markdown, HTML, and JSON reports that carry the Trust Score and full provenance references, with secret redaction always on.
Scenario 1 — the honest case. Agent: "All tests pass." RECEIPTS re-runs the full suite fresh, in the sandbox, on the final repo state. Exit code 0. Verdict: VERIFIED, with the exact command and output attached.
Scenario 2 — stale success.
Agent: "All tests pass." But the agent made one more edit after the last green run. RECEIPTS re-runs the suite against the actual final state and gets a failing assertion. Verdict: REFUTED — evidence: assert -1 == 5, captured fresh.
Scenario 3 — the classic trick. Agent: "Fixed the race condition — both tests pass." RECEIPTS's AST-based tamper detector diffs the session's test files and finds a removed test function and its assertion. Two tests pass because one test is gone. Verdict: REFUTED — evidence: the diff, with the deleted function named.
Scenario 4 — honestly unverifiable. Agent claims a behavior didn't change, but no deterministic before/after probe exists for that claim class in this release (the probe that could check it ships intentionally disabled — it's the one verifier capable of a false refutation). Verdict: UNVERIFIABLE("probe class disabled in v3") — not a pass, not a guess.
flowchart TD
A[Completed agent session<br/>Codex · Claude Code · OpenHands · OpenCode<br/>Gemini CLI · Copilot CLI · Cline · Amazon Q] --> B[Session Adapter]
B --> C[Normalized Session Package]
C --> D[Claim Extraction<br/>strict schema, LLM]
D --> E[Verification Planner<br/>deterministic, closed set]
E --> F[Docker Sandbox<br/>--network none · read-only mount · pinned digest]
F --> F1[pytest_exec]
F --> F2[test_tamper — AST diff]
F --> F3[file_stat]
F --> F4[scope_check]
F --> F5[build + lint]
F1 & F2 & F3 & F4 & F5 --> G[Adjudicator<br/>REFUTED requires executed evidence]
G --> H[(Evidence Graph<br/>SQLite + recursive CTE + FTS5)]
H --> I[Trust Score<br/>6 signals, versioned weights]
H --> J[Intelligence Insights]
H --> K[Ask RECEIPTS]
I & J & K --> L[Dashboard / Reports / Exports]
Every arrow in this diagram is a real module boundary in app/core/ and app/subsystems/ — subsystems depend only on the core, never on each other, so no feature can silently bypass adjudication.
| Layer | Technology | Why |
|---|---|---|
| Backend runtime | Python 3.12 + FastAPI (uvicorn) | Async-native, typed, fast to iterate on a strict pipeline |
| Persistence | SQLite (WAL mode), hand-written SQL repositories | Local-first, zero infra, additive migrations — no ORM lock-in |
| LLM gateway | LiteLLM Router (in-process) |
Battle-tested rotation/cooldown/fallback instead of reinventing it; RECEIPTS only auto-discovers keys and builds the model list |
| Token counting | tiktoken (o200k) / gpt-tokenizer in-browser |
Exact, deterministic counts — "measured token savings" has to mean measured |
| Near-dup detection | datasketch (MinHash/LSH) |
Deterministic redundant-instruction and prompt-similarity detection with no model call |
| Sandbox | Docker SDK for Python, digest-pinned image | No mutable tags, no network, no root — a verifier can't be a supply-chain vector |
| Plugin system | importlib.metadata entry points + pluggy |
The exact mechanism first-party adapters use — no privileged internal API |
| Secret storage | keyring (OS keychain) |
Provider keys never touch disk in plaintext or appear in logs |
| Observability | OpenTelemetry (GenAI semantic conventions), opt-in | Production-shaped tracing without leaking prompt content by default |
| Frontend | React 18 + TypeScript + Vite + Tailwind | Fast iteration, full type safety end to end |
| Server state | TanStack Query | Caching, dedupe, background refetch without hand-rolled state machines |
| Local UI state | Zustand | Ephemeral panel/selection state kept strictly separate from server data |
| Streaming | Server-Sent Events | Simplest transport that satisfies audit-progress and token streaming |
| Testing | pytest, ruff, mypy, vitest, eslint, tsc |
One command per layer, all gated in CI |
RECEIPTS has an unusual relationship with Codex: Codex built it, and Codex is its first audit subject.
- Built by Codex. The implementation was driven by Codex with GPT-5.6 during OpenAI Build Week, working module-by-module from a frozen engineering specification: the verification core, all eight session adapters, the sandbox and verifier catalog, the Evidence Graph, the Trust Score engine, the subsystems (Prompt Studio, Ask RECEIPTS, providers, dashboard, exports), the plugin SDK, and the React frontend. Codex made and recorded the key architectural decisions along the way — the LiteLLM gateway, the SQLite evidence graph, the digest-pinned Docker sandbox boundary, and the evidence-required adjudication policy (all logged as ADRs in DECISIONS.md).
- GPT-5.6 inside the product. The model policy is GPT-5.6 end to end: whole-session claim extraction defaults to
gpt-5.6-solat high reasoning effort — a single long-context pass over the entire transcript, which is what makes cross-timeline stale-success contradictions catchable at all — and per-claim planning, Ask RECEIPTS, and Prompt Studio run ongpt-5.6-terra, escalating only when the deterministic layer flags high complexity. - Codex as audit subject #1. The very first Tier-1 adapter parses Codex CLI
rollout-*.jsonltranscripts, and the golden fixture corpus is built from Codex-format sessions. The tool that Codex built is the tool that checks Codex's homework.
RECEIPTS treats every transcript and every repository it audits as hostile input, end to end:
- Sandboxed execution — every verifier command runs in Docker with
--network none, a non-root user, the repository mounted read-only, a writable tmpfs workdir, and CPU/memory/wall-clock limits. - Pinned image, not a tag — the sandbox image is referenced by immutable digest (
sha256:…); a mutable tag is rejected outright. - Allow-listed commands only — verifiers render commands from a fixed template catalog. No string taken from a transcript or a repository is ever interpolated into a shell.
- Instructions-as-data — text inside a transcript that looks like a command to RECEIPTS ("ignore previous instructions...") is cataloged as a claim, never executed.
- Secret redaction — API keys, tokens, and credential-shaped strings are stripped before anything reaches a model or an export, and never appear in logs.
- Provider isolation — LLM keys live in the OS keychain via
keyring; env-var keys are supported but never logged or rendered unmasked in the UI. - Typed error boundary — backend failures map to RFC-7807
problem+json; stack traces and secrets never reach the client.
Full threat-model notes: SECURITY.md.
Every number below is enforced by an automated gate, not asserted in prose:
| Check | Result |
|---|---|
| Backend tests | 114/114 passing |
| REFUTED precision (golden fixtures) | 1.000 — a single false refutation fails CI |
| REFUTED recall | 1.000 |
| Claim coverage | 1.000 |
Evidence-Graph trace(depth=6), 10k nodes |
well under the 150 ms budget |
| Cached dashboard aggregation | well under the 300 ms budget |
| SSE first byte | well under the 1 s budget |
| End-to-end offline audit | well under the 5-minute fixture budget |
| Frontend | eslint, tsc --noEmit, vitest all green |
| Dependency audit | pip-audit clean |
Run them yourself: make eval && make bench (needs GNU Make; on Windows, or without make, run the two steps directly: python eval/run_eval.py --fixtures fixtures/ --gate refuted-precision=1.0 then python benchmarks/run_benchmarks.py).
Every screenshot below shows real, freshly executed audits — not mockups. The verdicts, commands, exit codes, and diffs are the actual persisted evidence.
The claim vs. the evidence. The agent said "Fixed the race condition — all 2 tests pass." The verdict is REFUTED: pytest really does exit 0 with 2 passed, but the AST tamper scan caught that the diff removed test_concurrent_updates_do_not_reuse_a_sequence_number and its assertion. Two tests pass because one test is gone.
Stale success, caught by fresh execution. The agent claimed "All tests pass" — RECEIPTS re-ran the suite on the final repository state and got assert -1 == 5.
The evidence panel. Both execution records side by side: the green pytest run and the tamper scan that overrides it.
The Trust Score gate. One refuted claim caps the score at 3/10 no matter what else went well — and every dashboard card names the evidence node behind it.
You need: Python 3.12+, Node 22+, and Docker.
# 1. Get the code and install dependencies
git clone https://github.com/kanwa2006/RECEIPTS.git
cd RECEIPTS
python -m pip install -e ".[dev]"
npm --prefix web ci
npm --prefix web run build
# 2. Build the sandbox image, then pin its immutable ID into .env
docker build -t receipts-sandbox-runner:local -f docker/sandbox-runner/Dockerfile docker/sandbox-runner
cp .env.example .env
# Writes the built image's immutable digest into RECEIPTS_SANDBOX_IMAGE in
# .env. Identical in PowerShell and a POSIX shell — it shells out to Docker,
# not to a shell-specific command, so there's no manual copy-paste step.
python -c "import re, subprocess, pathlib; digest = subprocess.run(['docker', 'image', 'inspect', '--format', '{{.Id}}', 'receipts-sandbox-runner:local'], capture_output=True, text=True, check=True).stdout.strip(); env = pathlib.Path('.env'); env.write_text(re.sub(r'^RECEIPTS_SANDBOX_IMAGE=.*$', f'RECEIPTS_SANDBOX_IMAGE={digest}', env.read_text(), flags=re.M))"
# 3. (Recommended) Seed three real demo audits — no API key needed.
# This runs the full production pipeline (adapter → sandbox → verifiers →
# adjudication → Trust Score) against the bundled labeled fixtures.
python examples/seed_demo_audits.py
# 4. Run it
python -m uvicorn --env-file .env app.main:app --host 127.0.0.1 --port 8000Open http://127.0.0.1:8000. Prefer one command instead? make demo builds and starts the same stack in Docker Compose. make is GNU Make and isn't installed on Windows by default — the equivalent there is docker compose --profile sandbox up --build.
Windows note: the block above is copy-paste-safe in both PowerShell and a POSIX shell. If you're using PowerShell 7+ (
pwsh) rather than the default Windows PowerShell 5.1,&&chaining also works, so you can combine lines if you prefer.
Live claim extraction needs a provider key (e.g. OPENAI_API_KEY) in .env. Without one, every deterministic feature still works — RECEIPTS just says so honestly instead of faking a model-backed result.
A five-minute path built for judging:
- Open the app at
http://127.0.0.1:8000. The Intake panel discovers completed sessions from any supported agent CLI already on the machine — no upload step, no manual context pasting. - Load a session and start an audit. Watch the SSE progress stream through ingest → extract → verify → adjudicate → score in real time.
- Read the ledger. Every claim the agent made is listed with its verdict. Click a
REFUTEDclaim — the Evidence panel shows the exact command, exit code, and diff that contradicts it. - Check the Trust Score. Expand any signal — each one names its evidence nodes, and the score is reproducible: same audit, same score, every time.
- Ask RECEIPTS a question about the audit ("why was this claim refuted?", "what changed in the test suite?"). Watch it cite evidence nodes and refuse anything outside the selected session.
- Open Prompt Studio on any prompt from the session — see the deterministic analysis (token count, redundant instructions) land before any model call, and the mandatory diff on the rewrite.
- Export the report (Markdown/HTML/JSON) and note that the score and provenance travel with it — the export is not a fresh claim, it's the same evidence, portable.
For a fully offline, credential-free version of steps 2–4, run python examples/seed_demo_audits.py first — it audits the labeled fixtures under fixtures/ through the real sandbox and verifier pipeline, reproducing the honest/stale-success/test-removal scenarios from Real developer scenarios above with freshly executed, inspectable evidence.
Deliberately deferred — specified at the interface level so nothing blocks building them, but not shipped in V3:
- Tier-2 agent adapters (Cursor, Roo Code, Copilot-in-IDE, Aider) — the adapter interface is stable; these slot in without core changes.
- Behavior-probe verifier, currently shipped disabled — enabling it is gated on an eval proving zero false refutations, because it's the one verifier that could violate the evidence-required-refutation invariant.
- WASM plugin isolation (Extism) and third-party dashboard widgets — the current plugin permission model covers first-party-trusted extensibility; true sandboxed third-party plugins are a deliberate v2 of the plugin system.
- Desktop packaging (Tauri + PyInstaller) — the web-delivered baseline meets the current bar; a native shell is pure migration cost until offline/enterprise distribution is a committed requirement.
- PDF/DOCX/CSV exporters and a plugin install/enable UI — Markdown/HTML/JSON cover the audit-sharing need today; the exporter registry already supports adding these with no core change.
Everything above is a scoping decision recorded with its rationale in DECISIONS.md — not a gap that was missed.
| Neighbor | What it verifies | RECEIPTS' boundary |
|---|---|---|
| Runtime verification / auto-repair tools | Whether code works right now, and may fix it | Audits the agent's statements; never edits code or auto-repairs |
| Code review tooling | The diff | The transcript's claims about the diff, including narration that never touched a diff at all |
| Agent observability platforms | Session telemetry (tokens, latency, tool calls) | Adjudicates concrete claims against freshly executed evidence, not just logs what happened |
An auditor should stay independent of the agent it evaluates — RECEIPTS never repairs, merges, or edits; it only proves or disproves.
MIT. See LICENSE.