Every fact grounded in a verifiable, durably-archived source.
Source: NASA, "View of the Mission Control Center during Apollo 10 telecast," S69-34555, 19 May 1969. Public domain (US Government work).
factvault is a self-hostable research database where LLM hallucination at the retrieval layer is structurally impossible — not hoped-away. Every fact carries the verbatim excerpt, character offsets, URL, SHA-256 content hash, Wayback Machine snapshot, and ongoing verification status that produced it. An LLM that fabricates an excerpt during extraction gets rejected before the database INSERT, not discovered later in a post-mortem.
The whole project exists because of one non-negotiable commitment:
If the original URL dies tomorrow, the captured text, archived snapshot, and content hash remain authoritative forever.
1. Source Existence — the point of the project.
Every source captured with raw_text, archive_url, and SHA-256 content_hash. Periodic re-verification logged in an append-only table. Nothing deleted, overwritten, or tombstoned. Verbatim excerpt offsets verified deterministically before any statement touches the database.
2. Structured Facts — Wikidata-inspired statement model on Postgres.
Controlled property vocabulary (founded_in, ceo, raised_usd) prevents the silent-divergence failure mode where three synonymous slugs accumulate in an EAV database until nobody trusts any of them. Strict mode (default) queues unknown slugs for human review. Permissive mode available for rapid prototyping.
3. Cross-Source Corroboration — deterministic confidence from independent source count. One independent source: ceiling 0.50. Two: 0.85. Three or more: 0.95. Independence determined by publisher domain and trigram similarity — wire copy republished under a different masthead does not count twice. The LLM never sets confidence.
4. Story Assembly — one code path for all output.
bundle_assembler(entity_ids, depth) produces both pre-computed entity-keyed dossiers (depth=0) and on-demand query-keyed stories (depth=2–3). Every bundle carries full source-existence metadata on every fact.
git clone https://github.com/petersimmons1972/factvault
cd factvault
cp .env.example .env
# Host-run commands use localhost, not the in-compose service name.
# FACTVAULT_DATABASE_URL (app_user) is the runtime DSN for api/workers/mcp/init.
# FACTVAULT_MIGRATE_DATABASE_URL (superuser) is used only by `factvault migrate`
# (CREATE EXTENSION requires superuser). See docs/getting-started.md for the full contract.
export FACTVAULT_DATABASE_URL='postgres://app_user:dev_only_local_password@localhost:5432/factvault?sslmode=disable'
export FACTVAULT_MIGRATE_DATABASE_URL='postgres://factvault:factvault@localhost:5432/factvault?sslmode=disable'
export FACTVAULT_DEV_TENANT_ID='11111111-1111-1111-1111-111111111111'
make setupmake setup starts Postgres and the embedder, waits for readiness, builds the binary, runs migrations, and runs factvault init (keygen, health checks, example load). The init command is idempotent — re-running it will not overwrite existing keys or reload data.
The doctor command runs health checks — database reachability, RLS policies, Wayback API, embedding model, LLM endpoint, and a canary fact ingest end-to-end — and exits non-zero on any failure with a remediation command. Use doctor --required-only to exit 0 when only optional services (LLM, embedder, Wayback) are unavailable; the required checks (postgres, migrations, rls, canary) still gate the exit code.
postgres OK pgvector loaded
migrations OK schema version 6
rls OK cross-tenant row hidden
canary OK assembled 312 bytes
llm WARN connection refused (optional)
embedder OK dim=1024 norm=1.0000 model=BAAI/bge-m3
wayback OK https://web.archive.org/
If you prefer to run each step explicitly:
docker compose up -d postgres embedder
go build -o bin/factvault ./cmd/factvault
# Migrate runs as the Postgres superuser — CREATE EXTENSION requires superuser
# privileges. It reads FACTVAULT_MIGRATE_DATABASE_URL (exported above).
./bin/factvault migrate
# init (and everything after it) runs as app_user via FACTVAULT_DATABASE_URL —
# matches production, exercises the GRANTs. Do not pass password-bearing DSNs
# via --dsn: the flag rejects embedded passwords by design; use the env var.
./bin/factvault init --tenant "$FACTVAULT_DEV_TENANT_ID"
# Load a bundled example and assemble its first dossier.
./bin/factvault example load ai-startup-tracking \
--tenant "$FACTVAULT_DEV_TENANT_ID"
./bin/factvault worker dossier \
--tenant "$FACTVAULT_DEV_TENANT_ID" \
--limit 10For the full five-minute path from clone to a JWT-authenticated dossier query, see docs/getting-started.md. For day-two operations, see docs/operator-guide.md.
Testing guidance is documented in docs/testing.md.
A dossier answers "tell me about X" — pre-computed nightly for a registered entity, served from cache. A story answers "what's going on with this idea" — assembled on demand via recursive graph traversal, no entity pre-registration required.
Both share one assembler and one bundle JSON shape. See docs/superpowers/specs/2026-05-22-factvault-design.md §2 for three worked examples of each type.
| # | Stage | Key guarantee |
|---|---|---|
| 1 | Collect | Idempotent on (tenant_id, url). Raw HTML stored at INSERT; never re-fetched downstream. |
| 2 | Archive | raw_text extracted via Go stripHTML tag-stripper. Wayback SPN2 snapshot submitted. raw_html zlib-compressed. |
| 3 | Extract | Deterministic extractors run first. LLM runs on uncovered text only. Excerpt-offset check rejects hallucinations before INSERT. |
| 4 | Corroborate | Confidence recomputed from scratch on every run. Conflicts surfaced in v_conflicts, never silently resolved. |
| 5 | Verify | Daily CronJob. Append-only source_verifications log. No source or statement ever deleted. |
| 6 | Relate | relations table kept in sync with entity-valued statements. Embedding-near edges for graph traversal. |
Confidence is computed in internal/assembler/confidence.go. The formula is deterministic and auditable: independence is tested by publisher domain and trigram similarity, not guessed. No statement ever reaches 1.0 through the automated pipeline.
Every fact returned by GET /entities/{id}/dossier or POST /stories looks like this:
{
"property": { "slug": "acquired", "label": "Acquired" },
"value": { "entity": { "label": "Acme Corp" } },
"confidence": 0.85,
"sources": [
{
"url": "https://www.reuters.com/markets/deals/megacorp-acquires-acme-2025-11-14/",
"publisher": "reuters.com",
"content_hash": "a3f2c1d4e5b6a7f8...",
"archive_url": "https://web.archive.org/web/20251114183000/https://www.reuters.com/...",
"excerpt": "MegaCorp Inc. announced Tuesday it would acquire Acme Corp for $4.2 billion...",
"excerpt_offset_start": 1243,
"excerpt_offset_end": 1396,
"verification_status": "live"
}
]
}The excerpt_offset_start / excerpt_offset_end pair is not advisory metadata. It is a load-bearing guarantee: those character offsets are verified against raw_text before the row is written and re-verified on every daily verification run. If an LLM extractor fabricates an excerpt, the offsets will not match any text in the source body, and the statement is rejected.
REST API — standard Bearer JWT, RFC 9457 error responses:
# Dossier for a registered entity
GET /entities/{id}/dossier
# On-demand cross-entity story
POST /stories
{"query": "biotech CFO departures and SEC inquiries", "depth": 2, "max_facts": 500}
# Generate/list/get evidence briefs
POST /briefs/generate
GET /briefs
GET /briefs/{id}POST /briefs/generate now assembles the bundle server-side under the authenticated tenant. Client-supplied bundle JSON is rejected.
Example dossier brief payload:
{
"source_kind": "dossier",
"entity_id": "11111111-1111-1111-1111-111111111111"
}Illustrative story brief payload (adjust the query to your tenant data):
{
"source_kind": "story",
"query": "AI startup funding announcements"
}RSS ingestion worker:
./bin/factvault worker rss --feeds config/feeds.yaml --once
# loop mode (default): polls at configured interval/minimum interval
./bin/factvault worker rss --feeds config/feeds.yaml --interval 15mMCP server — works with Claude Desktop, Cursor, or any agent stack supporting MCP:
factvault__entity_lookup(entity_id="...", authorization="Bearer <jwt>")
factvault__story_query(query="acquisition chain narrative", depth=2, authorization="Bearer <jwt>")
factvault__fact_query(query="raised_usd", authorization="Bearer <jwt>")LLM backend is pluggable via OpenAI-compatible API. Default: Ollama at localhost:11434. Swap to any hosted provider with FACTVAULT_LLM_BASE_URL and FACTVAULT_LLM_API_KEY.
Before ingesting your first document, define your property vocabulary:
psql "$FACTVAULT_DATABASE_URL" <<'SQL'
INSERT INTO properties (id, tenant_id, slug, label, value_type) VALUES
(gen_random_uuid(), 'YOUR_TENANT_UUID', 'raised_usd', 'Raised (USD)', 'number'),
(gen_random_uuid(), 'YOUR_TENANT_UUID', 'ceo', 'Chief Executive Officer', 'entity_ref'),
(gen_random_uuid(), 'YOUR_TENANT_UUID', 'founded_in', 'Founded', 'date');
SQLThe controlled vocabulary is not bureaucracy — it prevents founded_in, founding year, and yearFounded from silently diverging into three properties that all mean the same thing. See docs/guides/defining-properties.md for the full authoring guide including examples for four domains.
| Document | What it covers |
|---|---|
| Design Spec | Complete architecture — all four pillars, all six pipeline stages, full DDL, bundle JSON shape, retrieval API, operational requirements |
| Source Existence | Why raw_text + archive_url + content_hash together; the verification lifecycle; what happens when URLs die |
| Facts and Sources | The statement model; how a fact differs from a source; what excerpt offsets mean and why they exist |
| Dossiers vs. Stories | Full treatment of both modes including all worked examples |
| Confidence and Corroboration | The deterministic confidence formula; what independence means; how to read v_conflicts |
| Defining Properties | The one mandatory authoring task before first ingest |
| 5-Minute Getting Started | Clone-to-dossier walkthrough for a fresh local operator |
| Operator Guide | Runtime components, configuration, health checks, backups, upgrades, and troubleshooting |
| Frontier Models | Explicit opt-in path and guardrails for hosted LLM extraction |
Active Go implementation. The repository now includes the Go CLI, migrations, workers, REST API, MCP server, doctor checks, example loader, Postgres store interfaces, and deploy scaffolding. Open issues continue to track the remaining backend and Tier 1 compose polish.
MIT.