Immutable Ledger for Agentic Systems

Proof chain and runtime trust infrastructure for agentic systems. Hash-chained entries prove what happened. Portable proof receipts prove a check ran so downstream services skip redundant guardrails. Cross-system verification correlates events across independent platforms without shared identity.

Guardrails get duplicated across enforcement points — AuthBridge runs PII scan, MCP Gateway runs it again, AI Gateway runs it a third time. The ledger solves this: centralized proof, decentralized enforcement, portable receipts between enforcement points.

What It Does

The ledger answers four questions for any agentic system:

1. What happened? Append-only event storage with per-source hash chains.
2. Can you prove it? SHA-256 hash chaining with cryptographic chain verification.
3. Did this check already run? Proof receipts travel with requests — downstream verifies the receipt instead of re-running the guardrail.
4. Can you correlate across systems? Query by agent ID, correlation ID, source, time range, or entry type across independent writers.

The Universal Contract

One gRPC call. Your identity. Your event format. Chained and verifiable.

// Core write + audit
rpc WriteEntry(WriteEntryRequest) returns (WriteEntryResponse);

// Proof receipts — runtime trust propagation
rpc IssueReceipt(WriteEntryRequest) returns (ProofReceipt);
rpc VerifyProof(VerifyProofRequest) returns (VerifyProofResponse);
rpc GetEntryByHash(GetEntryByHashRequest) returns (GetEntryResponse);

// Chain verification + query
rpc VerifyEntry(VerifyEntryRequest) returns (VerifyEntryResponse);
rpc VerifyChain(VerifyChainRequest) returns (VerifyChainResponse);
rpc QueryEntries(QueryEntriesRequest) returns (QueryEntriesResponse);

No shared identity system required. No event format standardization required. Each system keeps its own IDs and its own event schema. The ledger chains entries per entry_type, so each source maintains an independent, verifiable hash chain.

How It Works

• Append-only - database constraints enforce no UPDATE/DELETE on ledger entries. Startup verification confirms permissions.
• Per-type hash chains - each entry_type forms its own SHA-256 chain. Independent verification per source system.
• V2 canonical proof envelope - hashes commit to entry ID, metadata, content, idempotency key, chain position, timestamp, and previous hash using length-delimited fields.
• Concurrent-safe - connection pool (deadpool-postgres, configurable max size) with per-entry_type PostgreSQL advisory locks. Writes to different chains run in parallel. Integrity violations trigger exponential backoff retry (up to 10 attempts, configurable) with auto-recovery after 60 seconds.
• Idempotent - optional idempotency keys prevent duplicate entries on retry; reusing a key with different content or metadata returns a conflict.
• Cross-system queries - QueryEntries filters by agent_id, correlation_id, source_id, entry_type prefix, and time range. One query returns entries from all sources for the same agent or request.
• Hardened admin surface - /shutdownz is disabled unless ARE_LEDGER_SHUTDOWN_TOKEN is set and requires a bearer token when enabled. gRPC bearer-token auth can be enabled with ARE_LEDGER_API_TOKEN.
• Proof receipts - IssueReceipt writes an entry and returns a compact ProofReceipt (hash, type, position, timestamp). VerifyProof validates a receipt by hash without knowing the entry ID. Receipts travel as HTTP headers so downstream services verify a check ran without re-executing it.
• Writer signatures - optional writer_signature (opaque bytes) + signer_key_reference (key ID, SPIFFE SVID, DID). The ledger stores but doesn't verify — downstream checks the signature against the writer's public key. Proves WHO wrote the entry.
• Attestation reports - optional attestation_report (opaque bytes — SGX quote, SEV-SNP report, RATS EAT token). Proves WHERE the entry was written (verified runtime). Three layers of proof, all stored, none interpreted by the ledger.

Three Layers of Proof

Layer 1: entry_hash            → content wasn't modified (ledger verifies)
Layer 2: writer_signature      → who wrote it (verifier checks against public key)
         signer_key_reference  → which key to use
Layer 3: attestation_report    → where it was written (verifier checks against hardware root)

All optional. All backward compatible. All identity-neutral — the ledger stores opaque bytes for Ed25519, ECDSA, SPIFFE SVIDs, SGX quotes, or any format the writer uses.

Proof Receipts

Receipts solve the redundant-check problem in multi-hop agentic architectures. When AuthBridge runs a guardrail, it issues a receipt. The MCP Gateway verifies the receipt and skips the same guardrail. The MCP Server does the same.

AuthBridge runs guardrail
  → IssueReceipt(entry_type="guardrail.pii_scan", content={result: "clean"})
  → Gets ProofReceipt {entry_hash: "abc123...", chain_position: 42}
  → Attaches header: X-Proof-Receipt: base64({h:"abc123...", t:"guardrail.pii_scan"})
  → Forwards request

MCP Gateway receives request
  → Reads X-Proof-Receipt
  → VerifyProof(entry_hash="abc123...", entry_type="guardrail.pii_scan")
  → Response: {valid: true, agent_id: "authbridge", written_ts: ...}
  → Skips re-running the guardrail

Receipts are NOT credentials — they prove a check ran, they don't grant authority. The V2 hash commits to all entry fields (content, agent_id, correlation_id, entry_id, chain_position, timestamp, previous_hash). Changing any field breaks verification.

Performance

Benchmarked on Podman-hosted PostgreSQL 16 (single node, no tuning) using CPEX-shaped workloads via REST API (scripts/perf/cpex-latency-bench.py):

Scenario	p50	p99	Throughput	Errors
IssueReceipt (4 parallel chains, 100 req/s)	4.4ms	57.6ms	87/s	0
IssueReceipt (single chain, 100 req/s)	4.2ms	8.1ms	88/s	0
VerifyProof (under write load, 100 req/s)	2.8ms	6.6ms	—	0
Receipt round-trip (IssueReceipt + VerifyProof, 50 req/s)	8.0ms	13.0ms	—	0

Raw gRPC numbers (no REST overhead): WriteEntry p50=1.7ms p99=4.3ms ~520/sec, VerifyProof p50=0.6ms p99=1.7ms ~1,400/sec.

Known Scale Considerations

Concern	Current behavior	Mitigation path
Advisory lock contention	Writes to the same entry_type serialize via SHA-256-derived advisory locks. Single-chain: 88/s at 100 req/s with 0 errors.	Use distinct entry_type per source. Parallel chains scale linearly. Split hot chains by tool or instance.
Chain verification on long chains	VerifyChain walks chains in batches of 500 entries (bounded memory). Verification checkpoints not yet implemented.	Add checkpoints for long-chain skip-ahead. Streaming already prevents OOM.
Storage growth	Each entry stores full content bytes (up to 1 MiB). High-volume systems generate significant storage.	Content compression. Content-addressed storage (store hash, external blob). TTL-based archival.
gRPC message size	QueryEntries can return large result sets. Default 4MB gRPC limit hit at ~3K entries.	Pagination (already implemented). Client must page through results.

Quick Start

cd demo
make up        # Start ledger + postgres
make smoke     # Write sample entries and verify chains
make demo      # Full cross-system demo with OpenShell + Kagenti

Evidence & Metrics

The repository keeps the proof surface close to the code:

• tests/EVIDENCE_MATRIX.md summarizes automated, live, and not-yet-automated coverage.
• tests/evidence-results.json records the latest evidence runner output.
• tests/SECURITY_TESTING.md documents red-team and hardening checks.
• contracts/fleet-ecosystem-integration-contract.md defines the proof-only boundary and canonical API mapping for deepfield-fleet, governed-cognitive-loop, and fleet-llm-d integration.
• proof-explorer/proof.py verify --all independently verifies stored chains through the public API.

Current checked-in evidence shows 146/146 automated checks GREEN. The matrix still keeps design-level/manual items as YELLOW until those checks are automated in tests/run_evidence.py.

Useful local verification commands:

cargo test --all --locked
cargo clippy --all-targets --all-features --locked -- -D warnings
python tests/run_evidence.py
python proof-explorer/proof.py verify --all

The service exposes Prometheus metrics at /metrics on ARE_LEDGER_METRICS_PORT:

• are_ledger_write_total — write attempts by outcome (ok, invalid, error)
• are_ledger_write_duration_seconds — write latency histogram
• are_ledger_verify_duration_seconds — verification latency histogram
• are_ledger_chain_integrity_retries — retry attempts per write due to chain contention
• are_ledger_chain_verify_failure_total — chain verification detected invalid link or hash
• are_outbox_publish_failure_total — outbox HTTP publish failures

The standalone binary can deliver committed outbox events to an HTTP event sink. Set ARE_LEDGER_OUTBOX_HTTP_ENDPOINT to enable delivery, optionally set ARE_LEDGER_OUTBOX_HTTP_BEARER_TOKEN, and use ARE_LEDGER_OUTBOX_HTTP_TIMEOUT_SECONDS to override the 10-second request timeout. Each request contains the stored JSON payload plus Idempotency-Key (the outbox ID) and X-Ledger-Entry-ID headers. Successful 2xx responses mark the row DELIVERED; transport errors and non-2xx responses leave it PENDING for retry. Consumers must deduplicate by Idempotency-Key because delivery is at least once. When the endpoint is unset, publishing is disabled and rows remain PENDING.

Connection pool and chain recovery are configurable:

Variable	Default	Purpose
ARE_LEDGER_POOL_MAX_SIZE	16	PostgreSQL connection pool max size
ARE_LEDGER_CHAIN_MAX_RETRIES	10	Retry attempts per write before chain halt
ARE_LEDGER_CHAIN_HALT_RECOVERY_SECONDS	60	Auto-recovery timeout for halted chains

Hash compatibility note: this pre-release standalone ledger uses the V2 canonical proof envelope as its initial public contract. No production data has been written with the earlier experimental hash shape; if you have local demo data from before V2, reload it.

Security Notes

For shared deployments, put the gRPC listener behind TLS/mTLS-capable infrastructure and set ARE_LEDGER_API_TOKEN; clients can pass the token explicitly or through the same environment variable. Set ARE_LEDGER_SHUTDOWN_TOKEN only for controlled graceful-shutdown drills, and call /shutdownz with Authorization: Bearer <token>.

The optional Flask REST gateway is local-dev by default: it binds to 127.0.0.1, runs with debug disabled, and only allows localhost Vite origins unless GATEWAY_CORS_ORIGINS is set. For shared use, set GATEWAY_API_TOKEN, keep it behind TLS/auth-aware infrastructure, and only widen GATEWAY_HOST or CORS origins intentionally.

Demo: Cross-System Proof

The demo shows three independent systems writing to the same ledger without knowing about each other:

TIME          SOURCE      TYPE                         AGENT_ID         DETAIL
10:00:00.500  kagenti     kagenti.agent.deployed       spiffe://demo    image: model-agent:v3
10:00:00.800  openshell   openshell.sandbox.created    sbx-demo-001     policy: github-readonly
10:00:01.200  kagenti     kagenti.tool.call            spiffe://demo    tool: check-model  trace: aaa
10:00:01.205  openshell   openshell.http_activity      sbx-demo-001     GET api.github.com  trace: aaa
10:00:02.100  kagenti     kagenti.tool.call            spiffe://demo    tool: promote-model  trace: bbb
10:00:02.105  openshell   openshell.network_activity   sbx-demo-001     DENY POST  trace: bbb

Three identity systems. Three event formats. Three independent hash chains. One verifiable timeline.

# Cross-system query by trace ID
python proof-explorer/proof.py query --correlation-id trace-aaa
# Returns entries from both OpenShell and Kagenti for the same request

# Verify all chains
python proof-explorer/proof.py verify --all
# 3 chains verified, 0 tampered

# Detect authorization gaps
python proof-explorer/proof.py drift --agent-id agt-demo-001
# "POST api.github.com denied by OpenShell but no governance scope evaluation found"

Adapters

Thin bridges for existing agentic systems:

Adapter	Source System	Input Format	Entry Type Namespace
adapters/ocsf/	NVIDIA OpenShell	OCSF v1.7.0 JSONL	openshell.*
adapters/otel/	Kagenti / any OTEL system	OTLP JSON spans	kagenti.*
Direct gRPC	Any system	Any bytes	Your namespace

Architecture

System A ──→ adapter ──→ ┌─────────────────────┐
                         │  Immutable Ledger   │
System B ──→ adapter ──→ │  (gRPC :19292)      │ ←── proof-explorer CLI
                         │                     │
System C ──→ direct  ──→ │  PostgreSQL (chains)│
                         └─────────────────────┘

Each adapter is 100-150 lines of Python. Direct gRPC integration is ~30 lines. The ledger doesn't interpret event content — it chains raw bytes and makes them queryable by metadata.

Scaling Roadmap

1. Pool PostgreSQL connections. Done. deadpool-postgres (configurable via ARE_LEDGER_POOL_MAX_SIZE, default 16). SHA-256-derived 64-bit advisory lock keys replace SQL hashtext (was int4, risked false serialization). Measured: single-chain at 100 req/s went from 738 errors / 11 req/s to 0 errors / 88 req/s, p99 from 220ms to 8.1ms.
2. Measure the bottlenecks. Done. Prometheus histograms for write duration, verification duration, and chain integrity retries. CPEX-shaped latency bench at scripts/perf/cpex-latency-bench.py.
3. Bounded chain verification. Done. VerifyChain walks chains in batches of 500 entries (bounded memory). VerifyEntry fetches only the predecessor instead of the full chain. No risk of OOM on long chains.
4. Partition ledger storage. Partition ledger_entries by time, tenant, or chain namespace once volume grows, and keep indexes aligned to entry_type, agent_id, source_id, correlation_id, and written_ts queries.
5. Add verification checkpoints. Periodically persist signed/checkpointed chain tips or Merkle roots so long-chain verification can resume from known-good anchors instead of replaying from genesis every time.
6. Separate large payloads when needed. Keep small event content inline; for large payloads, store a content hash in the ledger and move raw bytes to object storage.
7. Define synthetic scale gates. Run local smoke, hot-chain stress, multi-chain stress, query/read stress, restart/recovery, and long-soak drills, then publish their outputs alongside the evidence matrix.

Project Structure

proto/                     The universal contract (9 RPCs)
src/                       Ledger server (Rust, gRPC, PostgreSQL)
migrations/                Database schema (append-only constraints + hash index)
contracts/                 Integration contracts (fleet ecosystem, CPEX/AuthBridge/Praxis)
sdks/python/               Python client SDK (WriteEntry, IssueReceipt, VerifyProof, GetEntryByHash)
adapters/ocsf/             OpenShell OCSF event bridge
adapters/otel/             Kagenti/OTEL span bridge
proof-explorer/            Query, verify, timeline, and drift CLI
api/                       REST gateway for frontend (Flask)
frontend/                  7-act narrative proof explorer (React + Vite + motion)
demo/                      Self-contained demo with compose (includes joint CPEX/AuthBridge scenarios)
scripts/perf/              Latency benchmarks (k6 + Python harness)
tests/                     Evidence matrix and 146 automated checks

Why This Exists

Every agentic platform logs events. None of them provide cross-system, cryptographically verifiable proof chains. The gap matters because:

• Compliance (EU AI Act August 2026, NIST AI RMF) requires auditable, tamper-evident decision records for autonomous systems.
• Cross-system correlation is impossible when OpenShell logs to JSONL, Kagenti logs to OTEL, and governance systems log to their own databases.
• Observability is not proof. Logs can be edited. Traces can be deleted. Hash-chained entries with independent verification are tamper-evident.

This ledger is the missing persistence and verification layer underneath protocol standards (MCP), runtime sandboxes (OpenShell), orchestration platforms (Kagenti), and per-framework governance (AGT).

Origin

Open-sourced as standalone neutral infrastructure for the agentic ecosystem.

License

Apache License 2.0. See LICENSE.