Add bencher_replica: in-process SQLite replication to replace Litestream

[epompeii]

Why

Litestream 0.5.13 blocked all Bencher Cloud API writes for ~5.5 minutes on 2026-07-10: when its verify step decides a full re-snapshot is needed, it copies the entire 4.35 GB database into a local LTX file while holding the SQLite write lock (via its _litestream_lock table), and no configuration knob reaches that code path. The same stall recurs on process restarts, and LTX compaction churns whole-database S3 transfers several times a day. Upstream 0.5.14 does not fix it.

bencher_replica is an in-process replacement. Being in-process is the structural fix: the app funnels writes through a single writer mutex, so the replicator coordinates checkpoints with the app's own write scheduling instead of fighting a separate process for locks. The prime invariant (I5): the SQLite write lock is only ever held for O(WAL-tail) work, never O(database). All six governing invariants are documented in plus/bencher_replica/src/lib.rs.

What

• WAL parser with full salt and cumulative checksum-chain verification, cross-validated in both directions against SQLite itself
• Storage: local filesystem XOR S3-compatible (endpoint override for R2/MinIO) behind one contract, plus a scripted fault-injection wrapper for tests
• Sync engine: step-driven core; checkpoints run PASSIVE while the replicator holds BEGIN IMMEDIATE, closing the ship-vs-checkpoint race without ever needing blocking RESTART/TRUNCATE checkpoints
• Snapshots: generation-based, sourced from a single-step SQLite online backup into a scratch file (transactionally consistent under concurrent checkpoints), throttled zstd multipart upload, snapshot.json as the atomic commit marker with a mandatory-replay boundary offset
• Restore: latest-only, in the same startup handshake slot Litestream used; every epoch WAL is chain-pre-validated before application and checkpoint consumption is verified frame-for-frame
• Verification: restore-and-compare at a pinned position (default daily, 0 disables); the backstop for externally-caused divergence
• Shadow mode: with both plus.litestream and plus.replica configured, Litestream keeps checkpoint ownership and restore precedence during the burn-in
• Fix: standalone sweep connections (stats, credit grants) now disable wal_autocheckpoint when replication is configured; previously the credit sweep could checkpoint and restart the WAL behind Litestream's back
• Observability: Replica* otel counters and a critical-section histogram; JsonLitestream.metrics_port plus [[metrics]] in the Fly configs so litestream_* Prometheus metrics are scraped during the shadow period

Testing

• 275 crate tests: WAL parser fixtures (real and synthetic, both checksum byte orders, golden cross-version canary), three-backend storage contract suite, 8 fault-injection scenarios, 6 crash-recovery kill points, 8 seeded 200-op randomized equivalence workloads, ignored soak and live-S3 tiers
• 4 server-level integration tests: real API writes replicate and restore to a logically equivalent database; reboot-with-missing-DB auto-restores; shadow coexistence
• An adversarial multi-agent review confirmed 18 findings; all are fixed with regression tests. Highlights: salt-match resume now proves CONTENT against the replica tip (post-power-loss rewind forks), the meta-verified resume path requires salt1 continuity (buried WAL cycles), restore hard-fails instead of soft-stopping when local errors could leave torn state, and shutdown checkpoints after the final ship so SQLite's checkpoint-on-close WAL deletion does not force a full snapshot on every deploy
• Full gates: cargo nextest run --all-features (2047 passed), cargo test --doc, cargo clippy --no-deps --all-targets --all-features -- -Dwarnings, cargo check --no-default-features, cargo gen-types

Deployment plan

1. Shadow burn-in: add plus.replica (S3 target, its own bucket/prefix) alongside the existing plus.litestream config; watch replica.verify.pass and replica.divergence metrics
2. Run the ignored live-S3 tier against a real bucket before the shadow deploy: cargo nextest run -p bencher_replica --features plus,testing --run-ignored ignored-only with BENCHER_REPLICA_TEST_S3_* env vars
3. Cutover: remove the plus.litestream section (forces one clean generation); the Litestream removal checklist (binary, Dockerfile stage, config types) follows after the burn-in

Notes for review

• One new lint suppression needs sign-off per CLAUDE.md: #[expect(clippy::too_many_lines)] on ApiCounter::description() in bencher_otel (matches existing precedent for exhaustive metadata matches)
• cargo deny check fails on two PRE-EXISTING advisories via bencher_plot's tree (ttf-parser unmaintained, crossbeam-epoch RUSTSEC-2026-0204), unrelated to this change
• Documented residual limitation: an external writer that buries a WAL cycle AND deletes the WAL file while the server is down is undetectable at resume; the daily verification is the backstop

[github-actions]

🤖 Claude Code Review

PR: #929
Base: devel
Head: u/ep/bencher-replica
Commit: d5070f6f47c6105b46e44e10150b35769dbc8ff8

---

All four subsystem agents plus my own integration pass are complete. Here's the consolidated review.

---

PR Review: bencher_replica (in-process SQLite replication replacing Litestream)

Overall

This is an exceptionally high-quality, heavily-tested change (~15k lines of new crate + ~1k of tests, golden WAL fixtures, cross-validation against SQLite's own recovery in both byte orders, fault/crash/restore matrices). Across the WAL/checksum core, S3/storage/restore state machine, config/checkpoint safety, and API startup/shutdown wiring I found no high-severity correctness bugs, no data-loss hazards, and no secret-exposure defects. CLAUDE.md standards are met throughout: no anyhow/Box<dyn Error>, thiserror variants wrap original error types (not String), struct destructuring for exhaustiveness, OTel counters for adverse events, no emdashes, Dockerfile stubs and CI path filters correct, and cargo gen-types was run.

Verified load-bearing logic is correct: WAL checksum fold and endianness match SQLite exactly; salt/torn-tail/pgno-0 handling stops the trusted prefix safely; checkpoints are PASSIVE-only under both the writer mutex and BEGIN IMMEDIATE with a ship-before-checkpoint re-scan; restore ordering, the mandatory-replay wal_boundary guard, and zip-bomb/truncated-listing defenses all hold; path traversal is blocked via validate_key/validate_prefix; no panics are reachable from untrusted WAL/remote bytes (all unwrap/expect are test-only).

Security is improved, not regressed

GET /v0/server/config now returns .sanitized() instead of the raw config, masking security/DB/SMTP/OAuth/Litestream/replica secrets unconditionally in both debug and release. New Secret typing + Sanitize impls cover the replica S3 credentials, with dedicated tests. Good.

Findings (all low / medium — points to confirm, not blockers)

1. MEDIUM — Fly kill_timeout = "6s" is tight against the new in-process shutdown budget. services/api/fly/fly.toml:4 is unchanged, but shutdown is now fully in-process: connection drain, then final WAL ship (shutdown_sync_timeout default 4s), then an unbounded epoch-sealing checkpoint. Drain + 4s can reach 6s before the checkpoint runs, so the fast-resume optimization will often be SIGKILLed. The 4s default is deliberately documented to fit inside 6s and this is lag-not-loss (crash recovery covers it), but worth confirming 6s is intentional.

2. LOW-MEDIUM — Only SIGINT is handled; SIGTERM bypasses the final replica sync. main.rs races on tokio::signal::ctrl_c() with no SignalKind::terminate handler. Fly sends SIGINT (fine for prod), but self-hosted docker stop/Kubernetes send SIGTERM and exit without the final WAL ship. This matters more now than with external Litestream, since the tail-ship depends entirely on the graceful path. Still lag not loss (WAL persists locally).

3. LOW — In sole mode a fatal replica error takes down a healthy server. main.rs replica_fatal wins the race() when !shadow, crashing an otherwise-healthy API on a backup-subsystem failure. Consistent with existing Litestream JoinHandle behavior and explicitly exempted in shadow mode; flagging as a durability-vs-availability decision to confirm.

4. LOW — Symlinked DB file (not directory) diverges restore vs replicator paths. replica_db_path canonicalizes the parent when the file is missing (restore) but the full path once it exists (start_replica). These agree for a symlinked data directory (the documented case) but resolve differently if the DB file itself is a symlink. Unusual config, worth a one-line doc note.

5. LOW — No TLS warning on plaintext http:// S3 endpoints. s3.rs accepts http:// endpoints silently; credentials/bytes could transit in plaintext on operator misconfig. A warn! on non-https would be cheap. (Note: replicator.rs already has such a warning, but it only matches a literal 7-byte http:// prefix and misses scheme-less endpoints.)

6. LOW — normalize_prefix doesn't reject ./... Unlike validate_key/validate_prefix, s3.rs normalize_prefix only strips empty components, so a configured prefix like a/../b is preserved. Harmless for S3 (opaque keys, used symmetrically) but diverges from the rigor applied everywhere else.

7. LOW — Raw String where strong types exist. SnapshotMeta.created (has bencher_json::DateTime), SnapshotMeta.sha256, and ReplicaMeta.generation (has GenerationId) are raw strings. Defensible as serde wire types (and generation-as-String tolerates unparsable values for forced-new-generation detection), but it's the raw-primitive-in-model pattern CLAUDE.md steers away from.

Nits (optional)

• SyncError::WalIo interpolates its io::Error in the Display string but omits #[source], unlike WalError::Read — slightly poorer error-chain reporting.
• pub fn wal_checksum (wal.rs) assert!s on non-multiple-of-8 length; safe for all in-crate callers, but a panic-on-misuse contract for a pub fn (consider non-pub or debug_assert! + early return).
• #[expect(clippy::too_many_lines)] in api_meter.rs description() is outside a test module; CLAUDE.md asks for explicit approval before suppressing a lint outside tests (it's an exhaustive one-line-per-counter match, so likely fine to approve).

Recommendation

Approve pending confirmation of the two MEDIUM/LOW-MEDIUM operational items (#1 Fly kill_timeout headroom and #2 SIGTERM handling for self-hosted). The remaining items are polish and can be follow-ups. The shadow-mode burn-in design (Litestream stays authoritative until cutover) is a sound rollout strategy that de-risks the migration.

---

_{Model: claude-opus-4-8}

[github-actions]

Bencher Report

Project	Bencher
Branch	u/ep/bencher-replica
Testbed	intel-v1

🚨 2 Alerts

Benchmark	Measure Units	View	Benchmark Result (Result Δ%)	Upper Boundary (Limit %)
Adapter::Rust	Latency microseconds (µs)	📈 plot 🚷 threshold 🚨 alert (🔔)	3.89 µs (+10.89%) Baseline: 3.51 µs	3.71 µs (104.87%)
Adapter::RustBench	Latency microseconds (µs)	📈 plot 🚷 threshold 🚨 alert (🔔)	3.89 µs (+10.93%) Baseline: 3.50 µs	3.71 µs (104.86%)

Click to view all benchmark results

Benchmark	Latency	Benchmark Result microseconds (µs) (Result Δ%)	Upper Boundary microseconds (µs) (Limit %)
Adapter::Json	📈 view plot 🚷 view threshold	4.87 µs (+4.48%) Baseline: 4.66 µs	4.92 µs (98.98%)
Adapter::Magic (JSON)	📈 view plot 🚷 view threshold	4.72 µs (+4.13%) Baseline: 4.53 µs	4.73 µs (99.74%)
Adapter::Magic (Rust)	📈 view plot 🚷 view threshold	26.39 µs (+2.73%) Baseline: 25.69 µs	26.72 µs (98.76%)
Adapter::Rust	📈 view plot 🚷 view threshold 🚨 view alert (🔔)	3.89 µs (+10.89%) Baseline: 3.51 µs	3.71 µs (104.87%)
Adapter::RustBench	📈 view plot 🚷 view threshold 🚨 view alert (🔔)	3.89 µs (+10.93%) Baseline: 3.50 µs	3.71 µs (104.86%)

🐰 View full continuous benchmarking report in Bencher