Live-write ingest path (spec 9.4): micro-batched single-writer flushes

[tenequm]

Context

Spec 9.4 defers live-write. This issue activates it with the smallest design that is correct on both local and S3 storage, based on the storage research from the 2026-06-10 design session.

The write problem on object storage is churn, not latency: commits run ~1.2s p50 on S3 Standard (lance-format/lance#6752 benchmarks), commit cost grows with fragment count (lance-format/lance#5194), and per-turn writes from live sessions would produce exactly that fragment/version/OCC churn. Ingest latency itself is not user-facing - no agent blocks on pond storing its transcript.

Design: micro-batched single-writer ingest

• One pond ingest process per data dir tails/receives live events and buffers them in memory.
• Flush one merge_insert batch per interval (~30-60s) or size threshold, whichever first.
• Single writer per data dir means no OCC contention from live-write itself; multi-host cron sync remains the existing OCC story.
• Identical behavior against local and object-store URLs - no architecture fork.

This is deliberately one tier below MemWAL. MemWAL solves many-independent-writers contention at read-path, ops, and maturity costs (experimental spec; correctness fixes for flushed-memtable search visibility only landed in lance v7.1 betas - pond is pinned to 7.0.0). Adoption trigger, recorded so it is not relitigated: adopt MemWAL only when (a) hosted multi-writer ingest exists - many independent processes committing to the same tables - and (b) pond is off the 7.0.0 pin onto a line where the 7.1 read-path fixes have stabilized.

Hygiene that must hold (mostly already true)

• Auto-cleanup stays gated (PR refactor(maintenance)!: gate compaction, drop unsafe vacuum, carve out [maintenance] config #37 / 002ddb3 lineage); cleanup spikes cost +1.2-1.5s every 20th commit when enabled (perf: disable auto-cleanup by default — hook adds multi-second latency every 20 commits and usually deletes nothing lance-format/lance#6728).
• Compaction scheduled, never inline with flushes.
• Dataset handles reused by the long-running process (reopen-per-write multiplies IOPs).

Unlocks

• Claude Agent SDK SessionStore backend (separate issue) - the SDK transcript_mirror stream is the natural input.
• Row-level host provenance (Track ingest host provenance on stored session rows #40) assumes live-write row granularity; the designs are compatible.