Add durable_sessions(last_seen_at_ms DESC, session_key) index to fix hydration temp B-tree spill (#168)

[ekalb81]

Fixes the mechanism described in #168: stream_sessions's snapshot walk (ORDER BY d.last_seen_at_ms DESC, d.session_key) had no supporting index, so SQLite built a temp B-tree to satisfy the sort. This connection never sets PRAGMA temp_store (unlike scan_cache.rs's connection), so the sorter spilled to disk-backed temp pages and peak private memory scaled with spilled PMA count rather than session_json payload — the issue measured 2,952 MB against the real 3.5 GB / 4,826-session corpus, versus 36 MB with the ORDER BY removed.

The change

Ledger schema migration v8 → v9 (SQL-only, per AGENTS.md): CREATE INDEX IF NOT EXISTS durable_sessions_last_seen_idx ON durable_sessions(last_seen_at_ms DESC, session_key). No backfill needed — an index has no historical data to reconcile, unlike every prior migration step in this file.

Verification

• stream_sessions_query_plan_has_no_temp_btree_for_order_by asserts on EXPLAIN QUERY PLAN's actual output (not just the index's existence — an index that exists but isn't chosen is the real failure mode and looks identical to success from the outside). It also includes a self-check: with the index dropped, the identical query DOES regress to a temp B-tree, proving the assertion isn't vacuous.
• stream_sessions_row_order_is_unchanged_by_the_last_seen_index proves the real StoredSession key sequence from stream_sessions is identical with and without the index, including under last_seen_at_ms ties broken by session_key — the ordering the doc comment calls load-bearing for surfacing a missing-snapshot invariant violation per-key.
• probe_index_migration_one_time_cost_on_realistic_corpus (ignored perf probe) times the migration's CREATE INDEX IF NOT EXISTS against a corpus shaped like the real one (4,826 sessions, ~3 GB db file): 3.74 ms. The index only touches the narrow durable_sessions table, never the multi-GB session_snapshots blob table, so the one-time cost on existing installs is negligible — not the "real time" concern the task raised.
• Existing migration invariants (schema_fingerprint_matches_committed_expected_value, fresh_and_fully_migrated_databases_produce_identical_schema_shapes, migration_from_every_prior_version_reaches_schema_version_with_no_gap_or_duplicate, migration_step_count_matches_steps_migrate_actually_runs, interrupting_migration_between_steps_resumes_cleanly_at_the_right_version) all updated and passing.

EXPLAIN QUERY PLAN, before and after (literal SQLite output against the real schema, empty tables so the planner's choice is schema-driven, not row-count-driven)

Before:

QUERY PLAN
|--SCAN d USING INDEX sqlite_autoindex_durable_sessions_1
|--SEARCH s USING INDEX sqlite_autoindex_session_snapshots_1 (session_key=? AND version=?) LEFT-JOIN
`--USE TEMP B-TREE FOR ORDER BY

After:

QUERY PLAN
|--SCAN d USING INDEX durable_sessions_last_seen_idx
`--SEARCH s USING INDEX sqlite_autoindex_session_snapshots_1 (session_key=? AND version=?) LEFT-JOIN

The temp B-tree step disappears entirely; d is scanned directly in final order via the new index.

The win, measured

src-tauri/examples/hydration_memory.rs, 2,500-session / 1.13 GB synthetic corpus (same PRNG seed both runs, so byte-identical corpora), release build, resident_only pass (the first pass, so its peak is attributable to stream_sessions alone with no carryover from a later pass):

	before (shipped)	after (indexed)
peak RSS (baseline → after_hydration)	116.6 MB → 403.3 MB (+286.7 MB)	117.1 MB → 117.1 MB (+0 MB)
overall peak RSS across all 3 passes	406.3 MB	117.1 MB
stream_sessions wall time (this pass)	8.568 s	3.792 s (2.3×)

Rust heap deltas (heap_bytes) are byte-identical between the two runs (e.g. +12,395,531 / +12,394,475 for resident_only), confirming the corpora and workload are identical and only the SQLite-side memory changed. private_bytes, sampled only immediately before and after each pass, doesn't show this delta — SQLite releases the temp B-tree's pages once the statement is fully drained, before the "after" sample is taken, matching the issue's own "released incrementally" observation. peak_rss (Windows' tracked high-water mark, monotonic since process start) is what actually captures the transient spike, and it does.

At the default 1,200-session / 603 MB corpus the same pattern holds at smaller scale: peak RSS 111.0 → 267.4 MB before, essentially flat after.

The mechanism the issue described reproduces cleanly in this harness and the fix eliminates it — this is not a stretched result.

Scope check

Grepped the whole crate for the same last_seen_at_ms ordering. stream_sessions is the only production usage of ORDER BY d.last_seen_at_ms DESC, d.session_key (or any last_seen_at_ms ordering at all — the column touches no other query). The only other occurrence in the file is an old_style_load_sessions test helper (a pre-existing before/after allocation-shape comparison, unrelated to this fix) that queries the same order for oracle purposes. No other query benefits from or regresses against the new index; ordinary session writes now maintain one additional narrow index alongside the existing two (durable_sessions_identity_idx, durable_sessions_project_idx), which is expected, unavoidable overhead for this fix and not something this PR tries to avoid.

Anything the spec didn't anticipate

• The one-time migration cost concern (CREATE INDEX IF NOT EXISTS on an existing multi-GB table "takes real time") turned out to be a non-issue: durable_sessions itself is a narrow table (no blob columns) even when the database file is multiple GB, because all the size lives in the separate session_snapshots.session_json blob column the index never touches. 3.74 ms, not something worth budgeting for.
• private_bytes's instantaneous before/after sampling in the harness doesn't show the SQLite-side spike at all (see above) — only peak_rss does. Worth knowing if this harness is reused for a future SQLite-memory investigation.

Validation

All six AGENTS.md commands pass: npm run check, npm test (174 passed), npm run build, cargo fmt --check, cargo clippy -D warnings, cargo test --locked (372 passed, all crates).

Not merging — opening for review per the task.

🤖 Generated with Claude Code

[codecov]

Codecov Report

❌ Patch coverage is 74.65753% with 37 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
src-tauri/src/history_store.rs	74.65%	37 Missing ⚠️

📢 Thoughts on this report? Let us know!