feat(loader): offline batch loader + DuckDB→DuckDB transfer (carry embeddings, no re-embed)

[jrosskopf]

Summary

Offline batch loader + DuckDB→DuckDB transfer that loads a large corpus (e.g.
~20k PDFs) without hitting the live server's per-tenant Embeds quota or paying
to re-embed in production. The expensive step (extract → chunk → embed)
happens once, offline, in a throwaway loader instance; the result transfers
into the live tenant carrying the embeddings as data — prod never re-embeds.

Design note: docs/notes/2026-06-23-batch-loader.md.
Runbook: docs/operations.md → "Bulk-loading a large corpus".

escurel-loader build    --src <dir> --out <loader_dir> --skill <name> --embedder hash
escurel-loader transfer --from <loader_dir> --to <live_data_dir> \
                        --tenant <t> --expect-model hash --on-collision skip

What's in it (7 logical changes, one branch)

1. Embedder::model_id() — pins the embedding space (Zero/Hash/Candle/Gemini/Reloadable).
2. Split materialize_document into embed → write_document_blocks(page_id, overlay, chunks, vectors) — lets the transfer supply precomputed vectors (the no-re-embed path).
3. escurel-loader crate — LoaderBuilder runs the real extract→chunk→embed→materialise pipeline in-process against a standalone DuckDB; writes manifest.json (model_id/dim/schema_version). Adds Indexer::reindex_vectors().
4. Indexer::merge_from_attached(alias, on_collision) — the DuckDB→DuckDB transfer: drop HNSW → INSERT … BY NAME SELECT (skip/replace/error) → recreate HNSW → refresh FTS. Vectors copy verbatim.
5. copy_files — blobs + overlay markdown LaneStore→LaneStore, files-first (crash-safe ordering).
6. transfer fn + escurel-loader operator binary — validates the manifest against the live tenant's --expect-model before touching anything; fails closed on mismatch.
7. Docs — design note, ops runbook, discovered note.

Notable decisions / findings

• The binary lives in escurel-loader, not escurel-cli: the transfer must ATTACH two DuckDB files on disk, which can't go through the gateway HTTP client surface that escurel-cli is built on. escurel-cli stays a pure presentation layer. (Deviation from the plan's "escurel-cli/src/admin.rs" — host-side disk ops are a hard mismatch for the all-HTTP CLI.)
• Bug fixed (caught by the idempotent re-transfer test): a transfer into an already-migrated tenant re-ran Migrator::up → "Table pages already exists". Now gates up on fresh = !db_path.exists(), mirroring the server boot recipe.
• Gemini Batch-API embedder (~half price) is deferred — --embedder already abstracts the space and the manifest gate makes adding gemini-batch:<model> safe later.

Test plan

All no-mock E2E (real DuckDB files, real FsStore, real ATTACH, real extraction):

• crates/escurel-index/tests/write_document_blocks.rs — supplied vectors stored byte-equal; count-mismatch fails closed.
• crates/escurel-index/tests/merge_from_attached.rs::{merge_skip_imports_new_pages_keeps_existing_and_copies_vectors_verbatim, merge_replace_overwrites_colliding_pages, merge_error_aborts_on_collision} — target uses a ZeroEmbedder so the byte-identical imported vector is the no-re-embed proof.
• crates/escurel-loader/tests/build.rs — loader build materialises docs with non-NULL vectors + manifest.
• crates/escurel-loader/tests/copy_files.rs — blobs byte-equal + overlays re-keyed + idempotent.
• crates/escurel-loader/tests/transfer.rs::{transfer_carries_embeddings_verbatim_and_is_idempotent, transfer_aborts_on_embedder_model_mismatch} — capstone: build (HashEmbedder, non-zero vectors) → transfer → live dense_vec byte-identical to loader's, instances search-hittable, re-transfer idempotent, model-mismatch aborts before any rows move.
• Plus model_id unit tests across escurel-embed.

Four-check gate green locally: cargo fmt --check, cargo clippy --workspace --all-targets -- -D warnings, cargo test --workspace --all-targets, cargo build --workspace --release.

🤖 Generated with Claude Code