Support for dictionary in ZSTD compression

[Eugene Vignanker (brain-eugenevignanker)]

Summary

Adds zstd doc-store dictionary support. The dictionary is a recorded, self-describing property of the index:

• ZstdCompressor gains dictionary: Option<ZstdDictionaryDescriptor>, persisted in meta.json's index_settings.docstore_compression (e.g. zstd(dictionary_path=dict.bin.zst)).
• ZstdDictionaryDescriptor holds a single field, path — a Directory-relative path to a sibling file holding the dictionary's bytes (zstd-compressed on disk). No content hash. A dictionary can be multiple megabytes; hashing it on every doc store open (every segment, every reader reload) is real, avoidable cost with no enforcement value if the hash can't reliably be checked cheaply.
• Dictionary resolution is entirely tantivy-side now: Compressor::resolve_dictionary(&dyn Directory) reads the path via the existing Directory::atomic_read, decompresses it, and returns Arc<[u8]>. Callers (SegmentReader, SegmentSerializer/segment_writer's resort path) call this themselves from settings().docstore_compression — no new Directory trait method, no capability plumbing.
• Integrity/corruption detection relies on zstd's own frame checksum (CParameter::ChecksumFlag), enabled only when compressing with a dictionary and verified automatically by the decompressor on every block read — a few bytes + a sub-microsecond XXH64 pass per block, not proportional to dictionary size. A missing/mismatched dictionary now fails loudly at first decompress, not silently.
• StoreWriter/StoreReader/Compressor/Decompressor take an explicit dictionary: Option<&[u8]> (or Option<Arc<[u8]>>) parameter through the compress/decompress path.
• New compress_whole/decompress_whole helpers (whole-buffer zstd::stream::encode_all/decode_all) for the dictionary file itself — distinct from the existing per-block length-prefixed format, since the dictionary file has no skip-index seeking into it.
• Doc store footer format is unchanged (no version bump, no reserved-byte usage) — the redesign has no on-disk footer impact.
• Added tests: dictionary compress/decompress round-trip, dictionary path serde round-trip (including combined with compression_level), dictionary-mismatch/missing detection at decompress time, and confirming a store with no dictionary never resolves one.

Why

Different datasets need different dictionaries, and a reader has no way to detect a missing/mismatched dictionary at read time otherwise — a silent-corruption risk. This makes the dictionary a first-class, self-describing part of the index, with the embedding application (Brainstore) only ever needing to supply it at index creation — every later open derives it purely from that index's own meta.json.

Public API changes

IndexBuilder::new/open_or_create and Index::open — the entry points Brainstore actually calls to create/open an index — are untouched (zero diff vs base in index/index_builder.rs, index/index.rs). Every change below is one level down, inside tantivy::store:

• New public type ZstdDictionaryDescriptor { pub path: String }; ZstdCompressor gains a new field dictionary: Option<ZstdDictionaryDescriptor> (#[serde(default)], backward-compatible). Its zstd(...) meta.json string gains a matching dictionary_path=<path> option.
• New methods Compressor::resolve_dictionary(&self, directory: &dyn Directory) -> io::Result<Option<Arc<[u8]>>> / Compressor::has_dictionary(&self) -> bool, and new functions compress_whole/decompress_whole.
• StoreWriter::new, StoreReader::open, Compressor::compress_into, Decompressor::decompress/decompress_into each gain one new dictionary parameter.
• Compressor/ZstdCompressor no longer derive Copy (still Clone), since the new dictionary/path field isn't Copy.
• No changes to the Directory trait, ManagedDirectory, or the doc store footer format/version.

Defense in depth

Merge stacking (indexer/merger.rs) copies compressed blocks byte-for-byte and never decompresses, so it can't exercise the dictionary's zstd checksum. If the fixed-for-index-life dictionary invariant is ever violated externally (bug, race, manual meta.json edit) while the compressor family stays the same, stacking is now explicitly disabled whenever the output compressor has a dictionary (Compressor::has_dictionary), forcing the decompress/recompress path instead — so a violation fails loudly at merge time rather than silently corrupting the merged segment. Covered by a new regression test that rotates the dictionary path between commits before merging.

Not in scope / follow-ups

• No cross-open in-process caching of dictionary bytes yet — every StoreReader/SegmentReader construction re-reads and re-decompresses the dictionary file. Left as a separate future concern.
• No dictionary rotation: a dictionary is fixed for the life of an index once created (enforced on the Brainstore side, not in this fork).