diff --git a/CHANGELOG.md b/CHANGELOG.md index bfa5ce55..7ff112e0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,15 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +### Documentation + +- **README pre-release lifts:** a caveated operating-point table from the + committed synthetic benchmark; the Bitmap prefilter's hypergeometric null; + a "Security: index-file trust" callout; a "Research collaboration" (paper + co-authorship) section; a "Scope" section; `RankQuantFastscan` (doc-hidden) + and the `experimental` `MultiBucketBitmap` surfaced with caveats; in-progress + Python bindings noted; and a more precise MSRV rationale. + ### Added - **CI fuzz smoke** (`.github/workflows/fuzz.yml`): a bounded cargo-fuzz run on diff --git a/README.md b/README.md index f823ca1a..a46926f9 100644 --- a/README.md +++ b/README.md @@ -41,10 +41,12 @@ vector on its own: prefilter feeds an exact rerank — the coarse→fine pipeline ships as library primitives. -ordvec is a compressed **flat-scan** substrate (optionally two-stage), not -a navigable-graph or billion-scale ANN index: it pairs small codes with -fast runtime-dispatched SIMD (AVX-512/AVX2, NEON, wasm128) rather than -graph traversal. +ordvec is a compressed **flat-scan** substrate (optionally two-stage): small +codes scored by fast SIMD — AVX-512/AVX2 runtime-dispatched on x86_64, baseline +NEON on aarch64, and `simd128` on wasm32. It +is the code-and-scan layer, not a navigable-graph index — but the codes are +small and index-agnostic, so they compose *under* an ANN or sharding layer for +large-scale serving rather than competing with one. ## Ordinal index family @@ -57,6 +59,33 @@ graph traversal. - **`SignBitmap`** — a sign bitmap per document for sign-cosine candidate generation, feeding an exact rerank stage. +Two further paths, for callers who need them: + +- **`RankQuantFastscan`** *(`#[doc(hidden)]` — reachable as + `ordvec::RankQuantFastscan`, but the API is not yet stable)* — an optional + b=2 FastScan kernel (block-32 PQ-LUT) for absolute-minimum scan latency, at + 2× the RankQuant b=2 footprint (`dim/2` bytes/doc). Surfaced here so + latency-critical callers know it exists. +- **`MultiBucketBitmap`** *(behind `--features experimental`)* — the + multi-bucket bilinear-overlap probe behind the research-side decomposition; + a scaffold for the theory, not a production path. + +## The bitmap prefilter has a closed-form null + +The `Bitmap` prefilter scores candidates by `popcount(Q AND D)` over each +document's fixed-size top-bucket set. Two *unrelated* documents — modelled as +independent uniform top-bucket sets — overlap **hypergeometrically**, +`H(dim, n_top, n_top)`, with expected overlap `n_top² / dim` (e.g. 16 at +`dim = 256`, `n_top = 64`). So the filter's **selectivity** — how often an +unrelated document clears a given overlap threshold — is closed-form and +data-independent, not a tuned cutoff. (Whether *true* neighbours clear the bar +is empirical; this is an exact candidate-generation null, not a +retrieval-optimality theorem.) The invariance underneath it — that the rank +transform is unchanged by any strictly monotone reparametrisation of the +coordinates — is separately machine-checked in Lean, with the formalisation +accompanying the paper. Details in +[`docs/RANK_MODES.md`](docs/RANK_MODES.md). + ## Quickstart The crate is being prepared for its first crates.io release. Until then, @@ -87,27 +116,81 @@ For the sub-linear two-stage path (`Bitmap` / `SignBitmap` candidate generation → `RankQuant` rerank) and the full mode comparison, see [`docs/RANK_MODES.md`](docs/RANK_MODES.md). +### Python + +PyO3/maturin bindings are **in progress** — the same `Rank` / `RankQuant` / +`Bitmap` / `SignBitmap` API from Python, shipping to PyPI as `ordvec` (the +coordinated PyPI release is pending). Until then, build from source with +`maturin develop` in [`ordvec-python/`](ordvec-python/). + ## Documentation - **Design deep-dive & reproducible benchmark tables:** [`docs/RANK_MODES.md`](docs/RANK_MODES.md) +- **Design alternatives evaluated and cut:** + [`docs/ALTERNATIVES_CONSIDERED.md`](docs/ALTERNATIVES_CONSIDERED.md) +- **Index-file trust model:** + [`docs/INDEX_PROVENANCE.md`](docs/INDEX_PROVENANCE.md), + [`THREAT_MODEL.md`](THREAT_MODEL.md) - **API docs:** *(available after the first crates.io release)* -- **Paper (OrdVec / RankQuant):** _link TBD. Collaborators welcome (see - [Contributing](#contributing))._ +- **Paper (OrdVec / RankQuant):** _link TBD — see + [Research collaboration](#research-collaboration)._ ## Reproducible benchmark -The head-to-head benchmark generates a seeded synthetic corpus -in-process, so the quality numbers (R@10, candidate-recall, bytes/vec) -are regenerable from a clean checkout with no external corpus file: +The head-to-head benchmark generates a seeded synthetic corpus in-process, so +the **quality numbers (R@10, candidate-recall, bytes/vec) are deterministic** +and regenerable from a clean checkout with no external corpus file: ```sh cargo run --release --example bench_rank ``` -A committed capture of one run lives at -[`benchmarks/rank_modes_results.txt`](benchmarks/rank_modes_results.txt). +A few operating points from the committed run +([`benchmarks/rank_modes_results.txt`](benchmarks/rank_modes_results.txt)): + +| Mode | bytes/vec | p50 (ms) | Mdocs/s | R@10 | +|------|----------:|---------:|--------:|-----:| +| `Rank` asym (full-precision reference) | 512 | 3.71 | 8 | 0.845 | +| `RankQuant` b=4 asym | 128 | 0.31 | 96 | 0.806 | +| `RankQuant` b=2 asym | 64 | 0.24 | 126 | 0.572 | +| `RankQuant` b=2 FastScan | 128 | 0.09 | 333 | 0.570 | +| Two-stage b=2 (M=500, CR=1.000) | 96 | 0.11 | 275 | 0.572 | + +*One representative run on a **synthetic** corpus (dim=256, n=30k, seed=1), +AMD Ryzen 9 9950X (AVX-512), 32 threads, single-thread scan. **R@10 is +deterministic** run-to-run; **throughput/latency vary** with hardware and run. +R@10 is measured against FP32 brute-force cosine on this synthetic corpus — +the broader real-corpus evaluation lives in the paper (in progress).* + +## Scope + +ordvec is a **library and substrate**, not a turnkey service: small +ordinal/sign codes, fast SIMD scoring, and a built-in two-stage prefilter — +the code-and-scan layer of a retrieval system. It is not a navigable-graph +index (HNSW) on its own — yet — and not a serving tier at all: ordvec is the +substrate other systems build on, so its small, index-agnostic codes slot +**under** an ANN or sharding layer for large-scale serving rather than +replacing it. Encoding is training-free and data-oblivious by design — +no codebook fit — so you index the first vector with no prior data and never +refit as the corpus grows. + +Quality evidence in this repo is the reproducible synthetic benchmark above; +the broader real-corpus evaluation is in the paper (in progress). + +## Security: index-file trust + +The on-disk formats (`.tvr` / `.tvrq` / `.tvbm` / `.tvsb`) carry **no built-in +checksum, MAC, or signature — by design.** The loaders validate *structure* +(magic, version, bounds, exact-length payload) but not *origin*: a +structurally valid file can still be untrusted. If an index file crosses a +trust boundary (network transfer, shared storage), verifying it is the +caller's responsibility — e.g. a SHA-256 manifest, artifact-store integrity, +or Sigstore attestation. No in-format crypto is shipped because it would add +key management the library can't own. See +[`docs/INDEX_PROVENANCE.md`](docs/INDEX_PROVENANCE.md) and +[`THREAT_MODEL.md`](THREAT_MODEL.md). ## Provenance @@ -125,17 +208,37 @@ sign-cosine intuition and engineering polish. Thanks to Mike Singleton ([@singleton2787](https://github.com/singleton2787)) for mathematical assistance and mentorship. +## Research collaboration + +ordvec is the reference implementation for an in-progress paper on **ordinal +retrieval** — using the rank and sign structure of embeddings, rather than +their floating-point magnitudes, as the retrieval signal. The repository is +open specifically to grow a group of collaborators, **including named +co-authorship on the paper** — a different invitation than "send a PR." +Collaboration we're actively seeking: + +- **Real-corpus evaluation** — running the modes against public corpora + (GloVe, MTEB / BEIR, OpenAI embedding dumps) beyond the synthetic benchmark. +- **Theory** — formalising the hypergeometric candidate-generation null and + the rank-cosine invariants. +- **Independent reproduction** — re-running the benchmark on other hardware + and reporting the numbers. + +If that's your area, see [GOVERNANCE.md](GOVERNANCE.md) and open an issue or a +discussion. + ## Contributing -Contributions to the code, the docs, and the accompanying paper are all -welcome — see [CONTRIBUTING.md](CONTRIBUTING.md). The crate is going -public specifically to invite collaboration on polishing the OrdVec / -RankQuant paper. +Contributions to the code, the docs, and the paper are all welcome — see +[CONTRIBUTING.md](CONTRIBUTING.md). ## Minimum supported Rust version -ordvec's MSRV is **Rust 1.89** — the release that stabilized the AVX-512 -intrinsics the SIMD kernels rely on. Raising the MSRV is treated as a +ordvec's MSRV is **Rust 1.89** — the release that stabilized the specific +AVX-512 intrinsics the SIMD kernels compile against (it also clears the 1.87 +floor from `is_multiple_of`). Because the kernels are built against those +intrinsics, this is a hard compile floor, not just a convenience pin: a +toolchain below 1.89 won't build the crate. Raising the MSRV is treated as a minor-version change. ## License