From 602b20ef8b11b3bf85e14ff3e4836f363ad93cf5 Mon Sep 17 00:00:00 2001 From: Nelson Spence Date: Tue, 26 May 2026 09:24:40 -0500 Subject: [PATCH 1/3] docs: README pre-release lifts (numbers, hypergeometric, scope, security, research) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Surfaces what the README was underselling, staying fiction-free — every number is verified against the committed synthetic benchmark, and no paper / real-corpus numbers are used (those are mid-re-run): - Reproducible-benchmark section gains a 5-row operating-point table from the committed synthetic run (benchmarks/rank_modes_results.txt), with explicit caveats: synthetic corpus, specific hardware, and R@10-deterministic / throughput-varies. - Hypergeometric null of the Bitmap prefilter (H(D,n_top,n_top), E=n_top^2/D) surfaced as the principled-test framing (from RANK_MODES.md). - "Security: index-file trust" callout — formats are unsigned by design; verify origin externally (links INDEX_PROVENANCE.md + THREAT_MODEL.md). - "Research collaboration" section — explicit named-co-authorship invite for the in-progress ordinal-retrieval paper (real-corpus eval, theory, repro). - "Scope" — library/substrate that composes under an ANN/sharding layer for scale (not "can't scale"); training-free. - RankQuantFastscan surfaced with caveats (#[doc(hidden)], unstable, 2x storage); MultiBucketBitmap (experimental) noted. - Python bindings noted as in-progress (PyPI pending). - MSRV 1.89 rationale made precise (AVX-512 intrinsics + u64::is_multiple_of; hard compile floor). - Documentation section gains ALTERNATIVES_CONSIDERED + index-trust links. Does not assert the paper's "third category" positioning (still settling). Signed-off-by: Nelson Spence --- CHANGELOG.md | 9 ++++ README.md | 131 ++++++++++++++++++++++++++++++++++++++++++++------- 2 files changed, 123 insertions(+), 17 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index bfa5ce55..7ff112e0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,15 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +### Documentation + +- **README pre-release lifts:** a caveated operating-point table from the + committed synthetic benchmark; the Bitmap prefilter's hypergeometric null; + a "Security: index-file trust" callout; a "Research collaboration" (paper + co-authorship) section; a "Scope" section; `RankQuantFastscan` (doc-hidden) + and the `experimental` `MultiBucketBitmap` surfaced with caveats; in-progress + Python bindings noted; and a more precise MSRV rationale. + ### Added - **CI fuzz smoke** (`.github/workflows/fuzz.yml`): a bounded cargo-fuzz run on diff --git a/README.md b/README.md index f823ca1a..53d884de 100644 --- a/README.md +++ b/README.md @@ -41,10 +41,11 @@ vector on its own: prefilter feeds an exact rerank — the coarse→fine pipeline ships as library primitives. -ordvec is a compressed **flat-scan** substrate (optionally two-stage), not -a navigable-graph or billion-scale ANN index: it pairs small codes with -fast runtime-dispatched SIMD (AVX-512/AVX2, NEON, wasm128) rather than -graph traversal. +ordvec is a compressed **flat-scan** substrate (optionally two-stage): small +codes scored by fast runtime-dispatched SIMD (AVX-512/AVX2, NEON, wasm128). It +is the code-and-scan layer, not a navigable-graph index — but the codes are +small and index-agnostic, so they compose *under* an ANN or sharding layer for +large-scale serving rather than competing with one. ## Ordinal index family @@ -57,6 +58,29 @@ graph traversal. - **`SignBitmap`** — a sign bitmap per document for sign-cosine candidate generation, feeding an exact rerank stage. +Two further paths, for callers who need them: + +- **`RankQuantFastscan`** *(`#[doc(hidden)]` — reachable as + `ordvec::RankQuantFastscan`, but the API is not yet stable)* — an optional + b=2 FastScan kernel (block-32 PQ-LUT) for absolute-minimum scan latency, at + 2× the RankQuant b=2 footprint (`dim/2` bytes/doc). Surfaced here so + latency-critical callers know it exists. +- **`MultiBucketBitmap`** *(behind `--features experimental`)* — the + multi-bucket bilinear-overlap probe behind the research-side decomposition; + a scaffold for the theory, not a production path. + +## The bitmap prefilter has a closed-form null + +The `Bitmap` prefilter scores candidates by `popcount(Q AND D)` over each +document's fixed-size top-bucket set. Because both sets are fixed size, the +overlap of two *unrelated* documents is **hypergeometric** — +`H(D, n_top, n_top)`, with expected overlap `n_top² / D` under the null (e.g. +exactly 16 at `D = 256`, `n_top = 64`). So the prefilter is a principled +statistical test rather than a tunable heuristic: it has a closed-form +false-positive rate, and an observed overlap can be read against a +hypergeometric p-value. Details in +[`docs/RANK_MODES.md`](docs/RANK_MODES.md). + ## Quickstart The crate is being prepared for its first crates.io release. Until then, @@ -87,27 +111,80 @@ For the sub-linear two-stage path (`Bitmap` / `SignBitmap` candidate generation → `RankQuant` rerank) and the full mode comparison, see [`docs/RANK_MODES.md`](docs/RANK_MODES.md). +### Python + +PyO3/maturin bindings are **in progress** — the same `Rank` / `RankQuant` / +`Bitmap` / `SignBitmap` API from Python, shipping to PyPI as `ordvec` (the +coordinated PyPI release is pending). Until then, build from source with +`maturin develop` in [`ordvec-python/`](ordvec-python/). + ## Documentation - **Design deep-dive & reproducible benchmark tables:** [`docs/RANK_MODES.md`](docs/RANK_MODES.md) +- **Design alternatives evaluated and cut:** + [`docs/ALTERNATIVES_CONSIDERED.md`](docs/ALTERNATIVES_CONSIDERED.md) +- **Index-file trust model:** + [`docs/INDEX_PROVENANCE.md`](docs/INDEX_PROVENANCE.md), + [`THREAT_MODEL.md`](THREAT_MODEL.md) - **API docs:** *(available after the first crates.io release)* -- **Paper (OrdVec / RankQuant):** _link TBD. Collaborators welcome (see - [Contributing](#contributing))._ +- **Paper (OrdVec / RankQuant):** _link TBD — see + [Research collaboration](#research-collaboration)._ ## Reproducible benchmark -The head-to-head benchmark generates a seeded synthetic corpus -in-process, so the quality numbers (R@10, candidate-recall, bytes/vec) -are regenerable from a clean checkout with no external corpus file: +The head-to-head benchmark generates a seeded synthetic corpus in-process, so +the **quality numbers (R@10, candidate-recall, bytes/vec) are deterministic** +and regenerable from a clean checkout with no external corpus file: ```sh cargo run --release --example bench_rank ``` -A committed capture of one run lives at -[`benchmarks/rank_modes_results.txt`](benchmarks/rank_modes_results.txt). +A few operating points from the committed run +([`benchmarks/rank_modes_results.txt`](benchmarks/rank_modes_results.txt)): + +| Mode | bytes/vec | p50 (ms) | Mdocs/s | R@10 | +|------|----------:|---------:|--------:|-----:| +| `Rank` asym (full-precision reference) | 512 | 3.71 | 8 | 0.845 | +| `RankQuant` b=4 asym | 128 | 0.31 | 96 | 0.806 | +| `RankQuant` b=2 asym | 64 | 0.24 | 126 | 0.572 | +| `RankQuant` b=2 FastScan | 128 | 0.09 | 333 | 0.570 | +| Two-stage b=2 (M=500, CR=1.000) | 96 | 0.11 | 275 | 0.572 | + +*One representative run on a **synthetic** corpus (dim=256, n=30k, seed=1), +AMD Ryzen 9 9950X (AVX-512), 32 threads, single-thread scan. **R@10 is +deterministic** run-to-run; **throughput/latency vary** with hardware and run. +R@10 is measured against FP32 brute-force cosine on this synthetic corpus — +the broader real-corpus evaluation lives in the paper (in progress).* + +## Scope + +ordvec is a **library and substrate**, not a turnkey service: small +ordinal/sign codes, fast SIMD scoring, and a built-in two-stage prefilter — +the code-and-scan layer of a retrieval system. It is not a navigable-graph +index (HNSW) or a distributed serving tier on its own; because the codes are +small and index-agnostic, they slot **under** an ANN or sharding layer for +large-scale serving. Encoding is training-free and data-oblivious by design — +no codebook fit — so you index the first vector with no prior data and never +refit as the corpus grows. + +Quality evidence in this repo is the reproducible synthetic benchmark above; +the broader real-corpus evaluation is in the paper (in progress). + +## Security: index-file trust + +The on-disk formats (`.tvr` / `.tvrq` / `.tvbm` / `.tvsb`) carry **no built-in +checksum, MAC, or signature — by design.** The loaders validate *structure* +(magic, version, bounds, exact-length payload) but not *origin*: a +structurally valid file can still be untrusted. If an index file crosses a +trust boundary (network transfer, shared storage), verifying it is the +caller's responsibility — e.g. a SHA-256 manifest, artifact-store integrity, +or Sigstore attestation. No in-format crypto is shipped because it would add +key management the library can't own. See +[`docs/INDEX_PROVENANCE.md`](docs/INDEX_PROVENANCE.md) and +[`THREAT_MODEL.md`](THREAT_MODEL.md). ## Provenance @@ -125,17 +202,37 @@ sign-cosine intuition and engineering polish. Thanks to Mike Singleton ([@singleton2787](https://github.com/singleton2787)) for mathematical assistance and mentorship. +## Research collaboration + +ordvec is the reference implementation for an in-progress paper on **ordinal +retrieval** — using the rank and sign structure of embeddings, rather than +their floating-point magnitudes, as the retrieval signal. The repository is +open specifically to grow a group of collaborators, **including named +co-authorship on the paper** — a different invitation than "send a PR." +Collaboration we're actively seeking: + +- **Real-corpus evaluation** — running the modes against public corpora + (GloVe, MTEB / BEIR, OpenAI embedding dumps) beyond the synthetic benchmark. +- **Theory** — formalising the hypergeometric candidate-generation null and + the rank-cosine invariants. +- **Independent reproduction** — re-running the benchmark on other hardware + and reporting the numbers. + +If that's your area, see [GOVERNANCE.md](GOVERNANCE.md) and open an issue or a +discussion. + ## Contributing -Contributions to the code, the docs, and the accompanying paper are all -welcome — see [CONTRIBUTING.md](CONTRIBUTING.md). The crate is going -public specifically to invite collaboration on polishing the OrdVec / -RankQuant paper. +Contributions to the code, the docs, and the paper are all welcome — see +[CONTRIBUTING.md](CONTRIBUTING.md). ## Minimum supported Rust version -ordvec's MSRV is **Rust 1.89** — the release that stabilized the AVX-512 -intrinsics the SIMD kernels rely on. Raising the MSRV is treated as a +ordvec's MSRV is **Rust 1.89** — the release that stabilized the specific +AVX-512 intrinsics the SIMD kernels compile against (it also clears the 1.87 +floor from `u64::is_multiple_of`). Because the kernels are built against those +intrinsics, this is a hard compile floor, not just a convenience pin: a +toolchain below 1.89 won't build the crate. Raising the MSRV is treated as a minor-version change. ## License From 0b0ffd80f9b21f72652e7bc7204f1ae86e4a73d7 Mon Sep 17 00:00:00 2001 From: Nelson Spence Date: Tue, 26 May 2026 09:40:26 -0500 Subject: [PATCH 2/3] docs: tighten Bitmap hypergeometric claim; signal graph-ready MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Codex stop-gate: the hypergeometric section overstated the claim. The null fixes the filter's *selectivity* (false-positive rate for an unrelated doc) in closed form; whether *true* neighbours clear the bar is empirical, not combinatorial — it's a candidate-generation null, not a retrieval-optimality theorem. Dropped the 'principled statistical test' / p-value-detector framing. Surfaced the deeper guarantee instead (not underselling): the order-only rank invariance is machine-checked in Lean (formalisation accompanies the paper). Also: 'not a navigable-graph index ... on its own, yet.' (graph-ready). Signed-off-by: Nelson Spence --- README.md | 22 +++++++++++++--------- 1 file changed, 13 insertions(+), 9 deletions(-) diff --git a/README.md b/README.md index 53d884de..b92b2d87 100644 --- a/README.md +++ b/README.md @@ -72,13 +72,17 @@ Two further paths, for callers who need them: ## The bitmap prefilter has a closed-form null The `Bitmap` prefilter scores candidates by `popcount(Q AND D)` over each -document's fixed-size top-bucket set. Because both sets are fixed size, the -overlap of two *unrelated* documents is **hypergeometric** — -`H(D, n_top, n_top)`, with expected overlap `n_top² / D` under the null (e.g. -exactly 16 at `D = 256`, `n_top = 64`). So the prefilter is a principled -statistical test rather than a tunable heuristic: it has a closed-form -false-positive rate, and an observed overlap can be read against a -hypergeometric p-value. Details in +document's fixed-size top-bucket set. Two *unrelated* documents — modelled as +independent uniform top-bucket sets — overlap **hypergeometrically**, +`H(D, n_top, n_top)`, with expected overlap `n_top² / D` (e.g. 16 at +`D = 256`, `n_top = 64`). So the filter's **selectivity** — how often an +unrelated document clears a given overlap threshold — is closed-form and +data-independent, not a tuned cutoff. (Whether *true* neighbours clear the bar +is empirical; this is an exact candidate-generation null, not a +retrieval-optimality theorem.) The invariance underneath it — that the rank +transform is unchanged by any strictly monotone reparametrisation of the +coordinates — is separately machine-checked in Lean, with the formalisation +accompanying the paper. Details in [`docs/RANK_MODES.md`](docs/RANK_MODES.md). ## Quickstart @@ -164,8 +168,8 @@ the broader real-corpus evaluation lives in the paper (in progress).* ordvec is a **library and substrate**, not a turnkey service: small ordinal/sign codes, fast SIMD scoring, and a built-in two-stage prefilter — the code-and-scan layer of a retrieval system. It is not a navigable-graph -index (HNSW) or a distributed serving tier on its own; because the codes are -small and index-agnostic, they slot **under** an ANN or sharding layer for +index (HNSW) or a distributed serving tier on its own, yet. The codes are +small and index-agnostic, so they slot **under** an ANN or sharding layer for large-scale serving. Encoding is training-free and data-oblivious by design — no codebook fit — so you index the first vector with no prior data and never refit as the corpus grows. From 530de9ddcae389b3b6a867f77cfbb1a79d2a04dd Mon Sep 17 00:00:00 2001 From: Nelson Spence Date: Tue, 26 May 2026 09:44:13 -0500 Subject: [PATCH 3/3] docs: address PR #75 review (notation, dispatch scope, MSRV, yet) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Bot findings (gemini/copilot): - D was overloaded (population size vs 'popcount(Q AND D)' document) — the hypergeometric is now over 'dim': H(dim, n_top, n_top), n_top^2/dim. - 'runtime-dispatched SIMD' wrongly covered NEON/wasm — dispatch is x86-only; NEON (aarch64) is baseline and wasm uses compile-time simd128. Reworded. - 'u64::is_multiple_of' -> 'is_multiple_of' (stabilized 1.87 across int types; the crate uses it on usize). Also (Nelson): 'yet' now attaches to the navigable-graph index only — ordvec is explicitly not a serving tier (it's the substrate, not a vendor competitor). Signed-off-by: Nelson Spence --- README.md | 16 +++++++++------- 1 file changed, 9 insertions(+), 7 deletions(-) diff --git a/README.md b/README.md index b92b2d87..a46926f9 100644 --- a/README.md +++ b/README.md @@ -42,7 +42,8 @@ vector on its own: library primitives. ordvec is a compressed **flat-scan** substrate (optionally two-stage): small -codes scored by fast runtime-dispatched SIMD (AVX-512/AVX2, NEON, wasm128). It +codes scored by fast SIMD — AVX-512/AVX2 runtime-dispatched on x86_64, baseline +NEON on aarch64, and `simd128` on wasm32. It is the code-and-scan layer, not a navigable-graph index — but the codes are small and index-agnostic, so they compose *under* an ANN or sharding layer for large-scale serving rather than competing with one. @@ -74,8 +75,8 @@ Two further paths, for callers who need them: The `Bitmap` prefilter scores candidates by `popcount(Q AND D)` over each document's fixed-size top-bucket set. Two *unrelated* documents — modelled as independent uniform top-bucket sets — overlap **hypergeometrically**, -`H(D, n_top, n_top)`, with expected overlap `n_top² / D` (e.g. 16 at -`D = 256`, `n_top = 64`). So the filter's **selectivity** — how often an +`H(dim, n_top, n_top)`, with expected overlap `n_top² / dim` (e.g. 16 at +`dim = 256`, `n_top = 64`). So the filter's **selectivity** — how often an unrelated document clears a given overlap threshold — is closed-form and data-independent, not a tuned cutoff. (Whether *true* neighbours clear the bar is empirical; this is an exact candidate-generation null, not a @@ -168,9 +169,10 @@ the broader real-corpus evaluation lives in the paper (in progress).* ordvec is a **library and substrate**, not a turnkey service: small ordinal/sign codes, fast SIMD scoring, and a built-in two-stage prefilter — the code-and-scan layer of a retrieval system. It is not a navigable-graph -index (HNSW) or a distributed serving tier on its own, yet. The codes are -small and index-agnostic, so they slot **under** an ANN or sharding layer for -large-scale serving. Encoding is training-free and data-oblivious by design — +index (HNSW) on its own — yet — and not a serving tier at all: ordvec is the +substrate other systems build on, so its small, index-agnostic codes slot +**under** an ANN or sharding layer for large-scale serving rather than +replacing it. Encoding is training-free and data-oblivious by design — no codebook fit — so you index the first vector with no prior data and never refit as the corpus grows. @@ -234,7 +236,7 @@ Contributions to the code, the docs, and the paper are all welcome — see ordvec's MSRV is **Rust 1.89** — the release that stabilized the specific AVX-512 intrinsics the SIMD kernels compile against (it also clears the 1.87 -floor from `u64::is_multiple_of`). Because the kernels are built against those +floor from `is_multiple_of`). Because the kernels are built against those intrinsics, this is a hard compile floor, not just a convenience pin: a toolchain below 1.89 won't build the crate. Raising the MSRV is treated as a minor-version change.