docs: README pre-release lifts (numbers, hypergeometric, scope, security, research)

[Fieldnote-Echo]

README pre-release lifts

Surfaces what the README was underselling, from the Perplexity feedback + my read of it. Two guardrails held throughout:

1. No paper / real-corpus numbers — they're being re-run across three seeds at n=1800. The only numbers here are the committed synthetic in-repo benchmark, every figure verified against benchmarks/rank_modes_results.txt.
2. Don't undersell — scope is framed as composable to scale, not "can't scale," and recall is never framed as a weakness. Also does not assert the paper's "third retrieval category" positioning (still settling).

What landed

• Benchmark excerpt — a 5-row operating-point table (full-precision ref → b=4 → b=2 → FastScan → two-stage) with explicit caveats: synthetic corpus, Ryzen 9 9950X / AVX-512 hardware, and the determinism split (R@10 deterministic; throughput/latency vary).
• Hypergeometric null — the Bitmap prefilter's H(D,n_top,n_top), E=n_top²/D framing (from docs/RANK_MODES.md) as the real intellectual core. Repo framing, no unverifiable superlatives.
• Security: index-file trust — formats are unsigned by design; verify origin externally. Links docs/INDEX_PROVENANCE.md + THREAT_MODEL.md.
• Research collaboration — explicit named-co-authorship invite (real-corpus eval, theory, reproduction), upgrading the buried footnote.
• Scope — library/substrate that composes under an ANN/sharding layer; training-free.
• RankQuantFastscan surfaced with caveats (#[doc(hidden)], unstable API, 2× storage) per your call; MultiBucketBitmap (experimental) noted.
• Python bindings noted as in-progress (PyPI pending) per your call.
• MSRV rationale made precise (AVX-512 intrinsics + u64::is_multiple_of; hard compile floor).
• Docs section gains ALTERNATIVES_CONSIDERED + index-trust links.

Verification

All table figures cross-checked against benchmarks/rank_modes_results.txt (e.g. b=2 asym 0.2382 ms → 0.24; FastScan 332.9 Mdocs/s → 333; R@10 0.5715 → 0.572). Caught + fixed one of my own slips pre-commit (FastScan footprint is dim/2 B/doc, not dim). Docs-only — no code, no build impact.

[qodo-code-review]

Review Summary by Qodo

README documentation enhancements with benchmark, security, and research sections

📝 Documentation

Walkthroughs

Description

• Adds reproducible benchmark table with operating points from synthetic corpus
• Surfaces Bitmap prefilter's hypergeometric null as principled statistical test
• Introduces Security section on index-file trust and format validation
• Adds Research collaboration section with named co-authorship invitation
• Clarifies scope as composable library layer under ANN/sharding systems
• Documents RankQuantFastscan and MultiBucketBitmap with stability caveats
• Notes in-progress Python bindings and PyPI release pending
• Refines MSRV 1.89 rationale with precise intrinsic and API requirements

Diagram

flowchart LR
  A["README Content"] --> B["Benchmark Table"]
  A --> C["Hypergeometric Null"]
  A --> D["Security: Index Trust"]
  A --> E["Research Collaboration"]
  A --> F["Scope Clarification"]
  A --> G["API Stability Notes"]
  B --> H["Operating Points"]
  C --> I["Statistical Test Framing"]
  D --> J["Format Validation"]
  E --> K["Co-authorship Invite"]
  F --> L["Composable Layer"]
  G --> M["RankQuantFastscan & MultiBucketBitmap"]

Loading

File Changes

1. README.md 📝 Documentation +114/-17

Comprehensive README documentation enhancements and clarifications

• Reframes scope as composable library layer under ANN/sharding systems
• Adds 5-row operating-point benchmark table with caveats (synthetic corpus, specific hardware,
 deterministic R@10)
• Introduces hypergeometric null section explaining Bitmap prefilter as principled statistical test
• Documents RankQuantFastscan and MultiBucketBitmap with stability/experimental caveats
• Adds Python bindings section noting in-progress PyPI release
• Introduces Security section on index-file trust and format validation
• Adds Research collaboration section with explicit named co-authorship invitation
• Expands Documentation section with ALTERNATIVES_CONSIDERED and INDEX_PROVENANCE links
• Refines MSRV 1.89 rationale with precise intrinsic and API requirements

README.md

---

2. CHANGELOG.md 📝 Documentation +9/-0

CHANGELOG entry for README documentation updates

• Adds Documentation section under Unreleased documenting all README pre-release lifts
• Summarizes benchmark table, hypergeometric null, security callout, research collaboration, scope,
 API stability notes, Python bindings, and MSRV rationale

CHANGELOG.md

---

[qodo-code-review]

Code Review by Qodo

🐞 Bugs (1) 📘 Rule violations (0)

1. Broken packaged doc link 🐞 Bug ⚙ Maintainability

Description

README links to docs/ALTERNATIVES_CONSIDERED.md, but Cargo.toml excludes that file from the
crate package, so consumers installing from crates.io won’t have that referenced document available.
This creates a broken/undiscoverable documentation path at release time.

Code

README.md[R125-126]

Evidence

The README adds a relative link to docs/ALTERNATIVES_CONSIDERED.md, but Cargo.toml explicitly
excludes that same file from the packaged crate contents, making the referenced doc unavailable to
crate consumers.

README.md[123-129]
Cargo.toml[14-23]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
`README.md` links to `docs/ALTERNATIVES_CONSIDERED.md`, but that file is excluded from the crate package via `[package].exclude`. This means users reading the README from the published crate (or viewing crate sources) won’t be able to follow the link.

### Issue Context
The file exists in the repo, so the link works on GitHub, but it is intentionally excluded from the published crate payload.

### Fix Focus Areas
- README.md[121-129]
- Cargo.toml[14-23]

### Fix options (pick one)
1. **Preferable:** Change the README link to an **absolute GitHub URL** (so it works for both GitHub and crates.io readers).
2. Or **ship the doc** by removing `docs/ALTERNATIVES_CONSIDERED.md` from `[package].exclude` (if you want it included in the published crate).
3. Or remove the README bullet if you intentionally don’t want to expose it to crate consumers.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

[gemini-code-assist]

Code Review

This pull request updates the project's documentation in README.md and CHANGELOG.md to prepare for a pre-release. The updates include adding a synthetic benchmark operating-point table, explaining the hypergeometric null of the bitmap prefilter, detailing the security trust model, and introducing a research collaboration section. The reviewer pointed out a minor inaccuracy in the MSRV rationale section of the README, which references u64::is_multiple_of instead of usize::is_multiple_of as used in the codebase.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[Copilot]

Pull request overview

Documentation-focused update to the project’s README/Changelog to better reflect ordvec’s intended positioning (flat-scan substrate that composes under ANN/sharding), expose key intellectual framing (hypergeometric null for bitmap overlap), and add operational/security/research context ahead of a first release.

Changes:

• Expands README with clarified scope/positioning, a benchmark operating-point excerpt, and a hypergeometric-null explanation for Bitmap.
• Adds sections on index-file trust/security, research collaboration, Python bindings status, and additional docs links.
• Updates CHANGELOG with a summary entry for the README documentation lift.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File	Description
README.md	Adds/reshapes multiple README sections: scope framing, benchmark excerpt, hypergeometric-null explanation, security/index trust guidance, Python status, and research collaboration.
CHANGELOG.md	Records the README-focused documentation lift under Unreleased.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.