From 178e0b0ed180c6545aaab3d7f0e205d02726365e Mon Sep 17 00:00:00 2001 From: Harsha Vardhan Simhadri Date: Mon, 4 May 2026 12:54:08 -0700 Subject: [PATCH 1/8] update readme and review instructions --- .github/copilot-instructions.md | 67 +++++++++++++++++++++++++++++++++ .github/instructions.md | 11 ------ README.md | 46 +++++++++++++++------- 3 files changed, 100 insertions(+), 24 deletions(-) create mode 100644 .github/copilot-instructions.md delete mode 100644 .github/instructions.md diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md new file mode 100644 index 000000000..59c30b2a5 --- /dev/null +++ b/.github/copilot-instructions.md @@ -0,0 +1,67 @@ +When performing a code review, check that: +- No unit tests were eliminated without a strong reason. +- Additional dependencies introduced have a strong justification. +- Changes are not likely to increase build times. +- Each file has a license header. +``` +/* + * Copyright (c) Microsoft Corporation. + * Licensed under the MIT license. + */ +``` + +## SemVer and API Compatibility + +- The workspace obeys SemVer. Removing or changing public API signatures (functions, types, re-exports) is a breaking change and requires a major version bump or a deprecated compatibility shim. +- Re-exports are part of the public API surface — removing them is also a breaking change. +- If changing public behavior, explain migration impact in the PR description. + +## Error Handling + +- Do not introduce `panic!` paths for recoverable errors — propagate with `Result` instead. +- Keep error types small. Avoid large enums/structs that blow up the stack; look for ways to reduce field sizes (e.g., compute derivable fields, use enums instead of `&'static str`). +- Prefer `ANNError::new(ANNErrorKind::…, e)` over the old `log`-style constructors, which force eager string formatting. +- When using `thiserror`, rely on `#[from]` for automatic `Error::source` chaining — do not format the inner error in the `#[error("…")]` display string. +- Include relevant context values (e.g., the kind, key, or dimension) in error messages for debuggability. + +## Documentation + +- Doc comments and README examples must match actual API signatures and serialized shapes. Stale examples that fail to compile or deserialize are treated as bugs. +- Do not leave dead references to APIs that no longer exist. +- When changing a function signature or removing a parameter, update all doc comments that mention the old signature. + +## Constants and Assumptions + +- Do not hardcode magic values — make them configurable with sensible defaults and document the rationale. +- If using `wrapping_add` or other wrapping arithmetic, justify why overflow is expected or acceptable. +- Add assertions for invariants that callers or maintainers would otherwise have to discover by reading the implementation. + +## SIMD and Platform Portability + +- Do not assume specific SIMD lane widths (e.g., `f32s::LANES == 8`). Code must be correct on AVX2, AVX-512, and ARM/NEON. +- When touching architecture-specific intrinsics, verify cross-platform behavior per `diskann-wide/README.md`. + +## Testing + +- Keep test helpers close to the code they exercise, typically in a `mod tests` at the bottom of the file or in an adjacent test module, guarded with `#[cfg(test)]`. +- Do not add tests for derived traits (`Clone`, `Debug`, `PartialEq`) or enums unless they have explicit behavior beyond the derive. +- Test edge cases like empty inputs (e.g., empty iterators) to lock in defined behavior and prevent divide-by-zero or NaN results. + +## Rayon and Parallelism + +- Never use the global Rayon thread pool. Always execute parallel work within the provided `RayonThreadPool` or `RayonThreadPoolRef`. +- Preserve deadlock-avoidance intent when modifying nested parallel loops. Be aware that combining blocking synchronization (e.g., mutex acquisition) with Rayon work-stealing can cause deadlocks. + +## Unsafe Code and Safety + +- Every `unsafe` block must have a `// SAFETY:` comment directly above it explaining why the operation is sound. This is enforced by the `undocumented_unsafe_blocks = "warn"` workspace lint. +- Safety comments must be specific and verifiable — state the concrete precondition that makes the operation safe (e.g., `// SAFETY: i + width <= len ensures this read is in-bounds`). Do not use vague justifications like `// SAFETY: this is safe`. +- Safety contracts on `unsafe fn` signatures must be internally consistent — if the documented precondition says `scratch.len() >= n`, ensure the implementation does not write beyond `n` elements (e.g., due to rounding up to a panel/block size). +- Prefer safe abstractions over raw `unsafe` when possible. Use `unsafe` only when there is a measurable performance benefit or when interfacing with FFI/intrinsics. +- For pointer arithmetic, prefer `offset_from` to express bounds rather than `wrapping_add` unless wrapping behavior is intentionally needed — and document why. +- When calling SIMD intrinsics or FFI, list the specific preconditions being satisfied (alignment, length, non-null, valid initialization). + +## Naming + +- Use names that reflect the current architecture, not historical ones. Rename outdated terms when refactoring. +- Struct and type names should map clearly to their domain concepts for easier mental mapping. diff --git a/.github/instructions.md b/.github/instructions.md deleted file mode 100644 index 784882c81..000000000 --- a/.github/instructions.md +++ /dev/null @@ -1,11 +0,0 @@ -When performing a code review, check that: -- No unit tests were eliminated without a strong reason. -- Additional dependencies introduced have a strong justification. -- Changes are not likely to increase build times. -- Each file has a license header. -``` -/* - * Copyright (c) Microsoft Corporation. - * Licensed under the MIT license. - */ -``` diff --git a/README.md b/README.md index 4dc168d2b..f59474763 100644 --- a/README.md +++ b/README.md @@ -1,20 +1,29 @@ # DiskANN -[![DiskANN Main](https://github.com/microsoft/DiskANN/actions/workflows/push-test.yml/badge.svg?branch=main)](https://github.com/microsoft/DiskANN/actions/workflows/push-test.yml) -[![PyPI version](https://img.shields.io/pypi/v/diskannpy.svg)](https://pypi.org/project/diskannpy/) -[![Downloads shield](https://pepy.tech/badge/diskannpy)](https://pepy.tech/project/diskannpy) -[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT) +The new main branch implements DiskANN3 in Rust, and used by many search, recommendation, RAG and vector DB platforms across Microsoft. +DiskANN3 is a library for bringing scalable, accurate and cost-effective vector indexing to stores and databases. +See our [wiki](https://github.com/microsoft/DiskANN/wiki/DiskANN-Project-and-Research-Overview-(2018%E2%80%90present)) for a comrehensive overview of research underlying this project. -[![DiskANN Paper](https://img.shields.io/badge/Paper-NeurIPS%3A_DiskANN-blue)](https://papers.nips.cc/paper/9527-rand-nsg-fast-accurate-billion-point-nearest-neighbor-search-on-a-single-node.pdf) -[![DiskANN Paper](https://img.shields.io/badge/Paper-Arxiv%3A_Fresh--DiskANN-blue)](https://arxiv.org/abs/2105.09613) -[![DiskANN Paper](https://img.shields.io/badge/Paper-Filtered--DiskANN-blue)](https://harsha-simhadri.org/pubs/Filtered-DiskANN23.pdf) +DiskANN3 is written to be extensible to a new store via the implementation of `Provider` API. +The library provides and update and query API to users and translate the requests into reads and mutations to the underlying store via the Provider API. + +The library supports the following algorithmic features +- Real-time updates while allowing stable recall under long update streams -- no merges, rebuilds, patches needed. +- A diverse set of distance functions and quantizers (PQ, MinMax, Scalar, Spherical) implemented for x86 and aarch64. +- Hooks to allow simple filters (predicate) processsing in conjuction with vector search. + +This repo has several Provider implementations. +- In-memory providers. These are volatile and not intended for use in databses. DiskANN3 with in-memory providers outperforms HNSWlib on throughput. +- Disk provider. This is intended to match the performormance of the first version of DiskANN reported in NeurIPS'19 paper. +- Garnet provider for high-through scale up vector search +- Bf-tree provider. +A provider for Cosmos DB NoSQL is not included but documented in this paper at [https://www.vldb.org/pvldb/vol18/p5166-upreti.pdf](aka.ms/CosmosDB/VectorSearch). -> [!IMPORTANT] -> We are currently in the process of updating this repository with a new version of the code written in Rust. +## Getting Started + +- To benchmark this library, please start with [diskann-benchmarks](/diskann-benchmark/README.md). This also allows you to build, store and load indices +- To add a new backend, implement the [Provider API](/diskann/src/provider.rs) contract for your store/DB. -DiskANN is a suite of scalable, accurate and cost-effective approximate nearest neighbor search algorithms for large-scale vector search that support real-time changes and simple filters. -This code is based on ideas from Microsoft's [DiskANN](https://aka.ms/AboutDiskANN). -The main branch now contains a rearchitected project written in Rust. This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/). For more information see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) or @@ -24,7 +33,18 @@ See [guidelines](CONTRIBUTING.md) for contributing to this project. ## Legacy C++ Code -Older C++ code is retained on the `cpp_main` branch, but is not actively developed or maintained. +[![DiskANN Main](https://github.com/microsoft/DiskANN/actions/workflows/push-test.yml/badge.svg?branch=main)](https://github.com/microsoft/DiskANN/actions/workflows/push-test.yml) +[![PyPI version](https://img.shields.io/pypi/v/diskannpy.svg)](https://pypi.org/project/diskannpy/) +[![Downloads shield](https://pepy.tech/badge/diskannpy)](https://pepy.tech/project/diskannpy) +[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT) + + +Older C++ code is retained on the `cpp_main` branch, and implements the following papers, but is not actively developed or maintained. + +[![DiskANN Paper](https://img.shields.io/badge/Paper-NeurIPS%3A_DiskANN-blue)](https://papers.nips.cc/paper/9527-rand-nsg-fast-accurate-billion-point-nearest-neighbor-search-on-a-single-node.pdf) +[![DiskANN Paper](https://img.shields.io/badge/Paper-Arxiv%3A_Fresh--DiskANN-blue)](https://arxiv.org/abs/2105.09613) +[![DiskANN Paper](https://img.shields.io/badge/Paper-Filtered--DiskANN-blue)](https://harsha-simhadri.org/pubs/Filtered-DiskANN23.pdf) + The legacy C++ code was forked off from [code for NSG](https://github.com/ZJULearning/nsg) algorithm. If you use the C++ version in your software please cite the following: From eae10c4a06f71781f814121e79b0d10b28454af2 Mon Sep 17 00:00:00 2001 From: Harsha Vardhan Simhadri Date: Mon, 4 May 2026 17:21:07 -0700 Subject: [PATCH 2/8] update agents.md --- .github/copilot-instructions.md | 2 +- agents.md | 189 ++++++++++++++++++++++++-------- diskann/README.md | 8 +- 3 files changed, 151 insertions(+), 48 deletions(-) diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md index 59c30b2a5..b656eca0c 100644 --- a/.github/copilot-instructions.md +++ b/.github/copilot-instructions.md @@ -20,7 +20,7 @@ When performing a code review, check that: - Do not introduce `panic!` paths for recoverable errors — propagate with `Result` instead. - Keep error types small. Avoid large enums/structs that blow up the stack; look for ways to reduce field sizes (e.g., compute derivable fields, use enums instead of `&'static str`). -- Prefer `ANNError::new(ANNErrorKind::…, e)` over the old `log`-style constructors, which force eager string formatting. +- Prefer `ANNError::new(ANNErrorKind::…, e)` over the old `log_*`-style constructors, which force eager string formatting and double-log errors. - When using `thiserror`, rely on `#[from]` for automatic `Error::source` chaining — do not format the inner error in the `#[error("…")]` display string. - Include relevant context values (e.g., the kind, key, or dimension) in error messages for debuggability. diff --git a/agents.md b/agents.md index 0feeca54d..b3e9587b4 100644 --- a/agents.md +++ b/agents.md @@ -1,17 +1,46 @@ # DiskANN Repository - Agent Onboarding Guide -**Last Updated**: 2026-02-11 (based on v0.45.0, Rust 1.92) +**Last Updated**: 2026-05-04 (based on v0.50.1, Rust 1.92) This guide helps coding agents understand how to work efficiently with the DiskANN repository. +> **Relationship with `.github/copilot-instructions.md`**: This file (`agents.md`) is the operational manual — project structure, commands, workflows, and detailed conventions. The [`.github/copilot-instructions.md`](.github/copilot-instructions.md) file contains concise review-time rules that are automatically injected into Copilot code review context. Keep both in sync when conventions change. + +--- + +## Quick Reference + +```bash +# Build +cargo build --workspace + +# Test +cargo test # all tests +cargo test -p diskann # single crate +cargo test -p diskann -- --exact test_name # single test +cargo test --profile ci # CI profile (faster) + +# Lint & Format +cargo fmt --all --check # check formatting +cargo fmt --all # apply formatting +cargo clippy --workspace --all-targets -- -Dwarnings # lint (matches CI) + +# Regenerate test baselines +DISKANN_TEST=overwrite cargo test -p diskann +``` + +**Note**: `rustfmt` and `clippy` are not installed by default. Run `rustup component add rustfmt clippy` if needed. + --- ## Table of Contents -1. [Repository Overview](#repository-overview) -2. [Repository Structure](#repository-structure) -3. [Testing](#testing) -4. [Code Quality & Linting](#code-quality--linting) +1. [Quick Reference](#quick-reference) +2. [Repository Overview](#repository-overview) +3. [Repository Structure](#repository-structure) +4. [Boundaries](#boundaries) +5. [Testing](#testing) +6. [Code Quality & Linting](#code-quality--linting) --- @@ -22,7 +51,7 @@ This guide helps coding agents understand how to work efficiently with the DiskA - **Language**: Rust (Edition 2021), toolchain version in [`rust-toolchain.toml`](rust-toolchain.toml) - **License**: MIT (see [`LICENSE.txt`](LICENSE.txt)) - **Version**: See [`Cargo.toml`](Cargo.toml) -- **Architecture**: Cargo workspace with 15+ crates +- **Architecture**: Cargo workspace with 17 crates (resolver = "3") - **Legacy Code**: Older C++ code is on the `cpp_main` branch (not maintained) ### Key Resources @@ -56,6 +85,7 @@ The repository uses a Cargo workspace with crates organized into functional tier - `diskann-providers/` - Storage abstraction layer - `diskann-disk/` - Disk-based indexing with io_uring support - `diskann-label-filter/` - Inverted index for filtered search +- `diskann-garnet/` - Garnet (Redis-compatible) DataProvider and FFI endpoints for vector sets **Tier 4: Infrastructure & Tools** - `diskann-benchmark-runner/` - Test runner infrastructure @@ -63,6 +93,7 @@ The repository uses a Cargo workspace with crates organized into functional tier - `diskann-benchmark-simd/` - SIMD-specific benchmarks - `diskann-benchmark/` - Benchmark definitions and runners - `diskann-tools/` - CLI utilities (autotuner, etc.) +- `vectorset/` - Garnet client for benchmarking vector set workloads --- @@ -79,6 +110,34 @@ The repository uses a Cargo workspace with crates organized into functional tier --- +## Boundaries + +### 🚫 Never + +- Modify files in `diskann/tests/generated/` by hand — these are auto-generated baselines. Regenerate with `DISKANN_TEST=overwrite`. +- Modify `rust-toolchain.toml`, `.github/workflows/`, or `.codecov.yml` without explicit approval. +- Use the global Rayon thread pool — use `RayonThreadPool`/`RayonThreadPoolRef` (enforced by `clippy.toml` disallowed methods). +- Use `rand::thread_rng` — use the project's `random.rs` utilities instead (enforced by `clippy.toml`). +- Use `vfs::PhysicalFS::new` or `VirtualStorageProvider::new_physical()` in tests — use `VirtualStorageProvider::new_overlay()`. +- Remove or weaken existing tests without a strong, documented reason. +- Commit secrets, credentials, or API keys. + +### ⚠️ Ask First + +- Adding new workspace dependencies to `Cargo.toml` — justify the addition. +- Changing public API signatures in any `diskann-*` crate — requires SemVer analysis (may need a major version bump). +- Modifying tier dependency rules (e.g., adding a Tier 3 dependency to a Tier 4 benchmark crate). +- Changing `clippy.toml` or `rustfmt.toml` lint/formatting configuration. + +### ✅ Always + +- Include the MIT license header in every new source file. +- Run `cargo fmt --all` and `cargo clippy --workspace --all-targets -- -Dwarnings` before committing. +- Update doc comments when changing function signatures or removing parameters. +- Add a `// SAFETY:` comment above every `unsafe` block with specific, verifiable preconditions. + +--- + ## Testing ### Test Execution @@ -104,9 +163,17 @@ cargo test --doc ### Test Baseline Caching System -DiskANN uses a baseline caching system for regression detection. See [`diskann/README.md`](diskann/README.md) for a high-level overview of how the baseline system works. For implementation and API details, refer directly to: -- [`diskann/src/test/cache.rs`](diskann/src/test/cache.rs) — core baseline caching APIs -- [`diskann/src/test/cmp.rs`](diskann/src/test/cmp.rs) — `VerboseEq` and related helpers for better test error messages +DiskANN uses a baseline caching system for regression detection. Test results are serialized as JSON into `diskann/tests/generated/` and compared against on subsequent runs. Any difference is flagged as a test failure. + +- To regenerate baselines: run tests with `DISKANN_TEST=overwrite` +- Before checking in: delete `diskann/tests/generated/` first, then regenerate to prune unused baselines +- Regenerated JSON files should be inspected via `git diff` during review + +The APIs are **`pub(crate)`** (internal to the `diskann` crate only): +- [`diskann/src/test/cache.rs`](diskann/src/test/cache.rs) — `get_or_save_test_results`, `TestRoot`, `TestPath` +- [`diskann/src/test/cmp.rs`](diskann/src/test/cmp.rs) — `VerboseEq` trait, `verbose_eq!` macro, `assert_eq_verbose!` + +See [`diskann/README.md`](diskann/README.md) for additional details. ### AVX-512, Aarch64, and multi-platform @@ -130,23 +197,62 @@ Low-level crates should use bespoke, precise, non-allocating error types. Use `t `diskann::ANNError` is not a suitable low-level error type. +```rust +// ✅ Good — bespoke error type with thiserror, uses #[from] for source chaining +#[derive(Debug, thiserror::Error)] +pub enum SgemmError { + #[error("dimension overflow: {expected_rows}×{expected_cols} exceeds usize")] + DimensionOverflow { expected_rows: usize, expected_cols: usize }, + + #[error(transparent)] + Allocation(#[from] AllocatorError), +} + +// ❌ Bad — single crate-level enum, formats inner error in display string +#[derive(Debug, thiserror::Error)] +pub enum MyLibError { + #[error("sgemm failed: {0}")] // Don't format the inner error here + Sgemm(#[source] SgemmError), + #[error("io failed: {0}")] + Io(#[from] std::io::Error), + // ... 20 more variants — too broad +} +``` + #### Mid-Level (diskann algorithms) Use `diskann::ANNError` and its context machinery. This type: -- Has a small size and `Drop` implementation, so is efficient in function ABIs. -- Records stack trace of its first creation under `RUST_BACKTRACE=1`. -- Precisely records line numbers of creation. -- Has a context layering machinery to add additional information as an error is passed up the stack. +- Is 16 bytes with niche optimization for `Option`, allowing return in registers. +- Records stack trace of its first construction under `RUST_BACKTRACE=1`. +- Precisely records file and line of creation via `#[track_caller]`. +- Has context layering machinery to add additional information as an error is passed up the stack. +- Is backed internally by `anyhow::Error`. When converting to `ANNError`, use `#[track_caller]` for better source reporting. +Prefer `ANNError::new(ANNErrorKind::…, e)` over the old `log_*`-style constructors, which force eager string formatting and double-log errors. + +```rust +// ✅ Good — deferred formatting, precise kind, track_caller +#[track_caller] +fn process_vectors(data: &[f32]) -> Result<(), ANNError> { + validate(data).map_err(|e| ANNError::new(ANNErrorKind::IndexError, e))?; + Ok(()) +} + +// ❌ Bad — eager string formatting, double-logs on creation +fn process_vectors(data: &[f32]) -> Result<(), ANNError> { + validate(data).map_err(|e| ANNError::log_index_error(format!("validation failed: {e}")))?; + Ok(()) +} +``` Traits with associated error types should consider constraining with `diskann::error::ToRanked` instead of `Into` if non-critical errors should be supported. -`diskann::ANNError` should be used only for unrecoverable errors. +`ANNError` is the mid-level propagated error type; use `ToRanked` and `RankedError` to distinguish transient/recoverable failures from fatal ones. #### High Level (tooling) -At this level `anyhow::Error` is an appropriate type to use. +At this level `anyhow::Error` is appropriate for binaries and CLI entry points. Note that some tooling helpers still use `ANNError` for compatibility. #### Do Not @@ -159,36 +265,35 @@ Do not use a single crate-level error enum. Problems: ### Formatting -**Note**: `rustfmt` is not installed by default. Run `rustup component add rustfmt` if needed. - -```bash -# Check formatting (matches CI) -cargo fmt --all --check - -# Apply formatting to all crates -cargo fmt --all -``` - -See [`rustfmt.toml`](rustfmt.toml) for formatting configuration. +See [`rustfmt.toml`](rustfmt.toml) for formatting configuration. Commands are in [Quick Reference](#quick-reference). ### Clippy (Linting) -**Note**: `clippy` is not installed by default. Run `rustup component add clippy` if needed. - ```bash -# Basic clippy check -cargo clippy --workspace --all-targets - -# CI-style check (warnings as errors) -cargo clippy --workspace --all-targets --config 'build.rustflags=["-Dwarnings"]' - # Check with no default features (for specific crates) cargo clippy -p diskann --no-default-features ``` See [`clippy.toml`](clippy.toml) for linting rules, including: - Disallowed methods (rayon global thread pool, rand::thread_rng, etc.) -- Required documentation for unsafe blocks + +The workspace-level lint in [`Cargo.toml`](Cargo.toml) enforces documentation for unsafe blocks: +- `undocumented_unsafe_blocks = "warn"` + +```rust +// ✅ Good — specific, verifiable precondition +// SAFETY: `i + width <= len` ensures this read is in-bounds. +let val = unsafe { ptr.add(i).read() }; + +// ✅ Good — FFI with listed preconditions +// SAFETY: `a` and `b` are non-null, properly aligned, and do not alias. +// `m`, `n`, `k` match the actual matrix dimensions. +unsafe { ffi::sgemm(m, n, k, a.as_ptr(), b.as_ptr(), c.as_mut_ptr()) }; + +// ❌ Bad — vague, unverifiable +// SAFETY: this is safe +let val = unsafe { ptr.add(i).read() }; +``` ### Code Coverage @@ -216,15 +321,13 @@ CI workflow is defined in [`.github/workflows/ci.yml`](.github/workflows/ci.yml) ## Pre-commit Checklist -Before committing changes, always run: - -```bash -# Format all code -cargo fmt --all +Before committing changes, always run the format and clippy commands from [Quick Reference](#quick-reference). -# Run clippy with warnings as errors -cargo clippy --workspace --all-targets -- -D warnings -``` +### Points to Add +how to create a new runtime +how to write error handling +how to configure providers +how to write a new benchmark --- diff --git a/diskann/README.md b/diskann/README.md index 1ca0587b8..1ff46e940 100644 --- a/diskann/README.md +++ b/diskann/README.md @@ -6,7 +6,7 @@ Stay tuned for more updates! ### Test Baselines -Developers are strongly encouraged to consider the [caching infrastructure]() +Developers are strongly encouraged to consider the [caching infrastructure](src/test/cache.rs) when writing index tests to provide an early warning of algorithmic changes. This infrastructure serializes test results into a file in `diskann/tests/generated` @@ -23,7 +23,7 @@ during the review process to flag regressions early. Before checking in new test results, it's a good idea to completely delete `diskann/tests/generated` to ensure that unused baselines get removed from the repository. -The API for registering and retrieving test results is in `diskann/src/tests/cache` +The API for registering and retrieving test results is in [`diskann/src/test/cache.rs`](src/test/cache.rs) and consists of: * `fn get_or_save_test_results(test_name: &str, results: &R) -> R`: Get the results for @@ -37,11 +37,11 @@ and consists of: The above API will return the previously saved baseline in the normal test mode, which can be compared with the `results` argument. -When comparing baselines, developers should use the `diskann::tests::cmp::VerboseEq` +When comparing baselines, developers should use the `diskann::test::cmp::VerboseEq` which provides more diagnostics regarding the source of structural inequality than the standard libraries `PartialEq` trait. Additional utilities include -* `diskann::tests::cmp::verbose_eq!`: A trait for automatically implementing `VerboseEq`. +* `diskann::test::cmp::verbose_eq!`: A macro for automatically implementing `VerboseEq`. This macro can be used until a proper `derive` macro is implemented: ```rust use diskann::test::cmp::verbose_eq; From 1a9a59270097d54760cb4c91d7dec5b1043b2e02 Mon Sep 17 00:00:00 2001 From: Harsha Vardhan Simhadri Date: Thu, 7 May 2026 09:04:08 -0700 Subject: [PATCH 3/8] updated readme --- README.md | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/README.md b/README.md index f59474763..72d989a17 100644 --- a/README.md +++ b/README.md @@ -1,7 +1,6 @@ # DiskANN -The new main branch implements DiskANN3 in Rust, and used by many search, recommendation, RAG and vector DB platforms across Microsoft. -DiskANN3 is a library for bringing scalable, accurate and cost-effective vector indexing to stores and databases. +The main branch implements DiskANN3, a library for bringing scalable, accurate and cost-effective vector indexing to stores and databases. It is meant to be used in search, recommendation, RAG and vector DB platforms. See our [wiki](https://github.com/microsoft/DiskANN/wiki/DiskANN-Project-and-Research-Overview-(2018%E2%80%90present)) for a comrehensive overview of research underlying this project. DiskANN3 is written to be extensible to a new store via the implementation of `Provider` API. @@ -10,18 +9,19 @@ The library provides and update and query API to users and translate the request The library supports the following algorithmic features - Real-time updates while allowing stable recall under long update streams -- no merges, rebuilds, patches needed. - A diverse set of distance functions and quantizers (PQ, MinMax, Scalar, Spherical) implemented for x86 and aarch64. +- Choice of memory tiers to allow operation at different price-performance points. - Hooks to allow simple filters (predicate) processsing in conjuction with vector search. -This repo has several Provider implementations. -- In-memory providers. These are volatile and not intended for use in databses. DiskANN3 with in-memory providers outperforms HNSWlib on throughput. -- Disk provider. This is intended to match the performormance of the first version of DiskANN reported in NeurIPS'19 paper. -- Garnet provider for high-through scale up vector search -- Bf-tree provider. -A provider for Cosmos DB NoSQL is not included but documented in this paper at [https://www.vldb.org/pvldb/vol18/p5166-upreti.pdf](aka.ms/CosmosDB/VectorSearch). +This repo offers the following Provider implementations as illustrative examples: +- In-memory providers. These are volatile and not intended for use in databases. DiskANN3 with in-memory providers outperforms HNSWlib on throughput. +- Disk provider. This is intended to match the performormance of the first version of DiskANN reported in [NeurIPS'19 Paper](https://img.shields.io/badge/Paper-NeurIPS%3A_DiskANN-blue)](https://papers.nips.cc/paper/9527-rand-nsg-fast-accurate-billion-point-nearest-neighbor-search-on-a-single-node.pdf). +- Garnet provider for high-throughput scale up vector search, and as an example of mapping to a k-v store. +- Bf-tree provider as an illustration of how to connect to a B-tree in your database. +The provider for [https://learn.microsoft.com/en-us/azure/cosmos-db/vector-search](Cosmos DB NoSQLVector Search) is not included but documented in the [https://www.vldb.org/pvldb/vol18/p5166-upreti.pdf](VLDB'25 paper). ## Getting Started -- To benchmark this library, please start with [diskann-benchmarks](/diskann-benchmark/README.md). This also allows you to build, store and load indices +- Please start with [diskann-benchmarks](/diskann-benchmark/README.md) to benchmark this library and its concrete implementations. This also allows you to build, store and load indices. - To add a new backend, implement the [Provider API](/diskann/src/provider.rs) contract for your store/DB. From 89bfe12ef3d95173ba017f23f5cc8f10f2dadfcb Mon Sep 17 00:00:00 2001 From: Harsha Vardhan Simhadri Date: Thu, 7 May 2026 09:47:18 -0700 Subject: [PATCH 4/8] update instructions.ms --- .github/copilot-instructions.md | 28 +++++++------ agents.md | 69 +++------------------------------ 2 files changed, 19 insertions(+), 78 deletions(-) diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md index b656eca0c..de2a4c97c 100644 --- a/.github/copilot-instructions.md +++ b/.github/copilot-instructions.md @@ -1,14 +1,4 @@ When performing a code review, check that: -- No unit tests were eliminated without a strong reason. -- Additional dependencies introduced have a strong justification. -- Changes are not likely to increase build times. -- Each file has a license header. -``` -/* - * Copyright (c) Microsoft Corporation. - * Licensed under the MIT license. - */ -``` ## SemVer and API Compatibility @@ -16,6 +6,10 @@ When performing a code review, check that: - Re-exports are part of the public API surface — removing them is also a breaking change. - If changing public behavior, explain migration impact in the PR description. +## Dependency and Build Hygiene +- Check for changes that introduce new dependencies or increase build times. If a dependency is added, it must be justified in the PR description. + + ## Error Handling - Do not introduce `panic!` paths for recoverable errors — propagate with `Result` instead. @@ -42,7 +36,7 @@ When performing a code review, check that: - When touching architecture-specific intrinsics, verify cross-platform behavior per `diskann-wide/README.md`. ## Testing - +- Do not drop existing unit tests without a strong reason. - Keep test helpers close to the code they exercise, typically in a `mod tests` at the bottom of the file or in an adjacent test module, guarded with `#[cfg(test)]`. - Do not add tests for derived traits (`Clone`, `Debug`, `PartialEq`) or enums unless they have explicit behavior beyond the derive. - Test edge cases like empty inputs (e.g., empty iterators) to lock in defined behavior and prevent divide-by-zero or NaN results. @@ -61,7 +55,11 @@ When performing a code review, check that: - For pointer arithmetic, prefer `offset_from` to express bounds rather than `wrapping_add` unless wrapping behavior is intentionally needed — and document why. - When calling SIMD intrinsics or FFI, list the specific preconditions being satisfied (alignment, length, non-null, valid initialization). -## Naming - -- Use names that reflect the current architecture, not historical ones. Rename outdated terms when refactoring. -- Struct and type names should map clearly to their domain concepts for easier mental mapping. +## License Headers +- Each file has a license header. +``` +/* + * Copyright (c) Microsoft Corporation. + * Licensed under the MIT license. + */ +``` \ No newline at end of file diff --git a/agents.md b/agents.md index b3e9587b4..50d6a4ee6 100644 --- a/agents.md +++ b/agents.md @@ -2,63 +2,6 @@ **Last Updated**: 2026-05-04 (based on v0.50.1, Rust 1.92) -This guide helps coding agents understand how to work efficiently with the DiskANN repository. - -> **Relationship with `.github/copilot-instructions.md`**: This file (`agents.md`) is the operational manual — project structure, commands, workflows, and detailed conventions. The [`.github/copilot-instructions.md`](.github/copilot-instructions.md) file contains concise review-time rules that are automatically injected into Copilot code review context. Keep both in sync when conventions change. - ---- - -## Quick Reference - -```bash -# Build -cargo build --workspace - -# Test -cargo test # all tests -cargo test -p diskann # single crate -cargo test -p diskann -- --exact test_name # single test -cargo test --profile ci # CI profile (faster) - -# Lint & Format -cargo fmt --all --check # check formatting -cargo fmt --all # apply formatting -cargo clippy --workspace --all-targets -- -Dwarnings # lint (matches CI) - -# Regenerate test baselines -DISKANN_TEST=overwrite cargo test -p diskann -``` - -**Note**: `rustfmt` and `clippy` are not installed by default. Run `rustup component add rustfmt clippy` if needed. - ---- - -## Table of Contents - -1. [Quick Reference](#quick-reference) -2. [Repository Overview](#repository-overview) -3. [Repository Structure](#repository-structure) -4. [Boundaries](#boundaries) -5. [Testing](#testing) -6. [Code Quality & Linting](#code-quality--linting) - ---- - -## Repository Overview - -**DiskANN** is a Rust implementation of scalable approximate nearest neighbor (ANN) search algorithms. The project is a rewrite from C++ to Rust. - -- **Language**: Rust (Edition 2021), toolchain version in [`rust-toolchain.toml`](rust-toolchain.toml) -- **License**: MIT (see [`LICENSE.txt`](LICENSE.txt)) -- **Version**: See [`Cargo.toml`](Cargo.toml) -- **Architecture**: Cargo workspace with 17 crates (resolver = "3") -- **Legacy Code**: Older C++ code is on the `cpp_main` branch (not maintained) - -### Key Resources -- **Contributing**: See [`CONTRIBUTING.md`](CONTRIBUTING.md) (requires CLA) -- **Code of Conduct**: See [`CODE_OF_CONDUCT.md`](CODE_OF_CONDUCT.md) - ---- ## Repository Structure @@ -78,14 +21,14 @@ The repository uses a Cargo workspace with crates organized into functional tier **Tier 2: Core Libraries** - `diskann-linalg/` - Linear algebra operations - `diskann-utils/` - Shared utilities (Reborrow, MatrixView traits) -- `diskann-quantization/` - Vector quantization (PQ, SQ) +- `diskann-quantization/` - Vector quantization -**Tier 3: Algorithm & Storage** -- `diskann/` - Core ANN graph algorithm and in-memory indexing (CENTRAL crate) -- `diskann-providers/` - Storage abstraction layer -- `diskann-disk/` - Disk-based indexing with io_uring support +**Tier 3: Algorithm & Proivders** +- `diskann/` - Core indexing logic +- `diskann-providers/` - Hodge-podge of stuff, will be dismantled +- `diskann-disk/` - Disk-based provider for the index with io_uring support - `diskann-label-filter/` - Inverted index for filtered search -- `diskann-garnet/` - Garnet (Redis-compatible) DataProvider and FFI endpoints for vector sets +- `diskann-garnet/` - Garnet (Redis-compatible) Provider and FFI endpoints for vector sets **Tier 4: Infrastructure & Tools** - `diskann-benchmark-runner/` - Test runner infrastructure From 43995980d9b66c423fcd4e605f7890ebb507d755 Mon Sep 17 00:00:00 2001 From: Harsha Vardhan Simhadri Date: Thu, 7 May 2026 09:50:48 -0700 Subject: [PATCH 5/8] moved agents.md to AGENTS.md --- agents.md => AGENTS.md | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename agents.md => AGENTS.md (100%) diff --git a/agents.md b/AGENTS.md similarity index 100% rename from agents.md rename to AGENTS.md From 8863e158e9ed7ce69fa22e94a687b345dbd08b84 Mon Sep 17 00:00:00 2001 From: Harsha Vardhan Simhadri Date: Sun, 10 May 2026 16:12:19 -0700 Subject: [PATCH 6/8] Revert README.md changes Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- README.md | 46 +++++++++++++--------------------------------- 1 file changed, 13 insertions(+), 33 deletions(-) diff --git a/README.md b/README.md index 72d989a17..4dc168d2b 100644 --- a/README.md +++ b/README.md @@ -1,29 +1,20 @@ # DiskANN -The main branch implements DiskANN3, a library for bringing scalable, accurate and cost-effective vector indexing to stores and databases. It is meant to be used in search, recommendation, RAG and vector DB platforms. -See our [wiki](https://github.com/microsoft/DiskANN/wiki/DiskANN-Project-and-Research-Overview-(2018%E2%80%90present)) for a comrehensive overview of research underlying this project. - -DiskANN3 is written to be extensible to a new store via the implementation of `Provider` API. -The library provides and update and query API to users and translate the requests into reads and mutations to the underlying store via the Provider API. - -The library supports the following algorithmic features -- Real-time updates while allowing stable recall under long update streams -- no merges, rebuilds, patches needed. -- A diverse set of distance functions and quantizers (PQ, MinMax, Scalar, Spherical) implemented for x86 and aarch64. -- Choice of memory tiers to allow operation at different price-performance points. -- Hooks to allow simple filters (predicate) processsing in conjuction with vector search. - -This repo offers the following Provider implementations as illustrative examples: -- In-memory providers. These are volatile and not intended for use in databases. DiskANN3 with in-memory providers outperforms HNSWlib on throughput. -- Disk provider. This is intended to match the performormance of the first version of DiskANN reported in [NeurIPS'19 Paper](https://img.shields.io/badge/Paper-NeurIPS%3A_DiskANN-blue)](https://papers.nips.cc/paper/9527-rand-nsg-fast-accurate-billion-point-nearest-neighbor-search-on-a-single-node.pdf). -- Garnet provider for high-throughput scale up vector search, and as an example of mapping to a k-v store. -- Bf-tree provider as an illustration of how to connect to a B-tree in your database. -The provider for [https://learn.microsoft.com/en-us/azure/cosmos-db/vector-search](Cosmos DB NoSQLVector Search) is not included but documented in the [https://www.vldb.org/pvldb/vol18/p5166-upreti.pdf](VLDB'25 paper). +[![DiskANN Main](https://github.com/microsoft/DiskANN/actions/workflows/push-test.yml/badge.svg?branch=main)](https://github.com/microsoft/DiskANN/actions/workflows/push-test.yml) +[![PyPI version](https://img.shields.io/pypi/v/diskannpy.svg)](https://pypi.org/project/diskannpy/) +[![Downloads shield](https://pepy.tech/badge/diskannpy)](https://pepy.tech/project/diskannpy) +[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT) -## Getting Started +[![DiskANN Paper](https://img.shields.io/badge/Paper-NeurIPS%3A_DiskANN-blue)](https://papers.nips.cc/paper/9527-rand-nsg-fast-accurate-billion-point-nearest-neighbor-search-on-a-single-node.pdf) +[![DiskANN Paper](https://img.shields.io/badge/Paper-Arxiv%3A_Fresh--DiskANN-blue)](https://arxiv.org/abs/2105.09613) +[![DiskANN Paper](https://img.shields.io/badge/Paper-Filtered--DiskANN-blue)](https://harsha-simhadri.org/pubs/Filtered-DiskANN23.pdf) -- Please start with [diskann-benchmarks](/diskann-benchmark/README.md) to benchmark this library and its concrete implementations. This also allows you to build, store and load indices. -- To add a new backend, implement the [Provider API](/diskann/src/provider.rs) contract for your store/DB. +> [!IMPORTANT] +> We are currently in the process of updating this repository with a new version of the code written in Rust. +DiskANN is a suite of scalable, accurate and cost-effective approximate nearest neighbor search algorithms for large-scale vector search that support real-time changes and simple filters. +This code is based on ideas from Microsoft's [DiskANN](https://aka.ms/AboutDiskANN). +The main branch now contains a rearchitected project written in Rust. This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/). For more information see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) or @@ -33,18 +24,7 @@ See [guidelines](CONTRIBUTING.md) for contributing to this project. ## Legacy C++ Code -[![DiskANN Main](https://github.com/microsoft/DiskANN/actions/workflows/push-test.yml/badge.svg?branch=main)](https://github.com/microsoft/DiskANN/actions/workflows/push-test.yml) -[![PyPI version](https://img.shields.io/pypi/v/diskannpy.svg)](https://pypi.org/project/diskannpy/) -[![Downloads shield](https://pepy.tech/badge/diskannpy)](https://pepy.tech/project/diskannpy) -[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT) - - -Older C++ code is retained on the `cpp_main` branch, and implements the following papers, but is not actively developed or maintained. - -[![DiskANN Paper](https://img.shields.io/badge/Paper-NeurIPS%3A_DiskANN-blue)](https://papers.nips.cc/paper/9527-rand-nsg-fast-accurate-billion-point-nearest-neighbor-search-on-a-single-node.pdf) -[![DiskANN Paper](https://img.shields.io/badge/Paper-Arxiv%3A_Fresh--DiskANN-blue)](https://arxiv.org/abs/2105.09613) -[![DiskANN Paper](https://img.shields.io/badge/Paper-Filtered--DiskANN-blue)](https://harsha-simhadri.org/pubs/Filtered-DiskANN23.pdf) - +Older C++ code is retained on the `cpp_main` branch, but is not actively developed or maintained. The legacy C++ code was forked off from [code for NSG](https://github.com/ZJULearning/nsg) algorithm. If you use the C++ version in your software please cite the following: From 981bfcee0e6c8f9b0fd056ca86f1d94055c4e6bf Mon Sep 17 00:00:00 2001 From: Harsha Vardhan Simhadri Date: Sun, 10 May 2026 16:55:05 -0700 Subject: [PATCH 7/8] shorten and split agents.md --- AGENTS.md | 130 +++++------------------------------------ diskann-wide/AGENTS.md | 7 +++ diskann/AGENTS.md | 15 +++++ 3 files changed, 38 insertions(+), 114 deletions(-) create mode 100644 diskann-wide/AGENTS.md create mode 100644 diskann/AGENTS.md diff --git a/AGENTS.md b/AGENTS.md index 50d6a4ee6..0c3bee9d9 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -3,15 +3,7 @@ **Last Updated**: 2026-05-04 (based on v0.50.1, Rust 1.92) -## Repository Structure - -The repository uses a Cargo workspace with crates organized into functional tiers. See [`Cargo.toml`](Cargo.toml) for: -- Workspace members and their dependencies -- Shared dependency versions -- Build profiles (release, ci) -- Workspace-level lints - -### Crate Organization +## Crate Organization **Tier 1: Foundation** - `diskann-wide/` - Low-level SIMD, bit manipulation, type width abstractions @@ -40,9 +32,8 @@ The repository uses a Cargo workspace with crates organized into functional tier --- -## Dependencies +### Internal Dependencies -### Internal - Tier 1 and Tier 2 crates may be added as dependencies of any internal crate - `diskann` may be added as a dependency of any equal or higher tier internal crate except those below @@ -55,7 +46,7 @@ The repository uses a Cargo workspace with crates organized into functional tier ## Boundaries -### 🚫 Never +### Never - Modify files in `diskann/tests/generated/` by hand — these are auto-generated baselines. Regenerate with `DISKANN_TEST=overwrite`. - Modify `rust-toolchain.toml`, `.github/workflows/`, or `.codecov.yml` without explicit approval. @@ -65,14 +56,14 @@ The repository uses a Cargo workspace with crates organized into functional tier - Remove or weaken existing tests without a strong, documented reason. - Commit secrets, credentials, or API keys. -### ⚠️ Ask First +### Ask First - Adding new workspace dependencies to `Cargo.toml` — justify the addition. - Changing public API signatures in any `diskann-*` crate — requires SemVer analysis (may need a major version bump). - Modifying tier dependency rules (e.g., adding a Tier 3 dependency to a Tier 4 benchmark crate). - Changing `clippy.toml` or `rustfmt.toml` lint/formatting configuration. -### ✅ Always +### Always - Include the MIT license header in every new source file. - Run `cargo fmt --all` and `cargo clippy --workspace --all-targets -- -Dwarnings` before committing. @@ -81,63 +72,17 @@ The repository uses a Cargo workspace with crates organized into functional tier --- -## Testing - -### Test Execution - -```bash -# Run all tests -cargo test - -# Run tests for specific crate -cargo test -p diskann - -# Run specific test -cargo test -p diskann -- --exact test_name - -# Run with CI profile (faster) -cargo test --profile ci - -# Run doc tests -cargo test --doc -``` - -**Note**: CI uses `cargo-nextest` for running tests. See [`.cargo/nextest.toml`](.cargo/nextest.toml) for test configuration (timeouts, retries, etc.). - -### Test Baseline Caching System - -DiskANN uses a baseline caching system for regression detection. Test results are serialized as JSON into `diskann/tests/generated/` and compared against on subsequent runs. Any difference is flagged as a test failure. - -- To regenerate baselines: run tests with `DISKANN_TEST=overwrite` -- Before checking in: delete `diskann/tests/generated/` first, then regenerate to prune unused baselines -- Regenerated JSON files should be inspected via `git diff` during review - -The APIs are **`pub(crate)`** (internal to the `diskann` crate only): -- [`diskann/src/test/cache.rs`](diskann/src/test/cache.rs) — `get_or_save_test_results`, `TestRoot`, `TestPath` -- [`diskann/src/test/cmp.rs`](diskann/src/test/cmp.rs) — `VerboseEq` trait, `verbose_eq!` macro, `assert_eq_verbose!` -See [`diskann/README.md`](diskann/README.md) for additional details. - -### AVX-512, Aarch64, and multi-platform - -When touching architecture-specific intrinsics, run cross-platform validation per `diskann-wide/README.md`: - -- Testing AVX-512 code on non-AVX-512 capable x86-64 machines. -- Testing Aarch64 code on x86-64 machines. -- Testing code compiled for and running on the `x86-64` CPU (no AVX/AVX2) does not execute unsupported instructions. - ---- -## Code Quality & Linting -### Error Handling +## Error Handling There are three regimes of error handling and the strategy to use depends on the regime. -#### Low-Level - -Low-level crates should use bespoke, precise, non-allocating error types. Use `thiserror` for boilerplate. Chain with `std::error::Error::source`. +### Low-Level +Low-level crates should use bespoke, precise, non-allocating error types. +Use `thiserror` for boilerplate. Chain with `std::error::Error::source`. `diskann::ANNError` is not a suitable low-level error type. ```rust @@ -162,7 +107,7 @@ pub enum MyLibError { } ``` -#### Mid-Level (diskann algorithms) +### Mid-Level (diskann algorithms) Use `diskann::ANNError` and its context machinery. This type: @@ -193,11 +138,11 @@ fn process_vectors(data: &[f32]) -> Result<(), ANNError> { Traits with associated error types should consider constraining with `diskann::error::ToRanked` instead of `Into` if non-critical errors should be supported. `ANNError` is the mid-level propagated error type; use `ToRanked` and `RankedError` to distinguish transient/recoverable failures from fatal ones. -#### High Level (tooling) +### High Level (tooling) At this level `anyhow::Error` is appropriate for binaries and CLI entry points. Note that some tooling helpers still use `ANNError` for compatibility. -#### Do Not +### Do Not Do not use a single crate-level error enum. Problems: @@ -206,22 +151,8 @@ Do not use a single crate-level error enum. Problems: - Generates large structs that blow up the stack - Branch-heavy `Drop` implementations which bloat code -### Formatting - -See [`rustfmt.toml`](rustfmt.toml) for formatting configuration. Commands are in [Quick Reference](#quick-reference). - -### Clippy (Linting) - -```bash -# Check with no default features (for specific crates) -cargo clippy -p diskann --no-default-features -``` - -See [`clippy.toml`](clippy.toml) for linting rules, including: -- Disallowed methods (rayon global thread pool, rand::thread_rng, etc.) -The workspace-level lint in [`Cargo.toml`](Cargo.toml) enforces documentation for unsafe blocks: -- `undocumented_unsafe_blocks = "warn"` +## Document unsafe usage ```rust // ✅ Good — specific, verifiable precondition @@ -238,42 +169,13 @@ unsafe { ffi::sgemm(m, n, k, a.as_ptr(), b.as_ptr(), c.as_mut_ptr()) }; let val = unsafe { ptr.add(i).read() }; ``` -### Code Coverage - -Code coverage of changes is required for PRs. See [`.codecov.yml`](.codecov.yml) for coverage policy and thresholds. -### CI Pipeline +## CI Pipeline CI workflow is defined in [`.github/workflows/ci.yml`](.github/workflows/ci.yml). Key jobs include: - Format and clippy checks - Tests on multiple platforms (Linux, Windows) -- Code coverage +- [Code coverage](.codecov.yml) - Architecture compatibility (SDE) -### Test Patterns - -**DO**: -- Look for existing setup/execution infrastructure -- Factor out common patterns - -**DON'T**: -- Add tests for derived traits (Clone, Debug, PartialEq) -- Add tests for enums unless they have explicit functionality - ---- - -## Pre-commit Checklist - -Before committing changes, always run the format and clippy commands from [Quick Reference](#quick-reference). - -### Points to Add -how to create a new runtime -how to write error handling -how to configure providers -how to write a new benchmark - ---- - -**End of Agent Onboarding Guide** - -*This guide should be updated when major changes occur to the repository structure or development workflows.* +**Note**: CI uses `cargo-nextest` for running tests. See [`.cargo/nextest.toml`](.cargo/nextest.toml) for test configuration (timeouts, retries, etc.). diff --git a/diskann-wide/AGENTS.md b/diskann-wide/AGENTS.md new file mode 100644 index 000000000..6b6d9c781 --- /dev/null +++ b/diskann-wide/AGENTS.md @@ -0,0 +1,7 @@ +### Multi-platform support + +When touching architecture-specific intrinsics, run cross-platform validation per `diskann-wide/README.md` and test: + +- AVX-512 code on non-AVX-512 capable x86-64 machines. +- Aarch64 code on x86-64 machines. +- code compiled for and running on the `x86-64` CPU (no AVX/AVX2) does not execute unsupported instructions. diff --git a/diskann/AGENTS.md b/diskann/AGENTS.md new file mode 100644 index 000000000..411b61bc9 --- /dev/null +++ b/diskann/AGENTS.md @@ -0,0 +1,15 @@ +### Test Baseline Caching System + +DiskANN uses a baseline caching system for regression detection. +Test results are serialized as JSON into `diskann/tests/generated/` and compared against on subsequent runs. +Any difference is flagged as a test failure. + +- To regenerate baselines: run tests with `DISKANN_TEST=overwrite` +- Before checking in: delete `diskann/tests/generated/` first, then regenerate to prune unused baselines +- Regenerated JSON files should be inspected via `git diff` during review + +The APIs are **`pub(crate)`** (internal to the `diskann` crate only): +- [`diskann/src/test/cache.rs`](diskann/src/test/cache.rs) — `get_or_save_test_results`, `TestRoot`, `TestPath` +- [`diskann/src/test/cmp.rs`](diskann/src/test/cmp.rs) — `VerboseEq` trait, `verbose_eq!` macro, `assert_eq_verbose!` + +See [`diskann/README.md`](diskann/README.md) for additional details. \ No newline at end of file From ba47b1650095ab5c182c95ca7459d2d7511f0b54 Mon Sep 17 00:00:00 2001 From: Harsha Vardhan Simhadri Date: Sun, 10 May 2026 17:03:41 -0700 Subject: [PATCH 8/8] update review instructions --- .github/copilot-instructions.md | 13 ++++++------- AGENTS.md | 3 ++- 2 files changed, 8 insertions(+), 8 deletions(-) diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md index de2a4c97c..cb6004c7f 100644 --- a/.github/copilot-instructions.md +++ b/.github/copilot-instructions.md @@ -2,9 +2,7 @@ When performing a code review, check that: ## SemVer and API Compatibility -- The workspace obeys SemVer. Removing or changing public API signatures (functions, types, re-exports) is a breaking change and requires a major version bump or a deprecated compatibility shim. -- Re-exports are part of the public API surface — removing them is also a breaking change. -- If changing public behavior, explain migration impact in the PR description. + The workspace obeys SemVer. Removing or changing public API signatures (functions, types, re-exports) requires a version bump, suggested migration impact in the PR description. ## Dependency and Build Hygiene - Check for changes that introduce new dependencies or increase build times. If a dependency is added, it must be justified in the PR description. @@ -20,14 +18,14 @@ When performing a code review, check that: ## Documentation -- Doc comments and README examples must match actual API signatures and serialized shapes. Stale examples that fail to compile or deserialize are treated as bugs. +- Doc comments and README examples must match actual API signatures and serialized shapes. +- Stale examples that fail to compile or deserialize are treated as bugs. - Do not leave dead references to APIs that no longer exist. - When changing a function signature or removing a parameter, update all doc comments that mention the old signature. ## Constants and Assumptions - Do not hardcode magic values — make them configurable with sensible defaults and document the rationale. -- If using `wrapping_add` or other wrapping arithmetic, justify why overflow is expected or acceptable. - Add assertions for invariants that callers or maintainers would otherwise have to discover by reading the implementation. ## SIMD and Platform Portability @@ -44,11 +42,12 @@ When performing a code review, check that: ## Rayon and Parallelism - Never use the global Rayon thread pool. Always execute parallel work within the provided `RayonThreadPool` or `RayonThreadPoolRef`. -- Preserve deadlock-avoidance intent when modifying nested parallel loops. Be aware that combining blocking synchronization (e.g., mutex acquisition) with Rayon work-stealing can cause deadlocks. +- Preserve deadlock-avoidance intent when modifying nested parallel loops. +- Be aware that combining blocking synchronization (e.g., mutex acquisition) with Rayon work-stealing can cause deadlocks. ## Unsafe Code and Safety -- Every `unsafe` block must have a `// SAFETY:` comment directly above it explaining why the operation is sound. This is enforced by the `undocumented_unsafe_blocks = "warn"` workspace lint. +- Every `unsafe` block must have a `// SAFETY:` comment directly above it explaining why the operation is sound. - Safety comments must be specific and verifiable — state the concrete precondition that makes the operation safe (e.g., `// SAFETY: i + width <= len ensures this read is in-bounds`). Do not use vague justifications like `// SAFETY: this is safe`. - Safety contracts on `unsafe fn` signatures must be internally consistent — if the documented precondition says `scratch.len() >= n`, ensure the implementation does not write beyond `n` elements (e.g., due to rounding up to a panel/block size). - Prefer safe abstractions over raw `unsafe` when possible. Use `unsafe` only when there is a measurable performance benefit or when interfacing with FFI/intrinsics. diff --git a/AGENTS.md b/AGENTS.md index 0c3bee9d9..b916bc394 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -140,7 +140,8 @@ Traits with associated error types should consider constraining with `diskann::e ### High Level (tooling) -At this level `anyhow::Error` is appropriate for binaries and CLI entry points. Note that some tooling helpers still use `ANNError` for compatibility. +At this level `anyhow::Error` is appropriate for binaries and CLI entry points. + Note that some tooling helpers still use `ANNError` for compatibility. ### Do Not