Add `diskann-benchmark-multi-vector` crate by suri-kumkaran · Pull Request #1027 · microsoft/DiskANN

suri-kumkaran · 2026-05-06T21:17:18Z

New benchmark crate for the Chamfer and MaxSim multi-vector distance
operations from diskann-quantization. Plugs into the existing
diskann-benchmark-runner regression workflow; mirrors the structure of
diskann-benchmark-simd.

What's new

Workspace member diskann-benchmark-multi-vector + benchmark-multi-vector binary.
Four registered regression kernels — (f32, f16) × (optimized, reference):
- optimized drives QueryComputer (architecture-dispatched SIMD).
- reference drives the Chamfer / MaxSim fallback used by the
  multi_vector unit tests.
Example configs: a benchmark matrix (multi-vector.json), a smoke
config consumed by integration tests (test.json), and a default 5%
tolerance (tolerance.json).
README documenting the kernels, the ns/IP @ Dim metric, and the
run / check verify / check run workflow.

Design notes

Two-axis dispatch (element type × implementation). Arch selection
is delegated to QueryComputer; not re-exposed here.
QueryComputer::new is hoisted out of the timed loop to preserve
the production amortization model. Reference path is hoisted
symmetrically.
ns/IP @ Dim = min_latency_µs * 1000 / (Q * D * loops).
Invariant under Q, D, and loop budget; approximately linear in
Dim. Per-shape regression detection is robust because the Dim
factor cancels.
Repetitive boilerplate collapsed into macros so adding an element type
is a one-liner.

Tests

8 unit tests (tolerance boundary, mismatched-run guard, zero-baseline
rejection, rendered table content) + 2 integration tests driving the
CLI.

Copilot

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Adds a new diskann-benchmark-multi-vector crate to benchmark and regression-check multi-vector distance ops (Chamfer, MaxSim) across f32/f16 and optimized/reference implementations, integrated with diskann-benchmark-runner.

Changes:

Introduces benchmark library with dispatcher registration, benchmark execution, and regression checking logic.
Adds benchmark-multi-vector CLI + integration tests and example job/tolerance configs used by the runner workflow.
Updates workspace to include the new crate.

Reviewed changes

Copilot reviewed 8 out of 9 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
diskann-benchmark-multi-vector/src/lib.rs	Core benchmark/regression logic, dispatch rules, fixtures, and unit tests.
diskann-benchmark-multi-vector/src/bin.rs	CLI entrypoint wiring registry + basic integration tests.
diskann-benchmark-multi-vector/examples/tolerance.json	Default tolerance config consumed by `check verify/run`.
diskann-benchmark-multi-vector/examples/test.json	Small smoke input for integration tests.
diskann-benchmark-multi-vector/examples/multi-vector.json	Full benchmark matrix input example.
diskann-benchmark-multi-vector/README.md	Documents kernels, metric normalization, and runner workflows.
diskann-benchmark-multi-vector/Cargo.toml	New crate manifest + deps/bin target.
Cargo.toml	Adds new crate to workspace members.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+#[derive(Debug, Error)]
+#[error("this kernel handles a different implementation than {0}")]
+pub(crate) struct ImplementationMismatch(Implementation);


+            match change {
+                Ok(change) => {
+                    row.insert(format!("{:.3} %", change * 100.0), 6);
+                    if change > c.tolerance.min_time_regression.get() {
+                        row.insert("FAIL", 7);
+                    }
+                }
+                Err(err) => {
+                    row.insert("invalid", 6);
+                    row.insert(err, 7);
+                }
+            }


+                match relative_change(before_min, after_min) {
+                    Ok(change) => {
+                        if change > tolerance.min_time_regression.get() {
+                            passed = false;
+                        }
+                    }
+                    Err(_) => passed = false,
+                };


+            let min_latency = r
+                .latencies
+                .iter()
+                .min()
+                .copied()
+                .unwrap_or(MicroSeconds::new(u64::MAX));


+fn run_loops<F>(run: &Run, mut body: F) -> RunResult
+where
+    F: FnMut(),
+{
+    let mut latencies = Vec::with_capacity(run.num_measurements.get());
+
+    for _ in 0..run.num_measurements.get() {
+        let start = std::time::Instant::now();
+        for _ in 0..run.loops_per_measurement.get() {
+            body();
+        }
+        latencies.push(start.elapsed().into());
+    }
+
+    let percentiles = percentiles::compute_percentiles(&mut latencies).unwrap();
+    RunResult {
+        run: run.clone(),
+        latencies,
+        percentiles,
+    }
+}


+            .join("tolerance.json");
+
+        let stdout = run_check_test(&input_path, &tolerance_path);
+        println!("stdout = {}", stdout);


codecov-commenter · 2026-05-07T06:57:02Z

Codecov Report

❌ Patch coverage is 85.55759% with 79 lines in your changes missing coverage. Please review.
✅ Project coverage is 89.44%. Comparing base (c50fb2b) to head (6b33719).
⚠️ Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
diskann-benchmark-multi-vector/src/lib.rs	84.78%	75 Missing ⚠️
diskann-benchmark-multi-vector/src/bin.rs	92.59%	4 Missing ⚠️

❌ Your patch status has failed because the patch coverage (85.55%) is below the target coverage (90.00%). You can increase the patch coverage or adjust the target coverage.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1027      +/-   ##
==========================================
- Coverage   89.51%   89.44%   -0.08%     
==========================================
  Files         460      462       +2     
  Lines       85466    85916     +450     
==========================================
+ Hits        76508    76849     +341     
- Misses       8958     9067     +109

Flag	Coverage Δ
miri	`89.44% <85.55%> (-0.08%)`	⬇️
unittests	`89.29% <85.55%> (-0.08%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
diskann-benchmark-multi-vector/src/bin.rs	`92.59% <92.59%> (ø)`
diskann-benchmark-multi-vector/src/lib.rs	`84.78% <84.78%> (ø)`

... and 11 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

This reverts commit 96d17b3.

hildebrandmw

Thanks Suryansh, this will be useful for performance optimization. One big question I had is whether or not this needs to be its own crate, or whether it can be folded into diskann-benchmark (which will need some work to put the normal indexing code behind a feature gate to keep compile times low) or as a binary in diskann-quantization directly.

The pattern established by diskann-benchmark-simd is a little unfortunate so now is the time to think about our approach before we have to deal with a crate proliferation. We also might be able to bury all benchmark related binary crates in a subdirectory to keep the top level organized.

In either case, I don't think we should publish this to crates.io - and we should probably stop publishing diskann-benchmark-simd as well.

hildebrandmw · 2026-05-07T15:34:05Z

+    ($f:ident, $field:tt, $($expr:tt)*) => {
+        writeln!($f, "{:>18}: {}", $field, $($expr)*)
+    }
+}


Since #996, manual alignment shouldn't be needed any more.

hildebrandmw · 2026-05-08T16:40:45Z

+    macro_rules! register {
+        ($impl:ident, $t:ty, $tag:literal) => {
+            dispatcher.register_regression($tag, Kernel::<$impl, $t>::new());
+        };


Nit: The macro is offering only a moderate benefit in terseness. Might be better off without it?

hildebrandmw · 2026-05-08T16:41:47Z

+    }
+
+    // Optimized (architecture-dispatched QueryComputer).
+    register!(Optimized, f32, "multi-vector-op-f32-optimized");


We should probably provide a way of overriding the runtime architecture to test both AVX512 and AVX2 kernels on the same machine. Otherwise, we have no way to compare such implementations against each other ...

hildebrandmw · 2026-05-08T16:46:00Z

+            }
+        }
+    };
+}


This might be cleaner to get rid of DispatchRule and use an entirely bespoke, smaller trait. Like:

impl TryFrom<Implementation> for Optimized { type Error = FailureScore; fn try_from(i: Implementation) -> Result<Self, Self::Error> { match i { Implementation::Optimized => Ok(Self), _ => Err(FailureScore(1)), } } }

and write matching via TryFrom.

hildebrandmw · 2026-05-08T16:48:09Z

+const RNG_SEED: u64 = 0x12345;
+
+struct Data<T> {
+    query_data: Box<[T]>,


We spend so much time writing matrix abstractions. Why not use them? If the constructors are insufficient, that's a sign we have work to do!

hildebrandmw · 2026-05-08T16:50:06Z

+        results.push(result);
+    }
+    Ok(results)
+}


These two run loops are nearly identical. Perhaps a tweak to factor out the common parts? Maybe consider keeping the loop nest the same, but taking a trait object (to reduce compile times) for the distance implementation?

hildebrandmw · 2026-05-08T16:51:28Z

+}
+
+impl_kernel_for!(f32);
+impl_kernel_for!(f16);


Since the abstractions these macros are stamping were also introduced in this PR, consider whether or not there's a simpler way to express things via generics that doesn't require macros.

hildebrandmw · 2026-05-08T16:52:58Z

+| `multi-vector-op-f16-reference`  | `f16`   | `Chamfer` / `MaxSim` |
+
+The **optimized** path constructs a `QueryComputer` once per shape (which
+internally selects the best available SIMD kernel for the host) and calls


If we're using this as a development tool ... it's probably a good idea to allow an override for the architecture.

hildebrandmw · 2026-05-08T16:54:45Z

+bench run --input-file examples/multi-vector.json --output-file before.json
+```
+
+### Regression check workflow


The section on regression checks seems to be just a copy+paste of how diskann-benchmark-runner works. Does this need to be spelled out in the README like this?

hildebrandmw · 2026-05-08T16:57:26Z

+The crate registers four kernels — one per `(element_type, implementation)`
+pair:
+
+| Tag                              | Element | Implementation       |


While it's fine to have a list like this, the chances of this getting updated as kernels are added is nearly zero. Maybe instead rely on the inputs behavior of the runner APP and ensure that the returned descriptions are sufficient?

suri-kumkaran and others added 4 commits May 7, 2026 01:18

Add benchmark crate for multi-vector

77cc936

Move some repetetive code to macros and add more benchmark cases

54a21ec

Move some repetetive code to macros and add more benchmark cases

f3a5d9f

Move some repetetive code to macros and add more benchmark cases

8efdbcd

suri-kumkaran self-assigned this May 6, 2026

suri-kumkaran requested review from a team and Copilot May 6, 2026 21:17

suri-kumkaran added this to DiskANN backlog May 6, 2026

suri-kumkaran linked an issue May 6, 2026 that may be closed by this pull request

Add benchmarking suite for multi-vector operations #993

Open

4 tasks

Copilot AI reviewed May 6, 2026

View reviewed changes

Copilot started reviewing on behalf of suri-kumkaran May 6, 2026 21:57 View session

suri-kumkaran added 2 commits May 7, 2026 11:54

Merge branch 'main' into users/suryangupta/multi-vector-benchmark

74d1cae

Add Cargo.lock

3a89c37

suri-kumkaran added 2 commits May 8, 2026 01:48

Remove unused scalar benchmark config file

96d17b3

Revert "Remove unused scalar benchmark config file"

6b33719

This reverts commit 96d17b3.

hildebrandmw reviewed May 8, 2026

View reviewed changes

harsha-simhadri requested a review from suhasjs May 8, 2026 19:33

Conversation

suri-kumkaran commented May 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What's new

Design notes

Tests

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

codecov-commenter commented May 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

hildebrandmw left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

suri-kumkaran commented May 6, 2026 •

edited

Loading

codecov-commenter commented May 7, 2026 •

edited

Loading