Coverage hardening: regression floor, experimental coverage, and adversarial red-team

[Fieldnote-Echo]

Summary

Follow-up to the test-coverage push. Investigating the Codecov badge (showing 80%) revealed it was stale — actual line coverage on main is 89.1% (verified three ways; the lcov CI uploads computes to 89.10%), already past the OpenSSF silver test_statement_coverage80 bar. So this is less "reach 80%" and more lock it in, close the weak spots, and harden the boundary ahead of going public.

Three parts:

1. ci: coverage regression floor. The coverage job uploaded to Codecov but enforced no threshold, so a regression could land silently. Adds an Enforce coverage floor step (cargo llvm-cov report --fail-under-lines 85, reusing the lcov step's profdata — no re-run) so line coverage can't erode below 85%.

2. test: experimental multi_bucket coverage. MultiBucketBitmap was the weakest module (64.7% fn / 85.5% line); top_m_bilinear, the candidate-generation primitive, was entirely untested. Adds a tie-robust top-m correctness test (boundary property + m==0 / m>n clamps), a diagonal-weight test that exercises the weight==0 skip branch, and an accessor sweep → 100% line / 100% function.

3. test: Cipher adversarial red-team suites. Two "throw garbage at it" offensive-security passes (run on Opus), pinning the FFI/loader boundary as regressions ahead of the public flip:

   • tests/redteam_delta.rs — 18 tests (core: malformed loader headers, integer overflow at dim=u16::MAX, search_asymmetric_subset candidate-list edges, empty-index search, documented fail-louds).
   • ordvec-python/tests/test_redteam_fuzz.py — 210 tests (bindings: negative/>=2**64 int scalars, dtype confusion, signaling/quiet NaN patterns, non-contiguous arrays, loader corruption, PyO3 borrow-flag reentrancy; abort-class probes subprocess-isolated).

   Verdict: zero genuine bugs. Every one of ~240 adversarial probes hit a clean typed error, an intentional fail-loud assert, or a correct result. Notable confirmations: the ~137 GiB-implied loader DoS is rejected in microseconds before allocating; negative / >=2**64 Python ints raise OverflowError (no wrap-to-usize OOM); the duplicate-candidate gather contract is pinned.

Coverage impact (core crate, --all-features)

	line	function
before	89.10%	88.11%
after	90.86%	90.56%

multi_bucket.rs 85.5%→100% line; rank.rs 92.8%→95.2%; quant.rs 83.4%→86.4%; bitmap.rs 89.6%→91.0%. (fastscan.rs 81.5% unchanged — #[doc(hidden)] optional path, not targeted; sign_bitmap.rs ~84% is capped by the AVX-512 kernels the hosted coverage runner can't exercise without SDE.)

Notes / follow-ups (not in this PR)

• OpenSSF silver test_statement_coverage80 is substantively met (90.9% ≫ 80%) and now floored — it can be marked met on bestpractices.dev (project 12977) with the Codecov dashboard as the justification URL. (Your call — it's your account.)
• The Codecov badge reads 80% (stale); it should refresh to ~91% once this lands and the coverage job re-uploads.
• Both red-team agents independently flagged one optional micro-hardening: the pub rank_norm / rankquant_norm primitives accept degenerate d ∈ {0,1} (returning 0 / -0) rather than rejecting — no crash, just an asymmetry with the fail-loud rank_to_bucket. Unreachable from the index API (constructors and loaders reject dim < 2). Candidate for a small follow-up issue alongside consistency: rank_to_bucket clamps rank >= d instead of rejecting (mirror #26 bucket_centre fail-loud) #28.

Test plan

• cargo fmt --all --check
• cargo clippy --all-targets --all-features -- -D warnings (clean)
• cargo test + cargo test --features experimental (all green)
• cargo test --no-default-features
• python -m pytest ordvec-python/tests — 365 passed (155 existing + 210 new), debug + release builds
• coverage re-measured at 90.86% line; floor step (--fail-under-lines 85) verified exit 0
• CI matrix (MSRV 1.89, AVX-512 under SDE, fuzz build, Python wheel matrix) — unaffected by these test-only + CI changes; confirmed by the PR run

[qodo-code-review]

Review Summary by Qodo

Coverage hardening: regression floor, multi_bucket expansion, and adversarial red-team suites

🧪 Tests ✨ Enhancement

Walkthroughs

Description

• Add coverage regression floor (85% line coverage) to CI workflow
• Expand multi_bucket test coverage to 100% line/function
  - top_m_bilinear correctness with tie-robust boundary checks
  - diagonal-weight matrix exercising zero-weight skip branch
  - accessor sweep before and after add
• Add 18-test adversarial red-team suite for core loaders and search
  - Malformed headers, DoS allocation attempts, integer overflow at u16::MAX
  - search_asymmetric_subset edge cases and documented fail-loud contracts
  - Empty-index/empty-input search paths
• Add 210-test Python FFI adversarial fuzz suite (Cipher)
  - Integer scalar abuse, dtype confusion, NaN bit patterns
  - Loader corruption and forged file handling
  - PyO3 borrow-flag reentrancy contract verification

Diagram

flowchart LR
  A["CI Coverage Job"] -->|"Enforce floor"| B["85% Line Coverage Gate"]
  C["multi_bucket Tests"] -->|"100% coverage"| D["top_m_bilinear + Accessors"]
  E["Core Red-Team Suite"] -->|"18 tests"| F["Loaders + Search Boundaries"]
  G["Python FFI Red-Team"] -->|"210 tests"| H["Integer/dtype/NaN/Reentrancy Guards"]
  B --> I["Regression Prevention"]
  D --> I
  F --> I
  H --> I

Loading

File Changes

1. .github/workflows/coverage.yml ⚙️ Configuration changes +12/-4

Add coverage regression floor enforcement step

• Updated workflow description to mention regression floor enforcement
• Added "Enforce coverage floor" step using cargo llvm-cov report --fail-under-lines 85
• Reuses profdata from lcov step (no re-run) for efficiency
• Prevents silent coverage regression below 85% threshold

.github/workflows/coverage.yml

---

2. tests/index/multi_bucket.rs 🧪 Tests +104/-0

Expand multi_bucket test coverage to 100%

• Added multi_bucket_top_m_bilinear_matches_bruteforce test with tie-robust correctness checks
• Added multi_bucket_bilinear_diagonal_weights_skip_zeros test exercising weight==0 skip branch
• Added multi_bucket_accessors test covering all index accessors before/after add
• Achieves 100% line and function coverage for MultiBucketBitmap module

tests/index/multi_bucket.rs

---

3. tests/redteam_delta.rs 🧪 Tests +672/-0

Add 18-test core adversarial red-team suite

• 18 adversarial tests covering loader boundary conditions (DELTA-A through DELTA-E)
• DELTA-A: malformed headers, huge declared n_vectors DoS, dim boundaries, version bytes, all-0xFF
 files
• DELTA-B: integer overflow at dim=u16::MAX ceiling in symmetric/asymmetric search
• DELTA-C: search_asymmetric_subset edge cases (empty list, k==0, k>m, duplicates, out-of-range)
• DELTA-D: empty-index/empty-input search paths and buffer integrity
• DELTA-E: documented fail-loud contracts (byte-LUT b=1 panic)
• All tests verify clean error handling without panics or OOM

tests/redteam_delta.rs

---

View more (1)

4. ordvec-python/tests/test_redteam_fuzz.py 🧪 Tests +1099/-0

Add 210-test Python FFI adversarial fuzz suite

• 210 adversarial fuzz tests of Python FFI boundary (Cipher suite)
• Integer scalar abuse: negative, 2**64, overflow handling for k/m/batch_size/idx
• dtype confusion: wrong element dtypes rejected as TypeError
• NaN encodings: signaling/quiet NaN bit patterns rejected by finite guard
• Loader corruption: truncated/extended/forged files raise clean IOError
• PyO3 reentrancy: __index__ callbacks re-entering &mut methods raise RuntimeError
• Subprocess isolation for abort-class probes to catch segfaults as exit codes
• All tests assert correct guarded behavior; zero genuine bugs found

ordvec-python/tests/test_redteam_fuzz.py

---

[qodo-code-review]

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 📎 Requirement gaps (0)

1. Coverage floor flag mismatch ✓ Resolved 🐞 Bug ☼ Reliability

Description

The coverage regression gate runs cargo llvm-cov report without --all-features, so the enforced
threshold may be computed against a different feature set than the generated/uploaded lcov report.
This weakens the regression signal and can produce inconsistent “coverage floor” behavior across
feature-gated code.

Code

.github/workflows/coverage.yml[R41-49]

Evidence

The workflow generates coverage with --all-features but enforces the threshold via `cargo llvm-cov
report without --all-features`, so the floor is not guaranteed to correspond to the uploaded
report’s feature surface.

.github/workflows/coverage.yml[41-49]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The CI step that enforces the coverage floor uses different flags than the step that generates the lcov report. This can cause the floor to be enforced on a different compilation surface than what gets uploaded.

## Issue Context
`Generate coverage (lcov)` uses `--all-features`, but `Enforce coverage floor` does not.

## Fix Focus Areas
- Update the floor step to use the same feature set (and any other relevant flags) as the lcov step.

- .github/workflows/coverage.yml[41-49]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

2. Time-based DoS assertion ✓ Resolved 🐞 Bug ☼ Reliability

Description

The loader DoS regression test asserts rejection occurs within 2 seconds, which can be flaky on slow
or highly contended CI runners even when correctness is intact. This risks intermittent failures
unrelated to functional regressions.

Code

tests/redteam_delta.rs[R137-154]

Evidence

The test explicitly measures elapsed wall time and fails if it exceeds 2 seconds, which is a common
source of flaky CI failures due to runtime variability.

tests/redteam_delta.rs[119-154]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
A Rust test uses a hard wall-clock upper bound (`< 2s`) as a correctness gate.

## Issue Context
Wall-clock timing is inherently variable in CI (shared runners, load spikes). The test can keep its intent (ensure header rejection happens before any large allocation) using a deterministic assertion instead (e.g., assert the returned error corresponds to the size/payload mismatch guard), or by making the timing bound substantially more tolerant.

## Fix Focus Areas
- Replace or relax the elapsed-time assert to avoid flaky CI failures while still validating the intended guard behavior.

- tests/redteam_delta.rs[137-154]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

3. Hard-coded nonexistent path ✓ Resolved 🐞 Bug ⚙ Maintainability

Description

test_write_to_nonexistent_directory_io_error uses a hard-coded absolute path to simulate a missing
directory, which is less portable and less hermetic than constructing a guaranteed-nonexistent path
under tmp_path. This can make the test suite more environment-dependent than necessary.

Code

ordvec-python/tests/test_redteam_fuzz.py[R1005-1009]

Evidence

The test currently relies on /nonexistent_dir_ordvec_xyz/idx.tvr being invalid on the host; using
tmp_path would make the test fully hermetic and portable.

ordvec-python/tests/test_redteam_fuzz.py[1005-1009]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
A Python test asserts an IOError by writing to a hard-coded absolute path that is assumed not to exist.

## Issue Context
Pytest already provides `tmp_path`, which can be used to create a path that is guaranteed to be nonexistent (e.g., `tmp_path / 'no_such_dir' / 'idx.tvr'`) without relying on host filesystem layout.

## Fix Focus Areas
- Replace the hard-coded `/nonexistent_dir_ordvec_xyz/...` with a nested path under `tmp_path` that is not created.

- ordvec-python/tests/test_redteam_fuzz.py[1005-1009]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

[gemini-code-assist]

Code Review

This pull request introduces comprehensive adversarial and red-team fuzzing tests to harden the ordvec FFI boundary and Rust core API. It adds a Python fuzzing suite (test_redteam_fuzz.py) targeting integer overflow, NaN handling, type confusion, and loader corruption, alongside Rust-native integration tests (multi_bucket.rs and redteam_delta.rs) verifying boundary constraints. The review feedback highlights critical cross-platform compatibility issues on 32-bit architectures where hardcoded 64-bit integer values cause unexpected OverflowError exceptions, and suggests robust resource cleanup strategies using context managers and drop guards to prevent temporary file leaks during test execution.

[Copilot]

Pull request overview

This PR hardens the project’s quality gates and boundary robustness ahead of publication by enforcing a CI coverage regression floor and adding adversarial “red-team” test suites for both the Rust core and the Python bindings (plus expanded experimental MultiBucketBitmap coverage).

Changes:

• Add a CI coverage floor check to prevent silent coverage regressions.
• Add targeted integration tests to bring MultiBucketBitmap’s candidate-generation and bilinear scoring paths to full coverage.
• Add adversarial red-team suites: a Rust loader/overflow/subset/search-edge suite and a Python FFI-boundary fuzz suite covering dtype/shape/int-scalar abuse and loader corruption.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

File	Description
.github/workflows/coverage.yml	Adds an “Enforce coverage floor” step after generating lcov coverage.
tests/index/multi_bucket.rs	Adds correctness + branch-coverage tests for top_m_bilinear, diagonal weights, and accessors.
tests/redteam_delta.rs	Introduces a new Rust red-team regression suite targeting loaders, overflow boundaries, subset edge cases, and fail-loud contracts.
ordvec-python/tests/test_redteam_fuzz.py	Introduces a large Python red-team suite probing the PyO3/numpy boundary and loader corruption behavior.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[Fieldnote-Echo]

/gemini review

[Fieldnote-Echo]

/review

[qodo-code-review]

PR Reviewer Guide 🔍

Warning

/review is deprecated. Use /agentic_review instead (removal date not yet scheduled).

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 4 🔵🔵🔵🔵⚪

🧪 PR contains tests

🔒 No security concerns identified

⚡ Recommended focus areas for review Test Logic In the candidate-set membership check for search_asymmetric_subset, the expression mixes a negative-id guard with a contains check that casts IDs, which can become a no-op or misleading depending on whether the returned id type is signed or unsigned. Consider making the assertion type-consistent (e.g., assert all returned ids are in the candidate list, and separately assert non-negativity if the API uses sentinel negatives). fn delta_c3_subset_k_greater_than_m_clamps() { let dim = 64; let n = 32; let corpus = make_corpus(8301, n, dim); let mut idx = RankQuant::new(dim, 2); idx.add(&corpus); let query = make_corpus(8302, 1, dim); let cands: Vec<u32> = vec![3, 3, 9]; let (scores, global) = idx.search_asymmetric_subset(&query, &cands, 10); assert_eq!(scores.len(), 3, "k must clamp to candidate count m"); assert_eq!(global.len(), 3); // Returned globals are drawn only from the candidate set. for &g in &global { assert!( g < 0 || cands.contains(&(g as u32)), "result id {g} not in candidate set {cands:?}" ); } } Redundancy Temporary-file cleanup is implemented via the TempFile Drop guard, but several tests also manually call remove_file on the same path. This is harmless but redundant and can obscure failures (e.g., if you later want to assert a file still exists after an operation). Prefer relying on the RAII guard consistently. /// RAII guard that removes its temp file on drop, so a panicking test never /// leaks a file in `$TMPDIR` (the per-test cleanup below is skipped if an /// assertion fails first). Derefs / `AsRef`s to `Path`, so it passes straight to /// the loaders. struct TempFile(std::path::PathBuf); impl std::ops::Deref for TempFile { type Target = std::path::Path; fn deref(&self) -> &std::path::Path { &self.0 } } impl AsRef<std::path::Path> for TempFile { fn as_ref(&self) -> &std::path::Path { &self.0 } } impl Drop for TempFile { fn drop(&mut self) { let _ = std::fs::remove_file(&self.0); } } /// Write `bytes` to a uniquely-named temp file and return a self-deleting guard. /// Mirrors the `forge` helper in `src/rank_io.rs`'s test module /// (pid + nanosecond nonce + suffix) so concurrent test binaries never /// collide, and uses only `std::fs` / `std::env::temp_dir` (no /// `tempfile` dev-dependency). fn forge(suffix: &str, bytes: &[u8]) -> TempFile { let mut p = std::env::temp_dir(); let nonce = std::time::SystemTime::now() .duration_since(std::time::UNIX_EPOCH) .unwrap() .as_nanos(); p.push(format!( "ordvec_redteam_delta_{}_{}_{}", std::process::id(), nonce, suffix )); std::fs::File::create(&p).unwrap().write_all(bytes).unwrap(); TempFile(p) } /// Run all four `T::load` entry points against one forged file and assert /// each returns `Err` without panicking. `catch_unwind` enforces the /// no-panic half of the contract (a malformed file must never abort the /// process); `is_err` enforces the rejection half. fn assert_all_loaders_reject(path: &std::path::Path, label: &str) { let p = path.to_path_buf(); let r1 = std::panic::catch_unwind(|| Rank::load(&p)); assert!(r1.is_ok(), "Rank::load panicked on {label}"); assert!(r1.unwrap().is_err(), "Rank::load accepted {label}"); let r2 = std::panic::catch_unwind(|| RankQuant::load(&p)); assert!(r2.is_ok(), "RankQuant::load panicked on {label}"); assert!(r2.unwrap().is_err(), "RankQuant::load accepted {label}"); let r3 = std::panic::catch_unwind(|| Bitmap::load(&p)); assert!(r3.is_ok(), "Bitmap::load panicked on {label}"); assert!(r3.unwrap().is_err(), "Bitmap::load accepted {label}"); let r4 = std::panic::catch_unwind(|| SignBitmap::load(&p)); assert!(r4.is_ok(), "SignBitmap::load panicked on {label}"); assert!(r4.unwrap().is_err(), "SignBitmap::load accepted {label}"); } // ===================================================================== // DELTA-A — loaders: adversarial header geometry. // ===================================================================== /// DELTA-A1 (DoS): a forged TVR1 header declaring `n_vectors == /// MAX_VECTORS` with a valid `dim` but **no payload bytes**. The implied /// payload is `1024 * 64Mi * 2 ≈ 137 GiB`; a naive loader that sizes a /// buffer from the declared length before checking it against the file /// would attempt a 137 GiB allocation. `check_payload_matches_file` runs /// *before* any allocation, so the loader must reject this in negligible /// time. We bound the wall-clock to catch a regression that re-orders the /// allocation ahead of the size check. #[test] fn delta_a1_loader_rejects_huge_declared_nvectors_with_empty_payload() { let mut v = Vec::new(); v.extend_from_slice(b"TVR1"); v.push(1); // version v.extend_from_slice(&1024u32.to_le_bytes()); // dim (valid) v.extend_from_slice(&MAX_VECTORS_U32.to_le_bytes()); // n_vectors at the cap // No payload bytes — declared payload ~137 GiB, file is header-only. let p = forge("dos_huge_nvectors.tvr", &v); let start = std::time::Instant::now(); let r = std::panic::catch_unwind(|| Rank::load(&p)); let elapsed = start.elapsed(); std::fs::remove_file(&p).ok(); assert!(r.is_ok(), "Rank::load panicked on the DoS header"); assert!( r.unwrap().is_err(), "Rank::load must reject a header declaring a gigabyte payload over an empty file" ); // The size check is O(1) (a `stream_position` + integer compare); a // generous ceiling still catches a regression that tries to allocate // ~137 GiB first. assert!( elapsed < std::time::Duration::from_secs(2), "loader took {elapsed:?} to reject a tiny-file/huge-payload header — \ a size guard must precede allocation", ); } CI Runtime The suite adds a large number of parametrized tests and also spawns subprocess-based probes with relatively high timeouts. It’s worth validating CI runtime and flake risk (especially on slower runners) and considering grouping/marking the slowest probes (or lowering the timeout) if runtime becomes an issue. _CHILD_PREAMBLE = ( "import numpy as np\n" "from ordvec import Rank, RankQuant, Bitmap, SignBitmap\n" "def uv(n,d,s=0):\n" " rng=np.random.default_rng(s); v=rng.standard_normal((n,d)).astype(np.float32)\n" " v/=np.linalg.norm(v,axis=1,keepdims=True)+1e-9; return v\n" ) def _run_isolated(body: str) -> subprocess.CompletedProcess: """Run a probe body in a child interpreter; return the completed process. The child prints ``OK`` on a clean finish. The caller asserts ``returncode == 0`` so a hard abort (negative rc = killed by signal) or an uncaught ``PanicException`` (rc = 1, traceback on stderr) fails loudly here instead of crashing the pytest session. """ src = _CHILD_PREAMBLE + body + "\nprint('OK')\n" return subprocess.run( [sys.executable, "-c", src], capture_output=True, text=True, timeout=120, ) def _assert_clean_child(proc: subprocess.CompletedProcess) -> None: assert proc.returncode == 0, ( f"child crashed: rc={proc.returncode} " f"(negative = killed by signal {-proc.returncode} → abort/segfault)\n" f"stderr:\n{proc.stderr}" ) assert "PanicException" not in proc.stderr, ( f"core panic leaked across the FFI boundary:\n{proc.stderr}" ) assert proc.stdout.strip().endswith("OK"), ( f"child did not finish cleanly:\nstdout:{proc.stdout}\nstderr:{proc.stderr}" )

[gemini-code-assist]

Code Review

This pull request introduces extensive red-team and adversarial fuzzing test suites in both Python and Rust to harden the FFI boundaries, input guards, file loaders, and edge cases. Feedback on these additions suggests using tmp_path instead of a hardcoded root-level path in Python tests, utilizing std::fs::write for more idiomatic Rust file writing, and dynamically bounding a loop in multi_bucket.rs to prevent potential out-of-bounds panics if the corpus size changes.

[Fieldnote-Echo]

Cleared the qodo review findings in d1a4826:

• Coverage floor flag mismatch: folded the floor into the single --all-features cargo llvm-cov run, so it is computed on exactly the uploaded data. The separate report step could not take --all-features (a run-time flag), which is why the two steps appeared to differ.
• Time-based DoS assertion: bumped the wall-clock bound 2s → 30s (a generous regression guard, not a perf SLA) so it will not flake on loaded runners.
• Test Logic (subset membership): split the assertion into a separate i64 sentinel check and a u32 membership check, for type-consistency.
• Subprocess timeouts: lowered the abort-probe timeout 120s → 30s (the probes finish in well under a second).
• Hard-coded path: already resolved earlier (tmp_path).

Gate green locally: clippy -D warnings, redteam_delta 18, pytest 210.