feat(mem-search): ISA est_hint no-search launch + .isa inverse-SA sidecar (build/load/FFI)

[nh13]

PR #6 of the v0.2 stack (carries commit 3acd729). Base is feat/v0.2-alloc-sweep (#18) — stacked on #5. Tracked in V0.2_PR_STACK.md row #6.

What this does

Reintroduces the inverse-SA .isa sidecar (removed back at #3) and wires the ISA "no-search launch" hint into mem_search:

• .isa sidecar (isa_file.rs, magic PMIS) — write_isa_file builds inv[sa[i]] = i directly into the packed 5-byte layout; IsaFileReader mmaps it for O(1) refpos → SA-index lookups. Opt-in: emitted only by prmi build --with-isa (~+32 GB at hg38), loaded best-effort.
• ISA est_hint no-search launch — mem_search/mem_search_backward accept an exact inverse-SA hint (prmi_isa_at(refpos)); the launch skips the model search and is byte-identical to est_hint == 0.
• FFI — prmi_has_isa, prmi_isa_at, and est_hint params on the mem-search calls (ffi_isa.rs tests); isa_build.rs integration tests.

Carry-forward of the #14 .isa hardening (the recorded debt)

The tracker's "#6 must re-apply" note flagged that #3 deleted .isa along with my #14 hardening. Auditing #6's fresh .isa design against that debt:

• Already present in feat(prmi): cleanroom trainer (uniform weighting) + shared lookup math #6's design (no action needed): Error::Io for all I/O; load_isa_best_effort cross-validates .isa num_entries against the .sa (warns + ignores on mismatch); isa_for_refpos is bounds-checked (refpos < num_entries).
• Re-applied — the one real gap: validate_isa_header computed expected_len with unchecked arithmetic (a crafted huge num_entries could wrap it and accept a truncated file, or panic in debug). Now uses usize::try_from + checked_mul/checked_add → SizeMismatch. Added a regression test (isa_rejects_overflowing_num_entries, mirroring the .sa one).
• ISA_MAGIC magic-contract assertion restored in sidecar_magic.rs, adapted to feat(prmi): cleanroom trainer (uniform weighting) + shared lookup math #6's renamed magic (PMIS = 0x53494D50, not feat(prmi): 2× forward+RC suffix array, PRMIv2 sidecar, and the spectrum runtime #14's PISA).
• train_sidecar_writer .isa-exists assertion: intentionally NOT restored. feat(prmi): cleanroom trainer (uniform weighting) + shared lookup math #6 made .isa opt-in (if config.with_isa), so the default build has no .isa (default_build_has_no_isa asserts exactly this). .isa build coverage lives in feat(prmi): cleanroom trainer (uniform weighting) + shared lookup math #6's isa_build.rs instead.

Conflict resolution

One conflict (lib.rs): merged my mem-search -2 doc (pac_num_bases overflow, from #4/#5) with #6's added est_hint >= sa_num -2 reason and the est_hint explanation paragraphs.

Green-pass

• cargo build --workspace ✅
• cargo clippy --workspace --all-targets --all-features -- -D warnings ✅
• cargo +nightly fmt --all -- --check ✅
• cargo test --workspace ✅ — 157 lib tests + all integration/FFI (.isa round-trip, overflow rejection, est_hint ≡ unhinted byte-identity across Rust + FFI, ISA_MAGIC contract); only Plan-3 deferrals ignored.

This clears the isa-hardening debt recorded in the tracker.

Summary by CodeRabbit

• New Features

  • Optional inverse suffix array (ISA) sidecars for exact-position hinting to accelerate indexed searches
  • CLI flag to opt in to writing ISA during index builds
  • Search APIs now accept and honor SA-index hints for faster "confirm-only" paths and preserve identical results to unhinted searches

• Bug Fixes / Behavior

  • Improved validation and explicit error codes for out-of-range or missing hints

• Tests

  • Expanded unit/integration tests and a benchmark comparing hinted vs. unhinted search performance

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 9a9a4572-3495-4b7e-a23d-c38bfff20d03

📥 Commits

Reviewing files that changed from the base of the PR and between 6003088 and ef59aab.

📒 Files selected for processing (14)

• prmi-sys/src/lib.rs
• prmi-sys/tests/ffi_isa.rs
• prmi-sys/tests/spectrum_ffi.rs
• prmi/examples/bench_est_hint.rs
• prmi/src/cli.rs
• prmi/src/index/mod.rs
• prmi/src/index/spectrum.rs
• prmi/src/sidecar/isa_file.rs
• prmi/src/sidecar/magic.rs
• prmi/src/sidecar/mod.rs
• prmi/src/train/config.rs
• prmi/src/train/mod.rs
• prmi/tests/isa_build.rs
• prmi/tests/sidecar_magic.rs

🚧 Files skipped from review as they are similar to previous changes (11)

• prmi/src/sidecar/mod.rs
• prmi/src/sidecar/magic.rs
• prmi/src/cli.rs
• prmi/tests/sidecar_magic.rs
• prmi-sys/tests/ffi_isa.rs
• prmi/src/train/config.rs
• prmi/examples/bench_est_hint.rs
• prmi/src/index/spectrum.rs
• prmi/src/index/mod.rs
• prmi/tests/isa_build.rs
• prmi-sys/src/lib.rs

---

📝 Walkthrough

Walkthrough

Adds optional inverse-suffix-array (.isa) sidecars, integrates them into LearnedIndex with O(1) refpos→SA lookups, implements hint-driven forward/backward search paths and FFI helpers, wires build/CLI to emit .isa, and provides tests and a benchmark validating equivalence and performance.

Changes

ISA Sidecar & Hint-Based Search

Layer / File(s)	Summary
ISA File Format & Serialization prmi/src/sidecar/magic.rs, prmi/src/sidecar/isa_file.rs, prmi/src/sidecar/mod.rs	Defines the .isa on-disk header/magic and packed 5-byte entries, implements write_isa_file() and IsaFileReader (mmap-backed) with header validation and unit tests; integrates .isa into SidecarPaths.
Index ISA Loading & Query Accessors prmi/src/index/mod.rs	Adds optional isa: Option<IsaFileReader> to LearnedIndex, best-effort file-backed loading (load_isa_best_effort), has_isa() and isa_at(refpos) accessors, and SHM-open behavior.
Spectrum Fast-Path Search Methods prmi/src/index/spectrum.rs	Adds mem_search_from_hint and mem_search_backward_from_hint implementing single-compare/hinted left-walk fast paths, packed-pac validation, optional interval reconstruction, and tests ensuring fail-closed and equivalence properties.
Build Pipeline & CLI Integration prmi/src/train/config.rs, prmi/src/train/mod.rs, prmi/src/cli.rs	Adds with_isa: bool to TrainerConfig (default false), wires --with-isa CLI flag, cleans stale .isa during build, and conditionally writes .isa when building.
FFI API Layer & Hint Routing prmi-sys/src/lib.rs	Exports prmi_has_isa(handle) and prmi_isa_at(handle, refpos, out_sa_index) with explicit error codes; updates prmi_mem_search() and prmi_mem_search_backward() to validate/route est_hint (0 → unhinted launch, non-zero → exact hint → *_from_hint) and return -2 on out-of-range hints.
FFI Correctness Tests prmi-sys/tests/ffi_isa.rs, prmi-sys/tests/spectrum_ffi.rs	Adds FFI tests validating presence/absence, null/out-of-range error codes for prmi_isa_at, byte-identity of hinted vs unhinted results across SA indices, and backward-search equivalence; includes PAC packing helper for tests.
Integration Tests & Benchmarks prmi/tests/isa_build.rs, prmi/examples/bench_est_hint.rs, prmi/tests/sidecar_magic.rs	End-to-end tests verify ISA generation/loading and hint-based search equivalence for forward/backward; benchmark measures per-call latency and speedups for hinted vs unhinted search.

🎯 4 (Complex) | ⏱️ ~60 minutes

Possibly related PRs

• fg-labs/prmi#17: Modifies the same FFI entry points prmi_mem_search and prmi_mem_search_backward, and this PR implements est_hint routing and validation there.
• fg-labs/prmi#4: Introduced sidecar magic/constants that this PR extends by adding ISA_MAGIC.
• fg-labs/prmi#5: Related changes to SidecarPaths construction; this PR builds upon that to add .isa path handling.

Poem

🐰 I packed the suffixes tight,

Little hints to speed the flight,
From refpos to SA I leap,
Searches wake without the deep,
Quick as carrots, small and bright.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and specifically describes the main change: adding ISA (inverse suffix array) est_hint support for no-search fast-path launches, the .isa sidecar file (build, load, FFI), which aligns with the changeset scope.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[nh13]

@coderabbitai review

[coderabbitai]

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

[nh13]

@coderabbitai review

[coderabbitai]

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

[nh13]

@coderabbitai review

[coderabbitai]

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

[nh13]

@coderabbitai full review

[coderabbitai]

✅ Action performed

Full review finished.

[coderabbitai]

Actionable comments posted: 5

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@prmi/examples/bench_est_hint.rs`:
- Around line 100-104: Validate inputs before performing start-index math:
ensure qlen is > 0 and strictly less than ref_len (otherwise ref_len - qlen
becomes zero or negative) and ensure n_queries > 0; in the code that computes
stride and start (variables stride, ref_len, qlen, n_queries and the loop over k
producing start = (k * stride) % (ref_len - qlen)), add an early check that
returns an error or clamps qlen (e.g., require qlen < ref_len and n_queries >=
1) and handle the invalid case (panic with a clear message or adjust qlen to
ref_len - 1) so you never compute a modulo/division by zero or a negative range
when calculating (ref_len - qlen) and stride.

In `@prmi/src/index/mod.rs`:
- Around line 387-395: In load_isa_best_effort remove the isa_path.exists()
precheck and call IsaFileReader::open(isa_path) directly; match on the Err
variant and if the error.kind() is std::io::ErrorKind::NotFound return None
silently, otherwise emit the existing log::warn! (including the error) and
return None—use IsaFileReader::open and the error's kind() to distinguish
NotFound from other I/O failures so real permission/metadata errors aren't
masked.

In `@prmi/src/index/spectrum.rs`:
- Around line 1466-1468: The code performs unchecked arithmetic with anchor_len
in the backward hint path (calculate anchor_end = pivot + anchor_len as usize
and later use anchor_len + left_ext), which can overflow/truncate; update the
logic around pivot, anchor_len, anchor_end and left_ext (the backward-hint
branch that returns a MemMatch) to use checked arithmetic (e.g.,
usize::checked_add/checked_sub) and bail out by returning an appropriate
MemMatch when any checked operation returns None or when the resulting length
would exceed u32::MAX before casting; ensure you validate anchor_end and
combined lengths with checked_add and bounds checks instead of raw +/as casts to
prevent overflow and FFI truncation.
- Around line 1322-1347: The hint fast-path mem_search_from_hint omits the
packed-PAC size guard used by non-hint paths and can panic when
PacEncoding::Packed { num_bases } has an undersized pac buffer; add the same
validation before calling compare helpers: when enc is PacEncoding::Packed {
num_bases } compute the expected packed buffer length and if pac.len() is too
small return the zero MemMatch early instead of proceeding, and apply the
identical check to the other hint fast-path function in this file (the second
hint path around lines 1454–1490) so both hint codepaths fail closed on
undersized Packed pac buffers.

In `@prmi/src/sidecar/isa_file.rs`:
- Around line 54-58: write_isa_file currently allocates a full in-memory buffer
`packed = vec![0u8; n * BYTES_PER_PACKED_ENTRY]` and copies packed entries in
the loop over `sa`, which blows memory for large n; change this to write
directly to a pre-sized file (or a writable mmap) instead of building the full
Vec: create/truncate the target file, set its length to `n *
BYTES_PER_PACKED_ENTRY`, obtain a writable mmap or use positioned writes, then
for each `(i, &p) in sa.iter().enumerate()` compute `off = (p as usize) *
BYTES_PER_PACKED_ENTRY` and write `pack_position(i as u64)` directly into the
file/mmap at `off`; remove the large `packed` allocation and ensure proper
sync/close semantics.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: d7754b3a-0c52-4243-83d4-4b3ec78977dd

📥 Commits

Reviewing files that changed from the base of the PR and between f98f429 and 6003088.

📒 Files selected for processing (14)

• prmi-sys/src/lib.rs
• prmi-sys/tests/ffi_isa.rs
• prmi-sys/tests/spectrum_ffi.rs
• prmi/examples/bench_est_hint.rs
• prmi/src/cli.rs
• prmi/src/index/mod.rs
• prmi/src/index/spectrum.rs
• prmi/src/sidecar/isa_file.rs
• prmi/src/sidecar/magic.rs
• prmi/src/sidecar/mod.rs
• prmi/src/train/config.rs
• prmi/src/train/mod.rs
• prmi/tests/isa_build.rs
• prmi/tests/sidecar_magic.rs

[nh13]

@coderabbitai review

[coderabbitai]

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.