fix: stream-scoping parity across BM25 retrieval lanes

[vvooki-sys]

What & why

Enforces a single retrieval invariant across the whole search surface: every BM25 lane applies stream scoping at index-query time, not post-hoc. The vector, graph and plain-BM25 lanes already did this; the BM25 date-range and entity lanes queried the global Tantivy index without a stream predicate and relied on downstream stages. This brings both lanes to parity with their siblings and adds a fail-closed guard as defense-in-depth.

Loomem-side counterpart of the sibling-engine change (Loomem-Port: planned, commit d0c84c2 on claude/implementation-request-coxdgz) — design ported, adapted to Loomem's module layout.

Changes

P0 — stream predicate at the source

• search_with_date_range / search_with_entity (loomem-core/src/tantivy_index.rs) take Option<&str> stream and add a MUST stream clause, mirroring search_with_stream / search_with_agent. None = unfiltered (backward-compatible for pre-scoping callers). A small push_stream_clause helper factors the shared term-query construction.
• Bm25Leaf::date / Bm25Leaf::entity (loomem-server/src/handlers/search.rs) now carry stream instead of hardcoding None — this also closes the agent-scoped date/entity branch (which routes through search_with_agent with leaf.stream).
• bm25_retrieve routes date / entity through a per-stream merge helper (bm25_over_streams): single-stream fast path, multi-stream max-score union, None unscoped. The plain branch's dual original/expanded query now shares the same helper.

P1 — defense-in-depth

• filter_and_truncate runs a fail-closed retain on stream membership before top-K truncation. A result whose chunk is missing, unreadable, or carries no in-scope stream attribution is dropped, not passed through — so a future lane that forgets the predicate cannot leak cross-stream chunks. Mirrors the graph lane's gate, but fail-closed.

NLOC: the stream clause would have pushed both core functions past NLOC≤100, so the duplicated inline doc→SearchResult loops were replaced with the existing collect_results helper (behavior-identical). This also removed the pre-existing as i32 level casts on both paths.

Tests (deterministic, tempdir/in-memory — no network, no LLM)

• Core (tantivy_index.rs): date-lane and entity-lane isolation; None-parity union (backward compat); date-only (empty query) scoping; and a high-BM25 L1 twin in another stream that must not surface in a stream-scoped date query (guards the invariant against ranking-stage reorder).
• Handler e2e (through search_handler): time_filter and entity isolation, plus a non-vacuous check that the dropped twin is retrievable when scoped to its own stream.

Gates

• cargo fmt --check ✓
• cargo clippy --workspace -- -D warnings (canonical CI gate) ✓
• New tests green: core tantivy_index (20), handler handlers::search (40), agent_filter (5), new e2e (3), core integration_test (6).

Note: two pre-existing mcp/dispatcher::tests::ac6_3_* failures (MCP tool-definition count drift) exist on main and are unrelated to this change — reported, not touched, per the surgical-changes rule. The clippy --all-targets variant also surfaces two pre-existing test-only lints in llm.rs/dispatcher.rs, likewise untouched.

Out of scope

MCP tool schema (time_filter semantics), ranking/boost changes (L1 ×1.5 untouched), and the already-conformant vector/graph/plain-BM25 lanes.

No merge — merge is Łukasz's call.

🤖 Generated with Claude Code

---

Generated by Claude Code

Greptile Summary

This PR makes BM25 stream scoping consistent across date-range and entity retrieval. The main changes are:

• Adds optional stream predicates to search_with_date_range and search_with_entity.
• Routes plain, date, and entity BM25 branches through a shared per-stream merge helper.
• Adds a fail-closed stream membership guard before final top-K truncation.
• Updates core and handler tests for scoped, unscoped, and cross-stream twin cases.

Confidence Score: 5/5

Safe to merge with low risk.

The changed retrieval paths preserve unscoped None behavior while adding source-level stream constraints for scoped searches. Tests cover date, entity, empty-query date, high-score cross-stream twins, and handler-level behavior. No functional or security issues were identified in the changed files.

No files require special attention.

T-Rex Logs

What T-Rex did

• The environment checks were run and reported llvm-config not found, clang not found, and no libclang entries.
• The targeted test command was executed but clang-sys panicked due to missing libclang libraries after llvm-config became unavailable.
• A fallback formatting gate was attempted but blocked because cargo-fmt is not installed for the stable toolchain.
• Artifacts documenting the test run, environment check, and cargo-fmt check were captured and uploaded for review.

_{Ran code and verified through T-Rex}

Important Files Changed

Filename	Overview
loomem-core/src/tantivy_index.rs	Adds optional stream predicates to date-range and entity BM25 searches, refactors result collection, and covers scoped/unscoped behavior with focused unit tests.
loomem-server/src/handlers/search.rs	Routes BM25 plain/date/entity branches through per-stream source-scoped retrieval and adds a final fail-closed stream guard before top-K truncation, with handler e2e tests.
loomem-core/tests/integration_test.rs	Updates the entity-search integration test to use the new backward-compatible unscoped None stream argument.

Sequence Diagram

%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
participant Client
participant Handler as search_handler/bm25_retrieve
participant Merge as bm25_over_streams
participant Leaf as bm25_leaf
participant Tantivy as TantivyIndex
participant Guard as filter_and_truncate

Client->>Handler: SearchRequest with streams/date/entity
Handler->>Merge: streams + branch-specific leaf builder
alt no stream scope
    Merge->>Leaf: run_leaf(None)
else one stream
    Merge->>Leaf: run_leaf(Some(stream))
else multiple streams
    loop each stream
        Merge->>Leaf: run_leaf(Some(stream))
        Leaf-->>Merge: scoped BM25 hits
    end
    Merge->>Merge: merge by id, max score wins
end
Leaf->>Tantivy: search_with_date_range/search_with_entity(..., stream)
Tantivy->>Tantivy: add MUST stream TermQuery when Some(stream)
Tantivy-->>Handler: SearchResult candidates
Handler->>Guard: hybrid results + ctx.stream_list
Guard->>Guard: retain only chunks whose stored stream is in scope
Guard-->>Client: top-K scoped results

Loading

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
participant Client
participant Handler as search_handler/bm25_retrieve
participant Merge as bm25_over_streams
participant Leaf as bm25_leaf
participant Tantivy as TantivyIndex
participant Guard as filter_and_truncate

Client->>Handler: SearchRequest with streams/date/entity
Handler->>Merge: streams + branch-specific leaf builder
alt no stream scope
    Merge->>Leaf: run_leaf(None)
else one stream
    Merge->>Leaf: run_leaf(Some(stream))
else multiple streams
    loop each stream
        Merge->>Leaf: run_leaf(Some(stream))
        Leaf-->>Merge: scoped BM25 hits
    end
    Merge->>Merge: merge by id, max score wins
end
Leaf->>Tantivy: search_with_date_range/search_with_entity(..., stream)
Tantivy->>Tantivy: add MUST stream TermQuery when Some(stream)
Tantivy-->>Handler: SearchResult candidates
Handler->>Guard: hybrid results + ctx.stream_list
Guard->>Guard: retain only chunks whose stored stream is in scope
Guard-->>Client: top-K scoped results

Loading

_{Reviews (3): Last reviewed commit: "fix: scope BM25 date/entity lanes at the..." | Re-trigger Greptile}

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
loomem-site	Ready	Preview, Comment	Jul 10, 2026 8:11pm