feat(retrieval): BM25 hybrid retrieval with reciprocal rank fusion

[AyushAnand413]

Summary

• Adds BM25Okapi (sparse) retrieval alongside existing FAISS (dense) in retrieval/retriever.py
• Both retrievers independently fetch initial_top_k candidates; results merged with Reciprocal Rank Fusion (k=60) — parameter-free, robust to score-scale differences between sparse and dense signals
• Final merged list is capped at initial_top_k and passed to Reranker unchanged — zero breaking changes downstream
• rank_bm25==0.2.2 added to requirements.txt

How it works

query
 ├── FAISS (cosine sim)  → top-25 ranked list
 └── BM25Okapi           → top-25 ranked list
          ↓
   Reciprocal Rank Fusion  (score = Σ 1/(60 + rank))
          ↓
   merged top-25 → Reranker → final answer

Tests

10 new unit tests in tests/test_hybrid_retrieval.py covering:

• _tokenize edge cases
• _reciprocal_rank_fusion (single list, agreement boost, exclusive chunks, empty input, custom k)
• BM25 scoring correctness (relevant chunk ranked higher, zero score for unrelated query)

All 10 pass without requiring faiss or sentence_transformers in the test environment.

Test plan

• pip install rank_bm25==0.2.2
• Upload a PDF and run a query — verify chunks returned without errors
• Keyword-exact queries should surface the matching chunk higher than before
• pytest tests/test_hybrid_retrieval.py → 10 passed

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
ragify-ai	Ready	Preview, Comment	May 5, 2026 10:08am