Promote beta → main: keyword RAG ranks before it truncates - #524
Merged
Conversation
chore: sync beta with main
keywordSearchRagTable ORed the per-term ILIKE patterns and then applied LIMIT 20 with no ORDER BY, so Postgres could return any 20 of the matching rows. rankByTerms then ranked that arbitrary sample precisely. The comment claimed "over-fetch, then rank precisely" and did neither. Measured on the prod corpus for golden case adr-langgraph (7 terms, 256 of 478 chunks match at least one): before 20 candidates, 0 chunks of the target doc, overlap 1-2 of 7 after 20 candidates, 3 chunks of the target doc, overlap 2-6 of 7 The target was never in the candidate pool, which is why keyword-only recall@5 measured 0.0% on 37/37 golden queries with no error and no empty result to make it visible. Fix: build the SQL mirror of scoreByTerms (one CASE arm per term, summed), ORDER BY match_count DESC, content — the second key keeps ties deterministic per rule #16 — then LIMIT. rankByTerms still refines the top candidates. Gate: 298 files / 3198 tests green; all six architecture gates at baseline.
fix(rag): rank keyword candidates in SQL before the LIMIT truncates them
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
One change on top of
main: #523.Fixes the 0.0% keyword-only recall that #515 measured.
keywordSearchRagTableappliedLIMIT 20to an unordered OR-match, sorankByTermsranked an arbitrary sample of up to 256 matching chunks.Measured on the live prod corpus (golden case
adr-langgraph, 7 terms, 256 of 478 chunks match):Read-path only — no migration, no schema change.
Evidence
pnpm gate: 298 files / 3198 tests greenpnpm verify:arch: all six gates= baselinetests/unit/db/keyword-rag-ranking.test.ts, 3 of 4 assertions red before the fix)Not verified
pnpm eval:retrievalrecall@5/MRR has not been re-run — it needs a local corpus + Ollama and this machine has no local Postgres up. Whether RRF fusion is still negative-value (hybrid MRR 0.674 vs vector-only 0.761) is therefore still open.