fix(rag): rank keyword candidates in SQL before the LIMIT truncates them

[pushkarverma3698]

Fixes the 0.0% keyword recall measured by #515 (AG-010). #515 measured the defect; this fixes it.

The bug

keywordSearchRagTable ORed the per-term ILIKE patterns, then applied LIMIT 20 with no ORDER BY. Postgres was free to return any 20 matching rows. rankByTerms then ranked that arbitrary sample precisely.

The comment said // over-fetch, then rank precisely. It did neither.

Nothing errored and nothing came back empty, so from the outside a dead keyword lane and an unhelpful one were indistinguishable — for weeks.

Failing test first

tests/unit/db/keyword-rag-ranking.test.ts captures the SQL the function actually sends and asserts the ranking happens before the truncation. Before the fix it captured, verbatim:

SELECT content, metadata
FROM brain.turicks_brain
WHERE content ILIKE $1 OR content ILIKE $2 OR content ILIKE $3 OR content ILIKE $4
LIMIT $5

3 of 4 assertions failed. All 4 pass after.

Measured on the prod corpus, not asserted

Golden case adr-langgraph — 7 significant terms, 256 of 478 chunks match at least one:

	candidates	target-doc chunks in pool	term overlap
before	20	0	1–2 of 7
after	20	3	2–6 of 7

The target document was never in the pool. recall@5 = 0% was structurally guaranteed, not a ranking weakness.

The fix

Build the SQL mirror of scoreByTerms — one CASE WHEN content ILIKE … THEN 1 ELSE 0 END per term, summed as match_count — then ORDER BY match_count DESC, content before LIMIT. The second sort key keeps ties deterministic (rule #16: retrieval is reproducible, two identical runs cannot disagree). rankByTerms still refines the top candidates in JS, and now receives the right ones.

Test plan

• pnpm gate — 298 files / 3198 tests green
• pnpm verify:arch — all six gates = baseline
• pnpm lint — exit 0
• Before/after measured against the live prod corpus (above)
• NOT VERIFIED: full pnpm eval:retrieval recall@5/MRR re-run. It needs a local corpus + Ollama; this machine has no local Postgres up. The numbers to beat are hybrid 83.8% / MRR 0.674 vs vector-only 83.8% / 0.761 — fusion is currently negative-value because one input is noise. Re-run before judging RRF.