feat: SimpleRetriever (~50 LOC) matches the ~2,800 LOC stack — simplification validated#31
Open
Ven-Z8 wants to merge 9 commits into
Open
feat: SimpleRetriever (~50 LOC) matches the ~2,800 LOC stack — simplification validated#31Ven-Z8 wants to merge 9 commits into
Ven-Z8 wants to merge 9 commits into
Conversation
…-encoder) Per the 4-workflow simplification consensus: one strong dense embedder (bge-large-en-v1.5, fastembed, with query/passage prefixes) + one cross-encoder (bge-reranker-base) over top-N — the documented two-stage shape, ~50 LOC, zero new deps. Targets the proven @5<->@10 ordering gap (0.668 vs 0.789). Wired as --pipeline simple for ablation vs the 0.67 enterprise stack. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…load OOM'd) The eval builds a fresh SimpleRetriever per doc, which reloaded bge-large (~1.3GB) + reranker every document -> OOM (EXIT=137) at doc 2. Share embed/rerank models via a module cache; only the per-doc Qdrant index is fresh. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… doc 4) Each doc opened a local QdrantClient that was never closed -> handle/memory accumulation -> OOM (EXIT=137). Add SimpleRetriever.close() and call it + gc.collect() after each doc in the eval loop. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…OOM'd locally) bge-large (1024d, ~1.3GB) repeatedly OOM'd the local Mac at doc 4 on the 10-doc set. Switch to bge-base-en-v1.5 (768d, ~440MB) — still a major upgrade over the 33M bge-small, fastembed-native, fits memory; the cross-encoder is the main ordering lever regardless. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Head-to-head (same 4 docs, 25 Q): simple two-stage (bge-base + cross-encoder, ~50 LOC) page-Recall@5=0.795 vs enterprise (SPLADE+ColBERT+RRF+adaptive+router, ~2,800 LOC)=0.808 — within noise. The heavy stack buys ~nothing. Adopt simple, delete ~2,000 LOC. Path to 0.85 = stronger embedder (OOMs locally) + maybe visual, not more architecture. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
There was a problem hiding this comment.
Ven-Z8 has reached the 50-review limit for trial accounts. To continue receiving code reviews, upgrade your plan.
Each doc runs in a fresh subprocess so peak RAM is reclaimed between docs - fixes the 16GB-local OOM that killed the simple pipeline at doc 5 (accumulation, not single-doc). Adds --doc-index/--embed-model/--dim/--records-out and a scripts/eval_mmlb_isolated.py wrapper that loops per-doc and aggregates page-Recall + lift. Unblocks running a STRONGER embedder (bge-large) on the full 10-doc set. 1-doc bge-large: 0.667@5, no OOM. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
# Conflicts: # docs/benchmarks-mmlongbench.md # scripts/eval_mmlongbench.py # src/contextiq/evals/mmlongbench/runner.py
… comment) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Your trial has ended. Reactivate Greptile to resume code reviews.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The rethink/un-complication pass. 4 layer workflows converged: the heavy retrieval stack is over-engineered. Adds
SimpleRetriever(retrieval/simple.py, ~50 LOC: one FastEmbed dense embedder + one FastEmbed cross-encoder, proper query/passage prefixes), wired as--pipeline simple.Head-to-head (same 4 docs, 25 Q, 0 empty)
Within noise → the 150-function stack buys ~nothing. Recommend: adopt simple, delete ~2,000 LOC.
Key findings (in
docs/benchmarks-mmlongbench.md)Status: NOT for merge yet — paused for regroup. Captures the validated simplification + a working simple pipeline.