feat: Claude-vision extractor + pivotal finding (retrieval, not extraction, is the bottleneck)#30
Open
Ven-Z8 wants to merge 3 commits into
Open
feat: Claude-vision extractor + pivotal finding (retrieval, not extraction, is the bottleneck)#30Ven-Z8 wants to merge 3 commits into
Ven-Z8 wants to merge 3 commits into
Conversation
…ievable text) Renders each page to an image and has Claude transcribe full-page markdown including chart/figure/table descriptions, tagged per page. Targets the visual evidence gap (Figure 0.47 / Layout 0.33) that the text pipeline and the small granite VLM both missed. Concurrent per-page calls; wired as --extractor claude. 1-doc smoke: 0.88@5, blocks page-tagged, 0 empty. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…bottleneck Standard/granite-VLM/Claude-vision extraction all give ~0.67 page-Recall@5 (identical per-source). Strong figure/chart transcription doesn't move the number -> the gold page isn't being ranked top-5 by single-shot retrieval. Next lever is retrieval (expansion/rerank/biasing/agentic), not extraction. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
There was a problem hiding this comment.
Ven-Z8 has reached the 50-review limit for trial accounts. To continue receiving code reviews, upgrade your plan.
…tractor # Conflicts: # docs/benchmarks-mmlongbench.md # scripts/eval_mmlongbench.py
There was a problem hiding this comment.
Your trial has ended. Reactivate Greptile to resume code reviews.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a
ClaudeVisionExtractor(renders pages → Claude transcribes full-page markdown incl. chart/figure descriptions) behind theExtractorprotocol, wired as--extractor claude. More importantly, it produced the key finding of the extraction phase.Pivotal result (10-doc representative set)
Claude richly transcribes every chart/figure, yet the same pages rank top-5. → Extraction is not the bottleneck; single-shot retrieval ranking is. The evidence is in the index; the retriever just doesn't surface the gold page for Figure/Layout questions.
Implication
Stop extraction work. The lever is retrieval intelligence — query reformulation / iterative / tree-guided (agentic) retrieval — which is exactly sub-projects #2/#3 of the original architecture. See updated
docs/benchmarks-mmlongbench.md.Test Plan