Local-LLM correspondence timing benchmark (Gemma 3 27B)#633
Closed
efahnestock wants to merge 1 commit into
Closed
Conversation
Measures the cost of replacing the learned correspondence model with a local LLM that matches a panorama's landmarks directly against a city's full OSM landmark set. Streams the 28,582 unique Chicago landmark tag bundles through the model in chunks and times single-stream prefill/decode latency on the local GPU, extrapolating to per-landmark / per-panorama / full-city cost. Backs the paper's timing comparison: Gemma 3 27B ~5.4 min/panorama (no thinking) and ~7.6 min/panorama (chain-of-thought), vs the 3.7 s learned correspondence step. - correspondence_scaling_common.py: stockpile/query loading, match prompt, JSON validation, sweep records, extrapolation/report. - benchmark_correspondence_scaling_ollama.py: gemma3:27b sweep via ollama. - correspondence_scaling_README.md: setup + reproduction.
Collaborator
Author
efahnestock
added a commit
that referenced
this pull request
Jul 2, 2026
…scripts/timing/ (#634) Stacked on #632. ## What 1. **Introduces `experimental/overhead_matching/swag/scripts/timing/`** as the home for pipeline timing benchmarks, so they don't accumulate in the `scripts/` catch-all. Moves `measure_pipeline_latency.py` (from #632) into it. 2. **Adds the local-LLM correspondence timing benchmark** (from the now-superseded #633): measures how long a local LLM would take to replace the learned correspondence model, by matching a panorama's landmarks directly against a city's full OSM landmark set. ## Files (in `scripts/timing/`) - `measure_pipeline_latency.py` — moved from `scripts/` (unchanged). - `correspondence_scaling_common.py` — loads the Chicago OSM stockpile (28,582 unique tag bundles) + panorama query landmarks, builds the match prompt, validates JSON output, extrapolates per-call timings to per-landmark / per-panorama / full-city cost. - `benchmark_correspondence_scaling_ollama.py` — `gemma3:27b` sweep via ollama (candidate-fill × query-batch × thinking on/off). - `correspondence_scaling_README.md` — setup + reproduction. - `BUILD` — the three targets. `scripts/BUILD`: drops the `measure_pipeline_latency` target (moved) and widens `evaluate_histogram_on_paths` visibility so the relocated target can depend on it. ## Result (backs the paper) | | per panorama | |---|---| | Gemma 3 27B, no thinking | ~5.4 min | | Gemma 3 27B, chain-of-thought | ~7.6 min | | learned correspondence step | 3.7 s | ## Notes - Requires a local GPU + ollama (`gemma3:27b` pulled on first run) and VIGOR/pano_v2 data under `/data/overhead_matching/datasets/` — not CI-reproducible; commands in the README. - Reference sweep results live at `/data/overhead_matching/evaluation/timing/`. - Supersedes #633 (same benchmark, but stacked on #632 and relocated into `scripts/timing/`).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds a benchmark measuring the cost of replacing the learned correspondence model with a local LLM that matches a panorama's landmarks directly against a city's full OSM landmark set. This backs the timing comparison in the paper.
The correspondence model cheaply scores each panorama landmark against the whole city stockpile; an LLM must instead read candidate OSM landmarks into a finite context window. So we stream the 28,582 unique Chicago landmark tag bundles through the model in chunks and time single-stream prefill+decode latency on the local GPU (RTX 5090), then extrapolate to per-landmark / per-panorama / full-city cost.
Result (backs the paper)
→ roughly two orders of magnitude slower.
Files
correspondence_scaling_common.py— loads the Chicago OSM stockpile (VIGORv4_202001) + panorama query landmarks (pano_v2panov2_tuned_prompt), builds the match prompt, validates JSON output, and extrapolates per-call timings to a whole city.benchmark_correspondence_scaling_ollama.py— runnable sweep forgemma3:27bvia ollama (candidate-fill × query-batch × thinking on/off).correspondence_scaling_README.md— setup + reproduction commands.Reproduction / notes
gemma3:27bpulled on first run, ~17 GB) and the VIGOR/pano_v2 data under/data/overhead_matching/datasets/— not CI-reproducible. Commands are in the README./data/overhead_matching/evaluation/timing/(gemma3_27b_thinking_off_per_panorama.jsonl,gemma3_27b_thinking_on_cot_per_panorama.jsonl).completion+vision), so--thinking onemulates it via chain-of-thought prompting.