Skip to content

Correspondence-scaling timing benchmark + group timing scripts under scripts/timing/#634

Merged
efahnestock merged 1 commit into
mainfrom
correspondence_timing
Jul 2, 2026
Merged

Correspondence-scaling timing benchmark + group timing scripts under scripts/timing/#634
efahnestock merged 1 commit into
mainfrom
correspondence_timing

Conversation

@efahnestock

Copy link
Copy Markdown
Collaborator

Stacked on #632.

What

  1. Introduces experimental/overhead_matching/swag/scripts/timing/ as the home for pipeline timing benchmarks, so they don't accumulate in the scripts/ catch-all. Moves measure_pipeline_latency.py (from Add a script to measure latency #632) into it.
  2. Adds the local-LLM correspondence timing benchmark (from the now-superseded Local-LLM correspondence timing benchmark (Gemma 3 27B) #633): measures how long a local LLM would take to replace the learned correspondence model, by matching a panorama's landmarks directly against a city's full OSM landmark set.

Files (in scripts/timing/)

  • measure_pipeline_latency.py — moved from scripts/ (unchanged).
  • correspondence_scaling_common.py — loads the Chicago OSM stockpile (28,582 unique tag bundles) + panorama query landmarks, builds the match prompt, validates JSON output, extrapolates per-call timings to per-landmark / per-panorama / full-city cost.
  • benchmark_correspondence_scaling_ollama.pygemma3:27b sweep via ollama (candidate-fill × query-batch × thinking on/off).
  • correspondence_scaling_README.md — setup + reproduction.
  • BUILD — the three targets.

scripts/BUILD: drops the measure_pipeline_latency target (moved) and widens evaluate_histogram_on_paths visibility so the relocated target can depend on it.

Result (backs the paper)

per panorama
Gemma 3 27B, no thinking ~5.4 min
Gemma 3 27B, chain-of-thought ~7.6 min
learned correspondence step 3.7 s

Notes

Base automatically changed from latency_script to main July 2, 2026 15:00
…er scripts/timing/

Introduces experimental/overhead_matching/swag/scripts/timing/ as the home for
pipeline timing benchmarks, so they aren't scattered in the scripts/ catch-all.

- Moves measure_pipeline_latency.py (from #632) into scripts/timing/.
- Adds the local-LLM correspondence timing benchmark:
    correspondence_scaling_common.py (stockpile/query loading, match prompt,
    JSON validation, extrapolation/report) and
    benchmark_correspondence_scaling_ollama.py (gemma3:27b sweep via ollama),
    plus a README with setup + reproduction.
- Adds scripts/timing/BUILD with the three targets; removes the
  measure_pipeline_latency target from scripts/BUILD and widens
  evaluate_histogram_on_paths visibility so the relocated target can depend on it.

Backs the paper's timing numbers: pipeline ~36.6 s/observation, and the LLM
alternative ~5.4/7.6 min per panorama (Gemma 3 27B, no/with thinking).
@efahnestock efahnestock force-pushed the correspondence_timing branch from 1ced5ff to 39748c2 Compare July 2, 2026 15:39
@efahnestock efahnestock merged commit 70809ea into main Jul 2, 2026
3 checks passed
@efahnestock efahnestock deleted the correspondence_timing branch July 2, 2026 15:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant