Feature: Extraction Accuracy Lab - compare ingestion accuracy across models/strategies before committing

[jaysanderson]

Pitch

Let a user upload document(s), run them through several extraction strategies / vision models, and see a side-by-side comparison of ingestion accuracy - so they can choose the right strategy for a corpus before committing to a full ingest.

Why

Ingestion quality varies widely by extractor/model, especially on scanned/complex PDFs, and choosing an extract strategy today is a blind, global, all-or-nothing decision. A manual bake-off during the build proved the top vision models differ materially in accuracy and cost. Productising it turns a hidden risk into a buyer-facing capability.

Shape

• Admin/tool surface: upload test doc(s) -> pick strategies/models -> run.
• Per-strategy results side by side: extracted-text yield (chars/page), structure/table fidelity, a quality score, cost, latency. Highlight the winner per doc.
• Then apply the chosen strategy to a real ingest.

Plugs into

ARAG already supports named extract strategies (extract_strategies, vllm_config vision path) and exposes the model catalogue via GET /kb/{id}/schema. Work is productising the (manual) comparison behind RetrievalProvider.

Risks

• Test uploads must not pollute a real corpus - scratch/sandbox KB or per-run isolation + cleanup.
• Objective scoring is hard - ground-truth or model-as-judge; be explicit it is advisory.
• Cost controls - cap pages/models per run.
• Platform note: the json:true DA generator is buggy (see docs/ARAG-DEV.md) - verify the working path first.

Ref: docs/HANDOVER.md section 8, docs/BACKLOG.md idea bank.