Observed on the 2026-08-17 server run: each dev query has 1 positive + up to 3 hard negatives (30,272 pairs), so ranking top-10 always includes every candidate and development_recall_at_10 is 1.0 for ANY ranker (trained model and TF-IDF both score 1.0). Meaningful alternatives computed by mw-baseline.py: trained MRR=0.824 / P@1=0.698 vs TF-IDF MRR=0.556 / P@1=0.316. Proposed fix: replace the runner metric with positive-vs-candidates rank metrics (MRR, P@1, recall@k over the full dev document corpus), and record both in the execution receipt.
Observed on the 2026-08-17 server run: each dev query has 1 positive + up to 3 hard negatives (30,272 pairs), so ranking top-10 always includes every candidate and development_recall_at_10 is 1.0 for ANY ranker (trained model and TF-IDF both score 1.0). Meaningful alternatives computed by mw-baseline.py: trained MRR=0.824 / P@1=0.698 vs TF-IDF MRR=0.556 / P@1=0.316. Proposed fix: replace the runner metric with positive-vs-candidates rank metrics (MRR, P@1, recall@k over the full dev document corpus), and record both in the execution receipt.