feat: measure the shortlist funnel loss, then size the shortlist per patient

[majdabd]

Two commits: measure the pipeline's largest loss, then act on its dominant cause.

---

1. feat(eval): expose the shortlist funnel loss

recall@k measures the first-level candidate list. nDCG@k measures the ranked list. Neither can see the stage between them, which is where the pipeline loses the most: the shortlist handed to the eligibility reasoner is far shorter than the candidate list, and a relevant trial dropped there can never be ranked, however good the reasoning is.

Run	Track	First level	Shortlist	Size	Loss
qwen36_medcpt	2021	0.866	0.577	196	−33.4%
h100_baichuan	2021	0.866	0.581	197	−32.9%
qwen36_medcpt	2022	0.813	0.517	155	−36.4%
l40_trec23	2023	0.644	0.254	243	−60.6%

The 2023 row shows why this stayed hidden. That run has the best nDCG@10 of any run (0.881) and the worst shortlist recall (0.254). Condensed nDCG only orders the judged trials that survive the funnel, so it cannot report the three quarters that never arrived.

evaluate() now reports shortlist_recall, shortlist_size, funnel_depth_loss and shortlist_selection_delta whenever top_trials.txt is present.

Where the loss comes from (TREC 2021):

Stage	Recall	Cause
Full first level (2000)	0.866	—
First-level top-200	0.595	Depth: −0.272 (94% of the loss)
Actual shortlist (196)	0.577	Ordering: −0.017 (6% of the loss)

shortlist_selection_delta is negative on all four runs (−0.032, −0.027, −0.022, −0.009): at the same depth, the criterion reranker plus RRF fusion currently select worse than taking the first-level top-N and doing nothing. Small beside the depth loss, but free to recover — and it means the second level should be re-tested rather than assumed helpful. Not addressed here.

---

2. feat(search): per-patient shortlist depth

Depth is 94% of the loss, and no single number fits. The depth a patient needs to reach 90% of its own first-level recall ranges from 50 to 1550 trials, spread evenly — 17 of 75 patients need ≤200, 21 need >700. Sizing for the worst case wastes about 65% of the reasoner's compute; sizing for the median drops the hard patients.

New search.shortlist.policy:

• fixed — existing divisor sizing, unchanged. Still the default.
• relative_to_max — keep trials scoring ≥ alpha × this patient's top score.

A peaked first-level score curve means retrieval was confident and few trials are plausible; a flat curve means many are. The cut is relative to the patient's own maximum because RRF scores are comparable only within one patient.

Tuning

Done offline by replaying first_level_scores.json from the completed runs — no GPU time. Compared against fixed depth at equal mean depth, which is the only fair test: a policy must spend compute better, not merely spend more.

Track	Gain at equal compute
TREC 2021	+0.017 to +0.029 recall (peak at mean depth ~305)
TREC 2022	+0.007 to +0.025 recall
TREC 2023	−0.010 to +0.006 — questionnaire topics, no gain

Two other policies were tried and rejected: cumulative score mass tracked fixed depth to within ±0.003 everywhere, and half-max curve width won only below ~350 mean depth and lost above it.

Two effects, reported separately

At alpha=0.25 the policy also chooses to go deeper (196 → 305 trials on 2021), lifting shortlist recall 0.609 → 0.710. That part is bought with compute. The equal-cost table above is the free part. Both are real and they should not be conflated.

Safety

• Default fixed, so enabling this is an explicit A/B, never a silent change. Verified: existing configs parse to policy: fixed.
• Degrades to the fixed depth when first-level scores are absent (resumed runs).
• Never exceeds what the reasoner can actually consume (rag.max_trials_rag), since trials past that cap get no eligibility output and vanish from the ranking.
• Writes shortlist_depth.json recording the decision and the depth the old sizing would have chosen.

---

Verification

• 433 tests pass; ruff check src/ tests/ clean.
• 12 new tests for the depth policy, 2 for the funnel metrics.
• The production choose_shortlist_depth() was replayed over the real runs and reproduces the offline simulator's depths and recalls exactly.
• No behaviour change under default config; runs predating top_trials.txt still evaluate, with funnel keys None.

Not in this PR

The negative shortlist_selection_delta, constraint/calculator tools for numeric criteria, and any iterative retrieval loop. This PR deliberately contains no agent loop — the measurements say depth allocation is the dominant lever, so it comes first and gets evaluated on its own.

🤖 Generated with Claude Code

[Copilot]

Pull request overview

This PR extends the TREC qrels evaluation to expose “funnel” metrics that quantify recall loss between the first-level candidate list and the second-level shortlist (when top_trials.txt exists), and adds tests to validate the new outputs.

Changes:

• Add parsing of top_trials.txt and compute shortlist/funnel metrics in evaluate().
• Include new funnel metrics in the returned per-query rows and mean aggregate.
• Add unit tests covering funnel metric reporting and behavior when the shortlist file is absent.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File	Description
tests/test_qrels_eval.py	Adds tests asserting new funnel metrics and their absence (None) when top_trials.txt is missing.
src/trialmatchai/trec/qrels.py	Implements shortlist parsing and aggregates new funnel metrics into evaluate() results.

Suppressed comments (1)

src/trialmatchai/trec/qrels.py:245

• Funnel metrics are currently computed only when shortlist is truthy and retrieved is truthy. This skips reporting when top_trials.txt exists but is empty (or when the first-level list is empty), despite the docstring/PR intent to report whenever the shortlist file is present. Using an explicit None sentinel (see _shortlist_for_patient) and checking is not None avoids conflating “absent” with “empty” and still keeps metrics None when the file truly doesn’t exist.

        shortlist = _shortlist_for_patient(patient_dir)
        if shortlist and retrieved:
            depth = len(shortlist)
            short_r = recall_at_k(shortlist, rel_set, depth)
            first_r = recall_at_k(retrieved, rel_set, depth)

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.