feat(benchmark): Signal-to-noise analysis layer: SNR, scenario-health audit, contamination defense

[tcballard]

Applies the lessons from OpenAI's "Separating signal from noise in coding evaluations" and the aligned Ai2 "Signal and Noise" framework. Everything here is reporting/analysis-layer and results-neutral — verified byte-identical arms/cells/stats/per_scenario before/after; only a second paired series is added. Full suite green (462 passed, 6 skipped).

What the articles say, and where we stood

Lesson	Before → After
Separate genuine failures from harness/grading noise	Already done (error taxonomy + exclusion; deterministic scorer avoids LLM-judge noise) — now cited in Limitations
Report signal-to-noise, not bare point estimates	New — scoring/snr.py
Per-subtask decomposition; flag low-signal subtasks	New — scoring/health.py discrimination audit
Broken/ambiguous tasks distort the ceiling	New — health audit classifies broken/tie scenarios
Contamination (real PEPs/RFCs are pretrained)	New — no_grounding floor surfaced as the contamination control

Changes

Both-contrasts paired diff (scoring/crossover.py) — _paired now computes the within-seed paired adherence difference for a list of arm pairs, defaulting to DEFAULT_PAIRS = rac-vs-naive_rag (H1) and rac-vs-no_grounding (H2). A contrast whose arms aren't both in the sweep is silently skipped; pair= still accepted for back-compat.

Signal-to-noise (scoring/snr.py) — SNR = |paired diff_mean| / across-seed diff_std per contrast per N. The article's core caveat is baked in: noise cannot be estimated from one seed, so n_seeds < 2 reports "not estimable" rather than a fabricated ratio. Distinguishes noise-dominated (SNR < 1), clean separation (σ=0, nonzero gap), and no-effect. Headline quoted at the largest N.

Scenario-health audit (scoring/health.py) — classifies every scenario as broken (ceiling context_dump never adheres), contaminated (floor no_grounding adheres → parametric memory suffices), tie (no separation), or discriminating, with summary counts — the analogue of OpenAI's "X% of tasks are broken" ceiling analysis.

Surfacing (scripts/report.py, runner/dashboard.py) — a Signal-to-noise section and a Scenario-health section in the report; a new "Signal / noise" dashboard tab (with the template CSS/tab indices kept in lockstep, guarded by a test). All .get-guarded so legacy/single-seed/no-rac datasets render gracefully.

Methodology + wiring (spec/methodology-signal-to-noise.md, Makefile, run_real.sh, Limitations prose) — documents the SNR definition, the health classes, and the contamination defense (synthetic scenarios are contamination-proof; the floor detects it in real ones), mapping each article lesson to a mechanism. An additive methodology note, not a pre-registration amendment — no gold labels, gates, or the frozen H1/H2 tests change. The real-crossover arms already include rac + no_grounding, so both contrasts populate; the docs now flag that SEEDS≥2 is required for the SNR section.

Testing

• New: test_snr.py, test_scenario_health.py, test_report_snr.py; extensions to test_real_curve.py (both contrasts) and test_dashboard.py (new tab + tab/section lockstep).
• Results-neutrality checked explicitly (old single-pair vs new default → adherence/cells/stats identical).
• Offline e2e: a 3-seed sweep renders the SNR + health sections; the deterministic stub shows "clean (σ=0)" (perfect separation every seed), and a single-seed run shows "noise not estimable".

Independent of PR #20 (roster Step 1); branched from main.