Skip to content

feat(d2): normalized gain + paired-difference significance#49

Merged
DCCA merged 1 commit into
mainfrom
feat/normalized-gain-paired-lift
Jul 2, 2026
Merged

feat(d2): normalized gain + paired-difference significance#49
DCCA merged 1 commit into
mainfrom
feat/normalized-gain-paired-lift

Conversation

@DCCA

@DCCA DCCA commented Jul 2, 2026

Copy link
Copy Markdown
Owner

Summary

Two statistical upgrades to the effectiveness (D2) dimension, the two highest value-per-effort items from the deep-research competitor survey (SkillsBench, and Anthropic's "Adding Error Bars to Evals", arXiv 2411.00640):

  1. Hake normalized gaing = (skill − baseline) / (1 − baseline), reported alongside absolute lift in the scorecard D2 line and JSON. Corrects for baseline headroom: a +5pp lift on a 0.9 baseline (g=0.5) reads as the strong result it is, not a rounding error next to a +5pp lift on a 0.4 baseline (g≈0.08).
  2. Paired-difference significance — significance now runs on the per-eval paired difference when ≥2 evals appear in both configs. Shared per-eval difficulty cancels, shrinking the SE so a genuine lift clears the noise with fewer trials. Falls back to the unpaired se_of_difference for a single eval (no between-eval variance to exploit).

baseline_lift and its unpaired SE are unchanged, so existing consumers keep working; the new normalized_gain / paired_lift / paired_se fields are additive.

Design notes

  • New stats.normalized_gain and stats.paired_diff are pure, dependency-free, and unit-tested; aggregate/dimensions/scorecard thread the fields through.
  • Significance is still decided at full precision (before rounding) so display rounding can't flip the verdict.

Test evidence

169 passed in 0.53s          # +6 new tests (stats, aggregate, scorecard)
50 files already formatted    # ruff format --check
All checks passed!            # ruff check
self-validation: 100 / A / Ship

Dogfooded end-to-end on a two-eval benchmark — D2 line renders:
pass 68% ± 25%, lift +20% (significant), gain +0.38, and the paired path flips a lift that the unpaired SE would call n.s. to significant.

🤖 Generated with Claude Code

…ficance

Two statistical upgrades to the effectiveness (D2) dimension, both grounded in
the deep-research survey (SkillsBench + Miller 2024, arXiv 2411.00640):

- Normalized gain g = (skill - baseline) / (1 - baseline) corrects the raw lift
  for baseline headroom, so a small lift on a high baseline reads as the strong
  result it is. Reported in the scorecard D2 line and JSON.
- Significance now runs on the per-eval PAIRED difference when >=2 evals are
  present in both configs: shared per-eval difficulty cancels, shrinking the SE
  so a real lift clears the noise with fewer trials. Falls back to the unpaired
  se_of_difference for a single eval (no between-eval variance to exploit).

New stats helpers (normalized_gain, paired_diff) are pure and unit-tested;
aggregate/dimensions/scorecard thread the new fields through. baseline_lift and
its unpaired SE are unchanged for backward compatibility.
@DCCA
DCCA merged commit 02c0108 into main Jul 2, 2026
1 check passed
@DCCA
DCCA deleted the feat/normalized-gain-paired-lift branch July 2, 2026 16:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants