feat(d2): normalized gain + paired-difference significance#49
Merged
Conversation
…ficance Two statistical upgrades to the effectiveness (D2) dimension, both grounded in the deep-research survey (SkillsBench + Miller 2024, arXiv 2411.00640): - Normalized gain g = (skill - baseline) / (1 - baseline) corrects the raw lift for baseline headroom, so a small lift on a high baseline reads as the strong result it is. Reported in the scorecard D2 line and JSON. - Significance now runs on the per-eval PAIRED difference when >=2 evals are present in both configs: shared per-eval difficulty cancels, shrinking the SE so a real lift clears the noise with fewer trials. Falls back to the unpaired se_of_difference for a single eval (no between-eval variance to exploit). New stats helpers (normalized_gain, paired_diff) are pure and unit-tested; aggregate/dimensions/scorecard thread the new fields through. baseline_lift and its unpaired SE are unchanged for backward compatibility.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two statistical upgrades to the effectiveness (D2) dimension, the two highest value-per-effort items from the deep-research competitor survey (SkillsBench, and Anthropic's "Adding Error Bars to Evals", arXiv 2411.00640):
g = (skill − baseline) / (1 − baseline), reported alongside absolute lift in the scorecard D2 line and JSON. Corrects for baseline headroom: a +5pp lift on a 0.9 baseline (g=0.5) reads as the strong result it is, not a rounding error next to a +5pp lift on a 0.4 baseline (g≈0.08).se_of_differencefor a single eval (no between-eval variance to exploit).baseline_liftand its unpaired SE are unchanged, so existing consumers keep working; the newnormalized_gain/paired_lift/paired_sefields are additive.Design notes
stats.normalized_gainandstats.paired_diffare pure, dependency-free, and unit-tested;aggregate/dimensions/scorecardthread the fields through.Test evidence
Dogfooded end-to-end on a two-eval benchmark — D2 line renders:
pass 68% ± 25%, lift +20% (significant), gain +0.38, and the paired path flips a lift that the unpaired SE would call n.s. to significant.🤖 Generated with Claude Code