Add between-subjects support to compare() (rudimentary) - #11
Merged
ianarawjo merged 30 commits intoAug 17, 2026
Conversation
Real-data validation (ppi_real.py) surfaced two bugs the synthetic harness never triggered: - wilcoxon's cross-fit degenerate-variance guard (_walsh_theta_fold_lambda and 4 sites sharing the same idiom in evalstats/ppi.py) only checked the labeled-side variance relative to the judge-side variance, missing the case where the judge side is ALSO exactly degenerate (common with real Likert-tied data at small fold sizes). A spurious lambda=0 in one fold silently zeroed out the other fold's entire contribution to both the point estimate and its variance, driving real-data Type-I error as high as 0.515 (nominal 0.05) on some judge pairs. - _ppi_kruskal_wallis_pairwise (and its mnar_experimental sibling) crashed with a ZeroDivisionError whenever a genuine, strong real effect drove its bootstrap covariance to exact zero (np.linalg.pinv can't represent "infinite precision", collapsing wald_stat/df to 0). The real-data harness silently counted every crash as "failed to detect", collapsing reported power from 0.83 (uncorrected) to 0.20 (corrected). Also added a realism improvement to the real-data harness's paired/ repeated Type-I null checks: they previously copied the exact same human label onto every arm (Y_lab identically 0), which is not just what triggered the wilcoxon bug but an unrealistically idealized worst case -- real independent human ratings never agree to floating-point precision. generate_real_paired_null_cell/generate_real_omnibus_repeated_null_cell now mix in independent small-noise label copies (90% of reps, sigma=0.03) alongside the exact-tie construction (10%), still a valid null (mean-zero noise keeps the true difference exactly 0) but more representative. Full official real-data re-run (reps=200, all checks): wilcoxon corr max 0.515->0.105, mean 0.098->0.052; kruskal corrected power 0.196->1.000; Holm-confirmed miscalibrated cells 37/2208->1/2208. Added regression tests for both degenerate cases. See Addendum 39 in simulations/out/results_why_ppi_shrink_1_over_0.md for the full investigation. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
_ppi_anova_independent_f_stat's power_tune=True branch had each of the k groups independently estimate its own PPI power-tuning lambda from that group's own labeled subsample. Since lambda is chosen specifically to minimize that group's own reported variance using that same finite sample's noisy moments, the reported variance carries a systematic "argmin-then-evaluate-at-the-argmin" optimism bias, distinct from lambda's own sampling uncertainty (which _lambda_var_inflation already corrects for separately). Confirmed directly on real data: mean(denom) was ~14% too small relative to mean(ss_between)/(k-1) under a genuine null (ratio 1.136, should be ~1.0) -- absent under power_tune=False (ratio 0.970), isolating this specifically to adaptive power-tuning. Fix: estimate lambda ONCE, pooled across all k groups' labeled+unlabeled data (new evalstats/ppi.py:_pooled_k_group_lambda, generalizing the existing 2-group _pooled_two_group_lambda ttest already uses), instead of each group independently. Pooling increases the effective sample lambda is estimated from, shrinking the optimism gap. This is a different mechanism from Addendum 37's earlier (rejected) k-group-pooled-lambda attempt, which was evaluated only against a distinct MNAR point-estimate bias that pooling doesn't address -- this fix targets an MCAR-relevant variance bias instead, and is validated accordingly. Validated via ground-truth Monte Carlo: 18 real-data MCAR null cells across 5 datasets (never worse than the per-group construction, several cells dropping from ~1.4-1.5x nominal to within noise of nominal), 7 synthetic null scenarios including 2 MNAR (no regression, some improvement), and power checks on both real and synthetic data (power unchanged or mildly improved everywhere tested, no cost). 30/30 anova_ind unit tests and the full 378-test tests/test_ppi_corrections.py + tests/test_p_values.py suite pass. Official synthetic harness re-check (reps=500 Type-I + 3 power sweeps) confirms: corr max 0.086/mean 0.051, 0 Holm-confirmed miscalibrated cells, smooth well-calibrated power curves. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
wilcoxon's Walsh-average theta is proportion-like on [-0.5, 0.5], so (as
with a binomial's p(1-p)) its sampling variance is maximal at theta=0 and
collapses toward the boundaries -- measured 0.0166 at true theta=0 vs 8e-6
at 0.499, giving corr(sqrt(var), |theta_hat|) = -0.88..-0.95. The plug-in
("Wald") variance is evaluated AT the observed estimate, so a large
|estimate| mechanically arrives with a small se and a two-sided test is
inflated in both tails. This is a property of the ESTIMAND, not of lambda,
which is why fixed lambda=1 was always well calibrated: adaptive lambda
shrinks the correction toward f_lab and concentrates the statistic on the
small labeled sample where the coupling bites, exposing it rather than
creating it.
Evaluate the human term's variance UNDER H0 instead (a score rather than a
Wald construction -- the same reason this package prefers Wilson over Wald
for binary and Tango for paired binary). Under H0 the Walsh count is the
Wilcoxon signed-rank statistic, whose null law is distribution-free;
sign-flip randomization obtains it exactly, handling the heavy ties where
the closed form (2n+1)/(6n(n+1)) is 3.5x wrong on real 88%-tied judge data.
A null variance is constant w.r.t. theta_hat, so unlike a variance-
stabilizing transform it cannot diverge at the boundary where real effects
live. The substitution keeps the estimated correlation and rescales only the
human side, since replacing var_lab alone breaks the quadratic form's
Cauchy-Schwarz consistency (measured: 9% of samples clamp to ~0 se).
Full official ppi_real (reps=200, seed=46, all six corpora), wilcoxon over
192 matched cells: Type-I max 0.1050 -> 0.0800, pooled 0.0522 -> 0.0428
(z=-6.09), cells>0.075 10 -> 1, CI coverage 0.941 -> 0.945, CI width 0.2199
-> 0.1621 (-26%), power unchanged at 1.000. Whole table: Holm-confirmed
miscalibrated cells 1/2208 -> 0/2208. A narrower interval with better
coverage is the direct evidence that cross-fitting's 5-17% SE inflation was
wasteful rather than protective. Of the twelve tests exactly two moved --
wilcoxon (this change) and anova_ind (whose baseline predates 3d64d1f); the
other ten are bit-for-bit identical.
Removes _walsh_theta_fold_lambda and _WILCOXON_CROSSFIT_COV_COEF;
_cross_fit_satterthwaite_df is retained (ttest's two-sample path uses it).
power_tune=False is deliberately untouched -- that path is long-validated
and serves as the harness's classical baseline. Nine other approaches were
tried and rejected first, including an arcsine transform that looked best
synthetically then collapsed real power to 0.462. See Addenda 41/42 in
simulations/out/results_why_ppi_shrink_1_over_0.md.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
ianarawjo
force-pushed
the
worktree-between-subjects-support
branch
from
August 16, 2026 15:53
2a59d1e to
c554c82
Compare
… figures The label-efficiency multiplier is not measured directly -- it is obtained by INVERTING the classical reference curve (equiv_n_lab = interp(ppi_power, power_grid, n_grid)). That inversion's gain dN/dP is 800-1250 labels per unit power wherever the curve is flat, so a binomial SE of 0.02 on ppi_power becomes +/-16-25 equivalent labels -- at n_lab=15 that is +/-1.05x on the multiplier. Measured at the old single effect size (frac=0.15): predicted multiplier sd from binomial noise alone (1.41) EXCEEDED the observed scatter (0.56), so the sub-1.0x values visible in that sweep were inversion artifacts, not PPI underperforming a human-only test. Changes: * Sweep PPI_LABEL_EFF_EFFECT_FRACS = (0.15, 0.20, 0.25, 0.35) instead of a single frac. One effect size cannot keep the whole N_lab grid in the curve's steep middle; the eval types also peak at DIFFERENT fracs (binary ~0.20, continuous ~0.35, roughly 1.75x apart). Chosen by scanning all three eval types: reaching below 0.15 is near-dead for continuous (0/8 usable cells) and likert (0/8), while frac=0.50 degrades to 0/8 usable and 3/8 saturated at the 2.5-3.5x multiplier binary actually achieves. Both source builders take effect_frac; scenario names embed ".es=<frac>". * Smoothed, strictly-monotone reference curve (_smooth_monotone_power_curve). A BIAS fix, not cosmetic: at ref_n_mc=3000 the raw MC curve ties across adjacent grid points, and inverting a tie resolves to the lowest tied N, biasing equiv_n_lab downward exactly in the flat small-n_lab region. Measured: n_lab=15/r=0.30 goes 1.06x -> 1.52x. * CIs on every multiplier (_multiplier_ci), propagating ppi_power's binomial SE through the curve's local slope. These are wide and that is the point: at reps=200 most individual cells do not exclude 1.0x, so reporting point estimates alone overstates precision. * n_grid cap 500 -> 1500 (28 -> 36 points) in both the label-efficiency and N-formula checks. np.interp CLAMPS at the endpoints, so the cap was a hard ceiling on any reportable multiplier: binary's kappa=0.80 tier reaches ~4x, needing equiv ~800, but could only report 500/200 = 2.50x -- silently truncating the BEST-performing eval type into looking worse than likert. * Figures: save_ppi_label_efficiency_invariance_plot (multiplier vs effect size, one line per judge-quality tier -- flat lines mean the multiplier is a property of the judge, which is what licenses pooling across arms) and save_ppi_label_efficiency_threshold_plot (multiplier vs judge-human agreement, with a shaded <1.25x "not worth the trouble" band, because a multiplier can be statistically above 1.0 while being practically pointless -- at agreement 0.4 the medians are 1.14x/1.01x/1.03x). Both are emitted automatically alongside the pooled and per-arm plots. * effect_frac, multiplier_lo and multiplier_hi are now columns in the results CSV, and effect_frac in the raw per-method CSV, so the arms stay separable and the invariance check is reproducible from the artifacts. Validated on a reps=200 sweep: 823/840 arm-pairs agree within 95% CI (98.0%; continuous 100%, likert 98.2%, binary 95.7%), saturation down to 15/576, and median CI width falls monotonically with effect size (0.53 -> 0.23). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Follow-up polish to ebc964f's saturated-point handling: - Shrink the caret from 11pt to 7pt (and its edge from 0.8 to 0.7). At 11pt, a marker pinned at y_max with clip_on=False projected far enough above the axes to overlap the subplot titles. - Pad the subplot titles by 10pt, since the caret still deliberately straddles the axis line (that overhang is what makes it read as "runs off the chart" rather than "sits at the top of the chart"). - Shorten the legend entry to "saturated". The longer wording widened the legend box enough to squeeze the panels; the caret's meaning as a lower bound now belongs in the figure caption. - Raise the main title (suptitle y=0.99) and lift tight_layout's rect top from 0.94 to 0.96. The rect change is load-bearing: without it tight_layout reclaims the new headroom for the axes and the gap closes straight back up. Rendered against the reps=200 sweep to confirm no marker/title overlap remains in any panel. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…fficiency
The label-efficiency rule of thumb was stated in three different metrics
(kappa for binary, quadratic weighted kappa for likert, Pearson r for
continuous), which left open the reviewer question of whether the cross-type
agreement was an artifact of that choice. It was not, but the unifying
quantity turned out to be none of the three: it is rho^2, the squared
within-group correlation between judge score and human label.
PPI++ with tuned lambda is a control variate, so
saving = 1 / (1 - rho^2 * (1 - n_lab/N))
Validated over a 48-cell grid (3 eval types x 4 judge-noise x 4 judge-bias,
3000 reps each) against measured Var(human-subset)/Var(PPI): R^2=0.9968, mean
error -0.15%, max 5.5%, under ADAPTIVE power-tuned lambda. Pooled across all
three eval types rho^2 scores R^2=0.975 with a 1.07x spread at matched value,
against ICC/CCC at 0.703/1.58x and Krippendorff's alpha at 0.553/1.87x.
rho^2 wins because it is invariant to judge BIAS, correctly: PPI's rectifier
removes additive bias, so at fixed noise a 4x bias increase drops ICC
0.857 -> 0.532 while the realized saving holds at 3.30x -> 3.19x. An
agreement-metric threshold would discard a judge still worth 3.2x.
Changes:
- _alignment_metric_dict now carries pearson_r for ALL three eval types (it
was continuous-only) plus a derived rho2, so the axis is measurable on a
common footing. Also adds the full IRR panel a reviewer may ask for:
Krippendorff's alpha (nominal/ordinal/interval), Lin's CCC, Gwet's AC1,
PABAK, Kendall's tau-b, linear-weighted kappa, ICC(2,1) for binary.
Library implementations used wherever they exist (sklearn/scipy); only
alpha, AC1 and CCC are hand-rolled, each validated against an independent
reference rather than a recalled constant.
- _ppi_predicted_savings implements the formula, with the derivation and an
explicit warning against the asymptotic 1/(1-rho^2) form, which overstates
badly for a strong judge (claims 100x against a measured 40x at rho^2=0.99)
because sensitivity to the correction grows with judge quality.
- _calibrate_noise_for_alignment returns the full realized metric panel
instead of discarding all but the tuned metric; it already computed it.
- Calibration CSV gains rho2/pearson_r and the rest of the panel; results CSV
gains rho2, predicted_mult, predicted_mult_asymptotic.
- Main label-efficiency plot draws a per-tier dotted prediction curve. It sits
ABOVE the measured line at strong-judge tiers by construction: the
prediction is variance-scale, equiv_n_lab inverts a saturating power curve.
Two bugs found while validating the new metrics: Gwet's AC1 had a spurious
factor of 2 in its chance term (divide-by-zero on balanced binary marginals),
and PABAK used the binary-only 2*Po-1, returning -0.303 for a judge
simultaneously scoring weighted kappa 0.60; it now uses the K-category
Brennan-Prediger form, which reduces to 2*Po-1 at K=2.
Note kappa == quadratic weighted kappa == ICC(2,1) exactly for two raters
(residual scales as 1/n) -- a real unification, but not the right axis, and it
requires passing an explicit category grid or an absent category shifts kappa
by up to 0.044.
Existing primary metrics are unchanged and verified bit-identical across 355
values, so prior results remain comparable.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
ianarawjo
force-pushed
the
worktree-between-subjects-support
branch
from
August 16, 2026 23:47
c554c82 to
1f925b2
Compare
…l types The axis was calibrated and labeled per eval type (kappa for binary, quadratic weighted kappa for likert, Pearson r for continuous), so a shared "IRR~=0.8" legend entry meant three different things: those tiers realize rho^2 = 0.667 / 0.683 / 0.640. Close enough to look alignable while asserting an equivalence that does not hold, which made the panels not directly comparable. rho^2 is what actually predicts the multiplier (control-variate saving is 1/(1 - rho^2*(1 - n_lab/N))), so calibrating on it directly makes a tier mean the same judge quality in every panel and puts the axis in the units the rule of thumb is stated in. Predicted savings are now identical across eval types at each tier (3.23/3.22/3.22 at rho^2=0.70; 1.97/1.97/1.97 at 0.50), where before they differed by tier definition. - _LABEL_EFF_ALIGNMENT_METRIC: rho2 for all three; legend reads "ρ²~=". - Tier ladder retargeted onto the rho^2 scale: (0.70 ... 0.20), straddling the rho^2=0.5 rule-of-thumb threshold rather than sitting at an edge. The top is 0.70 because BINARY CANNOT EXCEED rho^2 ~= 0.733 at any noise level -- its held-constant flip-probability bias caps phi -- so a 0.8 tier would silently under-deliver on one arm. _NFORMULA_ALIGNMENT_TARGETS moved to match. - Threshold/invariance plot x-axis relabeled to name rho^2 explicitly. Two bugs fixed in the process: _calibrate_noise_for_alignment cannot bisect on rho^2 directly. For binary, llm_noise is a flip PROBABILITY, so past 0.5 the judge is systematically inverted and rho^2 -- which discards the sign -- climbs back toward 1 (measured: r goes 0.850 -> -0.010 at noise 0.50 -> -1.000 by 2.0, so rho^2 traces 0.72 -> 0.00 -> 1.00). The monotone-decrease assumption sailed past the zero crossing and converged on a perfectly ANTI-correlated judge reported as rho^2=1.0, on every binary tier. It now bisects on the SIGNED pearson_r against sqrt(target), which is monotone across the whole range, and reports rho^2 from the same final measurement. All 18 tiers now land within 0.0006. The old per-type metrics never hit this because kappa also goes negative. fit_nformula_rule_of_thumb.py computed 1 - alignment_value**2, which was already wrong (it squared kappa as though it were a correlation for binary and likert) and would now be wrong twice over, since alignment_value IS rho^2. It takes 1 - alignment_value. Also stops clamping the predicted-savings curve to the axis ceiling; clamped points drew a flat run along the top edge that read as a real measurement topping out. They are dropped instead. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Moves _detect_paired out of labeling.py into a new zero-dependency leaf module, evalstats/core/design.py, as a public detect_paired(). Pure refactor -- labeling.py re-exports the same function under its old private name so its one call site is unchanged. This lets compare()'s upcoming design="auto" routing share the exact same implementation without importing the labeling-CLI-focused module. Added tests/test_design.py since no test coverage existed for this function at all before now (pre-existing gap, not introduced here) -- covers paired/unpaired/threshold-boundary/no-factor/single-level cases, plus an identity check that labeling.py's re-export isn't a copy. Also checked in the finalized architecture plan (PLAN_between_subjects_ extension.md) and this session's own working instructions (IMPLEMENTATION_INSTRUCTIONS.md) for reference through the rest of this multi-phase implementation. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…rwise Two additive pieces of Phase 1 groundwork: 1. AUTO_UNPAIRED_METHOD_TABLE in config.py: the compare(design="unpaired") routing table. Two rows in practice -- binary -> anova_oneway (omnibus) + ttest (pairwise, Welch's); continuous/likert/grade -> kruskalwallis (omnibus + pairwise) + mannwhitney (k=2 special case). Matches the finalized decisions in PLAN_between_subjects_extension.md exactly, with the same reasoning inline as "reason" strings, following AUTO_ANALYZE_METHOD_TABLE's existing convention. 2. _ppi_kruskal_wallis_pairwise (evalstats/tests/__init__.py) now also returns "boots" and "pair_p" (per-pair two-sided bootstrap p-values, same convention as TestResult.corrected_p_value) alongside its existing keys -- purely additive, no existing caller reads new keys. Needed so the unpaired dispatcher can Holm-correct a family of per-pair p-values; the function only exposed a single omnibus wald_p before. Verified: the public kruskalwallis() wrapper (which doesn't read these new keys) is unaffected, and the 37 existing kruskal- related tests in tests/test_ppi_corrections.py still pass unchanged. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The core of Phase 1: compare_unpaired(), a sibling to core/paired.py's
machinery, not a branch inside it -- the paired path's whole design
assumes item-matched differences, which has no meaning for genuinely
disjoint groups.
- GroupStat / GroupDiffResult / GroupComparisonResult: new result
dataclasses (not PairedDiffResult/AnalysisBundle -- those carry
per_input_diffs, which doesn't exist here). .to_dict()/.to_frame()
included from the start (confirmed load-bearing -- evalstats is a
library API, not just a reporting tool). .plot() raises
NotImplementedError for now (deferred phase).
- Two test families per AUTO_UNPAIRED_METHOD_TABLE: binary ->
anova_oneway (omnibus, k>=3) + pairwise Welch's-style Δp CIs; everything
else -> kruskalwallis (omnibus, k>=3) + pairwise θ=P(a>b) CIs. Both
families' pairwise engines work uniformly at any k>=2 (Bonferroni/Holm
correction is a no-op at a family of 1 pair), so k=2 needs no special
case -- it's just the k>=3 pairwise machinery with an empty omnibus.
- Two independent correction axes matching the paired path's own
convention: Bonferroni for the pairwise CIs (not Šidák -- needs no
unverified independence assumption on this bootstrap's correlation
structure), Holm for the pairwise p-values (reusing
core/stats_utils.correct_pvalues, already used elsewhere).
- Non-PPI pairwise post-hocs for both families are new, small additions
(kruskalwallis() only ever populated its pairwise breakdown under PPI;
a plain-bootstrap and closed-form-Welch's analog didn't exist before).
- Synthetic-item-column fallback (mirrors labeling.py's) so data with no
natural row/reviewer id doesn't hard-error before this code runs.
- Per-group descriptive stats reuse the exact machinery
evalstats.quick.summarize() uses internally (resolve_auto_robustness_
method + robustness_metrics), called directly with multi_ci=True to get
gradient CI bands without touching summarize()'s own public signature.
- core/summary_unpaired.py: .summary() printer, reusing core/summary.py's
low-level rendering primitives (_choose_interval_line et al. -- these
already take plain floats, no bundle object needed) and mirroring the
paired path's pink PPI banner exactly, printing the caller's real
AlignmentResult inline.
Found and fixed one integration bug while smoke-testing: every
evalstats.tests function with labels prints its own internal alignment
report unconditionally (not gated by print_result -- existing, validated
behavior for direct callers, left untouched), which produced a confusing
duplicate/stale ("selection=unknown") report when called internally here
alongside our own correctly-disclosed one. Fixed by suppressing stdout
around just that internal call, not by touching evalstats.tests.
Verified end-to-end via direct smoke tests (not yet wired into compare()):
k=2 and k>=3, both families, with and without PPI, unbalanced groups, and
the synthetic-item-column fallback -- all produce correct, readable
output. Also added a brief clarifying note when the Bonferroni-CI verdict
and the independently Holm-corrected p-value rarely disagree right at the
boundary (both valid, just different corrections).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
_ppi_kruskal_wallis_pairwise's finite-sample F-correction divided by nu * df unconditionally once nu > df, but df (the covariance's own matrix rank) can come back 0 when the covariance is numerically degenerate -- most likely at k=2, where there's only one pair and a 1x1 covariance can collapse to ~zero variance for some data/seed combinations. Found by the between-subjects battle-test grid (6/192 cells crashed). Guard df == 0 explicitly and report no evidence of a difference (wald_p = 1.0) rather than dividing by zero; only the exact function on the call path is touched, not its MNAR-experimental twin.
Adds design: Literal["auto","paired","unpaired"] to compare(), inserted as a self-contained block right before the existing single-factor dispatch paths -- it either returns/raises immediately or falls through completely unchanged, so paired-path behavior is provably untouched for every existing call (design="paired" never evaluates the new branches; design="auto" on genuinely paired data calls the read-only detect_paired() and falls through). design="auto" on between-subjects data raises a clear ValueError instead of silently forcing a paired analysis onto it. Also folds in several fixes found during integration review and battle-testing of compare_unpaired(): - NaN in the metric column now drops-and-warns per group (matching evalstats.quick's convention) instead of silently poisoning CIs or crashing inside a Wilson-CI helper; group_arrays/group_lab_arrays are built from the same per-row mask so PPI labels stay aligned. - PPI dispatch now runs through the same label-sanitization guards every other PPI caller has (>=15-label floor, zero-labeled-group check) -- it was calling the private pairwise engines directly and skipping them. - method=/backend=/score_range= are now handled explicitly: score_range= threads through to the per-group CI's auto-method resolution; explicit method= overrides raise a clear "not supported" error instead of being silently dropped (the auto-detect warning was also actively misleading when this happened). - Multi-run (seeded) data and secondary_metric=+run_col combinations now raise clear guidance instead of silently inflating N or being ignored. - AUTO_UNPAIRED_METHOD_TABLE's family field is now the actual dispatch key (previously re-derived from pairwise_method by string comparison elsewhere, so editing the table didn't change behavior). - p_values=/omnibus= are now honored under design="unpaired", with unpaired-specific defaults of True (compare()'s own default is False, but unset here preserves the always-shown behavior this path was built and battle-tested with). Adds secondary_metric= (Pareto-front analysis) support for design="unpaired": pareto_bootstrap_unpaired() in core/pareto.py is a new function (the existing paired-path pareto_bootstrap() is untouched) that resamples each group's own rows independently rather than sharing one per-item index across entities, since disjoint groups have no shared item pool to preserve correlation through -- it still preserves each row's own primary/secondary pairing. classify_pareto_status() is reused unchanged. GroupComparisonResult.pareto_status/ pareto_frontier_probability mirror ComparisonResult's own attributes. Battle-tested: 192/192 crash-grid combinations (score_type x k x balance x PPI x seed), 48/48 Pareto-grid combinations, Type-I error calibration on target (~0.04-0.06 at nominal alpha=0.05) with high power under real effects.
The unpaired summary printer had grown its own parallel implementations of things the paired path already renders -- same visual style today, but two copies that could silently drift apart. Unifies all three: - _print_ppi_banner(): extracted from a copy-pasted block in both _print_bundle_summary and print_group_comparison_summary into one shared function. - _print_mean_advantage(): generalized to take plain per-entity arrays (labels/mean/std/ci_low/ci_high/multi_ci) instead of requiring a paired-specific RobustnessResult, so both paths call the literal same function. Verified byte-identical paired output before/after. - _print_pairwise_section(): the bigger piece. Refactored into _prepare_paired_pairwise_rows()/_prepare_unpaired_pairwise_rows(), each resolving their own design's method-specific logic (six CI/ p-value method families plus Friedman/Nemenyi for paired; one fixed Bonferroni-CI/Holm-p scheme for unpaired) into a common row+metadata shape, feeding one shared axis/legend/header/row-rendering core. Verified byte-identical paired output across three scenarios (default Wilcoxon/Romano-Wolf, binary/Newcombe, explicit bootstrap) plus a manual Nemenyi check, by diffing snapshots taken before the refactor. The Behavioral Agreement (McNemar-style pass/fail) subsection is pulled into its own _print_behavioral_agreement_section(), paired- only, since it needs the same item scored by both entities -- no between-subjects equivalent exists. - _print_pareto_section(): already dict-generic (only ever reads .labels/.mean/.ci_low/.ci_high off the pareto dict's robustness objects), so a small adapter (_GroupStatsAsRobustness in core/unpaired.py) lets it render the unpaired case -- including the ASCII scatterplot -- with no new display code at all. Net effect: the unpaired pairwise table now shows an interval-plot bar per comparison and the dominance family's theta as a signed deviation from its null (Deltatheta, so the shared zero-centered axis math applies uniformly) with p-values and significance stars, replacing the old text-only "Verdict: significant (A < B)" column. Numbers are unchanged (Deltatheta = theta - 0.5, same underlying estimate); only the presentation changed, to match the paired path exactly.
tests/test_unpaired.py: 52 tests covering compare_unpaired(), design= routing in compare(), NaN/PPI guards, p_values=/omnibus= toggles, Pareto-front support, and (for the shared rendering functions) a tripwire asserting they resolve to core.summary, not a reimplementation. simulations/investigate_unpaired_battle_test.py: crash/sanity grid (score_type x k x balance x PPI x seed), Type-I/power calibration check, and a Pareto-front crash grid -- kept in the repo matching the existing simulations/investigate_*.py convention. EXECUTIVE_SUMMARY.md: session report.
EXECUTIVE_SUMMARY.md, IMPLEMENTATION_INSTRUCTIONS.md, and PLAN_between_subjects_extension.md were sitting at the repo root, cluttering it alongside the actual package. Moved into a new notes/ folder; updated the handful of code comments that referenced their old root-level path.
… bands Two PR review requests addressed together, since they touch the same code: the between-subjects console printer (print_group_comparison_ summary) had grown into its own file mostly to avoid bloating summary.py further, but that made "is this actually shared or just parallel" harder to see at a glance. It now lives directly in core/summary.py next to the paired-path functions it calls -- deleted core/summary_unpaired.py entirely (its deletion landed in the prior commit due to a staging order accident; functionally it's part of this one). Also closes a real gap: GroupComparisonResult's own docstring said "no executive summary, critical-difference rank bands" -- unlike every other piece of unpaired rendering, these two were never connected to the paired path's existing machinery. Both turned out to be reusable outright: _critical_difference_groups/_assign_significance_groups/ _print_executive_summary only ever read a PairwiseMatrix-like .get(a,b) lookup and a RobustnessResult-like .mean/.ci_low/.ci_high -- no ranking-bootstrap dependency despite the paired path deriving their label order from one. Two small adapters in core/unpaired.py (_GroupDiffResultsAsPairwiseMatrix, _GroupComparisonResultAsBundle, alongside the existing _GroupStatsAsRobustness) let both paired-path functions render the unpaired case unmodified -- rank order comes from sorting by mean descending instead of a rank_dist bootstrap, and significance uses the same CI-exclusion check GroupDiffResult.significant already does (via a simultaneous_ci_method sentinel on the adapter that routes _critical_difference_groups away from its p-value-threshold branch, which this engine has no equivalent of). Positioned identically to the paired path: pairwise table (with CD bands in its footer) -> Pareto section, when secondary_metric= was passed -> executive summary (showing the Trade-off column when Pareto is present) -> Pareto callout. Verified on real data (k=2, k=3+, with/without Pareto) and via the full battle-test grid (192/192 crash-grid + 48/48 Pareto-grid combinations, all exercising .summary() end to end) plus the full regression suite (300 tests) and confirmation that the 23 pre-existing, unrelated test_pareto.py failures are unchanged.
PLAN_between_subjects_extension.md (design planning) and IMPLEMENTATION_INSTRUCTIONS.md (a scratch task brief for the autonomous implementation session) were working documents, not something worth keeping in the repo long-term. Deleted both. EXECUTIVE_SUMMARY.md is the one worth keeping -- renamed to HOW_BETWEEN_SUBJECTS_ADDED.md, a clearer name for what it actually is: a write-up of what got added and why. Also removed the resulting dangling "see notes/PLAN_between_subjects_ extension.md §X" pointers from code comments in config.py, summary.py, unpaired.py, and test_unpaired.py -- the surrounding rationale text stands on its own without them.
Both files were failing (26 tests total) for reasons unrelated to the between-subjects work, confirmed via git-stash comparison at the start of that effort. Root cause, found via git log -S on each renamed parameter: - 5df55f2 renamed compare()/tradeoff()'s secondary= to secondary_metric= but never touched tests/test_pareto.py (23 failures, all "unknown keyword argument 'secondary'"). - 12604ea renamed summarize()'s group_col=/value_col= to factor=/metric= but never touched tests/test_quick_primitives.py (3 failures). Both were genuine "renamed the API, forgot this one test file" gaps, not real regressions -- the local variable `secondary` in test_pareto.py's pareto_bootstrap() tests and the point_secondary=/ replicate_secondary= params elsewhere were left untouched, since neither is the renamed parameter. Updated the two now-misleadingly- named test functions in test_quick_primitives.py to match (test_summarize_dataframe_group_col -> ..._factor_metric, etc.); left test_pareto.py's test names alone since they were never parameter-specific. All 83 tests in both files now pass; full targeted regression suite (383 tests total) confirmed clean.
Stress-testing compare() across paired/unpaired designs, k=2..20, and malformed CSVs (missing columns, empty data, bad dtypes, etc.) turned up three real issues before this ships to developers: 1. load_from() crashed with a raw pandas TypeError on duplicate column names (a common CSV merge/export artifact) instead of a clear, attributed error. Now rejected upfront with EvalLoadError. 2. A NaN in the factor/grouping column silently became its own group at load time, only surfacing later as a misleading "scores contain N NaN cells" error that blamed the metric column instead of the real cause. Now rejected immediately in compare() with a correctly attributed message. 3. The Executive Summary's "Grp" rank column could go non-monotonic when critical-difference bands overlap/chain (A~B and B~C each individually non-significant, but A~C significant -- the classic CD-diagram transitivity caveat). _assign_significance_groups used to number CD-band members and leftover singletons in two separate passes, so a singleton sandwiched between two bands could get pushed behind both. Rewritten as a single rank-ordered pass that merges chained bands into one group, since this table needs exactly one ID per entity. This is shared code used by both the paired and unpaired reporting paths -- not new-feature-specific, just a pre-existing bug this grid happened to trigger. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The three bugs fixed in 234e01d had no permanent test coverage -- they were only caught by a standalone stress script, not pytest. Adds: - test_load_from_raises_on_duplicate_column_names: guards the EvalLoadError on duplicate column names in loader.py. - test_compare_raises_clear_error_on_nan_in_factor_column: guards the correctly-attributed ValueError on NaN in the factor column. - test_assign_significance_groups_merges_chained_bands_and_stays_monotonic: reproduces the exact chained-CD-band scenario (isolated performer sandwiched between two overlapping non-significance bands) and asserts group numbers stay non-decreasing down the rank-sorted list. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Standalone script exercising compare() across paired vs. unpaired design, k=2..20, n=15/30/50/200, and with/without a biased judge (PPI), plus a battery of deliberately malformed CSVs (missing columns, empty data, non-numeric/all-NaN scores, duplicate columns, NaN in the factor column, etc.). This is the script that surfaced the three bugs fixed in 234e01d. Kept for reuse as a regression check ahead of future releases: .venv/bin/python -m simulations.investigate_final_stress_test Includes an automated check that the Executive Summary's "Grp" column stays non-decreasing down the rank-sorted table, so a regression of the chained-CD-band bug would be caught by this grid even without rereading printed output by eye. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…able Delta-theta (stochastic dominance / probability of superiority) is the between-subjects path's primary pairwise statistic for continuous/ likert/grade data, but it doesn't say how far apart two groups are on the metric's own scale -- e.g. a 1.04-point gap on a 1-5 satisfaction score reads more intuitively than "P(A>B)=0.71". Add a secondary Delta-mean column (point estimate only, no separate CI -- same convention the paired path's own ES/rank-biserial column already uses), reusing the marginal means already computed for the "Mean Performance" section above the table. The binary/proportion-difference family is unaffected: its primary column already *is* the raw difference, so a second copy would be redundant. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
compare(design="unpaired", alignment=...)'s per-group marginal mean (the "Mean Performance" table, and the Delta-mean pairwise column added in 147fff2) was silently computed from raw judge scores regardless of alignment= -- only the pairwise Delta-theta/Delta-p was genuinely PPI- corrected. _compute_group_stats() called the generic robustness_metrics() helper, which has no PPI/alignment parameter at all, so the already- validated human-labeled subset (group_lab_arrays) was built and sanitized upstream but never used for the marginal estimate. Found by directly comparing compare()'s output with and without alignment= on real biased-judge App Store data and noticing the "PPI-corrected" group means were bit-for-bit identical to the uncorrected ones -- a check the PR's earlier stress test (234e01d) should have done and didn't (it only verified PPI output didn't crash and looked visually sane). Fix: wire the same PPI machinery the paired path already uses (_ppi_robustness_dispatch, resolved via is_binary_scores/ is_bounded_01_scores -> resolve_ppi_auto_methods, same GRADIENT_CI_ALPHAS sweep for the gradient CI bands) into _compute_group_stats, applied per between-subjects group. Every group is guaranteed at least one label by this point (compare_unpaired already validates that upstream), so unlike the paired path there's no "entity has zero labels, keep it uncorrected" fallback needed. Verified on real data: a biased judge's group mean moved from 1.56 (raw) to 2.61 (corrected), much closer to its true value (2.24), while a well-calibrated group barely moved -- matching the paired path's behavior. Confirmed the same correction direction/magnitude pattern holds for binary data. simulations/investigate_unpaired_ppi_calibration.py is the numerical (not just visual) stress test that should have caught this originally: 300-rep Monte Carlo empirical Type-I error and power, comparing raw vs. PPI-corrected pairwise significance under a deliberately biased judge. Results: a biased judge alone produces 100% false positives (continuous and likert); PPI correction brings that back to ~5.7% (continuous, essentially nominal) and ~10% (likert -- roughly 2x nominal, a real but pre-existing mild under-coverage in the ppi_t_interval/"unbounded" method, not introduced by this fix) while retaining 95-100% power when a real difference exists. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
… guard Rebased worktree-between-subjects-support onto ppi-power-tuning-tuning (4 new commits, including a99ae87 "Fix wilcoxon real-data Type-I inflation and kruskal power collapse"). That commit independently discovered and fixed the exact same _ppi_kruskal_wallis_pairwise ZeroDivisionError my own 28617d7 ("Fix ZeroDivisionError in PPI Kruskal-Wallis pairwise at k=2") had already patched -- both trace to the same underlying degenerate-covariance scenario, just found via different routes (my between-subjects battle-test grid vs. a99ae87's real-data harness). a99ae87's fix is more complete: it recognizes that exact-zero bootstrap variance around a nonzero effect is a maximally CONFIDENT result, not "no evidence" -- my simpler guard would have under-reported significance in exactly the strong-effect case a99ae87's own commit message describes collapsing kruskal's real-data power from 0.83 to 0.20. The rebase's merge conflict initially resolved by keeping BOTH fixes (mine layered inside a99ae87's else-branch, guarding a residual df==0 case that seemed theoretically possible even after a99ae87's check). Per direction to prefer the incoming, more rigorously real-data-tested version: verified empirically that a99ae87's fix ALONE (without my extra guard) still passes test_ppi_k2_pairwise_survives_degenerate_covariance_seeds (the exact regression test written for my own battle-test crashes) and the full PPI regression suite (test_unpaired.py, test_analyze.py, test_compound_ppi_fwer.py, test_ppi_corrections.py -- 440 tests, 0 failures). My guard was redundant; removing it here. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…ank 1 _assign_significance_groups (fixed for monotonicity in an earlier commit this session) let the #1 tier extend transitively through chained CD bands, same as #2+. That's wrong specifically for #1: _exec_verdict turns #1 membership into an explicit "tied with X as best" prose claim, so it has to mean "provably indistinguishable from the actual top performer" -- not "reachable from it via a chain of individually- nonsignificant neighbors" (the same Demsar 2006 transitivity caveat behind the earlier monotonicity fix, just biting a different part of the output this time). Found via a real case: FlipFlop and the top-ranked ClipCraze both showed "#1 / Tied with 5 others as best" in an unpaired executive summary, even though the pairwise table two lines above showed them significantly different (Delta-theta=-0.282, p<0.0001) -- ClipCraze was only connected to FlipFlop through two intermediate individually-nonsignificant links (ClipCraze~Loopster~Wavelength~...), not directly. A reader has no way to know a "#1" tag hides that. Fix: #1 membership is now restricted to the single maximal CD band that directly contains the rank-1 entity (no transitive extension). Everything past that band still gets its own tier via the existing (chain-merging) algorithm -- monotonicity is unaffected, and #2+ never made an individualized "tied with X" claim in the first place, so chaining there was never misleading the same way. Verified: the FlipFlop example's exec summary now shows ClipCraze/ Loopster as "#1, tied" and everyone else (including FlipFlop) as "#2, Significant drop-off" -- matching the pairwise table exactly. Full regression suite (test_unpaired.py, test_analyze.py, test_pareto.py, test_critical_difference_plot.py, test_compare.py -- 184 tests) passes with no other change in behavior; spot-checked a k=20 paired case to confirm #1 tiers still form sensible cliques there too. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
… output) For collecting more real data for the paper's FlipFlop example without touching the shared judge-bias simulation corpus. Clones just the RSS- fetching logic from collect_judge_bias_data.py's fetch_appstore_items (same endpoint, same retry/rate-limit handling), but: - Writes to its own gitignored simulations/out/appstore_scenario_ reviews.csv, never simulations/out/judge_bias_appstore*.csv (which the harness/tests read as shared ground truth -- growing it here would silently perturb every simulation depending on it). - Loops per app to a target N (default 300), instead of pooling every app's reviews and truncating to one global n_items. - Does no judge scoring at all -- that's a deliberate follow-up step the user runs themselves, separately. Re-running is safe and additive (dedupes against already-collected item_ids), since Apple's feed only serves recent reviews and won't hit 300/app in one run for most apps. Smoke-tested: fetched 5 real TikTok reviews, confirmed judge_bias_ appstore.csv was byte-identical before/after, confirmed re-running correctly skips already-collected items. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Scores reviews collected by collect_appstore_reviews_only.py with one or more LLM judges, writing to its own separate output (simulations/ out/appstore_scenario_judge_scores.csv + a merged item+human_label+ judge_score view) -- never judge_bias_appstore_scores.csv, same separation principle as the review collector itself. Reuses the actual judge-calling machinery (client construction, retry logic, the App Store prompt/response format) from collect_judge_bias_ data.py directly via import, rather than re-implementing it -- one place that knows how to talk to OpenRouter/Ollama, one prompt template. Supports mixing backends in a single run via --model-backends (e.g. thinkingmachines/inkling=openrouter alongside a free local Ollama model for variety), since OpenRouter needs a paid API key while Ollama models are free/local. Smoke-tested end-to-end with a local Ollama model (gemma3:1b, no API key/cost): scored 3 items, verified output schema, verified judge_bias_ appstore.csv stayed byte-identical, then verified resume/dedup behavior on a second run with the same --limit (correctly skipped the first 3 already-scored items and picked the next 3 unprocessed ones -- zero duplicate (item_id, model, run) rows). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Points at the standalone collector's output (appstore_scenario_reviews
.csv + appstore_scenario_judge_scores.csv) instead of the original
50/app single-judge dataset, and switches the judge to anthropic/
claude-haiku-4.5 -- selected from the 5 judges scored against this data
for having the widest per-app kappa spread (0.65 on FlipFlop vs. 0.88
on Wavelength) while keeping a solid pooled headline (0.78). All 5
judges independently showed their own worst calibration on FlipFlop,
so this isn't specific to one model's quirks.
Also documents in the module docstring that the in-paper explanation
for FlipFlop's miscalibration ("meme-fluent, ironic register") is
deliberately fictionalized narrative color, not a claim about the
actual mechanism -- the real driver (found by digging into the
mismatched reviews) is that a striking share of FlipFlop's 5-star
reviews contain bug reports/feature requests/account-complaint text,
which every judge reads as negative in isolation even though the star
rating is positive.
With real N=300/app (not 50), the result is now the cleanest version of
this story yet: raw judge scores show FlipFlop significantly worse
than Wavelength (mean diff -0.67, p<0.0001); PPI correction with just
15 labels/app (5% of the pool) collapses that to +0.01, p=1.0 --
matching the full-sample ground truth (-0.22, p=0.52) almost exactly.
Omnibus Kruskal-Wallis flips from p<0.0001 (raw) to p=0.90 (corrected).
Robustness check (10 independent label draws) unchanged: 0/10
reproduce the raw false positive.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…ample Carves out .gitignore exceptions for these two files (same pattern already used for judge_bias_privacy_judge*.csv / judge_bias_ iclr_metareview*.csv -- specific curated datasets committed despite the blanket simulations/out/* ignore rule), so paper_flipflop_example.py is reproducible for anyone who clones the repo without needing to re-run the two standalone collector scripts (which hit Apple's live RSS feed and a paid/local LLM judge -- neither guaranteed to reproduce the same sample or scores on a later run). appstore_scenario_reviews.csv: 1200 real App Store reviews (300/app, 4 apps: TikTok, Google Maps, Instagram, Facebook -- fictionalized as FlipFlop/Wavelength/Snippet/Bubblegum in the paper), collected via collect_appstore_reviews_only.py. Each review's real star rating is the human_label ground truth. appstore_scenario_judge_scores.csv: predictions from 5 independent LLM judges (anthropic/claude-haiku-4.5, ibm-granite/granite-4.1-8b, openai/gpt-oss-20b, qwen/qwen3.7-flash, thinkingmachines/inkling) against those reviews, collected via collect_appstore_judge_scores_ only.py. paper_flipflop_example.py uses claude-haiku-4.5 by default; switching JUDGE picks up any of the other four for comparison. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
ianarawjo
force-pushed
the
worktree-between-subjects-support
branch
from
August 17, 2026 00:33
1f925b2 to
eeb70c9
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
compare()/analyze()structurally assume within-subjects (paired) data — every entity scored on the same items. Between-subjects data (independent cohorts, no shared items — e.g. comparing competitor apps' review streams) had no valid path through them: the only workaround was faking a shared item index, which silently ran an invalid paired test on data with no such correspondence. This surfaced while writing a paper example and turned out to be a real structural gap, not an edge case.What changed
design="auto"|"paired"|"unpaired"param oncompare(). Auto-detects paired vs. unpaired data; raises a clear error on the mismatch instead of forcing an invalid analysis.design="unpaired"dispatches to a new engine (core/unpaired.py): Kruskal-Wallis/Mann-Whitney (continuous/likert/grade) or one-way ANOVA/Welch's t-test (binary), Bonferroni-corrected CIs + Holm-corrected p-values, PPI support, and nowsecondary_metric=Pareto-front analysis (via a new per-group joint bootstrap — disjoint groups have no shared item pool for the paired path's shared-index bootstrap to work).Paired-path safety
The existing paired path is unchanged for every current call pattern — verified by tracing every inserted line (each returns/raises immediately or touches only new locals) and by byte-identical output diffs across representative scenarios (default, binary/Newcombe, explicit bootstrap, Nemenyi) before/after the rendering refactor, plus the full existing test suite (~320 tests) passing throughout.
Testing
tests/test_unpaired.py)Known scope limits
design="unpaired"raise a clear error rather than attempting nested-run resampling.df["item"] = range(len(df))) beforeload_from(), since that function is shared by every call and wasn't touched.baseline=,pairwise_test=,show_rank_probabilities=have no effect underdesign="unpaired"(documented).Rendering unification, executive summary, housekeeping
Three follow-ups too, since the original description above:
summary_unpaired.pymerged intosummary.py. The between-subjects console printer had grown into its own file mostly to avoid bloatingsummary.pyfurther — moved back in, next to the paired-path functions it calls, so "is this actually shared or parallel" is easier to see at a glance.design="unpaired". Both were listed as explicitly out of scope inGroupComparisonResult's own docstring — turned out to be fully reusable from the paired path's existing machinery (_critical_difference_groups/_assign_significance_groups/_print_executive_summary), which only needs aPairwiseMatrix-like.get(a,b)lookup and aRobustnessResult-like means/CI object, not the ranking-bootstrap the paired path happens to derive its label order from. Two small adapters make it work unmodified; rank order comes from sorting by mean instead. Positioned identically to the paired path: pairwise table (CD bands in its footer) → Pareto section → executive summary (with a Trade-off column when Pareto is present) → Pareto callout.notes/cleanup. Trimmed down to a singleHOW_BETWEEN_SUBJECTS_ADDED.md.test_pareto.py/test_quick_primitives.py— two stale test files that were never updated after earlier parameter renames (secondary=→secondary_metric=,group_col=/value_col=→factor=/metric=).Also verified against Python 3.9 directly (this repo's CI matrix includes it) — a full
compare_unpaired()→.summary()run, including the new executive summary/CD-bands code, works cleanly.