A lot of changes - #13
Merged
Merged
Conversation
"judge_alignment" reads clearer at the call site than "validate_alignment" -- straight rename, same signature, no behavior change. Updated all ~130 call sites across the library, tests, examples, simulations, both READMEs, and one Jupyter notebook, plus a test class name that still spelled out the old name (TestValidateAlignmentBasic -> TestJudgeAlignmentBasic). Verified via py_compile on every touched file, running 4 of the affected example scripts end-to-end, and the full test_alignment.py (55/55) and test_compound_ppi_fwer.py (23/23) suites. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…diff rescale
nig_ci_1d's default b0=0.0625 (prior mean of sigma^2, sigma~=0.25) is
documented as calibrated for ci_single.py's rescale span [scale_lo,
scale_hi] ("weak knowledge that scores live in [0, 1]"). ci_paired.py
instead rescales paired diffs onto [-diff_span, diff_span] (needed so a
zero diff maps to 0.5, NIG's own prior centre) -- twice as wide a span as
ci_single's. Reusing b0=0.0625 unchanged there implies 2^2=4x the prior
variance in real diff units (variance scales with the square of a linear
rescale factor), producing persistent, substantial over-coverage that
isn't a deliberate safety margin -- just an unpropagated rescale-span
change, not a general flaw in NIG's prior.
Confirmed via direct comparison against ci_single.py's own likert usage
(same eval type, correct narrower rescale): already well-calibrated there
(0.947/0.952/0.932 at n=10/30/100, no systematic over-coverage) --
isolating the bug to ci_paired.py's wider rescale specifically, not NIG
generally.
Fixed by passing b0=0.0625/4 to nig_ci_1d in both ci_paired.py call sites
(flat-mode _run_cell and nested-mode _run_nested_pairwise_cell), via
functools.partial -- restores NIG's effective prior to the same absolute
variance ci_single.py already uses, not a new invented value. The
functools.partial wrapping required switching the flat-mode dispatch's
"which methods need rescaling" check from function identity (`fn is
nig_ci_1d`) to method identity (`method is NIG`), since a partial-wrapped
function is no longer identical to the original.
Re-validated at reps=300 (flat) / reps=150 (nested) across the corrected
icc range (0.01-0.95), continuous and likert:
- Flat, likert: coverage 0.983 -> 0.952 at n=10, width 23% narrower;
now narrower than logit_t's own width at every n.
- Flat, continuous: coverage 0.983 -> 0.964 at n=10 (still a touch
conservative but much closer), width matched to
logit_t's.
- Nested, likert: coverage 0.951, essentially identical to logit_t's
0.951, width narrower (0.585 vs 0.626).
- Nested, continuous: coverage 0.957, width narrower than logit_t's
(0.136 vs 0.143).
No warnings or errors in any re-validation run.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…n_ci
For users who don't want compare()'s full comparative report -- just a
trustworthy point estimate (or a few other common building blocks) as
plain data to hand to their own plotting library. All reuse the exact
auto method-selection/calibration machinery compare() uses internally
(extracted into router.resolve_auto_robustness_method, shared by
_analyze_single and the new primitives) rather than a second,
potentially-drifting calibration path.
- mean_ci(scores): calibrated mean + CI for a single array. Returns a
NamedTuple (attribute access or positional unpack).
- summarize(scores): the fuller descriptive + CI table (mean, median,
std, cv, iqr, percentiles, CI), batch-capable -- single array, a
{label: array} dict, or a long-format DataFrame + group_col/value_col.
Each group calibrated independently, so groups don't need to share a
rectangular design or even a data kind.
- stability(runs): standalone multi-run reliability (instability/ICC),
for one or more configs, without a full multi-model comparison.
- judge_debias_mean_ci(...): PPI-corrected mean + CI for judge scores
given a small human-labeled subset, raw arrays in -- a thin wrapper
around ppi.correct(np.mean, ...). Named to avoid the PPI acronym and
to make clear it corrects a mean, not per-item scores.
Also renames validate_alignment's array-only sibling into judge_alignment
itself: judge_alignment(human_labels, judge_labels) now works alongside
the existing judge_alignment(evaldata, llm_metric=..., human_groundtruth=...)
form (dispatched on the first argument's type), sharing a common core
(_judge_alignment_core) with the calibration-fitting and alignment-metric
logic so neither form can drift out of sync with the other. The array
form skips the DataFrame-specific representativeness checks (categorical
slice columns) but supports the score-distribution check when
all_judge_scores is provided; results built from raw arrays carry
placeholder column names and cannot be passed to compare(alignment=...).
Every result type follows the same convention: attribute access for the
common fields, .to_dict() for a JSON-friendly dict, .to_frame() (batch
results) for a pandas DataFrame -- mirroring ComparisonResult rather
than committing to one output shape.
Verified: 227 tests passing across the new tests/test_quick_primitives.py
(34 tests) plus the existing test_alignment.py (55), test_auto_ci_routing.py
(23), test_analyze.py (34), test_compare.py, and test_pareto.py (40) suites
-- confirming the router.py/alignment.py refactors preserve exact existing
behavior. Also ran examples/quick_primitives_demo.py and the pre-existing
alignment example scripts end-to-end.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- stability(): add a DataFrame form (config_col/run_col/item_col/value_col) with no orientation ambiguity, since each row names its own run/item -- the recommended path now. The raw-array/dict form keeps working but gets a heuristic warning when K > M (more "runs" than "items" is unusual for real eval data and is the signature of an un-transposed pivot table -- confirmed this exact mixup silently flips "very stable" into "near-random" on identical data with zero error). - mean_ci()/summarize(): strip NaN instead of silently propagating it into a NaN CI (confirmed: 10% missing data previously returned a real mean but ci_low=ci_high=nan with no warning). Warn how many were dropped; raise a clear, group-attributed error when nothing valid remains. - robustness_metrics(): statistic="median" combined with a mean/proportion -only closed-form CI method (wilson, logit_t, nig, t_interval, ...) previously silently produced a CI for the *mean* while reporting the *median* as the point estimate -- confirmed cases where the reported CI didn't even contain the reported point estimate. Now falls back to smooth_bootstrap (which respects `statistic`) with a warning explaining why, and always warns that statistic="median" hasn't been validated by the same simulation-based calibration testing as the mean path. Also fixed a related bug in the same function: the bootstrap_t path never forwarded `statistic` to bootstrap_t_ci_1d despite that function supporting it, and its multi_ci gradient loop used the fixed `alpha` instead of the per-band loop variable `a`. - judge_alignment(): reordered the array form to judge-first (judge_scores, human_scores), matching the EvalResults form's llm_metric-before-human_groundtruth convention -- previously the two forms disagreed, and swapping the array order silently produced a different, complete-looking report using a different statistical methodology with no error. Redesigned the array form's shape to match: both arrays are now the same length (every item), with human_scores NaN for unlabeled items -- mirrors the EvalResults form's sparse-column convention exactly, removes the manual-masking step, and gets the representativeness check for free when unlabeled items are present. - judge_debias_mean_ci(): same redesign, (judge_scores, human_scores) sparse pair replacing the old 3-separate-array signature. Added the hard-error/soft-warning thresholds already used by compare(alignment=...)'s PPI path (n_labeled >= 15, n_total >= 50 hard errors; n_labeled >= 30, n_total >= 100 soft warnings) plus a new hard error when every item is labeled (nothing left for PPI to correct). Now always warns that the labeled subset must be an unbiased, ideally uniform-random sample -- restating ppi.correct()'s own MNAR-labeling caveat, which the previous docstring-only version never surfaced. Also fixed two real stacklevel bugs in resolve_auto_robustness_method() found while verifying warning attribution for these changes: the resolve_score_bounds() delegation and the direct warnings.warn() call for the "unbounded" case need different stacklevel adjustments (one extra frame vs. none), but both were using the same value, overshooting past the user's actual call site for the direct-warn path in mean_ci()/summarize(). Verified: 444 tests passing across test_quick_primitives.py (61, up from 40), test_alignment.py, test_auto_ci_routing.py, test_analyze.py, test_compare.py, test_pareto.py, test_resampling.py, test_simultaneous_ci.py, test_cli.py, and test_compound_ppi_fwer.py. Re-verified the specific orientation-flip and NaN-CI repro cases from the audit by hand. Updated examples/quick_primitives_demo.py to the new signatures. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…lot()'s default plot_ci_forest() drew a single CI band per entity even though the data for the same 68/90/95/99% nested gradient the terminal's .summary() already shows (robustness.multi_ci) was already being computed by compare() by default -- just never exposed to the matplotlib path. Added a style="gradient"/"single" parameter (mirroring summary()'s own style= split): gradient draws the nested bands as increasingly-opaque bars toward the mean, falling back to single-band automatically per entity when multi_ci data isn't available (e.g. a Wald-type CI). compare_to overlay stays single-band regardless, to keep it legible. Extended ComparisonResult.entity_stats (the vis-compatibility shim) to also expose multi_ci per entity, and changed ComparisonResult.plot()'s default method from "bar" (the quick, uncorrected accuracy view) to "forest" -- result.plot() now shows the gradient CI picture by default, for quick inline use in a notebook. Verified: 8 new tests in test_ci_forest_plot.py, plus the existing critical-difference/scoreboard plot tests, test_compare.py, and test_pareto.py all still passing (98 total). Rendered and visually inspected the gradient plot, single-style plot, and compare_to overlay. Also smoke-tested the LMM path (a separate CI-computation branch). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Per feedback on the initial gradient plot render: - Added show_mean (default True), mean_marker="line"/"dot" (default "line", now a single plain black tick -- dropped the black+white double-line rendering), and show_ci_bracket (default False) to overlay a traditional bracket-style CI on top of the gradient bands. - Legend moved outside the axes (bbox_to_anchor to the right) so it never overlaps the bars; combined into one legend with tier colours, the gradient-band swatches, and the mean/bracket markers. - Tier legend labels simplified to "Unbeaten" / "Significantly worse". - Found and fixed a real bug while rebuilding the legend: the gradient band legend swatches were paired with _GRADIENT_BAND_ALPHAS in the wrong order (68% CI labeled as the lightest/widest band and vice versa) -- now correctly ordered widest/lightest (99%) to narrowest/darkest (68%), matching the actual drawing order. - Also fixed report.full_analysis.n_inputs never actually resolving (the attribute lives on .benchmark.n_inputs, not the bundle directly) -- N was silently missing from both the title and the new caption. - Added a self-contained methods caption (N, CI method, FWER correction, alpha, gradient legend) so the figure travels with its own provenance once copied out of evalstats into a paper, slide, or post, rather than only living in the surrounding terminal report. Verified: 14 tests in test_ci_forest_plot.py (6 new), plus the existing critical-difference/scoreboard/compare/pareto plot-adjacent suites (104 total) still passing. Rendered and visually inspected every option combination (gradient/single, mean line/dot/off, bracket on/off). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
For LaTeX embedding: a small caption-like line below the plot would
read as redundant once the figure gets its own \caption{} underneath.
Moved the CI-method/FWER-correction/alpha text to sit between the title
and the axes instead (N stays in the title itself, so nothing's lost) --
now reads as a subtitle that's part of the figure, not a second caption.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Switched the subtitle from axes-fraction to a points-based offset (consistent gap regardless of axes height) and increased that offset, which simultaneously pulls it away from the plot and closer to the title above it -- was hugging the axes top edge before. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…the title" This reverts commit 314cfd7.
… legend color_rule (default "tier", unchanged behavior) also accepts "factor" (each entity gets its own distinct color from a qualitative palette, stable across sort_by) or any matplotlib color spec (all bars use that one color). The tier-meaning legend only shows for color_rule="tier", since it's the only mode where color carries information beyond entity identity (which the y-axis labels already convey). Also fixed the whitespace complaint from the legend-outside-axes change: was reserving a fixed, overly generous right margin regardless of actual legend width, leaving blank canvas a reader would need to crop before using the figure in a paper. Now measures the legend's actual rendered extent after an initial draw and trims the figure's width to hug it (rescaling the axes' subplot fractions to preserve their exact pixel position/size on the narrower canvas) -- only applies to the standalone (own_fig) case; an externally-supplied ax= is left to the caller's own layout, as before. Verified: 19 tests in test_ci_forest_plot.py (5 new for color_rule), plus the broader plot-adjacent suite (109 total) still passing. Visually inspected all three color_rule modes, the whitespace trim on both gradient and single styles, the compare_to overlay, and confirmed the embedded ax= case is unaffected (as intended). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
compare_to previously always drew a single flat CI line regardless of style="gradient", and always used a fixed light-blue color unrelated to the row's actual tier/factor color -- so a red "significantly worse" row would show a light-blue comparison band beneath it, looking like an unrelated series rather than "the same entity, a second eval." Refactored the primary/comparison drawing into a shared _draw_ci_row() helper: compare_to now renders nested gradient bands (falling back to a single muted line when it has no multi_ci data) using the exact same color as its row, just at reduced alpha. Widened the per-row vertical offset and slightly shrunk the gradient band height when compare_to is active in gradient mode, so the two stacked bands don't heavily overlap. Updated the legend to a neutral full-vs-muted-alpha swatch pair (color now varies per row, so a single fixed comparison-color swatch no longer made sense), and removed the now-dead PALETTE["compare"] constant. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…, muted mean tick Per feedback: the comparison band should stand apart from the primary one, not just be a co-equal band in the same weight. Changes: - Added _lighten() -- a real RGB blend toward white, not just lower alpha (alpha fades toward whatever's underneath, not necessarily white, and reads more like a rendering artifact than an intentional "this is secondary" design choice). - Comparison bands are now visibly thinner (0.45x band height, 0.6x line width for the single-style fallback) than the primary's (0.9x when compare_to is active, full height otherwise). - Comparison's mean tick is now a neutral gray, not black, matching its lighter/thinner treatment; scales down naturally with the thinner band. - Legend swatches updated to mirror the actual thin+light vs. thick+full treatment instead of a same-weight alpha pair. Verified: 20 tests in test_ci_forest_plot.py (rewrote the same-hue test, which had encoded the old alpha-muting behavior and also had a zorder- range-overlap bug in its own filtering logic -- now splits primary vs. comparison bars by insertion order instead), plus the broader plot-adjacent suite (110 total) still passing. Rendered and visually inspected both gradient and single styles. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The previous offset (0.28, gradient mode) actually left the gap to the *next* entity's primary band (0.10) smaller than the gap to its own sibling (0.22) -- the comparison band could read as belonging to the row below it. Reduced the offset (0.2 gradient / 0.14 single) so the sibling gap is now clearly the smaller of the two, correctly grouping each row's primary+comparison pair before the next entity starts. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…for likert pairwise/marginal CIs Wires nig_ci_1d into the real evalstats package (analyze()/compare()), not just the simulation harness, as the auto-selected method for discrete/ordinal bounded data (Likert scales, integer percentage grades), superseding logit_t there specifically -- both an explicit eval_type="likert" override and a data-driven auto-detect fallback (with a UserWarning naming the detected grid step, silenceable via an explicit eval_type either way) are supported, per request. Context: AUTO_ANALYZE_METHOD_TABLE's "bounded_01" row used to default to nig/nig_nested before being superseded by logit_t in 85df093 ("Refining the sims for simultaneous cis and pvalue FWER correction"), citing fig:ci-decision-tree -- i.e. simulation results from this same harness. That comparison likely used the SAME uncorrected NIG prior fixed earlier in this branch (b6b4743: nig_ci_1d's default b0 is calibrated for a single-sample rescale, silently 4x too wide when reused unchanged on a paired diff's rescale span), which would have made NIG look needlessly conservative next to logit_t at the time. Re-validated post-fix (see b6b4743's message): NIG beats logit-t on likert score at every N up to 500 (17% better at n=10), while logit-t remains the marginally better choice for genuinely continuous bounded data -- exactly the split this commit encodes. Changes: - evalstats/core/resampling.py: new detect_quantization_step(), a production home for the GCD-style grid-detection logic from simulations/harness/cases/ci_paired.py's _detect_dither_halfwidth (same false-positive profile: 0% down to n=6 pooled values, verified up to n=1000). - evalstats/config.py: DataKind gains "likert"; new AutoAnalyzeRule row (pairwise/robustness -> nig/nig_nested); PPI_AUTO_METHOD_TABLE gets a fallback row to ppi_logit_t (no PPI-corrected NIG exists) so a "likert" data_kind can't raise there if it's ever threaded into that path. resolve_auto_simultaneous_ci_method needed no change -- it already collapses any non-binary data_kind to "numeric". - evalstats/core/paired.py: new method="nig" branch in pairwise_differences() (mirrors the existing logit_t branch, using the corrected _NIG_PAIRED_DIFF_B0); "nig" added to the method Literal in pairwise_differences/all_pairwise/vs_baseline. Also updates _simultaneous_cis_router -- all_pairwise() defaults to simultaneous_ci=True, which replaces individual pairwise CIs with a Sidak/joint-bootstrap-widened construction chosen by the router's OWN independent data_kind detection, entirely separate from AUTO_ANALYZE_METHOD_TABLE. Without this, the new default would have been silently bypassed for any k>=3 comparison (the common case, and the exact multi-model scenario this whole investigation is about). New eval_type param threads an already-resolved decision down from analyze() to avoid redundant re-detection/re-warning, while still supporting direct all_pairwise()/vs_baseline() callers via independent auto-detection. - evalstats/core/router.py: new eval_type: Optional[Literal["likert", "continuous"]] param on analyze() (and _analyze_single/ _analyze_multi_model), doing the explicit-override-or-auto-detect dance once per analyze() call and passing the resolved value down to all_pairwise() so the per-pair and simultaneous-CI paths stay consistent. Verified end-to-end through the real public API (analyze(), all_pairwise(), pairwise_differences()), not just unit-level: explicit eval_type='likert' routes to NIG with no warning; omitting it on genuinely discrete data auto-detects and warns exactly once; continuous [0,1] data is unaffected either way. Full existing suite (320 tests across test_analyze[_factorial].py, test_bootstrap_t_pairwise_ranking.py, test_bayes_binary_routing.py, test_p_values.py, test_permutation.py, test_wilson_newcombe.py, test_simultaneous_ci.py, test_compound_ppi_fwer.py) passes unchanged, with one new (expected, non-failing) warning in test_simultaneous_ci.py::test_compare_prompts_simultaneous_ci_true_by_default -- its 3-item hand-written fixture ([0.55, 0.6, ..., 0.8]) is coincidentally on a perfect step=0.05 grid, which the detector correctly flags; the test doesn't assert on which CI method was used, so it still passes. Also affects compare_e2e.py's own compare() calls going forward: it passes score_range=(1, 5) for likert without the new eval_type=, so a future rerun will auto-detect and switch to NIG there too (with a warning) -- the intended effect of this change, flagged here since it means compare_e2e.py's past likert results (validating logit_t) won't represent compare()'s default behavior anymore once this lands. NOT merged into compare-e2e yet -- committed as a draft for review, per explicit instruction, given the scope of this change (a real production default, not just simulation/paper-supporting code). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…relying on auto-detection
Threads an es_eval_type ("likert"/"continuous"/None, mapped from this
file's own broader eval_type which also has "binary"/"grades") through
to every es.compare() call site (the main per-rep call, plus both
oracle/subset-only reference-estimator calls via
_run_truth_only_compare) added in a0d8c6b.
Both call sites already wrap compare() in
warnings.catch_warnings()/simplefilter("ignore"), so the new
auto-detection warning was never actually visible during a run -- but
passing eval_type explicitly does more than silence a warning that was
already silenced: it pins down, in the test code itself, which method
(nig vs logit_t) each eval_type is deliberately exercising, rather than
leaving that resolution to a data-driven heuristic a future reader would
have to re-derive. Also makes this file's own compare() calls immune to
the same class of small-N/tiny-fixture false-detection risk noted in
a0d8c6b's commit message (irrelevant at this file's real sample sizes,
but no reason to depend on it holding).
Verified via _run_cell smoke test (both likert and continuous, 5 reps):
zero errors, and zero discreteness-detection warnings even OUTSIDE the
existing simplefilter("ignore") blocks, confirming eval_type is now
actually being passed rather than left to fall through to detection.
Still part of the same draft as a0d8c6b -- not merged into compare-e2e.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- style="single" fallback now draws standard error-bar caps instead of a bare rounded-cap line; the compare_to offset and mean-tick scale are retuned so the primary/comparison caps and ticks no longer overlap. - CI bands could hug the left/right spine due to barh/plot sticky edges overriding matplotlib's default autoscale margin. Now pads by 6% of the data span, clamped to the metric's resolved score bounds (or [0, 100] in percent mode) so a CI that genuinely sits at the true floor/ceiling still hugs it correctly.
a0d8c6b wired NIG into the auto default for ALL likert routing: single- and multi-run pairwise comparisons, single- and multi-run marginal (robustness) CIs, and the k>=3 simultaneous/family-wise (Sidak/joint- bootstrap-widened) construction. Only the first of those -- single-run pairwise -- was actually verified this session: - Multi-run/seeded pairwise: one nested-mode check looked consistent, but never went through the same adversarial stress-testing (boundary- clipping bias, detector edge cases) the single-run path did before being trusted. - Marginal/robustness CIs (the "nig"/"nig_nested" single-sample case in core/variance.py's robustness_metrics()): never tested directly at all. Checking simulations/harness/cases/ci_single.py's own separate reimplementation earlier this session is not a substitute for testing this actual production code path. - The k>=3 simultaneous-CI router (_simultaneous_cis_router): only ever tested with NIG's OLD, buggy prior (the very first standalone investigation, before core.paired._NIG_PAIRED_DIFF_B0 existed), never re-validated post-fix. Fixes: - config.py: AUTO_ANALYZE_METHOD_TABLE's "likert" row reverts robustness_method_single_run/seeded to "logit_t" (was "nig"/ "nig_nested"). resolve_auto_analyze_methods now falls the resolved pairwise method back from "nig" to "logit_t" whenever seeded=True, so the table itself can keep listing pairwise_method="nig" (still correct for the single-run case) without that leaking into multi-run. Reason text rewritten to state precisely what is and isn't covered. - core/paired.py: _simultaneous_cis_router's data_kind resolution drops the "likert" branch entirely -- bounded numeric data (discrete or continuous) always resolves to "bounded_01" there now, so the k>=3 construction always widens logit-t's formula. eval_type stays a parameter on all_pairwise()/_simultaneous_cis_router (unused for this purpose right now) rather than being ripped out, so re-enabling this later is a small, localized change instead of re-plumbing analyze() again. Docstrings on both functions corrected to say this explicitly instead of implying full likert support. - core/router.py: eval_type's docstring on analyze() states the actual current scope. The auto-detection UserWarning no longer claims "evalstats is using NIG-based methods" unconditionally -- it now says NIG applies to single-run pairwise comparisons specifically, and that other analyses on the same data still use logit-t. Re-verified end-to-end after rescoping: resolve_auto_analyze_methods returns ("nig","logit_t") for likert+single-run and ("logit_t","logit_t") for likert+seeded; analyze() on single-run likert data gives pairwise test_method="paired NIG", multi-run gives "paired logit-t"; a k=3 simultaneous-CI call on likert data shows per-pair NIG point estimates but a logit-t-widened simultaneous_ci_method="boot" construction, source- verified directly against _simultaneous_cis_router's body. Full existing suite (320 tests, same set as a0d8c6b) still passes, including the one expected warning in test_simultaneous_ci.py, now with corrected wording. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- New factors= param on plot_ci_forest/ComparisonResult.plot(): "auto" (default) upgrades to a grouped (model, prompt) view whenever a report genuinely has both axes with >1 level, "model"/"prompt" collapses to the marginal view over one axis, and ["model","prompt"] (or reversed) picks the grouped view explicitly with list order setting outer/inner grouping. Falls back to today's single-axis rendering unchanged otherwise. - New ComparisonResult.model_labels/.prompt_labels/.as_view() to support it. - color_rule default changed from "tier" to "auto" (tier for flat views, factor for grouped, since "tier" has no single meaning across a grid); color_rule="tier" on a grouped view now raises a clear error instead of silently doing something wrong. - compare_to and show_ci_bracket aren't supported together with a grouped view yet -- both raise clear errors. - Refactored _draw_ci_row/x-axis padding/legend-and-trim out of the single-axis code path into shared module-level helpers so the new grouped renderer (_plot_ci_forest_grouped) reuses the same visual language instead of duplicating it. - 16 new tests covering grouping, reversal, marginal overrides, and the new error paths.
- core/summary.py: _print_pairwise_section crashed (AttributeError on first_result.test_method) whenever a bundle had exactly one entity (no pairs to compare) -- e.g. a multi-model, single-prompt CLI/analyze() run. Now returns early with nothing to print, letting the rest of the bundle summary (mean/CI for the lone entity) render normally. Added a regression test reproducing the crashing shape via print_analysis_summary(). - vis/forest.py: PercentFormatter now uses decimals=0 -- axis ticks land on round gridline values, so "50%" reads better than the "50.0%" PercentFormatter(decimals=None) was auto-picking. - vis/forest.py: new font_scale= parameter on plot_ci_forest()/.plot(), uniformly scaling every text element (tick labels, axis/title, subtitle caption, legend) without touching layout -- for compensating when a figure gets shrunk to fit a paper column.
Three real models (via OpenRouter/Inspect AI) on BBQ: raw accuracies look like a smooth decline (78/71/69%), but pairwise significance shows the top two are tied despite a 7-point gap, the bottom two are ALSO tied despite only a 2-point gap, yet the top model significantly beats the bottom one -- "tied with" isn't transitive. Filters simulations/out/inspect_benchmarks.csv (already committed) down to 3 models x a fixed 100-question subsample, writes examples/bbq_results.csv, and runs the same comparison via the API. Run `python examples/compare_bbq_transitivity_demo.py` for the API path, or `evalstats analyze examples/bbq_results.csv --p-values` for the full terminal output (gradient plots, pairwise p-values, rank bands).
… fix misleading legend - core/summary.py: _print_multi_model_summary skipped nothing when a MultiModelBundle had only one template -- printed a "Cross-model per-template comparison" header for zero actual comparisons, then repeated every model's own (already-shown) single number again in a "Per-Model Summary" block per model. Both now gated on n_templates > 1. - cli.py: --correction defaulted to "fdr_bh" and didn't even list "auto", "shaffer", or "romano_wolf" as choices -- analyze()'s own default is "auto", which resolves to "shaffer" or "romano_wolf" and never "fdr_bh". CLI now matches the API default. - core/summary.py: gradient-style interval plots (the default everywhere) claimed a mean marker in their legend that the gradient renderer never actually draws -- only the non-default "line" style does. Fixed across all 5 call sites via a shared _mean_marker_legend() helper. - core/summary.py: "Pairwise Comparisons" header nested CI method / FWER correction / simultaneous-CI info into parentheses-within-parentheses, easy to miss and hard to parse. Header is now just the base test method; a clear "CI method: ... | FWER correction: ... | alpha=..." line prints explicitly below the table instead, mirroring the matplotlib forest plot's subtitle. - core/summary.py: "Robustness" section renamed to "Descriptive Statistics". - evalstats/tests/__init__.py: fixed a real regression (258 failing tests in test_ppi_corrections.py/test_compound_ppi_fwer.py) from judge_alignment()'s isinstance(x, EvalResults) dispatch -- _EvalStub was a duck-typed lookalike, not an actual EvalResults subclass, so every call fell through to the wrong branch. Made it a real (minimal) subclass instead. - examples/compare_bbq_transitivity_demo.py: removed a now-incorrect correction="fdr_bh" override, added line_width control, and switched to the lower-level print_analysis_summary() call (preserving "model"/ "models" labeling explicitly, since that path doesn't infer it the way ComparisonResult.summary() does).
- Explicit methods summary now sits directly above the p-value-method detail line (previously below the whole table): line 1 is "CI method: ... | p-value method: ... | alpha=...", line 2 is "FWER corrections: Simultaneous CIs: ... | p-values: ..." -- broken into two separate corrections since simultaneous CIs and p-values can use different methods, and simultaneous CI method was previously missing entirely from this reporting. - Section header now reads "Pairwise Comparisons (Tango CIs)" instead of the bare "(tango)" -- prettified via the same helper used for the new "CI method:" line. - Simplified the Romano-Wolf p-value description from "Romano-Wolf bootstrap step-down (FWER-controlled; no Wilcoxon-compatible joint form exists, see romano_wolf_stepdown_pvalues)" to "Romano-Wolf step-down (FWER-controlled)" -- the implementation-detail aside wasn't useful to a non-statistician reader. - Added _pretty_correction()/_pretty_simultaneous_ci() helpers with proper name mappings (simultaneous_ci_method has 4 real values -- max_t, sidak, boot, bonferroni -- not just the 2 I'd initially handled). - Dropped the ", boot-adjusted" suffix from "Statistically indistinguishable rank bands ... computed from 95% CI, boot-adjusted:" -- just noise.
NIG was previously scoped to single-run pairwise Likert comparisons only. Extend it to seeded/multi-run pairwise too, and fix the k>=3 simultaneous-CI router to widen NIG (not logit-t) for Likert data -- logit-t's paired-diff rounding-cancellation failure mode was still showing up there (fam.cov 10-26% at n=15,k=10 in an overnight compare_e2e sweep), since the router had never been updated after the original NIG rollout was scoped down. Also fixes two stale tests that predated Likert quantization auto-detection. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
correct()'s power_tune bootstrap branch previously always shrunk the raw lambda-hat estimate back toward 1 (vanilla PPI) as n_lab shrinks. This conflates "lambda-hat is imprecise because n_lab is small" with "the true lambda is probably close to 1" -- there's no reason the second should follow from the first, and it costs real power against a genuinely uninformative judge (up to ~2x, confirmed via simulation) without a matching Type-I benefit in that regime. Replaces the fixed target with an empirical one: splits the existing lambda-estimation bootstrap draw into batches (no extra resampling cost) to get P(lambda_hat < 0.5 | data), and shrinks toward target = 1 - that probability instead of always toward 1. Falls back to target=1 when Y_lab itself is near-degenerate, matching the existing degenerate-variance guard's intent. Screening-scale harness runs (Type-I sweep + 5-way estimator comparison, simulations/out/screening_adaptive_20260812_222247/) show this matching or beating shrink-to-1 at nearly every (eval_type, n_lab) cell tested, while also avoiding shrink-to-0's failure modes (MNAR-selection bias, degenerate/rare-event Y_lab). Full evalstats ppi pytest suite passes unchanged (379 tests). Investigation notes: simulations/out/results_why_ppi_shrink_1_over_0.md. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
_analytic_mean_point_se (shared by _analytic_mean_correct, _analytic_logit_t_correct, and every t_interval/logit_t test wrapper in evalstats/tests/__init__.py) still used the fixed shrink-to-1 formula after the bootstrap path was already switched to an adaptive target. Adds _analytic_shrink_target, which estimates the same "P(lambda_hat < 0.5)" target this backend's design needs, but adapted to having no bootstrap array to reuse: it draws a cheap internal micro-bootstrap of just the (Y_lab, Y_hat_lab) pair (var_unlab is already closed-form from the large unlabeled sample, no resampling needed there) under a FIXED internal seed, since this randomness is purely an implementation detail for approximating a shrinkage target, not a Monte Carlo quantity callers need to control -- keeps every existing call site's signature untouched and the function fully deterministic. Same degenerate-Y_lab guard as the bootstrap path, plus an explicit n_lab<=1 short-circuit this backend needs that the bootstrap path didn't. Full evalstats ppi pytest suite passes unchanged (379 tests). Re-ran the official-tier label-efficiency check (simulations/out/label_efficiency_analytic/) against the prior bootstrap-only-adaptive baseline: every N_lab>=30 cell is bit-for-bit identical (confirms the diff is cleanly isolated to the analytic path), while N_lab=15/20 -- where backend="auto" actually dispatches here -- improved at every point checked, most clearly likert N_lab=15 at kappa=0.40 going from a 1.00x label-efficiency multiplier (literally no benefit under the old fixed target) to 1.59x. Investigation notes: simulations/out/results_why_ppi_shrink_1_over_0.md. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
All three multi-run variants centre the interval at d_hat/(1 + z^2/n_items).
The denominator uses the ITEM count only -- never the run count R, never
R_eff -- so the shrinkage toward zero is invariant to how many runs are
collected: k = n/(n+z^2) is 0.722 at n=10, 0.929 at n=50. On lopsided
scenarios that bias is what breaks coverage, and more runs cannot touch it.
Bonett-Price shrinks by n/(n+2) instead (0.833 at n=10), roughly half as
hard, which is why it survives the same cells.
Measured consequence, on a lopsided cell with R=5 and high ICC:
n mj_floor_er cluster bonett(1 run) newcombe(1 run)
10 0.844 0.844 0.961 0.940
20 0.895 0.895 0.949 0.930
50 0.906 0.906 0.944 0.937
A single-run method applied to ONE of the five runs beats every multi-run
variant using all five. er and cluster are identical here, confirming the
R_eff step is inert exactly in the high-ICC regime where real eval data
lives (measured ICC median .748 over 48 corpora).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ants Wires in bonett_price_paired_ci_multirun_cluster and makes Bonett-Price a public method for paired binary data, single- and multi-run. The multi-run derivation rests on an exact restatement: Bonett-Price is the plain Wald interval on the paired differences over a sample augmented by two Laplace pseudo-items at delta = +1 and -1. The pseudo-items are ITEMS, not runs, so they do not scale with R -- scaling them per-run reinstates exactly the degeneracy the Laplace adjustment exists to prevent. Verified: exact reduction to single-run Bonett-Price at R=1 (2.2e-16), R identical copies reproduce the single-run interval (no free information), symmetric, in bounds, contains its point estimate. Retires mj_floor_er and mj_floor_mmnt from the sweep and repoints method='mj_floor' multi-run dispatch to the cluster variant. mj_floor_er's Kish term cancels exactly when its max() does not clamp and inflates variance up to 2.8x when it does -- inert in the high-ICC regime real eval data occupies, conservative elsewhere, and it makes width non-monotone in R. mj_floor_mmnt is algebraically the cluster interval whenever its floor does not clip. MJ_FLOOR_CLUSTER is kept as the single multi-run mj_floor comparator; it still carries the family's centre shrinkage d_hat/(1 + z^2/n), which uses the item count only and so is untouched by R, hence it inherits the lopsided-scenario tail. Comparator, not fallback. Measured on a lopsided cell at R=5: bonett_price_cluster covers .963/.952/.924 at n=10/20/50 against mj_floor_er's .871/.889/.895. Two bugs fixed: - order_present_methods() silently dropped any method missing from REPORT_METHOD_ORDER, so it would burn simulation time then produce zero rows with no diagnostic. Now raises with the fix in the message. - The *_mean reductions threshold the run mean at 0.5, which changes the estimand to a majority-vote difference; measured MinCov 0.000, degrading with R. Dead imports removed and the hazard documented on the function. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Implements the two methods the clustered matched-pair literature actually
recommends, so the multi-run recommendation is measured against published
baselines rather than only against our own variants:
- clustered_score: Yang, Sun & Hardin (2012) X^2_Score -- Tango's score
statistic with the Eliasziw-Donner variance inflation. It turns out to be
exactly our existing tango_scc quartic with z^2 -> z^2*(1+(n_c-1)rho),
so no new solver was needed.
- modified_obuchowski: Yang et al. (2010) X^2_MO, cluster-level and
assumption-free about the within-cluster correlation structure.
Formulas transcribed from rendered page images rather than the text layer
(the 1991 scan's OCR mangles both prose and math), then validated:
X^2_MO / X^2_O / X^2_D statistics vs R clust.bin.pair exact
clustered score CI vs Yang's published worked example exact
(-0.03829, 0.29140), and X^2_ICC + its Wald CI to 5 dp
score CI reduces to tango_scc(c=0) with no clustering 0.0e+00
bounds / symmetry 0 / 1.5e-14
inflation factor, independent vs correlated runs 1.0000 / 4.92
One ambiguity resolved: clust.bin.pair computes n_c from discordant cluster
sizes while Yang uses full cluster sizes. Both collapse to exactly n_c = R
for equal cluster sizes, which is always our case, so it is unreachable
here -- documented because it would bite on unequal clusters.
Real-data nested sweep (inspect, R=5, 130 scenarios, 10140 cells, 300 reps)
shows neither published method beats Bonett-Price:
method Cov MinCov Width Pen Score <.90 <.93
mj_floor_cluster 0.965 0.860 0.2583 0.0321 0.2904 9 34
bonett_price_cluster 0.980 0.933 0.3057 0.0158 0.3215 0 0
modified_obuchowski 0.930 0.613 0.2533 0.0851 0.3384 110 231
clustered_score 0.981 0.890 0.3300 0.0185 0.3485 3 9
modified_obuchowski is the unregularised Wald on item differences -- verified
identical to it at R=1, and it returns a zero-width interval at zero
discordance. That is the degeneracy Bonett-Price's Laplace pseudo-items
exist to prevent, and it explains MinCov .613. The clustered literature
supplies cluster-level variance estimators but no small-sample adjustment;
combining the two is what wins.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three Bonett-Price multi-run variants were three rows of the same interval: _er and _mmnt only add a floor to the item-level variance, neither floor ever fires (the Laplace pseudo-items already dominate it), and all three agreed to four decimals on the real-data nested sweep. Removes both. CLUSTER is the one kept because it is the only one with no floor at all, so it describes in a sentence: the single-run construction with the item as the unit of analysis. Drops modified_obuchowski from the sweep. It carries no small-sample adjustment -- verified bit-identical to the unregularised Wald on item differences at R=1, and it returns a zero-width interval at zero discordance -- giving MinCov .613 with 231 of 10140 real-data cells below .93. The implementation and its validation against the reference R package are retained in evalstats.core.resampling as a citable negative result, but it no longer clutters the tables. The nested binary sweep is now three methods: mj_floor_cluster (the retained mj_floor comparator), bonett_price_cluster (the recommendation), and clustered_score (Yang, Sun & Hardin's published competitor). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The two appendix FWER figures each stacked two separately-generated two-panel plots inside one figure*, carrying three copies of the same legend between them, at 0.583 pages apiece. Their labels were also effectively unreadable: the source PNGs are ~14.5in wide but drawn into 0.8\linewidth (~5.6in), a 0.38x reduction that renders 12pt type at roughly 4.6pt. Adds save_multiarm_fwer_panels_plot and save_simultaneous_ci_panels_plot to the pvalues case: FWER/coverage and power/width, each against n and against k, in one row with a single shared legend. Drawn at the final printed width (7in) with 7pt fonts, so nothing is downscaled and 7pt stays 7pt. Wired into the official-test plot paths so the figures the paper prints are reproduced by a normal harness run rather than by a one-off script. Sample-size ticks render 1000 as "1k" -- at four panels across the text width, "500" and "1000" collide otherwise. Both plots exclude the uncorrected baseline: it runs at FWER ~0.45 and compresses every corrected method into an unreadable band. The superseded vs_n/vs_k plots drop it for the same reason and are kept. Paper: 46 -> 44 pages, appendix 26 -> 24. The float area removed is only ~0.77 pages; the rest comes from draining the float-deferral queue, which had been stranding half-empty pages through the back of the appendix. simulations/papers/replot_fwer_panels.py is the exploratory version used to compare layouts (2x2 vs 1x4, full vs reduced method set); the harness functions are the ones that matter. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
compare() resolves correction="auto" to Romano-Wolf, which is the better
method and stays the default. But the paper validates and reports the
rank-based tests, so a reviewer could not check the REPORTED pathway
against the oracle and human-subset reference arms.
--classical-rank-path adds {"omnibus": True, "correction": "shaffer"};
Wilcoxon needs no flag, since pairwise_test="auto" already picks it for
any k. Threaded through _run_truth_only_compare as well, so the oracle
and human-subset arms use the same configuration -- switching only the
PPI arm would have left the references on Romano-Wolf and made the
comparison apples-to-oranges.
The official variants now list the reported path first, keeping
Romano-Wolf available as a second entry. No evalstats changes needed:
correction= already passes through compare()'s **kwargs and resolves
correctly (verified via PairwiseMatrix.correction_method).
Checked at k=5 (10 pairs), n=200, n_lab=40, reps=40 on likert, both
paths, same seed. PPI power lands at 90.0% either way, between the
human-subset floor (78.4%/79.4%) and the oracle ceiling (98.4%/96.9%),
with coverage 94.5-96.7% and Type-I 4.1-4.7% throughout. Switching to
the reported path costs essentially nothing here.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… PPI to Bonett-Price
Multi-run binary pairwise now defaults to bonett_price_paired_ci_multirun_shrunk.
The _cluster form pins its two Laplace pseudo-items at delta = +/-1, the largest
possible item-level discordance. That is exact at R=1, where every discordant
item has delta^2 = 1, but at R>1 real delta^2 shrinks toward the squared per-item
rate while the pseudo-mass stays at 2, so the pseudo-items become 3-25x heavier
than a real discordant item and the floor progressively swallows the variance.
The fix applies Bonett-Price's own device twice: once to the discordance rate
(the n+2 denominator, unchanged) and once to the discordance MAGNITUDE, shrunk
toward the R=1 reference with the same weight of two pseudo-items:
m2 = (sum_i delta_i^2 + 2) / (sum_i u_i + 2), u_i = mean_r |A_ir - B_ir|
Since delta_i^2 <= |delta_i| <= u_i, m2 lies in (0, 1] and cannot vanish; it is
equivalently a shrinkage estimator with weight sum(u)/(sum(u)+2) on the observed
magnitude and the rest on 1. The denominator must be sum(u), the EFFECTIVE count
of fully-discordant items -- a plain count of discordant items collapses m2 to
~0.2 when items flip sign across runs (MinCov .8268 vs .9296 on a 300-cell sweep).
Verified to machine precision: R=1 is bit-identical to bonett_price_paired_ci
with m2 = 1 by construction; replication invariance and arm-swap antisymmetry are
exactly 0; zero discordance matches the +/-1 form. Official synthetic nested sweep
(1536 cells, reps=500): MinCov .920 vs _cluster's .922, width .2481 vs .2605,
mean coverage .9740 vs .9793. Real inspect nested (reps=300): Score .5220 vs
.5387, MinCov .933 unflagged. It coincides with _cluster at ICC .99 and diverges
as run noise grows (9.6% narrower at ICC .50), which is the derived behaviour.
The three cells below .93 in that sweep were selection artefacts: rerun at
reps=20000 they return .948-.954. A MinCov taken over many cells at modest reps
is biased low, which is now noted in the docstring.
Also here, from the same line of work:
- clustered_score: when the ANOVA denominator is exactly 0 the ICC is 0/0 and
Yang's Remark 1(b) sets the inflation factor to 1, asserting independence of
all n*R units. Harmless at their cluster sizes (~2.4) but severe at ours: an
all-concordant table at n=15, R=20 got a 95% interval of width 0.025, narrowing
further with runs. Now returns n_c (full clustering) on that exact branch only,
a deliberate deviation from Remark 1 documented in the docstring. Verified inert
wherever rho is estimable: over 4000 random tables it changed only all-concordant
ones. Real-data MinCov .857 -> .893.
- Binary PPI pairwise routes to bonett_price rather than mj_floor, in both
PPI_AUTO_METHOD_TABLE and the non-PPI AUTO_ANALYZE_METHOD_TABLE, where the
binary rows collapse from two (bayes_binary below N=50, mj_floor above) to one.
mj_floor stays implemented and selectable, out of the official sweep.
- _bonett_price_augmented_interval takes a pseudo_m2 argument (default 1.0), so
_cluster is visibly the m2 = 1 special case.
- Removes tests/test_alignment.py::test_rubin_cis_converge_under_perfect_alignment.
It tested Rubin's-rules behaviour that no longer exists -- "rubin" appears
nowhere in evalstats -- and had been failing on this branch independently.
NOT included: simulations/harness/cases/pvalues.py carries the matching PPI
dispatch for bonett_price, but 1941 of its 1948 changed lines belong to another
working thread, so it is left uncommitted. Until it lands, a fresh checkout has
bonett_price in PPI_TEST_METHODS without its dispatch.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…X-table regenerator BINARY_PAIR_NESTED_OFFICIAL excludes BONETT_PRICE_CLUSTER from the default multi-run binary run. It is the same estimator as BONETT_PRICE_SHRUNK with the pseudo-item magnitude pinned at 1 rather than shrunk, so reporting both invites readers to treat a parameter setting as a competing method. It stays implemented and selectable (--methods bonett_price_cluster) as the ablation showing what the shrinkage buys. The nested dispatch loop now also skips methods absent from active_methods, which previously raised a KeyError once the default set shrank. simulations/relatex_ci_paired.py rebuilds ci_paired's LaTeX overall-summary table from a finished results CSV, the way replot_ci_paired_violins.py already rebuilds the violins. Useful when the method set changes but the sweep has not, or when --latex was not passed. It carries mean_pen_under/over, rejects and the timing sums through to SimResult -- dropping them silently zeroes the Penalty, Type-I, Power and Time columns, which is easy to miss. Verified against the run's own printed summary. compress_tables.py's ci_paired_nested caption updated for the new default. Numbers behind the swap, from runs at the paper's own grid (n=10..100, runs=5): real inspect reps=1000, bonett_price_shrunk Cov .976 / MinCov .947 / Width .2945 / Score .3140 -- the highest worst-case coverage of any method in that table, and 0 of 780 cells below .93. Synthetic reps=600: MinCov .927, 1 of 576 cells below .93. mj_floor_cluster attains a lower score (.2895 real, .2772 synth) but with 25/780 and 100/576 cells below .93. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…I CI set
PPI_BONETT_PRICE is now a distinct Method ("ppi_bonett_price") rather than
reusing ci_paired's BONETT_PRICE instance. The two estimate the same paired
binary difference, but they are reported in different sweeps and a shared name
made a `bonett_price` row ambiguous between the corrected and uncorrected
estimand -- the same reason PPI_WILSON is not named "wilson". order_present_
methods' guard caught the missing REPORT_METHOD_ORDER entry, which would
otherwise have dropped it silently from every table and plot.
All four PPI CI methods now share their non-PPI counterpart's colour
(bonett_price #556b2f, wilson #e377c2, t_interval #8c564b, logit_t #a6761d) so a
method reads the same across the ci_paired and PPI figures. Safe because the
colour test enforces distinctness only within co-plotted groups and no figure
draws a PPI method beside its non-PPI namesake; across the full co-plotted
nonstandard set the minimum separation is dE 20.2, floor is 12.
Two bugs the rename surfaced, both of which would have shipped:
- _PPI_CI_COMPARISON_TESTS still listed mj_floor, so the binary paired method was
dropped from ci_methods_comparison.png -- the figure that exists to show it.
The plot title also still read "(Tango / ...)", two renames stale.
- ppi_real's paired_bias dispatch passed a hardcoded [MJ_FLOOR.name], so the
real-data effect path emitted mj_floor even after _paired_methods_for was
switched. The PPI_BONETT_PRICE block added earlier was dead code. Removed the
now-unreachable MJ_FLOOR block and updated the docstring.
test_latex_tables' "nested binary" colour group was stale in the same way:
it listed bonett_price_cluster (no longer plotted) and omitted
bonett_price_shrunk (now the default), so the new method's colour was unchecked.
Nearest neighbour is mj_floor_flat at dE 19.4.
simulations/replot_ppi_effect.py rebuilds the PPI effect figure from a results
CSV, as relatex_ci_paired.py does for the ci_paired tables. Note --ci-comparison
and --nonstandard select different method sets; the real-data figure needs
--nonstandard alone or the paired/single split collapses.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
load_from() accepted a DataFrame or a list of dicts, so the first thing every user had to write was a pandas import and a read_csv. It now also accepts a str or PathLike and reads it: .tsv/.tab as tab-separated, .jsonl line-delimited, .json, .parquet, and anything else as CSV. A missing path raises EvalLoadError rather than a bare FileNotFoundError, and read failures are wrapped with the path in the message. This is what the paper's usage examples show, so the toolkit section's one-call story is now literally one call. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two changes to the terminal summary. Display order no longer forces the rank bootstrap. Four call sites read rank_dist.expected_ranks/p_best purely to decide row order, which meant the opt-in rank distribution was computed whether or not the caller asked for it. A new _display_order() sorts by descending mean with the label as tiebreak: free, deterministic, and already how the leaderboard sorts elsewhere. p_best/expected_ranks are now read only inside the show_rank_probabilities block, and label lookups moved from cross.rank_dist.labels to cross.labels so they need no distribution either. The documented gradient CI levels were wrong. config.py and _gradient_interval_line both described 90/95/99/99.9% bands, but GRADIENT_CI_ALPHAS = (0.32, 0.10, 0.05, 0.01) yields 68/90/95/99%, which is what the terminal legend has been printing all along. Corrected both, and noted that bands pair positionally to sorted(multi_ci) so a caller supplying its own alpha ladder still gets outermost-to-innermost shading. Also replaced the dead mean_idx local with the reason the mean is deliberately left unmarked: a point marker invites reading one value as the answer and the interval as decoration, which is the reading gradient plots exist to avoid. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…straps The bias check compares a correlation metric against ICC(2,1), so the only thing it can detect is a systematic shift or compression of the judge's raw scores. On a pass it printed "No sign of judge bias", which reads as a licence to skip PPI correction. It is not one. The bias PPI is most needed for errs in different directions across the conditions being compared, and those two errors cancel in any pooled statistic, including this check, so a clean result here is the least informative moment to conclude correction is unnecessary. PPI also absorbs the bias this check does see, so the result never changes what to do. Verified: a judge compressed to 0.55x with a +0.9 offset reports a true +0.50 effect as +0.27 raw, and the corrected estimate returns +0.51. The pass line is therefore gone from the compact summary; silence means nothing was flagged. The failing branch stays, since a compressed judge is worth knowing about, but is rescoped to "Possible judge scale bias" and now says the effect is on raw scores only. Both branches remain in summary(verbose=True), where the surrounding what/why/interpretation text supplies the context. Note the MCAR/representativeness check keeps its pass line, because PPI's validity genuinely depends on it. Separately, judge_alignment() gains ci=True. With ci=False every bootstrap CI on the alignment metrics is skipped and the bounds report NaN, keeping the closed-form point estimates. Each CI is 2000 resamples and they dominate the call's cost, so this is for callers that read only the estimates -- compare(alignment=...) reads nothing else -- and for large simulation sweeps. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The script's defaults had drifted from the scenario as written, so running it produced numbers that did not match §7.6. N_LAB was 15 while the paper labels 30 per app, the draw was seed 42, and the fourth app mapped to "Bubblegum" where the paper says "Razzletazz". Set N_LAB=30 and name the draw PAPER_SEED=8, which is the one the paper shows: pooled kappa .78, Pearson .79, Spearman .75, ICC .78 on the labeled subset, and theta .457 / p 1.000 / gap -0.413 on FlipFlop-Wavelength. Per-app kappa comes out 0.65 for FlipFlop against 0.77-0.88 for the others, which is the spread the paper's per-condition argument rests on. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
drift.log (a 500KB pvalues sweep dump) and texput.log (pdflatex scratch) had been sitting untracked for a while. Neither is worth versioning, and both regenerate. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
ResultReport._show_rank_probabilities already told an opt-in story about P(Best)/E[Rank] in the OUTPUT, but the work was unconditional: analyze() called bootstrap_ranks() eagerly, and a good deal of code read rank_dist.labels purely as the canonical label list, forcing a rank bootstrap to obtain a list of strings. LazyRankDistribution keeps labels/n_bootstrap free and runs the bootstrap once, cached, on the first read of rank_probs/expected_ranks/ p_best. Labels now come from AnalysisBundle.labels, sourced from benchmark.template_labels, which is the list core.router already feeds to every downstream construction; unpaired.py drops the SimpleNamespace(labels=...) stand-in it kept only to satisfy that read. Both the classical path (router._analyze_single) and the PPI path (api._run_alignment_ppi) construct the lazy form. The bit generator's state is snapshotted at construction rather than the live rng being held, so a deferred bootstrap draws exactly what an eager one would have. One behavioural consequence to be aware of: the parent rng is no longer advanced when the ranks go uncomputed, so downstream draws differ from the old always-compute behaviour. Results remain reproducible, but are not bit-identical to before this change. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…t hid
_simultaneous_cis_router re-listed the per-data-kind CI formulas inline,
and simulations/harness/cases/pvalues.py listed them a third time in its
own _canonical_ci_func. Three copies of the same table drift, and this
one had: at HEAD the harness returned mj_floor for binary and a rescaled
logit_t for the bounded/Likert case, while the library had since moved
binary to Bonett-Price and Likert to NIG. The simultaneous-CI numbers
were therefore measured on formulas evalstats no longer reports for
those data kinds.
canonical_pairwise_ci_func(data_kind, diff_bounds, method) is now the
single answer to "which interval does a pairwise difference get?". It
prefers the already-resolved pairwise method -- analyze() resolves
method="auto" once and passes the concrete name down -- so the
simultaneous CI widens the very interval the pairwise row reports rather
than forming a second, independently-derived opinion about the same
data. data_kind is the fallback, for resampling methods with no closed
form to widen. It returns None for "unbounded", whose construction needs
a degenerate-sample fallback the caller supplies.
bonett_price_paired_ci_from_diffs() supports this: the simultaneous path
works from each comparison's stored per_input_diffs and previously had
no diffs-based Bonett-Price to call, which is why it widened mj_floor
instead. Bonett-Price depends on the raw pairs only through the two
discordant counts and n, all three of which diffs in {-1, 0, 1}
determine, so it rebuilds a representative pair and delegates -- keeping
one copy of the formula.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…e paper's plotters The rho-drift control compared each method's implied rho^2 against its score-level rho^2 and failed on any deviation over 5%, with no notion of how noisy the measurement was. At the 200 replicates it defaulted to, that threshold is inside the noise: a variance from R replicates carries relative SE ~sqrt(2/R), and paired structures converge ~4x slower still, because D = truth_x - truth_y is heavier-tailed than either group's raw scores. paired_t's long-standing "residual LEVEL offset" was exactly this -- -17.6% at R=200, -3.8% at R=600, +0.3% at R=1500 on the same draws. STATUS item 3 is retired accordingly. RhoDriftPoint/PPIComparisonResult now carry rho2_implied_se from _var_ratio_bootstrap_se, a paired bootstrap over replicate indices (chunked to cap peak memory). The drift table gains a +-MC column propagated from its endpoints, and the control reports UNDERPOWERED, with the replicate count that would settle it, when a deviation sits within 3 sigma -- rather than claiming a failure it cannot support. --rho-drift-only now defaults to 2000 replicates, not 200. Also fixes a silent wrong-number bug: _method_rho2 took no shape_label, so it always computed against the default truth shape while being cached on a key that omitted which shape was asked for. It now takes and keys on shape_label, with --rho-drift-shape to drive it, and run_ppi_rho_drift_check takes only_methods to sweep one method at a time. pool.map became pool.imap so a long sweep reports per-cell progress and an ETA instead of going dark. compare_e2e passes ci=False to judge_alignment: compare()'s PPI correction reads only the alignment point estimates, so the per-metric bootstrap CIs are pure cost there -- verified output-identical, and ~72% of a PPI cell's runtime. New plotting scripts, each producing a figure the paper prints: plot_rho_drift_esinv.py (the 1x4 effect-size-invariance row, showing mean-type estimands flat and rank-type ones drifting from their recipe), replot_typeI_violin.py, replot_five_way.py, replot_labeleff_compact.py, and run_rho_drift_fig16.py. replot_label_efficiency.py gains the compact lookup-row and noise-family variants. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`boot` derives a joint critical value, converts it to an effective alpha through the normal CDF, and hands that to whichever closed-form CI resolved. That composition is exact only when the CI's own critical value is Phi^-1(1-alpha/2). Bonett-Price is normal-referenced, so binary is served exactly -- but NIG is a t interval at df=2*a_n and logit-t is symmetric on the logit scale, so those two inherit a mismatch. It shows: Likert family coverage sits at 0.943 pooled and 0.937 at n=15, worst-case 0.883, degrading as k grows. boot_cal removes the assumption by asking each formula for the level at which a resample's interval just covers a target, so no reference distribution of ours enters the calibration. Providers for Bonett-Price, NIG and logit-t are each verified exact against a brute-force bisection search. It stays method-agnostic -- a formula without a provider falls back to the scalar loop -- and costs O(B*k) rather than the nested B^2 a naive recalibration would need, because for an interval whose half-width scales with the critical value the crossing level has a closed form. It is NOT recommended and is deliberately absent from the paper: it recovers nominal coverage on Likert but does not beat Sidak on worst-case coverage or interval score, and a fifth construction is not worth the complexity. Reachable via prefer="boot_cal". Also corrects official_args_simultaneous_ci's docstring, which claimed the preset selects the smaller "standard" scenario catalog when the code sets "expanded" -- and counted that non-override among its overrides. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
AUTO_SIMULTANEOUS_CI_METHOD_TABLE encoded four rules (binary <50, binary >=50, numeric <30, numeric >=30). It is now one rule per data kind, both saying Sidak: no sample-size threshold, no data-type exception. Sidak was the only construction whose WORST-CASE family coverage held across the expanded scenario suite -- 0.913 to 0.943 for every eval type and N. The joint bootstrap is better centred on average and 3-5% narrower, but its worst case collapses to 0.50 on sparse/lopsided binary at n=15 (0.74 at n=30), and it under-covers Likert at every N. max-T is worse still under the alternative. The width Sidak gives up is small and bounded. Tukey's studentized range is the optimal equal-width procedure for all-pairwise comparisons and beats Sidak by only 1.8-3.0% -- a bound that applies here because the shared-arm contrast correlation really is 0.5, measured at 0.498-0.500 across the real eval corpora. Tukey itself needs normality and homoscedasticity (and sphericity in the repeated-measures form that applies to paired evals), which binary and Likert data violate. So Sidak sits within ~3% of the achievable optimum while assuming nothing. Nine tests encoded the old default. Rather than flip assertions, each was updated to keep its original intent: the joint-bootstrap route stays covered via an explicit prefer="boot" (including a new test_router_boot_still_reachable_when_preferred), and the PPI resample-sharing test now drives _run_alignment_ppi directly, since boot is no longer auto-reachable and compare() does not forward prefer= there. One test changed premise rather than expectation: under the old boot default, partial labeled-item overlap downgraded the WHOLE comparison to Bonferroni. Sidak needs none of that structure, so it no longer does -- a real improvement, now documented in the test. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
paper_flipflop_example.py shows one draw of 30 labeled items per app, and a reader is entitled to ask whether that draw was lucky. This re-runs the whole corrected analysis over many independent draws and reports the distribution, which is what the scenario's robustness footnote cites. The footnote it replaces rested on 8 draws and was underpowered enough to be wrong: it claimed the corrected omnibus "never rejects" (it rejects in 2 of 150), put the median p at 0.48 (0.71 over 150), and gave a range of 0.19-0.80 against a true 0.018-0.999. A min and a max from 8 draws will always understate a range, being the statistics most sensitive to n. At the paper's settings the shown draw turns out to be the median draw on theta -- rank 1 of 150, off by 0.0004 -- while its omnibus p sits at the 23rd percentile, so it is a slightly harder case for the correction than a typical draw. Both facts are now in the footnote. The labeling draw and the bootstrap seed vary together here on purpose: pinning the bootstrap would understate the spread a reader re-running the example sees. The opposite rule applies when SELECTING a draw rather than characterising the spread -- hold the bootstrap fixed there, or the selection lands on bootstrap noise -- and the docstring says so. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The builder put the whole shift on arm 0 and left arms 1..k-1 at baseline, which is what cases/pvalues.py's multiarm and simultaneous_ci sweeps want. cases/compare_e2e.py needs the other convention -- arm i shifted by i*delta, so the arms form a ladder and arm 0 vs arm k-1 is the widest gap, which is what its power column measures on a leaderboard. effect_mode="ramp" adds that without touching the existing default or the generate_scores signature: only the interpretation of `delta` changes (the per-arm step rather than the total shift), so nothing downstream needs to know which mode built a source. The ramp needs its own true_means. The arm0 variant caches two scalars and estimates each by calling generate_scores with k=1 -- which silently returns baseline under a ramp, since arange(1)*delta is zero. The ramp version draws the full k-arm array once and takes each arm's mean instead. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The no-PPI arm layered _apply_judge_noise over sample_group_truth's own icc-controlled noise. Both model rater unreliability, so the arm ran at r(arm_i, arm_j) = 0.37 -- below the lowest point (0.49) ci_paired's official icc sweep validates. The CI methods were being exercised outside the regime they were tuned in, and it showed: Likert family coverage sat at 93.7% (z=-1.9) at n=30, because NIG's prior scale is calibrated against the dispersion of a single noise layer. Judge noise inflated the paired- difference SD by ~35%, the prior shrank the posterior variance back toward its own scale, and the intervals came out too narrow. Judge noise now applies only on the PPI path, where it carries the differential bias PPI exists to remove. The no-PPI arm analyses the truth draw directly, which makes its DGP bit-for-bit identical to build_multiarm_sources(effect_mode="ramp") -- the same builder the simultaneous-CI and multiarm sweeps use. That is the claim the paper needs in order to read these columns next to those sweeps, so it is asserted by a test rather than left to prose. Likert n=30 goes 93.7% -> 95.8% (z=+1.3). Both arms now target truth_means: PPI cells always did, and the no-PPI arm's old llm_means target coincides with it once the noise is gone (measured max difference 0.003 at zero bias). The 200k-item second draw that computed llm_means is skipped rather than discarded. --icc-values sweeps the signal/noise split on the no-PPI arm, defaulting in official_args to 0.05/0.20/0.60 (+70% cells; the full five ci_paired uses would be +139%). A single icc is exactly what let NIG's dispersion sensitivity go unseen. 0.20 is the realistic point: cross-model correlation measured on the real corpora is 0.146 mean / 0.103 median, and icc=0.20 reproduces r=0.170. Note this is the CROSS-ARM correlation, not the multi-run ICC (~0.68), which is a different axis and is not exercised here at runs=1. icc is recorded on the result and in the CSV: without it two cells differing only in icc are indistinguishable in the saved data and pool silently in every aggregation. While adding it, the CSV became a lossless serialization of every CompareE2EResult field -- the reference-arm coverage counts appeared nowhere, and rebuilding width/score sums from the rounded means lost precision. A regenerated plot is now byte-identical to the run's own. Two tests guard that: one asserting no field is missing, one catching repr() on a numpy scalar, which writes "np.float64(...)" and silently reads back as a string. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
official_args carried icc_values while the --icc-values argparse default was None, so a bare CLI run built a 4232-cell single-icc grid while the preset built 7176 -- with nothing in the output saying which you got. A full-grid smoke run at reps=3 surfaced it: every check that had been written down passed (0 errors, every eval type, sizes to 1000, k to 10, PPI cells at the largest n, lossless CSV, LaTeX, plots) and the run still quietly reported "icc values: [0.2]". DEFAULT_ICC_VALUES is now the single constant both read, with the reasoning for the three points recorded on it, and a test asserts the two agree. That failure mode produces valid-looking numbers rather than an error, so it needs pinning rather than remembering. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
--eval-types defaulted to None in ci_paired and ci_single. The consuming code is `if args.eval_types:`, so None is falsy and no filter ever applied: a bare CLI run swept all of EVAL_TYPES, including "grades", while every official preset pins ["binary", "continuous", "likert"]. The failure is silent -- a larger sweep, not an error -- and it reached the paper. The multi-run pairwise CI table carried 15 grades rows across a full extra eval-type block that the official test would never have produced. scenarios.DEFAULT_EVAL_TYPES is now the one definition both the presets and the CLI defaults read, with the reason grades is excluded recorded on it (it is "continuous" rescaled to 0-100, so it costs a third of the runtime for coverage the continuous column already gives; "likert" stays as the genuinely distinct integer/few-level case). Passing --eval-types grades still works for anyone who wants it. compare_e2e and pvalues were already safe -- the former restricts `choices` to the three types, the latter has its own explicit default -- but the test covers all four cases, since the next case added is the one that will get this wrong. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two changes, both needed to regenerate fig:five-way-comparison-omnibus from its own CSV rather than leaving it as the only figure showing an eval type the official PPI preset excludes. ncol was len(ETS), fixed at three. The omnibus sweep has no binary arm and (post-fix) no grades, so it rendered a panel-wide empty gap. Columns now come from the eval types actually present. Verified a no-op for the main-text figure by A/B-ing patched against unpatched on identical inputs: byte-identical output when all three types are there. --effect-max clips the effect-size row's x axis. Every arm saturates well before the sweep's largest effect, so the tail is flat lines taking a third of the width; cutting it at 0.8 gives the region where the arms actually separate room to be read. Off by default. Scoped to that row only -- N_lab is a budget axis, not a saturating one, and the same cut there would drop signal rather than a flat tail. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The old page had grown out of sync with the paper's recommendations (sample-sigconf.tex) and was cluttered with stale simulation tables. Replace it with a web-native rendering of the two forest decision trees (CI method, p-value/FWER correction) plus a note that the full paper is forthcoming. Adds a reusable .dtree CSS component. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The old README's "Statistics" section described a set of pairwise/CI defaults (smoothed bootstrap, bayes_evals, Benjamini-Hochberg) that no longer match evalstats/config.py's actual method="auto" routing table (Wilson/Logit-t/Bonett-Price/NIG, Sidak/Shaffer/Romano-Wolf). Replace that section with the paper's two forest decision trees, rendered from sample-sigconf.tex via a standalone LaTeX build and rasterized to PNG. Also: add a Citation section, document the previously-undocumented `evalstats label` CLI command, merge duplicate screenshot sections, add a table of contents and badges, and collapse advanced/optional content (raw BenchmarkResult API, extra PPI test examples, pymer4/R setup) into <details> blocks to cut visual weight without losing content. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The site had drifted out of sync with the current package (last substantive update ~4 months ago). This pass: - Deletes choose.html: every one of its 9 generated code samples called report.quick_summary(), which doesn't exist (verified via the venv); it also hardcoded now-wrong defaults (fdr_bh, bayes_binary) throughout. Not worth salvaging piecemeal. - Removes the same stale "Bayesian paired below N<100, smooth bootstrap default" narrative from index.html, principles.html, and resources.html -- superseded by config.py's unified Bonett-Price / Logit-t / NIG routing. - Fixes a leftover "prompstats" typo and a "99% CIs by default" claim (actual default is 95%) in principles.html. - Regenerates usage.html's three terminal-output examples by actually running the package rather than hand-editing the old mockups -- the output format itself had changed (CI gradient bars, Sidak/Romano-Wolf labeling), not just the method names. - Refreshes roadmap.html to reflect PPI correction, judge auditing, and FWER control as shipped, and lists the real current gaps (between-subjects support, multi-run PPI). - Fixes two dead #debunking anchor links in resources.html. - Uncomments the loop that populates the site-wide top nav (was rendering completely empty on every page -- a pre-existing bug, not introduced here) and fixes three dead anchors it pointed to. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.