stats: pareto() collapses judge replications per answer and skips missing cost/tokens - #36
Open
ChrisW09 wants to merge 1 commit into
Open
stats: pareto() collapses judge replications per answer and skips missing cost/tokens#36ChrisW09 wants to merge 1 commit into
ChrisW09 wants to merge 1 commit into
Conversation
- _quality_by_config averaged raw rating items, so answers with more surviving judge replications weighed more (pseudo-replication) — the Pareto quality could disagree with attribute() on identical data (verified: 4.0 vs 3.0) and flip dominance decisions. Judge reps are now collapsed to one value per answer first, the same rule the analysis frame documents and attribute() follows. - _resources_by_config counted a missing cost_usd/tokens as 0.0, diluting the mean toward zero so a config with unreported costs looked spuriously cheap. Missing values are now skipped, exactly like the latency handling two lines above; a genuinely reported 0.0 still counts (and constant-0 objectives are dropped as before). Fixes #6 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #6.
Bug 1 — pseudo-replication.
_quality_by_configappended one value per rating item, i.e. per judge repetition._frame.pydocuments that judge reps must be collapsed per answer ("Treating them as independent observations would be pseudo-replication") andattribute()complies — so the same evaluation reported quality 3.0 in the descriptive layer and 4.0 in the Pareto table (verified withjudge_replications=3where one answer lost 2 verdicts to judge errors). Quality is now the mean over per-answer means, keyed byobs_key.Bug 2 — missing-as-zero.
_resources_by_configappended0.0for absentcost_usd/tokensmetadata while correctly skipping absent latency two lines above. A config whose adapter failed to report cost on half its observations had its mean cost halved; a fully untracked config got cost 0.0 and could dominate everything on cost. Missing values are now skipped like latency; a genuinely reported0.0still counts, and all-missing objectives stay constant-0 and are dropped from the dominance test as before.Both fixes come with regression tests pinning the exact scenarios from the issue.
Test:
pytest packages/cafe-core/tests/test_pareto.py— 6 passed; full suite green apart from pre-existing #4.🤖 Generated with Claude Code