stack_proportional -- weight models by relative score, not winner-take-all - #128
Merged
Conversation
…er-take-all stack_scores() is a linear program over the simplex, so it always puts all weight on the single best-performing model whenever there's any nonzero gap -- including a gap that's just measurement noise between two genuinely near-tied models. In practice this makes the resulting weights unstable: an unrelated upstream bugfix that shifts one model's score by a hair, well within replication noise, can flip the "winner" and swing weights from an even split to 100/0. stack_proportional() instead scales weight smoothly with relative performance: w_m proportional to mean_score_m (inverted via max-score for minimize, so a score of exactly 0 doesn't blow up). Falls back to a uniform split when every model scores identically.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
stack_scores()is a linear program over the simplex, so it always puts all weight on the single best-performing model whenever there's any nonzero gap -- including a gap that's just measurement noise between two genuinely near-tied models. In practice (pp-eigentest's ensemble stacking) this made weights unstable: an unrelated upstream bugfix that shifted one model's score by a hair, well within replication noise, flipped the "winner" and swung weights from an even 0.5/0.5 split to 1.0/0.0.stack_proportional(score_matrix, *, maximize=False)instead scales weight smoothly with relative performance:w_m ∝ mean_score_m(inverted viamax - scoreforminimize, so a score of exactly 0 doesn't blow up). Falls back to a uniform split when every model scores identically.Test plan
ruff format --preview --check/ruff check --preview-- cleanmypy --strict src-- cleanstack_scores), dominant-model ordering (bothminimize/maximize), uniform fallback on ties, zero-score doesn't blow up, shape, dtype -- all passpytest(full suite) -- all pass, no regressions