feat: replicate averaging in run_adaptive()/screen()/sobol_indices() - #124
Merged
Conversation
…122) run_grid(..., n_reps=N) (#112) detects rep-aware simulators and varies the replicate index; run_successive_halving/run_hyperband do too via the same _run_single/_generate_accepts_rep helpers. run_adaptive() (NSGA-II) and screen()/sobol_indices() (Morris/Sobol) didn't -- they always evaluated a single, fixed draw. Found deriving VBPCA hyperparameter recommendations via run_adaptive for extreme-aspect-ratio data regimes (yoavram-lab/VBPCApy#116): the same fragility class a different investigation already hit expensively (yoavram-lab/pp-eigentest#29 -- a single-draw Morris screen produced a "hyperparameter matters" finding that turned out to be a threshold artifact of one data draw, not a real effect). run_adaptive gains n_reps (mirrors run_grid's Simulator convention); screen()/sobol_indices() gain n_reps applied to the bare run_fn callable instead, via the same rep-keyword introspection convention. Both default to n_reps=1 (today's behavior). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #122.
run_grid(..., n_reps=N)(#112) andrun_successive_halving/run_hyperbandall supportrep-aware simulators (introspection-detected).run_adaptive()(NSGA-II) andscreen()/sobol_indices()(Morris/Sobol) didn't -- both always evaluated a single, fixed draw, no matter how many trials/trajectories were run.Why this matters
Found using
run_adaptiveto derive VBPCA hyperparameter recommendations for extreme-aspect-ratio regimes (yoavram-lab/VBPCApy#116). It's the same fragility a different investigation already hit expensively: yoavram-lab/pp-eigentest#29 -- a single-draw Morris screen (screen()doesn't varyrepeither) produced a "hyperparameter matters" finding that turned out to be a threshold artifact of one particular data draw, not a real effect.Changes
run_adaptive(..., n_reps=1): forn_reps>1, averages each trial's objective(s) overn_repsreplicate draws before returning them to Optuna. Detectsrep-aware simulators the same wayrun_griddoes (reuses_generate_accepts_rep).screen(..., n_reps=1)/sobol_indices(..., n_reps=1): same averaging, applied to the barerun_fncallable these take instead of aSimulator-- a new_run_fn_accepts_rep()mirrors the introspection convention forrun_fn(cfg, *, rep=0).n_reps=1, reproducing today's behavior exactly.Test plan
just ci(ruff format/check, mypy --strict, pytest --cov) passes at 99.4% coveragen_repsrejects<1; defaultn_reps=1matches pre-run_adaptive() and screen()/sobol_indices() don't support rep-aware simulators like run_grid does #122 output exactly; a non-rep-awarerun_fn/simulator gets calledn_repstimes with an identical draw (redundant compute, correct result); a rep-aware one receives incrementingrep=0..n_reps-1and the returned scores reflect the true mean across draws (verified numerically forrun_adaptive, and via observed-rep-set forscreen/sobol_indices)🤖 Generated with Claude Code