docs: re-validate shipped defaults.py hyperparameters with replication (#111) - #115
Merged
Conversation
VBPCASimulator.generate() had two seeding gaps compounding #111's "no replication" problem: the data-generation seed was pinned to a single fixed RNG_SEED for every trial (never varied), and it never passed random_state to VBPCA/select_n_components at all -- meaning that after #109 shipped, this pipeline actually got *worse*: VBPCA's new default (random_state=None) draws fresh, uncontrolled entropy on every trial instead of the old accidental seed-0 determinism. generate() now accepts an optional rep keyword (opting into trade-study's replicated-trials convention, jcm-sci/trade-study#112) and derives independent data and init seeds from it, so run_grid(..., n_reps=N) produces genuinely independent, reproducible replicates instead of N copies of one draw.
) v3_compare.py's replicated comparison (reps=10) never actually validated the exact configs recommend_config() ships today -- it reads optimized configs from older per-family JSON artifacts instead of the shipped module. This adds a targeted script that compares the raw library default against whatever recommend_config(n, p) returns, replicated (--n-reps, default 8) across the trade study's training and held-out validation regimes, using trade-study's new run_grid(n_reps=) support (jcm-sci/trade-study#112) and the previous commit's seeding fix. Both conditions share a base seed per regime (common random numbers) to isolate the config-choice effect from other randomness. Usage: python -m analysis.trade_study.validate_shipped_defaults
…ence The docstring claimed hp_va was "the dominant lever" behind correct rank recovery. The trade study's own marginal sensitivity data doesn't support that framing -- xprobe_fraction is the strongest, most significant, most consistent per-bucket predictor of rank_mae, and hp_va's marginal correlation is weak and non-significant in 2 of 3 buckets. Rather than assert a replacement attribution the surrogate's joint-optimum choice may still justify via factor interactions a marginal view can't see, the docstring now states plainly that which factor (if any) is dominant is unreconciled. Separately: ran the new validate_shipped_defaults.py (n_reps=8, seeded, training + held-out regimes) to check the practical question issue #111 actually cared about -- does what ships help. It does: replicated rank_mae for the shipped config is 28-58% lower than the library default across all three p-buckets, at a 0.4-3.8% cost in holdout RMSE. Documented in the docstring. Closes #111
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Per the agreed scoping (targeted re-validation, not a full Option A re-derivation): fixed the underlying seeding gap, built a replicated validation script for the exact configs
recommend_config()ships today, ran it, and corrected the docstring's unsupported claim using the results.What was found
Root cause confirmed:
VBPCASimulator.generate()had two seeding gaps. The data-generation seed was pinned to a single fixed constant for every trial (never varied across "replicates"), and it never passedrandom_statetoVBPCA/select_n_componentsat all — meaning after #109 shipped, this pipeline actually got worse: VBPCA's new default (random_state=None) draws fresh, uncontrolled entropy on every trial instead of the old accidental seed-0 determinism."Validated ≠ shipped" gap confirmed:
v3_compare.py's existing replicated comparison (reps=10) reads optimized configs from older per-family JSON artifacts, not fromdefaults.py's shipped_BUCKET_CONFIGS. No replicated validation existed anywhere for the exact values that actually ship.Changes
analysis/trade_study/_world.py:VBPCASimulator.generate()gains an optionalrepkeyword (opting into trade-study's replicated-trials convention,jcm-sci/trade-study#112), deriving independent data and VBPCA-init seeds from it.analysis/trade_study/validate_shipped_defaults.py(new): compares the raw library default against whateverrecommend_config(n, p)returns today, replicated (--n-reps, default 8) across the trade study's training + held-out validation regimes, using trade-study'srun_grid(n_reps=...). Paired seeding (common random numbers) isolates the config-choice effect from other randomness.src/vbpca_py/defaults.pydocstring: corrected the unsupported "hp_vais the dominant lever" claim — the trade study's own marginal sensitivity data doesn't support it (xprobe_fractionis the strongest per-bucket predictor ofrank_maein that view;hp_va's marginal correlation is weak/non-significant in 2 of 3 buckets). Rather than assert a replacement attribution, the docstring now says plainly that which factor is dominant is unreconciled.n_reps=8, seeded): replicatedrank_maefor the shipped config is 28-58% lower than the library default across all three p-buckets (smallp 0.34→0.14, trans 1.20→0.74, large 1.44→1.04), at a small cost in holdout RMSE (+0.4-3.8%). Documented in the docstring — this directly answers the practical question Re-validate defaults.py's bucketed hyperparameters with replication; docstring's "dominant lever" claim isn't supported by the study's own sensitivity data #111 raised ("does what ships actually help"), even though the deeper attribution question (which factor drives it) remains open.Results JSON (
analysis/results/optionA/shipped_defaults_validation.json) is a local artifact, consistent with the existing convention for this directory (none ofanalysis/results/optionA/*.jsonare git-tracked — all gitignored, regenerable).Closes #111
Test plan
just cigreen locally (lint, format, mypy --strict onsrc, coverage 90.21%) —analysis/isn't part of the mypy/coverage gate, consistent with existing conventionrepseeding directly: different reps produce different synthetic data and different VBPCA fits for both "default" and "shipped" conditionsn_reps=8, 368 trials, ~2.5 min on 24 cores) and verified the printed summary matches the JSON output