Skip to content

docs: re-validate shipped defaults.py hyperparameters with replication (#111) - #115

Merged
jc-macdonald merged 4 commits into
mainfrom
docs/111-revalidate-defaults
Aug 19, 2026
Merged

docs: re-validate shipped defaults.py hyperparameters with replication (#111)#115
jc-macdonald merged 4 commits into
mainfrom
docs/111-revalidate-defaults

Conversation

@jc-macdonald

Copy link
Copy Markdown
Collaborator

Summary

Per the agreed scoping (targeted re-validation, not a full Option A re-derivation): fixed the underlying seeding gap, built a replicated validation script for the exact configs recommend_config() ships today, ran it, and corrected the docstring's unsupported claim using the results.

What was found

Root cause confirmed: VBPCASimulator.generate() had two seeding gaps. The data-generation seed was pinned to a single fixed constant for every trial (never varied across "replicates"), and it never passed random_state to VBPCA/select_n_components at all — meaning after #109 shipped, this pipeline actually got worse: VBPCA's new default (random_state=None) draws fresh, uncontrolled entropy on every trial instead of the old accidental seed-0 determinism.

"Validated ≠ shipped" gap confirmed: v3_compare.py's existing replicated comparison (reps=10) reads optimized configs from older per-family JSON artifacts, not from defaults.py's shipped _BUCKET_CONFIGS. No replicated validation existed anywhere for the exact values that actually ship.

Changes

  1. analysis/trade_study/_world.py: VBPCASimulator.generate() gains an optional rep keyword (opting into trade-study's replicated-trials convention, jcm-sci/trade-study#112), deriving independent data and VBPCA-init seeds from it.
  2. analysis/trade_study/validate_shipped_defaults.py (new): compares the raw library default against whatever recommend_config(n, p) returns today, replicated (--n-reps, default 8) across the trade study's training + held-out validation regimes, using trade-study's run_grid(n_reps=...). Paired seeding (common random numbers) isolates the config-choice effect from other randomness.
  3. src/vbpca_py/defaults.py docstring: corrected the unsupported "hp_va is the dominant lever" claim — the trade study's own marginal sensitivity data doesn't support it (xprobe_fraction is the strongest per-bucket predictor of rank_mae in that view; hp_va's marginal correlation is weak/non-significant in 2 of 3 buckets). Rather than assert a replacement attribution, the docstring now says plainly that which factor is dominant is unreconciled.
  4. Ran the new validation script (n_reps=8, seeded): replicated rank_mae for the shipped config is 28-58% lower than the library default across all three p-buckets (smallp 0.34→0.14, trans 1.20→0.74, large 1.44→1.04), at a small cost in holdout RMSE (+0.4-3.8%). Documented in the docstring — this directly answers the practical question Re-validate defaults.py's bucketed hyperparameters with replication; docstring's "dominant lever" claim isn't supported by the study's own sensitivity data #111 raised ("does what ships actually help"), even though the deeper attribution question (which factor drives it) remains open.

Results JSON (analysis/results/optionA/shipped_defaults_validation.json) is a local artifact, consistent with the existing convention for this directory (none of analysis/results/optionA/*.json are git-tracked — all gitignored, regenerable).

Closes #111

Test plan

  • just ci green locally (lint, format, mypy --strict on src, coverage 90.21%) — analysis/ isn't part of the mypy/coverage gate, consistent with existing convention
  • Smoke-tested rep seeding directly: different reps produce different synthetic data and different VBPCA fits for both "default" and "shipped" conditions
  • Ran the full validation (n_reps=8, 368 trials, ~2.5 min on 24 cores) and verified the printed summary matches the JSON output

VBPCASimulator.generate() had two seeding gaps compounding #111's "no
replication" problem: the data-generation seed was pinned to a single
fixed RNG_SEED for every trial (never varied), and it never passed
random_state to VBPCA/select_n_components at all -- meaning that after
#109 shipped, this pipeline actually got *worse*: VBPCA's new default
(random_state=None) draws fresh, uncontrolled entropy on every trial
instead of the old accidental seed-0 determinism.

generate() now accepts an optional rep keyword (opting into
trade-study's replicated-trials convention, jcm-sci/trade-study#112)
and derives independent data and init seeds from it, so
run_grid(..., n_reps=N) produces genuinely independent, reproducible
replicates instead of N copies of one draw.
)

v3_compare.py's replicated comparison (reps=10) never actually
validated the exact configs recommend_config() ships today -- it reads
optimized configs from older per-family JSON artifacts instead of the
shipped module. This adds a targeted script that compares the raw
library default against whatever recommend_config(n, p) returns,
replicated (--n-reps, default 8) across the trade study's training and
held-out validation regimes, using trade-study's new run_grid(n_reps=)
support (jcm-sci/trade-study#112) and the previous commit's seeding
fix. Both conditions share a base seed per regime (common random
numbers) to isolate the config-choice effect from other randomness.

Usage: python -m analysis.trade_study.validate_shipped_defaults
…ence

The docstring claimed hp_va was "the dominant lever" behind correct
rank recovery. The trade study's own marginal sensitivity data doesn't
support that framing -- xprobe_fraction is the strongest, most
significant, most consistent per-bucket predictor of rank_mae, and
hp_va's marginal correlation is weak and non-significant in 2 of 3
buckets. Rather than assert a replacement attribution the surrogate's
joint-optimum choice may still justify via factor interactions a
marginal view can't see, the docstring now states plainly that which
factor (if any) is dominant is unreconciled.

Separately: ran the new validate_shipped_defaults.py (n_reps=8,
seeded, training + held-out regimes) to check the practical question
issue #111 actually cared about -- does what ships help. It does:
replicated rank_mae for the shipped config is 28-58% lower than the
library default across all three p-buckets, at a 0.4-3.8% cost in
holdout RMSE. Documented in the docstring.

Closes #111
@jc-macdonald
jc-macdonald merged commit 738a6fe into main Aug 19, 2026
7 checks passed
@jc-macdonald
jc-macdonald deleted the docs/111-revalidate-defaults branch August 19, 2026 14:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant