feat: migrate sensitivity_report.py to trade_study.sensitivity_from_table() - #119
Merged
jc-macdonald merged 1 commit intoAug 21, 2026
Merged
Conversation
Replaces the hand-rolled marginal Spearman correlation with trade_study.sensitivity_from_table() (jcm-sci/trade-study#113) for the 9 continuous VBPCA hyperparameters, per p-bucket. Unlike Spearman, this correctly detects non-monotonic (e.g. U-shaped) effects -- exactly the kind defaults.py's own docstring already documents for hp_va (a real nonlinear U-shape the marginal-correlation method this migration retires understates). surrogate_cv_r2 is reported alongside each bucket's indices so a poorly-fit surrogate's numbers aren't mistaken for a reliable finding. Sobol/Morris both require continuous factors (sensitivity_from_table silently drops anything else, matching screen()'s contract), so the 4 discrete VBPCA hyperparameters (niter_broadprior/maxiters/patience/ rmsstop_window) keep the original marginal-Spearman treatment, explicitly labeled "spearman_marginal" in the output rather than mixed in unlabeled with the Sobol results. Verified against the real, current surrogate_train.json: xprobe_fraction dominates rank_mae sensitivity in all three buckets (S1=0.46-0.77), corroborating -- via a completely different method -- the same marginal-Spearman finding defaults.py's docstring already documents ("xprobe_fraction is the strongest, most significant, and most consistent per-bucket predictor"), not hp_va despite it being called the "dominant lever" there. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
jc-macdonald
deleted the
feat/sensitivity-report-use-sensitivity-from-table
branch
August 21, 2026 12:30
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Replaces the hand-rolled marginal Spearman correlation in
sensitivity_report.pywithtrade_study.sensitivity_from_table()(jcm-sci/trade-study#113, built for exactly this) for the 9 continuous VBPCA hyperparameters, per p-bucket.Why this matters
Unlike Spearman, Sobol-via-surrogate correctly detects non-monotonic (e.g. U-shaped) effects -- exactly the kind
defaults.py's own docstring already documents forhp_va: "its tercile means do show a real, nonlinear U-shape a monotonic Spearman correlation understates." That gap was known but this script kept using the method that produces it.sensitivity_from_tableonly screens continuous factors (matchingscreen()'s own contract), so the 4 discrete VBPCA hyperparameters (niter_broadprior/maxiters/patience/rmsstop_window) keep the original marginal-Spearman treatment -- now explicitly labeled"spearman_marginal"in the output rather than mixed in unlabeled alongside the Sobol results.Verification
Ran against the real, current
surrogate_train.json(not synthetic data):xprobe_fractiondominates in all three buckets via this completely different method -- corroborating the same finding the old marginal-Spearman analysis already reported indefaults.py's docstring ("xprobe_fraction is the strongest, most significant, and most consistent per-bucket predictor of rank_mae"), nothp_vadespite it being called the "dominant lever" there. Figure regenerated cleanly (figure_sensitivity_rank_mae.pngetc.) with sane-looking bars.Test plan
just cipassessurrogate_train.json, inspected the regenerated figures and JSON output for sanity🤖 Generated with Claude Code