Problem
RegimeSurrogate.recommend()/predict_batch() return point predictions, and callers (e.g. analysis.trade_study._surrogate.recommend_ranked in VBPCApy's trade study) do a raw argmin over a candidate pool against those point predictions. When the underlying training data is itself noisy (see #112) and the surrogate's own accuracy is unquantified (see #114), an unconstrained point-argmin is exactly the wrong aggregation — it actively seeks out whichever candidate the surrogate is most wrong about in an optimistic direction, rather than the candidate that's robustly good.
Ask
Some combination of:
- Expose per-candidate prediction uncertainty from
predict_batch() (e.g. RF inter-tree spread, or GP posterior variance when method="gp"), not just point estimates.
- A
recommend() variant that optimizes a robust/pessimistic criterion under that uncertainty (e.g. upper-confidence-bound-style: minimize predicted_mean + k * predicted_std for a minimize-direction objective) rather than the raw point prediction, and/or supports bootstrap-resampled refits to check recommendation stability.
This would directly address a case in VBPCApy's trade study where the shipped hyperparameter recommendation for hp_va landed in what the project's own marginal sensitivity analysis calls the worse-performing tercile for 2 of 3 regime buckets (see also #113) — plausibly a real joint-optimum the marginal view can't see, but equally plausibly the point-argmin latching onto surrogate noise. There's currently no way to distinguish the two, and no tool in trade-study to help.
Problem
RegimeSurrogate.recommend()/predict_batch()return point predictions, and callers (e.g.analysis.trade_study._surrogate.recommend_rankedin VBPCApy's trade study) do a rawargminover a candidate pool against those point predictions. When the underlying training data is itself noisy (see #112) and the surrogate's own accuracy is unquantified (see #114), an unconstrained point-argmin is exactly the wrong aggregation — it actively seeks out whichever candidate the surrogate is most wrong about in an optimistic direction, rather than the candidate that's robustly good.Ask
Some combination of:
predict_batch()(e.g. RF inter-tree spread, or GP posterior variance whenmethod="gp"), not just point estimates.recommend()variant that optimizes a robust/pessimistic criterion under that uncertainty (e.g. upper-confidence-bound-style: minimizepredicted_mean + k * predicted_stdfor a minimize-direction objective) rather than the raw point prediction, and/or supports bootstrap-resampled refits to check recommendation stability.This would directly address a case in VBPCApy's trade study where the shipped hyperparameter recommendation for
hp_valanded in what the project's own marginal sensitivity analysis calls the worse-performing tercile for 2 of 3 regime buckets (see also #113) — plausibly a real joint-optimum the marginal view can't see, but equally plausibly the point-argmin latching onto surrogate noise. There's currently no way to distinguish the two, and no tool in trade-study to help.