Enable BestTrials for Online Experiments#5129
Conversation
Summary: Pass use_model_predictions=has_batch_trials to BestTrials in ResultsAnalysis. Online experiments (which use BatchTrials) get GP model predictions for noise smoothing via shrinkage. AutoML and other direct callers get the default (False). Per design meeting 3/26: 'If online experiment, set model_predictions=True. If AutoML, say False. Set based on if has batch trials or not. For now.' --- > Generated by [Metamate](https://fb.workplace.com/groups/metamate.feedback/) [Metamate Session](https://internalfb.com/intern/bunny?q=eggmate%20f5ba8eb9-b3ef-4a16-86ce-18bb5271aff5), [Trace](https://www.internalfb.com/confucius?session_id=f5ba8eb9-b3ef-4a16-86ce-18bb5271aff5&tab=Trace) Differential Revision: D99128865
|
@shrutipatel31 has exported this pull request. If you are a Meta employee, you can view the originating Diff in D99128865. |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #5129 +/- ##
=======================================
Coverage 96.40% 96.40%
=======================================
Files 613 613
Lines 68171 68193 +22
=======================================
+ Hits 65721 65743 +22
Misses 2450 2450 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
|
Hi @shrutipatel31! Thank you for your pull request. We require contributors to sign our Contributor License Agreement, and yours needs attention. You currently have a record in our system, but the CLA is no longer valid, and will need to be resubmitted. ProcessIn order for us to review and merge your suggested changes, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA. Once the CLA is signed, our tooling will perform checks and validations. Afterwards, the pull request will be tagged with If you have received this in error or have any questions, please contact us at cla@meta.com. Thanks! |
Summary:
Pass use_model_predictions=has_batch_trials to BestTrials in ResultsAnalysis.
Online experiments (which use BatchTrials) get GP model predictions for noise smoothing via shrinkage. AutoML and other direct callers get the default (False).
Per design meeting 3/26: 'If online experiment, set model_predictions=True. If AutoML, say False. Set based on if has batch trials or not. For now.'
Differential Revision: D99128865