Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 8 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,8 @@ historical reproduction.
- Stable audit artifacts: metrics, plots, raw reports, trade logs, config JSON,
run manifest, and optional QuantStats HTML.
- Walk-forward and train/test optimization designed to avoid leaking OOS data
into parameter selection.
into parameter selection, plus full-sample robust calibration for final
production parameter discovery.

## Performance Philosophy

Expand Down Expand Up @@ -178,9 +179,14 @@ service creates a run.
- `mode_1_decay`
- `mode_2_sbb`
- `mode_3_flat_minima`
- `mode_4_is_only_robust`
- `mode_5_full_robust`
- Endpoint-backed scoring for supported single-symbol modes, so objective
metrics match the actual QuantBT backtest route.
- Train-only robust candidate selection such as `is_plateau_robust`.
- Train-only robust candidate selection such as `is_plateau_robust` and
`is_only_robust`.
- Full-sample robust calibration selectors: `full_robust`,
`full_plateau_robust`, `full_temporal_robust`, and `full_best`.
- Optional trade-count penalty to avoid overfit low-trade Sharpe traps.

### Nautilus Validation Reports
Expand Down
3 changes: 3 additions & 0 deletions __init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,7 @@
benchmark_walkforward_kernels,
logging_callback,
score_strategy_output,
select_full_sample_robust_record,
select_is_plateau_robust_record,
select_is_only_robust_record,
select_flat_minima_record,
Expand Down Expand Up @@ -258,7 +259,9 @@
"logging_callback",
"score_strategy_output",
"select_flat_minima_record",
"select_full_sample_robust_record",
"select_is_only_robust_record",
"select_is_plateau_robust_record",
"stationary_bootstrap_sharpes",
"synthetic_walkforward_sharpes",
"stitch_oos_outputs",
Expand Down
2 changes: 1 addition & 1 deletion docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ Use this page as the first stop when deciding which QuantBT document to read.
| Portfolio matrix | `QuantBTEndpoint.portfolio(...)` | Multi-symbol positions with portfolio-level accounting |
| Pair/basket | `QuantBTEndpoint.basket(...)` | Frozen hedge-ratio units and package diagnostics |
| Arbitrage | `QuantBTEndpoint.arbitrage(...)` | Domain specs for basis, stat-arb, funding, carry, and index-basket routes |
| Walk-forward optimization | `QuantBTEndpoint.walk_forward(...)` | Folded OOS stitching and anti-leakage candidate selection |
| Walk-forward optimization | `QuantBTEndpoint.walk_forward(...)` | Folded OOS stitching, anti-leakage candidate selection, and full-sample robust calibration |
| Single holdout train/test | `QuantBTEndpoint.train_test_split(...)` | One train period and one test period using the WFO scoring stack |
| Third-party validation | `QuantBTEndpoint.nautilus_validation(...)` or `backend="nautilus"` | Independent event-driven accounting reports |

Expand Down
48 changes: 47 additions & 1 deletion docs/endpoint.md
Original file line number Diff line number Diff line change
Expand Up @@ -1203,6 +1203,9 @@ wfo = QuantBTEndpoint.walk_forward(
# "dispersion_penalty": 0.50,
# "temporal_weight": 0.65,
# "plateau_weight": 0.35,
# mode_5_full_robust:
# "candidate_selection_metric": "full_robust",
# alternatives: full_plateau_robust | full_temporal_robust | full_best
# crypto default annualization: 365; equities often use 252
"scoring_trading_days": 365,
# optional under-trading penalty; None disables it
Expand Down Expand Up @@ -1313,7 +1316,8 @@ Important rules:
`monthly`, and `weekly`; `single` is used by
`QuantBTEndpoint.train_test_split(...)` for one holdout fold;
- optimization modes are `mode_1_decay`, `mode_2_sbb`,
`mode_3_flat_minima`, and `mode_4_is_only_robust`;
`mode_3_flat_minima`, `mode_4_is_only_robust`, and
`mode_5_full_robust`;
- for all optimization modes, Optuna receives only in-sample or synthetic
in-sample objectives; OOS scoring is delayed until after the top IS candidate
set is frozen, reducing indirect look-ahead bias;
Expand Down Expand Up @@ -1350,6 +1354,17 @@ Important rules:
shard Sharpe stability, combines that with the existing plateau cluster
score, and selects the medoid/centroid before any OOS scoring. OOS is only
used afterward for reporting and final stitched validation;
- `mode_5_full_robust` is full-sample robust calibration, not WFO/OOS
validation. QuantBT treats the whole supplied history as one calibration
fold, uses Optuna plus robust selection across the full sample, and labels
metadata with `validation_claim="none_full_sample_calibration"`. Use this
after a strategy already passed separate validation, when the goal is to
choose one production parameter set that survived all available regimes.
Supported selectors are:
`full_robust` (default temporal plus plateau),
`full_plateau_robust` (dense parameter plateau only),
`full_temporal_robust` (best subperiod stability among top trials), and
`full_best` (best full-sample objective, highest overfit risk);
- numba accelerates repeated scoring/bootstrap loops when installed; Python /
NumPy fallback remains available for debug and equivalence tests.

Expand Down Expand Up @@ -1501,6 +1516,37 @@ wfo = QuantBTEndpoint.walk_forward(
)
```

Full-sample robust calibration example:

```python
cal = QuantBTEndpoint.walk_forward(
strategy_class=strategy,
target_mode="pct_equity",
optimization_mode="mode_5_full_robust",
optimization_config={
"candidate_selection_metric": "full_robust",
"top_is_fraction": 0.10,
"is_subperiods": 8,
"flat_eps": 0.12,
"flat_min_samples": 5,
"temporal_weight": 0.65,
"plateau_weight": 0.35,
"scoring_backend": "endpoint",
"scoring_trading_days": 365,
"min_trades_per_year": 100,
"trade_penalty_factor": 0.5,
},
optuna_trials=600,
random_seed=42,
initial_capital=20_000,
leverage=5,
alloc_per_trade=0.5,
fee=0.0005,
)
result = cal.backtest(data=df, param_ranges=param_ranges)
production_params = result.metadata["walk_forward"]["best_trial"]["params"]
```

Optional trade-frequency penalty:

```text
Expand Down
19 changes: 16 additions & 3 deletions endpoint.py
Original file line number Diff line number Diff line change
Expand Up @@ -518,7 +518,8 @@ def walk_forward(
then routed into an existing QuantBT backtest path, so boundary trades
are charged by the normal engine instead of averaging fold equities.
Supported optimization modes are `mode_1_decay`, `mode_2_sbb`,
`mode_3_flat_minima`, and `mode_4_is_only_robust`.
`mode_3_flat_minima`, `mode_4_is_only_robust`, and
`mode_5_full_robust`.
Fixed-parameter runs can leave
`optimization_mode="none"` and pass `params=...` to `backtest()`.
"""
Expand Down Expand Up @@ -548,7 +549,15 @@ def walk_forward(
candidate_selection_metric=str(
optimization_config.get(
"candidate_selection_metric",
"is_only_robust" if str(optimization_mode).lower().strip() == "mode_4_is_only_robust" else "robust_decay",
(
"is_only_robust"
if str(optimization_mode).lower().strip() == "mode_4_is_only_robust"
else (
"full_robust"
if str(optimization_mode).lower().strip() == "mode_5_full_robust"
else "robust_decay"
)
),
)
),
candidate_decay_lambda=optimization_config.get("candidate_decay_lambda"),
Expand Down Expand Up @@ -626,7 +635,8 @@ def train_test_split(
then the stitched holdout signal is routed into the selected QuantBT
target mode. `optimization_mode` accepts the same values as
walk-forward: `none`, `mode_1_decay`, `mode_2_sbb`,
`mode_3_flat_minima`, and `mode_4_is_only_robust`.
`mode_3_flat_minima`, `mode_4_is_only_robust`, and
`mode_5_full_robust`.
"""
return cls.walk_forward(
strategy_class=strategy_class,
Expand Down Expand Up @@ -1283,6 +1293,9 @@ def _run_walk_forward(
"candidate_table": wf_result.candidate_table,
"best_trial": wf_result.best_trial,
"optimization_mode": wf_result.metadata.get("optimization_mode"),
"validation_claim": wf_result.metadata.get("validation_claim"),
"full_sample_used_for_selection": wf_result.metadata.get("full_sample_used_for_selection"),
"oos_used_for_selection": wf_result.metadata.get("oos_used_for_selection"),
"data_hash": wf_result.metadata.get("data_hash"),
"config_hash": wf_result.metadata.get("config_hash"),
"random_seed": wf_result.metadata.get("random_seed"),
Expand Down
120 changes: 120 additions & 0 deletions tests/test_walkforward_phase1.py
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@
WalkForwardTrialRecord,
benchmark_walkforward_kernels,
score_strategy_output,
select_full_sample_robust_record,
select_is_only_robust_record,
select_is_plateau_robust_record,
select_flat_minima_record,
Expand Down Expand Up @@ -998,6 +999,125 @@ def strategy(data, params, train_index, test_index, fold):
assert trial_table["is_subperiod_count"].max() >= 1


def test_mode5_full_robust_selector_marks_full_sample_calibration():
records = [
WalkForwardTrialRecord(
0,
{"x": 95},
12.0,
12.0,
0.0,
0.0,
0.0,
[],
selection_metadata={"temporal_score": 1.0, "temporal_q25": -0.5, "temporal_median": 3.0},
),
WalkForwardTrialRecord(
1,
{"x": 20},
9.2,
9.2,
0.0,
0.0,
0.0,
[],
selection_metadata={"temporal_score": 7.0, "temporal_q25": 6.0, "temporal_median": 7.0},
),
WalkForwardTrialRecord(
2,
{"x": 21},
9.1,
9.1,
0.0,
0.0,
0.0,
[],
selection_metadata={"temporal_score": 7.1, "temporal_q25": 6.1, "temporal_median": 7.1},
),
WalkForwardTrialRecord(
3,
{"x": 22},
9.0,
9.0,
0.0,
0.0,
0.0,
[],
selection_metadata={"temporal_score": 7.2, "temporal_q25": 6.2, "temporal_median": 7.2},
),
]
cfg = WalkForwardConfig(
optimization_mode="mode_5_full_robust",
candidate_selection_metric="full_robust",
top_is_fraction=1.0,
flat_eps=0.04,
flat_min_samples=2,
is_subperiods=4,
temporal_weight=0.70,
plateau_weight=0.30,
)

selected = select_full_sample_robust_record(records, {"x": (0, 100, 1)}, config=cfg)

assert selected.params["x"] in {20, 21, 22}
assert selected.selection_metadata["selected_by"] == "full_robust"
assert selected.selection_metadata["oos_used_for_selection"] is False
assert selected.selection_metadata["full_sample_used_for_selection"] is True
assert selected.selection_metadata["validation_claim"] == "none_full_sample_calibration"


def test_walkforward_mode5_full_robust_uses_whole_history_without_oos_claim():
idx = _idx()
data = _bars(idx)
trend = pd.Series(np.linspace(100.0, 130.0, len(idx)), index=idx)
data["close"] = trend

def strategy(data, params, train_index, test_index, fold):
assert train_index.equals(test_index)
assert train_index.equals(data.index)
side = 1.0 if int(params["go_long"]) == 1 else -1.0
signal = pd.Series(side, index=test_index)
signal.iloc[0] = 0.0
return signal

bt = QuantBTEndpoint.walk_forward(
strategy_class=strategy,
target_mode="signal_notional",
optimization_mode="mode_5_full_robust",
optimization_config={
"top_is_fraction": 1.0,
"flat_eps": 1.0,
"flat_min_samples": 1,
"is_subperiods": 3,
"candidate_selection_metric": "full_robust",
"scoring_backend": "proxy",
"use_numba": True,
},
optuna_trials=4,
random_seed=11,
initial_capital=20_000.0,
leverage=5.0,
alloc_per_trade=1_000.0,
fee_rate=0.0,
use_funding=False,
)
result = bt.backtest(data=data, symbols=["BTC"], param_ranges={"go_long": (0, 1, 1)})
wf = result.metadata["walk_forward"]
best_meta = wf["best_trial"]["selection_metadata"]

assert wf["optimization_mode"] == "mode_5_full_robust"
assert wf["candidate_selection_metric"] == "full_robust"
assert wf["validation_claim"] == "none_full_sample_calibration"
assert wf["full_sample_used_for_selection"] is True
assert wf["oos_used_for_selection"] is False
assert wf["fold_table"].loc[0, "train_start"] == idx[0]
assert wf["fold_table"].loc[0, "test_start"] == idx[0]
assert best_meta["selected_by"] == "full_robust"
assert best_meta["oos_seen_by_optuna"] is False
assert best_meta["full_sample_used_for_selection"] is True
assert "is_subperiod_count" in wf["trial_table"].columns


def test_train_test_split_can_select_is_plateau_robust_without_oos_selection():
idx = _idx()
data = _bars(idx)
Expand Down
Loading
Loading