feat: backtest-honesty module (PBO/CSCV, MinTRL/MinBTL, overfitting verdict)

[manan-tech]

A new overfitting.py — the flagship "honesty" differentiator surfaced by the research: tell the user when a backtest is likely a fluke. The mainstream Python stack (PyPortfolioOpt, quantstats, bt, vectorbt, pyfolio) ships none of this; this compounds quant_reporter's existing PSR / deflated-Sharpe / walk-forward lead.

What

Surface	Purpose
probability_of_backtest_overfitting(returns_matrix, n_splits=16, ...)	PBO via CSCV — how often the in-sample-best config lands at/below the OOS median. Returns PBOResult (pbo, logits, performance-degradation slope, prob-OOS-loss).
min_track_record_length(returns, prob=0.95, ...)	MinTRL — observations needed for the Sharpe to be significant.
min_backtest_length(n_trials, sr_target_annual=1.0, ...)	MinBTL — years before N trials fake a Sharpe by chance.
assess_overfitting(returns_matrix=…, returns=…, n_trials=…)	One-call verdict (robust / caution / likely_overfit / inconclusive) bundling PBO + deflated Sharpe + MinTRL → OverfittingReport.
overfitting_section(report)	Embeddable HTML section (verdict-colored) for the validation / combined reports.

Consume-only (you pass the trial matrix, e.g. from backtest_many over a param set), offline, no new dependencies; reuses performance_stats' moment / deflated-Sharpe machinery.

Honesty posture

Verdict thresholds are documented heuristics, overridable via thresholds=, never asserted as decision rules — consistent with the "not-alpha" lane (the research refuted OOS-outperformance claims, so messaging stays on diagnostics).

Tests (TDD, 22 unit tests)

Known-answer PBO: noise averages ~0.5 (the finite-N selection effect), a genuine edge → PBO < 0.3, a constructed overfit matrix → PBO > 0.7; MinTRL monotonic in confidence and nan when the target is unreachable; MinBTL ↑ with trials, ↓ with target Sharpe; verdict wiring for every tier; overfitting_section renders through html_builder. Full suite: 445 passed, coverage 87.3%; ruff check src/ clean.

Docs

CHANGELOG.md (Added) and README.md (a "Backtest honesty (2.3)" row + subsection with examples) updated.

Design spec: docs/superpowers/specs/2026-07-01-backtest-honesty-design.md (local; that dir is gitignored).

🤖 Generated with Claude Code

https://claude.ai/code/session_011a6EibcjDDv3Ff1Fb5esS1

Summary by Sourcery

Add a backtest-honesty module providing overfitting diagnostics and integrate it into the public API, documentation, and tests.

New Features:

• Introduce probability_of_backtest_overfitting (PBO via CSCV), min_track_record_length (MinTRL), min_backtest_length (MinBTL), and assess_overfitting with OverfittingReport for backtest overfitting assessment.
• Add an overfitting_section HTML helper for embedding overfitting diagnostics into reports.

Documentation:

• Document the new backtest honesty capabilities in README, including usage examples, and record the feature in CHANGELOG.

Tests:

• Add a dedicated test suite for the overfitting module covering PBO behavior under noise, genuine edge, and constructed overfit scenarios, MinTRL/MinBTL properties, one-call verdict wiring, and HTML rendering.

[sourcery-ai]

Reviewer's Guide

Introduces a new backtest-honesty / overfitting diagnostics module that computes PBO via CSCV, minimum track record/backtest lengths, a combined overfitting verdict, and an embeddable HTML report section, wires it into the public API, and documents the new capability.

Sequence diagram for assess_overfitting verdict computation

sequenceDiagram
    participant Client
    participant quant_reporter as quant_reporter
    participant overfitting as overfitting_module
    participant performance_stats

    Client->>quant_reporter: assess_overfitting(returns_matrix, returns, n_trials)
    quant_reporter->>overfitting: assess_overfitting(returns_matrix, returns, n_trials)
    alt returns_matrix provided
        overfitting->>overfitting: _as_matrix(returns_matrix)
        overfitting->>overfitting: probability_of_backtest_overfitting(df, n_splits)
        overfitting->>overfitting: _best_series(df, periods_per_year)
    end
    alt returns provided
        overfitting->>performance_stats: deflated_sharpe_ratio(returns, n_trials, periods_per_year)
        overfitting->>overfitting: min_track_record_length(returns, periods_per_year)
    end
    overfitting-->>Client: OverfittingReport

    Client->>quant_reporter: overfitting_section(report)
    quant_reporter->>overfitting: overfitting_section(report)
    overfitting-->>Client: section dict (HTML banner)

Loading

File-Level Changes

Change	Details	Files
Add backtest-honesty / overfitting diagnostics module implementing PBO via CSCV, MinTRL, MinBTL, and a combined overfitting verdict with HTML rendering.	Implement min_track_record_length using existing moment/PSR machinery to compute the minimum observations needed for Sharpe significance and return NaN when unreachable. Implement min_backtest_length using expected maximum Sharpe across trials to estimate minimum backtest years before multiple trials can fake a target Sharpe. Implement probability_of_backtest_overfitting using CSCV over a cleaned returns matrix with configurable splits, combinatorial sampling, Sharpe-based metric, and summary statistics in PBOResult. Implement assess_overfitting to combine PBO, deflated Sharpe, and MinTRL into an OverfittingReport with heuristic, overridable thresholds and structured evidence. Implement overfitting_section to convert OverfittingReport into a styled HTML section compatible with the existing html_builder pipeline.	src/quant_reporter/overfitting.py
Expose new overfitting diagnostics in the public package API.	Import and re-export probability_of_backtest_overfitting, PBOResult, min_track_record_length, min_backtest_length, assess_overfitting, OverfittingReport, and overfitting_section from the new module.	src/quant_reporter/__init__.py
Document the new backtest-honesty capability and update changelog.	Add a README row and subsection documenting the backtest-honesty layer, example usage of assess_overfitting and the underlying primitives, and guidance on heuristic thresholds. Update CHANGELOG with a detailed entry describing the overfitting diagnostics module, its functions, behavior, and references.	README.md CHANGELOG.md
Add unit tests for overfitting diagnostics covering statistical behavior, edge cases, and HTML rendering.	Introduce random-data helpers to generate noise, dominant, and adversarial overfit matrices for PBO behavior tests. Test monotonicity and boundary behavior of min_backtest_length and min_track_record_length. Test PBOResult contents, error handling, and input flexibility for probability_of_backtest_overfitting. Test assess_overfitting verdict logic, thresholds override behavior, serialization, and text output. Test overfitting_section structure and end-to-end rendering via generate_html_report.	test/test_overfitting.py

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it. You can also reply to a
  review comment with @sourcery-ai issue to create an issue from it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time. You can also comment
  @sourcery-ai title on the pull request to (re-)generate the title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time exactly where you
  want it. You can also comment @sourcery-ai summary on the pull request to
  (re-)generate the summary at any time.
• Generate reviewer's guide: Comment @sourcery-ai guide on the pull
  request to (re-)generate the reviewer's guide at any time.
• Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
  pull request to resolve all Sourcery comments. Useful if you've already
  addressed all the comments and don't want to see them anymore.
• Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
  request to dismiss all existing Sourcery reviews. Especially useful if you
  want to start fresh with a new review - don't forget to comment
  @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

[sourcery-ai]

Hey - I've found 1 issue, and left some high level feedback:

• In probability_of_backtest_overfitting, all_combos = list(combinations(...)) can become very large before being subsampled; consider short-circuiting based on an upper bound on n_splits or generating a random subset of combinations without materializing the full list to avoid potential memory blowups.
• _as_matrix currently drops any row with a NaN in any configuration (dropna(how="any")), which can throw away a lot of data for long backtests; consider allowing per-series NaNs (e.g., forward-filling or dropping per-column) or at least making this behavior explicit/optional so users don’t unexpectedly lose observations.

Prompt for AI Agents

Please address the comments from this code review:

## Overall Comments
- In `probability_of_backtest_overfitting`, `all_combos = list(combinations(...))` can become very large before being subsampled; consider short-circuiting based on an upper bound on `n_splits` or generating a random subset of combinations without materializing the full list to avoid potential memory blowups.
- `_as_matrix` currently drops any row with a NaN in any configuration (`dropna(how="any")`), which can throw away a lot of data for long backtests; consider allowing per-series NaNs (e.g., forward-filling or dropping per-column) or at least making this behavior explicit/optional so users don’t unexpectedly lose observations.

## Individual Comments

### Comment 1
<location path="src/quant_reporter/overfitting.py" line_range="60-69" />
<code_context>
+    return 1.0 + max(var_factor, 0.0) * (z / (sr - sr_star_pp)) ** 2
+
+
+def min_backtest_length(n_trials, sr_target_annual=1.0, periods_per_year=252):
+    """Minimum Backtest Length (in years) before ``n_trials`` fake ``sr_target_annual``.
+
+    The expected maximum Sharpe of ``n_trials`` skill-less i.i.d. strategies grows
+    like ``Z_N / sqrt(T)``; solving for the horizon at which that expected fluke
+    stays below the annualized target gives ``MinBTL = (Z_N / sr_target_annual)**2``
+    years, where ``Z_N`` is the expected maximum of ``n_trials`` standard normals.
+    One trial cannot be an overfit selection, so returns 0.0. Returns ``nan`` for a
+    non-positive target.
+    """
+    if sr_target_annual <= 0:
+        return float("nan")
+    if n_trials is None or n_trials <= 1:
+        return 0.0
+    z_n = _expected_max_sr(n_trials, 1.0)  # expected max of n_trials standard normals
+    return float((z_n / sr_target_annual) ** 2)
+
</code_context>
<issue_to_address>
**nitpick:** The periods_per_year parameter is unused in min_backtest_length and may be confusing.

The function always computes MinBTL in years using a standard-normal assumption and ignores the periods_per_year argument, so callers cannot actually change the time scale. Please either incorporate periods_per_year into the calculation or remove it from the signature to avoid a misleading API.
</issue_to_address>

---

Sourcery is free for open source - if you like our reviews please consider sharing them ✨

• X
• Mastodon
• LinkedIn
• Facebook

_{Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.}