Skip to content

Harden statistical frontier promotion - #17

Merged
howdymary merged 2 commits into
mainfrom
codex/packet-7-statistical-promotion
Jul 15, 2026
Merged

Harden statistical frontier promotion#17
howdymary merged 2 commits into
mainfrom
codex/packet-7-statistical-promotion

Conversation

@howdymary

Copy link
Copy Markdown
Owner

What changed

  • replaces last-fold promotion with aggregate paired out-of-fold Brier skill against the contemporaneous market baseline
  • clusters uncertainty by independent event and uses a versioned one-sided finite-sample Student-t lower bound
  • enforces a trusted frontier floor of at least 30 independent events, familywise alpha at most 0.05, and Bonferroni correction
  • derives hypothesis counts from complete per-fold attempted-artifact manifests rather than trusting archive claims
  • binds observed rows, expected identities, attempted/evaluated folds, provider versions, artifact digests, final genomes, and trajectory hashes
  • records every attempt and explicit rejection reasons; rejects duplicate/reweighted/swapped rows, duplicate attempts, legacy schema 1, and provenance mismatches
  • preserves a small default local ratchet for exploration while independently rechecking the stronger trusted promotion policy
  • carries the typed contemporaneous market probability through real backtest/ratchet output

Why / root cause

The prior frontier path could select on one final fold, treat correlated rows as independent, undercount repeated hypotheses, and accept archive-supplied provenance. Those behaviors could promote selection artifacts rather than reproducible forecast skill.

User and developer impact

Promotion now requires aggregate held-out evidence that beats the market baseline after uncertainty and multiple-testing correction. Insufficient or inconsistent evidence fails closed with auditable reasons. Fewer candidates will promote; that is intentional.

Schema and migration

  • introduces the versioned method paired-event-clustered-brier-student-t-v2
  • advances archive/frontier evidence contracts and explicitly rejects schema-1 artifacts with migration guidance
  • adds typed BinaryForecast.market_probability
  • existing legacy promotion artifacts are not silently reinterpreted under the new methodology

Verification

  • independent reviewer: APPROVED after provenance, statistical, and archive-tamper attacks
  • focused final review: 60 passed
  • full suite: 363 passed
  • Black, compileall, focused mypy, and git diff --check: passed
  • sdist/wheel build and installed-wheel smoke: passed
  • custom Student-t quantiles match SciPy references to about 1e-14 at operational probabilities/degrees of freedom
  • exact 28-positive/2-loss, 20-hypothesis z-test counterexample is rejected under Student-t v2
  • fake zero-row final-fold artifact is rejected and writes no frontier

Residual risks

  • repeated independent search runs are documented as exploratory; familywise error is controlled within each archived attempt, not across unregistered external attempts
  • canonical hashes detect mutation but do not authenticate wholly fabricated archives; signed provenance is a later hardening opportunity
  • the finite-sample model assumes independent event clusters and reports that assumption explicitly

Tracks #2
Closes #10

Depends on #14.

@howdymary
howdymary force-pushed the codex/packet-7-statistical-promotion branch from 2bbbb5e to 6b3708b Compare July 15, 2026 00:07
@howdymary
howdymary force-pushed the codex/packet-7-statistical-promotion branch from 6b3708b to 4b56061 Compare July 15, 2026 00:17
@howdymary
howdymary changed the base branch from codex/packet-3-canonical-evaluation to main July 15, 2026 00:17
@howdymary
howdymary marked this pull request as ready for review July 15, 2026 00:17
@howdymary
howdymary merged commit f91eff1 into main Jul 15, 2026
18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Packet 7: harden statistical selection and frontier promotion

1 participant