Skip to content

Packet 3: canonical dataset evaluation - #14

Merged
howdymary merged 2 commits into
mainfrom
codex/packet-3-canonical-evaluation
Jul 15, 2026
Merged

Packet 3: canonical dataset evaluation#14
howdymary merged 2 commits into
mainfrom
codex/packet-3-canonical-evaluation

Conversation

@howdymary

Copy link
Copy Markdown
Owner

Summary

  • define the strict autopredict.dataset.v1 JSON-manifest + canonical-JSONL contract
  • separate forecast-safe observations from resolution labels and fail closed on leakage/integrity errors
  • add deterministic validate, evaluate, and deprecated canonical backtest CLI paths
  • add provenance-rich baseline evaluation reports and exact dataset documentation

Safety and integrity

  • enforces UTF-8/LF canonical bytes, hashes, counts, UTC timestamps, capture bounds, ordered non-crossed books, stable market-to-event identity, and resolution joins
  • partial datasets validate structurally but cannot be used as performance evidence
  • live execution remains disabled; no venue, credential, or order path is touched

Verification

  • 330 tests pass
  • all tracked/untracked Python files compile
  • Black and scoped mypy pass
  • isolated wheel import/CLI audit passes
  • independent reviewer approved; independent tester passed adversarial and installed-package matrices

Parent: #2
Closes #6
Depends on: #13

This is a stacked draft PR based on codex/packet-2-release-integrity; retarget to main after its prerequisites merge.

@howdymary
howdymary changed the base branch from codex/packet-2-release-integrity to main July 15, 2026 00:14
@howdymary
howdymary marked this pull request as ready for review July 15, 2026 00:14
@howdymary
howdymary merged commit cd65978 into main Jul 15, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Packet 3: define the canonical dataset schema and evaluation CLI

1 participant