diff --git a/config/sota_v3_models.json b/config/sota_v3_models.json index fa58409..004dd44 100644 --- a/config/sota_v3_models.json +++ b/config/sota_v3_models.json @@ -9,7 +9,7 @@ "session": false, "preset": "leaderboard", "repeats": 1, - "selection_status": "provisional-blocked", + "selection_status": "route-preflight-ready", "selection_frozen_at_utc": null, "selection_revision": "2026-08-03-public-catalog-cohort-v2", "catalog_snapshot_status": "frozen-public-metadata-only", diff --git a/docs/PUBLISH_READINESS.md b/docs/PUBLISH_READINESS.md index 47b3443..290880c 100644 --- a/docs/PUBLISH_READINESS.md +++ b/docs/PUBLISH_READINESS.md @@ -681,7 +681,7 @@ snapshot is in - [x] After these changes are committed, rerun the rehearsal from a clean checkout at the exact candidate SHA and record that SHA before any spend. **Verified unassisted 2026-08-03 at candidate SHA - `3be9432e10dd0f81c9e58f89bbaae1e0c5a7465f`.** A fresh clone from the GitHub + `02a069937d167810b261c21928c85cd3730f5461`.** A fresh clone from the GitHub remote with no `web/node_modules` runs `python3 scripts/sota_v3_rehearsal.py` to completion with no manual preparation: status `passed`, `spend_usd` 0.0, `evidence_class` `synthetic-non-evidence`, all seven mutations rejected, @@ -689,16 +689,23 @@ snapshot is in data byte-matching the checked-in frozen v2 dataset with the synthetic v3 row excluded, dependencies `installed` via `bun install --frozen-lockfile` (40 packages), and a successful staged build. The full suite passes in the - same clone (741 tests). No contract source was touched, so the fingerprint + same clone (742 tests). No contract source was touched, so the fingerprint remains `a523bdfcebe47bbd`, matching `config/sota_v3_lane.json`. This authorizes no spend; every lane gate remains false. - This re-verification supersedes the 2026-08-01 run at - `63f28e6897383fe73394c1e4354005dd404fb30b`, which recorded the same result - but predates the cohort-ten amendment. That amendment edits - `config/sota_v3_lane.json` and `config/sota_v3_models.json`, both of which - the rehearsal validates against, so the earlier SHA no longer evidences the - configuration a paid run would use. + **When this record goes stale.** The rehearsal stages `config/`, + `results/leaderboard/`, `results/analysis/`, and `web/`, so a commit + touching any of those — or any `_CONTRACT_SOURCES` file — invalidates this + verification and requires a rerun before spend. Commits confined to `docs/` + or `tests/` do not, since neither is staged nor fingerprinted. State the + rule rather than chasing the SHA: re-verify when a staged input changes, and + once more immediately before the first authorized spend. + + Earlier runs, superseded: `3be9432e10dd0f81c9e58f89bbaae1e0c5a7465f` + (2026-08-03, before the registry moved to `route-preflight-ready`) and + `63f28e6897383fe73394c1e4354005dd404fb30b` (2026-08-01, before the + cohort-ten amendment). Both recorded the same result, and both predate edits + to `config/sota_v3_models.json`, which the rehearsal validates against. The first attempt, at `a0fdec5493eaf5f702e71c519910e03f39e727f5`, **failed** — recorded here because the failure mode is the point. `_run_web_build` @@ -728,6 +735,15 @@ snapshot is in `python3 scripts/run_publication_matrix.py route-preflight --contract sota-v3`. This phase checks endpoint identity and parameters but cannot launch a model subprocess, reserve spend, or create run state. + **Route selection is done and the registry moved to `route-preflight-ready` + on 2026-08-03** (decision-log entry below), so the owner's + `route_preflight_authorized` grant is now the only remaining blocker on this + step. Granting it costs nothing and buys the answer to the question the + public catalog cannot answer: whether these ten exact routes are reachable + and acceptable under authentication. Run it before generating the seed + panel — a route failure here forces a cohort amendment, and a cohort size + change re-triggers the Holm family size, the allocation, and the + reservation. - [ ] After explicit owner authorization, use `scripts/seed_panel_commitment.py generate --lane config/sota_v3_lane.json --secret-file ` to draw the ordered 16-seed panel @@ -866,6 +882,7 @@ decision and why. | 2026-07-25 | Treat `sota-v3` as contract-ready and panel-blocked per consultant audit #93. | Independent review graded reproducibility A− but model discrimination D; one overlapping v2 tier; PR #92 contract economics still unmerged; scaffold-view unrun; site surfaces overclaim. | Merge #92, run scaffold-view at the frozen fingerprint, fix v2-site claim gaps, pre-register the v3 lane, and defer paid v3 panel spend until those gates pass. Do not publish ordinal model or baseline rankings. | | 2026-07-27 | Freeze v3 mechanics and reduce the pre-spend path to preregistration plus offline rehearsal. | PRs #92, #95, #98, #99, and #101 closed the mechanics, site-framing, same-view, version-dispatch, and claim-decomposition blockers. Continuing to add plausible realism changes now creates more schedule and evidence risk than it removes. The base SHA had no v3 lane/registry/manifest or v3 artifact; the working tree now has provisional fail-closed config files but still no selected model family or real/committed artifact. | Freeze score-affecting mechanics at `4f6ddddd6a6dd81c`; permit only one bounded, pre-data publication-parameter amendment if panel design requires it; preserve v2 literally; complete preregistration and a clean-checkout no-provider-call rehearsal; authorize no paid smoke or panel by this decision. | | 2026-07-30 | Reconcile the v3 pre-spend design across configs, policy, cost planning, seed commitment, and rehearsal. | The exact registered power procedure supports 15 independent seeds x 1 stochastic trajectory per model: base power 0.9461 and sensitivity power 0.8357 with Wilson lower bound 0.8283. One repeat is the registered estimand, not a dropped replicate. The prior configs disagreed on repeats and treated a live-readiness mismatch as non-fatal. No private seed was generated and no provider was called during reconciliation. | Bind the current lane to fingerprint `a523bdfcebe47bbd`, freeze the 15 x 1 statistical design, use a provisional 4,096/3,072/8,192 cap rule with whole-cohort invalidation and re-smoke on pressure, provide an unbiased private-seed generator, and make preregistration coherence a hard rehearsal gate. Keep every route, spend, execution, and publication authorization false. | +| 2026-08-03 | Move the ten-model registry from `provisional-blocked` to `route-preflight-ready` while `evidence_state` is still pre-data. | Everything registered about the ten routes comes from the public OpenRouter catalog, which by the registry's own admission "does not prove authenticated exact-route access or provider privacy and retention behavior." The v2 lane already lost Nemotron 3 Ultra and DeepSeek V4 Pro to bounded HTTP 404s on routes that looked healthy publicly, so a failed authenticated probe is a live possibility, not a hypothetical. Route preflight is the cheapest test of that assumption: it makes zero completion calls and cannot launch a model subprocess, reserve spend, or create run state. Discovering a dead route now costs a JSON regeneration; discovering it after the seed panel is committed means a committed panel attached to a design that then changed, because cohort size drives the Holm family size, which drives the allocation and the reservation. | Registry `selection_status` becomes `route-preflight-ready`; `selection_frozen_at_utc` stays `null`. This is strictly weaker than `frozen` and unlocks nothing that costs money: measured against the live configs, route-preflight readiness goes from two blockers to one — the owner's separate `route_preflight_authorized` grant — while the smoke and panel phases stay at an identical 60 blockers, still including "provider execution is locked until the model registry is frozen." Asserted by `test_route_preflight_readiness_unlocks_nothing_that_costs_money`. Cohort identity is **not** frozen by this decision; freezing it remains a separate later decision informed by preflight results. Every lane authorization remains false. | ## Experiment and release log diff --git a/tests/test_sota_v3_preregistration.py b/tests/test_sota_v3_preregistration.py index fe2cc6c..9b7aed3 100644 --- a/tests/test_sota_v3_preregistration.py +++ b/tests/test_sota_v3_preregistration.py @@ -12,6 +12,7 @@ from gm_bench.publication import ( SMOKE_MANIFEST_FORMAT, exact_sign_flip_feasibility, + publication_execution_issues, v3_preregistration_coherence_issues, ) @@ -92,7 +93,10 @@ def test_v3_registry_is_truthfully_provisional_and_contains_no_unverified_routes assert registry["contract"] == lane["contract"] assert registry["contract_fingerprint"] == lane["contract_fingerprint"] - assert registry["selection_status"] == "provisional-blocked" + # route-preflight-ready is strictly weaker than frozen: it clears the + # zero-call preflight readiness check and nothing else. The registry is + # still not frozen, so every paid phase stays locked. + assert registry["selection_status"] == "route-preflight-ready" assert registry["selection_frozen_at_utc"] is None assert registry["catalog_snapshot_status"] == "frozen-public-metadata-only" assert registry["catalog_checked_at_utc"] @@ -325,3 +329,31 @@ def test_runner_requires_explicit_contract_before_v2_preflight_can_run( assert exc_info.value.code == 2 assert "smoke requires an explicit --contract" in capsys.readouterr().err assert provider_access == [] + + +def test_route_preflight_readiness_unlocks_nothing_that_costs_money() -> None: + """`route-preflight-ready` must buy exactly one thing: the zero-call probe. + + This is the whole safety argument for making the flip before route + preflight has run, so it is asserted rather than described. The registry + is deliberately *not* `frozen`; every paid phase must remain as locked as + it was while the registry was `provisional-blocked`. + """ + lane = _read("sota_v3_lane.json") + registry = _read("sota_v3_models.json") + protocol = _read("sota_v3_publication_protocol.json") + pricing = _read("sota_v3_pricing_snapshot.json") + manifest = _read("sota_v3_smoke_manifest.json") + + def issues(reg: dict, phase: str) -> list[str]: + return publication_execution_issues(lane, reg, manifest, phase=phase, protocol=protocol, pricing=pricing) + + # The owner's separate zero-call authorization is the only thing left. + assert issues(registry, "route-preflight") == [ + "zero-call route preflight is locked while route_preflight_authorized is false" + ] + + blocked = dict(registry, selection_status="provisional-blocked") + for phase in ("smoke", "panel"): + assert issues(registry, phase) == issues(blocked, phase) + assert "provider execution is locked until the model registry is frozen" in issues(registry, phase) diff --git a/tests/test_sota_v3_route_catalog.py b/tests/test_sota_v3_route_catalog.py index 06d5b8c..550564e 100644 --- a/tests/test_sota_v3_route_catalog.py +++ b/tests/test_sota_v3_route_catalog.py @@ -42,7 +42,7 @@ def test_v3_catalog_freezes_exact_balanced_cohort_without_unlocking_execution() assert "meta/muse-spark-1.1" not in identities assert registry["catalog_snapshot_status"] == "frozen-public-metadata-only" - assert registry["selection_status"] == "provisional-blocked" + assert registry["selection_status"] == "route-preflight-ready" assert registry["selection_frozen_at_utc"] is None assert registry["catalog_checked_at_utc"] assert set(registry["required_smokes"]) == {model["id"] for model in models} @@ -170,8 +170,11 @@ def test_public_catalog_snapshot_cannot_unlock_any_provider_phase() -> None: protocol=protocol, pricing=pricing, ) - assert "zero-call route preflight is locked while route_preflight_authorized is false" in preflight_issues - assert "model registry is not ready for zero-call route preflight" in preflight_issues + # The registry-readiness blocker was deliberately cleared when the cohort + # moved to route-preflight-ready, so pin the stronger property instead: the + # owner's separate zero-call authorization must be the *only* thing left, + # which fails just as loudly if anything else silently unlocks. + assert preflight_issues == ["zero-call route preflight is locked while route_preflight_authorized is false"] smoke_issues = publication_execution_issues( lane,