From 02a069937d167810b261c21928c85cd3730f5461 Mon Sep 17 00:00:00 2001 From: Ned Cutler Date: Mon, 3 Aug 2026 16:20:29 -0400 Subject: [PATCH 1/2] Declare the ten-model registry ready for zero-call route preflight Everything registered about these ten routes comes from the public OpenRouter catalog, which the registry itself says does not prove authenticated route access or provider privacy behavior. The v2 lane already lost Nemotron 3 Ultra and DeepSeek V4 Pro to bounded 404s on routes that looked healthy publicly, so this is a live risk rather than a hypothetical one. Route preflight is the cheapest way to test that assumption: zero completion calls, no model subprocess, no reserved spend, no run state. It should happen before the seed panel is generated, because a dead route forces a cohort amendment and cohort size drives the Holm family size, which drives the allocation and the reservation -- the same cascade that just moved 15x1 to 16x1. selection_status therefore moves to route-preflight-ready. selection_frozen_at_utc stays null: this is strictly weaker than frozen and does not freeze cohort identity, which remains a separate later decision informed by preflight results. Measured against the live configs, the flip unlocks nothing that costs money. Route-preflight readiness goes from two blockers to one (the owner's separate route_preflight_authorized grant); smoke and panel stay at an identical 60 blockers each, still including "provider execution is locked until the model registry is frozen". test_route_preflight_readiness_unlocks_nothing_that_costs_money asserts that rather than leaving it as a claim in a commit message. The route-catalog guard previously pinned the registry-not-ready blocker that this change deliberately clears; it now pins the stronger property that the owner grant is the only remaining preflight blocker. --- config/sota_v3_models.json | 2 +- docs/PUBLISH_READINESS.md | 10 ++++++++ tests/test_sota_v3_preregistration.py | 34 ++++++++++++++++++++++++++- tests/test_sota_v3_route_catalog.py | 9 ++++--- 4 files changed, 50 insertions(+), 5 deletions(-) diff --git a/config/sota_v3_models.json b/config/sota_v3_models.json index fa58409..004dd44 100644 --- a/config/sota_v3_models.json +++ b/config/sota_v3_models.json @@ -9,7 +9,7 @@ "session": false, "preset": "leaderboard", "repeats": 1, - "selection_status": "provisional-blocked", + "selection_status": "route-preflight-ready", "selection_frozen_at_utc": null, "selection_revision": "2026-08-03-public-catalog-cohort-v2", "catalog_snapshot_status": "frozen-public-metadata-only", diff --git a/docs/PUBLISH_READINESS.md b/docs/PUBLISH_READINESS.md index 47b3443..2503564 100644 --- a/docs/PUBLISH_READINESS.md +++ b/docs/PUBLISH_READINESS.md @@ -728,6 +728,15 @@ snapshot is in `python3 scripts/run_publication_matrix.py route-preflight --contract sota-v3`. This phase checks endpoint identity and parameters but cannot launch a model subprocess, reserve spend, or create run state. + **Route selection is done and the registry moved to `route-preflight-ready` + on 2026-08-03** (decision-log entry below), so the owner's + `route_preflight_authorized` grant is now the only remaining blocker on this + step. Granting it costs nothing and buys the answer to the question the + public catalog cannot answer: whether these ten exact routes are reachable + and acceptable under authentication. Run it before generating the seed + panel — a route failure here forces a cohort amendment, and a cohort size + change re-triggers the Holm family size, the allocation, and the + reservation. - [ ] After explicit owner authorization, use `scripts/seed_panel_commitment.py generate --lane config/sota_v3_lane.json --secret-file ` to draw the ordered 16-seed panel @@ -866,6 +875,7 @@ decision and why. | 2026-07-25 | Treat `sota-v3` as contract-ready and panel-blocked per consultant audit #93. | Independent review graded reproducibility A− but model discrimination D; one overlapping v2 tier; PR #92 contract economics still unmerged; scaffold-view unrun; site surfaces overclaim. | Merge #92, run scaffold-view at the frozen fingerprint, fix v2-site claim gaps, pre-register the v3 lane, and defer paid v3 panel spend until those gates pass. Do not publish ordinal model or baseline rankings. | | 2026-07-27 | Freeze v3 mechanics and reduce the pre-spend path to preregistration plus offline rehearsal. | PRs #92, #95, #98, #99, and #101 closed the mechanics, site-framing, same-view, version-dispatch, and claim-decomposition blockers. Continuing to add plausible realism changes now creates more schedule and evidence risk than it removes. The base SHA had no v3 lane/registry/manifest or v3 artifact; the working tree now has provisional fail-closed config files but still no selected model family or real/committed artifact. | Freeze score-affecting mechanics at `4f6ddddd6a6dd81c`; permit only one bounded, pre-data publication-parameter amendment if panel design requires it; preserve v2 literally; complete preregistration and a clean-checkout no-provider-call rehearsal; authorize no paid smoke or panel by this decision. | | 2026-07-30 | Reconcile the v3 pre-spend design across configs, policy, cost planning, seed commitment, and rehearsal. | The exact registered power procedure supports 15 independent seeds x 1 stochastic trajectory per model: base power 0.9461 and sensitivity power 0.8357 with Wilson lower bound 0.8283. One repeat is the registered estimand, not a dropped replicate. The prior configs disagreed on repeats and treated a live-readiness mismatch as non-fatal. No private seed was generated and no provider was called during reconciliation. | Bind the current lane to fingerprint `a523bdfcebe47bbd`, freeze the 15 x 1 statistical design, use a provisional 4,096/3,072/8,192 cap rule with whole-cohort invalidation and re-smoke on pressure, provide an unbiased private-seed generator, and make preregistration coherence a hard rehearsal gate. Keep every route, spend, execution, and publication authorization false. | +| 2026-08-03 | Move the ten-model registry from `provisional-blocked` to `route-preflight-ready` while `evidence_state` is still pre-data. | Everything registered about the ten routes comes from the public OpenRouter catalog, which by the registry's own admission "does not prove authenticated exact-route access or provider privacy and retention behavior." The v2 lane already lost Nemotron 3 Ultra and DeepSeek V4 Pro to bounded HTTP 404s on routes that looked healthy publicly, so a failed authenticated probe is a live possibility, not a hypothetical. Route preflight is the cheapest test of that assumption: it makes zero completion calls and cannot launch a model subprocess, reserve spend, or create run state. Discovering a dead route now costs a JSON regeneration; discovering it after the seed panel is committed means a committed panel attached to a design that then changed, because cohort size drives the Holm family size, which drives the allocation and the reservation. | Registry `selection_status` becomes `route-preflight-ready`; `selection_frozen_at_utc` stays `null`. This is strictly weaker than `frozen` and unlocks nothing that costs money: measured against the live configs, route-preflight readiness goes from two blockers to one — the owner's separate `route_preflight_authorized` grant — while the smoke and panel phases stay at an identical 60 blockers, still including "provider execution is locked until the model registry is frozen." Asserted by `test_route_preflight_readiness_unlocks_nothing_that_costs_money`. Cohort identity is **not** frozen by this decision; freezing it remains a separate later decision informed by preflight results. Every lane authorization remains false. | ## Experiment and release log diff --git a/tests/test_sota_v3_preregistration.py b/tests/test_sota_v3_preregistration.py index fe2cc6c..9b7aed3 100644 --- a/tests/test_sota_v3_preregistration.py +++ b/tests/test_sota_v3_preregistration.py @@ -12,6 +12,7 @@ from gm_bench.publication import ( SMOKE_MANIFEST_FORMAT, exact_sign_flip_feasibility, + publication_execution_issues, v3_preregistration_coherence_issues, ) @@ -92,7 +93,10 @@ def test_v3_registry_is_truthfully_provisional_and_contains_no_unverified_routes assert registry["contract"] == lane["contract"] assert registry["contract_fingerprint"] == lane["contract_fingerprint"] - assert registry["selection_status"] == "provisional-blocked" + # route-preflight-ready is strictly weaker than frozen: it clears the + # zero-call preflight readiness check and nothing else. The registry is + # still not frozen, so every paid phase stays locked. + assert registry["selection_status"] == "route-preflight-ready" assert registry["selection_frozen_at_utc"] is None assert registry["catalog_snapshot_status"] == "frozen-public-metadata-only" assert registry["catalog_checked_at_utc"] @@ -325,3 +329,31 @@ def test_runner_requires_explicit_contract_before_v2_preflight_can_run( assert exc_info.value.code == 2 assert "smoke requires an explicit --contract" in capsys.readouterr().err assert provider_access == [] + + +def test_route_preflight_readiness_unlocks_nothing_that_costs_money() -> None: + """`route-preflight-ready` must buy exactly one thing: the zero-call probe. + + This is the whole safety argument for making the flip before route + preflight has run, so it is asserted rather than described. The registry + is deliberately *not* `frozen`; every paid phase must remain as locked as + it was while the registry was `provisional-blocked`. + """ + lane = _read("sota_v3_lane.json") + registry = _read("sota_v3_models.json") + protocol = _read("sota_v3_publication_protocol.json") + pricing = _read("sota_v3_pricing_snapshot.json") + manifest = _read("sota_v3_smoke_manifest.json") + + def issues(reg: dict, phase: str) -> list[str]: + return publication_execution_issues(lane, reg, manifest, phase=phase, protocol=protocol, pricing=pricing) + + # The owner's separate zero-call authorization is the only thing left. + assert issues(registry, "route-preflight") == [ + "zero-call route preflight is locked while route_preflight_authorized is false" + ] + + blocked = dict(registry, selection_status="provisional-blocked") + for phase in ("smoke", "panel"): + assert issues(registry, phase) == issues(blocked, phase) + assert "provider execution is locked until the model registry is frozen" in issues(registry, phase) diff --git a/tests/test_sota_v3_route_catalog.py b/tests/test_sota_v3_route_catalog.py index 06d5b8c..550564e 100644 --- a/tests/test_sota_v3_route_catalog.py +++ b/tests/test_sota_v3_route_catalog.py @@ -42,7 +42,7 @@ def test_v3_catalog_freezes_exact_balanced_cohort_without_unlocking_execution() assert "meta/muse-spark-1.1" not in identities assert registry["catalog_snapshot_status"] == "frozen-public-metadata-only" - assert registry["selection_status"] == "provisional-blocked" + assert registry["selection_status"] == "route-preflight-ready" assert registry["selection_frozen_at_utc"] is None assert registry["catalog_checked_at_utc"] assert set(registry["required_smokes"]) == {model["id"] for model in models} @@ -170,8 +170,11 @@ def test_public_catalog_snapshot_cannot_unlock_any_provider_phase() -> None: protocol=protocol, pricing=pricing, ) - assert "zero-call route preflight is locked while route_preflight_authorized is false" in preflight_issues - assert "model registry is not ready for zero-call route preflight" in preflight_issues + # The registry-readiness blocker was deliberately cleared when the cohort + # moved to route-preflight-ready, so pin the stronger property instead: the + # owner's separate zero-call authorization must be the *only* thing left, + # which fails just as loudly if anything else silently unlocks. + assert preflight_issues == ["zero-call route preflight is locked while route_preflight_authorized is false"] smoke_issues = publication_execution_issues( lane, From 0a2761db808eecbaa0e47fd046c3d2b26a5fe7f5 Mon Sep 17 00:00:00 2001 From: Ned Cutler Date: Mon, 3 Aug 2026 16:22:05 -0400 Subject: [PATCH 2/2] Re-anchor the clean-checkout rehearsal to the route-preflight-ready config The previous record sat at 3be9432, which predates the selection_status flip. config/sota_v3_models.json is staged by the rehearsal, so that SHA no longer evidenced the configuration a preflight or paid run would read. Re-verified from a fresh clone at 02a0699: status passed, spend_usd 0.0, seven of seven mutations rejected, sota_v2 rejected / sota_v3 accepted, site data byte-matching the frozen v2 dataset, dependencies installed via bun install --frozen-lockfile, successful staged build, 742 tests passing in the clone. Also records the invalidation rule, so the next person does not have to re-derive when this goes stale or chase the SHA on every docs commit: staged inputs (config/, results/leaderboard/, results/analysis/, web/) and contract sources invalidate it; docs/ and tests/ do not. --- docs/PUBLISH_READINESS.md | 23 +++++++++++++++-------- 1 file changed, 15 insertions(+), 8 deletions(-) diff --git a/docs/PUBLISH_READINESS.md b/docs/PUBLISH_READINESS.md index 2503564..290880c 100644 --- a/docs/PUBLISH_READINESS.md +++ b/docs/PUBLISH_READINESS.md @@ -681,7 +681,7 @@ snapshot is in - [x] After these changes are committed, rerun the rehearsal from a clean checkout at the exact candidate SHA and record that SHA before any spend. **Verified unassisted 2026-08-03 at candidate SHA - `3be9432e10dd0f81c9e58f89bbaae1e0c5a7465f`.** A fresh clone from the GitHub + `02a069937d167810b261c21928c85cd3730f5461`.** A fresh clone from the GitHub remote with no `web/node_modules` runs `python3 scripts/sota_v3_rehearsal.py` to completion with no manual preparation: status `passed`, `spend_usd` 0.0, `evidence_class` `synthetic-non-evidence`, all seven mutations rejected, @@ -689,16 +689,23 @@ snapshot is in data byte-matching the checked-in frozen v2 dataset with the synthetic v3 row excluded, dependencies `installed` via `bun install --frozen-lockfile` (40 packages), and a successful staged build. The full suite passes in the - same clone (741 tests). No contract source was touched, so the fingerprint + same clone (742 tests). No contract source was touched, so the fingerprint remains `a523bdfcebe47bbd`, matching `config/sota_v3_lane.json`. This authorizes no spend; every lane gate remains false. - This re-verification supersedes the 2026-08-01 run at - `63f28e6897383fe73394c1e4354005dd404fb30b`, which recorded the same result - but predates the cohort-ten amendment. That amendment edits - `config/sota_v3_lane.json` and `config/sota_v3_models.json`, both of which - the rehearsal validates against, so the earlier SHA no longer evidences the - configuration a paid run would use. + **When this record goes stale.** The rehearsal stages `config/`, + `results/leaderboard/`, `results/analysis/`, and `web/`, so a commit + touching any of those — or any `_CONTRACT_SOURCES` file — invalidates this + verification and requires a rerun before spend. Commits confined to `docs/` + or `tests/` do not, since neither is staged nor fingerprinted. State the + rule rather than chasing the SHA: re-verify when a staged input changes, and + once more immediately before the first authorized spend. + + Earlier runs, superseded: `3be9432e10dd0f81c9e58f89bbaae1e0c5a7465f` + (2026-08-03, before the registry moved to `route-preflight-ready`) and + `63f28e6897383fe73394c1e4354005dd404fb30b` (2026-08-01, before the + cohort-ten amendment). Both recorded the same result, and both predate edits + to `config/sota_v3_models.json`, which the rehearsal validates against. The first attempt, at `a0fdec5493eaf5f702e71c519910e03f39e727f5`, **failed** — recorded here because the failure mode is the point. `_run_web_build`