Declare the sota-v3 registry ready for zero-call route preflight - #108
Conversation
Everything registered about these ten routes comes from the public OpenRouter catalog, which the registry itself says does not prove authenticated route access or provider privacy behavior. The v2 lane already lost Nemotron 3 Ultra and DeepSeek V4 Pro to bounded 404s on routes that looked healthy publicly, so this is a live risk rather than a hypothetical one. Route preflight is the cheapest way to test that assumption: zero completion calls, no model subprocess, no reserved spend, no run state. It should happen before the seed panel is generated, because a dead route forces a cohort amendment and cohort size drives the Holm family size, which drives the allocation and the reservation -- the same cascade that just moved 15x1 to 16x1. selection_status therefore moves to route-preflight-ready. selection_frozen_at_utc stays null: this is strictly weaker than frozen and does not freeze cohort identity, which remains a separate later decision informed by preflight results. Measured against the live configs, the flip unlocks nothing that costs money. Route-preflight readiness goes from two blockers to one (the owner's separate route_preflight_authorized grant); smoke and panel stay at an identical 60 blockers each, still including "provider execution is locked until the model registry is frozen". test_route_preflight_readiness_unlocks_nothing_that_costs_money asserts that rather than leaving it as a claim in a commit message. The route-catalog guard previously pinned the registry-not-ready blocker that this change deliberately clears; it now pins the stronger property that the owner grant is the only remaining preflight blocker.
…onfig The previous record sat at 3be9432, which predates the selection_status flip. config/sota_v3_models.json is staged by the rehearsal, so that SHA no longer evidenced the configuration a preflight or paid run would read. Re-verified from a fresh clone at 02a0699: status passed, spend_usd 0.0, seven of seven mutations rejected, sota_v2 rejected / sota_v3 accepted, site data byte-matching the frozen v2 dataset, dependencies installed via bun install --frozen-lockfile, successful staged build, 742 tests passing in the clone. Also records the invalidation rule, so the next person does not have to re-derive when this goes stale or chase the SHA on every docs commit: staged inputs (config/, results/leaderboard/, results/analysis/, web/) and contract sources invalidate it; docs/ and tests/ do not.
|
Warning Review limit reached
Next review available in: 15 minutes You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (4)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Pull request overview
This PR advances the sota-v3 lane’s model registry from provisional-blocked to route-preflight-ready, enabling the zero-call route preflight readiness check while keeping all paid/execution authorizations locked (i.e., no spend or provider execution is enabled by this change).
Changes:
- Update
sota_v3_models.jsonto mark the registryselection_statusasroute-preflight-ready. - Adjust route-catalog and preregistration tests to reflect the new readiness state and to assert that only the owner’s
route_preflight_authorizedgate remains for preflight. - Document the updated clean-clone rehearsal SHA and add an explicit “when the record goes stale” invalidation rule plus a decision-log entry.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
tests/test_sota_v3_route_catalog.py |
Updates assertions for route-preflight-ready and strengthens the preflight blocker pin to ensure no other preflight unlock occurs. |
tests/test_sota_v3_preregistration.py |
Adds an explicit regression test ensuring route-preflight-ready does not change smoke/panel lock behavior versus provisional-blocked. |
docs/PUBLISH_READINESS.md |
Refreshes rehearsal provenance (new SHA/test count) and documents invalidation criteria + decision log entry for the readiness flip. |
config/sota_v3_models.json |
Declares the registry selection_status as route-preflight-ready while keeping selection_frozen_at_utc null. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Prepares the next gate in the v3 sequence. Grants no authorization —
route_preflight_authorizedand every other lane gate stayfalse, and this PR does not run preflight.Why this, and why now
Everything registered about the ten routes comes from the public OpenRouter catalog. The registry says so itself: it stayed
provisional-blockedbecause public metadata "does not prove authenticated exact-route access or provider privacy and retention behavior."That gap has bitten this project before. The v2 lane lost Nemotron 3 Ultra and DeepSeek V4 Pro to bounded HTTP 404s on routes that looked healthy publicly. So a failed authenticated probe is a live possibility.
Route preflight is the cheapest available test of that assumption — zero completion calls, no model subprocess, no reserved spend, no run state — and it should run before the seed panel is generated. A dead route forces a cohort amendment, and cohort size drives the Holm family size, which drives the allocation, which drives the reservation. That is the exact cascade that just moved 15×1 → 16×1 in #107. Discovering it now costs a JSON regeneration; discovering it after the seed panel is committed means a committed panel attached to a design that then changed.
The flip is strictly weaker than a freeze
selection_status→route-preflight-ready,selection_frozen_at_utcstaysnull. This does not freeze cohort identity; that remains a separate later decision informed by preflight results.Measured against the live configs:
route_preflight_authorizedgrantThe smoke/panel lists still include
provider execution is locked until the model registry is frozen.test_route_preflight_readiness_unlocks_nothing_that_costs_moneyasserts all of this, so it is checked rather than claimed.Note on the route-catalog guard
test_public_catalog_snapshot_cannot_unlock_any_provider_phasepinned the registry-not-ready blocker that this change deliberately clears. Rather than delete the line, it now pins the stronger property — that the owner grant is the only remaining preflight blocker — which fails just as loudly if anything else silently unlocks.Rehearsal re-anchored
config/sota_v3_models.jsonis staged by the rehearsal, so the prior record at3be9432no longer evidenced the config a preflight run would read. Re-verified from a fresh clone at02a0699: statuspassed,spend_usd0.0, 7/7 mutations rejected, site data byte-matching the frozen v2 dataset, dependencies installed viabun install --frozen-lockfile, successful staged build, 742 tests passing in the clone. The invalidation rule is now written down so the next person does not chase the SHA on every docs commit.Test plan
ruff check/ruff format --checkcleancontract_fingerprint()unchanged ata523bdfcebe47bbdfalse