Run the authenticated sota-v3 route preflight; fix the stale qwen endpoint tag - #109
Merged
Conversation
Owner granted route_preflight_authorized on 2026-08-03 and the zero-call probe ran against all ten registered routes at $0.00 spend. No completion call is possible in this phase: the runner validates the endpoint and continues before reaching subprocess.run. The "cannot create run state" property was also confirmed empirically -- the aborted first run left no new files behind. Nine of ten pins were correct on first contact. The tenth aborted the run: no healthy OpenRouter endpoint matches provider='Alibaba' tag='alibaba' name='Alibaba | qwen/qwen3.7-plus-20260602' Because the runner exits on first failure, the remaining three routes were checked directly against the authenticated endpoints API rather than assumed; all three match. The qwen defect is a stale label, not a lost model. The Alibaba endpoint is healthy at status 0 and 99.99% uptime, its provider_name and name match the registry exactly, and its prompt, completion, and 256k long-context override prices are identical to the recorded snapshot. Only the tag moved, from alibaba to alibaba/fp8. Corrected in the registry and in the pricing snapshot, whose provider_slug the route-catalog test binds to the endpoint tag. The cost estimator regenerates a byte-identical artifact, so the reservation holds at $89.845094 / $107.814113. The cohort therefore stays at ten and the 16x1 allocation is untouched. A genuinely dead route would have dropped the family to nine and re-triggered the Holm family size, the allocation, and the reservation -- which is the reason for running this free probe before generating the seed panel. Test updates reflect the new registered state without weakening the guards. The renamed catalog guard now asserts the distinction the lane is built on: the zero-call phase is open, and every phase that spends money is not. exact_route_acceptance is still unresolved and smoke is still blocked by 60 issues, the same count as before the probe.
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (1)
🚧 Files skipped from review as they are similar to previous changes (1)
📝 WalkthroughWalkthroughThe change authorizes authenticated, zero-call route preflight for ten SOTA v3 routes. It corrects the Qwen Alibaba route binding and updates readiness records and tests. Paid execution phases remain blocked. ChangesSOTA v3 route preflight
Estimated code review effort: 2 (Simple) | ~10 minutes Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Pull request overview
This PR records the completed authenticated sota-v3 route-preflight (zero-completion-call probe) and fixes a stale OpenRouter endpoint tag for the Qwen 3.7 Plus route by updating the registry and pricing snapshot to alibaba/fp8. It also updates tests and readiness docs to reflect that route-preflight can be authorized without unlocking any paid execution phases.
Changes:
- Update
config/sota_v3_lane.jsonto reflectroute_preflight_authorized: truewhile keeping all spend/execution/publication gatesfalse. - Fix the Qwen route’s
endpoint_tag/ provider slug fromalibaba→alibaba/fp8in the v3 registry and pricing snapshot. - Update tests and documentation to assert that completing route-preflight does not unlock smoke/panel phases, and add a run log describing the preflight results.
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| tests/test_sota_v3_route_catalog.py | Renames/updates the gate test to reflect route-preflight being authorized while paid phases remain blocked. |
| tests/test_sota_v3_preregistration.py | Updates preregistration assertions so route-preflight authorization is no longer treated as a fail-closed “paid” gate. |
| docs/run_logs/sota-v3-route-preflight-2026-08-03.md | Adds a detailed run log for the authenticated route-preflight and the Qwen tag mismatch diagnosis. |
| docs/PUBLISH_READINESS.md | Marks the route-preflight step as completed and records the Qwen tag correction rationale. |
| config/sota_v3_pricing_snapshot.json | Updates Qwen provider_slug to alibaba/fp8 to match the live endpoint tag. |
| config/sota_v3_models.json | Updates Qwen upstream_provider_slug and endpoint_tag to alibaba/fp8. |
| config/sota_v3_lane.json | Sets route_preflight_authorized to true while leaving all paid authorization gates false. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/run_logs/sota-v3-route-preflight-2026-08-03.md`:
- Line 27: Specify the language on the fenced diagnostic output block in the run
log by changing its opening fence to use text, resolving the MD040 Markdown lint
violation while leaving the block contents unchanged.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 8771c1a2-425f-46b8-bb33-6addb6ce7ac9
📒 Files selected for processing (7)
config/sota_v3_lane.jsonconfig/sota_v3_models.jsonconfig/sota_v3_pricing_snapshot.jsondocs/PUBLISH_READINESS.mddocs/run_logs/sota-v3-route-preflight-2026-08-03.mdtests/test_sota_v3_preregistration.pytests/test_sota_v3_route_catalog.py
A ticked box asserts a durable property. A passing route check is not durable: the first-party deepseek/deepseek-v4-flash-0731 route passed this very run and was deranked to status -5 the same evening, serving 78% of requests -- while its 24h availability figure still read 99.24%, so even the slow window did not show it. Record what the box actually means (the probe ran, its findings were acted on) and state the re-run requirement where someone reads it on the way to authorizing spend.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Owner granted
route_preflight_authorizedon 2026-08-03 and the zero-completion-call probe ran against all ten registered routes. $0.00 spend. No smoke or panel is authorized by this PR —spend_authorized,smoke_execution_authorized,panel_execution_authorized, andpublication_authorizedall remainfalse.Full detail:
docs/run_logs/sota-v3-route-preflight-2026-08-03.mdResult: 9/10 pins correct, 1 stale label, 0 dead routes
The first run aborted at cell 7 of 10:
The runner exits on first failure, so three routes were still untested. Rather than infer their status, all ten pins were checked directly against the authenticated endpoints API — the other nine match exactly, and all ten advertise
response_format.The qwen failure is a label, not a lost model
endpoint_tagalibabaalibaba/fp8provider_nameAlibabanameAlibaba | qwen/qwen3.7-plus-20260602Corrected in
config/sota_v3_models.jsonand inconfig/sota_v3_pricing_snapshot.json, whoseprovider_slugthe route-catalog test binds to the endpoint tag. Regenerating the cost estimator gives a byte-identical artifact — the reservation holds at $89.845094 / $107.814113.After the correction, all ten cells pass.
Why this matters more than the diff size suggests
A genuinely dead route would have dropped the Holm family to nine, re-triggering the family size → allocation → reservation cascade that moved 15×1 to 16×1 in #107. Because the only defect was a label, the cohort stays at ten and the 16×1 allocation is untouched. This is exactly the case for running the free probe before generating the seed panel: discovered later, the same correction would have landed against a committed panel.
It also means the public catalog snapshot was wrong about one route in ten. The authenticated check is now the stronger evidence for all ten.
The design claims, verified rather than assumed
route-preflightthe runner validates the endpoint thencontinues, never reachingsubprocess.run(command).results/publication*.exact_route_acceptanceis stillunresolved; smoke remains blocked by 60 issues, the same count as before the probe.Test changes
The four failures were all expected consequences of the grant and the tag fix. Updated to assert the new state without weakening the guards —
test_public_catalog_snapshot_cannot_unlock_any_provider_phaseis renamed totest_completed_route_preflight_cannot_unlock_any_paid_phaseand now pins the distinction the lane is built on: the zero-call phase is open, and every phase that spends money is not.Test plan
ruff check/ruff format --checkcleancontract_fingerprint()unchanged ata523bdfcebe47bbdSummary by CodeRabbit
New Features
Bug Fixes
Documentation
Tests