Refresh the sota-v3 lineup; gate on route availability, price drift, and the spend ceiling - #110
Merged
Merged
Conversation
Qwen 3.7 Plus -> Qwen 3.8 Max (released 2026-08-03), and the DeepSeek slot moves off the deranked first-party route onto Cloudflare FP8. Cohort size stays at ten, so the Holm family, the 16x1 allocation, and the power selection are all untouched. The DeepSeek substitution was chosen on 24h availability (99.75%, best of sixteen endpoints), not on spot health -- the first-party route read status 0 with the cohort's best throughput hours before it was deranked to -5. Same FP8 quantization, identical published rates, so the swap costs nothing. Reservation moves $107.81 -> $119.76. That is entirely the Max tier, which bills 6.25x/4.69x the Plus tier per token; a 55.1% discount that appeared on the GLM Novita route absorbs part of it. Call count is unchanged at 3,240. Both drifts were invisible until the snapshot was re-read, which is the case for checking prices rather than trusting them. Two runner changes, both mutation-tested: - The zero-call phase now probes every route and reports the complete set of failures. Exiting on the first one leaves every later route unchecked, which reads as "one route is broken" when it may be four -- that happened twice in two days. Paid phases still abort on the first bad route, since there the next cell costs money. - Endpoint eligibility now enforces availability floors on two windows. The 24h figure cannot see an outage in progress: the deranked DeepSeek route still read 99.24% over 24h while serving 78% of requests. The 30m figure is what moved. Floors are 90% (30m) and 95% (24h), both well below the noise band -- these readings drift half a point between polls, and a 99% 24h floor rejected two healthy cohort members while passing the route that had actually failed.
|
Warning Review limit reached
Next review available in: 26 minutes You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (6)
📝 WalkthroughWalkthroughThe SOTA-v3 lineup now uses updated Qwen, DeepSeek, and MiniMax routes. Pricing and cost artifacts reflect new list rates. Publication preflight enforces uptime, pricing, and spending ceilings. Tests cover these checks and isolate provider credentials. ChangesSOTA-v3 lineup and publication controls
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant Operator
participant PublicationRunner
participant OpenRouter
participant ExecutionCells
Operator->>PublicationRunner: set --max-spend-usd
PublicationRunner->>PublicationRunner: check committed operator ceiling
PublicationRunner->>OpenRouter: probe all configured routes
OpenRouter-->>PublicationRunner: return uptime, pricing, and eligibility results
PublicationRunner->>PublicationRunner: aggregate preflight failures
PublicationRunner->>ExecutionCells: launch cells when preflight succeeds
Possibly related PRs
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 1📝 Generate docstrings 💡
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
🟡 Not ready to approve
There are a few correctness/operational-paper-trail issues to address (notably stale provenance text in config/sota_v3_models.json and stdout error printing in route-preflight) before approval.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
This review doesn't count toward merge requirements. Sign up for the private preview to control whether Copilot approvals count.
Pull request overview
Updates the sota-v3 publication lineup and strengthens the publication runner’s safety/quality gates by (1) enforcing endpoint availability floors during endpoint eligibility checks and (2) ensuring the zero-call route-preflight probes all routes before failing, while paid phases still fail fast.
Changes:
- Refreshes sota-v3 pinned routes/pricing artifacts (Qwen slot to
qwen/qwen3.8-max, DeepSeek route to Cloudflare FP8) and updates cost estimates accordingly. - Adds endpoint eligibility gating on both 30m and 24h uptime floors in the runner.
- Extends runner tests to verify route-preflight checks every route (aggregating failures) and that paid phases still abort on first failure.
File summaries
| File | Description |
|---|---|
| tests/test_publication_runner.py | Adds coverage for aggregated route-preflight failures and uptime-floor enforcement behavior. |
| tests/test_publication_cost.py | Updates expected sota-v3 cost plan totals after repricing. |
| scripts/run_publication_matrix.py | Implements uptime-floor eligibility checks and aggregates route-preflight failures while keeping paid phases fail-fast. |
| results/analysis/sota-v3-pre-smoke-cost-estimate.json | Updates stored pre-smoke cost estimate numbers and timestamp for the refreshed lineup. |
| config/sota_v3_pricing_snapshot.json | Updates pinned pricing snapshot inputs (timestamp, provider slug/endpoint names, new Qwen tier). |
| config/sota_v3_models.json | Updates the pinned route lineup/metadata for sota-v3 (Qwen/DeepSeek substitutions and related catalog fields). |
Review details
Suppressed comments (1)
config/sota_v3_models.json:21
- The catalog snapshot timestamp was refreshed to 2026-08-04, but
selection_revision(and theselection_policynarrative immediately below) still refer to the 2026-08-03 cohort and don’t mention the Qwen/DeepSeek substitutions. For auditability, please update these fields to match the new lineup/snapshot provenance.
"selection_revision": "2026-08-03-public-catalog-cohort-v2",
"catalog_snapshot_status": "frozen-public-metadata-only",
"catalog_checked_at_utc": "2026-08-04T03:31:26Z",
"catalog_sources": [
"https://openrouter.ai/api/v1/models",
"https://openrouter.ai/api/v1/models/{model_id}/endpoints"
],
"selection_policy": "Ten-model pre-data cohort selected from the 2026-08-03 public OpenRouter catalog, superseding the eight-model 2026-07-28 cohort before any smoke or panel evidence exists. GPT-5.6 Luna replaces GPT-5.6 Luna Pro: the plain Luna route that was unhealthy at the prior snapshot has recovered, and the Pro variant carried an unresolved reasoning.mode inconsistency between its public description and the structured catalog. DeepSeek V4 Flash 0731 and Tencent Hy3 (both on first-party FP8 routes) are added as open-weight anchors. Thinking Machines Inkling Small was evaluated for the tenth slot and found ineligible at this snapshot: no healthy route advertises the response_format parameter the lane's frozen JSON-mode and require-parameters options demand. Exact public endpoint metadata and prices are pinned below, but the registry remains provisional-blocked because public metadata does not prove authenticated exact-route access or provider privacy and retention behavior.",
- Files reviewed: 6/6 changed files
- Comments generated: 2
- Review effort level: Lite
We're testing this review assessment. Please use 👍 or 👎 to tell us if it's correct.
| if args.phase != "route-preflight": | ||
| raise SystemExit(failure) from exc | ||
| preflight_failures.append(failure) | ||
| print(failure) |
| "OPENROUTER_REASONING_EFFORT" | ||
| ], | ||
| "role": "DeepSeek open-weight anchor on the first-party FP8 route" | ||
| "role": "DeepSeek open-weight anchor, substituted onto the Cloudflare FP8 route on 2026-08-04 after the first-party DeepSeek route was deranked to status -5 (30m uptime 77%). Same FP8 quantization and identical published rates; selected on 24h uptime (99.75%, best of 16 endpoints), not spot health." |
…account budget_policy.operator_ceiling_usd was declared in config and read by nothing. The only thing between a mistyped --max-spend-usd and an unbounded run was the operator retyping the right number from memory. It is now enforced ahead of the cell loop -- proven by asserting that a run over the cap reaches neither the endpoint probe nor a child process -- and set to the owner's $120 hard cap for sota-v3. A null ceiling stays permissive so contracts that have not chosen a number are not silently given one. Writing that test exposed something worse. The runner calls load_environment_files(ROOT) at startup, which reads the gitignored .env.local out of the working tree. Any test driving main() through a paid phase without stubbing the child process therefore runs the real benchmark against real routes and bills a real account -- and nothing fails, because the run succeeds. The test written to assert that a ceiling BLOCKS a run instead spent $0.436198 across 38 live calls to nine models. It resolved --contract sota-v3 to the shared fixture lane, which is fully unlocked by construction and inherits a null ceiling from the v2 protocol, so the guard correctly declined to fire and the smoke ran for real. An autouse fixture now neutralises the loader and clears provider keys for the whole suite, so no test can inherit a live credential. Tests that need a key present still set one explicitly with monkeypatch.setenv; what they can no longer do is pick one up by accident.
…ates The pricing snapshot is what the reservation was computed from, so nothing was comparing it to reality. Every probe now checks the pinned route's live base rates: an increase fails the route, a decrease is reported and allowed. The asymmetry is deliberate -- a rate that rose makes the budget wrong in the direction that costs money, and a rate that fell only means coming in under reserve. It found two things on its first live run. MiniMax's first-party route had been deranked to status -2. Substituted onto deepinfra/fp8 under the new written policy: same model, same FP8 quantization, highest 24h availability among eligible routes, identical rates. More seriously, openai/gpt-5.6-luna and z-ai/glm-5.2 were both pinned at 50%-off promotional rates. The GLM discount moved 55.1% -> 50% within hours of being recorded, which is what surfaced it. A promo is not a floor and a reservation computed from one is wrong the moment it ends, so the snapshot now pins undiscounted list rates throughout. That changes the picture: the reservation moves $119.76 -> $127.29, which EXCEEDS the committed $120.00 ceiling. The plan only ever appeared to fit because two routes were discounted. Projected actual spend remains ~$35-45 from July smoke telemetry, so the ceiling is a backstop rather than a forecast, but the committed number now needs an owner decision either way. Also adds docs/ROUTE_SUBSTITUTION_POLICY.md, which fixes eligibility, forbids price, throughput, and first-party status as substitution criteria, and requires route and privacy acceptance to be re-established for any new counterparty -- Cloudflare and DeepInfra have never been reviewed for this project. Throughput is called out specifically because the DeepSeek route had the cohort's best throughput hours before it failed. The 2026-08-03 run log is left unedited as the historical record; today's state is a new log.
The $120 ceiling was chosen against a $119.76 reservation that turned out to depend on 50%-off promotional rates on openai/gpt-5.6-luna and z-ai/glm-5.2. Pinning undiscounted list rates moved the reservation to $127.29 and put the committed plan over its own committed ceiling. Owner raised the cap to $150.00, which clears it with headroom for a further route substitution or list-price move without another ceiling decision. Nothing had detected the breach, because the reservation and the ceiling live in different files and nothing compared them. Two invariants now do, at zero cost, and both were verified to fail when broken: - the committed reservation must fit under the committed ceiling, so adding a model, substituting onto a pricier host, or losing a discount fails a test rather than surfacing when someone tries to authorize a run; - the committed cost artifact must match the configs it claims to describe, so the number the readiness docs and the ceiling decision quote cannot go stale against the plan that would actually run. No gate moves. spend_authorized, smoke_execution_authorized, panel_execution_authorized, and publication_authorized all remain false, the registry is still route-preflight-ready rather than frozen, and exact_route_acceptance is still unresolved. Projected actual spend remains ~$35-45, so the ceiling stays a backstop rather than a forecast.
nedcut
changed the title
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 5
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@config/sota_v3_models.json`:
- Line 304: Update the substitution-role descriptions at the entries around the
current line and the corresponding entry around line 452 to name the exact
outgoing routes: replace the generic first-party route references with
minimax/fp8 and deepseek/fp8 respectively, while preserving the remaining
history and rationale.
In `@scripts/run_publication_matrix.py`:
- Around line 497-503: Update the spend-limit validation around the CLI
max-spend value and the operator ceiling checks to reject non-finite numbers,
including NaN and infinities, before positivity or ceiling comparisons. Apply
this consistently to the CLI validation and _enforce_operator_ceiling,
preserving the existing positive-value and max-spend enforcement for finite
inputs.
- Around line 577-613: The _pricing_drift_issues check must fail closed when
pricing data or verification is incomplete. Update its snapshot loading and
endpoint lookup to treat read/parse failures and missing rates as blocking
issues, match endpoints using the full registered identity (provider_name, tag,
and name), and reject committed or live prompt/completion rates that are
malformed, non-finite, or negative before allowing the paid phase to proceed.
In `@tests/test_publication_cost.py`:
- Around line 95-99: Update the assertions in the publication cost test after
estimate to compare pricing_checked_at_utc and the complete committed["models"]
list against recomputed, rather than only comparing model-name sets. Preserve
the existing total-cost approximation assertion while ensuring provider, route,
rates, experiment_id, and ordering differences are detected.
In `@tests/test_publication_runner.py`:
- Around line 1366-1367: Update both main() invocations in
tests/test_publication_runner.py at lines 1366-1367 and 1401-1402 to include the
required --contract sota-v3 arguments for the route-preflight and smoke
commands, respectively, so the tests reach their intended mocked endpoint
validation behavior.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: a22f4fbe-b022-4533-aec9-2fae0611337c
📒 Files selected for processing (11)
config/sota_v3_models.jsonconfig/sota_v3_pricing_snapshot.jsonconfig/sota_v3_publication_protocol.jsondocs/PUBLISH_READINESS.mddocs/ROUTE_SUBSTITUTION_POLICY.mddocs/run_logs/sota-v3-lineup-refresh-2026-08-04.mdresults/analysis/sota-v3-pre-smoke-cost-estimate.jsonscripts/run_publication_matrix.pytests/conftest.pytests/test_publication_cost.pytests/test_publication_runner.py
Both found in review of the previous two commits; both verified before fixing. --max-spend-usd nan satisfied every guard while bounding nothing. NaN makes `nan <= 0`, `nan > ceiling`, and `spent >= nan` all false, so the positivity check, the operator ceiling, and the in-loop spend check each waved it through -- an unbounded paid run that looks fully authorized. Infinity is the same hole wherever no ceiling is configured. Non-finite limits and non-finite configured ceilings are now refused. _pricing_drift_issues failed open on every path that was not a clean comparison: unreadable snapshot, missing committed rate, missing or malformed live rate. "The price could not be verified" is not "the price is unchanged", and only one of those is safe to spend against, so each now blocks. It also matched endpoints on tag and name alone while the preflight pins provider_name, tag, and name; it now uses the full registered identity, so a same-tag endpoint from another provider can neither satisfy the check nor be compared against as though it were the pinned route. Also: preflight failures go to stderr rather than stdout, and both substitution roles now name the exact outgoing route and carry the correct observed 30m figure (78.93% for deepseek/fp8, 94.59% for minimax/fp8). All ten routes still pass the live probe. No gate moves.
…ss check Two route-preflight tests invoked main() without --contract, which is required of every operator invocation. They asserted real behaviour -- both still catch their own regression under mutation -- but they were not exercising the path an operator actually takes. Both now pass --contract sota-v3. The cost-artifact freshness check now also compares call counts, the pricing timestamp, and per-model cost-per-decision. The timestamp is what tells a reader which snapshot a number describes, so a stale one is its own defect even when the totals happen to agree.
29 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Four route events in twelve hours forced a lineup refresh. Turning the resulting judgement calls into enforced rules found three further problems, one of which billed a real account.
No gate moves.
spend_authorized,smoke_execution_authorized,panel_execution_authorized, andpublication_authorizedall remainfalse; the registry isroute-preflight-ready, notfrozen;exact_route_acceptanceis stillunresolved. Nothing here authorizes a paid run.Lineup
Cohort size stays at ten, so the Holm family size, the 16x1 allocation, and the power selection are untouched.
qwen/qwen3.7-plus@alibaba/fp8qwen/qwen3.8-max@alibabadeepseek/fp8cloudflare/fp8-5, 30m availability 78%minimax/fp8deepinfra/fp8-2The Qwen change is a cohort amendment; the other two follow the new written policy. DeepSeek recovered on its own within the day, so that substitution was not strictly necessary in hindsight — it is kept because Cloudflare holds the better 24h record, and the log says so.
What the new gates found
Each check was added, then immediately caught something real.
Route availability floors (90% over 30m, 95% over 24h). Two windows, because they detect different failures — the deranked DeepSeek route still read 99.24% over 24h while serving 78% of requests. A 99% 24h floor was measured rejecting two healthy cohort members while passing the route that had actually failed, so both floors sit well below the noise band; these readings drift half a point between consecutive polls.
Price drift on every probe. Found
openai/gpt-5.6-lunaandz-ai/glm-5.2both pinned at 50%-off promotional rates, surfaced when the GLM discount moved 55.1% → 50% within hours of being recorded. The snapshot now pins undiscounted list rates. An increase fails the route; a decrease is reported and allowed.The spend ceiling now binds.
budget_policy.operator_ceiling_usdhad sat in config and been read by nothing, so only the operator retyping the right number stood between a typo and an unbounded run.The probe checks every route. Exiting on the first failure had twice presented a partial picture as a complete one. Paid phases still abort on the first bad route, where the next cell costs money.
An incident
Writing the ceiling test exposed that the runner calls
load_environment_files(ROOT)at startup, reading the gitignored.env.localout of the working tree. Any test drivingmain()through a paid phase without stubbing the child process therefore runs the real benchmark against real routes and bills a real account — and nothing fails, because the run succeeds.The test written to assert that a ceiling blocks a run instead spent $0.436198 across 38 live calls to nine models. An autouse fixture now neutralises the loader and clears provider keys suite-wide; a test pins the guard.
Budget
Call count unchanged at 3,240. The $120 ceiling only ever fit because of the discounts; raising it clears the reservation with headroom for a further substitution.
Projected spend comes from July smoke telemetry reprojected at current rates: models emit 48–640 output tokens per decision against a 4,096-token reservation, and Grok's observed internal reasoning was 516 tokens against 4,096 reserved. The reserve is a backstop, not a forecast, and 42% of it is unused thinking headroom for Gemini and Grok.
Two new invariants stop the reservation and the ceiling drifting apart again, and stop the committed cost artifact going stale against the configs it describes.
Docs
docs/ROUTE_SUBSTITUTION_POLICY.md— new. Fixes eligibility; forbids price, throughput, and first-party status as substitution criteria; requires route and privacy acceptance to be re-established for any new counterparty. Throughput is called out because the DeepSeek route had the cohort's best throughput hours before it failed.docs/run_logs/sota-v3-lineup-refresh-2026-08-04.md— new. The 2026-08-03 log is left unedited as history.Follow-ups not in this PR
753 tests pass, ruff clean, contract fingerprint
a523bdfcebe47bbdunchanged.Summary by CodeRabbit
New Features
Updates
Documentation