Amend sota-v3 design: ten-model cohort, 16x1 allocation - #107
Merged
Conversation
Pre-data design amendment 2. Owner-directed cohort update before any smoke or panel evidence exists: - Swap GPT-5.6 Luna Pro for plain GPT-5.6 Luna on the first-party OpenAI route, removing the Pro reasoning.mode inconsistency and its unresolved-decision entry. - Add DeepSeek V4 Flash 0731 and Tencent Hy3 as open-weight anchors, both on healthy first-party FP8 routes. - Record Thinking Machines Inkling Small as evaluated-but-ineligible: no healthy route advertises response_format under the lane's frozen JSON-mode + require-parameters options. Kimi K3 passed the route requirements but was declined on cost. A family of ten tightens the Holm first step to 0.005; 15x1 fails the Wilson lower-bound rule at family ten (0.8020, lower 0.7941), so the allocation reselected with the identical frozen power machinery is 16 seeds x 1 repeat: base power 0.9527, sensitivity 0.8488 (lower bound 0.8416). Seed panel is now 16 pending authorized generation. Pre-smoke reservation regenerates to 3,240 calls, $107.81 at 1.2x. Contract fingerprint a523bdfcebe47bbd unchanged; every execution, spend, and publication authorization remains false.
|
Warning Review limit reached
Next review available in: 23 minutes You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (2)
📝 WalkthroughWalkthroughThe SOTA v3 preregistration changes to a ten-model cohort with 16 seeds and one repeat. Model routes, pricing, statistical estimates, cost projections, readiness documentation, and validation tests are updated. ChangesSOTA v3 cohort amendment
Estimated code review effort: 4 (Complex) | ~45 minutes Possibly related PRs
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 1📝 Generate docstrings 💡
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Pull request overview
This PR amends the pre-data sota-v3 preregistered design by expanding the frozen cohort from 8 to 10 models and updating the powered allocation from 15×1 to 16×1 (to maintain Holm-family feasibility and the Wilson lower-bound power rule under family size 10), while keeping the contract fingerprint unchanged.
Changes:
- Expand the frozen model registry/cohort to 10 models (swap in
openai/gpt-5.6-luna, adddeepseek/deepseek-v4-flash-0731andtencent/hy3) and update required smokes accordingly. - Reselect the panel allocation to 16 seeds × 1 repeat and propagate the new family size (Holm family = 10) through lane + protocol config and tests.
- Regenerate and pin the pricing snapshot + pre-smoke cost estimate artifacts; add a new run-log amendment record and update publish-readiness documentation.
Reviewed changes
Copilot reviewed 10 out of 10 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| tests/test_sota_v3_route_catalog.py | Updates invariants/expectations for the 10-model frozen cohort and revised anchor set. |
| tests/test_sota_v3_preregistration.py | Updates preregistration assertions for 16 seeds, 10-model family size, and registry/pricing lengths. |
| tests/test_publication_cost.py | Updates the expected call counts and total cost outputs for the 10-model, 16-seed plan. |
| results/analysis/sota-v3-pre-smoke-cost-estimate.json | Regenerated reservation artifact reflecting the new cohort size and allocation. |
| docs/run_logs/sota-v3-design-amendment-2026-08-03.md | New design amendment log documenting the cohort and allocation changes (pre-data). |
| docs/PUBLISH_READINESS.md | Updates published readiness narrative to reflect the ten-model cohort and 16×1 allocation and links the new amendment. |
| config/sota_v3_publication_protocol.json | Updates the frozen protocol’s family size and selected allocation/power figures. |
| config/sota_v3_pricing_snapshot.json | Updates pinned public-metadata pricing snapshot time and adds/updates model pricing entries. |
| config/sota_v3_models.json | Updates the frozen registry to the new 10-model cohort, required smokes, and resolved decisions. |
| config/sota_v3_lane.json | Updates lane amendment metadata and selected allocation (16 seeds) + minimum headline models (10). |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@config/sota_v3_lane.json`:
- Around line 71-94: Run scripts/panel_power.py using the documented simulation
seed and trial count, then compare its allocation, base_power_estimate,
base_power_wilson_ci95, sensitivity_power_estimate, and
sensitivity_power_wilson_ci95 outputs with the corresponding fields in the JSON
before finalizing the allocation or reservation.
In `@docs/run_logs/sota-v3-design-amendment-2026-08-03.md`:
- Around line 126-131: Update the reproduction pytest command to include
tests/test_publication_cost.py alongside the existing test files, ensuring the
documented cost recalculation assertions are executed.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: a755a9ff-5234-469a-b05d-1262d5fbad3a
📒 Files selected for processing (10)
config/sota_v3_lane.jsonconfig/sota_v3_models.jsonconfig/sota_v3_pricing_snapshot.jsonconfig/sota_v3_publication_protocol.jsondocs/PUBLISH_READINESS.mddocs/run_logs/sota-v3-design-amendment-2026-08-03.mdresults/analysis/sota-v3-pre-smoke-cost-estimate.jsontests/test_publication_cost.pytests/test_sota_v3_preregistration.pytests/test_sota_v3_route_catalog.py
Three pre-spend readiness gaps, none touching a _CONTRACT_SOURCES file: The scaffold-view checkbox in PUBLISH_READINESS.md was marked complete while its own body cited fingerprint 4f6ddddd6a6dd81c, and a later paragraph in the same document required a rerun under the current fingerprint. Rerunning is free and deterministic, so the compare was re-run under a523bdfcebe47bbd. Every headline mean, all eight per-seed scores, and the paired t reproduce exactly; neither scripted agent reads a field the intervening contract changes touched. The gate is now closed for this contract and the contradiction is resolved rather than papered over. The rehearsal's clean-checkout fix landed in #106 with three tests that all call _ensure_web_dependencies directly, so deleting the call site inside _run_web_build left the suite green while silently restoring the exit-127 regression on a fresh clone. Added two tests that drive _run_web_build itself and assert the install precedes the build. Verified by mutation: stubbing out the wiring line fails 2 of 11 tests, where it previously failed 0 of 9.
The recorded verification sat at 63f28e6, which predates the cohort-ten amendment. That amendment edits config/sota_v3_lane.json and config/sota_v3_models.json, both of which the rehearsal validates against, so the old SHA no longer evidenced the configuration a paid run would use. Re-verified from a fresh clone of the GitHub remote at 3be9432 with no web/node_modules: status passed, spend_usd 0.0, all seven mutations rejected, sota_v2 rejected / sota_v3 accepted, site data byte-matching the frozen v2 dataset, dependencies installed via bun install --frozen-lockfile (40 packages), successful staged build, and 741 tests passing in the same clone. Fingerprint remains a523bdfcebe47bbd. No gate is opened; all remain false. Also records the coverage gap the wiring tests closed, since it repeats the failure mode the surrounding entry already documents.
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
tests/test_sota_v3_rehearsal.py (1)
508-518: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick winAssert the build status on the reuse path.
The clean-checkout test checks
result["status"] == "passed"on Line 505. The existing-node_modulestest does not check the build status. Add the same assertion after the dependency assertion.Suggested assertion
assert result["dependencies"] == {"status": "reused", "source": "staged-directory"} + assert result["status"] == "passed"🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/test_sota_v3_rehearsal.py` around lines 508 - 518, Add an assertion in test_web_build_skips_the_install_when_dependencies_are_present verifying result["status"] equals "passed", alongside the existing dependencies assertion. Keep the current invocation and dependency-reuse assertions unchanged.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@tests/test_sota_v3_rehearsal.py`:
- Around line 476-485: The _record_bun_invocations helper currently records only
command arguments, so tests do not verify Bun’s working directory. Update
fake_run to record kwargs["cwd"] alongside each command, then assert staging /
"web" for both install and build invocations while preserving the existing argv
assertions.
---
Nitpick comments:
In `@tests/test_sota_v3_rehearsal.py`:
- Around line 508-518: Add an assertion in
test_web_build_skips_the_install_when_dependencies_are_present verifying
result["status"] equals "passed", alongside the existing dependencies assertion.
Keep the current invocation and dependency-reuse assertions unchanged.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 5296f692-5208-4972-be56-a18a843712d9
📒 Files selected for processing (3)
docs/PUBLISH_READINESS.mddocs/run_logs/scaffold-view-official-panel-2026-07-25.mdtests/test_sota_v3_rehearsal.py
Review finding on the wiring tests added in 3be9432: they recorded only the command line, so an install or build aimed at the repository's real web/ tree instead of the staging copy would run identical argv and pass. That defeats the staging isolation the rehearsal depends on, since a mutation could then touch the working tree it is supposed to leave alone. Confirmed against the previous test file: pointing either subprocess at ROOT/"web" left all 11 tests green. _record_bun_invocations now records (argv, cwd) pairs and both tests pin cwd to staging/"web". All three mutations are now caught -- wiring removed 2 failed, build cwd 2 failed, install cwd 1 failed.
This was referenced Aug 3, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Pre-data design amendment 2 (
docs/run_logs/sota-v3-design-amendment-2026-08-03.md), made whileevidence_stateis still pre-data so no observed v3 score could influence it.Cohort (8 → 10 models):
unresolved_decisionsentry is dropped as resolved-by-substitution)response_format, which the frozen lane requires viaOPENROUTER_JSON_MODE=true+OPENROUTER_REQUIRE_PARAMETERS=true. Kimi K3 passed route requirements but was declined on cost.Allocation (15x1 → 16x1): family of ten tightens Holm's first step to 0.005, and 15x1 fails the predeclared Wilson lower-bound rule at family ten (sensitivity 0.8020, lower bound 0.7941). Reselected with the identical frozen machinery (same variances, -100 planning effect, grid, 10k trials, seed 2026072800): 16 seeds x 1 repeat, base power 0.9527, sensitivity 0.8488 (lower bound 0.8416).
minimum_headline_modelsrises to 10 with the registered family.Cost: reservation regenerates to 3,240 calls, $89.85 unrounded / $107.81 at the committed 1.2x contingency (was $86.59 / $103.91).
Unchanged: contract fingerprint
a523bdfcebe47bbd(no_CONTRACT_SOURCESfile touched), directional claim, alpha/Holm/exact sign-flip machinery, output-cap policy, and every execution/spend/publication authorization remains false.Test plan
ruff format --check/ruff checkcleancontract_fingerprint()verified unchanged ata523bdfcebe47bbd--family-size 10, cost estimator regen)Summary by CodeRabbit
New Features
Documentation
Tests