Cut sota-v3 to eight reasoning-disabled models - #114
Open
nedcut wants to merge 1 commit into
Open
Conversation
Pre-data design amendment 3. No completion endpoint was called, no private seeds were regenerated, and panel and publication authorization both remain false. Contract fingerprint a523bdfcebe47bbd is unchanged: this touches only config, docs, results artifacts, and tests, none of which are fingerprint sources. Gemini 3.6 Flash and Grok 4.5 are withdrawn. They were the only two registered routes whose catalog marks reasoning mandatory, so they reintroduced the same cross-model reasoning inconsistency that already excluded both from the frozen sota-v2 panel. Turning reasoning off was never available: the other eight already run with it disabled, and these two are mandatory-minimum at the lowest effort OpenRouter offers. One cut, four results: - Reserved worst case falls $127.29 -> $73.40 with the 1.2x contingency, because the withdrawn pair also billed internal reasoning at the completion rate on top of output. Projected actual spend is ~$25-30. The operator ceiling drops $150 -> $100. - The cohort becomes uniformly reasoning-disabled, removing a confound from the primary contrast and making v2 and v3 more comparable. - The protocol's pending reasoning-policy verification is resolved by construction; there is no mandatory-reasoning route left to verify. - The only routes whose sota-v2 telemetry approached the 3,072-token cap-pressure threshold are gone. Grok recorded 937 output tokens per call against 2,064 reasoning tokens; every reasoning-disabled model peaked at or below 604 against the same 4,096-token cap. Holm family size goes 10 -> 8, which loosens the first step from 0.005 to 0.00625. Power was re-derived with the frozen machinery, changing only --family-size. The rule's smallest qualifying allocation at family eight is 15x1 (sensitivity 0.8357), exactly reproducing the design from when the family last had eight members. The panel keeps 16 seeds anyway: seed_panel is already frozen pre-data with a sha256, a hiding commitment, and its secret in the Keychain, and breaking a pre-data cryptographic commitment to save $4.53 is the worse trade. 16x1 at family eight gives sensitivity power 0.8727, Wilson lower 0.8660 -- strictly more powerful than the rule's minimum. The deviation is recorded rather than silent. The cap-pressure rule also gains the terminal case it never had. It said to amend the cap "once" but never defined a second trigger, so the only written instruction, read literally, was to amend again -- and config/*.json is outside _CONTRACT_SOURCES, so a second amendment leaves no fingerprint trace. It now pins max_cap_amendments: 1 and on_second_trigger: abort-sota-v3-and-repreregister. The ceiling is deliberately set below the 8,192-token fallback branch, which reserves $116.26. Previously the $150 ceiling was sized for the primary branch alone while the fallback reserved $219.83, so _enforce_operator_ceiling would have blocked the very remedy the rule prescribed -- the same way the plan crossed its own ceiling at $120. A trigger should halt the run for an explicit owner spend decision, not silently widen the budget it is measured against, and a test now asserts the fallback stays above the ceiling. Side effects worth naming: the cohort is no longer balanced, moving to 2 frontier-proprietary / 6 open-weight, so public presentation must describe it as predominantly open-weight; and ZDR routes fall 5 -> 4 because Grok held the xai/zdr pin. Both are pinned by tests so neither can drift further without a deliberate decision. Route acceptance evidence was regenerated for the eight remaining routes with collect_sota_v3_route_evidence.py --apply-registry, still with zero completion calls. test_v3_reasoning_enabled_route_requires_an_explicit_token_assumption now builds a synthetic reasoning-enabled route. It selected the real Grok row, so withdrawing Grok would have left the filter matching nothing and the guard passing vacuously -- a test that silently stops testing. The unpriced-reasoning trap is a property of the estimator, not of whoever is in the cohort this month.
📝 WalkthroughWalkthroughThe SOTA v3 amendment reduces the registered cohort from ten to eight by withdrawing Gemini 3.6 Flash and Grok 4.5. It updates statistical, cap, budget, route-evidence, readiness, cost-estimate, and test data for the revised lane. ChangesSOTA v3 amendment
Estimated code review effort: 3 (Moderate) | ~25 minutes Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 1📝 Generate docstrings 💡
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 5
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@config/sota_v3_lane.json`:
- Around line 15-21: The reserved-cost references are stale and must match the
regenerated value of $73.4032502784 with contingency. In
config/sota_v3_lane.json lines 15-21, replace both $68.87 references; make the
same replacement in config/sota_v3_publication_protocol.json lines 89-90,
preserving the surrounding policy text.
- Line 28: Update config/sota_v3_lane.json lines 28-28 to describe eight frozen
route-acceptance records instead of ten. In
config/sota_v3_publication_protocol.json lines 52-52, revise the adjacent
power_definition and remaining_blocker prose to refer to eight contrasts and
eight strict smokes.
In `@config/sota_v3_publication_protocol.json`:
- Around line 32-34: The configuration declares a one-amendment terminal policy
that the runner does not enforce. Update the runner’s output-cap handling to
persist amendment count outside the process, apply the amended cap during full
re-smokes, and check max_cap_amendments/on_second_trigger before any completion
call so a second trigger aborts with repreregistration. Ensure
output_policy_amendment_rule and the configured cap fields remain authoritative.
In `@docs/PUBLISH_READINESS.md`:
- Around line 10-49: Reconcile the critical-path status with the amendment-3
execution state described in the document: update the unchecked preregistration
task and any ten-model, $150 reservation references to the current eight-model
cohort and $100 ceiling, or explicitly mark those entries as superseded
historical context. Ensure operators see consistent guidance that spend and
strict-smoke execution are authorized while panel execution and publication
remain blocked until strict-smoke acceptance.
In `@tests/test_sota_v3_preregistration.py`:
- Around line 382-390: Replace the fallback reservation calculation in the test
around the existing cap, fallback, ceiling, and reserved assertions: clone the
models, lane, and pricing inputs, set each of the three output-cap inputs to
fallback, then call estimate and compare its contingent total against ceiling.
Remove the reserved * (fallback / cap) assertion while preserving the existing
cap/fallback ordering and ceiling checks.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: de30ec47-fe96-490d-93b1-defd2215cb56
📒 Files selected for processing (12)
config/sota_v3_lane.jsonconfig/sota_v3_models.jsonconfig/sota_v3_pricing_snapshot.jsonconfig/sota_v3_publication_protocol.jsondocs/PUBLISH_READINESS.mddocs/run_logs/sota-v3-design-amendment-2026-08-06.mdresults/analysis/sota-v3-pre-smoke-cost-estimate.jsonresults/analysis/sota-v3-route-acceptance-evidence.jsontests/test_publication_cost.pytests/test_publication_runner.pytests/test_sota_v3_preregistration.pytests/test_sota_v3_route_catalog.py
💤 Files with no reviewable changes (1)
- config/sota_v3_pricing_snapshot.json
Comment on lines
+15
to
+21
| "rationale": "Owner-directed pre-data cohort reduction before any smoke or panel evidence exists. Gemini 3.6 Flash and Grok 4.5 are withdrawn because they are the only two registered routes whose reasoning is mandatory: every other model runs reasoning-disabled, so the pair reintroduced the exact cross-model reasoning inconsistency that caused their exclusion from the frozen sota-v2 panel. Withdrawing them makes the cohort uniformly reasoning-disabled, resolves the pending reasoning-policy verification outright, and removes the only routes whose observed sota-v2 telemetry approached the 3,072-token cap-pressure threshold. It also cuts the reserved panel cost from $127.29 to $68.87 against a lowered $100 operator ceiling. Scores and apparent model quality played no part: no v3 smoke or panel data exists.", | ||
| "changes": [ | ||
| "Registered model family grows from eight to ten; Holm family size is now 10 and the first-step threshold is 0.05/10 = 0.005.", | ||
| "GPT-5.6 Luna Pro is replaced by plain GPT-5.6 Luna on the first-party OpenAI route, removing the unresolved Pro reasoning.mode inconsistency.", | ||
| "DeepSeek V4 Flash 0731 and Tencent Hy3 are added as open-weight anchors, both on first-party FP8 routes.", | ||
| "Selected allocation moves from 15 seeds x 1 repeat to 16 seeds x 1 repeat (16 episodes/model): sensitivity power 0.8488, Wilson 95% CI [0.841645, 0.855688]; base power 0.9527, Wilson 95% CI [0.948363, 0.95669].", | ||
| "The pending private seed panel is now 16 seeds; its identity remains unfrozen pending separately authorized generation." | ||
| "Registered model family shrinks from ten to eight; Holm family size is now 8 and the first-step threshold loosens from 0.05/10 = 0.005 to 0.05/8 = 0.00625.", | ||
| "Gemini 3.6 Flash (Google AI Studio) and Grok 4.5 (xAI) are withdrawn as the only mandatory-reasoning routes; the remaining eight are uniformly reasoning-disabled.", | ||
| "Allocation stays at 16 seeds x 1 repeat. At family 8 the rule's smallest qualifying allocation is 15x1 (sensitivity 0.8357, Wilson lower 0.828309), but the pre-data seed commitment is frozen at 16; 16x1 gives sensitivity power 0.8727, Wilson 95% CI [0.866024, 0.87909], base power 0.9629.", | ||
| "Operator spend ceiling lowered from $150.00 to $100.00; reserved worst-case cost is $68.87 with the 1.2x contingency.", | ||
| "The cap-pressure amendment rule gains an explicit terminal case: the cap may be amended at most once, and a second trigger aborts sota-v3 rather than amending again." |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win
Use the regenerated reserved-cost value.
results/analysis/sota-v3-pre-smoke-cost-estimate.json records $73.4032502784 with contingency. These policy records state $68.87. The amendment record and readiness document already use $73.40.
config/sota_v3_lane.json#L15-L21: replace both$68.87references with the regenerated reservation.config/sota_v3_publication_protocol.json#L89-L90: replace$68.87with the regenerated reservation.
📍 Affects 2 files
config/sota_v3_lane.json#L15-L21(this comment)config/sota_v3_publication_protocol.json#L89-L90
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@config/sota_v3_lane.json` around lines 15 - 21, The reserved-cost references
are stale and must match the regenerated value of $73.4032502784 with
contingency. In config/sota_v3_lane.json lines 15-21, replace both $68.87
references; make the same replacement in
config/sota_v3_publication_protocol.json lines 89-90, preserving the surrounding
policy text.
| "every execution, spend, and publication authorization remains false" | ||
| "planning effect -100 score points, directional trails-reference claim", | ||
| "the frozen 4,096-token output cap and 3,072-token cap-pressure trigger", | ||
| "contract fingerprint a523bdfcebe47bbd and all ten frozen route-acceptance records", |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win
Remove stale ten-route statements.
The active registry and exact-route acceptance data contain eight routes. The lane still describes ten frozen route-acceptance records. The protocol still describes ten contrasts and ten strict smokes near its updated eight-member family field.
config/sota_v3_lane.json#L28-L28: change the route-acceptance record count to eight.config/sota_v3_publication_protocol.json#L52-L52: update the adjacentpower_definitionandremaining_blockerprose to eight contrasts and eight strict smokes.
📍 Affects 2 files
config/sota_v3_lane.json#L28-L28(this comment)config/sota_v3_publication_protocol.json#L52-L52
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@config/sota_v3_lane.json` at line 28, Update config/sota_v3_lane.json lines
28-28 to describe eight frozen route-acceptance records instead of ten. In
config/sota_v3_publication_protocol.json lines 52-52, revise the adjacent
power_definition and remaining_blocker prose to refer to eight contrasts and
eight strict smokes.
Comment on lines
+32
to
+34
| "amendment_rule": "The common 4,096-token cap is frozen for the first strict smoke of every route. If any smoke is truncated, reaches 3,072 output tokens, or cannot satisfy mandatory minimum reasoning, invalidate all v3 smokes, amend the cap once before panel data, and re-smoke the entire family. Scores and apparent model quality are never cap-selection inputs. TERMINAL CASE: the cap may be amended at most once. If the re-smoke at the amended cap trips the trigger again, sota-v3 is aborted -- no panel is run and nothing is published under this contract -- and the lane must be re-preregistered from scratch with a cap and ceiling costed for both branches. A second amendment is never permitted. Because the amended-cap panel also exceeds the committed operator ceiling, any amendment additionally requires an explicit owner ceiling decision before spend.", | ||
| "max_cap_amendments": 1, | ||
| "on_second_trigger": "abort-sota-v3-and-repreregister" |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
rg -n -C 5 -g '*.py' \
'max_cap_amendments|on_second_trigger|fallback_output_token_cap|cap_pressure_threshold_tokens|amendment_rule' \
gm_bench scripts tests
rg -n -C 5 -g '*.py' \
'run_publication_matrix|publication_execution_issues|strict smoke|strict_smoke|cap.*trigger|trigger.*cap' \
gm_bench scripts testsRepository: nedcut/gm-bench
Length of output: 36018
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Inspect the protocol/lane JSON and targeted publication module sections.
printf 'Config files:\n'
for f in config/sota_v3_publication_protocol.json config/sota_v3_lane.json config/sota_v3_sm smoke manifest.json; do
if [ -f "$f" ]; then
echo "--- $f"
wc -l "$f"
fi
done
printf '\nProtocol output_policy:\n'
python3 - <<'PY'
import json
from pathlib import Path
p=Path("config/sota_v3_publication_protocol.json")
data=json.loads(p.read_text())
print(json.dumps(data.get("output_policy",{}), indent=2, sort_keys=True))
PY
printf '\npublication.py relevant sections:\n'
wc -l gm_bench/publication.py scripts/run_publication_matrix.py
sed -n '330,390p' gm_bench/publication.py
sed -n '467,540p' gm_bench/publication.py
sed -n '650,690p' gm_bench/publication.py
sed -n '138,160p' scripts/run_publication_matrix.py
sed -n '400,440p' scripts/run_publication_matrix.py
sed -n '630,670,860,890,1040p' scripts/run_publication_matrix.py
printf '\nAll references to amendment-related identifiers in tracked files:\n'
rg -n -C 3 'max_cap_amendments|on_second_trigger|output_policy_amendment|cap_pressure|amendment_rule|fallback_output_token_cap|abort-sota-v3-and-repreregister' .Repository: nedcut/gm-bench
Length of output: 13823
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '\nTracked files mentioning amendment identifiers:\n'
rg -n -C 3 'max_cap_amendments|on_second_trigger|output_policy_amendment|amendment_rule|abort-sota-v3-and-repreregister' --glob '!*.pyc' --glob '!*.json' . || true
printf '\nExecution issues body cap section:\n'
sed -n '467,555p' gm_bench/publication.py
printf '\nSmoke manifest validation cap triggers:\n'
sed -n '650,700p' gm_bench/publication.py
printf '\nPublication runner authorization and cap propagation:\n'
sed -n '138,210p' scripts/run_publication_matrix.py
sed -n '490,640p' scripts/run_publication_matrix.py
sed -n '840,870p' scripts/run_publication_matrix.py
printf '\nJSON config amendment fields:\n'
python3 - <<'PY'
import json
from pathlib import Path
for f in ("config/sota_v3_publication_protocol.json", "config/sota_v3_lane.json"):
data = json.loads(Path(f).read_text())
print(f"--- {f}")
for key in ("output_policy", "output_token_cap", "cap_pressure_threshold_tokens", "fallback_output_token_cap", "output_policy_amendment_rule"):
if key == "output_policy" and isinstance(data.get(key), dict):
for k, v in data[key].items():
if k in ("output_token_cap", "cap_pressure_threshold_tokens", "fallback_output_token_cap", "max_cap_amendments", "on_second_trigger", "amendment_rule"):
print(f" {key}.{k}: {v!r}")
elif key in data:
print(f" {key}: {data[key]!r}")
PY
printf '\nProgrammatic invariant: amendment policy fields absent from Python refs except config/test assertions?\n'
python3 - <<'PY'
import ast
from pathlib import Path
terms = {"max_cap_amendments", "on_second_trigger", "output_policy_amendment_rule", "amendment_rule"}
for path in sorted(Path(".").rglob("*.py")):
if any(part in {".git", "__pycache__", "node_modules"} for part in path.parts):
continue
try:
text = path.read_text(encoding="utf-8", errors="replace")
tree = ast.parse(text)
except SyntaxError:
continue
found = set()
for node in ast.walk(tree):
if isinstance(node, ast.Name) and node.id in terms:
found.add(node.id)
elif isinstance(node, ast.Constant) and node.value in terms:
found.add(node.value)
if found:
print(f"{path}: {sorted(found)}")
PYRepository: nedcut/gm-bench
Length of output: 25078
Persist and enforce the amended-cap terminal policy.
The runner still reads output_token_cap, cap_pressure_threshold_tokens, and fallback_output_token_cap, but only the test/doc references max_cap_amendments, on_second_trigger, and output_policy_amendment_rule. Track the amendment count outside the runner process, enforce the amended cap/value on re-smokes, and abort with repreregistration before another completion call after the second trigger.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@config/sota_v3_publication_protocol.json` around lines 32 - 34, The
configuration declares a one-amendment terminal policy that the runner does not
enforce. Update the runner’s output-cap handling to persist amendment count
outside the process, apply the amended cap during full re-smokes, and check
max_cap_amendments/on_second_trigger before any completion call so a second
trigger aborts with repreregistration. Ensure output_policy_amendment_rule and
the configured cap fields remain authoritative.
Comment on lines
+10
to
+49
| **Last reviewed:** 2026-08-06 | ||
| **Current target:** Preserve the published `sota-v2` study as frozen historical | ||
| evidence while pre-registering and rehearsing a finite `sota-v3` publication | ||
| lane. | ||
| **Current state:** The frozen phase-one public panel, blog, GitHub release, and | ||
| results-first site are published. Eight of ten | ||
| registered models produced route-matched, cost-complete `sota-v2` rows at the | ||
| shared 4,096-token native-minimum-reasoning cap, clearing the predeclared | ||
| minimum. Grok 4.5 and Mistral Medium 3.5 completed but remain diagnostic because | ||
| `sota-v2`-registered models produced route-matched, cost-complete `sota-v2` | ||
| rows at the shared 4,096-token native-minimum-reasoning cap, clearing the | ||
| predeclared minimum. Grok 4.5 and Mistral Medium 3.5 completed but remain diagnostic because | ||
| their artifacts lacked complete usage or cost coverage. The generated site now | ||
| exposes the eight eligible rows. All models overlap in one uncertainty tier, | ||
| and every eligible model trails `pick-trader`. The three P0 correctness and | ||
| artifact-integrity fixes landed in #85 as an explicit `sota-v3` contract. | ||
| Sunday's merged work closed the contract-economics, same-view, gap-diagnostic, | ||
| site-framing, statistical-tiering, and version-dispatched CI items. The current | ||
| working tree contains a ten-model public-catalog cohort, a frozen 16-seed x | ||
| 1-repeat statistical design, a frozen 4,096-token smoke ceiling, an empty | ||
| site-framing, statistical-tiering, and version-dispatched CI items. Pre-data | ||
| design amendment 3 (2026-08-06, | ||
| `docs/run_logs/sota-v3-design-amendment-2026-08-06.md`) withdrew Gemini 3.6 | ||
| Flash and Grok 4.5 — the only two mandatory-reasoning routes — leaving an | ||
| eight-model, uniformly reasoning-disabled cohort (2 frontier-proprietary / 6 | ||
| open-weight) at Holm family size 8. The current working tree contains that | ||
| cohort, a frozen 16-seed x 1-repeat statistical design (sensitivity power | ||
| 0.8727, Wilson lower 0.8660), a frozen 4,096-token smoke ceiling, an empty | ||
| not-started smoke manifest, explicit runner dispatch, and a zero-spend synthetic | ||
| rehearsal. Exact-route and synthetic-data privacy acceptance are recorded for | ||
| all ten routes, the private seed commitment is frozen with its secret in macOS | ||
| Keychain, and the latest authenticated route preflight plus Keychain-backed | ||
| dry-run pass without completion calls. Spend and strict-smoke execution are now | ||
| authorized under the committed $150 ceiling. Panel execution and publication | ||
| remain false until every strict smoke is accepted and the cap-pressure rule is | ||
| resolved. There is no real v3 smoke or leaderboard artifact yet. | ||
| all eight routes, the private 16-seed commitment is frozen with its secret in | ||
| macOS Keychain, and the latest authenticated route preflight plus | ||
| Keychain-backed dry-run pass without completion calls. Spend and strict-smoke | ||
| execution are authorized under a committed ceiling lowered to $100 against a | ||
| $73.40 reserved worst case (~$25-30 projected actual). | ||
|
|
||
| The cap-pressure rule is now resolved. It gains an explicit terminal case: the | ||
| cap may be amended at most once, and a second trigger aborts `sota-v3` rather | ||
| than amending again (`max_cap_amendments: 1`, `on_second_trigger: | ||
| "abort-sota-v3-and-repreregister"`). The 8,192-token fallback branch reserves | ||
| $116.26 and therefore sits deliberately above the ceiling, so a trigger halts | ||
| the run for an explicit owner spend decision instead of silently widening the | ||
| budget it is measured against. The reasoning-policy ambiguity is resolved by | ||
| construction, since no mandatory-reasoning route remains. Panel execution and | ||
| publication remain false until every strict smoke is accepted. There is no real | ||
| v3 smoke or leaderboard artifact yet. |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Reconcile the critical-path status in this document.
This section states that spend and strict-smoke execution are authorized. The critical-path section still has an unchecked preregistration task and retains the ten-model, $150 reservation state. This can give an operator conflicting execution guidance.
Replace that old task state with amendment-3 status, or mark it as superseded historical context.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/PUBLISH_READINESS.md` around lines 10 - 49, Reconcile the critical-path
status with the amendment-3 execution state described in the document: update
the unchecked preregistration task and any ten-model, $150 reservation
references to the current eight-model cohort and $100 ceiling, or explicitly
mark those entries as superseded historical context. Ensure operators see
consistent guidance that spend and strict-smoke execution are authorized while
panel execution and publication remain blocked until strict-smoke acceptance.
Comment on lines
+382
to
+390
| cap = lane["output_token_cap"] | ||
| fallback = policy["fallback_output_token_cap"] | ||
| ceiling = protocol["budget_policy"]["operator_ceiling_usd"] | ||
| reserved = json.loads(Path("results/analysis/sota-v3-pre-smoke-cost-estimate.json").read_text()) | ||
| reserved = reserved["costs_usd"]["total_with_1_2x_contingency"] | ||
|
|
||
| assert fallback > cap | ||
| assert reserved <= ceiling | ||
| assert reserved * (fallback / cap) > ceiling |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Calculate the fallback reservation with estimate.
Line 390 doubles the full reservation. Input-token cost does not scale with fallback / cap. This assertion does not validate the documented $116.26 fallback reservation.
Clone the models, lane, and pricing inputs. Set all three output-cap inputs to fallback. Then call estimate and compare its contingent total with ceiling.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@tests/test_sota_v3_preregistration.py` around lines 382 - 390, Replace the
fallback reservation calculation in the test around the existing cap, fallback,
ceiling, and reserved assertions: clone the models, lane, and pricing inputs,
set each of the three output-cap inputs to fallback, then call estimate and
compare its contingent total against ceiling. Remove the reserved * (fallback /
cap) assertion while preserving the existing cap/fallback ordering and ceiling
checks.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pre-data design amendment 3, closing the last open readiness gate before the paid smoke. Owner-directed: the goal was holding spend down on a hobby-scale project.
No spend, no data. No completion endpoint was called, no private seeds were regenerated, and panel/publication authorization both remain false. Contract fingerprint
a523bdfcebe47bbdis unchanged — this touches onlyconfig/,docs/,results/analysis/, and tests, none of which are_CONTRACT_SOURCES.What changed
Gemini 3.6 Flash and Grok 4.5 are withdrawn — the only two registered routes whose catalog marks reasoning mandatory. Turning reasoning off was never an option: the other eight already run with it disabled, and these two are
mandatory-minimumat the lowest effort OpenRouter offers.Power goes up: a smaller family loosens the Holm first step from 0.005 to 0.00625.
Why this one cut does four things
sota-v2panel for this exact reason. The cohort is now uniformly reasoning-disabled, making v2↔v3 more comparable, not less.reasoning_policy: "catalog-pinned-pending-strict-smoke-behavior-verification"had nothing left to verify.Cap-pressure rule: resolved
The rule said to amend the cap "once" but never defined a second trigger — so the only written instruction, read literally, was to amend again.
config/*.jsonsits outside the fingerprint, so a second amendment would leave no trace. Now pinned:max_cap_amendments: 1,on_second_trigger: "abort-sota-v3-and-repreregister".The ceiling is also deliberately set below the 8,192-token fallback branch ($116.26 reserved). Previously $150 was sized for the primary branch alone while the fallback reserved $219.83 — meaning
_enforce_operator_ceilingwould have blocked the very remedy the rule prescribed, the same way the plan crossed its own ceiling at $120. A trigger should halt for an explicit owner spend decision, not silently widen the budget it's measured against. A test asserts the fallback stays above the ceiling.Two judgment calls, both recorded rather than silent
16 seeds retained, not 15. At family 8 the frozen selection rule's smallest qualifying allocation is 15×1 (0.8357) — exactly reproducing the design from when the family last had eight members. But
seed_panelis already frozen pre-data with asha256, ahiding_commitment_sha256, and its secret in the Keychain. Breaking a pre-data cryptographic commitment to save $4.53 is the worse trade. 16×1 is strictly more powerful than the rule's minimum and never less conservative.Cohort balance was traded for reasoning uniformity. 2 frontier-proprietary / 6 open-weight, keeping OpenAI and Anthropic, losing Google and xAI. Public presentation must describe the family as predominantly open-weight. The 2/6 split and the ZDR count are both pinned by tests so neither drifts further without a deliberate decision.
A test that would have stopped testing
test_v3_reasoning_enabled_route_requires_an_explicit_token_assumptionselected the real Grok row to prove the estimator rejects unpriced reasoning. Withdrawing Grok would have left the filter matching nothing and the guard passing vacuously. It now builds a synthetic reasoning-enabled route — the unpriced-reasoning trap is a property of the estimator, not of whoever is in the cohort this month.Verification
ruff checkandruff format --checkcleana523bdfcebe47bbdOPENROUTER_REASONING_ENABLED=false--max-spend-usd 150now correctly rejected against the $100 ceilingcompletion_calls: 0Full rationale and reproduction commands in
docs/run_logs/sota-v3-design-amendment-2026-08-06.md.After this merges
Only the strict smoke remains before the panel:
Summary by CodeRabbit
Updates
Policy Changes