Freeze SOTA-v3 for strict smoke execution - #111
Conversation
|
Warning Review limit reached
Next review available in: 41 minutes You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (8)
📝 WalkthroughWalkthroughThe PR freezes SOTA-v3 route, seed, pricing, and publication metadata. It adds authenticated route-evidence collection, strict output-cap validation, and a Keychain-backed smoke launcher. Smoke execution is authorized under a $150 ceiling; panel execution and publication remain disabled. ChangesSOTA-v3 strict-smoke readiness
Estimated code review effort: 4 (Complex) | ~45 minutes Possibly related issues
Possibly related PRs
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 1📝 Generate docstrings 💡
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🧹 Nitpick comments (3)
tests/test_publication_runner.py (1)
1612-1631: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winAdd the numeric-cap case to the deferral test.
The test covers null cap rejected by default, null cap accepted with the deferral, and rejection when
max_tokenssupport disappears. It does not cover the case where the endpoint reports a numeric maximum below the cell cap while the deferral is present.That case matters because line 554 of
scripts/run_publication_matrix.pygates the exception onmaximum is None. A registry deferral must not rescue an endpoint that publishes a too-small numeric maximum. One assertion documents that boundary.💚 Proposed addition
payload["data"]["endpoints"][0]["supported_parameters"].remove("max_tokens") assert "cannot honor required parameters" in _endpoint_issues(deferred, payload)[0] + + # A registry deferral must not rescue a published numeric maximum below the cap. + payload["data"]["endpoints"][0]["supported_parameters"].append("max_tokens") + payload["data"]["endpoints"][0]["max_completion_tokens"] = 2048 + assert "cannot honor required parameters" in _endpoint_issues(deferred, payload)[0]🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/test_publication_runner.py` around lines 1612 - 1631, The test test_endpoint_preflight_allows_explicit_null_cap_deferral_only_until_strict_smoke should also verify that deferral does not allow a numeric endpoint maximum below the cell cap. With the deferred cell and max_tokens still supported, set max_completion_tokens to a numeric value less than the cell’s cap and assert _endpoint_issues reports the required-parameters failure.scripts/collect_sota_v3_route_evidence.py (1)
237-244: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueValidate the output path before the network reads, and apply the same rule to
--registry.Two small gaps:
- The containment check on line 240 runs after
collectperforms ten authenticated round trips. A mistyped--outputwastes the whole collection and writes nothing.--registryis read on line 237 and rewritten on line 244 with no containment check, while--outputgets one.♻️ Proposed fix
- registry = _read_json(args.registry) - evidence = collect(registry, headers) output = args.output.resolve() - if not output.is_relative_to(ROOT.resolve()): - parser.error("evidence output must be inside the repository") + registry_path = args.registry.resolve() + root = ROOT.resolve() + for label, path in (("evidence output", output), ("registry", registry_path)): + if not path.is_relative_to(root): + parser.error(f"{label} must be inside the repository") + registry = _read_json(registry_path) + evidence = collect(registry, headers) _write_json(output, evidence) if args.apply_registry: - _write_json(args.registry, apply_registry(registry, evidence, output), sort_keys=False) + _write_json(registry_path, apply_registry(registry, evidence, output), sort_keys=False)🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@scripts/collect_sota_v3_route_evidence.py` around lines 237 - 244, In the main flow around collect and the output-path validation, validate both args.output and args.registry are contained within ROOT before calling collect or reading either file. Reuse the existing repository-containment rule and parser.error behavior for each path, while preserving the subsequent evidence collection and optional registry update flow.scripts/run_publication_matrix.py (1)
554-562: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winDefine the
request-cap-pending-strict-smokecontract once. The same four-field dict is written out in four places across three files. No shared definition exists, so a change to any field name or value must be applied in all four places by hand. A missed copy fails closed in one path and open in another.
scripts/run_publication_matrix.py#L554-L562: replace the inline four-field re-check with a call to a shared predicate, for exampleis_pending_strict_smoke_cap(verification).scripts/run_publication_matrix.py#L259-L273: compare against the shared constant instead of the inline dict literal.scripts/collect_sota_v3_route_evidence.py#L115-L129: import the shared constant instead of the inline dict literal on lines 121-126.tests/test_publication_runner.py#L1619-L1627: build theoutput_cap_verificationfixture from the shared constant so the test tracks the production contract.Put the constant and the predicate in
gm_bench/publication.py, which both scripts already import.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@scripts/run_publication_matrix.py` around lines 554 - 562, Define the shared request-cap contract constant and is_pending_strict_smoke_cap predicate in gm_bench/publication.py. In scripts/run_publication_matrix.py lines 554-562, replace the inline field checks with the predicate; in lines 259-273, compare against the shared constant. In scripts/collect_sota_v3_route_evidence.py lines 115-129, import and use the shared constant, and in tests/test_publication_runner.py lines 1619-1627, construct output_cap_verification from that constant.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/PUBLISH_READINESS.md`:
- Line 908: Remove the blank line immediately before the new 2026-08-04
decision-log row so it remains part of the existing Markdown table and renders
under the table header.
In `@gm_bench/publication.py`:
- Around line 117-128: The v3 route acceptance validation must verify that both
evidence digests match their canonical payloads, not merely exist. In
v3_route_acceptance_issues(), load each relevant evidence_artifact and recompute
canonical_sha256(...) for route_evidence_sha256 and
privacy_acceptance.evidence_sha256, adding validation issues when either digest
does not match before smoke authorization proceeds.
In `@scripts/run_publication_matrix.py`:
- Around line 259-273: Move the output_cap_verification validation block from
_validate_models’ exact-route-only section to before the exact_routes=False
early return, so sweep models receive the same type, value, and max_tokens
support checks. Remove the now-duplicated validation block from the later
exact-route validation path while preserving the existing error behavior.
---
Nitpick comments:
In `@scripts/collect_sota_v3_route_evidence.py`:
- Around line 237-244: In the main flow around collect and the output-path
validation, validate both args.output and args.registry are contained within
ROOT before calling collect or reading either file. Reuse the existing
repository-containment rule and parser.error behavior for each path, while
preserving the subsequent evidence collection and optional registry update flow.
In `@scripts/run_publication_matrix.py`:
- Around line 554-562: Define the shared request-cap contract constant and
is_pending_strict_smoke_cap predicate in gm_bench/publication.py. In
scripts/run_publication_matrix.py lines 554-562, replace the inline field checks
with the predicate; in lines 259-273, compare against the shared constant. In
scripts/collect_sota_v3_route_evidence.py lines 115-129, import and use the
shared constant, and in tests/test_publication_runner.py lines 1619-1627,
construct output_cap_verification from that constant.
In `@tests/test_publication_runner.py`:
- Around line 1612-1631: The test
test_endpoint_preflight_allows_explicit_null_cap_deferral_only_until_strict_smoke
should also verify that deferral does not allow a numeric endpoint maximum below
the cell cap. With the deferred cell and max_tokens still supported, set
max_completion_tokens to a numeric value less than the cell’s cap and assert
_endpoint_issues reports the required-parameters failure.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 5593ea7c-0c9c-4785-bf03-67024acb0856
📒 Files selected for processing (18)
config/sota_v3_lane.jsonconfig/sota_v3_models.jsonconfig/sota_v3_pricing_snapshot.jsonconfig/sota_v3_publication_protocol.jsondocs/PUBLISH_READINESS.mddocs/production_benchmark.mddocs/run_logs/sota-v3-smoke-readiness-freeze-2026-08-04.mdgm_bench/publication.pyresults/analysis/sota-v3-route-acceptance-evidence.jsonscripts/collect_sota_v3_route_evidence.pyscripts/run_publication_matrix.pyscripts/run_sota_v3_smoke_from_keychain.pytests/test_publication_release.pytests/test_publication_runner.pytests/test_sota_v3_preregistration.pytests/test_sota_v3_route_catalog.pytests/test_sota_v3_route_evidence.pytests/test_sota_v3_smoke_keychain.py
| connection = http.client.HTTPSConnection( # nosemgrep: python.lang.security.audit.httpsconnection-detected.httpsconnection-detected | ||
| "openrouter.ai", timeout=30 | ||
| ) |
Summary
max_completion_tokensmetadata with a narrow request-cap-pending-smoke state while retaining the full post-smoke cap-pressure gateWhy
The prior route preflight passed eight routes and blocked Grok 4.5 plus Mistral Medium 3.5 because their exact endpoints advertise
max_tokensbut omitmax_completion_tokens. No same-model alternative endpoint supplies the missing maximum. Treating those routes as unbounded would be unsafe, but rejecting the smoke prevents the only bounded behavioral check that can resolve the metadata gap.This change allows that exact, registered exception only when
max_tokensis advertised and strict-smoke verification remains mandatory. Panel eligibility still requires complete finish-reason and usage telemetry, zero truncations, and peak output below the preregistered 3,072-token pressure threshold.The privacy decision is also explicit: GM-Bench inputs are synthetic and contain no personal or confidential data;
data_collection=denyprohibits provider training use; provider retention terms are accepted for this workload; and ZDR is recorded per exact endpoint rather than claimed universally. Five of ten registered routes were in OpenRouter's authenticated ZDR list at the freeze.Safety boundaries
GM_BENCH_WORKERS=1falseValidation
uv run pytest -q— 759 passeduv run ruff format --check gm_bench examples tests scriptsuv run ruff check gm_bench examples tests scriptsuv run python -m gm_bench validate-contractuv run python scripts/sota_v3_rehearsal.py— passed,$0.00,smoke_execution_issues: [], 13 expected panel blockersbun run lintandbun run buildinweb/Summary by CodeRabbit