th-1cc9fa: model-output ceiling clamp + raise starvation defaults (Python server parity)#194
Open
brentrager wants to merge 1 commit into
Open
th-1cc9fa: model-output ceiling clamp + raise starvation defaults (Python server parity)#194brentrager wants to merge 1 commit into
brentrager wants to merge 1 commit into
Conversation
…thon server parity) Python-server parity for the Rust server hardening (th-1cc9fa / th-562b6d): - Raise the anti-starvation turn sizing the runner applies: max_tokens 512 -> 8192 (DEFAULT_MAX_TOKENS), max_iterations 6 -> 20 (DEFAULT_MAX_ITERATIONS). The old chat-widget sizing starves reasoning models (empty replies / truncated tool loops). - New model_info.py: best-effort per-model output ceiling from the gateway's /model/info (map_model_info + model_output_ceiling), cached once per process, stdlib fetch, no extra deps. No key / any error / unknown model -> None -> unclamped. - Thread the resolved ceiling into AgentOptions.model_max_output so the engine clamps max_tokens to what the model can emit. Feature-detected against the pinned published core (which predates the field) so this stays green now and activates automatically once a core >= 1.3.2 is pinned. Tests: test_model_info.py (map + ceiling lookup: clamp/unknown/no-key/error/cache/url) and test_starvation_defaults.py (raised constants, runner sends 8192, iteration cap 20, + a forward-compat clamp test that skips on the pre-clamp core). 47 passed, 1 skipped. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The server's per-turn sizing (
max_tokens=512,max_iterations=6) is chat-widget sizing that starves reasoning models — they exhaust the token budget on reasoning and return empty, or run out of loop iterations mid-tool-use. And a budgetmax_tokenscan exceed what a model can physically emit. Rust parity for th-1cc9fa (raise defaults) + th-562b6d (clamp chatmax_tokensto the/model/infoceiling).Change (mirrors
rust/smooth-operator-server/src/{config,admin,handler,runner}.rs)DEFAULT_MAX_TOKENS512→8192,DEFAULT_MAX_ITERATIONS6→20.model_info.py— best-effort per-model output ceiling from the gateway's/model/info(map_model_info+model_output_ceiling), the Python analog ofadmin.rs. Cached once per process, stdliburllibfetch (no new deps). No gateway key / any transport error / unknown model / missing ceiling ⇒None⇒ engine leavesmax_tokensunclamped.AgentOptions.model_max_output(from the core parity PR) so the engine clampsmax_tokensto what the model can emit — the raised 8192 default is then safe.Release-ordering (important)
The server pins the published
smooai-smooth-operator-core(currently 1.3.1, which predates the clamp field). The ceiling threading is feature-detected (_CORE_SUPPORTS_CEILING) so this PR is green on the pinned core today and the clamp auto-activates once core ≥ 1.3.2 (the paired core PR) is published and the pin bumped. Until then: the raised defaults are live; the clamp is wired but dormant.Paired core PR: SmooAI/smooth-operator-core#73 (publish core 1.3.2, then bump this repo's
smooai-smooth-operator-corepin).Tests
test_model_info.py—map_model_info(positive/skip/garbage payloads) +model_output_ceiling(clamp lookup, unknown model, no-key skips fetch, error→None uncached, success cached once,/model/infoURL).test_starvation_defaults.py— raised constants, runner sends 8192, iteration cap is 20 (behavioral), + a forward-compat end-to-end clamp test that skips on the pre-clamp core (verified: passes when run against the local core-parity build).Full server suite: 47 passed, 1 skipped; ruff clean.
EPIC th-1cc9fa.
🤖 Generated with Claude Code