Pin SGLang runtime and production capacity defaults#596
Draft
wang-tong0 wants to merge 3 commits into
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
lmsysorg/sglang:v0.5.14image across SSH, Targon, CLI, and endpoint defaultsTERMINEL_SGLANG_DP_SIZEsetting only to Terminal environments so a session can stay on one DP rankWhy
This is the cleaned replacement for #594. Terminal requests no longer send
function.strict: true, so the request no longer enters SGLang's structural-tag/XGrammar constrained-decoding path undertool_choice=auto. The custom patched image, globalSGLANG_DIAG_DISABLE_TOOL_CONSTRAINTbypass, and Compose image override are therefore unnecessary and are intentionally absent.The experimental
num_continuous_decode_stepsandenable_mixed_chunkdefaults are also excluded. In the isolated B200 tests, the former was not read by the v0.5.14 scheduler hot path, while the latter increased GPU duty cycle and power without improving end-to-end request throughput. It has not been revalidated after removing strict tool constraints.Pinning a stock runtime still prevents behavior from changing silently when a floating
latestimage moves. Sticky DP routing preserves multi-turn prefix-cache locality, while the configurable dispatch budget keeps endpoint capacity explicit.Impact
TERMINEL_SGLANG_DP_SIZEremains opt-in and must match the endpoint's actual--dpvalueValidation
origin/main(4b3349f)python -m pytest -q