Skip to content

Pin SGLang runtime and production capacity defaults#596

Draft
wang-tong0 wants to merge 3 commits into
mainfrom
fix/pr594-remove-strict-workaround
Draft

Pin SGLang runtime and production capacity defaults#596
wang-tong0 wants to merge 3 commits into
mainfrom
fix/pr594-remove-strict-workaround

Conversation

@wang-tong0

Copy link
Copy Markdown
Collaborator

Summary

  • pin validator-managed SGLang deployments to the stock lmsysorg/sglang:v0.5.14 image across SSH, Targon, CLI, and endpoint defaults
  • forward the opt-in TERMINEL_SGLANG_DP_SIZE setting only to Terminal environments so a session can stay on one DP rank
  • make the executor per-endpoint dispatch budget configurable, retaining the production default of 600
  • cap the autoscaler's implicit default at one automatic instance
  • disable Memory sampling and scoring by default

Why

This is the cleaned replacement for #594. Terminal requests no longer send function.strict: true, so the request no longer enters SGLang's structural-tag/XGrammar constrained-decoding path under tool_choice=auto. The custom patched image, global SGLANG_DIAG_DISABLE_TOOL_CONSTRAINT bypass, and Compose image override are therefore unnecessary and are intentionally absent.

The experimental num_continuous_decode_steps and enable_mixed_chunk defaults are also excluded. In the isolated B200 tests, the former was not read by the v0.5.14 scheduler hot path, while the latter increased GPU duty cycle and power without improving end-to-end request throughput. It has not been revalidated after removing strict tool constraints.

Pinning a stock runtime still prevents behavior from changing silently when a floating latest image moves. Sticky DP routing preserves multi-turn prefix-cache locality, while the configurable dispatch budget keeps endpoint capacity explicit.

Impact

  • no custom SGLang fork or server-side constraint bypass is deployed
  • required or named tool-choice constraints remain available for other callers
  • existing endpoint rows that explicitly store a different image are not mutated by this PR
  • TERMINEL_SGLANG_DP_SIZE remains opt-in and must match the endpoint's actual --dp value
  • Memory and autoscaler defaults change as listed above

Validation

  • rebased on current origin/main (4b3349f)
  • python -m pytest -q
  • 877 passed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant