fix(sglang): clamp deterministic sampler endpoints - #271
Open
DreamEnding wants to merge 1 commit into
Open
Conversation
This was referenced Aug 13, 2026
DreamEnding
marked this pull request as ready for review
August 13, 2026 13:34
There was a problem hiding this comment.
Pull request overview
This PR backports SGLang 0.5.12.post1’s deterministic-sampler uint32 endpoint guard into Relax’s SGLang runtime, preventing hash==0xffffffff → u==1.0 → Gumbel(+inf) from letting a token incorrectly win argmax regardless of model probability.
Changes:
- Add a version-gated, idempotent patch that replaces
sglang.srt.layers.sampler.multinomial_with_seedwith a safe Gumbel transform for the affected SGLang version. - Route SGLang scheduler subprocess startup through a top-level, pickle-safe wrapper that applies the endpoint patch before optional routing-replay patching.
- Add focused unit tests covering endpoint numerics, version gating/signature drift, idempotency, and scheduler wrapper ordering/picklability.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
relax/backends/sglang/deterministic_sampler_patch.py |
Introduces the version-gated backport patch and safe deterministic multinomial implementation. |
relax/backends/sglang/sglang_engine.py |
Ensures scheduler subprocess runs via a top-level wrapper that applies runtime patches in the intended order. |
tests/backends/sglang/test_deterministic_sampler_patch.py |
Adds unit tests for numerics, version gating, signature validation, and idempotent install behavior. |
tests/backends/sglang/test_router_registration.py |
Adds tests verifying wrapper ordering (endpoint patch before routing patch) and picklability of the scheduler wrapper. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+57
to
+60
| gumbel = hashed.to(torch.float64) / torch.iinfo(torch.uint32).max | ||
| _uniform_hash_to_gumbel_(gumbel) | ||
| gumbel.add_(logprobs.to(torch.float64)) | ||
| return torch.argmax(gumbel, dim=1, keepdim=True) |
| launch_server(server_args, run_scheduler_process_func=_patched_run_scheduler_process) | ||
| else: | ||
| launch_server(server_args) | ||
| launch_server(server_args, run_scheduler_process_func=_patched_run_scheduler_process) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
0.5.12.post1的确定性采样会将哈希值0xffffffff映射为精确的1.0,在 Gumbel 变换中产生+inf,使某个 token 可能无视模型概率而赢得 argmax。Changes
relax/backends/sglang/deterministic_sampler_patch.py,校验安装版本与multinomial_with_seed签名后,以安全 Gumbel 变换替换受影响版本的确定性采样函数;relax/backends/sglang/sglang_engine.py增加顶层、可 pickle 的_launch_server_with_patches与_patched_run_scheduler_process,确保端点修复先于可选 routing-replay 补丁执行;0.5.12.post1,其余版本保持原行为;Verification
环境、硬件、commit:Windows 本地 CPU 环境;HEAD
218115e65ca2680155429ed5194983bdfc2c37f9。可复制命令:
单元/集成/端到端测试结果:端点补丁单元测试
6 passed;git diff --check通过。真实 SGLang scheduler 进程集成测试未在本地运行。性能 before/after(若适用):不适用;该变更修复极低频端点错误,不以吞吐或延迟优化为目标。
日志、曲线、profile 链接:无;待 Linux + SGLang
0.5.12.post1运行时验证后补充 scheduler 日志。Risk & Rollback
tests/backends/sglang/test_router_registration.py在当前 Windows 环境收集失败,因为上游relax.utils.s3_model_loader依赖 Linux 专用fcntl;pre-commit run --all-files会被既有 symlink/patch 文件的end-of-file-fixer改动中断,相关无关改动已还原。0.5.12.post1的 sampler 函数签名;若该版本运行时存在不同签名,将显式报错而不是静默替换。218115e。Checklist