Skip to content

Update DSv4 TRT image for B200/B300 (non-MTP) to the SWA-scratch-fix build (c914d6d) #1636

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Oseltamivir wants to merge 9 commits into
base: main
Choose a base branch
from
Open

Update DSv4 TRT image for B200/B300 (non-MTP) to the SWA-scratch-fix build (c914d6d) #1636

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
4bcefb7
Update DSv4 TRT image for B200/B300 (non-MTP) to feat-deepseek_v4-2dd…
Oseltamivir Jun 1, 2026
6b7558c
Backfill PR number in changelog pr-link
Oseltamivir Jun 1, 2026
bd3c94c
Merge branch 'main' into update-dsv4-trt-image-2dd03e6
Oseltamivir Jun 1, 2026
f441f9f
Try official TRT-LLM release image 1.3.0rc15.post1 for DSv4 B200/B300…
Oseltamivir Jun 1, 2026
4bc5592
Revert to custom feat/deepseek_v4-2dd03e6 image for DSv4 B200/B300 (n…
Oseltamivir Jun 2, 2026
1b0afeb
Merge branch 'main' into update-dsv4-trt-image-2dd03e6
Oseltamivir Jun 2, 2026
14a1bb3
Point DSv4 B200/B300 TRT (non-MTP) at the SWA-scratch-fix image
Oseltamivir Jun 2, 2026
242ab88
Merge remote-tracking branch 'origin/main' into update-dsv4-trt-image…
Oseltamivir Jun 2, 2026
e23a541
Revert DSv4 B200/B300 TRT (non-MTP) to 2dd03e6 + disable SWA scratch …
Oseltamivir Jun 2, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/configs/nvidia-master.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1801,7 +1801,7 @@ dsv4-fp4-b200-vllm-agentic:
- { tp: 8, ep: 8, dp-attn: true, offloading: cpu, conc-list: [64, 128, 256] }

dsv4-fp4-b200-trt:
image: ghcr.io#semianalysisai/trtllm-deepseek-v4:feat-deepseek_v4-9aa3715
image: ghcr.io#semianalysisai/trtllm-deepseek-v4:feat-deepseek_v4-2dd03e6
model: deepseek-ai/DeepSeek-V4-Pro
model-prefix: dsv4
runner: b200-dsv4
Expand Down Expand Up @@ -3049,7 +3049,7 @@ dsv4-fp4-b300-vllm-agentic:
- { tp: 8, ep: 8, dp-attn: true, offloading: cpu, conc-list: [128, 256, 512] }

dsv4-fp4-b300-trt:
image: ghcr.io#semianalysisai/trtllm-deepseek-v4:feat-deepseek_v4-9aa3715
image: ghcr.io#semianalysisai/trtllm-deepseek-v4:feat-deepseek_v4-2dd03e6
model: deepseek-ai/DeepSeek-V4-Pro
model-prefix: dsv4
runner: b300
Expand Down
7 changes: 7 additions & 0 deletions benchmarks/single_node/fixed_seq_len/dsv4_fp4_b200_trt.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,13 @@ sanitize_slurm_mpi_env_for_trtllm
export NCCL_NVLS_ENABLE="${NCCL_NVLS_ENABLE:-0}"
echo "NCCL_NVLS_ENABLE: $NCCL_NVLS_ENABLE"

# Disable DSv4 SWA scratch reuse: with attention-DP the V2 scheduler grows ctx KV
# (incl. SWA scratch) before delay batching defers a request, and the resulting
# revert_allocate_context -> resize(shrink) can't release the scratch of a
# never-forwarded request (no finish_event), crashing every dpa=true job.
export TRTLLM_DSV4_ENABLE_SWA_SCRATCH_REUSE="${TRTLLM_DSV4_ENABLE_SWA_SCRATCH_REUSE:-0}"
echo "TRTLLM_DSV4_ENABLE_SWA_SCRATCH_REUSE: $TRTLLM_DSV4_ENABLE_SWA_SCRATCH_REUSE"

if [[ "$MODEL" != /* ]]; then
hf download "$MODEL"
fi
Expand Down
7 changes: 7 additions & 0 deletions benchmarks/single_node/fixed_seq_len/dsv4_fp4_b300_trt.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,13 @@ sanitize_slurm_mpi_env_for_trtllm
export NCCL_NVLS_ENABLE="${NCCL_NVLS_ENABLE:-0}"
echo "NCCL_NVLS_ENABLE: $NCCL_NVLS_ENABLE"

# Disable DSv4 SWA scratch reuse: with attention-DP the V2 scheduler grows ctx KV
# (incl. SWA scratch) before delay batching defers a request, and the resulting
# revert_allocate_context -> resize(shrink) can't release the scratch of a
# never-forwarded request (no finish_event), crashing every dpa=true job.
export TRTLLM_DSV4_ENABLE_SWA_SCRATCH_REUSE="${TRTLLM_DSV4_ENABLE_SWA_SCRATCH_REUSE:-0}"
echo "TRTLLM_DSV4_ENABLE_SWA_SCRATCH_REUSE: $TRTLLM_DSV4_ENABLE_SWA_SCRATCH_REUSE"

nvidia-smi

SERVER_LOG="$PWD/server.log"
Expand Down
7 changes: 7 additions & 0 deletions perf-changelog.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3378,6 +3378,13 @@
- "Add MTP speculative-decoding sibling for dsv4-fp4-mi355x-vllm (model: deepseek-ai/DeepSeek-V4-Pro) on vllm/vllm-openai-rocm:v0.22.0, per vllm-project/vllm#43385"
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/1630

- config-keys:
- dsv4-fp4-b200-trt
- dsv4-fp4-b300-trt
description:
- "Revert the non-MTP TensorRT-LLM DeepSeek-V4-Pro image to ghcr.io/semianalysisai/trtllm-deepseek-v4:feat-deepseek_v4-2dd03e6 and disable DSv4 SWA scratch reuse via TRTLLM_DSV4_ENABLE_SWA_SCRATCH_REUSE=0 in the launcher. Root cause of the prior attention-DP hangs/crashes: the V2 scheduler grows a context request's KV cache (incl. SWA scratch slots) before delay batching can defer it, so revert_allocate_context -> resize(shrink) must release scratch slots of a never-forwarded request, which has no finish_event -> crash on every dpa=true job. Disabling scratch reuse stops those slots from being allocated so the revert shrinks cleanly."
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/1636

- config-keys:
- dsv4-fp4-mi355x-sglang-mtp
description:
Expand Down