Skip to content

Enable Rust frontend (VLLM_USE_RUST_FRONTEND=1) #1634

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
chunfangamd wants to merge 3 commits into
base: main
Choose a base branch
from
+9 −0
Open

Enable Rust frontend (VLLM_USE_RUST_FRONTEND=1) #1634

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
a79476d
Enable Rust frontend (VLLM_USE_RUST_FRONTEND=1)
chunfangamd Jun 1, 2026
7d601a0
Update per-changelog
chunfangamd Jun 1, 2026
aa63214
Merge branch 'main' into chun/minimax-rust-frontend
chunfangamd Jun 2, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions benchmarks/single_node/fixed_seq_len/minimaxm2.5_fp4_mi355x.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ if [ -n "$ROCR_VISIBLE_DEVICES" ]; then
fi

export VLLM_ROCM_USE_AITER=1
export VLLM_USE_RUST_FRONTEND=1
EXTRA_VLLM_ARGS=""
# if [ "$TP" -ge 4 ]; then
# # AITER CK fused MoE kernels lack compiled tiles for N=intermediate_size/TP
Expand Down
8 changes: 8 additions & 0 deletions perf-changelog.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3395,3 +3395,11 @@
description:
- "Add DeepSeek-V4-Pro FP4 MI355X ATOM MTP3 benchmark; image rocm/atom:rocm7.2.4_ubuntu24.04_py3.12_pytorch_release_2.10.0_atom0.1.3"
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/1627

- config-keys:
- minimaxm2.5-fp4-mi355x-vllm
description:
- "Enable vLLM Rust request frontend by exporting VLLM_USE_RUST_FRONTEND=1 in benchmarks/single_node/minimaxm2.5_fp4_mi355x.sh (v0.22.0 ROCm image ships the vllm-rs binary, so the flag engages it). Environment-only change; serve flags, TP/EP, attention/kernel settings unchanged"
- "The Rust frontend replaces only the Python serving/API layer (HTTP, tokenization, scheduling glue, detokenization) and spawns the same Python EngineCore, so GPU kernels/attention/MoE GEMM/KV cache are untouched"
- "A/B sweep (28 single-node points, 1k1k + 8k1k, TP 1/2/4) vs the Python-frontend baseline (run 26696260751): throughput Pareto-neutral (peak tok/s/GPU within <1.5%, frontiers coincident) and TPOT flat (+-0.5%); TTFT improves ~8% at 1k1k and ~22% at 8k1k (every point), the expected signature of lower frontend CPU latency before first token, scaling with input length"
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/1634
Loading