docs(llmkube): adopt AITER as the pinned attention backend by Tanguille · Pull Request #4869 · Tanguille/cluster

Tanguille · 2026-09-04T18:57:52Z

Promotes the time-boxed 09-01 AITER soak to a permanent decision. Comment-only: the rendered manifest is semantically identical (verified by parsing both revisions).

Why adopt, given parity

AITER is kept for determinism, not speed. Without the patch a KV connector forces TRITON_ATTN; image 0d07767 already reorders its backend preference list, so the current selection is fallout from auto-selection rather than a choice.

backend	decode M=2	decode M=3
AITER v0.1.19	55.31	64.07
TRITON_ATTN	55.72	64.88

Parity, both inside the within-arm spread (09-02, warm).

The soak's original exit criterion (short-context TTFT/ITL either side of 09-01) was never measured and is now unmeasurable — unrelated config changes land on both sides. The comment now says so rather than leaving the block looking unevaluated.

AITER v0.1.19 -> v0.1.21.post1

Image 0d07767 moves AITER two minor versions; Triton, PyTorch, ROCm base and flash-attention are all unchanged.

M	v0.1.19	v0.1.21.post1	delta	Welch t
2	55.31 ±0.68	57.47 ±0.14	+3.9%	6.19
3	64.07 ±1.12	66.71 ±0.35	+4.1%	4.50

4 warm reps each, concsweep.py. Uncached prefill unchanged (M<=1, Welch t 0.87, median -0.5%).

Combined AITER+vLLM delta — the Triton control arm was not re-run, so none of it is attributed to AITER alone.

TurboQuant

Not a candidate while the KV offload connector is on, and the blocker is not the 09-01 gfx1201 CK FMHA segfault: TurboQuant declares only KVCacheLayout.LBNHC, OffloadingConnector requires LBHNC. Using it means dropping --kv-transfer-config, which measured a halving of single-stream decode (31 -> 15.5 tok/s).

Inferred from declared layouts — vLLM logs actual rejection reasons at DEBUG only.

Stale comments refreshed

qwen36-27b-vllm.yaml was deleted in 5770d69; four comments still pointed at it
SGLang retired 2026-08-16; referenced as a live sibling
v0.28.0 (2026-08-26) superseded v0.27.1 as newest release

Verification

Both patches confirmed firing, not just installed: supports_kv_connector -> True and LDS_CAPACITY_ELEMENTS 32768 -> 39321
Neither is redundant upstream: LDS is still 32768 in rdna_hybrid_w4a16.py:66, and no supports_kv_connector override exists for the AITER backend
kustomize build passes; both revisions parse to identical structures

Summary by CodeRabbit

Documentation
- Updated comments to clarify the historical SGLang baseline and retirement details.
- Documented the nightly vLLM image version and gfx1201 support status.
- Revised notes on the adopted AITER KV-connector patch, related version updates, and verification.
- Clarified the blocks_per_chunk trial mechanism and evaluation metric.

Promotes the time-boxed 09-01 soak to a permanent decision. AITER measures parity with TRITON_ATTN (09-02 warm: M=2 55.31 vs 55.72, M=3 64.07 vs 64.88, inside the within-arm spread), so it is kept for DETERMINISM, not speed: without the patch a KV connector forces TRITON_ATTN, and image 0d07767 already reorders its preference list. The soak's short-context exit criterion was never measured and is now unmeasurable, which the comment now states outright rather than leaving the block looking unevaluated. Records the AITER v0.1.19 -> v0.1.21.post1 decode gain measured 2026-09-04 (M=2 55.31 -> 57.47, Welch t 6.19; M=3 64.07 -> 66.71, t 4.50; 4 warm reps each; uncached prefill unchanged, t 0.87) and notes it is a combined AITER+vLLM delta with no Triton control arm re-run. Documents why TURBOQUANT is not a candidate: it declares only KVCacheLayout LBNHC while OffloadingConnector requires LBHNC, so the blocker is the KV offload connector, not the 09-01 gfx1201 CK FMHA segfault. Refreshes stale comments: qwen36-27b-vllm.yaml was deleted in 5770d69, SGLang was retired 2026-08-16, and v0.28.0 has superseded v0.27.1 as the newest release. Comment-only -- the rendered manifest is semantically identical.

deepsource-io · 2026-09-04T18:58:29Z

DeepSource Code Review

We reviewed changes in 769e684...36880cc on this pull request. Below is the summary for the review, and you can see the individual issues we found as inline review comments.

See full review on DeepSource ↗

PR Report Card

Overall Grade	Security Reliability Complexity Hygiene

Code Review Summary

Analyzer	Status	Updated (UTC)	Details
JavaScript		Sep 4, 2026 6:58p.m.	Review ↗
Shell		Sep 4, 2026 6:58p.m.	Review ↗

Important

AI Review is run only on demand for your team. We're only showing results of static analysis review right now. To trigger AI Review, comment @deepsourcebot review on this thread.

coderabbitai · 2026-09-04T18:58:46Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Team

Run ID: c55c3695-423a-4cac-8e66-42098ffa7df4

📥 Commits

Reviewing files that changed from the base of the PR and between 769e684 and 36880cc.

📒 Files selected for processing (1)

kubernetes/apps/ai/llmkube/models/qwen38-27b-vllm.yaml

Included review availability: Your plan provides up to 1 included review per hour; 0 remain after this review.

📝 Walkthrough

Walkthrough

The pull request updates comments in the qwen38-27b-vllm.yaml Kubernetes manifest. It documents historical SGLang provenance, the nightly vLLM image, adopted AITER behavior, and benchmark interpretation. No declarations or resource values change.

Changes

Qwen vLLM manifest documentation

Layer / File(s)	Summary
Historical provenance and inherited settings `kubernetes/apps/ai/llmkube/models/qwen38-27b-vllm.yaml`	Comments identify the retired SGLang baseline and settings carried over from the replaced qwen36 configuration.
Runtime image and AITER status `kubernetes/apps/ai/llmkube/models/qwen38-27b-vllm.yaml`	Comments document the nightly vLLM image and the adopted AITER KV-connector patch, including its rationale, version changes, and verification command.
Benchmark and tuning notes `kubernetes/apps/ai/llmkube/models/qwen38-27b-vllm.yaml`	Comments clarify warmup controls and state that `blocks_per_chunk` trials must use total lookup seconds per request as the evaluation metric.

Estimated code review effort: 1 (Trivial) | ~5 minutes

Merge Risk: ⚪ Minimal · up to 36880

This updates documentation comments for the Qwen vLLM manifest without changing deployed configuration or rendered manifest behavior, so no current merge-blocking production risk remains.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly identifies the main change: documenting the adoption of AITER as the pinned attention backend for llmkube. It is concise and specific.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0…
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch feat/vllm-pin-aiter-backend

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

github-actions · 2026-09-04T18:58:48Z

AI Automated Review

Analysis engine: omniroute@http://litellm.ai.svc.cluster.local/v1 (openai)

Approve: clean documentation-only update formalizing the AITER attention backend adoption and refreshing retired engine references.

Tool Harness Findings

read_file on kubernetes/apps/ai/llmkube/models/qwen38-27b-vllm.yaml verified all diff lines are comment changes with zero runtime configuration or manifest modifications.
read_file on kubernetes/apps/ai/llmkube/models/resources/zz_aiter_kvconn_impl.py verified the monkey-patch behavior aligns with the documented supports_kv_connector log validation commands.
git_log confirmed commit 5770d6902 retired qwen36-27b-vllm.yaml as cited in the updated comments.

Sources

kubernetes/apps/ai/llmkube/models/qwen38-27b-vllm.yaml
kubernetes/apps/ai/llmkube/models/resources/zz_aiter_kvconn_impl.py
Git history for 5770d6902

tanguille-cluster Bot added the area/kubernetes label Sep 4, 2026

Navigation Menu

docs(llmkube): adopt AITER as the pinned attention backend - #4869

docs(llmkube): adopt AITER as the pinned attention backend#4869
Tanguille wants to merge 1 commit into
mainfrom
feat/vllm-pin-aiter-backend

Tanguille commented Sep 4, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

deepsource-io Bot commented Sep 4, 2026 •

edited

Loading

Uh oh!

coderabbitai Bot commented Sep 4, 2026 •

edited

Loading

Walkthrough

Changes

Uh oh!

github-actions Bot commented Sep 4, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

Tanguille commented Sep 4, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why adopt, given parity

AITER v0.1.19 -> v0.1.21.post1

TurboQuant

Stale comments refreshed

Verification

Summary by CodeRabbit

Uh oh!

deepsource-io Bot commented Sep 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

DeepSource Code Review

PR Report Card

Code Review Summary

Uh oh!

coderabbitai Bot commented Sep 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Uh oh!

github-actions Bot commented Sep 4, 2026

AI Automated Review

Tool Harness Findings

Sources

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Tanguille commented Sep 4, 2026 •

edited by coderabbitai Bot

Loading

deepsource-io Bot commented Sep 4, 2026 •

edited

Loading

coderabbitai Bot commented Sep 4, 2026 •

edited

Loading