test(e2e): add opt-in Qwen3.5-122B CUDA coverage

[shanyulu]

Purpose

Add repository-owned, opt-in CUDA E2E coverage for
Qwen/Qwen3.5-122B-A10B using the existing Qwen3.5/3.6 adapter family.

Scope

• CUDA
• vLLM 0.26.0 V1
• BF16
• text-only
• eager execution
• native DP4/TP1/EP4 baseline
• synchronous P2pNcclAFDConnector
• AFD 4A4F
• natural routing
• GSM8K-7 with eight-shot prompting
• eight-GPU hardware profile

The checkpoint declares num_experts=256 and num_experts_per_tok=8.

Test design

The profile is explicitly opt-in and requires:

AFD_E2E_BACKEND=gpu
AFD_E2E_LARGE_MODEL=1
AFD_GPU_E2E_MODEL=/path/to/Qwen3.5-122B-A10B
AFD_E2E_DEVICES=0,2,4,6,1,3,5,7

It never automatically downloads the approximately 234 GiB checkpoint and is
not wired into the default Buildkite PR/merge gate.

Cases:

• baseline-eager: native DP4/TP1/EP4 on devices 0,2,4,6
• afd-eager-4a4f: Attention DP4/TP1 on 0,2,4,6 and FFN DP4/TP1/EP4 on
  1,3,5,7

Correctness child environments remove benchmark-only forced-routing variables.

Validation

Exact candidate SHA: 934ccaf288eb2d40b0b5241af98df1dce31bee9f

Hardware:

• 8x NVIDIA RTX PRO 6000 Blackwell Server Edition
• 97,887 MiB/card
• Driver 595.71.05
• CUDA 13.2
• PCIe topology without NVLink

Software:

• Python 3.12.3
• PyTorch 2.11.0+cu130
• vLLM 0.26.0
• Transformers 5.8.1

Results:

• Native GSM8K strict match: 7/7
• AFD GSM8K strict match: 7/7
• Native and AFD runner exit code: 0
• No repository-runner cleanup failure
• No residual AFD/vLLM process
• Ports released
• GPU memory returned to 0 MiB
• CPU-safe suite: 795 passed / 2 upstream environment-capability failures / 50 skipped on Anaconda Python 3.12.3; the same two pidfd-dependent failures reproduce on pristine upstream/main. All remaining CPU-safe tests pass
• Complete pre-commit passed

Runtime observation: The AFD run emitted vLLM-internal worker shutdown
escalation and Python resource_tracker warnings during process teardown. The
repository E2E runner did not trigger its forced-SIGKILL failure path, exited
successfully, released all ports and GPU allocations, and left no residual
AFD/vLLM processes.

Performance characterization

A reviewer-requested, topology-scoped characterization was run on
934ccaf288eb2d40b0b5241af98df1dce31bee9f using the same local
Qwen3.5-122B-A10B BF16 checkpoint.

For resource parity, the primary comparison uses the same eight physical GPUs:

• Native: DP8/TP1/EP8
• AFD: 4A4F; Attention DP4/TP1 and FFN DP4/TP1/EP4

Workload:

• vLLM 0.26.0 V1
• eager execution
• natural routing
• random ISL 1024 / OSL 128
• 1024 requests per run
• request rate 5/s
• max concurrency 32
• temperature 0
• five formal repetitions after warmup

Metric	Native DP8/EP8	AFD 4A4F	Delta
Request throughput	2.4047 ± 0.0195 req/s	2.4295 ± 0.0092 req/s	+1.03%
Output throughput	307.80 ± 2.50 tok/s	310.98 ± 1.18 tok/s	+1.03%
TTFT	371.29 ± 13.39 ms	404.24 ± 6.16 ms	+8.87%
TPOT	100.13 ± 0.78 ms	98.88 ± 0.38 ms	-1.25%
ITL	100.13 ± 0.78 ms	98.88 ± 0.38 ms	-1.25%
E2E latency	13088.36 ± 109.82 ms	12962.50 ± 49.08 ms	-0.96%

All five formal runs on each topology completed 1024/1024 requests
successfully with zero request failures.

For this workload, steady-state throughput and decode latency are effectively
at parity. The main observed difference is approximately 9% higher TTFT for
AFD.

These measurements characterize this single-host PCIe topology, checkpoint,
and workload only and are not a general performance guarantee.

Limitations

• Requires eight CUDA devices with sufficient memory.
• Manual/opt-in only.
• Eager-only.
• Graph and DBO remain outside this eager-only large-model profile. [Bug]: FFN graph cache replays during eager prefill in FULL_DECODE_ONLY #261 was subsequently resolved by fix: gate GPU FFN graph replay on Attention run mode #270; this PR does not add graph performance coverage.
• Async, multi-node, quantized, and multimodal paths are not covered.

Relationship

• [RFC] Experimental Qwen3.5 MoE AFD support: CUDA correctness baseline #179 — early Qwen3.5 CUDA correctness context
• Add Qwen3.6 MoE CUDA AFD implementation support #181 — Qwen3.5/3.6 CUDA implementation
• test(e2e): add Qwen3.6 MoE CUDA coverage #256 — existing Qwen3.6 adapter-family CUDA E2E
• [Bug]: FFN graph cache replays during eager prefill in FULL_DECODE_ONLY #261 — resolved graph replay correctness issue
• fix: gate GPU FFN graph replay on Attention run mode #270 — merged fix for [Bug]: FFN graph cache replays during eager prefill in FULL_DECODE_ONLY #261

[jiangkuaixue123]

Thanks for adding the Qwen3.5 coverage! Do you have any performance data you could share, such as throughput and latency comparisons between the native baseline and AFD 4A4F?

[shanyulu]

Thanks for asking. I reran this as an equal-hardware comparison using the same eight physical GPUs for both configurations:

• Native: DP8/TP1/EP8
• AFD: 4A4F, with Attention DP4/TP1 and FFN DP4/TP1/EP4

The workload was Qwen3.5-122B-A10B in BF16 with vLLM 0.26.0 V1, eager execution, natural routing, temperature=0, random ISL=1024 / OSL=128, 1024 requests, request rate 5/s, and max concurrency 32. The values below are mean ± std over five formal runs after warmup.

Metric	Native	AFD 4A4F	Delta
Request throughput	2.4047 ± 0.0195 req/s	2.4295 ± 0.0092 req/s	+1.03%
Output throughput	307.80 ± 2.50 tok/s	310.98 ± 1.18 tok/s	+1.03%
TTFT	371.29 ± 13.39 ms	404.24 ± 6.16 ms	+8.87%
TPOT	100.13 ± 0.78 ms	98.88 ± 0.38 ms	-1.25%
ITL	100.13 ± 0.78 ms	98.88 ± 0.38 ms	-1.25%
E2E latency	13088.36 ± 109.82 ms	12962.50 ± 49.08 ms	-0.96%

All five runs completed 1024/1024 requests successfully.

For this workload I would characterize steady-state throughput and decode latency as effectively at parity rather than claiming AFD is faster. The main observed difference is ~9% higher TTFT for AFD on this single-host PCIe topology.

I also ran the repository's afd_balanced benchmark-only routing control separately, but since it changes routing semantics I am not using it as correctness or headline production-performance evidence.

These numbers are scoped to this checkpoint, workload, and topology rather than a general performance guarantee.