Add Qwen3.5 hybrid linear-attention MoE support (122B-A10B-FP8, 397B-A17B-FP8)

[yuxuandexter]

Adds support for the Qwen3.5 hybrid linear-attention MoE models — 122B-A10B-FP8 and
397B-A17B-FP8 — together with the presets needed to reproduce a correctness alignment
and a two-node AFD throughput run. FP8 only; the NVFP4 checkpoints are still being
worked on and are deliberately left out.

Six commits, ordered so each stands on its own:

	commit	what
1	Add Qwen3.5 hybrid linear-attention MoE support	the model, its kernels, and the AFD plumbing
2	Size the TRT-LLM capture page table with a ceiling division	standalone bug fix, reachable without Qwen3.5
3	Let the launcher scripts drive Qwen3.5 alignment runs	shared shell entrypoints — the only commit touching scripts other models use
4	Add Qwen3.5 FP8 correctness alignment presets	two presets over a shared driver
5	Add Qwen3.5 two-node AFD throughput presets	two presets over a shared no-TP driver
6	Document running Qwen3.5 122B and 397B	README and script map

What the model needed

Qwen3.5 interleaves gated DeltaNet with full attention at 3:1, so it needs a KV pool
holding recurrent state and paged KV side by side. The GDN decode path is a causal
conv1d that keeps its taps in scalar registers through a KERNEL_WIDTH cascade, plus a
recurrent gated-delta kernel whose grid puts the value block on the fastest-varying axis
so consecutive CTAs walk one request's state contiguously. 397B routes 512 experts at
top-k 10 against 122B's 256 at 8, which is what forced MegaMoE routing, weight loading
and the AFD expert plans to stop assuming the smaller shape.

The AFD workers gain --afd-async-pending-steps. Both loops previously retired the
oldest handle at a hardcoded len(pending) > 2; that constant decides how far
cudaGraphLaunch runs ahead of the CPU wait on the previous step's D2H copy, and the
right depth differs by model and topology.

The page-table fix (commit 2)

init_capture_graph sized the captured page table at max_seq_len // page_size while
replay built its table with a ceiling division. Whenever max_seq_len is not a multiple
of page_size the two disagree by one column, and replay fails on a sequence long
enough to need that last page:

RuntimeError: The expanded size of the tensor (132) must match the existing size (133)
at non-singleton dimension 1

The floor is wrong rather than merely inconsistent — an 8505-token sequence occupies 133
pages of 64. Both sites now route through one _num_pages helper, and an over-wide
replay table is rejected explicitly instead of being silently narrowed. This is reachable
on any model whose maximum sequence length is not page-aligned, Qwen3.5 or not.

Reproducing

Correctness, against a vLLM reference on the same weights:

./scripts/validate/qwen35_122b_a10b_fp8_fastafd_alignment.sh
./scripts/validate/qwen35_397b_a17b_fp8_fastafd_alignment.sh

Throughput, two nodes:

MODEL_PATH=/path/to/Qwen3.5-397B-A17B-FP8 RUN_VLLM_ALIGNMENT=0 NSYS=0 \
bash scripts/experiments/afd/qwen35/run_afd_qwen35_397b_a17b_fp8_8k_b320_2node_mb2_nsys_alignment.sh

Unlike the 235B and MiniMax presets, the Qwen3.5 ones fix their topology instead of
sizing it from the cluster: four attention workers (DP 4, no tensor parallelism) and four
expert workers (DP 4, EP 4). They ignore AFD_TOTAL_NODES and refuse to start on any
other worker count, so that a retained decode window contains no tensor-parallel
all-reduce traffic. AFD_NUM_MB has no default in the driver — 1 is the serialized
baseline and 2 is the ping-pong overlap, and silently picking either makes a throughput
comparison meaningless.

Verification

• python -m compileall python/minisgl passes.
• The alignment presets were restructured (the 397B one used to chain through the 122B
  one, into a driver named for an NVFP4 checkpoint). Proven behaviour-identical by
  stubbing the validator and diffing the final argv and environment of the old and new
  chains: 79 lines match exactly, per model.
• The throughput presets were checked against the driver the measurements were taken
  with, via its QWEN35_DRY_RUN=1 path: all 22 derived configuration values match at
  both batch sizes, including the prompt-pool SHA-256, the prefill-step bracket, the
  auto-raised token count and the Nsight window.
• Every repository path named in README.md and scripts/README.md exists on the branch.

Measured on two GB200 nodes, attention DP4/TP1 and MLP EP4/TP1, against a vLLM reference
on the same checkpoints. No throughput figures are stated in the README, since none have
been published yet.

Not included

Tests, the baseline/ study harness, NVFP4 presets, and development-only validators —
they depend on paths and checkpoints outside this repository.

[jiangkuaixue123]

Thanks for the work. I have a question about pipeline balance for hybrid-attention models such as Qwen3.5.

Linear-attention and full-attention layers can have quite different compute times on the attention group (AG), so the balance between AG work and expert-group (EG) MoE work is inherently layer-dependent. For example, suppose the microbatch size and topology make the AG computation of a linear-attention layer roughly balanced with its corresponding MoE computation on the EG. At a full-attention layer, the attention-side time and the FFN/expert-side time may differ substantially, leaving one side idle and introducing pipeline bubbles. This could waste resources and reduce the throughput benefit of AFD.

How do you view or address this layer-wise imbalance? Does the current implementation provide any mitigation, such as per-layer scheduling or microbatching, different overlap depths, or workload-aware partitioning? If not, have you measured how much the full-attention layers contribute to pipeline bubbles in the 122B or 397B runs, and is this something you plan to address in future work?