feat(qwen4_exp): stream the PLE n-gram table from disk

[jason-fxz]

What

Qwen3.8-Flash-Next has a huge PLE n-gram table (47.7 GiB as fp8 in the -FP8/-NVFP4 checkpoints). Instead of preloading it into pinned host RAM, this PR reads the needed rows from disk on each forward. --ple-backend disk is the new default; pinned keeps the old behavior.

Data path

checkpoint shards → TableFile → BatchReader → dedup → pinned staging → GPU

1. Rows are read in place from the checkpoint's fp8 safetensors shards; an extent table maps row id → (file, offset). No copy, no conversion.
2. TableFile: one O_DIRECT fd per file.
3. BatchReader: each fill becomes one batched read round — io_uring at constant QD64.
4. Duplicate rows in a fill are read once and copied to every destination. No RAM cache: an on/off A/B showed zero decode difference.
5. Rows land in fixed pinned staging; in the CUDA graph, lookup is just an H2D copy + fp8→bf16 dequant.

Row ids are hashed on the host from the request's token history, so prefix hits, restores and forks need no bookkeeping. The C++ store is stateless: stage(token run) + flush(signal).

Sync

Fast path: the decode graph launches first and waits on a cuStreamWaitValue64 flag right before consuming the rows; the host fills staging while the GPU runs embedding + layer 0, then sets the flag. Fallback: fill before launch. The engine hook is one context manager on the model (forward_host_ctx), a no-op for other models.

Decode / prefill costs

• Decode: one round of ≤16 reads per lane, ~0.1–0.4 ms, hidden behind embedding + layer 0. Disk metric: 4K random-read latency.
• Prefill: all chunk rows in one pipelined round; ≈0.05–0.16 ms per token of TTFT. Disk metric: random-read IOPS.

Performance

H100 (80 GB), -NVFP4 checkpoint. Decode: AIME problems. Prefill: real mixed text (tech report, blog, distinct GSM8K problems).

	decode tok/s	prefill TTFT 1K tok	4K	16K	32K
pinned	108.23	1.33 s	1.90 s	8.21 s	15.14 s
disk	105.37 (−2.6%)	1.46 s	2.38 s	9.39 s	16.87 s

RTX PRO 6000 Blackwell (sm_120), VRAM held to 32 GiB to model a consumer card, bare NVMe. Same workloads, except the prefill filler is unique tokens (rows dedup 1.00:1, the worst case); the MoE cache holds ~11% of the experts here, so every step already waits on expert traffic and the round trip hides completely.

	decode tok/s	prefill TTFT 1K tok	4K	16K	32K
pinned	66.08	1.25 s	1.28 s	4.37 s	8.77 s
disk	66.18 (+0.2%)	1.31 s	1.49 s	5.01 s	9.95 s

Fallbacks

O_DIRECT → buffered; io_uring (Linux ≥ 5.6) → 16-thread pread pool; stream memops → launch gating; disk → pinned. Windows is planned — the OS-specific parts sit behind these seams.

[andy-yang-1]

@jason-fxz Great work! On single RTX 5090:

Metric	disk	pinned	disk vs. pinned
Decode (AIME, 12K tokens)	68.3 tok/s	68.7 tok/s	-0.5%
TTFT (4K prompt)	1.30 s	1.27 s	+2%
TTFT (16K prompt)	4.20 s	3.65 s	+15%
Resident Memory	74 GB	120 GB	46 GB less
Load Time	28 s	68 s	40 s faster

[chbornman]

Was excited to see this PR so I tested on my box:

Hardware: RTX 5080 16 GB, Ryzen 9 9900X, 96 GB DDR5 (dual channel), checkpoint on a gen5 NVMe (Samsung 9100 PRO, ext4). Arch, driver 610.57, CUDA 13.3, this branch @ ccddade, RadixArk/Qwen3.8-Flash-Next-NVFP4.

Config that's serving right now, with an 80K window:

PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
ft serve --model RadixArk/Qwen3.8-Flash-Next-NVFP4 \
  --ple-backend disk --expert-load serial \
  --attention-backend qsa_sparse --nvfp4-backend triton \
  --moe-backend offload --page-size 64 --max-running-requests 1 \
  --memory-ratio 0.94 --max-seq-len-override 81920 --kv-reserve-tokens 81920 \
  --moe-cache-size 512 --disable-moe-prefill-overlap --max-prefill-length 2048

Numbers:

• KV pool allocates 81,472 tokens (1.92 GiB), 0.73 GiB free after CUDA graphs
• Decode: ~30 tok/s, and it stays there deep into context (29.6 tok/s at 72K)
• Fed it a 72,628-token prompt: prefill in 91 s (797 tok/s), and it correctly pulled a fact buried ~52K tokens deep
• Small-context configs decode faster if you don't need the window: 36 tok/s at 4K with the cache on auto
• Steady state host RAM: ~7 GiB available of 96 GB.

Running Pi agent at 32-35 tok/s with basic computer use tests.

Two things I hit on the way that might save others time:

1. --disable-moe-prefill-overlap is the unlock for big KV on 16 GB: the 1024-slot cache floor is really 2x512 double-buffering, so this frees ~1.3 GiB (+50K tokens of KV) and cost me 2-4 tok/s of decode, since decode at this cache size is PCIe-bound anyway.
2. At --memory-ratio 0.95 the first real multi-thousand-token prompt OOMs in gdn.py _conv_prefill (160 MiB alloc, 140 MiB free) and kills the worker. But 0.94 + --max-prefill-length 2048 has been rock solid (probably issue fix(engine): account for attention workspace in cache-pool sizing #303).

Happy to run more tests on this config if useful.