Summary
At long context, the decode rate a client actually receives is about half of what the scheduler's Decode batch … gen throughput (token/s) lines report for the same request. At short context the two agree within 5%. Whatever is spent per decode step outside the batch grows with context, and by ~75k tokens it dominates.
Environment: FreeToken git 4b94bdc (HEAD 2026-08-30, no local patches), pip install --no-build-isolation "freetoken[accel] @ git+…@4b94bdc"; torch 2.11.0+cu130, CUDA 13.0, driver 595.71.05; RTX 3090 24 GB (sm_86), PCIe 4.0 x16 (ft bench bw: H2D 23.8 GB/s), 32-core host, 503 GiB RAM; dealignai/Qwen3.8-Flash-Next-ABLITERATED-NVFP4 @ be794b99 (compressed-tensors NVFP4, same layout as the RadixArk stock checkpoint).
ft serve --max-seq-len-override 262144 --kv-reserve-tokens 262144 --moe-backend offload --moe-cache-auto --nvfp4-backend triton --max-running-requests 1 --max-prefill-length 8192 --sampling-defaults model --reasoning-parser qwen3 --tool-call-parser qwen3_coder
Measurement
Streaming POST /v1/chat/completions, temperature: 0, chat_template_kwargs: {"enable_thinking": false}, max_tokens: 400, one request at a time. Each prompt is prefixed with a fresh random nonce so the radix cache cannot serve any prefix. Decode rate = content tokens received ÷ (time of last token − time of first token), so prefill/TTFT is excluded by construction. Three runs per cell.
| prompt |
wall-clock decode (client) |
scheduler gen throughput (same runs, mean of the Decode batch lines) |
| ~70 tokens |
24.5–24.7 tok/s |
23.0 |
| 75,665 tokens |
11.4–13.0 tok/s (median 12.6) |
21.3 (min 0.5 on the first line, max 25.5) |
Same probe with --moe-backend hybrid (bench profile loaded, 33% fetch): short 31.6–31.9 vs 29.4; long 12.4–13.1 vs 26.5. So the gap is backend-independent and hides the whole MoE-backend difference at depth.
TTFT for the 75k prompt was ~70 s on every run (≈1,075 tok/s chunked prefill), consistent with the Prefill batch lines — prefill is not where the time goes.
What it is not
- not the radix cache (nonce'd prompts,
#cached-token: 0 on every prefill line)
- not prefill (excluded from the client number; TTFT matches the prefill lines)
- not the MoE backend (offload and hybrid show the same ratio)
- not the stream path per se (the short-prompt stream is fine at 24–32 tok/s)
Ask
Something per decode step scales with context and is not inside the timed decode batch: page/radix bookkeeping, the qsa_sparse block top-k selection, the hybrid_radix cache upkeep, detokenise/frontend hand-off … I have not profiled it. If a maintainer can name the suspect I can run py-spy/the torch profiler on this box during a 75k-context decode and post the trace. Related: #281 (a harness should report both numbers — the counter overstates end-to-end at depth).
Summary
At long context, the decode rate a client actually receives is about half of what the scheduler's
Decode batch … gen throughput (token/s)lines report for the same request. At short context the two agree within 5%. Whatever is spent per decode step outside the batch grows with context, and by ~75k tokens it dominates.Environment: FreeToken git
4b94bdc(HEAD 2026-08-30, no local patches),pip install --no-build-isolation "freetoken[accel] @ git+…@4b94bdc"; torch 2.11.0+cu130, CUDA 13.0, driver 595.71.05; RTX 3090 24 GB (sm_86), PCIe 4.0 x16 (ft bench bw: H2D 23.8 GB/s), 32-core host, 503 GiB RAM;dealignai/Qwen3.8-Flash-Next-ABLITERATED-NVFP4@be794b99(compressed-tensors NVFP4, same layout as the RadixArk stock checkpoint).ft serve --max-seq-len-override 262144 --kv-reserve-tokens 262144 --moe-backend offload --moe-cache-auto --nvfp4-backend triton --max-running-requests 1 --max-prefill-length 8192 --sampling-defaults model --reasoning-parser qwen3 --tool-call-parser qwen3_coderMeasurement
Streaming
POST /v1/chat/completions,temperature: 0,chat_template_kwargs: {"enable_thinking": false},max_tokens: 400, one request at a time. Each prompt is prefixed with a fresh random nonce so the radix cache cannot serve any prefix. Decode rate = content tokens received ÷ (time of last token − time of first token), so prefill/TTFT is excluded by construction. Three runs per cell.gen throughput(same runs, mean of theDecode batchlines)Same probe with
--moe-backend hybrid(bench profile loaded, 33% fetch): short 31.6–31.9 vs 29.4; long 12.4–13.1 vs 26.5. So the gap is backend-independent and hides the whole MoE-backend difference at depth.TTFT for the 75k prompt was ~70 s on every run (≈1,075 tok/s chunked prefill), consistent with the
Prefill batchlines — prefill is not where the time goes.What it is not
#cached-token: 0on every prefill line)Ask
Something per decode step scales with context and is not inside the timed decode batch: page/radix bookkeeping, the qsa_sparse block top-k selection, the hybrid_radix cache upkeep, detokenise/frontend hand-off … I have not profiled it. If a maintainer can name the suspect I can run
py-spy/the torch profiler on this box during a 75k-context decode and post the trace. Related: #281 (a harness should report both numbers — the counter overstates end-to-end at depth).