ft bench bw -o writes a profile that hybrid never loads (silent fetch cap of 1)

[benwilson]

Summary

ft bench bw -o /some/path.json writes a profile that ft serve --moe-backend hybrid never finds. The serve logs one WARNING and starts anyway with a fixed fetch cap of 1 — a different backend configuration from the one the bench recommended — unless FREETOKEN_BENCHBW_PATH is also exported.

Environment: FreeToken git 4b94bdc (HEAD 2026-08-30, no local patches), pip install --no-build-isolation "freetoken[accel] @ git+…@4b94bdc"; torch 2.11.0+cu130, CUDA 13.0, driver 595.71.05; RTX 3090 24 GB (sm_86), PCIe 4.0 x16 (ft bench bw: H2D 23.8 GB/s), 32-core host, 503 GiB RAM; dealignai/Qwen3.8-Flash-Next-ABLITERATED-NVFP4 @ be794b99 (compressed-tensors NVFP4, same layout as the RadixArk stock checkpoint).

ft serve --max-seq-len-override 262144 --kv-reserve-tokens 262144 --moe-backend offload --moe-cache-auto --nvfp4-backend triton --max-running-requests 1 --max-prefill-length 8192 --sampling-defaults model --reasoning-parser qwen3 --tool-call-parser qwen3_coder (the serve as above but --moe-backend hybrid)

Repro

1. ft bench bw --dtype nvfp4 -o /workspace/bench-bw.json → nvfp4 … 2.62x hybrid; overlapped: CPU-MoE 43.4 + PCIe 21.6 GB/s -> hybrid fetches 33.2% of misses, saved: /workspace/bench-bw.json
2. ft serve … --moe-backend hybrid → WARNING --moe-hybrid-max-fetch auto: no usable ft bench bw profile for 'nvfp4' experts; using a fixed fetch cap of 1
3. FREETOKEN_BENCHBW_PATH=/workspace/bench-bw.json ft serve … --moe-backend hybrid → INFO --moe-hybrid-max-fetch auto: fetching 33.2% of each decode step's expert misses over PCIe … the rest on the CPU

moe/bench_profile.py looks at FREETOKEN_BENCHBW_PATH, then benchbw/<gpu_uuid>.json, then the legacy benchbw.json — so -o opts the profile out of discovery, and nothing at bw time says so.

Measured consequence

Short-prompt decode on this box: offload 24.5 tok/s, hybrid with the profile 31.7, hybrid with the silent fetch cap 28.3–30.5 — close enough that a user would never notice they are running the un-benchmarked configuration; at long context the cap-1 run was the unstable one (9.5 / 29.1 tok/s on two runs vs 12.4–13.1 with the profile).

Ask (any of these; happy to PR whichever fits)

• bw -o prints the export FREETOKEN_BENCHBW_PATH=… line (or also writes the default location), or
• --moe-backend hybrid refuses to start without a usable profile unless --moe-hybrid-max-fetch is given explicitly, or
• a --benchbw-path flag on ft serve so the profile is part of the command line.