[Feature Request] Possible to support Qwen3.8-Flash-Next?

[SLAPaper]

https://huggingface.co/Qwen/Qwen3.8-Flash-Next

125B-A6B MoE + 51B embedding + 4B MTP

[jason-fxz]

Support for Qwen3.8-Flash-Next is actively in development! It's currently in a private branch.

Qwen3.8-Flash-Next-NVFP4 on RTX 5090 (32GB VRAM): Benchmarks with offloaded MoE are currently hitting ~50+ tok/s.

We are currently doing some internal code review and cleanup, and will open a PR to the public repository soon!

[pastoriomarco]

Any idea of what the minimum GPU VRAM size should be given the 6B active params? (considering FP8 or NVFP4 quants)
Are you also supporting offloading the 51B N-gram on disk?

Thank you, great repo BTW!

[jason-fxz]

PR is up: #257 — NVFP4 and the official block-FP8 checkpoints, text-only. See the PR description for the launch command and 4090/5090 numbers.

[jason-fxz]

@pastoriomarco

minimum GPU VRAM size

NVFP4: 16 GB can barely boot (tiny KV cache); 24 GB serves the full 262k context.
More VRAM = larger expert cache

offloading the 51B N-gram on disk

Not in this first PR — the table sits in pinned host RAM (~111 GiB total incl. the expert banks, >= 128 GB recommended). Disk offload is possible in principle, but the table is read on every decode step, so we'd expect a significant speed cost.

[pastoriomarco]

Thank you, I expected it would fit in 12GB since it's 6B active

[Patenator]

Will FreeToken manage the nvfp4 over my 4090's 24gb vram & system's 96gb ram (so total 120gb) with the native (or a not terribly smaller) context window without dying? (On Ubuntu, so not much ram wasted)

[jason-fxz]

@Patenator, I am investigating offloading PLE to disk, with that supported, 96GB RAM will be enough. (63 GiB NVFP4 expert bank pinned in RAM)

[bruefire]

I have a working low-RAM PLE path and measurements that may be useful for the disk-offload work.

Test system: WSL2 with 125 GiB RAM visible, RTX 5090 32 GiB, and RadixArk/Qwen3.8-Flash-Next-NVFP4. The #257 layout needs about 111.14 GiB pinned (47.68 GiB PLE + 63.46 GiB expert banks) and did not fit in its intended configuration on this machine. The cumulative CUDA pin ceiling was between 90 and 104 GiB.

The alternative keeps only the 63.46 GiB NVFP4 expert bank pinned and leaves the PLE safetensors mmap-backed. It applies MADV_RANDOM to the PLE mappings, then for each decode token deduplicates the 4 KiB pages containing its 16 requested rows and issues MADV_WILLNEED for those pages before copying the rows. Grouped prefill uses the same page batching. There is no full-table preload, thread pool, io_uring backend, or new checkpoint format.

Cold PLE-only replay through the production safetensors gather, using one fixed 899-token trajectory:

policy	elapsed	p99/token	major faults	disk read
normal mmap	2.25-2.54 s	10.55-11.48 ms	3,907-3,921	973-975 MiB
MADV_RANDOM	1.06-1.08 s	4.50-4.61 ms	4,119	16.09 MiB
random + page-deduped MADV_WILLNEED	0.201-0.208 s	0.702-0.724 ms	575-689	16.09 MiB

All arms produced the same checksum. The useful row payload was 2.20 MiB and the unique 4 KiB page floor was 16.09 MiB. Normal mmap was reading about 60.5x the page floor because of readahead. Random advice alone removed the amplification but left serialized synchronous faults; issuing all exact page reads ahead of the copies was the important part.

A one-load full-model fixed-trajectory cold/warm sweep measured:

policy	cold ms/token	cold tok/s	process / PLE major faults	warm ms/token	warm tok/s
normal mmap	13.833	72.29	951 / 911	13.062	76.56
MADV_RANDOM	14.095	70.95	992 / 952	13.027	76.77
random + MADV_WILLNEED	13.395	74.65	183 / 143	13.062	76.56

Every arm was forced through the same 899-token history and matched the reference output exactly. This reduces PLE decode major faults by 84.3%, removes 56.8% of the measured cold latency penalty, and reaches about 97.5% of warm throughput on the first cold pass. These are offline fixed-trajectory numbers, not a claim of universal server throughput; diverse-prompt server traffic still needs validation.

I have the implementation, focused replay benchmark, cold-reset hooks, and tests locally (112 relevant tests passing). My branch predates the merged #257 architecture, so opening it as-is would create a large competing diff. If useful, I can either port only the mmap PLE backend/prefetch policy onto current main as a focused PR, or test the implementation you are already developing on this constrained-pin machine. Please let me know which route would be more helpful.

[Patenator]

@jason-fxz I'm in awe of your work. 天才

[KeithVal]

Bare-metal datapoint to complement @bruefire's WSL2 report: Ubuntu 24.04 (no WSL2), 128 GB DDR4 (125.7 GiB visible), RTX 5070 Ti 16 GB, RadixArk/Qwen3.8-Flash-Next-NVFP4 on current main (#257).

Result: OOM-killed at expert bank 191/192 in all three configs I tried — parallel loader; --expert-load serial; and serial + cgroup MemoryHigh forced reclaim + 24 G swap allowance + --num-tokens 4096. In the last config ~2.4 GB did get swapped out and it still died at the same point, so on bare metal this is plain RAM exhaustion (roughly 2–4 GiB short including runtime overhead), not reclaim lag. (Runs were capped at MemoryMax=124G to keep the host alive; with 125.7 GiB total that cap is within ~1.7 GiB of the physical ceiling, so uncapping only changes which OOM killer fires.)

Notably I did not hit a pin ceiling at 90–104 GiB — pinning proceeded past ~110 GiB (PLE + 191 of 192 expert banks) — so that ceiling may be WSL2-specific rather than inherent.

Happy to test the PLE disk-offload / mmap path on this machine — 16 GB VRAM + 125.7 GiB RAM seems to be exactly the boundary config this feature targets.

[dandandelion]

Since pretty much this software is targeted towards low VRAM + decently sized RAM, is it possible for the implementation to pin the huge 51B N-gram to SSD (if RAM also cannot carry the whole thing?) as it was proven not too detrimental to inference speed anyways?

Also thank you so much for this piece of gem of a software. I appreciate you guys' hardwork

[Alieno79]

Hi, congratulations on your great work!
I have a question about resource usage. Is there a feature in Freetoken similar to Llama's RPC to split the workload across two PCs? Will it be implemented?
Unfortunately, my PC only has 64GB of RAM, and if I understand correctly, I can't run QWEN 3.8 Next (it requires at least 112GB), but when connected to the second PC, I get 128GB.
Thanks

[KeithVal]

Update to my report above: this is now solved for my machine by #279 (--ple-backend mmap).

Same box (bare metal Ubuntu 24.04, 125.7 GiB RAM, RTX 5070 Ti 16 GB, RadixArk/Qwen3.8-Flash-Next-NVFP4):

• pinned (main): OOM-killed at expert bank 191/192 in all three configs I tried
• mmap (feat(qwen4-exp): mmap PLE tables from disc #279): starts in 95 s, min MemAvailable 53.7 GiB, RSS 70 GiB, 18.69 tok/s decode

That is ~48 GiB of host residency removed, matching the 47.68 GiB PLE table. So @jason-fxz's "96 GB will be enough with PLE offloaded" looks right — and on 128 GB there is a lot of headroom left over.

Full numbers and the working invocation are in #279 (comment) . Posting here too since this thread is where the constraint was diagnosed.

[jason-fxz]

The disk offload mentioned above is up: #311. The PLE table is read straight from the checkpoint files (O_DIRECT + io_uring, fixed ~21 MiB staging) — the 47.7 GiB of pinned host RAM for the table is gone, and disk becomes the default backend. The expected speed cost turned out small: 2% for decode.

@Patenator With the table off RAM the big host item left is the ~63.5 GiB expert bank, so 96 GB boxes should now fit.