[Windows] Qwen3.6-35B-A3B-NVFP4 expert loading crashes — torch_cpu.dll access violation (0xc0000005)

[huangxianzhan]

[Windows] Qwen3.6-35B-A3B-NVFP4 expert loading crashes — torch_cpu.dll access violation (0xc0000005)

Environment

• OS: Windows 11, x86_64
• GPU: NVIDIA GeForce RTX 5080 Laptop (16 GB VRAM, SM120), driver 610.88
• RAM: 32 GB
• FreeToken: 0.1.2+g816c324d0 (engine wheel freetoken-0.1.2+g816c324d0-cp312-cp312-win_amd64.whl + freetoken_kernel_cache-0.1.2+cu130.g816c324d0-py3-none-win_amd64.whl, installed via engine/install.ps1)
• Python: 3.12.13 (managed by uv), torch 2.11.0 cu130, installed per install.ps1 defaults
• Model: nvidia/Qwen3.6-35B-A3B-NVFP4 (23.46 GB, 3 shards, downloaded from HF and verified locally)

Bug 1 (main): real NVFP4 expert loading crashes the scheduler worker

ft serve --model-path "E:\models\Qwen3.6-35B-A3B-NVFP4" --host 127.0.0.1 --port 1919

Log:

Loading mixed-fp8 weights: 100%|██████████| 3/3 [00:03<00:00, 1.07it/s]
INFO  --moe-cpu-layers auto: banks 16.93 GiB > pin budget 12.57 GiB; locking 11 head+tail MoE layers for CPU decode ([0, 1, 2, 3, 4, 5, 35, 36, 37, 38, 39])
INFO  expert banks: slow path (serial build)
Loading Qwen3.5 NVFP4 experts:   0%|          | 0/3 ...
Loading Qwen3.5 NVFP4 experts:  33%|███▎      | 1/3 [00:08<00:16, 8.39s/it]
ERROR Backend supervisor: backend worker freetoken-TP0-scheduler exited during load
ERROR Backend worker is gone and cannot be restarted; stopping the API server

Reproduced with all of these flag combinations (same crash point, ~1/3 through expert loading, i.e. on shard 2):

1. defaults (moe_backend=auto → offload, nvfp4_backend=triton)
2. --moe-backend offload --moe-cpu-layers 0
3. --moe-backend offload --nvfp4-backend marlin --expert-load parallel
   (note: parallel O_DIRECT reader unsupported on this platform (no os.O_DIRECT/preadv) -> serial build)

Windows Application event log (twice, matching both runs):

Faulting application: python.exe (3.12.13)
Faulting module: torch_cpu.dll  (in FreeToken venv site-packages/torch/lib)
Exception code: 0xc0000005  (access violation)

Crashes are on the CPU path (torch_cpu.dll), not a CUDA allocation error. 18+ GB RAM was free at crash time, so this is not a host OOM.

ft serve exits with code 15.

Bug 2 (minor): dummy-weight mode then fails with CUDA OOM allocating the MoE cache

ft serve --model-path <same> --dummy-weight

• /v1/models responds correctly (server comes up), but scheduling allocation fails:

INFO  --moe-cache-auto resolved moe_cache_size=5364 num_pages=8251 (prefill_overlap=False)
ERROR Backend supervisor: AcceleratorError: CUDA error: out of memory
  File "python/freetoken/moe/offload_cache.py", line 158, in OffloadMoeCache.__post_init__
torch.AcceleratorError: CUDA error: out of memory (cudaErrorMemoryAllocation)

16 GB VRAM, --memory-ratio default 0.9 → 5364 expert-cache slots could not be allocated. A 16 GB consumer card with a 35B-A3B-class model is exactly the advertised "offload" use case, so sizing here feels too aggressive or the reservation math doesn't account for the resident layers.

Expected behavior

• FreeToken (Windows, official win_amd64 wheels) should load Qwen3.6-35B-A3B-NVFP4 on a 16 GB RTX 5080 Laptop with experts in RAM + LRU exchange to VRAM (the README's edge-offload scenario), and serve /v1/chat/completions.

Extra info

• ft bench bw ran fine and saved a profile under ~/.cache/freetoken/benchbw/.
• Both engine wheels installed cleanly and ft --version works (0.1.2+g816c324d0).
• Happy to provide full serve*.log files or run any diagnostic commands.