Summary
With thinking on, temperature: 0 does not give reproducible output: three identical requests produced three different reasoning traces. With thinking off the same prompt is byte-identical across runs. Explicit sampling parameters are honoured (so this is not --sampling-defaults model overriding the request); it looks like a nondeterministic kernel flipping near-tied argmax picks, which content-heavy reasoning text exposes and peaked code output does not.
Environment: FreeToken git 4b94bdc (HEAD 2026-08-30, no local patches), pip install --no-build-isolation "freetoken[accel] @ git+…@4b94bdc"; torch 2.11.0+cu130, CUDA 13.0, driver 595.71.05; RTX 3090 24 GB (sm_86), PCIe 4.0 x16 (ft bench bw: H2D 23.8 GB/s), 32-core host, 503 GiB RAM; dealignai/Qwen3.8-Flash-Next-ABLITERATED-NVFP4 @ be794b99 (compressed-tensors NVFP4, same layout as the RadixArk stock checkpoint).
ft serve --max-seq-len-override 262144 --kv-reserve-tokens 262144 --moe-backend offload --moe-cache-auto --nvfp4-backend triton --max-running-requests 1 --max-prefill-length 8192 --sampling-defaults model --reasoning-parser qwen3 --tool-call-parser qwen3_coder
Repro
Prompt: Write a Python function to reverse a linked list. Output only code.
Request: {"temperature": 0.0, "max_tokens": 4096} (thinking on = template default), three times in a row, one at a time.
| run |
completion tokens |
reasoning sha1 (first 10) |
content sha1 |
| 1 |
206 |
7f4e39e460 |
1c6a96da5e |
| 2 |
218 |
bb6999e009 |
1c6a96da5e |
| 3 |
288 |
10921af11d |
df8677eeec |
First divergence between runs 1 and 2 is at character 210 of the reasoning, on a near-tie:
A: …nal only code no explanation. Could include class? Function reverse linked list. Need maybe iterativ…
B: …nal only code no explanation. Could include class? Function to reverse linked list. Need maybe itera…
Controls, same prompt, max_tokens: 1200, chat_template_kwargs: {"enable_thinking": false}, two runs each:
| params |
result |
temperature 0 |
IDENTICAL (91 tokens, same hash) |
temperature 0, top_k 1 |
IDENTICAL |
temperature 0, seed 7 |
IDENTICAL |
no params (--sampling-defaults model → temp 1.0/top_k 20/top_p 0.95) |
DIFFERENT — as documented |
Why it matters
Anyone using temp-0 for evals or regression tests on a thinking model gets a different trace (and sometimes a different answer: run 3 above) per run, with nothing in the response saying so. usage also carries no reasoning_tokens, so a truncated-by-cap reasoning run and a real answer look alike.
Ask
Which kernel is nondeterministic here — the offload MoE combine, the Triton NVFP4 dequant path, or the qsa_sparse attention? A --deterministic mode (accepting a throughput cost) would be ideal; failing that, a docs note that greedy ≠ reproducible with thinking on. Related: #223 (sampling-param validation), #281 (a regression harness needs to know this).
Summary
With thinking on,
temperature: 0does not give reproducible output: three identical requests produced three different reasoning traces. With thinking off the same prompt is byte-identical across runs. Explicit sampling parameters are honoured (so this is not--sampling-defaults modeloverriding the request); it looks like a nondeterministic kernel flipping near-tied argmax picks, which content-heavy reasoning text exposes and peaked code output does not.Environment: FreeToken git
4b94bdc(HEAD 2026-08-30, no local patches),pip install --no-build-isolation "freetoken[accel] @ git+…@4b94bdc"; torch 2.11.0+cu130, CUDA 13.0, driver 595.71.05; RTX 3090 24 GB (sm_86), PCIe 4.0 x16 (ft bench bw: H2D 23.8 GB/s), 32-core host, 503 GiB RAM;dealignai/Qwen3.8-Flash-Next-ABLITERATED-NVFP4@be794b99(compressed-tensors NVFP4, same layout as the RadixArk stock checkpoint).ft serve --max-seq-len-override 262144 --kv-reserve-tokens 262144 --moe-backend offload --moe-cache-auto --nvfp4-backend triton --max-running-requests 1 --max-prefill-length 8192 --sampling-defaults model --reasoning-parser qwen3 --tool-call-parser qwen3_coderRepro
Prompt:
Write a Python function to reverse a linked list. Output only code.Request:
{"temperature": 0.0, "max_tokens": 4096}(thinking on = template default), three times in a row, one at a time.First divergence between runs 1 and 2 is at character 210 of the reasoning, on a near-tie:
Controls, same prompt,
max_tokens: 1200,chat_template_kwargs: {"enable_thinking": false}, two runs each:temperature 0temperature 0, top_k 1temperature 0, seed 7--sampling-defaults model→ temp 1.0/top_k 20/top_p 0.95)Why it matters
Anyone using temp-0 for evals or regression tests on a thinking model gets a different trace (and sometimes a different answer: run 3 above) per run, with nothing in the response saying so.
usagealso carries noreasoning_tokens, so a truncated-by-cap reasoning run and a real answer look alike.Ask
Which kernel is nondeterministic here — the offload MoE combine, the Triton NVFP4 dequant path, or the qsa_sparse attention? A
--deterministicmode (accepting a throughput cost) would be ideal; failing that, a docs note that greedy ≠ reproducible with thinking on. Related: #223 (sampling-param validation), #281 (a regression harness needs to know this).