perf(qwen4): typical acceptance behind --typical-threshold (default off)

[davidtai]

Typical acceptance: an opt-in tokens-per-cycle lane for Qwen3.8 Flash-Next

Stacked on #391 and #475: this branch is the #475 head plus one commit. Only commit 4c963f88 is this pull request; the lane is off by default.

This lane is NOT distribution-exact. Unlike the exact decode lanes on this
stack, typical acceptance changes the emitted token stream: it trades
distribution-exactness for more accepted tokens per verify cycle. It is OFF
by default, it is its own opt-in switch, and it is gated on task quality, not
on matching the target distribution. Turn it on only when you want the speed
and accept the trade.

This pull request adds one decode lane, typical acceptance (Medusa-2), behind a
single operator knob that is unset by default. When unset, the exact
speculative-sampling law runs byte-for-byte unchanged. When set to a positive
threshold, the lane accepts more draft tokens per cycle and decode gets faster.

Headline, at the canonical 16,384 / 1,024 cell, three seeds: at threshold 0.09
decode is 107.95 tok/s against 81.78 with the lane off, +32.0%, clearing the
100 tok/s target; tokens per cycle rise from 2.54 to 3.73. Quality holds within
measurement noise (Section 5). A stricter threshold 0.20 gives +19.8%.

Terms:

• verify cycle: one target forward pass that checks a block of draft tokens.
• tokens/cycle: committed tokens divided by verify cycles; higher is faster.
• delta, eps, H: the acceptance floor is min(eps, delta*exp(-H)), where H is the
  entropy of the target row.
• tok/s: decode tokens per second.

---

1. Chart

Three arms at the 16,384 / 1,024 cell: exact (lane off), threshold 0.09, threshold
0.20. The dashed line is the 100 tok/s target. Threshold 0.09 clears it; threshold
0.20 is a stricter setting between exact and 0.09. The full data is in
docs/perf/pr391-typical-acceptance.md. (Replace 4c963f8829a5f3403bd9ac26650fa1719cd36a7e with the pushed SHA.)

---

2. How it works, in plain terms

The exact rule verifies each draft token with a rejection coin. It accepts a draft
token x with probability min(1, p(x)/q(x)), where p is the target row and
q is the draft row. On a rejection it resamples that position from the residual
(p - q)+ and ends the block. This reproduces the target distribution exactly.

Typical acceptance changes the accept test. It accepts a draft token when the
token is "typical" under the target's own distribution:

p(x) > min(eps, delta * exp(-H(p)))

It accepts the longest run of typical positions in the block. At the first
position that is not typical, it resamples that one position from the target row
p itself (not from the residual) and ends the block. The per-position test uses
no coin; it is deterministic.

The floor adapts to the target's entropy. A confident target (low H) has a high
floor, so only a strong token is typical. An uncertain target (high H) has a low
floor, so more tokens are typical. This accepts more tokens per cycle than the
exact coin, which is where the speed comes from, and why the emitted stream no
longer matches the target distribution. At temperature 0 the argmax is always
typical, so the lane is a no-op there and engages only at temperature > 0.

Reference: Cai, Li, Geng, Peng, Lee, Chen, Dao, "Medusa: Simple LLM Inference
Acceleration Framework with Multiple Decoding Heads," 2024, arXiv:2401.10774,
Section 2.3.1 "Typical Acceptance" (criterion
p_original(x) > min(epsilon, delta * exp(-H(p_original)))). Medusa adapts the
typicality idea from Hewitt, Manning, Liang, "Truncation Sampling as Language
Model Desmoothing," 2022, arXiv:2210.15191.

---

3. What to merge

One lane, one knob, default off. The threshold is the operator control; it maps to
delta. GET /health reports the resolved state under typical_acceptance, and
each active request prints a [typical-accept] NOT distribution-exact; ... line.

Optimization	Knob	Class	Default	Effect at 16,384 tokens	Files
Medusa-2 typical acceptance: accept the longest prefix of draft positions that are typical under the target row p(x) > min(eps, delta*exp(-H)), resample the first atypical position from the target row	--typical-threshold / MTPLX_FABLE_TYPICAL_THRESHOLD (maps to delta)	NOT distribution-exact; gated on task quality	off (unset/0 = exact)	+32.0% decode at 0.09 (clears 100 tok/s); +19.8% at 0.20; quality within noise	mtplx/sampling.py, mtplx/generation.py, mtplx/server/openai.py; tests/test_typical_acceptance.py, tests/test_typical_acceptance_defaults_cpu.py, tests/test_typical_threshold_cli_health_cpu.py, docs/perf/pr391-typical-acceptance.md

Notes on this table:

• Default off, never armed by a served pack. The threshold is unset by
  default, so the exact law runs. The lane is not part of the served Flash-Next
  default stack; a served pack does not turn it on. A CPU test asserts the served
  default config leaves it off.
• --typical-threshold maps to delta; higher = stricter. A higher threshold
  raises the entropy floor, so fewer positions are typical and the arm sits closer
  to the exact rule. Unset or 0 is the exact law. As the threshold grows very
  large, every position is resampled from the target row and each cycle commits
  one exact token.
• eps is an advanced cap (MTPLX_FABLE_TYPICAL_EPS, default 1.0). With
  eps = 1.0 the cap never binds for any delta <= 1, so it is inert at the operating
  points here. A CPU test pins that the delta=0.09 numerics are identical under the
  old eps=0.3 and the new eps=1.0. Most operators never touch eps.
• Harness change kept in the PR. scripts/code_eval_gate.py gains an opt-in
  --save-completions sidecar (task_id + raw completion + extracted solution).
  Without the flag the scored report is byte-for-byte the same. This is how the
  HumanEval+ column below was produced offline on the exact same completions, the
  reproducibility the PR-391 quality screen needs on-box.

---

4. How to enable

The lane is off unless you set the threshold. Any of:

• Server flag: --typical-threshold 0.09
• Environment: MTPLX_FABLE_TYPICAL_THRESHOLD=0.09

Set it to 0 (or leave it unset) to keep the exact law. Raise it (for example
0.20) for a stricter, closer-to-exact setting. It engages only at
temperature > 0.

---

5. Benchmark method and results

Setting	Value
Model	Youssofal/Qwen3.8-Flash-Next-MTPLX-Optimized-Speed, revision 29ba90f82124961d0d902a9ea9bbb1034972af2f
Engine	MTPLX 2.10 on MLX 0.32.2, served from this branch
Prompt / output	16,384 templated tokens / 1,024 tokens
Sampler	temperature 1, top-p 0.95, top-k 20
Reasoning / MTP depth	xhigh / 3
Seeds	20260829, 20260830, 20260831
Memory cap / thermal	100 GiB / fans max, 40 C gate before each cell

Column sources: HE and HE+ are evalplus 0.3.1 greedy (the tool the PR-391 quality
screen used); MBPP and sampled HE are the repo gate. Each is internally consistent
across arms.

threshold	decode tok/s	tokens/cycle	HE (evalplus)	HE+ (evalplus)	MBPP (repo)	sampled HE (repo)
exact (off)	81.78	2.54	0.9329	0.9146	0.9016	0.9329
0.09	107.95	3.73	0.9329	0.9146	0.9016	0.9268
0.20	97.99	3.31	0.9329	0.9146	0.9016	0.9390

Prefill (~1329 tok/s), TTFT (~12.5 s) and peak memory (< 100 GiB) are flat across
arms: the lane touches only the decode accept decision.

6. Quality

Greedy HumanEval, HumanEval+ and MBPP are byte-identical across all three
arms. At temperature 0 the lane is a no-op (the argmax is always typical), so
greedy completions match exact. This is verified on the base tests and on the
extended HumanEval+ tests.

Sampled HumanEval moves by at most one or two tasks of 164, and the repo gate and
evalplus disagree on the direction (evalplus sampled HE+: exact 0.8963, 0.09
0.9207, 0.20 0.9268). The sampled effect is therefore within n=164 sampling and
harness noise, neither a gain nor a crater. On this battery, typical acceptance
preserves code-evaluation quality within measurement noise at both thresholds.

That is the trade: more tokens per cycle for a token stream that is not
distribution-exact, with a quality cost that stays inside noise here. The default
is the exact law; the threshold is how an operator opts in.

🤖 Generated with Claude Code