perf(qwen4): two exact decode lanes for Qwen3.8 Flash-Next (cached async PLE, pooled-key rowsel)

[davidtai]

Two stacked decode lanes for Qwen3.8 Flash-Next

This pull request adds two decode lanes on top of the Qwen3.8 Flash-Next stack.
Both lanes keep the output identical to the stock path. Both change only the
decode timing. The server arms both by default for a served Flash-Next pack, the
same way it arms the retained stack, and every lane has an off switch.

Headline, at the canonical 16,384 / 1,024 cell, three seeds, cold prefill, one
build: both lanes on decode 81.747 tok/s against 80.468 tok/s with both off, a
+1.59% gain, byte-identical output (section 4.2). This same-build pair is the
number this pull request cites.

The two lanes on their own, from the ABBA retest (section 4.1):

• Cached async PLE (ple_cached_aux): +1.54% decode over the bracketed control,
  and it separates cleanly from the window-to-window noise.
• Fixed-M4 pooled rowsel (qsa_pooled_rowsel): +1.03% decode, positive and
  stable but inside the window-to-window noise band at this shape.

Both lanes are exact by construction. In the same-build pair and in the retest,
the reasoning digest and the text digest match control on every seed.

Terms:

• PLE: per-layer embedding, fed from an n-gram sidecar table.
• QSA: Qwen Sparse Attention.
• M4: the fixed four-row speculative verify width.
• lane: the unit an operator turns off with one switch.
• tok/s: tokens per second.

---

1. Charts

The chart shows the six ABBA windows in run order (control, cached, pooled,
pooled, cached, control) with one line per seed. Each cached and pooled window
sits above both control windows at every seed. The complete data and the
did-not-work table are in docs/perf/pr391-aux-lanes.md.

---

2. What to merge

The server arms both lanes by default for a served Flash-Next pack. It stamps two
environment keys. Three controls turn a lane off:

• Set the key to 0, for example MTPLX_FABLE_PLE_CACHED_AUX=0.
• Set MTPLX_FABLE_DISABLE=<lane>[,<lane>] or MTPLX_FABLE_DISABLE=all.
• Pass --disable-optimization <lane> or --disable-optimization all.

GET /health reports the resolved state under aux_lane_defaults, and each
lane prints an install verdict. Class is exact when the outputs are identical to
the stock path. Effect is measured at the 16,384 / 1,024 cell.

Optimization	Env key	Lane	Class	Default	Effect at 16,384 tokens	Files	Recommendation
Cached async PLE auxiliary: a native provider stages the fixed 64 M4 n-gram rows and the auxiliary embedding plane is produced with mx.async_eval outside the compiled verifier	MTPLX_FABLE_PLE_CACHED_AUX	ple_cached_aux	exact	on; =0	+1.54% decode, separated from noise; byte-identical	mtplx/ple_cached_aux.py, mtplx/ple_cached_row_handoff.py, native_extensions/ple_cpu_rows/, mtplx/native/__init__.py; tests/test_pr391_ple_cached_aux_cpu.py, tests/test_pr391_ple_cached_row_handoff_cpu.py, tests/test_pr391_cached_sidecar_primitive_cpu.py, tests/test_pr391_cached_sidecar_producer_cpu.py	merge
Fixed-M4 pooled-key rowsel: the twelve QSA indexers' pooled-key preparation binds the pool kernel metadata once per indexer and shares one inv_freq object	MTPLX_FABLE_QSA_POOLED_ROWSEL	qsa_pooled_rowsel	exact	on; =0	+1.03% decode, inside the ~1.5% window noise; byte-identical	mtplx/qsa_pooled_rowsel.py; tests/test_pr391_fixed_m4_pool_install_cpu.py	merge (marginal at this shape)

Notes on this table:

• The cached lane needs the native extension native_extensions/ple_cpu_rows.
  scripts/fable/setup_over100_venv.sh builds it the same way it builds the QSA
  sparse-decode extension. When the extension is not built, the lane declines
  with a printed reason and the server serves the stock path. Any other failure
  fails the model load, so an armed-but-inert lane is unreachable.
• The pooled lane is exact by construction: the install checks the 48-layer QSA
  layout, the per-indexer geometry, the RMS-norm epsilon, the RoPE scale, the
  shared inv_freq object identity, and the rope and bank op-diet items, and it
  reports bank mode rowsel with no weight copies. A contract failure fails the
  model load.
• Both lanes stay out of the retained 44-key stack, the committed flag files and
  the full-stack self-check, so the PR-391 battery counts do not move. They share
  the retained stack's off switches only.
• A note for the width-parameterized verify lane that lands separately
  (MTPLX_QWEN4_FIXED_VERIFY_ROWS): the pooled method derives the number of new
  pooled blocks from the write width, not from an assumed single block, so it
  stays correct when a step writes 5 or 6 rows.

---

3. Benchmark method

Setting	Value
Model	Youssofal/Qwen3.8-Flash-Next-MTPLX-Optimized-Speed, revision 29ba90f82124961d0d902a9ea9bbb1034972af2f
Engine	MTPLX 2.10.2 on MLX 0.32.2, served from this branch
Prompt	16,384 templated tokens
Output	1,024 tokens
Sampler	temperature 1, top-p 0.95, top-k 20
Reasoning effort	xhigh
MTP depth	3
Seeds	20260829, 20260830, 20260831
Memory cap	100 GiB
Prefill state	cold; new prefill tokens equal prompt tokens
Thermal state	fans at maximum, a 40 degree Celsius gate before every cell

---

4. Results

4.1 The 16,384 / 1,024 ABBA retest

Six guarded windows ran in the order control, cached, pooled, pooled, cached,
control (c1, a1, b1, b2, a2, c2). The bracketing controls cancel the linear drift
across the run.

Window	Arm	20260829	20260830	20260831	Mean
c1	control	80.357	75.851	82.757	79.655
a1	cached	82.261	77.439	84.219	81.306
b1	pooled	81.325	76.683	83.920	80.642
b2	pooled	82.310	77.720	84.569	81.533
a2	cached	82.825	77.574	84.680	81.693
c2	control	81.691	76.973	83.953	80.872

Arm	Decode mean	vs control 80.264	17,408 (history)
control (c1, c2)	80.264	n/a	n/a
cached async PLE (a1, a2)	81.500	+1.236 (+1.54%)	+1.20%
fixed-M4 pooled (b1, b2)	81.088	+0.824 (+1.03%)	+0.77%

The two control windows differ by about 1.52%, which is the load-bearing noise.
The cached lane clears it: every cached window beats every control window, per
seed and per window-mean. The pooled lane overlaps it: its slower window falls
below control's faster window. Prefill and peak memory are flat, and the output
is byte-identical to control on all three seeds for both arms.

4.2 Same-build served pair

Two guarded windows ran on one build of this worktree, on the canonical cell.
The stack-both window serves the defaults, so the server arms both lanes; the
control window serves the same build with both lanes off
(MTPLX_FABLE_PLE_CACHED_AUX=0, MTPLX_FABLE_QSA_POOLED_ROWSEL=0). The server
log confirms engagement on the stack-both window: [fable] ple_cached_aux
installs variant=async_aux, and [fable] qsa_pooled_rowsel installs 12 rowsel
bindings with one shared inv_freq object.

Seed	Arm	prefill tok/s	decode tok/s	peak GB	wall s	TTFT s	gen (finish)	reasoning sha	text sha
20260829	stack-both	1,284.2	82.952	89.20	25.32	12.96	1024 (length)	cfc57ad86ebd	e3b0c44298fc
20260829	control	1,287.3	81.232	89.20	25.55	12.93	1024 (length)	cfc57ad86ebd	e3b0c44298fc
20260830	stack-both	1,350.4	77.530	93.50	20.19	12.30	610 (stop)	a69cd27623b6	cf5e14d1a99c
20260830	control	1,353.2	76.571	93.50	20.26	12.27	610 (stop)	a69cd27623b6	cf5e14d1a99c
20260831	stack-both	1,351.5	84.761	95.17	24.02	12.32	990 (stop)	0b28bbfa9fad	2baf608e1946
20260831	control	1,353.4	83.602	95.17	24.17	12.30	990 (stop)	0b28bbfa9fad	2baf608e1946
mean	stack-both	1,328.7	81.747	92.62	23.17	12.52	n/a	n/a	n/a
mean	control	1,331.3	80.468	92.62	23.33	12.50	n/a	n/a	n/a

The two lanes together add +1.279 decode tok/s, +1.59% over the same build
with both lanes off (per seed +2.12% / +1.25% / +1.39%). Prefill and peak memory
are flat, and TTFT is flat. The output is byte-identical: the reasoning digest
and the text digest match between the two arms on all three seeds. This
same-build pair is the number this pull request cites.

---

5. Original numbers at 17,408 tokens (history)

The two lanes come from an external optimization pass, first measured at a
different shape: 17,408 templated prompt tokens and 1,024 output tokens. Those
numbers are kept as history in
docs/perf/pr391-aux-lanes.md.

Lane	Candidate	Control	Delta
Cached async PLE	80.8518 tok/s	79.8925 tok/s	+1.20%
Fixed-M4 pooled rowsel	80.4197 tok/s	79.8085 tok/s	+0.77%

---

6. Changes that did not work

Every row below is measured and rejected. The code of these rows is not in this
pull request.

Optimization	Measured effect	Reason
Native sidecar sync raw PLE	73.03 tok/s (about -8.5%)	The synchronous native read sits on the critical path; the cached lane supersedes it.
Native sidecar async raw PLE	78.34 tok/s (about -1.9%)	The uncached predecessor; the cached lane supersedes it.
GDN conv/norm fused rows	-0.53 tok/s, +0.12 s wall	The component win did not survive the full model.
Empty finalization	-0.33% tok/s	Near flat, and it needs a custom profiler MLX build.
Native compiled-graph replay slot plan	-0.07% tok/s	An MLX-core change that measured flat to slightly slower.
Command-buffer timing extension	-0.18% tok/s	Measurement instrument, not a speedup.
GPU-stream PLE transport	-1.88% tok/s	The GPU-stream factory did not beat the CPU queue transport.
Deferred CPU PLE serving	-1.93% tok/s	The schedule trades old overlap for new overlap and shows no gain.
Queued sampled-D3 selector	-4.49% tok/s	The queued GPU-to-CPU-to-GPU boundary adds draft cost.
MTP depth 4	55.77 tok/s (about -30%)	The deeper draft costs more than it accepts.
Draft temperature 0.85	about 77.76 tok/s against 79.79	The benchmark contract is temperature 1.

---

7. How to run and how to disable

Build the venv and both native extensions:

scripts/fable/setup_over100_venv.sh

Serve the pack. The server arms both lanes by default:

mtplx serve \
  --model ~/.mtplx/models/Youssofal--Qwen3.8-Flash-Next-MTPLX-Optimized-Speed \
  --model-id mtplx-flash-next-optimized-speed

Read GET /health and check the aux_lane_defaults block. Confirm the
[fable] ple_cached_aux and [fable] qsa_pooled_rowsel verdicts name both lanes.

Turn one lane off, or both:

mtplx serve --disable-optimization ple_cached_aux --model ...
MTPLX_FABLE_QSA_POOLED_ROWSEL=0 mtplx serve --model ...
mtplx serve --disable-optimization ple_cached_aux,qsa_pooled_rowsel --model ...

---

8. File map

Area	Files
Runtime	mtplx/ple_cached_aux.py, mtplx/ple_cached_row_handoff.py, mtplx/qsa_pooled_rowsel.py, mtplx/qwen4_aux_lanes.py, mtplx/native/__init__.py, mtplx/full_stack_env.py, mtplx/runtime.py
Native extension	native_extensions/ple_cpu_rows/
Server	mtplx/server/openai.py, mtplx/server/__init__.py, mtplx/profiles.py
Tests	tests/test_pr391_ple_cached_aux_cpu.py, tests/test_pr391_ple_cached_row_handoff_cpu.py, tests/test_pr391_cached_sidecar_primitive_cpu.py, tests/test_pr391_cached_sidecar_producer_cpu.py, tests/test_pr391_fixed_m4_pool_install_cpu.py, tests/test_qwen4_aux_lanes.py
Documentation	docs/perf/pr391-aux-lanes.md, docs/perf/pr391-charts/pr391-aux-lanes-decode.svg

🤖 Generated with Claude Code

https://claude.ai/code/session_01C5oDbNTShwy1QS9dT8VDQD

[davidtai]

Superseded by youssofal#475 (same head, opened against upstream main).