Flash-Next n-gram sidecar: the table cannot be replaced by a function; the miss path pays three page-ins per row

[davidtai]

Summary

Question that started this: on a machine that cannot hold the Qwen3.8 Flash-Next n-gram sidecar (29.8 GiB) in memory, can the SSD read be replaced by a computed function, fitted polynomial, frequency-domain transform, or small network that re-expresses the rows?

Answer from measuring the table: no. The rows carry no structure that any of those can use, and lossless compression tops out near 13%. What remains is the cost of a cold row read, and the sidecar's miss path pays three page-ins per row, which is the one exact improvement this issue proposes.

1. The table has no structure a function can use

Probe on table shard 22 of Vontra/Qwen3.8-Flash-Next-MLX-4bit-MTP rev 327c8a60 (2,500,012 rows, 160 values per row, 4-bit affine group 32, 100 bytes per row). The MTPLX pack stores the same 4-bit table.

Measurement	Result	Meaning
Nibble entropy	3.84 of 4 bits	Codes are 96% incompressible
zstd -19 / xz on raw codes	1.04x / 1.03x	No lossless gain
Exact duplicate rows in 2.5M	0	No dictionary gain
Adjacent-row cosine vs random pair	0.065 vs 0.065	Row order carries nothing
SVD rank for 82% of energy	128 of 160	Near isotropic; no low-rank basis
DCT energy, first 20 bins vs last 20	12.5% vs 12.4%	Flat spectrum along the row
DCT coefficients for bit-exact rows	144 of 160	Exactness needs the whole row
Scale/bias entropy	83 of 160 bits per row	The only lossless slack, about 13% of the row

Product quantization with resident codebooks is the lossy floor:

Bytes per row	Table size	Energy lost	Values bit-exact
20	6.0 GiB	33%	19%
40	11.9 GiB	10%	34%
80	23.8 GiB	0.7%	89%
100 (checkpoint)	29.8 GiB	0	100%

Conclusions:

• A polynomial over the dimension index is a subspace of the lowest DCT coefficients, which carry their flat 1/160 share of energy each. Frequency-domain, low-rank and per-row-seed schemes fail for the same reason.
• Fitting the bf16 original does not change this. The Q4 step is 0.002 against a per-dimension spread of 0.0074, so quantization noise is about 0.6% of row energy. The flat spectrum is the signal.
• A dense network that reproduces the rows exactly would need at least the table's entropy in parameters, about 57B to 100B at 2 to 3.6 bits per parameter. Only a lossy function of the n-gram tokens is possible, and it changes model output.

2. Decode-time prefetch already measured in PR #391

PR #391 records these in its not-in-this-PR table, so they are not proposed again here:

• Candidate-row prefetch for the next cycle: cycle +0.72% on every seed, decode 69.6 to 69.1 tok/s, 7% hit rate.
• Early full-window PLE prefetch: -0.1%, bookkeeping equals saving.
• PLE boundary skip: resident fraction 0.11 to 0.13, so 85 to 93% of a cycle's rows are new.

What did land, all exact: prefill lookahead, first-chunk gather at request arrival, and --ngram-prewarm (decode 56 tok/s cold to 68.8 warm).

3. The miss path costs three page-ins per row

On PR #391's head a5e38bb, mtplx/models/qwen4_exp.py:

• The sidecar holds a hot-row LRU of raw row bytes, MTPLX_NGRAM_HOT_MB, default 1024 (line 4746 onward).
• A decode-sized gather (up to _HOT_PATH_MAX_ROWS = 4096, line 4698) takes the LRU path in _rows_matrices (line 4908 onward). Misses go to _warm (line 4818), which submits chunked os.pread calls to a 16-thread pool and blocks on every future, then fancy-indexes the now-warm memmaps.
• Each row is three preads, one per map, because weights, scales and biases live in three separate regions of ngram-table.safetensors (the touch loop in _submit_warm). The rows are hash-scattered, so each pread lands on its own page.

For a depth-3 cycle with 64 rows and 85 to 93% new, that is roughly 170 random page-ins per cycle instead of about 58. With PR #391's own measurement that a row costs a whole page either way, the sidecar's I/O per cycle is about three times what the rows need.

4. Proposal

1. One record per row. Lay the sidecar out as a single 100-byte record per row (weights, scales, biases interleaved) instead of three maps. Same bytes, same rows, one page-in per miss instead of three. This is a pack-side change plus a reader change and stays bit-exact.
2. Measure the cold cycle after the change. Decode tok/s cold versus --ngram-prewarm warm, with the per-cycle miss count and warm-pass wall time from the existing counters. The 19% cold-versus-warm gap in PR perf(qwen4): restack Flash-Next optimizations on MTPLX 2.10.1 #391 is the number to close.
3. Seed the hot LRU from ngram-hotness.npy at load if it is not already, so the first cycles of a request start at steady-state hit rate instead of empty.

Out of scope: any lossy table. It changes outputs and would need its own quality baseline.

[davidtai]

This really should just be a proposal to optimize the paging in 4.1 to reduce miss cost but agent is going to agent.

[youssofal]

Thank you for measuring instead of guessing. The entropy, duplicate, SVD and DCT results settle the "replace the table with a function" question for good, and the three-page-ins-per-row diagnosis is exact: in the reader, _submit_warm does one os.pread per map for every row (weights, scales and biases live in three regions of ngram-table.safetensors), so a hash-scattered miss really costs three pages.

2.11.2 ships your interleaved layout as an opt-in derived cache. python -m mtplx.ngram_row_layout <ngram-table.safetensors> --out <cache> converts the table and checks every row bit for bit, and MTPLX_NGRAM_ROW_FILE=<cache> serves from it. The pack itself is untouched and the cache is tied to that exact file. There is no default and no speed claim yet. The next step is the cold-versus-warm measurement on a Mac that cannot keep the table resident, together with your item 3, seeding the hot-row cache (MTPLX_NGRAM_HOT_MB) at load instead of starting it empty. Details are in docs/diagnostics/ngram-row-cache.md. This thread stays open for the measurement and the default decision, with your name on the analysis.