Skip to content

The VQ4P apply is NEON-only, so §47's tuning does not travel to x86 #32

Description

@marcobambini

src/model.c:2137 gates the vectorized VQ4P apply:

#if (defined(__ARM_NEON) || defined(__aarch64__)) && !defined(WASTE_P6_SCALAR)

The vectorized side is four vqtbl4q_s8 per 16-row tile — a 64-byte table
lookup — over a vld3q_u8 deinterleave, with widening adds into int16. An x86
build takes the else branch: one row at a time, four scalar table loads per
row. It is not a narrower kernel, it is the absence of one — and on the same
build VQ3R's table gets simd_avx512.c.

Reported by @fab2s in #11, while trying to run the 64-entry crossover
docs/LEARNED.md §50 left open. Confirmed here by reading rather than by
measurement: this repo has no x86 machine that is not emulated, which is also
why docs/BACKENDS.md records AVX-512 as "compiled and dispatched, never
executed".

What is being assumed because of it

  • The 64-entry crossover cannot be measured on an x86 CPU at all. §50 left
    open whether VQ4P's 72 KB int8 table inverts the LUT-versus-reconstruct
    result it measured at 256 entries. On x86 that comparison puts a scalar apply
    against a vectorized one, so the number would be answering an ISA question
    and not a table-size one.
  • §47's tuning table is an ARM result. WASTE_XPAR, WASTE_XPAR_BATCH,
    WASTE_P6_CHUNK and the thread counts were measured against the vectorized
    apply, and CLAUDE.md carries them as engine-wide guidance. On x86 the kernel
    underneath them is a different kernel, and the inversion between Kimi-Linear
    and K3 has never been retested there.
  • VQ4P is the format convert.py produces by default, so this is the path
    on every x86 build, not a corner case.

What might close it

vqtbl4q_s8 looks up 64 bytes across four registers. The AVX-512 analogue is
VBMI's vpermi2b, which indexes 128 bytes across two zmm registers — the
same primitive with room to spare for a 64-entry table. Zen 4 and later, and
Ice Lake and later, have VBMI. src/simd_avx512.c already exists as the place
it would go, and dispatch resolves once at init, so this is a slot to fill
rather than an #ifdef in model.c.

Unmeasured in both directions, and worth stating before anyone spends a
weekend on it: the path may still be bound by the deinterleave rather than by
the lookup, in which case the table primitive buys much less than it looks
like it should. Cross-lane byte permutes are also not uniformly cheap across
x86 microarchitectures. This is a hypothesis with an obvious first experiment,
not a plan.

The doc sentence to qualify either way

docs/BACKENDS.md says:

The third hot path, the VQ gather, gets nothing — no x86 SIMD helps it
either, for the same reason NEON does not.

That was written about VQ3R's 256-entry table, where it holds: 256 entries fit
no byte-permute primitive. It is now being read as covering VQ4P, where the
premise it rests on is false. The sentence needs qualifying whether or not
anyone writes the kernel.

Acceptance, if someone does write it

  • Bit-identity, not "close". -DWASTE_P6_SCALAR already makes the ARM path
    self-checking against the portable one, and §43 says why the bar is that
    high: an int8 table makes the engine discontinuous, so an approximate match
    is not a match. An x86 path has to clear the same bar against the same
    reference.
  • The check that decides it is tests/run.sh's SIMD backend matches the
    CPU baseline
    , which runs WASTE_BACKEND=cpu against the dispatched path and
    compares logits. It has to run on an --index-bits 6 container: the suite's
    default WASTE_REF_MODEL is a VQ3R conversion, and per CLAUDE.md running
    the suite on the wrong container shape is exactly how a load path once stayed
    broken through green runs.
  • No machine here can execute it. Whoever writes it is also the only person
    who can test it, so the measurement matters more than the patch.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requesthelp wantedExtra attention is needed

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions