[XPU][FP8] Reuse block MoE prefill descriptor buffers - #573
Conversation
|
Validation completed on XPU 0,1. Code state:
AccuracyGSM8K configuration: 30 rounds x 100 questions, 5-shot, temperature 0,
Per-block performanceSame accuracy-warmed service, exact 32k input / 2k output, one round:
The random numerical correctness test for the FP8 block-scale MoE prefill kernel passed. The exhaustive E4M3 encoding test still exposes the existing raw The wheel build completed successfully after one full retry. The first attempt hit a transient Conclusion: no per-block accuracy or performance regression was observed, and no cross-workload accuracy regression was observed for online per-tensor FP8. This PR is ready to merge based on the completed validation. |
Summary
tile_expertsandtile_rowstensors across block-scaled MoE prefill callsStream safety
A single thread-local buffer is not sufficient because one host thread can enqueue work on multiple XPU streams while prior work remains asynchronous. This PR keeps a persistent cache entry for each
(device_index, sycl::queue)pair. Calls on the same stream reuse ordered storage; calls on different streams use distinct tensors.Scope
This is a kernel-only performance optimization following #566 and the schema-only #572. The existing operator and framework dispatch are unchanged.
Testing
Not run as part of PR creation.