Optimize to_bitmask() on NEON - #344
Conversation
|
x86 has dedicated |
|
The complexity is isolated to NEON so it seems worthwhile to add this. Marking as ready for review. |
| lo | (hi << 8) | ||
| let rotated = vextq_u8::<8>(bits, bits); | ||
| let paired = vzip1q_u8(bits, rotated); | ||
| vaddvq_u16(vreinterpretq_u16_u8(paired)) as u64 |
There was a problem hiding this comment.
Will this work correctly on big-endian targets? Not sure if we care about those? Maybe we should run those in CI as well, as far as I can tell we currently don't.
There was a problem hiding this comment.
Probably doesn't matter, just wondering!
There was a problem hiding this comment.
Big-endian Aarch64? No, we definitely don't care. Support for it was removed from Linux 7.4 because it is never used.
LaurenzV
left a comment
LaurenzV
left a comment
There was a problem hiding this comment.
Otherwise fine from my side, without having tried to understand the new approach!
|
The old approach was splitting 256/512-bit vectors into 2/4 128-bit vectors, reducing their values to bits individually and combining the result. The new approach instead narrows elements first (u32->u16, u16->u8, etc) to reduce the width of the vector in bits, until it reaches either 128 bits or u8 elements, and then goes through the handwritten u8 codepaths. Does that make sense? |
|
Oh yeah it does, I meant more like I didn't spend time trying to verify all the used intrinsics, haha. 😄 |
2 participants
A stab at #342 to see what it would take to optimize it.
Summary of changes:
The narrowing reduction into base 8-bit kernels helps a lot; mask16x16 sees 1.5x to 2.x throughput improvement depending on the CPU.
The 256-bit kernel was discovered by having Codex generate a dozen candidates and select the best one according to llvm-mca. It saves 4-5 cycles of latency and provides 30% better throughput compared to the generic split-combine lowering into 128-bit ops. It is tested exhaustively on all possible valid mask patterns.
I'm not entirely convinced this is worth optimizing, so marking it as draft. If this really is an important operation for some algorithms, similar optimizations should also be extended to AVX2 and possibly SSE4.2; AVX-512 is already zero-cost.