Skip to content

Optimize 64-bit int<->float conversions on AVX2, add more tests - #348

Merged
Shnatsel merged 4 commits into
linebender:mainfrom
Shnatsel:float-conversion-optimizations
Aug 25, 2026
Merged

Optimize 64-bit int<->float conversions on AVX2, add more tests#348
Shnatsel merged 4 commits into
linebender:mainfrom
Shnatsel:float-conversion-optimizations

Conversation

@Shnatsel

@Shnatsel Shnatsel commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

Closes #311

Summary:

  1. Replace a branch with a xor in the 32-bit codepath, which is free on anything but haswell thanks to ILP
  2. Route f64x2 -> i64x2 conversions through non-precise scalar conversion intrinsic _mm_cvttsd_si64 which is faster than SIMD formulations thanks to ILP (for 2-element vectors only) and cheaper than as casts thanks to not handling saturation
  3. Add AVX2 specializations for 64-bit element conversions, some lifted from LLVM and other kernels selected by Codex for performance on llvm-mca out of a large pool of candidates.

llvm-mca reciprocal throughput for 256-bit vectors on AVX2, lower is better:

Conversion Haswell Skylake Alder Lake Zen 3
i64 → f64, old → new 10 → 2.0 10 → 1.5 8 → 1.3 4 → 2.0
u64 → f64, old → new 2.5 → 2.0 1.7 → 1.5 1.7 → 1.3 2.0 → 2.0
f64 → i64, approximate 12.3 → 4.5 12 → 3.0 12 → 3.0 8 → 4.0
f64 → i64, precise 12.3 → 6.0 12 → 4.0 12 → 4.3 8 → 4.5
f64 → u64, approximate 20 → 3.0 16 → 2.0 16 → 2.0 12.5 → 3.5
f64 → u64, precise 20 → 3.8 16 → 2.5 16 → 2.5 12.5 → 4.0

Up to 8x speedup!

Also adds in-depth tests for various edge cases against a from-scratch integer-only reference implementation because I don't trust anything anymore after the libm and musl FMA fiasco, and testing an algorithm from LLVM against itself is silly.

@LaurenzV LaurenzV left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As you already know I'm not a huge fan of big tests with large loops and magic numbers, but if you think it makes sense then let's just keep it. 😄

@Shnatsel

Copy link
Copy Markdown
Contributor Author

Yeah, sorry, these are kind of necessary. 64 bits is a very large input space, so throwing a bunch of handwritten inputs at it isn't very useful, and you can't write a simple exhaustive test either (which works fine for 32 bits). I'd rather verify the edge cases like this than ship a buggy implementation accidentally.

@Shnatsel
Shnatsel added this pull request to the merge queue Aug 25, 2026
Merged via the queue into linebender:main with commit 3ac40f9 Aug 25, 2026
22 checks passed
@Shnatsel
Shnatsel deleted the float-conversion-optimizations branch August 25, 2026 08:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Optimize float<->integer conversions further

2 participants