ir: Fix shift overflow in AND(SHR, C) constant folding

[weltling]

The redundancy rules for AND of a shifted value against a mask computed the shifted all ones value without masking the shift count, which is undefined behavior when the count reaches the type width.

Mask the shift count per type width to match the existing SHR constant folding.

[dstogov]

This seems right!

How did you find the problem.
Nor AddressSanitizer neither valgrind report problems to me on your test.

[dstogov]

I understood. C doesn't perform shifts on 8-bit data. It first extends operands to int and as result we don't have the undefined behavior. To demonstrate the issue, the test should use uint32_t instead of uint8_t (with modified constants).

Could you please modify the test.

[weltling]

Thanks for checking. I've been experimenting with a fuzzer that goes beyond the text parser and mutates the IR data structures directly. This was found by an early prototype run under a UBSan build.

On the test, I initially chose uint8_t on purpose, since it gets promoted to int before the shift, so it was actually failing on a plain release build. I'll adjust the test case as requested and file a draft PR with the fuzzer prototype later today for review.

Thanks!

[weltling]

@dstogov the test is rewritten with uint32_t now and adapted values.

Thanks

[dstogov]

Thanks!