Overall Problem
Velox’s cuDF expression path was failing decimal tests and tripping compute-sanitizer on expressions like greatest(a, literal, b) / least(...). The trigger is null-aware cuDF binary ops (NULL_MAX, NULL_MIN, etc.) when one operand is a multi-row column and the other is a scalar (e.g. a folded decimal literal).
cuDF’s compiled binop path treats the scalar as a 1-row column_view and dispatches null-aware kernels that call column_device_view::is_valid() on both sides. That path is broken for scalars (see below). Under sanitizer you get a 4-byte OOB read from a 1-byte allocation; without sanitizer you can get silent UB and CUDA stream corruption, which may cause unrelated-looking failures later (e.g. divide tests).
Possible Velox workaround (and why it’s a bad long-term idea)
What was suggested: add CudfBinaryOpUtils helpers that make_column_from_scalar to broadcast the scalar, then call column-vs-column cudf::binary_operation. Wired into:
- GreatestLeastFunction (the real trigger)
- BinaryFunction for all decimal column-vs-scalar ops
- BetweenFunction for decimal literal bounds
- scatterNullsAtZeroDivisor for EQUAL vs zero
Why this is problematic:
-
Fixes the symptom in the wrong layer. The bug is in cuDF’s scalar wrapping; Velox shouldn’t permanently paper over it for every decimal binop.
-
Over-broad. Analysis showed only null-aware ops actually read scalar validity in the kernel (NULL_MAX, NULL_MIN, NULL_EQUALS, …). Native column-vs-scalar paths for ADD/SUB/MUL/MOD, comparisons, and EQUAL are fine and already covered by cuDF tests. Broadcasting everything adds extra GPU allocations and kernel work on hot arithmetic paths for no correctness gain.
-
Duplicates cuDF’s own fix. The workaround is essentially “call make_column_from_scalar before binop” — exactly what cuDF should do inside scalar_as_column_view (they already do this correctly for struct_view scalars).
-
Hides the bug from the ecosystem. Other RAPIDS/cuDF callers hitting column-vs-scalar NULL_MAX/NULL_MIN still see UB; Velox-only workaround doesn’t help them and delays upstream fix.
-
Maintenance debt. Needs to be narrowed or removed once cuDF is fixed; easy to forget and leave inefficient paths in place.
A minimal interim Velox workaround would only broadcast for null-aware ops in GreatestLeastFunction (and any other null-aware scalar paths). That’s defensible as a short-term unblocker; broadcasting all decimal column-scalar ops is not.
Apparent cuDF bug
Location: cpp/src/binaryop/compiled/binary_ops.cu, scalar_as_column_view (fixed-width types including fixed_point).
When building a temporary 1-row column_view from a scalar, cuDF does:
reinterpret_cast<bitmask_type const*>(s.validity_data())
But scalar validity is a single device bool (scalar.hpp → device_scalar), not a column null bitmask. Null-aware kernels in binary_ops.cuh (ops_wrapper for NullMax, NullMin, NullEquals, …) call is_valid() even for the scalar side (index 0), which goes through bit_is_set and reads a 32-bit word from a 1-byte buffer.
Confirmed by:
- compute-sanitizer on Velox repro and cuDF unit tests: Invalid global read of size 4 bytes in ops_wrapper on fixed_point, nearest allocation size 1 byte
- Stack: binary_operation(column, scalar) → apply_binary_op
Not affected: non-null-aware ops (ADD, DIV, EQUAL, comparisons, etc.) don’t read operand validity inside the device loop, so column-vs-scalar fixed_point works for those — which is why cuDF’s existing tests missed this (they had column/column NULL_MAX/NULL_MIN and column/scalar ADD/DIV, but not column/scalar null-aware fixed_point with N > 1).
Likely Fix
in scalar_as_column_view, materialize a proper 1-bit null mask (or use make_column_from_scalar(s, 1, …) as already done for struct scalars) instead of casting bool* to bitmask_type*.
See attached full Cursor agent transcript.
Steps/Code to reproduce bug
New tests added to binop-compiled-fixed_point-test.cpp. Instructions given.
Overall Problem
Velox’s cuDF expression path was failing decimal tests and tripping compute-sanitizer on expressions like greatest(a, literal, b) / least(...). The trigger is null-aware cuDF binary ops (NULL_MAX, NULL_MIN, etc.) when one operand is a multi-row column and the other is a scalar (e.g. a folded decimal literal).
cuDF’s compiled binop path treats the scalar as a 1-row column_view and dispatches null-aware kernels that call column_device_view::is_valid() on both sides. That path is broken for scalars (see below). Under sanitizer you get a 4-byte OOB read from a 1-byte allocation; without sanitizer you can get silent UB and CUDA stream corruption, which may cause unrelated-looking failures later (e.g. divide tests).
Possible Velox workaround (and why it’s a bad long-term idea)
What was suggested: add CudfBinaryOpUtils helpers that
make_column_from_scalarto broadcast the scalar, then call column-vs-columncudf::binary_operation. Wired into:Why this is problematic:
Fixes the symptom in the wrong layer. The bug is in cuDF’s scalar wrapping; Velox shouldn’t permanently paper over it for every decimal binop.
Over-broad. Analysis showed only null-aware ops actually read scalar validity in the kernel (NULL_MAX, NULL_MIN, NULL_EQUALS, …). Native column-vs-scalar paths for ADD/SUB/MUL/MOD, comparisons, and EQUAL are fine and already covered by cuDF tests. Broadcasting everything adds extra GPU allocations and kernel work on hot arithmetic paths for no correctness gain.
Duplicates cuDF’s own fix. The workaround is essentially “call make_column_from_scalar before binop” — exactly what cuDF should do inside scalar_as_column_view (they already do this correctly for struct_view scalars).
Hides the bug from the ecosystem. Other RAPIDS/cuDF callers hitting column-vs-scalar NULL_MAX/NULL_MIN still see UB; Velox-only workaround doesn’t help them and delays upstream fix.
Maintenance debt. Needs to be narrowed or removed once cuDF is fixed; easy to forget and leave inefficient paths in place.
A minimal interim Velox workaround would only broadcast for null-aware ops in GreatestLeastFunction (and any other null-aware scalar paths). That’s defensible as a short-term unblocker; broadcasting all decimal column-scalar ops is not.
Apparent cuDF bug
Location:
cpp/src/binaryop/compiled/binary_ops.cu,scalar_as_column_view(fixed-width types including fixed_point).When building a temporary 1-row column_view from a scalar, cuDF does:
But scalar validity is a single device bool (scalar.hpp → device_scalar), not a column null bitmask. Null-aware kernels in binary_ops.cuh (ops_wrapper for NullMax, NullMin, NullEquals, …) call is_valid() even for the scalar side (index 0), which goes through bit_is_set and reads a 32-bit word from a 1-byte buffer.
Confirmed by:
Not affected: non-null-aware ops (ADD, DIV, EQUAL, comparisons, etc.) don’t read operand validity inside the device loop, so column-vs-scalar fixed_point works for those — which is why cuDF’s existing tests missed this (they had column/column NULL_MAX/NULL_MIN and column/scalar ADD/DIV, but not column/scalar null-aware fixed_point with N > 1).
Likely Fix
in scalar_as_column_view, materialize a proper 1-bit null mask (or use make_column_from_scalar(s, 1, …) as already done for struct scalars) instead of casting bool* to bitmask_type*.
See attached full Cursor agent transcript.
Steps/Code to reproduce bug
New tests added to
binop-compiled-fixed_point-test.cpp. Instructions given.