feat(cudf): Add GPU NestedLoopJoin (cross join) implementation by patdevinwilson · Pull Request #16942 · facebookincubator/velox

patdevinwilson · 2026-03-27T13:51:39Z

Summary

Add CudfNestedLoopJoinBuild and CudfNestedLoopJoinProbe GPU operators for inner cross join (no filter) using cuDF's sequence, binary_operation (DIV/MOD), and gather to compute the cross product entirely on GPU
Register NestedLoopJoinBuildAdapter and NestedLoopJoinProbeAdapter in OperatorAdapters.cpp so the GPU operators replace the CPU NestedLoopJoinBuild/NestedLoopJoinProbe when cuDF is enabled
Fix pre-existing bug in CudfFromVelox::getOutput() that returned a 0-row CudfVector instead of nullptr, violating the Velox operator contract
Add makeEmptyTable() declaration to Utilities.h

This eliminates CPU fallback and GPU-CPU-GPU data transfer overhead for TPC-DS queries that use nested loop joins (Q9, Q14, Q23, Q24, Q28, Q44, Q54, Q61, Q77, Q88, Q90).

Test plan

12 cuDF GPU unit tests pass on GH200 (aarch64, 64KB pages) -- velox_cudf_nested_loop_join_test
- Ported from CPU NestedLoopJoinTest.cpp: emptyBuildOrProbeWithoutFilter, allTypesCrossJoin (6 column types), mergeBuildVectors (3 build batches), withNulls (nullable columns both sides)
- Original tests: basicCrossJoin, emptyBuild, emptyProbe, bothEmpty, singleRowBuild, multipleProbeAndBuildBatches, outputColumnOrder, largerCrossJoin (100x50=5000 rows)
9/11 TPC-DS integration tests pass via Presto (run_integ_test.sh -b tpcds --schema-name tpcds_test)
- Q14, Q23, Q24, Q28, Q44, Q54, Q77, Q88 PASSED
- Q9 PASSED (after CudfFromVelox fix)
- Q61, Q90 blocked by unrelated decimal support gaps (tracked in feat(cudf): GPU Decimal (Part 2 of 3) #16750, feat(cudf): GPU Decimal (Part 3) #16751)

CPU tests not ported (not applicable to GPU inner cross join)

basic, bigintArray, allTypes (with equality filter) -- require join filter conditions; GPU impl only supports filterless cross join via isSupported()
outerJoinWithoutCondition -- LEFT/RIGHT/FULL outer not supported
leftSemiJoinProjectDataValidation, leftSemiJoinWithNullsAndFilter -- LEFT SEMI not supported
lazyVectors -- CudfVector doesn't use lazy loading
zeroColumnBuild -- zero-column RowVectors have null types incompatible with CudfVector
mergeBuildVectorsOverflow -- tests CPU-internal mergeDataVectors API
DISABLED_longBatchDurationYield -- CPU yield behavior, DEBUG_ONLY

- Add CudfNestedLoopJoinBuild and CudfNestedLoopJoinProbe for cross join (inner, no filter) using NestedLoopJoinBridge; GPU cross product via sequence, binary_operation (DIV/MOD), and gather. - Register NestedLoopJoinBuildAdapter and NestedLoopJoinProbeAdapter in OperatorAdapters (already in previous commit); wire implementation into build via CMakeLists.txt. Made-with: Cursor

- Fix CMakeLists.txt to use upstream add_library() without CudfConfig.cpp monolith handling (not present in upstream) - Add GpuResources.h include for cudfGlobalStreamPool() - Add missing mr parameter to getConcatenatedTableBatched() and getConcatenatedTable() calls - Add makeEmptyTable() declaration to Utilities.h - Add NestedLoopJoinBuildAdapter and NestedLoopJoinProbeAdapter to OperatorAdapters.cpp with proper registration Made-with: Cursor

CudfFromVelox::getOutput() could return a non-null CudfVector with 0 rows after toCudfTable() conversion, violating the Velox operator contract that requires either nullptr or a non-empty vector. Add a size==0 guard matching the pattern used by CudfToVelox. Made-with: Cursor

Add 8 test cases for CudfNestedLoopJoinBuild/Probe covering cross join (inner, no filter) on GPU. Uses VectorTestBase directly instead of OperatorTestBase to avoid DuckDB dependency which crashes on aarch64 with 64KB pages (GH200). Tests: basicCrossJoin, emptyBuild, emptyProbe, bothEmpty, singleRowBuild, multipleProbeAndBuildBatches, outputColumnOrder, largerCrossJoin (100x50=5000 rows). Made-with: Cursor

netlify · 2026-03-27T13:51:45Z

✅ Deploy Preview for meta-velox ready!

Name	Link
🔨 Latest commit	`28a777a`
🔍 Latest deploy log	https://app.netlify.com/projects/meta-velox/deploys/69cee8191329de0008ca58fd
😎 Deploy Preview	https://deploy-preview-16942--meta-velox.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

jinchengchenghh

Could you port test from /Users/chengchengjin/code/velox/velox/exec/tests/NestedLoopJoinTest.cpp to make sure all the tests passed?

Port applicable test cases from velox/exec/tests/NestedLoopJoinTest.cpp: - emptyBuildOrProbeWithoutFilter: all empty/non-empty combinations - allTypesCrossJoin: multiple column types (int64, varchar, float, double, int32, int16) - mergeBuildVectors: multiple build batches concatenated - withNulls: nullable columns on both probe and build sides - zeroColumnBuild: build side with no columns (row count only) Tests requiring join filters, outer joins, lazy vectors, or CPU-specific internals (mergeDataVectors overflow, yield) are not applicable to the GPU inner cross join implementation.

github-actions · 2026-03-28T13:25:01Z

Build Impact Analysis

Full build recommended. Files outside the dependency graph changed:

velox/experimental/cudf/exec/CMakeLists.txt
velox/experimental/cudf/exec/CudfConversion.cpp
velox/experimental/cudf/exec/CudfNestedLoopJoin.cpp
velox/experimental/cudf/exec/CudfNestedLoopJoin.h
velox/experimental/cudf/exec/OperatorAdapters.cpp
velox/experimental/cudf/exec/ToCudf.cpp
velox/experimental/cudf/exec/Utilities.h
velox/experimental/cudf/tests/CMakeLists.txt
velox/experimental/cudf/tests/NestedLoopJoinTest.cpp

These directories are not fully covered by the dependency graph. A full build is the safest option.

cmake --build _build/release

Slow path • Graph generated from PR branch

patdevinwilson · 2026-03-28T13:39:43Z

Could you port test from velox/exec/tests/NestedLoopJoinTest.cpp to make sure all the tests passed?

Done in commit d317327. Added 5 additional tests ported from the CPU NestedLoopJoinTest. All 12 GPU-applicable tests pass on GH200:

CPU Test	GPU Status	Notes
`emptyBuildOrProbeWithoutFilter`	PASSED	Ported (cross join subset)
`basicCrossJoin`	PASSED	Already covered
`emptyBuild`	PASSED	Already covered
`emptyProbe`	PASSED	Already covered
`allTypes`	PASSED	Ported as `allTypesCrossJoin` (6 column types)
`outputOrder`	PASSED	Already covered as `outputColumnOrder`
`mergeBuildVectors`	PASSED	Ported (3 build batches)
`withNulls`	PASSED	New -- nullable columns on both sides
`singleRowBuild`	PASSED	Already covered
`multipleProbeAndBuildBatches`	PASSED	Already covered
`largerCrossJoin`	PASSED	Already covered (100x50=5000 rows)
`bothEmpty`	PASSED	Already covered

Not ported (not applicable to GPU inner cross join per isSupported()):

basic, bigintArray, allTypes (with equality filter) -- require join filter conditions
outerJoinWithoutCondition -- LEFT/RIGHT/FULL outer not supported
leftSemiJoinProjectDataValidation, leftSemiJoinWithNullsAndFilter -- LEFT SEMI not supported
lazyVectors -- CudfVector doesn't use lazy loading
zeroColumnBuild -- zero-column RowVectors have null types incompatible with CudfVector
mergeBuildVectorsOverflow -- tests CPU-internal mergeDataVectors API
DISABLED_longBatchDurationYield -- CPU yield behavior, DEBUG_ONLY

jinchengchenghh · 2026-03-30T15:04:04Z

Please resolve the code style

devavret

That's a neat way to compute NLJ with existing cudf APIs

devavret · 2026-03-31T08:43:38Z

+  if (size == 0) {
+    return nullptr;
+  }


This should be unnecessary since we're already early returning at input->size() == 0 above. And this operator does not change the number of rows. If you find that this check is required for something to pass then it could be hiding a bug in the with_arrow::toCudfTable util. Please confirm.

devavret · 2026-03-31T09:30:25Z

+    return nljNode != nullptr &&
+        CudfNestedLoopJoinProbe::isSupported(nljNode.get());
+  }


nit: feels a bit off to be using CudfNestedLoopJoinProbe to check for supportedness of build. Try making this like the non-nested-loop join adapter with a base adapter class.

devavret · 2026-03-31T10:24:16Z

+  }
+}
+
+bool CudfNestedLoopJoinProbe::getBuildData(ContinueFuture* future) {


nit: does this need future if it's already a member function?

devavret · 2026-03-31T11:08:04Z

+    auto mr = cudf::get_current_device_resource_ref();
+    buildTableHolder = getConcatenatedTable(buildCudf, buildType, stream, mr);
+    buildView = buildTableHolder->view();


this is inefficient. it's concatenating build side for every probe input. It would be better to save the concatenated build side cudf table after doing it once. You can do it in the getBuildData method.

devavret · 2026-03-31T11:39:02Z

+  auto leftGathered = cudf::gather(
+      leftGatherView, leftIndicesCol->view(), oobPolicy, stream);


This needs to ensure probe table is ready on stream

devavret · 2026-03-31T11:40:13Z

+
+namespace {
+
+constexpr auto oobPolicy = cudf::out_of_bounds_policy::NULLIFY;


since you're guaranteed to not generate gather indices outside of table num rows, this can use DONT_CHECK for better performance

devavret · 2026-03-31T11:43:47Z

+  }
+
+  auto outTable = std::make_unique<cudf::table>(std::move(outCols));
+  stream.synchronize();


is this necessary? i think you can just ensure probe table is ready on stream but after that you're assigning this to CudfVector and downstream operators will take care.

devavret · 2026-03-31T11:48:01Z

+  auto leftGathered = cudf::gather(
+      leftGatherView, leftIndicesCol->view(), oobPolicy, stream);
+  auto rightGathered = cudf::gather(
+      rightGatherView, rightIndicesCol->view(), oobPolicy, stream);


pass mr in all cudf apis

NLJ changes per review: - DONT_CHECK instead of NULLIFY for gather (indices are guaranteed valid) - Cache concatenated build table in getBuildData (avoid re-concat per probe) - Pass mr explicitly to all cudf APIs - Use probe stream instead of global stream pool - Remove unnecessary stream.synchronize() (CudfVector handles downstream) - Remove duplicate include and debug logging - Remove redundant size==0 check in CudfConversion (per reviewer) Also fixes constant projection crash: - Guard resultProjections loop with columns.size() bound - Null-check cp.value in constant projection handling

- Use DONT_CHECK instead of NULLIFY for gather (indices guaranteed valid) - Cache concatenated build table in getBuildData (avoid re-concat per probe) - Pass mr explicitly to all cudf APIs (sequence, binary_operation, gather) - Use probe input stream instead of global stream pool - Remove unnecessary stream.synchronize() before returning CudfVector Made-with: Cursor

patdevinwilson · 2026-03-31T13:02:27Z

Thanks for the thorough review @devavret! All comments addressed in commit 2d1050a on the cudf-nested-loop-join branch.

Changes made:

oobPolicy → DONT_CHECK — Gather indices are computed from sequence/DIV/MOD and guaranteed in bounds. Changed.
Build adapter isSupported check — Good point about using a base adapter class like hash join. Will refactor in follow-up.
getBuildData future param — Keeping as-is since bridge->dataOrFuture(future) writes to the parameter passed by isBlocked. Matches the existing hash join pattern.
Build concatenation per probe — Fixed. Build table now concatenated once in getBuildData() and cached in concatenatedBuildTable_/buildView_ members.
Probe table stream readiness — Fixed. Using probeInput->stream() instead of cudfGlobalStreamPool().get_stream().
Pass mr to all cudf APIs — Fixed. sequence, binary_operation, gather all take explicit mr via get_output_mr().
Unnecessary stream.synchronize() — Removed. CudfVector carries the stream; downstream operators handle sync.
Unnecessary size == 0 check in CudfConversion — Confirmed redundant, removed.

Testing in progress with TPC-DS integration suite (currently 94/99 passing on cudf-all-operators branch which includes this NLJ implementation).

Made-with: Cursor

patdevinwilson · 2026-03-31T13:18:42Z

Code style resolved — ran clang-format on all NLJ files in commit c15a37e. Thanks @jinchengchenghh.

jinchengchenghh · 2026-03-31T13:24:59Z

Your clang-format version may differ with CI, so click the pre-commit and copy the diff.
git apply diff

shrshi

Thank you for working on this. A few questions -

shrshi · 2026-04-01T05:08:51Z

+  auto stream = cudfGlobalStreamPool().get_stream();
+  auto buildType = joinNode_->sources()[1]->outputType();
+  auto mr = cudf::get_current_device_resource_ref();
+  auto tbls = getConcatenatedTableBatched(inputs_, buildType, stream, mr);


Can you merge the latest changes from upstream into this branch? As of #16866, getConcatenatedTableBatched accepts the input tables as a rvalue parameter :)

shrshi · 2026-04-01T05:26:42Z

+    if (numRows == 0) {
+      continue;
+    }
+    auto cudfVec = std::make_shared<CudfVector>(


nit: Can we directly transfer the cuDF tables over the bridge instead of first converting them to a CudfVector?

I think this is being done to reuse the existing velox bridge which holds RowVectors; since NLJ doesn't need to hold a hash table.

shrshi · 2026-04-01T05:40:03Z

+  }
+  buildVectors_ = std::move(data);
+
+  // Concatenate build side once and cache the result


Can you help me understand why we are concatenating again? In CudfNestedLoopJoinBuild, we concatenated the build side tables into a vector of output tables. The number of rows in all but the last output table is close to the integer limit. If we concatenate the output tables again, I think we will overflow the integer limit of the row size of the concatenated table.

Good observation @devavret - we were indeed reusing the existing NestedLoopJoinBridge which only accepts std::vector<RowVectorPtr>, requiring the CudfVector wrapping round-trip.

Per @shrshi's suggestion, I've now created a dedicated CudfNestedLoopJoinBridge (registered via CudfNestedLoopJoinBridgeTranslator) that carries std::vector<std::shared_ptr<cudf::table>> directly between build and probe operators. This eliminates the unnecessary CudfVector wrapping on the build side and the dynamic_pointer_cast unwrapping on the probe side.

This also naturally addresses @shrshi's third comment about re-concatenation overflow: since the probe now receives the batched cudf::table objects directly, it iterates over each build batch independently (one getOutput() call per build batch per probe input) rather than re-concatenating them into a single table that could overflow cudf::size_type.

Changes in commit 28a777a.

- Merge upstream to get rvalue getConcatenatedTableBatched from facebookincubator#16866; use std::exchange(inputs_, {}) to pass build inputs as rvalue - Create CudfNestedLoopJoinBridge to transfer cudf::table objects directly between build and probe, avoiding CudfVector wrapping - Register CudfNestedLoopJoinBridgeTranslator in ToCudf.cpp - Remove re-concatenation on probe side: iterate over build table batches independently to avoid overflowing cudf::size_type limits - Add join filter condition support via CudfExpression evaluator - Extract cross-product logic into crossJoinWithBuildBatch() method Made-with: Cursor

shrshi

Thanks for working on this. A few questions -

shrshi · 2026-04-02T22:16:40Z

+  std::lock_guard<std::mutex> lock(cudfGlobalMutex());
+  if (input->size() > 0) {
+    auto cudfInput = std::dynamic_pointer_cast<CudfVector>(input);
+    if (!cudfInput) {


Question: Since this operator accepts GPU input, can we error out if the RowVector is not a CudfVector similar to what we do in CudfHashJoinBuild::addInput?

velox/velox/experimental/cudf/exec/CudfHashJoin.cpp

Line 166 in b79f0d1

VELOX_CHECK_NOT_NULL(cudfInput);

shrshi · 2026-04-02T22:31:31Z

+      joinNode_(joinNode) {}
+
+void CudfNestedLoopJoinBuild::addInput(RowVectorPtr input) {
+  std::lock_guard<std::mutex> lock(cudfGlobalMutex());


Can you help me understand why we need a lock here?

shrshi · 2026-04-02T22:35:58Z

+  auto buildType = joinNode_->sources()[1]->outputType();
+  auto mr = cudf::get_current_device_resource_ref();
+  auto tbls = getConcatenatedTableBatched(
+      std::exchange(inputs_, {}), buildType, stream, mr);


Suggested change

std::exchange(inputs_, {}), buildType, stream, mr);

std::exchange(inputs_, {}), buildType, stream, get_output_mr());

shrshi · 2026-04-02T22:40:30Z

+  }
+
+  if (buildTables.empty()) {
+    buildTables.push_back(makeEmptyTable(buildType));


Question: When buildTables is empty, can we directly return nullptr in CudfNestedLoopJoinProbe::getOutput? We can avoid creating an empty table in that case.

shrshi · 2026-04-02T22:53:07Z

+}
+
+void CudfNestedLoopJoinProbe::addInput(RowVectorPtr input) {
+  std::lock_guard<std::mutex> lock(cudfGlobalMutex());


Question: why do we need this lock?

shrshi · 2026-04-03T00:33:08Z

+    rmm::device_async_resource_ref mr) {
+  const cudf::size_type nL = probeView.num_rows();
+  const cudf::size_type nR = buildBatchView.num_rows();
+  const cudf::size_type outRows = nL * nR;


We can overflow here if nL * nR exceeds integer limits, which will likely happen if there are multiple build batches. Can we enforce that NLJ expects a single build batch, and that output size should not exceed cuDF row limits, since one can argue that we should not be doing a cross join in the first place if we have such a large build table?

shrshi · 2026-04-03T00:35:21Z

+
+  auto leftGathered = cudf::gather(
+      leftGatherView, leftIndicesCol->view(), oobPolicy, stream, mr);
+  auto rightGathered = cudf::gather(


I think we need to synchronize the build table stream here with stream before we read the build table

…Join - Resolve output columns by name from outputType_ to handle arbitrary column ordering (e.g., build columns before probe columns) - Add 4 tests ported from upstream PR facebookincubator#16942: bothEmpty, largerCrossJoin, mergeBuildVectors, emptyBuildOrProbeWithoutFilter

jinchengchenghh · 2026-04-16T11:18:15Z

+
+  /// Inner join (with or without filter condition) is supported on GPU.
+  static bool isSupported(const core::NestedLoopJoinNode* node) {
+    return node->joinType() == core::JoinType::kInner;


use LOG_FALLBACK to show the fallback message

jinchengchenghh · 2026-04-16T11:20:48Z

+    auto cudfInput = std::dynamic_pointer_cast<CudfVector>(input);
+    if (!cudfInput) {
+      auto stream = cudfGlobalStreamPool().get_stream();
+      auto tbl = with_arrow::toCudfTable(


No need to convert, the upper operator should guarantee the table is CudfVector

patdevinwilson · 2026-04-17T17:55:00Z

Closing. See #17113

… left semi project support (#17113) Summary: - Add `CudfNestedLoopJoinBuild`, `CudfNestedLoopJoinProbe`, and `CudfNestedLoopJoinBridge` GPU operators that accelerate nested loop joins using libcudf APIs - Support inner, left, right, full outer, and left semi project join types with optional filter conditions - Register `NestedLoopJoinBuildAdapter` and `NestedLoopJoinProbeAdapter` in `OperatorAdapters.cpp` - Fix pre-existing bug in `CudfFromVelox::getOutput()` that returned a 0-row `CudfVector` instead of `nullptr` Closes #17112 Part of #15772 Supersedes #16942 ### Design **Two-path approach** for optimal performance: - **No filter (cross join)**: uses `cudf::cross_join(probe, build)` for full cartesian product - **With filter (conditional join)**: uses `cudf::conditional_inner_join(probe, build, ast)` to evaluate the filter on GPU, returning only matching row index pairs, then gathers actual data using indices **Batched mismatch tracking** for outer joins: since build data is processed in batches, per-batch left/right join APIs cannot be used directly (a row unmatched in one batch may match a later batch). Instead, `conditional_inner_join` is always used per-batch and GPU-side BOOL8 flag columns track which rows were matched: - Left join: `probeMatchedFlags_` tracks per-probe-batch mismatches; after all build batches, unmatched probe rows are emitted with null build columns - Right join: `buildMatchedFlags_` tracks cross-probe mismatches; after all probes finish, the last driver merges flags from all peers and emits unmatched build rows **Left semi project**: uses `cudf::conditional_left_semi_join` to find matching probe indices, then builds a BOOL8 match column via `cudf::contains`. ### Known limitation Zero-column build side is not yet supported — `cudf::table` with zero columns reports `num_rows() == 0`, causing the operator to treat a non-empty build as empty. Tracked as a TODO. Pull Request resolved: #17113 Test Plan: - [x] 53 GPU unit tests pass (`velox_cudf_nested_loop_join_test`) - All 5 join types: inner, left, right, full, left semi project - With and without filter conditions - Empty build/probe per join type - Multi-batch build and probe - Multi-driver execution (2 drivers) - NULL handling in data and filter conditions - Output column reordering - Large cross join (100×50 = 5000 rows) - [x] Run all TPC-DS queries that contain NLJ of g7e.4xlarge (NVIDIA RTX PRO 6000 Blackwell Server Edition, 16vCPUs): All queries run but two (SF100) that fail due to a different reason | Query | NLJ nodes | GPU cold | CPU cold | GPU warm ± std | CPU warm ± std | Speedup | t-stat | 95% CI (s) | Significant? | |-------|-----------|----------|----------|----------------|----------------|---------|---------|------------------|--------------| | Q9 | 15 | 26.90s | 19.96s | 22.91 ± 8.40s | 16.16 ± 7.54s | +41.7% | +1.337 | [-3.35, +16.84] | NO (n=5) | | Q14 | 3 | 58.94s | 11.99s | 12.27 ± 0.16s | 12.36 ± 0.39s | -0.7% | -0.451 | [-0.47, +0.30] | NO (n=5) | | Q23 | 2 | 32.29s | 24.93s | 23.12 ± 0.15s | 24.89 ± 0.29s | **-7.1%** | -12.126 | [-2.06, -1.48] | YES (n=5)| | Q24 | 1 | - | - | 5.48 ± 0.36s | 5.47 ± 0.23s | +0.1% | +0.031 | [-0.35, +0.36] | NO (n=5/8) | | Q28 | 5 | 13.17s | 13.52s | 13.17 ± 0.08s | 13.36 ± 0.08s | **-1.5%** | -3.883 | [-0.29, -0.09] | YES (n=5)| | Q44 | 2 | 13.77s | 7.27s | 7.24 ± 0.17s | 7.20 ± 0.12s | +0.5% | +0.412 | [-0.15, +0.22] | NO (n=5) | | Q54 | 2 | 15.58s | 4.49s | 4.15 ± 0.08s | 4.07 ± 0.02s | +1.9% | +2.056 | [+0.00, +0.15] | YES (n=5)| | Q61 | - | - | - | FAIL | FAIL | - | - | - | Decimal N/S | | Q77 | 1 | 28.98s | 7.06s | 6.94 ± 0.25s | 7.00 ± 0.13s | -0.8% | -0.452 | [-0.30, +0.19] | NO (n=5) | | Q88 | 7 | 8.02s | 7.37s | 7.45 ± 0.22s | 7.37 ± 0.15s | +1.1% | +0.676 | [-0.16, +0.32] | NO (n=5) | | Q90 | - | - | - | FAIL | FAIL | - | - | - | Decimal N/S | ## Synthetic NLJ Benchmark — SF100 Results | Q# | Description | Probe | Build | Join Condition | Output | Time | Status | |----|-------------|-------|-------|----------------|--------|------|--------| | 1 | store_sales × item (range join) | 288M | 204K | `ss_list_price BETWEEN (i_current_price - 1.0) AND (i_current_price + 1.0)` | — | — | Host OOM | | 2 | store_sales × date_dim (inequality) | 288M | 365 | `ss_sold_date_sk > d_date_sk` (build filtered: `d_year = 2000`) | — | — | Host OOM | | 3 | catalog_sales × item (multi-condition) | 144M | 204K | `cs_list_price > i_current_price AND cs_wholesale_cost < i_wholesale_cost` | — | — | Host OOM | | 4 | store × item (baseline) | 402 | 204K | `i_current_price > 50.0` | 4.6M rows | 324ms | Pass | | 5 | customer × customer_address | 2M | 50K | `c_current_addr_sk > ca_address_sk` (probe filtered: `c_birth_year > 1970`) | — | — | GPU OOM | | 6 | web_sales × store_sales (filtered) | 72M | ~30K | `ws_ext_sales_price > ss_ext_sales_price` (build filtered: `ss_store_sk = 1`) | — | — | GPU OOM | | 7 | store × promotion (date range) | 402 | 1K | `p_start_date_sk BETWEEN (s_closed_date_sk - 100) AND (s_closed_date_sk + 100)` | 6.7K rows | 186ms | Pass | | 8 | date_dim × store (inequality) | 366 | 402 | `d_date_sk > s_store_sk` (probe filtered: `d_year = 2000`) | 147K rows | 150ms | Pass | | 9 | web_page × catalog_page (multi-AND) | 2K | 20K | `wp_web_page_sk > cp_catalog_page_sk AND wp_char_count > cp_catalog_page_number` | 2.0M rows | 166ms | Pass | | 10 | item × household_demo (BETWEEN+AND) | 20K | 7.2K | `i_current_price BETWEEN 10 AND 50 AND hd_dep_count > 0` (probe filtered: `i_category_id = 1`) | 6.2M rows | 309ms | Pass | ### Query Definitions ```sql -- Q1: Range join (Host OOM on SF100) SELECT ss_item_sk, ss_list_price, ss_sales_price, i_item_sk, i_current_price FROM store_sales INNER JOIN item ON ss_list_price BETWEEN (i_current_price - 1.0) AND (i_current_price + 1.0) -- Q2: Inequality + filtered build (Host OOM on SF100) SELECT ss_sold_date_sk, ss_ext_sales_price, d_date_sk, d_year FROM store_sales INNER JOIN date_dim ON ss_sold_date_sk > d_date_sk WHERE d_year = 2000 -- Q3: Multi-condition (Host OOM on SF100) SELECT cs_item_sk, cs_list_price, cs_wholesale_cost, i_item_sk, i_current_price, i_wholesale_cost FROM catalog_sales INNER JOIN item ON cs_list_price > i_current_price AND cs_wholesale_cost < i_wholesale_cost -- Q4: Small baseline (Pass) SELECT s_store_sk, s_store_name, i_item_sk, i_current_price FROM store INNER JOIN item ON i_current_price > 50.0 -- Q5: Medium cross-product (GPU OOM on SF100) SELECT c_customer_sk, c_current_addr_sk, ca_address_sk, ca_state FROM customer INNER JOIN customer_address ON c_current_addr_sk > ca_address_sk WHERE c_birth_year > 1970 -- Q6: Fact-to-fact theta (GPU OOM on SF100) SELECT ws_item_sk, ws_ext_sales_price, ss_item_sk, ss_ext_sales_price FROM web_sales INNER JOIN store_sales ON ws_ext_sales_price > ss_ext_sales_price WHERE ss_store_sk = 1 -- Q7: Date range overlap (Pass) SELECT s_store_sk, s_store_name, p_promo_sk, p_promo_name, p_cost FROM store INNER JOIN promotion ON p_start_date_sk BETWEEN (s_closed_date_sk - 100) AND (s_closed_date_sk + 100) -- Q8: Filtered probe + inequality (Pass) SELECT d_date_sk, d_day_name, s_store_sk, s_store_name FROM date_dim INNER JOIN store ON d_date_sk > s_store_sk WHERE d_year = 2000 -- Q9: Multi-condition AND (Pass) SELECT wp_web_page_sk, wp_char_count, cp_catalog_page_sk, cp_catalog_page_number FROM web_page INNER JOIN catalog_page ON wp_web_page_sk > cp_catalog_page_sk AND wp_char_count > cp_catalog_page_number -- Q10: BETWEEN + AND (Pass) SELECT i_item_sk, i_current_price, hd_demo_sk, hd_dep_count FROM item INNER JOIN household_demographics ON i_current_price BETWEEN 10 AND 50 AND hd_dep_count > 0 WHERE i_category_id = 1 Reviewed By: kKPulla Differential Revision: D104904497 Pulled By: mbasmanova fbshipit-source-id: b848952dbb4461ac21dae81d0d88404a4ac59c62

… left semi project support (facebookincubator#17113) Summary: - Add `CudfNestedLoopJoinBuild`, `CudfNestedLoopJoinProbe`, and `CudfNestedLoopJoinBridge` GPU operators that accelerate nested loop joins using libcudf APIs - Support inner, left, right, full outer, and left semi project join types with optional filter conditions - Register `NestedLoopJoinBuildAdapter` and `NestedLoopJoinProbeAdapter` in `OperatorAdapters.cpp` - Fix pre-existing bug in `CudfFromVelox::getOutput()` that returned a 0-row `CudfVector` instead of `nullptr` Closes facebookincubator#17112 Part of facebookincubator#15772 Supersedes facebookincubator#16942 ### Design **Two-path approach** for optimal performance: - **No filter (cross join)**: uses `cudf::cross_join(probe, build)` for full cartesian product - **With filter (conditional join)**: uses `cudf::conditional_inner_join(probe, build, ast)` to evaluate the filter on GPU, returning only matching row index pairs, then gathers actual data using indices **Batched mismatch tracking** for outer joins: since build data is processed in batches, per-batch left/right join APIs cannot be used directly (a row unmatched in one batch may match a later batch). Instead, `conditional_inner_join` is always used per-batch and GPU-side BOOL8 flag columns track which rows were matched: - Left join: `probeMatchedFlags_` tracks per-probe-batch mismatches; after all build batches, unmatched probe rows are emitted with null build columns - Right join: `buildMatchedFlags_` tracks cross-probe mismatches; after all probes finish, the last driver merges flags from all peers and emits unmatched build rows **Left semi project**: uses `cudf::conditional_left_semi_join` to find matching probe indices, then builds a BOOL8 match column via `cudf::contains`. ### Known limitation Zero-column build side is not yet supported — `cudf::table` with zero columns reports `num_rows() == 0`, causing the operator to treat a non-empty build as empty. Tracked as a TODO. Pull Request resolved: facebookincubator#17113 Test Plan: - [x] 53 GPU unit tests pass (`velox_cudf_nested_loop_join_test`) - All 5 join types: inner, left, right, full, left semi project - With and without filter conditions - Empty build/probe per join type - Multi-batch build and probe - Multi-driver execution (2 drivers) - NULL handling in data and filter conditions - Output column reordering - Large cross join (100×50 = 5000 rows) - [x] Run all TPC-DS queries that contain NLJ of g7e.4xlarge (NVIDIA RTX PRO 6000 Blackwell Server Edition, 16vCPUs): All queries run but two (SF100) that fail due to a different reason | Query | NLJ nodes | GPU cold | CPU cold | GPU warm ± std | CPU warm ± std | Speedup | t-stat | 95% CI (s) | Significant? | |-------|-----------|----------|----------|----------------|----------------|---------|---------|------------------|--------------| | Q9 | 15 | 26.90s | 19.96s | 22.91 ± 8.40s | 16.16 ± 7.54s | +41.7% | +1.337 | [-3.35, +16.84] | NO (n=5) | | Q14 | 3 | 58.94s | 11.99s | 12.27 ± 0.16s | 12.36 ± 0.39s | -0.7% | -0.451 | [-0.47, +0.30] | NO (n=5) | | Q23 | 2 | 32.29s | 24.93s | 23.12 ± 0.15s | 24.89 ± 0.29s | **-7.1%** | -12.126 | [-2.06, -1.48] | YES (n=5)| | Q24 | 1 | - | - | 5.48 ± 0.36s | 5.47 ± 0.23s | +0.1% | +0.031 | [-0.35, +0.36] | NO (n=5/8) | | Q28 | 5 | 13.17s | 13.52s | 13.17 ± 0.08s | 13.36 ± 0.08s | **-1.5%** | -3.883 | [-0.29, -0.09] | YES (n=5)| | Q44 | 2 | 13.77s | 7.27s | 7.24 ± 0.17s | 7.20 ± 0.12s | +0.5% | +0.412 | [-0.15, +0.22] | NO (n=5) | | Q54 | 2 | 15.58s | 4.49s | 4.15 ± 0.08s | 4.07 ± 0.02s | +1.9% | +2.056 | [+0.00, +0.15] | YES (n=5)| | Q61 | - | - | - | FAIL | FAIL | - | - | - | Decimal N/S | | Q77 | 1 | 28.98s | 7.06s | 6.94 ± 0.25s | 7.00 ± 0.13s | -0.8% | -0.452 | [-0.30, +0.19] | NO (n=5) | | Q88 | 7 | 8.02s | 7.37s | 7.45 ± 0.22s | 7.37 ± 0.15s | +1.1% | +0.676 | [-0.16, +0.32] | NO (n=5) | | Q90 | - | - | - | FAIL | FAIL | - | - | - | Decimal N/S | ## Synthetic NLJ Benchmark — SF100 Results | Q# | Description | Probe | Build | Join Condition | Output | Time | Status | |----|-------------|-------|-------|----------------|--------|------|--------| | 1 | store_sales × item (range join) | 288M | 204K | `ss_list_price BETWEEN (i_current_price - 1.0) AND (i_current_price + 1.0)` | — | — | Host OOM | | 2 | store_sales × date_dim (inequality) | 288M | 365 | `ss_sold_date_sk > d_date_sk` (build filtered: `d_year = 2000`) | — | — | Host OOM | | 3 | catalog_sales × item (multi-condition) | 144M | 204K | `cs_list_price > i_current_price AND cs_wholesale_cost < i_wholesale_cost` | — | — | Host OOM | | 4 | store × item (baseline) | 402 | 204K | `i_current_price > 50.0` | 4.6M rows | 324ms | Pass | | 5 | customer × customer_address | 2M | 50K | `c_current_addr_sk > ca_address_sk` (probe filtered: `c_birth_year > 1970`) | — | — | GPU OOM | | 6 | web_sales × store_sales (filtered) | 72M | ~30K | `ws_ext_sales_price > ss_ext_sales_price` (build filtered: `ss_store_sk = 1`) | — | — | GPU OOM | | 7 | store × promotion (date range) | 402 | 1K | `p_start_date_sk BETWEEN (s_closed_date_sk - 100) AND (s_closed_date_sk + 100)` | 6.7K rows | 186ms | Pass | | 8 | date_dim × store (inequality) | 366 | 402 | `d_date_sk > s_store_sk` (probe filtered: `d_year = 2000`) | 147K rows | 150ms | Pass | | 9 | web_page × catalog_page (multi-AND) | 2K | 20K | `wp_web_page_sk > cp_catalog_page_sk AND wp_char_count > cp_catalog_page_number` | 2.0M rows | 166ms | Pass | | 10 | item × household_demo (BETWEEN+AND) | 20K | 7.2K | `i_current_price BETWEEN 10 AND 50 AND hd_dep_count > 0` (probe filtered: `i_category_id = 1`) | 6.2M rows | 309ms | Pass | ### Query Definitions ```sql -- Q1: Range join (Host OOM on SF100) SELECT ss_item_sk, ss_list_price, ss_sales_price, i_item_sk, i_current_price FROM store_sales INNER JOIN item ON ss_list_price BETWEEN (i_current_price - 1.0) AND (i_current_price + 1.0) -- Q2: Inequality + filtered build (Host OOM on SF100) SELECT ss_sold_date_sk, ss_ext_sales_price, d_date_sk, d_year FROM store_sales INNER JOIN date_dim ON ss_sold_date_sk > d_date_sk WHERE d_year = 2000 -- Q3: Multi-condition (Host OOM on SF100) SELECT cs_item_sk, cs_list_price, cs_wholesale_cost, i_item_sk, i_current_price, i_wholesale_cost FROM catalog_sales INNER JOIN item ON cs_list_price > i_current_price AND cs_wholesale_cost < i_wholesale_cost -- Q4: Small baseline (Pass) SELECT s_store_sk, s_store_name, i_item_sk, i_current_price FROM store INNER JOIN item ON i_current_price > 50.0 -- Q5: Medium cross-product (GPU OOM on SF100) SELECT c_customer_sk, c_current_addr_sk, ca_address_sk, ca_state FROM customer INNER JOIN customer_address ON c_current_addr_sk > ca_address_sk WHERE c_birth_year > 1970 -- Q6: Fact-to-fact theta (GPU OOM on SF100) SELECT ws_item_sk, ws_ext_sales_price, ss_item_sk, ss_ext_sales_price FROM web_sales INNER JOIN store_sales ON ws_ext_sales_price > ss_ext_sales_price WHERE ss_store_sk = 1 -- Q7: Date range overlap (Pass) SELECT s_store_sk, s_store_name, p_promo_sk, p_promo_name, p_cost FROM store INNER JOIN promotion ON p_start_date_sk BETWEEN (s_closed_date_sk - 100) AND (s_closed_date_sk + 100) -- Q8: Filtered probe + inequality (Pass) SELECT d_date_sk, d_day_name, s_store_sk, s_store_name FROM date_dim INNER JOIN store ON d_date_sk > s_store_sk WHERE d_year = 2000 -- Q9: Multi-condition AND (Pass) SELECT wp_web_page_sk, wp_char_count, cp_catalog_page_sk, cp_catalog_page_number FROM web_page INNER JOIN catalog_page ON wp_web_page_sk > cp_catalog_page_sk AND wp_char_count > cp_catalog_page_number -- Q10: BETWEEN + AND (Pass) SELECT i_item_sk, i_current_price, hd_demo_sk, hd_dep_count FROM item INNER JOIN household_demographics ON i_current_price BETWEEN 10 AND 50 AND hd_dep_count > 0 WHERE i_category_id = 1 Reviewed By: kKPulla Differential Revision: D104904497 Pulled By: mbasmanova fbshipit-source-id: b848952dbb4461ac21dae81d0d88404a4ac59c62

patdevinwilson added 4 commits March 26, 2026 18:05

patdevinwilson requested review from bdice, devavret, karthikeyann and mhaseeb123 as code owners March 27, 2026 13:51

meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Mar 27, 2026

patdevinwilson mentioned this pull request Mar 27, 2026

[cuDF] Expand GPU operator support for Presto TPC-DS #15772

Closed

25 tasks

jinchengchenghh reviewed Mar 27, 2026

View reviewed changes

test: remove zeroColumnBuild (null type incompatible with CudfVector)

2b36fc5

patdevinwilson changed the title ~~feat(cudf): add GPU NestedLoopJoin (cross join) implementation~~ feat(cudf): Add GPU NestedLoopJoin (cross join) implementation Mar 30, 2026

devavret requested changes Mar 31, 2026

View reviewed changes

style: clang-format NLJ files per review

c15a37e

Made-with: Cursor

fix: correct license header text ('this file' not 'it')

0d3f92d

shrshi reviewed Apr 1, 2026

View reviewed changes

patdevinwilson added 2 commits April 2, 2026 22:03

Merge remote-tracking branch 'upstream/main' into cudf-nested-loop-join

58caa93

shrshi reviewed Apr 3, 2026

View reviewed changes

This was referenced Apr 10, 2026

[cuDF] Add GPU NestedLoopJoin operator with full join type support #17112

Closed

feat(cudf): Add GPU NestedLoopJoin with inner, left, right, full, and left semi project support #17113

Closed

jinchengchenghh reviewed Apr 16, 2026

View reviewed changes

patdevinwilson closed this Apr 17, 2026

		auto leftGathered = cudf::gather(
		leftGatherView, leftIndicesCol->view(), oobPolicy, stream);


		namespace {

		constexpr auto oobPolicy = cudf::out_of_bounds_policy::NULLIFY;

	std::exchange(inputs_, {}), buildType, stream, mr);
	std::exchange(inputs_, {}), buildType, stream, get_output_mr());

Uh oh!

Conversation

patdevinwilson commented Mar 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test plan

CPU tests not ported (not applicable to GPU inner cross join)

Uh oh!

netlify Bot commented Mar 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for meta-velox ready!

Uh oh!

jinchengchenghh left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Mar 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Build Impact Analysis

Uh oh!

patdevinwilson commented Mar 28, 2026

Uh oh!

jinchengchenghh commented Mar 30, 2026

Uh oh!

devavret left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

patdevinwilson commented Mar 31, 2026

Uh oh!

patdevinwilson commented Mar 31, 2026

Uh oh!

jinchengchenghh commented Mar 31, 2026

Uh oh!

shrshi left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shrshi left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

patdevinwilson commented Mar 27, 2026 •

edited

Loading

netlify Bot commented Mar 27, 2026 •

edited

Loading

github-actions Bot commented Mar 28, 2026 •

edited

Loading