Skip to content

[Data] Bump shuffle related test to sf 1000 - #65200

Open
owenowenisme wants to merge 10 commits into
ray-project:masterfrom
owenowenisme:data/release-sf1000-shuffle-v2-inline8kb
Open

[Data] Bump shuffle related test to sf 1000 #65200
owenowenisme wants to merge 10 commits into
ray-project:masterfrom
owenowenisme:data/release-sf1000-shuffle-v2-inline8kb

Conversation

@owenowenisme

@owenowenisme owenowenisme commented Aug 4, 2026

Copy link
Copy Markdown
Member

Description

  • Bump all the shuffle release tests for hash shuffle v2 except map groups on column08 column13 column14(84groups), because there are data skews in the test, will make some ofthe reduce tasks unschedulable therefore we keep the test sf100 before we solve the data skews.
  • V1 still tested with sf 100
  • Bump the number of partitions of tpch tests to 200
  • Add _cast_strings_to_large in groupby benchmark because now some of the groups's string data can exceed 2GB per column, overflowing Arrow's int32 string offsets when the shuffle reduce sorts the partition into one contiguous table.

Now the release data test yaml looked messy but we can remove the v1 part after we deprecate v1

Related issues

Link related issues: "Fixes #1234", "Closes #1234", or "Related to #1234".

Additional information

Optional: Add implementation details, API changes, usage examples, screenshots, etc.

Signed-off-by: You-Cheng Lin <mses010108@gmail.com>
@owenowenisme owenowenisme added the data Ray Data-related issues label Aug 4, 2026
owenowenisme and others added 7 commits August 5, 2026 01:44
Signed-off-by: You-Cheng Lin <mses010108@gmail.com>
Signed-off-by: You-Cheng Lin <106612301+owenowenisme@users.noreply.github.com>
Signed-off-by: You-Cheng Lin <mses010108@gmail.com>
Signed-off-by: You-Cheng Lin <mses010108@gmail.com>
@owenowenisme owenowenisme added the go add ONLY when ready to merge, run all tests label Aug 9, 2026
@owenowenisme
owenowenisme marked this pull request as ready for review August 9, 2026 08:26

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request scales up TPC-H nightly benchmarks and groupby benchmarks to Scale Factor 1000 (SF1000) for the hash_shuffle_v2 strategy. To handle this larger scale, it increases the default join partition counts to 200, introduces a utility to cast string columns to large_string to prevent Arrow's int32 offset overflow, and adjusts various runtime environment parameters. Feedback on the changes includes correcting a factual error in a comment regarding the original partition count in tpch_q13.py and fixing a typo in release_data_tests.yaml.

Comment thread release/nightly_tests/dataset/tpch/tpch_q13.py Outdated
Comment thread release/release_data_tests.yaml Outdated

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

Reviewed by Cursor Bugbot for commit a94154d. Configure here.

Comment thread release/nightly_tests/dataset/groupby_benchmark.py
Signed-off-by: You-Cheng Lin <mses010108@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

data Ray Data-related issues go add ONLY when ready to merge, run all tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Ray fails to serialize self-reference objects

1 participant