[Data] Bump shuffle related test to sf 1000 - #65200
Conversation
…00-shuffle-v2-inline8kb
Signed-off-by: You-Cheng Lin <mses010108@gmail.com>
Signed-off-by: You-Cheng Lin <106612301+owenowenisme@users.noreply.github.com>
There was a problem hiding this comment.
Code Review
This pull request scales up TPC-H nightly benchmarks and groupby benchmarks to Scale Factor 1000 (SF1000) for the hash_shuffle_v2 strategy. To handle this larger scale, it increases the default join partition counts to 200, introduces a utility to cast string columns to large_string to prevent Arrow's int32 offset overflow, and adjusts various runtime environment parameters. Feedback on the changes includes correcting a factual error in a comment regarding the original partition count in tpch_q13.py and fixing a typo in release_data_tests.yaml.
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
Reviewed by Cursor Bugbot for commit a94154d. Configure here.

Description
_cast_strings_to_largein groupby benchmark because now some of the groups's string data can exceed 2GB per column, overflowing Arrow's int32 string offsets when the shuffle reduce sorts the partition into one contiguous table.Now the release data test yaml looked messy but we can remove the v1 part after we deprecate v1
Related issues
Additional information