Is your feature request related to a problem? Please describe.
All existing bloom-filter tests only catches a filter that prunes too much (rows would go missing) but never one that prunes nothing.
We cannot assert it in C++, since libcudf's Parquet writer cannot emit bloom filters, so a test cannot construct its own input. In Python we have committed fixtures, but cudf.read_parquet returns only a DataFrame, so the counter never reaches the test. And we don't want to work around that by calling pylibcudf directly from cuDF's test suite, that file should stick to cuDF Python APIs.
Describe the solution you'd like
Add bloom filter write support to libcudf's Parquet writer.
The verification then becomes trivial in C++, since the counters are already exposed on table_with_metadata and already asserted in cpp/tests/io/parquet_reader_test.cpp. A test would write a file with a bloom filter, then assert that a present value keeps the row groups while an absent value prunes them to zero. The in-range part is what statistics cannot prune, so it isolates the bloom filter's contribution.
Describe alternatives you've considered
- Embedding a hexdump of an externally written file in the C++ test: rejected as unmaintainable.
- Committing binary fixtures and testing from Python: the current state, but cannot assert the counter.
- Exposing the counters through cuDF's Python API: new public API surface added only for testing.
- Adding pylibcudf tests that read cuDF's fixtures: adding fragile cross-package-path.
- Raising the pyarrow pin to ≥24 and generating fixtures at runtime: solves fixture generation but a length-absent header still needs a committed file.
Additional context
Bloom-filter fixtures live in python/cudf/cudf/tests/data/parquet/ (bloom_filter_alignment*.parquet, data_index_bloom_encoding_*.parquet, mixed_card_ndv_*_bf_*.parquet).
Related: #22901 (bloom filter fetch and header-parsing fix), #23393 (Python bindings and pytest coverage for the hybrid scan reader), #23515 (length-absent bloom filters), #22861 (where the hexdump-based C++ test was removed in favor of pytests).
Is your feature request related to a problem? Please describe.
All existing bloom-filter tests only catches a filter that prunes too much (rows would go missing) but never one that prunes nothing.
We cannot assert it in C++, since libcudf's Parquet writer cannot emit bloom filters, so a test cannot construct its own input. In Python we have committed fixtures, but
cudf.read_parquetreturns only aDataFrame, so the counter never reaches the test. And we don't want to work around that by calling pylibcudf directly from cuDF's test suite, that file should stick to cuDF Python APIs.Describe the solution you'd like
Add bloom filter write support to libcudf's Parquet writer.
The verification then becomes trivial in C++, since the counters are already exposed on
table_with_metadataand already asserted incpp/tests/io/parquet_reader_test.cpp. A test would write a file with a bloom filter, then assert that a present value keeps the row groups while an absent value prunes them to zero. The in-range part is what statistics cannot prune, so it isolates the bloom filter's contribution.Describe alternatives you've considered
Additional context
Bloom-filter fixtures live in
python/cudf/cudf/tests/data/parquet/(bloom_filter_alignment*.parquet,data_index_bloom_encoding_*.parquet,mixed_card_ndv_*_bf_*.parquet).Related: #22901 (bloom filter fetch and header-parsing fix), #23393 (Python bindings and pytest coverage for the hybrid scan reader), #23515 (length-absent bloom filters), #22861 (where the hexdump-based C++ test was removed in favor of pytests).