Skip to content

fix(cbq): read_index / MmapReader crash on zero-record CBQ files - #101

Open
alejandrogzi wants to merge 1 commit into
ArcInstitute:mainfrom
alejandrogzi:fix/empty-cbq-index
Open

fix(cbq): read_index / MmapReader crash on zero-record CBQ files#101
alejandrogzi wants to merge 1 commit into
ArcInstitute:mainfrom
alejandrogzi:fix/empty-cbq-index

Conversation

@alejandrogzi

Copy link
Copy Markdown

fix(cbq): read_index / MmapReader crash on zero-record CBQ files

Summary

Index::from_bytes fails on empty input, so a valid CBQ file containing
zero records (e.g. a run where an upstream filter discarded every read) cannot
have its index read: both the streaming Reader::read_index and
MmapReader::new return CbqError::IndexCastingError
("Unable to cast bytes to Index - likely an alignment error"). This makes
bqtools decode (and any other index-reading consumer) crash on such files.

Root cause

A zero-record CBQ has an empty index: u_bytes = 0 and a 9-byte zstd frame of
zero-length content. After decompression the buffer is an empty Vec<u8>,
whose pointer is dangling and aligned only to 1.
bytemuck::try_cast_slice::<u8, BlockRange> checks pointer alignment even for
empty slices, so it returns TargetAlignmentGreaterAndInputNotAligned.

The bytes on disk are canonical — they are exactly what ColumnarBlockWriter
produces for empty input — so this is purely a reader-side issue.

Fix

Guard the empty case before casting (src/cbq/core/index.rs):

if bytes.is_empty() {
    return Ok(Self { ranges: Vec::default() });
}

Test

Added cbq::read::tests::test_read_index_empty_file
(src/cbq/read.rs): writes a zero-record CBQ via ColumnarBlockWriter,
reads it back with the streaming Reader, and asserts the index loads with
0 records / 0 blocks.

  • Fails on current code with CbqError(IndexCastingError) — passes with the fix.
  • Full suite green: cargo test (327 tests + doctests), cargo fmt --check, cargo clippy.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request adds an empty guard in Index::from_bytes to handle zero-record CBQ files, preventing bytemuck::try_cast_slice from failing on empty decompressed buffers due to alignment checks on dangling pointers. It also adds a unit test to verify this behavior for the streaming Reader. The reviewer suggests expanding the test coverage to include MmapReader and optimizing the test by avoiding an unnecessary .to_vec() allocation.

Comment thread src/cbq/read.rs
Comment on lines +406 to +410
let mut reader = Reader::new(Cursor::new(writer.inner_data().to_vec())).unwrap();
while reader.read_block().unwrap().is_some() {}
let index = reader.read_index().unwrap().expect("index should exist");
assert_eq!(index.num_records(), 0);
assert_eq!(index.num_blocks(), 0);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The PR description mentions that both the streaming Reader::read_index and MmapReader::new crash on zero-record CBQ files. However, the added test only covers the streaming Reader path.

We should expand this test to also cover MmapReader to prevent future regressions in both reader implementations. Additionally, we can avoid the unnecessary .to_vec() allocation on writer.inner_data() by passing the slice directly to Cursor::new.

        let data = writer.inner_data();

        // Test streaming reader
        let mut reader = Reader::new(Cursor::new(data)).unwrap();
        while reader.read_block().unwrap().is_some() {}
        let index = reader.read_index().unwrap().expect("index should exist");
        assert_eq!(index.num_records(), 0);
        assert_eq!(index.num_blocks(), 0);

        // Test MmapReader
        let temp_dir = std::env::temp_dir();
        let temp_path = temp_dir.join(format!("empty_test_{}.cbq", std::process::id()));
        std::fs::write(&temp_path, data).unwrap();
        let mmap_reader = MmapReader::new(&temp_path).unwrap();
        assert_eq!(mmap_reader.num_records(), 0);
        assert_eq!(mmap_reader.num_blocks(), 0);
        std::fs::remove_file(temp_path).unwrap();

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant