Summary
chunk_file guards against binary content with a NUL-byte scan and returns no chunks. The
graph-enabled walk path never calls chunk_file, so that guard is bypassed: a valid-UTF-8 blob
containing NUL bytes is windowed and its raw NUL bytes land in Chunk::content.
NUL is a legal Unicode codepoint, so the file passes the UTF-8 check and reaches
chunk::chunk_text -> window_chunks, which has no binary guard at all.
Reproduction
// A valid-UTF-8 blob containing NUL codepoints, with a graphed-language extension.
let blob = format!("{}binary{}payload{}{}", NUL, NUL, NUL, "x\n".repeat(200));
fs::write(dir.join("blob.rs"), &blob)?;
chunk_file("blob.rs", &blob, "rust", IndexTuning::default());
walk_checkout(dir, &WalkOptions::builder().build_graph(true).build())?;Observed:
chunk_file() directly -> 0 chunks
walk_checkout(graph=true) -> 3 chunks
chunk NUL-contaminated: true
chunk NUL-contaminated: false
chunk NUL-contaminated: false
The two paths disagree about identical bytes.
Why it matters
The graph-enabled path is the path hosts actually run. Chunks are handed to an embedding model and
persisted; NUL bytes in text payloads are a common source of downstream failures (Postgres text
rejects the NUL codepoint outright).
Suggested direction
Apply the same binary rejection in the graph-enabled branch of walk_checkout, or move the guard
down into chunk_text/window_chunks so it cannot be bypassed by construction. The second is
preferable — the guard belongs where the chunk is produced, not at one of two call sites.
Notes
Pre-existing, inherited verbatim from the monorepo export.
Covered by a currently-passing test that documents the wrong behaviour:
tests/robustness.rs::build_graph_path_leaks_raw_nul_bytes_into_a_window_chunk_flagged_not_fixed.
Invert it when fixing.
Summary
chunk_fileguards against binary content with a NUL-byte scan and returns no chunks. Thegraph-enabled walk path never calls
chunk_file, so that guard is bypassed: a valid-UTF-8 blobcontaining NUL bytes is windowed and its raw NUL bytes land in
Chunk::content.NUL is a legal Unicode codepoint, so the file passes the UTF-8 check and reaches
chunk::chunk_text->window_chunks, which has no binary guard at all.Reproduction
Observed:
The two paths disagree about identical bytes.
Why it matters
The graph-enabled path is the path hosts actually run. Chunks are handed to an embedding model and
persisted; NUL bytes in text payloads are a common source of downstream failures (Postgres
textrejects the NUL codepoint outright).
Suggested direction
Apply the same binary rejection in the graph-enabled branch of
walk_checkout, or move the guarddown into
chunk_text/window_chunksso it cannot be bypassed by construction. The second ispreferable — the guard belongs where the chunk is produced, not at one of two call sites.
Notes
Pre-existing, inherited verbatim from the monorepo export.
Covered by a currently-passing test that documents the wrong behaviour:
tests/robustness.rs::build_graph_path_leaks_raw_nul_bytes_into_a_window_chunk_flagged_not_fixed.Invert it when fixing.