Skip to content

Derive the data-files cutoff warning from the real cutoff signal, not from empty manifests #125

Description

@YanivZalach

Context

TableInventory._warn_if_data_cutoff_happened infers "the MAX_DATA_FILES_TO_COLLECT limit truncated this manifest" from len(manifest.child_files) == 0. That proxy is wrong: a manifest also ends up with zero child files when its avro failed to read, or when its added_snapshot_timestamp is NULL (manifests_extractor.py left-joins timestamps against only the collected snapshots, so any manifest added by a snapshot outside the selected range gets NULL, and its exclusive data files are then dropped by the latest_snapshot_timestamp > snapshot_timestamp_cutoff filter in data_files_extractor.py).

A guard was added in the PR that introduced per-file warnings: manifests with an error or a NULL added_snapshot_timestamp are now skipped. That removes the known false positives but keeps the proxy, so any other cause of an empty manifest still shows a false "limit of N data files was reached" warning, and a genuinely truncated manifest with a NULL timestamp gets no warning at all.

Goal

Use the signal the extractor already computes. DataFilesExtractor._find_cutoff_snapshot_timestamp returns epoch 0 when the limit was never reached and a real timestamp when it was, so the truth is available at the source.

Spec

  • DataFilesExtractor.extract_dataframe(self) -> tuple[DataFrame, bool] (or an equivalent accessor) — also reports whether max_data_files_to_collect was exceeded.
  • FilesCollection gains data_files_cutoff_reached: bool = False, set by CollectDataFiles.collect.
  • TableInventory._warn_if_data_cutoff_happened returns immediately unless that flag is set; the per-manifest and table-level warnings then keep their current text and the error guard, but drop the NULL-timestamp guard for the per-manifest warning.

Notes

  • Collecting the cutoff dataframe costs an extra Spark action; check whether it can be folded into the existing _find_included_data_files join instead.

Metadata

Metadata

Assignees

No one assigned

    Labels

    backendbugSomething isn't working

    Projects

    Status
    Todo

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions