Context
TableInventory._warn_if_data_cutoff_happened infers "the MAX_DATA_FILES_TO_COLLECT limit truncated this manifest" from len(manifest.child_files) == 0. That proxy is wrong: a manifest also ends up with zero child files when its avro failed to read, or when its added_snapshot_timestamp is NULL (manifests_extractor.py left-joins timestamps against only the collected snapshots, so any manifest added by a snapshot outside the selected range gets NULL, and its exclusive data files are then dropped by the latest_snapshot_timestamp > snapshot_timestamp_cutoff filter in data_files_extractor.py).
A guard was added in the PR that introduced per-file warnings: manifests with an error or a NULL added_snapshot_timestamp are now skipped. That removes the known false positives but keeps the proxy, so any other cause of an empty manifest still shows a false "limit of N data files was reached" warning, and a genuinely truncated manifest with a NULL timestamp gets no warning at all.
Goal
Use the signal the extractor already computes. DataFilesExtractor._find_cutoff_snapshot_timestamp returns epoch 0 when the limit was never reached and a real timestamp when it was, so the truth is available at the source.
Spec
DataFilesExtractor.extract_dataframe(self) -> tuple[DataFrame, bool] (or an equivalent accessor) — also reports whether max_data_files_to_collect was exceeded.
FilesCollection gains data_files_cutoff_reached: bool = False, set by CollectDataFiles.collect.
TableInventory._warn_if_data_cutoff_happened returns immediately unless that flag is set; the per-manifest and table-level warnings then keep their current text and the error guard, but drop the NULL-timestamp guard for the per-manifest warning.
Notes
- Collecting the cutoff dataframe costs an extra Spark action; check whether it can be folded into the existing
_find_included_data_files join instead.
Context
TableInventory._warn_if_data_cutoff_happenedinfers "theMAX_DATA_FILES_TO_COLLECTlimit truncated this manifest" fromlen(manifest.child_files) == 0. That proxy is wrong: a manifest also ends up with zero child files when its avro failed to read, or when itsadded_snapshot_timestampis NULL (manifests_extractor.pyleft-joins timestamps against only the collected snapshots, so any manifest added by a snapshot outside the selected range gets NULL, and its exclusive data files are then dropped by thelatest_snapshot_timestamp > snapshot_timestamp_cutofffilter indata_files_extractor.py).A guard was added in the PR that introduced per-file warnings: manifests with an
erroror a NULLadded_snapshot_timestampare now skipped. That removes the known false positives but keeps the proxy, so any other cause of an empty manifest still shows a false "limit of N data files was reached" warning, and a genuinely truncated manifest with a NULL timestamp gets no warning at all.Goal
Use the signal the extractor already computes.
DataFilesExtractor._find_cutoff_snapshot_timestampreturns epoch 0 when the limit was never reached and a real timestamp when it was, so the truth is available at the source.Spec
DataFilesExtractor.extract_dataframe(self) -> tuple[DataFrame, bool](or an equivalent accessor) — also reports whethermax_data_files_to_collectwas exceeded.FilesCollectiongainsdata_files_cutoff_reached: bool = False, set byCollectDataFiles.collect.TableInventory._warn_if_data_cutoff_happenedreturns immediately unless that flag is set; the per-manifest and table-level warnings then keep their current text and theerrorguard, but drop the NULL-timestamp guard for the per-manifest warning.Notes
_find_included_data_filesjoin instead.