Show Iceberg table and partition statistics files in the graph

[YanivZalach]

Background

IceGraph currently visualizes table metadata JSON files, snapshot manifest lists, manifests, data files, and delete files. It does not collect or render Iceberg statistics artifacts.

Observed on the running Spark Connect table default.orders:

This is an Iceberg metadata relationship, not a manifest relationship:

metadata JSON
  ├─ current-snapshot-id -> snapshot manifest list
  ├─ statistics[]
  │    └─ statistics-path -> Puffin table-statistics file
  └─ partition-statistics[]
       └─ statistics-path -> Parquet partition-statistics file

statistics files contain snapshot-id as an association, but the snapshot does not point to them.

Official Iceberg references:

• Table Statistics and Partition Statistics metadata fields
• Puffin file format
• Spark compute_table_stats and compute_partition_stats procedures

Desired behavior

Expose registered Iceberg table-statistics and partition-statistics files as first-class graph nodes. Keep the existing metadata-to-snapshot and branch connections unchanged. Add a second direct connection type from each metadata JSON node to every file registered in its statistics[] and partition-statistics[] fields.

The graph therefore has two independent outgoing relationship types from a metadata node:

metadata JSON -> snapshot manifest list      (existing lineage)
metadata JSON -> table statistics file       (new direct metadata reference)
metadata JSON -> partition statistics file   (new direct metadata reference)

A statistics record may display its snapshot-id in the details panel, but that ID is an association only. Do not draw a snapshot-to-statistics edge.

[YanivZalach]

Implementation outline

Backend collection and graph payload

1. Add distinct file types for table statistics and partition statistics in backend/constants.py. Keep them separate because the payloads and formats differ.
2. Extend get_metadata_row_slim_df_from_path and CollectMetadata to retain optional statistics and partition-statistics arrays from every selected metadata JSON file. Metadata files without either field must continue to work.
3. Parse and normalize the entries into records keyed by statistics-path:
   • Table-statistics record: path, snapshot ID, file size, footer size, and blob-metadata.
   • Partition-statistics record: path, snapshot ID, and file size.
4. Deduplicate graph nodes by file path, while appending the path to every referring MetadataFileRecord.child_files. This preserves the fact that a file may remain registered across multiple metadata versions.
5. Add the two record collections to TableInventoryResult and include them in GraphNormalizer. Use the registered metadata values only. The initial implementation must not parse Puffin blobs or scan the partition-statistics Parquet file.
6. Preserve snapshot-range behavior: collect statistics references only from metadata files in the selected metadata window. A file that is absent from all selected metadata files must not be emitted.

Frontend graph and details

1. Add matching FileType values, labels, colors, and graph levels in frontend/src/graphConstants.js. Position the two statistics node types alongside snapshots, as metadata-derived side artifacts rather than downstream data files.
2. Update buildEdgesFromNodes in frontend/src/pages/TableLayout.jsx. Metadata nodes are currently handled specially and emit direct metadata-to-snapshot and branch edges. Preserve those existing edges. Add direct metadata-to-child edges only when the target is a table-statistics or partition-statistics node. This is an additional connection type, not a replacement.
3. Ensure the normal Graph page side panel shows the record fields. In particular:
   • Table Statistics: path, snapshot ID, file size, footer size, and blob metadata including blob type, field IDs, sequence number, snapshot ID, and properties such as NDV.
   • Partition Statistics: path, snapshot ID, and file size.
4. Do not put either type in the partition File Tree. They are metadata artifacts, not data or delete files.

[YanivZalach]

Mock view:

The files can be to the right or at the same level with the snapshots, which ever is more simple code wise and more clear.