Add a new Insights page that tells the user what IceGraph noticed about the table: file sizing, partition skew, delete pressure, metadata fragmentation, and how well the data is clustered.
First iteration is logic based only — deterministic heuristics over metadata already collected. No AI/LLM involvement (that is a later iteration, separate from #98).
Blocked by #100 (frontend philosophy and refactor) and #44 (Spark enhanced data file info).
Delivery
Insights are computed as part of the existing POST /api/v1/graph-data job and returned as a new top-level field on the job result (data.insights). It is independent of data.errors / data.warnings and does not feed the Issues panel.
Insights
Group A — computable from the current data-file record
Uses file_size_in_bytes, record_count, partition, content, sort_order_id, file_format, equality_ids, plus table properties, default-spec-id and default-sort-order-id.
- File size vs target — avg / median / p95 file size against
write.target-file-size-bytes, reported as a percentage delta from the target.
- Small-file concentration — partitions ranked by file count and by the share of their files below a fraction of the target size.
- Partition skew — file count and total bytes per partition; ratio of the largest partition to the median.
- Delete-file pressure — delete-to-data-file ratio overall and per partition, split by position deletes and equality deletes.
- Manifest fragmentation — data files per manifest, and manifests carrying only a handful of entries.
- Declared sort order compliance — table declares a sort order but a share of files were written with
sort_order_id = 0.
- Spec and format drift — files spanning more than one partition spec, or more than one file format.
- Commit cadence — snapshots per day and average files added per commit.
Group B — depends on #44 (column bounds from .all_files)
- Clustering score per column — 0-100 score derived from how much per-file min/max ranges overlap; lower overlap scores higher. Reported alongside the number of files a point lookup on that column would need to touch.
- Null-heavy and constant columns — columns where nulls dominate, or where min equals max across most files.
- Row size anomaly — files whose bytes-per-row deviate sharply from the table median.
Thresholds
- Where Iceberg already exposes the intent as a table property (
write.target-file-size-bytes), read it from the table and report the deviation from it.
- Every other threshold gets a default in
backend/constants.py, settable by environment variable, and documented in README.md.
- Not user-adjustable from the UI in this iteration.
Add a new Insights page that tells the user what IceGraph noticed about the table: file sizing, partition skew, delete pressure, metadata fragmentation, and how well the data is clustered.
First iteration is logic based only — deterministic heuristics over metadata already collected. No AI/LLM involvement (that is a later iteration, separate from #98).
Blocked by #100 (frontend philosophy and refactor) and #44 (Spark enhanced data file info).
Delivery
Insights are computed as part of the existing
POST /api/v1/graph-datajob and returned as a new top-level field on the job result (data.insights). It is independent ofdata.errors/data.warningsand does not feed the Issues panel.Insights
Group A — computable from the current data-file record
Uses
file_size_in_bytes,record_count,partition,content,sort_order_id,file_format,equality_ids, plus tableproperties,default-spec-idanddefault-sort-order-id.write.target-file-size-bytes, reported as a percentage delta from the target.sort_order_id = 0.Group B — depends on #44 (column bounds from
.all_files)Thresholds
write.target-file-size-bytes), read it from the table and report the deviation from it.backend/constants.py, settable by environment variable, and documented inREADME.md.