Skip to content

Insights page: logic-based table insights #122

Description

@YanivZalach

Add a new Insights page that tells the user what IceGraph noticed about the table: file sizing, partition skew, delete pressure, metadata fragmentation, and how well the data is clustered.

First iteration is logic based only — deterministic heuristics over metadata already collected. No AI/LLM involvement (that is a later iteration, separate from #98).

Blocked by #100 (frontend philosophy and refactor) and #44 (Spark enhanced data file info).

Delivery

Insights are computed as part of the existing POST /api/v1/graph-data job and returned as a new top-level field on the job result (data.insights). It is independent of data.errors / data.warnings and does not feed the Issues panel.

Insights

Group A — computable from the current data-file record

Uses file_size_in_bytes, record_count, partition, content, sort_order_id, file_format, equality_ids, plus table properties, default-spec-id and default-sort-order-id.

  • File size vs target — avg / median / p95 file size against write.target-file-size-bytes, reported as a percentage delta from the target.
  • Small-file concentration — partitions ranked by file count and by the share of their files below a fraction of the target size.
  • Partition skew — file count and total bytes per partition; ratio of the largest partition to the median.
  • Delete-file pressure — delete-to-data-file ratio overall and per partition, split by position deletes and equality deletes.
  • Manifest fragmentation — data files per manifest, and manifests carrying only a handful of entries.
  • Declared sort order compliance — table declares a sort order but a share of files were written with sort_order_id = 0.
  • Spec and format drift — files spanning more than one partition spec, or more than one file format.
  • Commit cadence — snapshots per day and average files added per commit.

Group B — depends on #44 (column bounds from .all_files)

  • Clustering score per column — 0-100 score derived from how much per-file min/max ranges overlap; lower overlap scores higher. Reported alongside the number of files a point lookup on that column would need to touch.
  • Null-heavy and constant columns — columns where nulls dominate, or where min equals max across most files.
  • Row size anomaly — files whose bytes-per-row deviate sharply from the table median.

Thresholds

  • Where Iceberg already exposes the intent as a table property (write.target-file-size-bytes), read it from the table and report the deviation from it.
  • Every other threshold gets a default in backend/constants.py, settable by environment variable, and documented in README.md.
  • Not user-adjustable from the UI in this iteration.

Metadata

Metadata

Assignees

No one assigned

    Projects

    Status
    Todo

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions