Skip to content

Replace backend dataclasses with Pydantic models #141

Description

@YanivZalach

Problem

Every internal record in backend/ is a plain @dataclass. Type hints on these are documentation only, nothing is enforced at construction. Records are built from Spark Connect rows and parsed JSON metadata, so a schema drift, a None where an int is expected, or a renamed Iceberg summary key flows silently through the collectors and only surfaces later as a confusing TypeError in GraphNormalizer or as a malformed graph in the UI.

Goal

Convert the backend record models to pydantic.BaseModel so bad data fails loudly at the boundary where it is constructed, with a ValidationError that names the field and the offending value.

Scope

Convert these 13 models:

File Model
backend/base_classes/base_file.py HiddenFile, BaseFile
backend/collectors/collector.py FilesCollection
backend/collectors/collect_snapshots.py SnapshotRecord
backend/collectors/collect_manifests.py HiddenManifestMetadata, ManifestRecord
backend/collectors/collect_data_files.py HiddenDataFileMetadata, DataFileRecord
backend/collectors/collect_metadata.py MetadataFileRecord
backend/table_inventory/table_inventory.py TableInventoryResult
backend/search_cutoff/find_search_cutoff.py SearchCutoff
backend/table_list_catalog/table_list_catalog.py CacheEntry
backend/snapshot_analyzer/constants.py ReplaceSubOperation

Also:

  • Add pydantic (v2) to backend/pyproject.toml.
  • Replace dataclasses.asdict / dataclasses.fields call sites with the BaseModel equivalents.
  • frozen=True dataclasses map to model_config = ConfigDict(frozen=True).
  • Tighten the Dict[str, Any] / bare Optional fields where the real shape is known, so the migration actually buys validation rather than re-encoding the same permissiveness.

Metadata

Metadata

Assignees

No one assigned

    Labels

    backendenhancementNew feature or requestneeds-triageMaintainer needs to evaluate this issue

    Projects

    Status
    Todo

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions