Problem
Every internal record in backend/ is a plain @dataclass. Type hints on these are documentation only, nothing is enforced at construction. Records are built from Spark Connect rows and parsed JSON metadata, so a schema drift, a None where an int is expected, or a renamed Iceberg summary key flows silently through the collectors and only surfaces later as a confusing TypeError in GraphNormalizer or as a malformed graph in the UI.
Goal
Convert the backend record models to pydantic.BaseModel so bad data fails loudly at the boundary where it is constructed, with a ValidationError that names the field and the offending value.
Scope
Convert these 13 models:
| File |
Model |
backend/base_classes/base_file.py |
HiddenFile, BaseFile |
backend/collectors/collector.py |
FilesCollection |
backend/collectors/collect_snapshots.py |
SnapshotRecord |
backend/collectors/collect_manifests.py |
HiddenManifestMetadata, ManifestRecord |
backend/collectors/collect_data_files.py |
HiddenDataFileMetadata, DataFileRecord |
backend/collectors/collect_metadata.py |
MetadataFileRecord |
backend/table_inventory/table_inventory.py |
TableInventoryResult |
backend/search_cutoff/find_search_cutoff.py |
SearchCutoff |
backend/table_list_catalog/table_list_catalog.py |
CacheEntry |
backend/snapshot_analyzer/constants.py |
ReplaceSubOperation |
Also:
- Add
pydantic (v2) to backend/pyproject.toml.
- Replace
dataclasses.asdict / dataclasses.fields call sites with the BaseModel equivalents.
frozen=True dataclasses map to model_config = ConfigDict(frozen=True).
- Tighten the
Dict[str, Any] / bare Optional fields where the real shape is known, so the migration actually buys validation rather than re-encoding the same permissiveness.
Problem
Every internal record in
backend/is a plain@dataclass. Type hints on these are documentation only, nothing is enforced at construction. Records are built from Spark Connect rows and parsed JSON metadata, so a schema drift, aNonewhere anintis expected, or a renamed Iceberg summary key flows silently through the collectors and only surfaces later as a confusingTypeErrorinGraphNormalizeror as a malformed graph in the UI.Goal
Convert the backend record models to
pydantic.BaseModelso bad data fails loudly at the boundary where it is constructed, with aValidationErrorthat names the field and the offending value.Scope
Convert these 13 models:
backend/base_classes/base_file.pyHiddenFile,BaseFilebackend/collectors/collector.pyFilesCollectionbackend/collectors/collect_snapshots.pySnapshotRecordbackend/collectors/collect_manifests.pyHiddenManifestMetadata,ManifestRecordbackend/collectors/collect_data_files.pyHiddenDataFileMetadata,DataFileRecordbackend/collectors/collect_metadata.pyMetadataFileRecordbackend/table_inventory/table_inventory.pyTableInventoryResultbackend/search_cutoff/find_search_cutoff.pySearchCutoffbackend/table_list_catalog/table_list_catalog.pyCacheEntrybackend/snapshot_analyzer/constants.pyReplaceSubOperationAlso:
pydantic(v2) tobackend/pyproject.toml.dataclasses.asdict/dataclasses.fieldscall sites with theBaseModelequivalents.frozen=Truedataclasses map tomodel_config = ConfigDict(frozen=True).Dict[str, Any]/ bareOptionalfields where the real shape is known, so the migration actually buys validation rather than re-encoding the same permissiveness.