Skip to content

feat: add Spark schema converter - #354

Merged
Tomoscorbin merged 2 commits into
mainfrom
agent/add-spark-schema-converter
Aug 13, 2026
Merged

feat: add Spark schema converter#354
Tomoscorbin merged 2 commits into
mainfrom
agent/add-spark-schema-converter

Conversation

@Tomoscorbin

Copy link
Copy Markdown
Owner

Summary

  • add delta_engine.databricks.to_spark_schema() as a lazy public Spark interop helper
  • keep native PySpark conversion inside the Spark adapter while preserving column order, spelling, types, and modeled nullability
  • document metadata and array/map nullability semantics and preserve the dependency-free declaration/import surface
  • cover every modeled scalar type, nested schemas, unsupported types, public imports, and missing-PySpark behavior

Why

ETL code can now reuse a DeltaTable declaration as its authoritative DataFrame schema without adding PySpark to DeltaTable or leaking Spark types into the backend-neutral schema API.

Validation

  • uv run pytest — 1,245 passed, 78 deselected; 97.15% coverage
  • uv run ruff check src tests
  • uv run ruff format --check src tests
  • uv run mypy src
  • uv run lint-imports — all 7 contracts kept
  • uv run --group docs sphinx-build -W -b html docs <temporary-output>

@Tomoscorbin
Tomoscorbin marked this pull request as ready for review August 13, 2026 09:02
@Tomoscorbin
Tomoscorbin merged commit a77a41c into main Aug 13, 2026
8 checks passed
@Tomoscorbin
Tomoscorbin deleted the agent/add-spark-schema-converter branch August 13, 2026 09:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant