docs: spec generating a DeltaTable declaration from a live table

[Tomoscorbin]

Design only — no code. Records the design for delta-engine generate CATALOG.SCHEMA.TABLE, which reads one live Unity Catalog table and prints an importable DeltaTable declaration.

Why

The engine has no adoption on-ramp. Bringing an existing table under management means hand-transcribing its catalog state — every column, type, nullability, comment, tag, property, and key.

Transcription errors are punished asymmetrically. The engine owns the full column set and the full key set, so an omission is not a smaller declaration but a destructive one: a column left out becomes DropColumn, a key left out becomes DropForeignKey. Some of this is caught — ColumnMappingRequiredForDrop blocks a column drop without column mapping — but nothing blocks dropping a constraint. Hand transcription is therefore most dangerous on exactly the tables that most need adopting: the large, old, key-bearing ones.

The design: raise, then render

The translation is half-built already. _lower_declaration (api/delta_table.py:550) is exactly DeltaTable -> DesiredTable; nothing goes the other way. And the public surface is close to round-trippable: every DeltaTable.__init__ parameter except scope has a matching read-only property, and ObservedColumn's fields are a subset of DesiredColumn's.

api/codegen.py       pure: ObservedTable -> Python source. No I/O, no backend.
databricks.py        build_sql_reader(connection) — mirrors build_sql_engine.
cli/generate.py      wires them; owns stdout/stderr and exit codes.

raise_declaration validates by construction — if DeltaTable(...) builds, the emitted source imports. render_declaration accepts any DeltaTable, so it also normalises hand-written declaration modules.

The correctness oracle already exists too: a generated declaration is correct if and only if diff_table(generated.to_desired_table(), observed) is empty. That is the primary test, not a nice-to-have.

Foreign keys: a knowingly accepted trap

ForeignKey.references holds a DeltaTable object, not a name, so a single-table module cannot construct one. And omission is not free — _diff_foreign_keys (domain/plan/diff.py:518) has no unmanaged branch, so an undeclared key becomes DropForeignKey. No scope avoids this: METADATA_ASPECTS includes FOREIGN_KEYS, and out-of-scope drift fails validation rather than being ignored.

Decision: emit the keys as a commented block with a warning naming the constraint that planning would drop. The module stays importable and immediately plan-able; the cost is that planning it as written drops the keys until the user wires them up.

Three defences, and the spec is explicit that none of them stops a determined > redirect: the comment block, a stderr warning (which is why generate_module returns source and warnings separately rather than a bare str), and a test pinning that a generated FK-bearing table produces exactly those DropForeignKey actions and nothing else.

The alternative that produces a clean-planning module — following the references and emitting the closure — is recorded as the natural follow-on alongside schema-wide discovery, where the closure is already being read.

Why there is no serialisation layer

Recorded because the question will recur: the codebase has nine translation sites and they look like they belong together.

Four wire formats, five target languages, and no two sites share both endpoints. The single true inverse pair is render_data_type / data_type_from_json, already co-located in one file for that reason. What the nine share is a verb, not logic.

A shared serialisation/ package also has no legal position. The layer contract (pyproject.toml:205) is cli -> databricks | schema | adapters | api -> application -> domain, exhaustive = true; a package importable by all of them sits at or below domain, which would put Databricks DDL rendering and AS JSON parsing below the domain — inverting the hexagon that backend-imports-stay-in-adapters exists to protect. It would worsen locality too: changing Delta property management today touches application/properties.py and its callers, with policy and meaning together.

to_x/from_x methods on domain classes are worse — they weld Databricks vocabulary onto a deliberately format-neutral domain (the Iceberg direction), and they invert the expression-problem trade-off: DataType is a closed set of 17 while the operations are what grows, so methods mean every new format edits all 17 classes. sql/types.py:12-19 already reasons about exactly this criterion.

What the repo actually does is visible in diff_entries.py: extract the shared interpretation, leave each consumer its own emission. Codegen shares only a verb with render_data_type, but shares an interpretation with _lower_declaration — which is why the raise lands beside the lowering.

Two numbers, verified rather than estimated

The vocabulary-pin test depends on them: schema.__all__ has 23 names, and DataType has 17 concrete subclasses — matching render_data_type's 17 match arms exactly, so that renderer is currently exhaustive.

Location

The brainstorming skill writes specs to docs/superpowers/specs/, which is gitignored here (.gitignore:47, zero files tracked). Placed in docs/todo/ instead, where this project's committed design docs live and where todo.md links them, with a matching backlog entry.

Verification

Docs-only; no src/ or tests/ change, so the code gates are unexercised and I make no claim about them. docs/todo/ is excluded from the Sphinx build (docs/conf.py:81), and the only pre-commit hook is commitizen on commit-msg, so no linter applies to this file.

What was checked: every file:line citation in the spec was read rather than recalled, and the two counts above were computed against the installed package.

---

Implementation plan

docs/todo/2026-07-30-catalog-to-declaration-codegen-plan.md — nine TDD tasks, each with runnable test code and its own commit.

#	Task	Deliverable
1	DeltaTable.scope	Completes the public surface; without it the renderer silently turns a scope="metadata" declaration into scope="full"
2	Type source rendering	render_data_type_source + the exhaustiveness pin
3	render_declaration	Whole declaration, defaults omitted, derived import line
4	raise_declaration	The inversion, including the streaming-table scope rule
5	generate_module	Module assembly, FK warning, and the round-trip oracle
6	build_sql_reader	The reader seam
7	generate command	CLI, exit codes, stdout/stderr split
8	Documentation	reference-cli.md, README, design-doc correction
9	Live verification	Credentialed generate-then-plan against a real workspace

Four things checked rather than assumed

Writing runnable test code surfaced four facts that would each have cost an implementation cycle:

• Click 8.4.2 removed CliRunner(mix_stderr=...) — the signature is now (charset, env), and stdout/stderr are captured separately by default. Four call sites in the draft plan would have failed with TypeError.
• ReadError has no .message — it passes the message to Exception.__init__ and keeps only .exception_type. The CLI error path uses str(error).
• tests/live/ auto-marks by directory via pytest_collection_modifyitems, so no pytestmark is needed; live_catalog() / live_schema() are plain functions rather than fixtures, and qualified_table() returns a dotted string that cannot be passed to fetch_state.
• Struct accepts a list and normalises to a tuple in __post_init__, so generated source can emit Struct([StructField(...)]) without tuple syntax.

One deviation from the design

The command goes in cli/app.py, not a new cli/generate.py. That is where the Typer app and its shared _anticipated_errors / _engine_logging helpers live, and tests/cli/conftest.py already monkeypatches names on cli_app; a separate module would force a second patch target for no benefit. Task 8 corrects the design doc's file table so the two do not disagree.

Not carried over

The design lists a golden-file test. The plan discharges it differently and says so: Task 3 asserts the complete rendered text character-for-character and Task 5 pins byte-identical regeneration, so a format change already fails a diff-reviewable assertion. Task 5 additionally runs ruff check --isolated over real generated output, since generated code a formatter immediately rewrites is a defect.