Remove legacy compatibility catalogs

CHANGELOG.md

@@ -5,6 +5,18 @@ All notable changes to this reference implementation are documented here.
 ## Unreleased
 
 Planned adapter release: `0.5.0`, after lifecycle integration and final specification conformance.
+### Fixed
+
+- Removed the temporary `*_catalog` compatibility schema and made SAV/ZSAV
+  import, validation, export, fidelity reporting, and in-place metadata edits
+  use the normative UUID-keyed OpenStatSpec catalog exclusively.
+- Existing databases that contain former compatibility relations must be
+  remediated manually before further operations.
+- Imports now reject dataset names that collide with normative UUID identifiers,
+  cleanup drops only a physical table actually created by the failing import,
+  and non-preserved SPSS compatible variable names are reported as an explicit
+  loss requiring export consent.
+
 
 ### Added
 
@@ -24,8 +36,8 @@ Planned adapter release: `0.5.0`, after lifecycle integration and final specific
 
 - Bumped the canonical transformation-plan and SPSS frontend contracts to
   `v0.2`; canonical JSON and hashes include every sequential operation.
-- In-place apply now records variable label, value labels, `F` print/write
-  format, and measurement level in both normative and compatibility catalogs.
+- In-place apply records variable label, value labels, `F` print/write
+  format, and measurement level only in the normative catalog.
 - Dolt still requires an exact branch, exact HEAD, and clean working set;
   successful apply leaves an inspectable diff and never calls `DOLT_COMMIT`.
 - Dolt declaration validation is implemented by the Python adapter. The

README.md

@@ -11,10 +11,10 @@ For each supported import, one source dataset becomes one dedicated wide SQL
 table. Cases are rows and source variables are physical SQL columns. The
 singular UUID-keyed tables from the specification (`dataset`, `variable`,
 `operation`, `fidelity_event`, and related metadata tables) are the public
-catalog contract. Historical `*_catalog` tables are an internal compatibility
-layer for the current exporter and are not the standard database interface.
-The adapter does not reshape data, create EAV
-or long-form tables, or harmonize studies or waves.
+catalog contract and the only catalog representation created or read by the
+adapter. Databases created with the former `*_catalog` compatibility schema
+must be remediated to the normative schema before use. The adapter does not
+reshape data, create EAV or long-form tables, or harmonize studies or waves.
 
 Unsupported source features, SQL targets, or export paths fail explicitly.
 There is no silent truncation, type conversion, metadata loss, or partial

docs/release-readiness.md

@@ -70,8 +70,8 @@ The gate must prove that:
 - the exact bounded `COMPUTE`/`IF` program compiles to all seven ordered
   operations without dropping `FORMATS`, `VARIABLE LEVEL`, or `EXECUTE`;
 - boolean data results match the equivalent expression and the target's label,
-  0/1 value labels, `F1.0` print/write format, and nominal level exist in both
-  normative and compatibility catalogs;
+  0/1 value labels, `F1.0` print/write format, and nominal level exist in the
+  normative catalog;
 - injected schema, data, catalog, and audit failures leave no partial apply;
 - compensation tracks only newly created targets and never drops or rewrites a
   pre-existing target;

docs/transformations.md

@@ -141,15 +141,15 @@ schema/table identity. It creates no derived dataset, output table, full-table
 copy, staging table, snapshot, rollback artifact, or recovery/history layer.
 Assignments and recodes use ordered `UPDATE` statements; later operations see
 earlier results. Label, value-label, format, and measurement-level operations
-update both the normative and compatibility catalogs.
+update the normative catalog.
 
 A numeric create target is supported atomically on SQLite and PostgreSQL.
 MySQL, MariaDB, and Dolt reject `target_mode=create` before mutation. On those
 profiles a separate versioned stage must first provision the nullable numeric
-physical column and both catalog representations; the transformation executor
-then sees a pre-existing target and performs no schema DDL.
-The public operation reports success only after physical data, both metadata
-representations, and the compact audit row are mutually complete.
+physical column and normative variable row; the transformation executor then
+sees a pre-existing target and performs no schema DDL.
+The public operation reports success only after physical data, normative
+metadata, and the compact audit row are mutually complete.
 
 Before Dolt mutation, the executor verifies the expected branch and `HEAD` and
 requires clean `dolt_status`. Success changes the same working set without

src/openstatspec/__init__.py

@@ -15,7 +15,8 @@
 from .sql.workflow import TransformationError
 from .transform import (
     AssignOperation, BooleanExpression, ComparisonExpression,
-    ConditionalAssignOperation, ExecuteOperation, Operand, PredicateExpression,
+    ConditionalAssignOperation, CreateVariableOperation, DeleteVariableOperation,
+    ExecuteOperation, Operand, PredicateExpression,
     RecodeMatch, RecodeOperation, RecodeResult, RecodeRule,
     ReplaceValueLabelsOperation, SetFormatOperation,
     SetMeasurementLevelOperation, SetVariableLabelOperation,
@@ -26,7 +27,7 @@
 __all__ = [
     "AssignOperation", "BooleanExpression", "ComparisonExpression",
     "ConditionalAssignOperation", "ExecuteOperation", "Operand",
-    "PredicateExpression",
+    "PredicateExpression", "CreateVariableOperation", "DeleteVariableOperation",
     "CapabilityDeclaration", "DoltConformanceSource", "LossReport",
     "SpssFrontendCompilation", "TransformationError",
     "TransformationFrontendError",

src/openstatspec/api.py

@@ -84,7 +84,7 @@ def capability_matrix(
             "multiple_response_sets": "supported",
             "variable_alignment": "supported",
             "variable_sets": "supported",
-            "compatible_variable_names": "supported",
+            "compatible_variable_names": "requires-explicit-loss-consent",
             "custom_attributes": {
                 "scalar_values": "supported",
                 "ordered_value_arrays": "supported",

src/openstatspec/frontends/spss/__init__.py

@@ -1,7 +1,12 @@
 """SPSS syntax frontend for canonical OpenStatSpec transformation plans."""
 
 from .binding import bind_spss_syntax
-from .compiler import SpssFrontendCompilation, compile_spss_syntax
+from .compiler import (
+    SPSS_FRONTEND_CONTRACT,
+    SPSS_FRONTEND_SCHEMA_CHANGE_CONTRACT,
+    SpssFrontendCompilation,
+    compile_spss_syntax,
+)
 from .syntax import (
     SpssSyntaxProgram,
     normalize_spss_source,
@@ -11,10 +16,9 @@
 )
 
 
-SPSS_FRONTEND_CONTRACT = "openstatspec-spss-syntax-frontend-v0.2"
-
 __all__ = [
     "SPSS_FRONTEND_CONTRACT",
+    "SPSS_FRONTEND_SCHEMA_CHANGE_CONTRACT",
     "SpssFrontendCompilation",
     "SpssSyntaxProgram",
     "bind_spss_syntax",

src/openstatspec/frontends/spss/binding.py

@@ -8,10 +8,12 @@
 from ...transform.errors import SourceSpan, frontend_error
 from ...transform.plan import (
     AssignOperation, BooleanExpression, ComparisonExpression,
-    ConditionalAssignOperation, ExecuteOperation, Operand, PlanOperation,
+    ConditionalAssignOperation, CreateVariableOperation, DeleteVariableOperation,
+    ExecuteOperation, Operand, PlanOperation,
     PredicateExpression, RecodeMatch, RecodeOperation, RecodeResult, RecodeRule,
     ReplaceValueLabelsOperation, SetFormatOperation,
     SetMeasurementLevelOperation, SetVariableLabelOperation,
+    TRANSFORMATION_PLAN_SCHEMA_CHANGE_CONTRACT,
     TRANSFORMATION_PLAN_V1_CONTRACT,
     TransformationPlan, TypedValue, ValueLabel,
 )
@@ -21,10 +23,11 @@
 from ...transform.validation import bind_transformation_plan
 from .syntax import (
     BooleanSyntax, ComparisonSyntax, ComputeCommandSyntax, ExecuteCommandSyntax,
-    FormatsCommandSyntax, IfCommandSyntax, OperandSyntax, PredicateSyntax,
+    DeleteVariablesCommandSyntax, FormatsCommandSyntax, IfCommandSyntax,
+    OperandSyntax, PredicateSyntax,
     RecodeCommandSyntax, RecodeMatchSyntax, RecodeResultSyntax,
     SpssSyntaxProgram, SyntaxLiteral, ValueLabelsCommandSyntax,
-    VariableLabelsCommandSyntax, VariableLevelCommandSyntax,
+    StringCommandSyntax, VariableLabelsCommandSyntax, VariableLevelCommandSyntax,
 )
 
 
@@ -290,8 +293,6 @@ def _bind_recode(
         # replace intentionally preserves the existing variable metadata. A later
         # VALUE LABELS command replaces value labels explicitly.
     return operations, spans
-
-
 def bind_spss_syntax(
     program: SpssSyntaxProgram, schema: VariableSchema, *, input_alias: str = "parent",
 ) -> BoundTransformation:
@@ -305,6 +306,50 @@ def bind_spss_syntax(
     operations: list[PlanOperation] = []
     spans: list[SourceSpan] = []
     for command in program.commands:
+        if isinstance(command, StringCommandSyntax):
+            for variable_token in command.variables:
+                if variable_token.text.startswith("__"):
+                    raise frontend_error(
+                        "reserved_target_name",
+                        f"Target name {variable_token.text!r} is reserved.",
+                        span=variable_token.span,
+                        target=variable_token.text,
+                    )
+                if any(
+                    variable.name.casefold() == variable_token.text.casefold()
+                    for variable in variables
+                ):
+                    raise frontend_error(
+                        "target_already_exists",
+                        f"Target name {variable_token.text!r} already exists.",
+                        span=variable_token.span,
+                        target=variable_token.text,
+                    )
+                operations.append(CreateVariableOperation(
+                    variable_token.text, "string", command.width,
+                ))
+                spans.append(command.span)
+                variables.append(VariableDefinition(
+                    variable_token.text, "string",
+                    declared_string_width=command.width,
+                ))
+            continue
+        if isinstance(command, DeleteVariablesCommandSyntax):
+            for variable_token in command.variables:
+                index, variable = _resolve(
+                    variables, variable_token.text, variable_token.span,
+                )
+                if len(variables) == 1:
+                    raise frontend_error(
+                        "cannot_delete_last_variable",
+                        "DELETE VARIABLES cannot remove the final dataset variable.",
+                        span=variable_token.span,
+                        variable=variable.name,
+                    )
+                operations.append(DeleteVariableOperation(variable.name))
+                spans.append(command.span)
+                del variables[index]
+            continue
         if isinstance(command, RecodeCommandSyntax):
             recodes, recode_spans = _bind_recode(command, variables)
             operations.extend(recodes)
@@ -447,10 +492,15 @@ def bind_spss_syntax(
     v01_types = (
         RecodeOperation, SetVariableLabelOperation, ReplaceValueLabelsOperation,
     )
+    schema_change_types = (CreateVariableOperation, DeleteVariableOperation)
     contract = (
-        TRANSFORMATION_PLAN_V1_CONTRACT
-        if all(isinstance(operation, v01_types) for operation in operations)
-        else "openstatspec-transformation-plan-v0.2"
+        TRANSFORMATION_PLAN_SCHEMA_CHANGE_CONTRACT
+        if any(isinstance(operation, schema_change_types) for operation in operations)
+        else (
+            TRANSFORMATION_PLAN_V1_CONTRACT
+            if all(isinstance(operation, v01_types) for operation in operations)
+            else "openstatspec-transformation-plan-v0.2"
+        )
     )
     plan = TransformationPlan(
         tuple(operations), contract=contract, input_alias=input_alias,

src/openstatspec/frontends/spss/compiler.py

@@ -4,7 +4,10 @@
 
 from dataclasses import dataclass
 
-from ...transform.plan import TransformationPlan
+from ...transform.plan import (
+    TRANSFORMATION_PLAN_SCHEMA_CHANGE_CONTRACT,
+    TransformationPlan,
+)
 from ...transform.schema import BoundTransformation, VariableSchema
 from .binding import bind_spss_syntax
 from .syntax import (
@@ -14,6 +17,10 @@
 )
 
 
+SPSS_FRONTEND_CONTRACT = "openstatspec-spss-syntax-frontend-v0.2"
+SPSS_FRONTEND_SCHEMA_CHANGE_CONTRACT = "openstatspec-spss-syntax-frontend-v0.3"
+
+
 @dataclass(frozen=True)
 class SpssFrontendCompilation:
     """One source artifact and its fully bound canonical plan."""
@@ -30,6 +37,12 @@ def plan(self) -> TransformationPlan:
     def plan_hash(self) -> str:
         return self.plan.sha256()
 
+    @property
+    def frontend_contract(self) -> str:
+        if self.plan.contract == TRANSFORMATION_PLAN_SCHEMA_CHANGE_CONTRACT:
+            return SPSS_FRONTEND_SCHEMA_CHANGE_CONTRACT
+        return SPSS_FRONTEND_CONTRACT
+
 
 def compile_spss_syntax(
     source: str,

src/openstatspec/frontends/spss/execution.py

@@ -10,7 +10,6 @@
     _run_in_place_submission,
     load_transformation_schema,
 )
-from . import SPSS_FRONTEND_CONTRACT
 from .compiler import compile_spss_syntax
 
 
@@ -37,7 +36,7 @@ def prepare(connection: Any, live_dataset_id: str) -> InPlacePlanSubmission:
             plan=compilation.plan,
             source_kind="spss_syntax",
             source_hash=compilation.source_hash,
-            frontend_contract=SPSS_FRONTEND_CONTRACT,
+            frontend_contract=compilation.frontend_contract,
         )
 
     return _run_in_place_submission(

src/openstatspec/frontends/spss/syntax.py

@@ -137,6 +137,19 @@ class ExecuteCommandSyntax:
     span: SourceSpan
 
 
+@dataclass(frozen=True)
+class StringCommandSyntax:
+    variables: tuple[Token, ...]
+    width: int
+    span: SourceSpan
+
+
+@dataclass(frozen=True)
+class DeleteVariablesCommandSyntax:
+    variables: tuple[Token, ...]
+    span: SourceSpan
+
+
 
 @dataclass(frozen=True)
 class VariableLabelSyntax:
@@ -175,6 +188,7 @@ class ValueLabelsCommandSyntax:
     RecodeCommandSyntax | ComputeCommandSyntax | IfCommandSyntax
     | VariableLabelsCommandSyntax | ValueLabelsCommandSyntax
     | FormatsCommandSyntax | VariableLevelCommandSyntax | ExecuteCommandSyntax
+    | StringCommandSyntax | DeleteVariablesCommandSyntax
 )
 
 
@@ -549,6 +563,43 @@ def execute(self, start: Token) -> ExecuteCommandSyntax:
         end = self.expects("period", "Expected '.' after EXECUTE.")
         return ExecuteCommandSyntax(_joined_span(start.span, end.span))
 
+    def string(self, start: Token) -> StringCommandSyntax:
+        variables = self.variable_list(
+            stop_kinds=frozenset({"left_paren", "period", "eof"}),
+        )
+        self.expects("left_paren", "Expected '(' before a STRING width.")
+        width_token = self.expects(
+            "identifier", "STRING requires a width such as A20.",
+        )
+        match = re.fullmatch(r"A([0-9]+)", width_token.text, re.IGNORECASE)
+        if match is None:
+            raise frontend_error(
+                "spss_syntax_error",
+                "STRING supports only character widths such as A20.",
+                span=width_token.span,
+            )
+        width = int(match.group(1))
+        if not 1 <= width <= 32767:
+            raise frontend_error(
+                "invalid_string_width",
+                "STRING width must be between 1 and 32767.",
+                span=width_token.span,
+                width=width,
+            )
+        self.expects("right_paren", "Expected ')' after a STRING width.")
+        end = self.expects("period", "Expected '.' after STRING.")
+        return StringCommandSyntax(
+            variables, width, _joined_span(start.span, end.span),
+        )
+
+    def delete_variables(self, start: Token) -> DeleteVariablesCommandSyntax:
+        self.expects_keyword("VARIABLES")
+        variables = self.variable_list(stop_kinds=frozenset({"period", "eof"}))
+        end = self.expects("period", "Expected '.' after DELETE VARIABLES.")
+        return DeleteVariablesCommandSyntax(
+            variables, _joined_span(start.span, end.span),
+        )
+
 
     def recode_result(self) -> RecodeResultSyntax:
         if (token := self.accepts_keyword("SYSMIS")) is not None:
@@ -692,6 +743,10 @@ def parse(self) -> SpssSyntaxProgram:
                 commands.append(self.formats(start))
             elif command == "execute":
                 commands.append(self.execute(start))
+            elif command == "string":
+                commands.append(self.string(start))
+            elif command == "delete":
+                commands.append(self.delete_variables(start))
             elif command == "variable":
                 if self.current.kind == "identifier" and self.current.text.casefold() == "level":
                     commands.append(self.variable_level(start))