Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
76 changes: 76 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -138,3 +138,79 @@ jobs:
env:
OPENSTATSPEC_MARIADB_URL: mysql+pymysql://openstatspec:openstatspec@localhost:3306/openstatspec
steps: *sql-test-steps

dolt-integration:
name: Dolt 2.2.2 integration
runs-on: ubuntu-latest
env:
OPENSTATSPEC_DOLT_URL: mysql+pymysql://openstatspec:openstatspec@127.0.0.1:13308/openstatspec
steps:
- uses: actions/checkout@v7
- name: Checkout OpenStatSpec specification fixtures
uses: actions/checkout@v7
with: *specification-checkout
- name: Checkout required SPSS engine
uses: actions/checkout@v7
with: *engine-checkout
- uses: actions/setup-python@v7
with:
python-version: "3.12"
- name: Initialize Dolt 2.2.2
run: |
mkdir -p "$RUNNER_TEMP/openstatspec-dolt"
docker run --rm \
--volume "$RUNNER_TEMP/openstatspec-dolt:/var/lib/dolt" \
--workdir /var/lib/dolt \
dolthub/dolt:2.2.2 init \
--name "OpenStatSpec CI" --email "ci@openstatspec.org"
docker run --rm \
--volume "$RUNNER_TEMP/openstatspec-dolt:/var/lib/dolt" \
--workdir /var/lib/dolt \
dolthub/dolt:2.2.2 sql \
-q "CREATE DATABASE openstatspec"
docker run --rm \
--volume "$RUNNER_TEMP/openstatspec-dolt:/var/lib/dolt" \
--workdir /var/lib/dolt \
dolthub/dolt:2.2.2 sql \
-q "CREATE USER 'openstatspec'@'%' IDENTIFIED BY 'openstatspec'; GRANT ALL PRIVILEGES ON openstatspec.* TO 'openstatspec'@'%'"
docker run --detach --name openstatspec-dolt \
--publish 127.0.0.1:13308:3306 \
--volume "$RUNNER_TEMP/openstatspec-dolt:/var/lib/dolt" \
--workdir /var/lib/dolt \
dolthub/dolt:2.2.2 sql-server -H 0.0.0.0 -l warning
- run: python -m pip install --upgrade pip
- run: python -m pip install ./openstatspec-pyspssio
- run: python -m pip install -e ".[dev,sql]"
- name: Wait for Dolt
run: |
python - <<'PY'
import time
import pymysql

for attempt in range(30):
try:
connection = pymysql.connect(
host="127.0.0.1",
port=13308,
user="openstatspec",
password="openstatspec",
database="openstatspec",
)
except pymysql.MySQLError:
if attempt == 29:
raise
time.sleep(2)
else:
connection.close()
break
PY
- run: python -m pytest -m services
- name: Show Dolt logs on failure and stop server
if: always()
run: |
if docker inspect openstatspec-dolt >/dev/null 2>&1; then
if [ "${{ job.status }}" != "success" ]; then
docker logs openstatspec-dolt
fi
docker rm --force openstatspec-dolt
fi
5 changes: 3 additions & 2 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,8 +14,9 @@ SPSS profile.
- Export of supported dataset semantics to SAV and ZSAV.
- Consistent legacy compatible-name rewriting across type-2, subtype-13, and
VLS subtype-14 records, with fail-closed SAV/ZSAV validation.
- SQLite, PostgreSQL, MySQL, and MariaDB profiles, including service-backed
PostgreSQL 17/18, MySQL 8.4/9.7, and MariaDB 11.4/11.8/12.3 CI coverage.
- SQLite, PostgreSQL, MySQL, MariaDB, and independent Dolt profiles, including
service-backed PostgreSQL 17/18, MySQL 8.4/9.7, MariaDB 11.4/11.8/12.3, and
pinned Dolt 2.2.2 CI coverage. Dolt Transformation Workflow support is not claimed.
- Preflight checks for target profile limits, atomic imports, validation, a
command-line interface, and machine-readable capability and loss reports.

Expand Down
9 changes: 7 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,12 +62,17 @@ engine. Its import module remains `pyspssio`; the exact source commit is recorde
in operation metadata. There is no fallback reader or writer. It supports unencrypted SAV and ZSAV import and
SAV/ZSAV export for the semantics exposed by that engine. SQLite is the local
reference path.
PostgreSQL, MySQL, and MariaDB are each covered by separate service-backed CI
conformance checks. Use these explicit SQLAlchemy URLs:
PostgreSQL, MySQL, MariaDB, and Dolt are each covered by separate service-backed CI
conformance checks. Dolt support is an independent core profile pinned to 2.2.2;
other Dolt versions and unknown MySQL-wire products fail closed. Use these explicit SQLAlchemy URLs:

- SQLite: `sqlite:///dataset.sqlite`
- PostgreSQL: `postgresql+psycopg://user:password@host/database`
- MySQL/MariaDB: `mysql+pymysql://user:password@host/database`
- Dolt 2.2.2: `mysql+pymysql://user:password@host/database` (detected by server identity)

The Dolt core profile supports strict wide-table import, validation, and export;
the separate Transformation Workflow is unsupported.

Run `openstatspec capabilities` before an integration to inspect the
machine-readable feature matrix. Export is deliberately strict: if known
Expand Down
12 changes: 7 additions & 5 deletions docs/sav-profile.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,12 +13,14 @@ unencrypted .sav and .zsav files, and writes both formats through the same imple

## Verified SQL profiles

SQLite has a local reference fixture. PostgreSQL, MySQL, and MariaDB have
SQLite has a local reference fixture. PostgreSQL, MySQL, MariaDB, and Dolt have
separate service-backed conformance checks in GitHub Actions that import,
validate, and export the supported fixture. MySQL 8.4 and MariaDB 11.4 exercise
the shared profile contract; this does not claim coverage for every server
configuration. An import that exceeds a target's strict single-table column
limit fails before it creates a dataset.
validate, and export the supported fixture. Dolt is an independent core profile
pinned to exact server version 2.2.2 and detected over `mysql+pymysql` by active
server identity; other Dolt versions and unknown MySQL-wire products fail closed.
The core Dolt profile does not claim support for the separate Transformation
Workflow. An import that exceeds a target's strict single-table column or row
envelope fails before it creates a dataset.

## Fidelity contract

Expand Down
2 changes: 1 addition & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ openstatspec = "openstatspec.cli:main"
[tool.pytest.ini_options]
testpaths = ["tests"]
addopts = "-q"
markers = ["services: requires a configured PostgreSQL or MySQL/MariaDB service"]
markers = ["services: requires a configured PostgreSQL, MySQL, MariaDB, or Dolt service"]

[tool.hatch.build.targets.wheel]
packages = ["src/openstatspec"]
8 changes: 7 additions & 1 deletion src/openstatspec/spss/sav.py
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,8 @@
variable_attribute_pairs,
)
from ..core import UnsupportedOperationError
from ..sql.capabilities import effective_profile
from ..sql.profiles import preflight
from ..sql.wide import (
create_wide_dataset,
physical_name,
Expand Down Expand Up @@ -262,7 +264,11 @@ def export_sav_dataset(
destination_path = Path(destination)
if destination_path.suffix.lower() not in {".sav", ".zsav"}:
raise UnsupportedOperationError("Export destinations must use the .sav or .zsav extension.")
dataset, variables, rows = read_wide_dataset(database_url=database_url, dataset_id=dataset_id)
profile, _active = effective_profile(database_url)
dataset, variables, rows = read_wide_dataset(
database_url=database_url, dataset_id=dataset_id, profile=profile,
)
preflight(profile, variables, rows=rows)
validate_spss_catalog(
variables,
case_weight_variable=dataset.get("case_weight_variable"),
Expand Down
Loading