Skip to content

tomodian/deesql

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

20 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

CodeQL Copilot code review Dependency Graph docker release test.e2e test.unit

deesql

The missing toolkit for Amazon Aurora DSQL -- schema migrations, compatibility checking, and a local development proxy. Available as a CLI and as a Go library.

deesql compares your desired schema (.sql files) against a live Aurora DSQL cluster and generates a migration plan. It also ships a local proxy server that lets you develop against a standard PostgreSQL container while enforcing DSQL compatibility at the wire protocol level.

No migration history table, no temp databases -- just declarative schema diffing and a DSQL-native development workflow.

Why?

Aurora DSQL is a fantastic choice for building modern applications. It gives you a serverless, virtually unlimited, active-active distributed SQL database with strong consistency -- all at an affordable pay-per-request price point. And because it speaks the PostgreSQL wire protocol, your existing tools, drivers, and ORMs just work.

The one catch: DSQL supports a subset of PostgreSQL. Features like CREATE EXTENSION, triggers, PL/pgSQL, and FOREIGN KEY constraints aren't available. This is a reasonable tradeoff for the scalability and simplicity DSQL provides, but it means existing migration tools (designed for full PostgreSQL) can generate SQL that DSQL rejects at apply time.

deesql bridges that gap:

  • plan / apply -- Stateless, declarative migrations built specifically for DSQL. Parses your .sql files, introspects the live cluster, diffs, and applies -- respecting DSQL's DDL constraints at every step.
  • verify -- Catches DSQL-incompatible SQL in your schema files before you ever connect to a cluster.
  • proxy -- A local TCP proxy that sits between your app and a standard PostgreSQL container, intercepting and rejecting unsupported SQL with real DSQL error codes. Develop locally with full confidence that your SQL will work on DSQL.

Comparison

Feature deesql Atlas Flyway
Aurora DSQL support First-class Pro Plan required Generic PostgreSQL
Migration approach Declarative (desired-state diffing) Declarative + versioned Versioned (sequential migrations)
Migration history table None (stateless) Required (atlas_schema_revisions) Required (flyway_schema_history)
DSQL compatibility checking Built-in (verify command) Pro Plan required No
Local DSQL proxy Built-in (proxy command) No No
CREATE INDEX ASYNC Native support No awareness No awareness
1 DDL per transaction Handled automatically Manual workaround Manual workaround
IAM authentication Built-in (AWS SDK chain) Manual DSN config Manual DSN config
Unsupported ALTER TABLE detection Errors at plan time Errors at apply time Errors at apply time
Temp database required No (in-process parsing) Yes (for some providers) No
Language Go (single binary) Go (single binary) Java (JVM required)

Install

Go

go install github.com/tomodian/deesql@latest

Docker

docker run --rm ghcr.io/tomodian/deesql:latest --help

To run with AWS credentials:

docker run --rm \
  -v ~/.aws:/home/nonroot/.aws:ro \
  -e AWS_PROFILE \
  -v $(pwd)/schema:/schema:ro \
  ghcr.io/tomodian/deesql:latest \
  plan --endpoint <cluster>.dsql.<region>.on.aws --schema /schema

Build from source

git clone <repo>
cd migrate
make build

Quick Start

  1. Define your desired schema in .sql files:
-- schema/users.sql
CREATE TABLE users (
    id         TEXT PRIMARY KEY DEFAULT gen_random_uuid()::TEXT,
    email      TEXT NOT NULL UNIQUE,
    name       TEXT NOT NULL,
    created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);

CREATE INDEX ASYNC idx_users_email ON users (email);
  1. Check compatibility:
deesql verify --schema ./schema
  1. Preview changes:
deesql plan --endpoint <cluster>.dsql.<region>.on.aws --schema ./schema
  1. Apply:
deesql apply --endpoint <cluster>.dsql.<region>.on.aws --schema ./schema
  1. Develop locally with docker compose:
# compose.yml
services:
  postgres:
    image: postgres:17-alpine
    environment:
      POSTGRES_USER: admin
      POSTGRES_PASSWORD: password
      POSTGRES_DB: postgres

  deesql:
    image: ghcr.io/tomodian/deesql:latest
    command: ["proxy", "--listen", ":5432", "--upstream", "postgres:5432"]
    ports:
      - "15432:5432"
    environment:
      POSTGRES_USER: admin
      POSTGRES_PASSWORD: password
      POSTGRES_DB: postgres
    depends_on:
      - postgres
docker compose up -d
psql -h localhost -p 15432 -U admin -d postgres

Go Library

deesql can be imported as a Go library to integrate DSQL compatibility checks and migrations into your own code (e.g. unit tests, CI pipelines).

go get github.com/tomodian/deesql

Verify schema compatibility

Use the verify package to check .sql files or raw SQL strings for DSQL compatibility -- no database connection needed.

import "github.com/tomodian/deesql/verify"

// Check .sql files in directories
violations, err := verify.CheckDirs([]string{"./schema"})

// Check specific files
violations, err := verify.CheckFiles([]string{"./schema/users.sql"})

// Check raw SQL (e.g. in a unit test)
violations := verify.CheckSQL("test.sql", `
    CREATE TABLE users (
        id SERIAL PRIMARY KEY,
        data JSONB
    );
`)
// violations[0].Rule == "SERIAL type not supported (use GENERATED AS IDENTITY)"
// violations[1].Rule == "jsonb type not supported as column type ..."

Plan and apply migrations

Use the migrate package to generate and apply migration plans programmatically. You provide your own *sql.DB connection.

import "github.com/tomodian/deesql/migrate"

// Generate a migration plan
planOut, err := migrate.GeneratePlan(ctx, migrate.GeneratePlanInput{
    DB:          db, // your *sql.DB
    SchemaFiles: []string{"./schema/users.sql"},
})

// Inspect the plan
for _, stmt := range planOut.Plan.Statements {
    fmt.Printf("%s %s: %s\n", stmt.Action, stmt.Resource, stmt.DDL)
}
fmt.Println(migrate.PlanSummary(planOut.Plan))

// Check for hazards
err = migrate.CheckHazards(ctx, migrate.CheckHazardsInput{
    Plan:           planOut.Plan,
    AllowedHazards: []string{"INDEX_BUILD"},
})

// Apply the plan
err = migrate.Apply(ctx, migrate.ApplyInput{
    DB:             db,
    Plan:           planOut.Plan,
    MaxRetries:     5,
    RetryBaseDelay: 2 * time.Second,
})

Example: DSQL compatibility test

func TestSchemaIsDSQLCompatible(t *testing.T) {
    violations, err := verify.CheckDirs([]string{"./schema"})
    require.NoError(t, err)
    for _, v := range violations {
        t.Errorf("%s:%d: %s\n  %s", v.File, v.Line, v.Rule, v.Context)
    }
}

How It Works

Schema Migrations

.sql files ──parse──> Desired Schema ──┐
                                       ├──diff──> Migration Plan ──apply──> DSQL
Live DSQL  ──introspect──> Current Schema──┘
  1. Parse .sql files into a schema model (no SQL execution, no temp database)
  2. Introspect the live DSQL cluster via pg_catalog
  3. Diff the two schemas and generate ordered DDL statements
  4. Apply each statement to the live cluster

Local Proxy

App / psql ──> deesql proxy (:15432) ──> PostgreSQL (:5432)
                    │
               Intercepts SQL
               Blocks unsupported operations
               Returns DSQL-compatible errors (SQLSTATE 0A000)

The proxy speaks the PostgreSQL wire protocol, inspecting Query and Parse messages. Unsupported SQL is rejected immediately with the same error codes Aurora DSQL would return, while allowed SQL is forwarded to the backend.

Output

Plans use Terraform-style change indicators:

Migration plan (3 statement(s)):
------------------------------------------------------------

  + table.orders
    -- Statement 1
    CREATE TABLE orders (...);

  ~ table.users
    -- Statement 2
    ALTER TABLE users ADD COLUMN bio TEXT;

  + index.idx_orders_user_id
    -- Statement 3
    CREATE INDEX ASYNC idx_orders_user_id ON orders (user_id);
    -- ⚠ INDEX_BUILD: Building index idx_orders_user_id asynchronously

------------------------------------------------------------
Plan: 2 to create, 1 to update.
Symbol Action Description
+ Create A new resource will be provisioned
~ Update An existing resource will be modified in place
- Destroy An existing resource will be deleted
+/- Replace A resource will be destroyed and recreated

Commands

plan

Generate and display a migration plan without applying it.

deesql plan --endpoint <endpoint> --schema ./schema

apply

Generate and apply a migration plan.

deesql apply --endpoint <endpoint> --schema ./schema [--force] [--allow-hazards DELETES_DATA,INDEX_BUILD]

verify

Check schema files for Aurora DSQL compatibility (no database connection needed).

deesql verify --schema ./schema

proxy

Start a DSQL-filtering proxy between a PostgreSQL client and backend.

deesql proxy [--listen :15432] [--upstream localhost:5432]

The proxy rewrites CREATE INDEX ASYNC to CREATE INDEX CONCURRENTLY for the PostgreSQL backend, and warns about transaction rule violations (multiple DDL, mixed DDL+DML).

When POSTGRES_USER, POSTGRES_PASSWORD, and POSTGRES_DB environment variables are set, the proxy handles authentication automatically -- apps configured for DSQL IAM auth work without credential changes.

The proxy intercepts and blocks 40+ unsupported SQL patterns including:

  • Unsupported DDL: CREATE DATABASE, CREATE EXTENSION, CREATE TRIGGER, CREATE TYPE, CREATE PROCEDURE, CREATE RULE, CREATE UNLOGGED TABLE, CREATE MATERIALIZED VIEW, CREATE TABLE AS SELECT
  • Table restrictions: INHERITS, PARTITION BY, COLLATE, FOREIGN KEY, EXCLUDE
  • Index restrictions: synchronous CREATE INDEX (must use ASYNC), CONCURRENTLY, non-btree types, ASC/DESC ordering
  • Unsupported statements: TRUNCATE, ALTER SYSTEM, VACUUM, SAVEPOINT, LISTEN/NOTIFY, LOCK TABLE
  • Session restrictions: SET statement_timeout, SET lock_timeout, non-REPEATABLE-READ isolation levels, bare ANALYZE
  • Function restrictions: non-SQL languages (plpgsql, plv8, etc.)

sql

Execute a raw SQL file against Aurora DSQL (for cleanup, seeding, etc.).

deesql sql cleanup.sql --endpoint <endpoint>

Flags

Global

Flag Description Default
--endpoint Aurora DSQL cluster endpoint (required for plan/apply)
--region AWS region auto-detected from endpoint
--user Database user admin
--schema Directories with .sql files (repeatable) (required)
--profile AWS profile name $AWS_PROFILE
--role-arn IAM role ARN to assume (none)
--connect-timeout Connection timeout 10s

apply / sql

Flag Description Default
--allow-hazards Hazard types to permit (apply only) (none)
--force Skip confirmation prompt (apply only) false
--retries Max retries on OCC conflict (SQLSTATE 40001) 5
--retry-delay Initial delay between retries (doubles each attempt) 2s

proxy

Flag Description Default
--listen Address to listen on :15432
--upstream Backend PostgreSQL address localhost:5432

AWS Authentication

Credentials are resolved via the standard AWS SDK default credential chain:

  1. Environment variables (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN)
  2. Shared credentials file (~/.aws/credentials)
  3. AWS config file with --profile or $AWS_PROFILE
  4. IAM role assumption with --role-arn (via STS)
  5. EC2/ECS instance metadata (IMDS)

Supported DDL

Schema file syntax

Feature Syntax Notes
Create table CREATE TABLE name (...) With columns, inline constraints
Create index CREATE [UNIQUE] INDEX ASYNC name ON table (cols) DSQL requires ASYNC
Primary key PRIMARY KEY (cols) Inline or table-level
Unique constraint UNIQUE (cols) Inline or table-level
Check constraint CHECK (expr) Inline or table-level
Identity column GENERATED {ALWAYS|BY DEFAULT} AS IDENTITY Not SERIAL
Default value DEFAULT expr Literals or functions (now(), gen_random_uuid())

Supported column types

Category Types
Text TEXT, VARCHAR(N), CHAR(N)
Numeric INTEGER, BIGINT, SMALLINT, REAL, DOUBLE PRECISION, NUMERIC
Boolean BOOLEAN
Date/Time TIMESTAMPTZ, TIMESTAMP, DATE, TIME, INTERVAL
Binary BYTEA
Other UUID

Migration operations

Operation Supported
Create table Yes
Drop table Yes
Add column Yes
Drop column No (DSQL limitation)
Add/drop index Yes (CREATE INDEX ASYNC)
Add/drop check constraint No (DSQL limitation)
Add/drop unique constraint No (DSQL limitation)
Change column type No (DSQL limitation)
Change NOT NULL No (DSQL limitation)
Change default No (DSQL limitation)
Change primary key No (requires table recreation)

Hazard Types

Hazards warn about potentially dangerous operations:

Type Description
DELETES_DATA Drops a table or column and its data
INDEX_BUILD Async index build in progress
INDEX_DROPPED Dropping an index may degrade query performance

Use --allow-hazards to permit specific types:

deesql apply --endpoint <endpoint> --schema ./schema --allow-hazards DELETES_DATA,INDEX_BUILD

Design Principles

  • Stateless -- No migration history table. The plan is always computed fresh from the diff between desired and live schemas.
  • No temp database -- SQL files are parsed in-process. No need for a secondary PostgreSQL or DSQL cluster.
  • DSQL-native -- Built to complement Aurora DSQL's strengths (async indexes, IAM auth, single-DDL transactions) so you can focus on your application, not migration plumbing.
  • Safe by default -- Hazardous operations require explicit opt-in via --allow-hazards.
  • Local-first -- The proxy brings DSQL's behavior to your local PostgreSQL, so you develop with confidence from day one.

License

MIT

About

Local Aurora DSQL emulator that runs in front of your PostgreSQL. Run migration, test locally, then switch to real DSQL when ready.

Topics

Resources

License

Stars

Watchers

Forks

Packages

 
 
 

Contributors