Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

## [Unreleased]

### Added
- Warehouse presets and guides for Redshift, Postgres, SQL Server, Azure SQL, MySQL, DuckDB,
MotherDuck, and Athena.

## [1.1.0] — 2026-05-18

### Added
Expand Down
11 changes: 10 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,8 @@ uv tool install dbt-spec-kit
dbt-specify --version
```

Supported warehouse presets: `snowflake`, `databricks`, `trino`, and `bigquery`.
Supported warehouse presets: `snowflake`, `databricks`, `trino`, `bigquery`, `redshift`,
`postgres`, `sqlserver`, `azure-sql`, `mysql`, `duckdb`, `motherduck`, and `athena`.

## What init adds

Expand Down Expand Up @@ -145,6 +146,14 @@ Use `dbt-specify ci` when the lifecycle and dbt artifact checks should block a P
- [Databricks guide](docs/warehouse-guides/databricks.md)
- [Trino guide](docs/warehouse-guides/trino.md)
- [BigQuery guide](docs/warehouse-guides/bigquery.md)
- [Redshift guide](docs/warehouse-guides/redshift.md)
- [Postgres guide](docs/warehouse-guides/postgres.md)
- [SQL Server guide](docs/warehouse-guides/sqlserver.md)
- [Azure SQL guide](docs/warehouse-guides/azure-sql.md)
- [MySQL guide](docs/warehouse-guides/mysql.md)
- [DuckDB guide](docs/warehouse-guides/duckdb.md)
- [MotherDuck guide](docs/warehouse-guides/motherduck.md)
- [Athena guide](docs/warehouse-guides/athena.md)

## OSS project

Expand Down
5 changes: 3 additions & 2 deletions docs/getting-started.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,8 @@ This creates:
- `CLAUDE.md` — the agent orientation file (or `CLAUDE.md.dbt-specify-suggested` if you already have one)
- `specs/` — empty directory for your first spec

Supported warehouse presets: `snowflake`, `databricks`, `trino`, and `bigquery`.
Supported warehouse presets: `snowflake`, `databricks`, `trino`, `bigquery`, `redshift`,
`postgres`, `sqlserver`, `azure-sql`, `mysql`, `duckdb`, `motherduck`, and `athena`.

## Compose with dbt-labs/dbt-agent-skills

Expand Down Expand Up @@ -102,5 +103,5 @@ dbt-specify doctor
- Use the [team onboarding playbook](team-onboarding-playbook.md) when presenting the workflow to an analytics engineering team.
- Read [methodology.md](methodology.md) for the full four-phase loop.
- Read [skills-and-sub-agents.md](skills-and-sub-agents.md) for dbt Labs skills composition and safe sub-agent delegation.
- Read [warehouse-guides/snowflake.md](warehouse-guides/snowflake.md) or [databricks.md](warehouse-guides/databricks.md) for your warehouse's preset.
- Read the [warehouse guides](warehouse-guides/snowflake.md) for preset-specific planning rules.
- See [`examples/jaffle-shop-staging-overhaul/`](../examples/jaffle-shop-staging-overhaul/) for a complete worked example.
26 changes: 26 additions & 0 deletions docs/warehouse-guides/athena.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
# Athena guide

Use this preset when dbt targets Amazon Athena and S3-backed tables.

## What the preset adds

- S3 table and file-layout planning
- partitioning and partition-projection decisions
- Glue catalog and Lake Formation/IAM governance review
- workgroup, output location, and scan-cost guardrails
- Iceberg versus Hive-style table format decisions

## Good fit

- lakehouse tables queried through Athena
- Iceberg or external Hive-style datasets
- teams that need scan-cost and S3 layout evidence in PRs

## Use

```bash
dbt-specify init analytics --warehouse athena
```

dbt-spec-kit does not query Athena. dbt, the adapter, AWS credentials, and workgroup configuration
remain outside this toolkit.
25 changes: 25 additions & 0 deletions docs/warehouse-guides/azure-sql.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
# Azure SQL guide

Use this preset when dbt targets Azure SQL Database or Azure SQL Managed Instance.

## What the preset adds

- service tier, elastic pool, CPU, IO, and log-pressure planning
- Query Store evidence expectations for performance-sensitive work
- index, columnstore, and incremental load guardrails
- Microsoft Entra, role, RLS, dynamic masking, and view-boundary review
- firewall, private endpoint, and deployment-window callouts

## Good fit

- Azure-native enterprise teams
- shared elastic pool environments
- governed marts using Azure SQL serving layers

## Use

```bash
dbt-specify init analytics --warehouse azure-sql
```

This is a planning preset. It does not configure Azure credentials, networking, or dbt profiles.
6 changes: 6 additions & 0 deletions docs/warehouse-guides/bigquery.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,12 @@
The BigQuery preset adds planning requirements for partitioning, clustering, materialization, cost,
policy tags, authorized views, and BI Engine expectations.

Use it with:

```bash
dbt-specify init analytics --warehouse bigquery
```

## Partitioning

Large fact-like tables should partition by the date or timestamp column that bounds routine
Expand Down
26 changes: 26 additions & 0 deletions docs/warehouse-guides/duckdb.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
# DuckDB guide

Use this preset when dbt targets DuckDB for local analytics, CI fixtures, or file-based workflows.

## What the preset adds

- local file and object-store path review
- memory and single-process limits
- DuckDB extension planning
- Parquet staging and export expectations
- governance handling for local files and CI artifacts

## Good fit

- local development examples
- lightweight analytics projects
- CI smoke tests with deterministic data files

## Use

```bash
dbt-specify init analytics --warehouse duckdb
```

This preset is useful for AI-agent demos because it makes local file dependencies and reproducibility
explicit.
26 changes: 26 additions & 0 deletions docs/warehouse-guides/motherduck.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
# MotherDuck guide

Use this preset when dbt targets MotherDuck or a hybrid DuckDB/MotherDuck workflow.

## What the preset adds

- local versus cloud execution boundary review
- database, share, role, and collaborator planning
- file and object-store reproducibility checks
- cost/quota guardrails
- sensitive data movement review

## Good fit

- collaborative DuckDB-style analytics
- shared demo and lightweight team projects
- local-to-cloud workflows where data movement must be explicit

## Use

```bash
dbt-specify init analytics --warehouse motherduck
```

The preset does not configure MotherDuck authentication. It only adds planning and governance
guidance.
25 changes: 25 additions & 0 deletions docs/warehouse-guides/mysql.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
# MySQL guide

Use this preset when dbt reads from or writes to MySQL and operational safety matters.

## What the preset adds

- primary versus replica versus analytics database review
- index and `EXPLAIN` expectations for heavy joins
- incremental watermark, batch size, and lock planning
- engine, charset, collation, and timezone assumptions
- grants and PII access review

## Good fit

- operational-source staging projects
- MySQL replicas used for analytics
- teams that need to prevent AI agents from overloading OLTP databases

## Use

```bash
dbt-specify init analytics --warehouse mysql
```

dbt-spec-kit does not run MySQL queries. dbt and the selected adapter own execution.
25 changes: 25 additions & 0 deletions docs/warehouse-guides/postgres.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
# Postgres guide

Use this preset when dbt targets Postgres or uses Postgres as an analytics-safe serving database.

## What the preset adds

- OLTP safety checks for primary versus replica versus analytics database
- index and materialization planning
- lock and transaction impact review
- schema/grant/access planning
- extension and non-portable SQL callouts

## Good fit

- smaller analytics projects on Postgres
- product analytics replicas
- local or self-managed dbt deployments where lock safety matters

## Use

```bash
dbt-specify init analytics --warehouse postgres
```

The preset does not connect to Postgres. dbt and its adapter handle database execution.
26 changes: 26 additions & 0 deletions docs/warehouse-guides/redshift.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
# Redshift guide

Use this preset when dbt runs on Amazon Redshift or Redshift is the analytics serving database.

## What the preset adds

- distribution and sort key planning
- vacuum and analyze expectations
- Spectrum/external table scan boundaries
- workload queue and full-refresh cost guardrails
- schema, grants, late binding view, and restricted-data review

## Good fit

- enterprise marts on Redshift
- Redshift Spectrum plus internal table joins
- teams that need PR evidence for dist/sort decisions

## Use

```bash
dbt-specify init analytics --warehouse redshift
```

dbt still owns execution through the Redshift adapter and the user's normal profile. This preset only
changes planning guidance and generated agent context.
26 changes: 26 additions & 0 deletions docs/warehouse-guides/sqlserver.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
# SQL Server guide

Use this preset when dbt targets Microsoft SQL Server.

## What the preset adds

- clustered, nonclustered, and columnstore index planning
- incremental load and transaction log guardrails
- schema, owner, role, and access-path review
- tempdb and concurrency risk checks
- T-SQL compatibility, collation, and date precision callouts

## Good fit

- SQL Server analytics marts
- enterprise teams with strict schema and role ownership
- incremental models where blocking and log pressure matter

## Use

```bash
dbt-specify init analytics --warehouse sqlserver
```

The preset guides planning. dbt adapter configuration and database credentials remain outside
dbt-spec-kit.
24 changes: 24 additions & 0 deletions presets/athena/constitution-additions.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
## Article A1 — S3 file layout is part of model design

Athena models document table format, file format, target file size, and small-file risk. Plans for
large tables explain how files are compacted or kept query-efficient.

## Article A2 — Partitioning and projection are explicit

Large fact-like tables document partition columns, partition projection usage, and expected partition
pruning. Unpartitioned large scans require an approved justification.

## Article A3 — Glue and Lake Formation governance is reviewed

Plans identify Glue catalog/database/table ownership and Lake Formation or IAM access boundaries for
governed outputs.

## Article A4 — Workgroups and query cost guardrails are required

Plans name the Athena workgroup, expected scanned data, output location, and cost guardrails for
large models.

## Article A5 — Iceberg versus Hive table format is deliberate

Plans choose Iceberg, Hive-style external tables, or another format deliberately and document
incremental, compaction, and schema evolution behavior.
25 changes: 25 additions & 0 deletions presets/athena/plan-additions.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
## Athena-specific concerns

### Table and file layout

| Model | Table format | File format/size | Small-file risk |
|---|---|---|---|
| <model> | Iceberg / Hive external | Parquet/ORC/other, <MB> | low/medium/high |

### Partitioning and pruning

| Model | Partition columns | Projection used? | Expected scan boundary |
|---|---|---|---|
| <model> | <cols> | yes/no | <predicate> |

### Governance

| Output | Glue database/table | Lake Formation/IAM boundary | Sensitive? |
|---|---|---|---|
| <model> | <db.table> | <policy/role> | yes/no |

### Workgroup and cost

| Job | Workgroup | Output location | Scan guardrail |
|---|---|---|---|
| <job> | <workgroup> | s3://... | <limit/mitigation> |
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
---
name: athena-partition-file-layout-decisions
description: Use when planning Athena dbt models that depend on S3 layout, partitions, Glue/Lake Formation governance, and scan cost.
---

# Athena partition and file-layout decisions

## When to use this skill

Use this for Athena-backed dbt models, especially large tables, Iceberg tables, external sources, or
governed S3 data.

## Decision routine

1. Choose table format deliberately: Iceberg for managed evolution/upserts, Hive-style external
tables for simple append/read patterns.
2. Prefer columnar files such as Parquet or ORC.
3. Document partition columns and expected pruning predicates.
4. Check small-file risk and compaction expectations.
5. Name Glue database/table ownership and Lake Formation or IAM access boundaries.
6. Name workgroup, query result location, and scan cost guardrails.

## Common failures

- Partitioning by a high-cardinality column that creates too many S3 prefixes.
- Generating many tiny files that make every query expensive.
- Running unbounded scans because partition projection was not planned.
- Publishing governed outputs without Glue/Lake Formation ownership review.
24 changes: 24 additions & 0 deletions presets/azure-sql/constitution-additions.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
## Article AZ1 — Service tier and workload impact are documented

Plans identify Azure SQL service tier, elastic pool impact, expected workload, and whether the change
can cause CPU, IO, or log pressure.

## Article AZ2 — Query Store evidence guides performance changes

Performance-sensitive changes document expected query patterns and use Query Store or equivalent
evidence after deployment when available.

## Article AZ3 — Index, columnstore, and incremental choices are explicit

Models document clustered index, nonclustered indexes, columnstore usage, incremental strategy, and
transaction log impact.

## Article AZ4 — Security uses Azure-native controls

Governed outputs document Microsoft Entra authentication assumptions, roles, row-level security,
dynamic data masking, or views used for access boundaries.

## Article AZ5 — Operational limits are respected

Plans call out firewall/private endpoint assumptions, cross-database access, tempdb pressure, and
deployment windows for full refreshes.
Loading
Loading