dbt-mesh-platform

A production-grade dbt Mesh reference implementation on Snowflake Sample Data (TPC-H), built to accompany the Medium blog series:

"AI-Augmented dbt Best Practices at the Data Platform Level" by Kiran Pothina · TechMinion Academy

Blog series

#	Post	Status
1	Scaffolding a Production dbt Project in 60 Seconds with AI	✅ Published
2	Four AI Agents That Keep Your dbt Project Honest	✅ Published
3	Scaling dbt Across Teams — Mesh, Contracts and Versioning	✅ Published
4	One Truth, Five Teams — Governing the dbt Semantic Layer	✅ Published
5	Ship Fast, Break Nothing — CI/CD, Monitoring and Model Health	✅ Published

What's in this repo

Component	Description	Post
`agents/generate_boilerplate.py`	Scaffolds full staging layer from INFORMATION_SCHEMA in < 60s	1
`agents/description_agent.py`	Enriches model + column descriptions using Claude API	2
`agents/test_agent.py`	Profiles Snowflake views and generates 74 data tests	2
`agents/lineage_agent.py`	Reads manifest.json · flags blast radius · orphaned sources	2
`agents/constraint_agent.py`	Adds contracts · freshness SLAs · accepted values	2
`agents/scorecard.py`	Scores all models across 5 health dimensions · avg 80.4	5
`dbt_platform/`	8 TPC-H staging views · source YAML · shared macros	1–2
`dbt_commercial/`	fct_orders_v1 · dim_customers_v1 · versioned public contracts	3
`dbt_finance/`	fct_revenue_v1 · monthly revenue stub · reads commercial	3–4
`dbt_product/`	fct_usage_v1 · feature usage proxy · reads platform	3–4
`dbt_marketing/`	fct_attribution_v1 · segment attribution · reads commercial	3–4
`dbt_analytics/`	fct_orders_enriched · MetricFlow time spine · 8 metrics	4
`.github/workflows/`	12 GitHub Actions workflows — slim CI + deploy per domain	5
`governance/`	Snowflake setup SQL · roles · databases · cross-domain grants	All

Project topology

dbt-mesh-platform/
├── .github/workflows/     ← 12 CI/CD workflow files (slim CI + deploy × 6 domains)
├── governance/            ← Snowflake setup SQL · cross-domain grants
├── agents/                ← 6 AI agent scripts
├── profiles.yml.example   ← Connection profiles for all six domain projects
│
├── dbt_platform/          ← Platform team · TPC-H staging · shared macros
├── dbt_commercial/        ← Commercial team · fct_orders_v1 · dim_customers_v1
├── dbt_finance/           ← Finance team · fct_revenue_v1
├── dbt_product/           ← Product team · fct_usage_v1
├── dbt_marketing/         ← Marketing team · fct_attribution_v1
└── dbt_analytics/         ← Analytics team · cross-domain marts · 8 MetricFlow metrics

Each domain project is independent with its own Snowflake database, warehouse, role, and CI/CD pipeline.

Key metrics

Metric	Value
dbt models	24 across 6 projects
Data tests	74 (auto-generated by test agent)
Build result	PASS=83 WARN=0 ERROR=0
MetricFlow metrics	8 (gross_revenue · net_revenue · discount_rate · AOV · active_customers · …)
CI/CD workflows	12 (slim CI + deploy per domain)
Scorecard avg	80.4 — A:18 · B:1 · C:5 · D:0
PR feedback time	3–8 minutes (vs 45 min full rebuild)

Architecture

SNOWFLAKE_SAMPLE_DATA (TPC-H)
        ↓ source()
  dbt_platform          8 staging views · 74 tests
        ↓ source('platform', …)
  ┌─────┼──────────┬──────────┐
  ↓     ↓          ↓          ↓
finance  product  marketing  commercial
  fct_   fct_     fct_       fct_orders_v1
  revenue usage   attribution dim_customers_v1
  _v1    _v1      _v1        (public · enforced · versioned)
        ↓ source() all domain contracts
  dbt_analytics
  fct_orders_enriched (1.5M rows · cross-domain join)
  metricflow_time_spine (1,855 days)
        ↓
  8 MetricFlow metrics → mf query / BI tools / Python

Snowflake setup

Full setup scripts are in governance/. Run them in this order:

Step 1 — One-time infrastructure

-- Run as ACCOUNTADMIN
-- governance/snowflake_setup.sql
-- Creates: 6 warehouses · 12 roles · 12 databases · all grants

Domain	Dev database	Prod database	Warehouse
Platform	`PLATFORM_DEV`	`PLATFORM_PROD`	`PLATFORM_WH`
Commercial	`COMMERCIAL_DEV`	`COMMERCIAL_PROD`	`COMMERCIAL_WH`
Finance	`FINANCE_DEV`	`FINANCE_PROD`	`FINANCE_WH`
Product	`PRODUCT_DEV`	`PRODUCT_PROD`	`PRODUCT_WH`
Marketing	`MARKETING_DEV`	`MARKETING_PROD`	`MARKETING_WH`
Analytics	`ANALYTICS_DEV`	`ANALYTICS_PROD`	`ANALYTICS_WH`

Step 2 — RSA key (once per developer)

mkdir -p ~/.dbt
openssl genrsa 2048 | openssl pkcs8 -topk8 -inform PEM \
  -nocrypt -out ~/.dbt/rsa_key.p8
openssl rsa -in ~/.dbt/rsa_key.p8 -pubout -out ~/.dbt/rsa_key.pub
chmod 600 ~/.dbt/rsa_key.p8

Register in Snowflake:

ALTER USER <your_username> SET RSA_PUBLIC_KEY='<contents of rsa_key.pub>';

Step 3 — Environment variables

export SNOWFLAKE_ACCOUNT="abc12345.us-east-1"
export SNOWFLAKE_USER="your_snowflake_username"
export SNOWFLAKE_PRIVATE_KEY_PATH="$HOME/.dbt/rsa_key.p8"
export SNOWFLAKE_PRIVATE_KEY_PASSPHRASE=""   # leave empty if no passphrase
export DBT_USER_SCHEMA="yourname"            # scopes dev schemas: yourname_staging

Step 4 — Profiles

cp profiles.yml.example ~/.dbt/profiles.yml

Target	Database	Schema	When used
`dev`	`PLATFORM_DEV`	`kiran_staging`	Local development
`ci`	`PLATFORM_DEV`	`pr_12_staging`	GitHub Actions PR
`prod`	`PLATFORM_PROD`	`staging`	Merge to main

Quick start

git clone https://github.com/TechPopsicles/dbt-mesh-platform.git
cd dbt-mesh-platform

pip install dbt-core==1.11.8 dbt-snowflake==1.11.4
pip install -r agents/requirements.txt

# Build platform staging layer
cd dbt_platform && dbt deps && dbt build

# Build commercial domain (reads from platform)
export PLATFORM_DATABASE=PLATFORM_DEV
export PLATFORM_SCHEMA=KIRAN_STAGING
cd ../dbt_commercial && dbt deps && dbt build

# Run the AI agents (Post 2)
cd ..
python agents/description_agent.py --project dbt_platform
python agents/test_agent.py --project dbt_platform
python agents/lineage_agent.py --manifest dbt_platform/target/manifest.json
python agents/constraint_agent.py --project dbt_platform

# Run the scorecard across all projects (Post 5)
python agents/scorecard.py --all

AI agents

`generate_boilerplate.py` — Post 1

Connects to Snowflake, reads INFORMATION_SCHEMA, generates complete staging layer in under 60 seconds.

python agents/generate_boilerplate.py \
  --database SNOWFLAKE_SAMPLE_DATA \
  --schema   TPCH_SF1 \
  --source   tpch \
  --project  dbt_platform \
  --out-dir  dbt_platform/models/staging/tpch

`description_agent.py` — Post 2

Calls Claude API to enrich every model and column with business context descriptions.

`test_agent.py` — Post 2

Profiles actual Snowflake staging views and generates data tests in dbt 1.9+ format with arguments: nesting.

`lineage_agent.py` — Post 2

Reads manifest.json and flags: orphaned sources · untested models · high blast radius · thin descriptions. Writes agents/reports/lineage_report.md + lineage_report.json.

`constraint_agent.py` — Post 2

Adds contract: enforced · freshness SLAs · accepted_values tests. Reads and writes YAML only.

`scorecard.py` — Post 5

Scores every model on 5 dimensions (20 pts each):

Dimension	What it checks
Documentation	Model description + % columns described
Test coverage	Data tests per column
Contracts	`enforced: true` + `data_type` on all columns
Freshness	Upstream source freshness config
Blast radius	Downstream model count (penalised above threshold)

python agents/scorecard.py --all                    # score all 6 projects
python agents/scorecard.py --manifest dbt_platform/target/manifest.json
python agents/scorecard.py --all --output json      # → agents/reports/scorecard_report.json

dbt_commercial — public contracts

Two versioned public contracts consumed by Finance, Marketing, and Analytics:

# fct_orders_v1  →  FINANCE_DEV_ROLE · ANALYTICS_DEV_ROLE
# dim_customers_v1  →  MARKETING_DEV_ROLE · ANALYTICS_DEV_ROLE
config:
  contract:
    enforced: true
  grants:
    select: ['FINANCE_DEV_ROLE', 'ANALYTICS_DEV_ROLE']

Versioning demo — fct_orders_v2 renames gross_revenue to total_revenue:

cd dbt_commercial
dbt parse  # fires: "fct_orders.v1 slated for deprecation on 2026-07-08"
dbt build  # both v1 and v2 coexist during migration window

dbt_analytics — semantic layer

8 MetricFlow metrics defined in models/marts/core/metrics/:

cd dbt_analytics
dbt deps && dbt build

# List all metrics
mf list metrics

# Query metrics
mf query --metrics gross_revenue --group-by metric_time__month --order metric_time__month
mf query --metrics gross_revenue,net_revenue --group-by order__market_segment
mf query --metrics avg_order_value --group-by order__customer_tier
mf query --metrics discount_rate --group-by order__market_segment

CI/CD — slim model pattern

12 GitHub Actions workflows in .github/workflows/. Each domain has:

{domain}_slim_ci.yml — fires on PR, builds only state:modified+
{domain}_deploy.yml — fires on merge to main, full prod build

Four-stage slim CI pattern (see Post 5 for full explanation):

# Stage 1: full git history
fetch-depth: 0

# Stage 2: state manifest from live prod (no stale S3 artifacts)
git checkout main → dbt parse --target prod → state/manifest.json

# Stage 3: zero-copy clone for incremental correctness
dbt clone --select "state:modified+,config.materialized:incremental,state:old"

# Stage 4: slim build
dbt build --select "state:modified+" --defer --state state

PR schema pattern: COMMERCIAL_DEV.pr_47_staging · COMMERCIAL_DEV.pr_47_marts

Schema naming

{DOMAIN}_{ENV}.{PREFIX}_{LAYER}

PLATFORM_DEV.KIRAN_STAGING    ← dev   (DBT_USER_SCHEMA=kiran)
PLATFORM_DEV.PR_12_STAGING    ← CI    (DBT_USER_SCHEMA=pr_12)
PLATFORM_PROD.STAGING         ← prod  (DBT_USER_SCHEMA='')

Repository conventions

Convention	Pattern	Example
Database	`{DOMAIN}_{ENV}`	`FINANCE_DEV`
Warehouse	`{DOMAIN}_WH`	`FINANCE_WH`
Role	`{DOMAIN}_{ENV}_ROLE`	`FINANCE_DEV_ROLE`
Schema	`{PREFIX}_{LAYER}`	`KIRAN_STAGING`
Staging model	`stg_{source}__{entity}`	`stg_tpch__orders`
Intermediate	`int_{verb}_{entity}`	`int_order_items`
Fact mart	`fct_{event}_v{n}`	`fct_orders_v2`
Dimension mart	`dim_{entity}_v{n}`	`dim_customers_v1`

Tech stack

Tool	Version
dbt-core	1.11.8
dbt-snowflake	1.11.4
Snowflake	TPC-H SF1 sample data
MetricFlow	0.13.0
Python	3.12
Claude API	claude-sonnet-4-6
CI/CD	GitHub Actions

License: MIT

Navigation Menu

Repository files navigation

dbt-mesh-platform

Blog series

What's in this repo

Project topology

Key metrics

Architecture

Snowflake setup

Step 1 — One-time infrastructure

Step 2 — RSA key (once per developer)

Step 3 — Environment variables

Step 4 — Profiles

Quick start

AI agents

`generate_boilerplate.py` — Post 1

`description_agent.py` — Post 2

`test_agent.py` — Post 2

`lineage_agent.py` — Post 2

`constraint_agent.py` — Post 2

`scorecard.py` — Post 5

dbt_commercial — public contracts

dbt_analytics — semantic layer

CI/CD — slim model pattern

Schema naming

Repository conventions

Tech stack

About

Releases

Packages

Contributors

Languages

Name		Name	Last commit message	Last commit date
Latest commit History 15 Commits
.github/workflows		.github/workflows
agents		agents
dbt_analytics		dbt_analytics
dbt_commercial		dbt_commercial
dbt_finance		dbt_finance
dbt_marketing		dbt_marketing
dbt_platform		dbt_platform
dbt_product		dbt_product
governance		governance
.gitignore		.gitignore
README.md		README.md
profiles.yml.example		profiles.yml.example

Folders and files

Latest commit

History

Repository files navigation

dbt-mesh-platform

Blog series

What's in this repo

Project topology

Key metrics

Architecture

Snowflake setup

Step 1 — One-time infrastructure

Step 2 — RSA key (once per developer)

Step 3 — Environment variables

Step 4 — Profiles

Quick start

AI agents

generate_boilerplate.py — Post 1

description_agent.py — Post 2

test_agent.py — Post 2

lineage_agent.py — Post 2

constraint_agent.py — Post 2

scorecard.py — Post 5

dbt_commercial — public contracts

dbt_analytics — semantic layer

CI/CD — slim model pattern

Schema naming

Repository conventions

Tech stack

About

Topics

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages

`generate_boilerplate.py` — Post 1

`description_agent.py` — Post 2

`test_agent.py` — Post 2

`lineage_agent.py` — Post 2

`constraint_agent.py` — Post 2

`scorecard.py` — Post 5