Skip to content

Config scripts for ASQL Mirrored database ELT pattern - #80

Merged
bennyaustin merged 2 commits into
mainfrom
Mirror-ELT-Pattern
Apr 19, 2026
Merged

Config scripts for ASQL Mirrored database ELT pattern#80
bennyaustin merged 2 commits into
mainfrom
Mirror-ELT-Pattern

Conversation

@bennyaustin

Copy link
Copy Markdown
Owner

This pull request introduces support for ingesting and transforming data from an Azure SQL Mirror source system ("WWI-mirror") into the ELT pipeline. It adds new post-deployment scripts and updates existing indexes and deployment logic to accommodate the new data source. The changes also generalize some transformation compute names and update index definitions to include the OutputDWTable column for uniqueness.

Azure SQL Mirror integration:

  • Added a new ASQLMirror folder and three post-deployment scripts: ASQLMirror_IngestDefinition.sql, ASQLMirror_L1TransformDefinition.sql, and ASQLMirror_L2TransformDefinition.sql, which define ingestion and transformation logic for mirrored Azure SQL tables and ensure idempotent (re-runnable) deployment using MERGE statements. [1] [2] [3] [4] [5]
  • Updated the main post-deployment script (Script.PostDeployment.sql) to include and execute the new ASQLMirror scripts, enabling their automatic application during deployments.

Index and schema updates:

  • Modified unique index definitions for L1TransformDefinition, L1TransformInstance, L2TransformDefinition, and L2TransformInstance to include the OutputDWTable column, ensuring uniqueness when multiple output tables are involved. [1] [2] [3] [4]

Generalization and consistency improvements:

  • Changed the compute name for L1 transforms in RestAPI and UploadFiles sources from 'L1Transform-Generic-Synapse' to 'L1Transform-Generic-Fabric' and set ComputePath to NULL for consistency with new patterns. [1] [2] [3]

#
Update L1 temp table inserts for Fabric compute

Changed [ComputePath] from 'L1Transform' to NULL in all L1 temp table INSERTs. Updated [ComputeName] from 'L1Transform-Generic-Synapse' to 'L1Transform-Generic-Fabric' in AzureRestAPI and PurviewRestAPI scripts to reflect migration to Fabric compute environment.
@bennyaustin
bennyaustin merged commit cdcc4f1 into main Apr 19, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant