Config scripts for ASQL Mirrored database ELT pattern - #80
Merged
Conversation
Update L1 temp table inserts for Fabric compute Changed [ComputePath] from 'L1Transform' to NULL in all L1 temp table INSERTs. Updated [ComputeName] from 'L1Transform-Generic-Synapse' to 'L1Transform-Generic-Fabric' in AzureRestAPI and PurviewRestAPI scripts to reflect migration to Fabric compute environment.
…2 transformations
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request introduces support for ingesting and transforming data from an Azure SQL Mirror source system ("WWI-mirror") into the ELT pipeline. It adds new post-deployment scripts and updates existing indexes and deployment logic to accommodate the new data source. The changes also generalize some transformation compute names and update index definitions to include the
OutputDWTablecolumn for uniqueness.Azure SQL Mirror integration:
ASQLMirrorfolder and three post-deployment scripts:ASQLMirror_IngestDefinition.sql,ASQLMirror_L1TransformDefinition.sql, andASQLMirror_L2TransformDefinition.sql, which define ingestion and transformation logic for mirrored Azure SQL tables and ensure idempotent (re-runnable) deployment usingMERGEstatements. [1] [2] [3] [4] [5]Script.PostDeployment.sql) to include and execute the new ASQLMirror scripts, enabling their automatic application during deployments.Index and schema updates:
L1TransformDefinition,L1TransformInstance,L2TransformDefinition, andL2TransformInstanceto include theOutputDWTablecolumn, ensuring uniqueness when multiple output tables are involved. [1] [2] [3] [4]Generalization and consistency improvements:
'L1Transform-Generic-Synapse'to'L1Transform-Generic-Fabric'and setComputePathtoNULLfor consistency with new patterns. [1] [2] [3]