Mirror elt pattern - #74
Merged
Merged
Conversation
Co-authored-by: bennyaustin <20385499+bennyaustin@users.noreply.github.com> Agent-Logs-Url: https://github.com/bennyaustin/elt-framework/sessions/0451983c-05cf-4c13-b039-33032b8494b3
Co-authored-by: bennyaustin <20385499+bennyaustin@users.noreply.github.com> Agent-Logs-Url: https://github.com/bennyaustin/elt-framework/sessions/5e9a1692-7aec-4a2e-8411-abb49452ccf9
… clause Co-authored-by: bennyaustin <20385499+bennyaustin@users.noreply.github.com> Agent-Logs-Url: https://github.com/bennyaustin/elt-framework/sessions/ca0f9d41-5104-418e-ac7c-35925626eda9
…nches Co-authored-by: bennyaustin <20385499+bennyaustin@users.noreply.github.com> Agent-Logs-Url: https://github.com/bennyaustin/elt-framework/sessions/4893a51d-92e0-4576-9ea8-7144c15a2f5b
Replace non-SARGable OR predicates with separate file/table query branches in InsertTransformInstance_L1
Fix NULL-unsafe WHERE predicates in InsertIngestInstance
Fix NULL-unsafe predicates in InsertIngestInstance WHERE clauses
Fix NULL-unsafe WHERE predicates in InsertTransformInstance_L1 UPDATE clause
Merge pull request #68 from bennyaustin/Mirror-ELT-Pattern
Added a check constraint to L1TransformInstance to enforce mutually exclusive input source specification (file or table). Updated UI_IngestInstance and UI_L1TransformDefinition indexes to be UNIQUE. Included the new constraint script in the project for deployment.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request introduces several improvements to the ELT ControlDB schema and procedures, focusing on enforcing data integrity, enhancing uniqueness constraints, and optimizing how input sources are handled for transformations. The changes include new check constraints, updates to unique indexes, improved handling of NULL comparisons for matching records, and more efficient logic in stored procedures for distinguishing between file-based and table-based inputs.
Schema and Data Integrity Improvements:
CC_L1TransformInstance_InputSourceon the[ELT].[L1TransformInstance]table to ensure that either file-based or table-based input fields are set, but not both. [1] [2]UI_IngestInstanceandUI_L1TransformDefinitionto be unique, enforcing data uniqueness at the database level. [1] [2]Stored Procedure Logic Enhancements:
InsertTransformInstance_L1.sqlto separate logic for file-based and table-based inputs, improving SARGability and efficiency. Existence checks and updates now use equality predicates instead of complex OR/NULL logic, making queries faster and more reliable. [1] [2]InsertTransformInstance_L1.sqlto match records more precisely using explicit NULL-safe comparisons, reducing the risk of incorrect updates.Improved NULL Handling in Record Matching:
InsertIngestInstance.sqlto use explicit NULL-safe equality checks when matching on destination fields, ensuring correct identification of existing records and avoiding issues with NULL comparisons. [1] [2] [3]