Add Sensitive Tier destination table pipeline with configurable participant eligibility filtering and suppression - #30
Open
gloria-trivitt wants to merge 67 commits into
Open
Add Sensitive Tier destination table pipeline with configurable participant eligibility filtering and suppression#30gloria-trivitt wants to merge 67 commits into
gloria-trivitt wants to merge 67 commits into
Conversation
…ransfer them to a new dataset table, and apply an initial hardcoded filter via a CTE.
- Generate SQL programmatically using CTE-based filtering - Replace hardcoded filters with configurable filter_profiles - Support dynamic base and join table selection via JSON config - Enforce non-null Connect_ID sourced from base table - Add schema validation and missing column reporting - Export generated SQL and reports to GCS - Improve logging and error handling across the pipeline
…the classification data
…tion and update subset config for testing
…d of after loading it
Prevents destroy=No/withdraw=Yes/Revoke=No rows from being misclassified as DATA_QUALITY_EXCLUSION when they should be UNDEFINED. Updates synthetic participant TEST_EXCLUSION_WITHDRAW_NO_TS to include revoke=Yes so the test actually covers this clause.
Previously, render_subset_expression emitted a full CASE/WHEN block for every censored output column, repeating the same status/timestamp/ cutoff condition once per column that shared a gate (e.g. the same module2 condition was duplicated 3x, menstrual 6x, etc.). This made the generated SQL large and repetitive, and meant BigQuery had to re-evaluate identical boolean logic multiple times per row. This commit moves that logic into the CTE as a set of precomputed boolean flags (one per unique status_col), built by the new build_unique_eligibility_rules() helper. Each censored column now just references its flag via IF(flag, col, NULL) instead of restating the full condition. build_unique_eligibility_rules() dedupes MODULE_CENSOR_RULES by status_col (since some CIDs, e.g. the two menstrual survey CIDs, legitimately share one gate) and raises ValueError if two CIDs share a status_col but disagree on completion_ts/cutoff, to catch config mistakes that would otherwise silently miscompute eligibility for consent-sensitive columns. Verified equivalent output against the synthetic dataset via bidirectional EXCEPT DISTINCT between the old and new generated tables (zero rows either direction).
…nfer INT64, so all columns in the generated table have a consistent type
…ssified as UNDEFINED)
…ons, so generated SQL has deterministic column order across runs
…DEFINED), setting ehr_cutoff to withdraw_ts since there is no ambiguity window when the two timestamps are equal
…avior and exercise the cutoff boundary across four modules
…logging statement
- Rename _build_censorship_case_block -> _build_reason_cte_block; the block now keys on status_col instead of column_name, so CIDs that share a status_col (e.g. the two menstrual survey CIDs) collapse to a single reason block instead of generating identical CASE logic twice. - Join deduped reasons back to every output column via a new column_status_map CTE, so each column (including duplicate CIDs) still gets its own row in the final result. - Pass classification_table as a real parameter to _build_reason_cte_block instead of templating it via a placeholder + string.replace(). - Reword reason messages: quote status values, add "(empty string)" annotation, distinguish NULL vs "too late" as separate sub-reasons.
Extract render_eligibility_condition() as the single source of truth for
"is {status_col} eligible" used both to build the *_eligible flags in
build_subset_query's CTE and to gate the censored/not-censored branch in
_build_reason_cte_block. Previously these were two hand-written copies of
the same condition that could silently drift apart; now a change to one
always propagates to the other.
…ilder and update callers
…lls through the censorship module
- build_censorship_summary_sql: own the full CREATE OR REPLACE TABLE
(...) statement, including header comment, instead of returning a
bare SELECT for transforms.py to wrap
- build_censorship_summary_sql: add output_table param, use it in
"/* Censorship summary query for {output_table} -> {destination_table} */"
header comment
- create_censorship_summary_table (transforms.py): drop the local
CREATE TABLE wrapper and pass output_table/destination_table through
to build_censorship_summary_sql directly, matching the pattern used
by destination_table_builder.build_destination_table_query
…ter (allowed_cases is not empty)
…t use the classification_filter
…bles without a classification_filter, instead of generating invalid SQL
…ification_filter configured
Contributor
Author
Current scope: required changesThese edits are applicable to the
Edits specific to
|
…M, BU, and Mouthwash modules and timestamps are commented out
…cross classify_participants, the destination table query builder, the censorship summary SQL, Flask endpoints, and destination_config.json (allowed_cases -> allowed_consent_groups)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR builds the pipeline that produces the Sensitive Tier destination table(s) and their suppression summaries. Participant eligibility/suppression is applied per destination table via a
classification_filterconfig flag. Tables that do not need consent-based filtering can omit it and skip that part of the pipeline entirely.The workflow performs the following steps:
filter_profile. A table can usefilter_profile: "no_filters"to skip this step.Connect_ID IS NOT NULLis always enforced automatically, regardless of the selected profile.classification_filter:constants.pyclassification_filter, none of the three items above are applied; the table is built as a plain, filtered (perfilter_profile) subset of its base table(s), and columns that would otherwise be suppressed pass through unchanged.NULLby default).classification_filterconfigured, creates a suppression summary table to explain why a column was suppressed. Tables without aclassification_filterskip this step entirely; there is no suppression to summarize. Note: the summary does not specify if a column was alreadyNULLto begin with or did not exist in the source table.