From df13f6deea363485b74b912acda1bed851cdb4ad Mon Sep 17 00:00:00 2001 From: Kenan Salim Date: Tue, 16 Jun 2026 00:14:58 +0300 Subject: [PATCH 1/3] =?UTF-8?q?test(data):=20silver=20business-rule=20data?= =?UTF-8?q?=20test=20=E2=80=94=20non-negative=20activity=20counts=20(#1321?= =?UTF-8?q?)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit First custom singular data test for the silver layer: activity counts on class_collab_document_activity can never be negative (a negative value means a broken transform / bad source row that would corrupt any summed metric). dbt fails the build if it returns rows; honest NULLs are not flagged. Pattern for the silver-integrity rollout under #1321/#1323 (extend to the other class_* metric tables). Enforced data_type contracts are intentionally NOT added here yet — the union models' exact types (e.g. MD5()->FixedString(16) vs String) can't be proven without a dbt run, so they land once the warehouse-gated CI is wired (the prerequisite step). Signed-off-by: Kenan Salim --- ...rt_collab_document_counts_non_negative.sql | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) create mode 100644 src/ingestion/dbt/tests/assert_collab_document_counts_non_negative.sql diff --git a/src/ingestion/dbt/tests/assert_collab_document_counts_non_negative.sql b/src/ingestion/dbt/tests/assert_collab_document_counts_non_negative.sql new file mode 100644 index 000000000..0ccd1029b --- /dev/null +++ b/src/ingestion/dbt/tests/assert_collab_document_counts_non_negative.sql @@ -0,0 +1,19 @@ +-- Business-rule data test (#1321 silver-layer integrity). +-- Activity counts can never be negative; a negative value means a broken +-- transform or bad source row and would corrupt any metric that sums them. +-- dbt fails the build if this returns any rows. NULL (not-ingested, e.g. +-- visited_page_count on OneDrive) is intentionally NOT flagged — NULL < 0 is +-- NULL, not a violation; honest NULLs are handled separately. +SELECT + unique_key, + viewed_or_edited_count, + synced_count, + shared_internally_count, + shared_externally_count, + visited_page_count +FROM {{ ref('class_collab_document_activity') }} +WHERE viewed_or_edited_count < 0 + OR synced_count < 0 + OR shared_internally_count < 0 + OR shared_externally_count < 0 + OR visited_page_count < 0 From 9cbd155a80066fbde61676cecc4a27155edacb34 Mon Sep 17 00:00:00 2001 From: Kenan Salim Date: Wed, 17 Jun 2026 11:47:28 +0300 Subject: [PATCH 2/3] test(data): adopt data_quality catalog convention for non-negative counts (#1321) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Aligns the silver business-rule test with the documented data-quality convention (tests/README.md / ADR 0005) instead of a bare build-blocking singular test: - add config(tags=['data_quality'], severity='warn', store_failures=true, meta{...}) so the check joins the monitored catalog, keeps failing rows in an audit table for drill-down, and emits a finding — rather than hard-failing dbt build with no stored rows. Mirrors the existing physical_bound check assert_meeting_duration_caps. - read class_collab_document_activity FINAL (RMT) per the README, so transient duplicates can't surface as repeated violations. - move under tests/collaboration/ to match the domain layout. Not changed: no dbt_utils/dbt-expectations (the project intentionally ships no packages.yml — singular SQL is the standard); no magic-number upper bound (statistical anomaly detection is a separate scheduled check); no shared<=viewed cross-column rule (the Graph activity counts are independent, so that invariant is unverified and would false-fire). Signed-off-by: Kenan Salim --- ...rt_collab_document_counts_non_negative.sql | 19 ----------- ...rt_collab_document_counts_non_negative.sql | 32 +++++++++++++++++++ 2 files changed, 32 insertions(+), 19 deletions(-) delete mode 100644 src/ingestion/dbt/tests/assert_collab_document_counts_non_negative.sql create mode 100644 src/ingestion/dbt/tests/collaboration/assert_collab_document_counts_non_negative.sql diff --git a/src/ingestion/dbt/tests/assert_collab_document_counts_non_negative.sql b/src/ingestion/dbt/tests/assert_collab_document_counts_non_negative.sql deleted file mode 100644 index 0ccd1029b..000000000 --- a/src/ingestion/dbt/tests/assert_collab_document_counts_non_negative.sql +++ /dev/null @@ -1,19 +0,0 @@ --- Business-rule data test (#1321 silver-layer integrity). --- Activity counts can never be negative; a negative value means a broken --- transform or bad source row and would corrupt any metric that sums them. --- dbt fails the build if this returns any rows. NULL (not-ingested, e.g. --- visited_page_count on OneDrive) is intentionally NOT flagged — NULL < 0 is --- NULL, not a violation; honest NULLs are handled separately. -SELECT - unique_key, - viewed_or_edited_count, - synced_count, - shared_internally_count, - shared_externally_count, - visited_page_count -FROM {{ ref('class_collab_document_activity') }} -WHERE viewed_or_edited_count < 0 - OR synced_count < 0 - OR shared_internally_count < 0 - OR shared_externally_count < 0 - OR visited_page_count < 0 diff --git a/src/ingestion/dbt/tests/collaboration/assert_collab_document_counts_non_negative.sql b/src/ingestion/dbt/tests/collaboration/assert_collab_document_counts_non_negative.sql new file mode 100644 index 000000000..6c4eb0f1f --- /dev/null +++ b/src/ingestion/dbt/tests/collaboration/assert_collab_document_counts_non_negative.sql @@ -0,0 +1,32 @@ +{{ config( + tags=['data_quality'], + severity='warn', + store_failures=true, + meta={ + 'title': 'Collab document activity counts are non-negative', + 'domain': 'collab', + 'category': 'physical_bound', + 'tier': 'error', + 'remediation': 'A negative activity count is physically impossible and points to a broken transform or a bad source row; any metric that sums the column is corrupted. Inspect the stored failing rows by unique_key, trace back to the m365 sharepoint/onedrive feeder mapping, and fix the source. A NULL count (a column a product never emits, e.g. visited_page_count on OneDrive) is not a violation.' + } +) }} +-- Business-rule data test (#1321 silver-layer integrity). +-- Activity counts can never be negative; a negative value means a broken +-- transform or bad source row and would corrupt any metric that sums them. +-- NULL (not-ingested, e.g. visited_page_count on OneDrive) is intentionally +-- NOT flagged — NULL < 0 is NULL, not a violation; honest NULLs are handled +-- separately. Read FINAL so transient ReplacingMergeTree duplicates can't +-- surface as repeated violations of the same row. +SELECT + unique_key, + viewed_or_edited_count, + synced_count, + shared_internally_count, + shared_externally_count, + visited_page_count +FROM {{ ref('class_collab_document_activity') }} FINAL +WHERE viewed_or_edited_count < 0 + OR synced_count < 0 + OR shared_internally_count < 0 + OR shared_externally_count < 0 + OR visited_page_count < 0 From a803a97c2a9cefcb172de1bb36cdd5c0f5e46366 Mon Sep 17 00:00:00 2001 From: Kenan Salim Date: Wed, 17 Jun 2026 19:12:34 +0300 Subject: [PATCH 3/3] test(data): add date to non-negative failure rows for triage (#1321) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Address PR review: include `date` in the stored-failure SELECT so on-call gets the activity day of the offending count directly in the audit rows, without a second lookup. Kept as context only — not a filter: a negative count is corruption on any partition, so the integrity check stays unbounded over the full table (no date window). Signed-off-by: Kenan Salim --- .../assert_collab_document_counts_non_negative.sql | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/src/ingestion/dbt/tests/collaboration/assert_collab_document_counts_non_negative.sql b/src/ingestion/dbt/tests/collaboration/assert_collab_document_counts_non_negative.sql index 6c4eb0f1f..f4f282b15 100644 --- a/src/ingestion/dbt/tests/collaboration/assert_collab_document_counts_non_negative.sql +++ b/src/ingestion/dbt/tests/collaboration/assert_collab_document_counts_non_negative.sql @@ -16,9 +16,13 @@ -- NULL (not-ingested, e.g. visited_page_count on OneDrive) is intentionally -- NOT flagged — NULL < 0 is NULL, not a violation; honest NULLs are handled -- separately. Read FINAL so transient ReplacingMergeTree duplicates can't --- surface as repeated violations of the same row. +-- surface as repeated violations of the same row. `date` is selected purely +-- as triage context in the stored-failure rows (the activity day of the bad +-- count) — it is not a filter: a negative count is corruption on any partition, +-- so the check stays unbounded over the full table. SELECT unique_key, + date, viewed_or_edited_count, synced_count, shared_internally_count,