Skip to content

feat(ingestion): cross-connector silver-contract parity gate (git commits) #1344

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
mitasovr wants to merge 1 commit into
base: main
Choose a base branch
from
Draft

feat(ingestion): cross-connector silver-contract parity gate (git commits) #1344

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
57 changes: 57 additions & 0 deletions .github/workflows/silver-contract-parity.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
name: Silver Contract Parity

# Offline cross-connector silver-contract consistency gate.
#
# Verifies that every connector feeding a `silver:<class>` tag declares exactly
# the columns the silver class contract declares — same names, ORDER and types
# (catches "column added/reordered/retyped in one connector, forgotten in
# another" before merge). Reads the dbt manifest produced by `dbt parse` — NO
# warehouse needed, runs in seconds.
#
# This is the offline declaration-consistency gate (layer 2). The dbt contracts
# themselves (layer 1: SQL output == declared schema) are enforced wherever the
# models are built with `dbt build` (e2e / Argo).
#
# No `paths:` filter on purpose: meant to be a required status check on main —
# a path-filtered required check never reports on PRs outside the filter, which
# leaves them stuck on "Expected".
#
# Script: src/ingestion/scripts/check_silver_contract_parity.py

on:
pull_request:
branches: [main]
workflow_dispatch:

jobs:
parity:
name: Silver contract parity
runs-on: ubuntu-latest
timeout-minutes: 5
steps:
- uses: actions/checkout@v4

- uses: actions/setup-python@v5
Comment on lines +32 to +34

@coderabbitai coderabbitai Bot Jun 15, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "Unpinned action references in workflows:"
rg -n '^\s*-\s*uses:\s*[^@]+@v[0-9]+' .github/workflows

echo
echo "checkout steps missing persist-credentials hardening:"
rg -n 'uses:\s*actions/checkout@' .github/workflows -C 3

Repository: constructorfabric/insight

Length of output: 9507

Pin GitHub Actions to immutable SHAs and disable checkout credential persistence.

Using moving major tags (@v4, @v5) weakens supply-chain guarantees. Add persist-credentials: false to the checkout action unless authenticated git operations are required later in the job.

Affected lines 
      - uses: actions/checkout@v4

      - uses: actions/setup-python@v5

Replace with:

      - uses: actions/checkout@v4
        with:
          persist-credentials: false

      - uses: actions/setup-python@v5

And pin both actions to full commit SHAs instead of version tags.

🧰 Tools
🪛 zizmor (1.25.2)

[warning] 32-32: credential persistence through GitHub Actions artifacts (artipacked): does not set persist-credentials: false

(artipacked)

[error] 32-32: unpinned action reference (unpinned-uses): action is not pinned to a hash (required by blanket policy)

(unpinned-uses)

[error] 34-34: unpinned action reference (unpinned-uses): action is not pinned to a hash (required by blanket policy)

(unpinned-uses)

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/silver-contract-parity.yml around lines 32 - 34, The
GitHub Actions in the workflow are using mutable version tags (`@v4` and `@v5`)
instead of immutable commit SHAs, which weakens supply-chain security
guarantees. Additionally, the checkout action is missing the persist-credentials
configuration. Fix this by replacing the moving major version tags with full
commit SHAs for both the actions/checkout and actions/setup-python actions. Add
a `with:` block to the actions/checkout action that includes
`persist-credentials: false` to disable credential persistence unless
authenticated git operations are explicitly required later in the job.

Source: Linters/SAST tools

with:
python-version: "3.12"

# Pinned so an unrelated dbt-core/adapter release can't break this gate.
- name: Install dbt-clickhouse
run: pip install "dbt-core==1.10.15" "dbt-clickhouse==1.9.6"

# Self-test the gate's own logic (no warehouse, no manifest needed).
- name: Unit-test the parity gate
run: python3 src/ingestion/scripts/tests/test_check_silver_contract_parity.py

- name: dbt parse (offline, no warehouse)
working-directory: src/ingestion/dbt
env:
# profiles.yml renders env_var('CLICKHOUSE_PASSWORD'); `dbt parse`
# never opens a connection, so any non-empty value lets it render.
CLICKHOUSE_PASSWORD: parse-only
run: dbt parse --profiles-dir .

- name: Check silver contract parity
run: |
python3 src/ingestion/scripts/check_silver_contract_parity.py \
--manifest src/ingestion/dbt/target/manifest.json
7 changes: 4 additions & 3 deletions src/ingestion/connectors/git/bitbucket-cloud/dbt/bitbucket_cloud__commits.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
materialized='incremental',
unique_key='unique_key',
order_by=['unique_key'],
on_schema_change='append_new_columns',
settings={'allow_nullable_key': 1},
schema='staging',
tags=['bitbucket-cloud', 'silver:class_git_commits']
Expand All @@ -22,9 +23,9 @@ SELECT
'' AS committer_email,
COALESCE(c.message, '') AS message,
parseDateTimeBestEffortOrNull(c.date) AS date,
COALESCE(fc.files_changed, 0) AS files_changed,
COALESCE(fc.lines_added, 0) AS lines_added,
COALESCE(fc.lines_removed, 0) AS lines_removed,
toInt64(COALESCE(fc.files_changed, 0)) AS files_changed,
toInt64(COALESCE(fc.lines_added, 0)) AS lines_added,
toInt64(COALESCE(fc.lines_removed, 0)) AS lines_removed,
if(JSONLength(COALESCE(toString(c.parent_hashes), '[]')) > 1, 1, 0) AS is_merge_commit,
'insight_bitbucket_cloud' AS data_source,
toUnixTimestamp64Milli(now64()) AS _version,
Expand Down
48 changes: 48 additions & 0 deletions src/ingestion/connectors/git/bitbucket-cloud/dbt/schema.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,8 +25,56 @@ models:
- name: bitbucket_cloud__repository_branches
description: "Bitbucket Cloud branches -> staging, feeds class_git_repository_branches"

# CONTRACT must match the `class_git_commits` silver contract exactly — name,
# order and data_type. Enforced by scripts/check_silver_contract_parity.py in
# CI: you cannot add/reorder/retype a column here without doing the same in the
# silver contract (and therefore every other connector feeding this class).
- name: bitbucket_cloud__commits
description: "Bitbucket Cloud commits -> staging, feeds class_git_commits"
config:
contract:
enforced: true
columns:
- name: tenant_id
data_type: String
- name: source_id
data_type: String
- name: unique_key
data_type: String
- name: project_key
data_type: String
- name: repo_slug
data_type: String
- name: commit_hash
data_type: String
- name: branch
data_type: String
- name: author_name
data_type: String
- name: author_email
data_type: String
- name: committer_name
data_type: String
- name: committer_email
data_type: String
- name: message
data_type: String
- name: date
data_type: Nullable(DateTime)
- name: files_changed
data_type: Int64
- name: lines_added
data_type: Int64
- name: lines_removed
data_type: Int64
- name: is_merge_commit
data_type: UInt8
- name: data_source
data_type: String
- name: _version
data_type: Int64
- name: _airbyte_extracted_at
data_type: DateTime64(3)

- name: bitbucket_cloud__file_changes
description: "Bitbucket Cloud file changes -> staging, feeds class_git_file_changes"
Expand Down
7 changes: 4 additions & 3 deletions src/ingestion/connectors/git/github/dbt/github__commits.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@
materialized='incremental',
unique_key='unique_key',
order_by=['unique_key'],
on_schema_change='append_new_columns',
settings={'allow_nullable_key': 1},
schema='staging',
tags=['github', 'silver:class_git_commits']
Expand All @@ -21,9 +22,9 @@ SELECT
COALESCE(committer_email, '') AS committer_email,
COALESCE(message, '') AS message,
parseDateTimeBestEffortOrNull(committed_date) AS date,
COALESCE(changed_files, 0) AS files_changed,
COALESCE(additions, 0) AS lines_added,
COALESCE(deletions, 0) AS lines_removed,
toInt64(COALESCE(changed_files, 0)) AS files_changed,
toInt64(COALESCE(additions, 0)) AS lines_added,
toInt64(COALESCE(deletions, 0)) AS lines_removed,
if(length(parent_hashes) > 1, 1, 0) AS is_merge_commit,
'insight_github' AS data_source,
toUnixTimestamp64Milli(now64()) AS _version,
Expand Down
48 changes: 48 additions & 0 deletions src/ingestion/connectors/git/github/dbt/schema.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,8 +26,56 @@ models:
- name: github__repository_branches
description: "GitHub branches -> staging, feeds class_git_repository_branches"

# CONTRACT must match the `class_git_commits` silver contract exactly — name,
# order and data_type. Enforced by scripts/check_silver_contract_parity.py in
# CI: you cannot add/reorder/retype a column here without doing the same in the
# silver contract (and therefore every other connector feeding this class).
- name: github__commits
description: "GitHub commits -> staging, feeds class_git_commits"
config:
contract:
enforced: true
columns:
- name: tenant_id
data_type: String
- name: source_id
data_type: String
- name: unique_key
data_type: String
- name: project_key
data_type: String
- name: repo_slug
data_type: String
- name: commit_hash
data_type: String
- name: branch
data_type: String
- name: author_name
data_type: String
- name: author_email
data_type: String
- name: committer_name
data_type: String
- name: committer_email
data_type: String
- name: message
data_type: String
- name: date
data_type: Nullable(DateTime)
- name: files_changed
data_type: Int64
- name: lines_added
data_type: Int64
- name: lines_removed
data_type: Int64
- name: is_merge_commit
data_type: UInt8
- name: data_source
data_type: String
- name: _version
data_type: Int64
- name: _airbyte_extracted_at
data_type: DateTime64(3)

- name: github__file_changes
description: "GitHub file changes (PR + direct push) -> staging, feeds class_git_file_changes"
Expand Down
7 changes: 4 additions & 3 deletions src/ingestion/connectors/git/gitlab/dbt/gitlab__commits.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
materialized='incremental',
unique_key='unique_key',
order_by=['unique_key'],
on_schema_change='append_new_columns',
settings={'allow_nullable_key': 1},
schema='staging',
tags=['gitlab', 'silver:class_git_commits']
Expand Down Expand Up @@ -47,9 +48,9 @@ SELECT
COALESCE(c.committer_email, '') AS committer_email,
COALESCE(c.message, '') AS message,
parseDateTimeBestEffortOrNull(c.committed_date) AS date,
COALESCE(f.files_changed, 0) AS files_changed,
COALESCE(c.stats_additions, 0) AS lines_added,
COALESCE(c.stats_deletions, 0) AS lines_removed,
toInt64(COALESCE(f.files_changed, 0)) AS files_changed,
toInt64(COALESCE(c.stats_additions, 0)) AS lines_added,
toInt64(COALESCE(c.stats_deletions, 0)) AS lines_removed,
if(COALESCE(c.parent_count, 0) > 1, 1, 0) AS is_merge_commit,
'insight_gitlab' AS data_source,
toUnixTimestamp64Milli(now64()) AS _version,
Expand Down
48 changes: 48 additions & 0 deletions src/ingestion/connectors/git/gitlab/dbt/schema.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,8 +20,56 @@ models:
- name: gitlab__repository_branches
description: "GitLab branches -> staging, feeds class_git_repository_branches"

# CONTRACT must match the `class_git_commits` silver contract exactly — name,
# order and data_type. Enforced by scripts/check_silver_contract_parity.py in
# CI: you cannot add/reorder/retype a column here without doing the same in the
# silver contract (and therefore every other connector feeding this class).
- name: gitlab__commits
description: "GitLab commits -> staging, feeds class_git_commits"
config:
contract:
enforced: true
columns:
- name: tenant_id
data_type: String
- name: source_id
data_type: String
- name: unique_key
data_type: String
- name: project_key
data_type: String
- name: repo_slug
data_type: String
- name: commit_hash
data_type: String
- name: branch
data_type: String
- name: author_name
data_type: String
- name: author_email
data_type: String
- name: committer_name
data_type: String
- name: committer_email
data_type: String
- name: message
data_type: String
- name: date
data_type: Nullable(DateTime)
- name: files_changed
data_type: Int64
- name: lines_added
data_type: Int64
- name: lines_removed
data_type: Int64
- name: is_merge_commit
data_type: UInt8
- name: data_source
data_type: String
- name: _version
data_type: Int64
- name: _airbyte_extracted_at
data_type: DateTime64(3)

- name: gitlab__file_changes
description: "GitLab commit file changes -> staging, feeds class_git_file_changes"
Expand Down
Loading
Loading