From 7c9a831f9688b218c2eda98ae2707ef5722a3ccb Mon Sep 17 00:00:00 2001
From: Roman Mitasov <Roman.Mitasov@constructor.tech>
Date: Thu, 16 Jul 2026 11:43:41 +0300
Subject: [PATCH] docs(data-flow): propose silver->gold data contract (RFC)

Draft contract for the planned DB split (ingestion bronze/staging/silver
vs gold). Defines five datasets based on the current silver schemas
(commits, pull requests, PR comments, task field history, chat activity)
plus two auxiliary dimensions (repository branches, task statuses), each
with a JSON Schema, field descriptions, examples, and a growing list of
interpretation cases (patch_id first/last rules, statusCategory close
detection, enriched events vs raw deltas, honest-NULL chat counters).

Normative additions over current state are limited to person_id on every
dataset and patch_id on commits; all other gaps are listed in Appendix A.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Roman Mitasov <Roman.Mitasov@constructor.tech>
---
 .../specs/SILVER-GOLD-DATA-CONTRACT.md        | 762 ++++++++++++++++++
 1 file changed, 762 insertions(+)
 create mode 100644 docs/domain/ingestion-data-flow/specs/SILVER-GOLD-DATA-CONTRACT.md

diff --git a/docs/domain/ingestion-data-flow/specs/SILVER-GOLD-DATA-CONTRACT.md b/docs/domain/ingestion-data-flow/specs/SILVER-GOLD-DATA-CONTRACT.md
new file mode 100644
index 000000000..4bff186c6
--- /dev/null
+++ b/docs/domain/ingestion-data-flow/specs/SILVER-GOLD-DATA-CONTRACT.md
@@ -0,0 +1,762 @@
+# Silver → Gold Data Contract
+
+**Status:** draft v0.2 (2026-07-16)
+**Scope:** the fixed interface between the ingestion database (bronze + staging + silver) and the analytics database (gold).
+
+Schemas in this document are based on the **current silver tables** (real column names and types). The contract adds exactly two normative requirements on top of what exists today: `person_id` on every dataset ([§2.2](#22-person_id)) and `patch_id` on commits ([§3](#3-commits)). Everything else describes data as it is produced now. Known deltas between the contract and the implementation are listed in [Appendix A](#appendix-a-gaps-between-this-contract-and-the-current-implementation).
+
+<!-- toc -->
+
+- [1. Purpose](#1-purpose)
+- [2. Common conventions](#2-common-conventions)
+  - [2.1 Envelope fields](#21-envelope-fields)
+  - [2.2 person_id](#22-person_id)
+  - [2.3 Deduplication](#23-deduplication)
+- [3. Commits](#3-commits)
+- [4. Pull / Merge Requests](#4-pull--merge-requests)
+- [5. PR / MR Comments](#5-pr--mr-comments)
+- [6. Task Field-Change History](#6-task-field-change-history)
+- [7. Messenger Chat Activity](#7-messenger-chat-activity)
+- [8. Extending this contract](#8-extending-this-contract)
+- [Appendix A. Gaps between this contract and the current implementation](#appendix-a-gaps-between-this-contract-and-the-current-implementation)
+
+<!-- /toc -->
+
+## 1. Purpose
+
+The warehouse is being split into two databases:
+
+- **Ingestion DB** — bronze, staging, and silver layers. Owned by the ingestion pipeline (Airbyte connectors, dbt, Rust enrich binaries). Knows about vendor APIs, sync mechanics, and identity resolution.
+- **Gold DB** — metric and reporting models. Owned by analytics. Knows nothing about vendors, sync mechanics, or identity resolution.
+
+This document is the contract between them. It defines the datasets the ingestion side publishes and the gold side consumes:
+
+| Dataset | Section | Silver table |
+|---|---|---|
+| `commits` | [§3](#3-commits) | `silver.class_git_commits` |
+| `repository_branches` (aux dimension) | [§3.5](#35-auxiliary-dimension-repository_branches) | `silver.class_git_repository_branches` |
+| `pull_requests` | [§4](#4-pull--merge-requests) | `silver.class_git_pull_requests` |
+| `pull_request_comments` | [§5](#5-pr--mr-comments) | `silver.class_git_pull_requests_comments` |
+| `task_field_history` | [§6](#6-task-field-change-history) | `silver.class_task_field_history` |
+| `task_statuses` (aux dimension) | [§6.5](#65-auxiliary-dimension-task_statuses) | `silver.class_task_statuses` |
+| `chat_activity` | [§7](#7-messenger-chat-activity) | `silver.class_collab_chat_activity` |
+
+Rules of the boundary:
+
+1. **Gold reads only contract datasets.** No gold model may reach into bronze, staging, or non-contract silver tables.
+2. **Records arrive identity-resolved.** Every record carries a `person_id` produced by Identity Resolution *before* it crosses the boundary. Gold never resolves emails, logins, or account ids itself.
+3. **The contract is versioned.** Adding a field or a case is a non-breaking change. Renaming/removing a field, changing a type, or changing the meaning of an existing field requires a new contract version and a migration window during which both shapes are published.
+4. **Each dataset section ends with Cases** — the agreed interpretation rules for that data. The case list will grow; when a new question comes up ("how do I count X?"), the answer is added here as a case, not decided ad hoc inside a gold model.
+
+## 2. Common conventions
+
+### 2.1 Envelope fields
+
+Every record carries a common envelope. Column names vary by dataset today (historical naming); the concept mapping:
+
+| Concept | git datasets (§3–§5) | task_field_history (§6) | chat_activity (§7) | Description |
+|---|---|---|---|---|
+| Tenant | `tenant_id` | — *(gap, see Appendix A)* | `tenant_id` | Multi-tenant isolation key. Every gold query MUST scope by it. |
+| Source instance | `source_id` | `insight_source_id` | `insight_source_id` | Connector instance id; distinguishes two instances of the same source type. |
+| Source type | `data_source` | `data_source` | `data_source` | Discriminator: `insight_github`, `insight_gitlab`, `insight_bitbucket_cloud`, `insight_jira`, `insight_youtrack`, `insight_slack`, `insight_m365`, `insight_zulip_proxy`. |
+| Record identity | `unique_key` | `unique_key` | `unique_key` | Stable logical record identity, `{tenant}-{source}-{natural_key_parts}` (per-dataset natural keys noted in each section). Exactly one logical record per `unique_key`. |
+| Person | `person_id` | `person_id` | `person_id` | The resolved acting person — [§2.2](#22-person_id). |
+| Version | `_version` | `_version` | `_version` | UInt64, ms epoch. When two physical rows share a `unique_key`, the greater `_version` wins. |
+
+### 2.2 person_id
+
+- `person_id` is the stable person UUID minted by Identity Resolution (UUIDv7 from the persons registry; minted on first observation, never re-derived).
+- Semantics per dataset: the **actor** of the record — commit author, PR author, comment author, field-change author, chat-activity user.
+- Resolution happens on the ingestion side (account id / email / login → person) before publishing. Gold treats `person_id` as opaque.
+- `person_id` is `NULL` when the actor could not be resolved (e.g. a bot, a departed contractor with no directory record, or a source that exposes no identity fields). The default gold policy: **exclude NULL-person rows from per-person metrics; include them in team/repo totals.** A dataset case may override this.
+- Raw identity fields (`author_email`, `author_name`, source account ids) are kept **for traceability and debugging only**. Gold MUST NOT join or group by them.
+
+### 2.3 Deduplication
+
+Contract datasets are stored as `ReplacingMergeTree(_version)` with `ORDER BY (unique_key)`; physical duplicates can exist until a background merge. Every gold read MUST dedup at read time using one of:
+
+```sql
+SELECT ... FROM <dataset> FINAL
+-- or
+SELECT ... FROM <dataset>
+QUALIFY ROW_NUMBER() OVER (PARTITION BY unique_key ORDER BY _version DESC) = 1
+```
+
+Updates to a logical record (an edited comment, a re-synced commit) reuse the same `unique_key` with a higher `_version`; they never create a second logical record.
+
+## 3. Commits
+
+One record per commit per branch observation, as unioned from GitHub / GitLab / Bitbucket staging into `silver.class_git_commits`. The same change may appear as several records: on several branches, or re-created by rebase / cherry-pick with a different `commit_hash`. The `patch_id` field groups those copies — see the cases below.
+
+### 3.1 JSON Schema
+
+```json
+{
+  "$schema": "https://json-schema.org/draft/2020-12/schema",
+  "$id": "insight/contracts/silver-gold/commit.schema.json",
+  "title": "Commit",
+  "type": "object",
+  "properties": {
+    "tenant_id":       { "type": "string", "minLength": 1 },
+    "source_id":       { "type": "string", "minLength": 1 },
+    "data_source":     { "type": "string", "pattern": "^insight_[a-z0-9_]+$" },
+    "unique_key":      { "type": "string", "minLength": 1 },
+    "person_id":       { "type": ["string", "null"], "format": "uuid" },
+    "_version":        { "type": "integer", "minimum": 0 },
+    "project_key":     { "type": "string" },
+    "repo_slug":       { "type": "string" },
+    "commit_hash":     { "type": "string", "pattern": "^[0-9a-f]{7,64}$" },
+    "patch_id":        { "type": ["string", "null"], "pattern": "^[0-9a-f]{40}$" },
+    "branch":          { "type": "string" },
+    "author_name":     { "type": "string" },
+    "author_email":    { "type": "string" },
+    "committer_name":  { "type": "string" },
+    "committer_email": { "type": "string" },
+    "message":         { "type": "string" },
+    "date":            { "type": ["string", "null"], "format": "date-time" },
+    "files_changed":   { "type": "integer", "minimum": 0 },
+    "lines_added":     { "type": "integer", "minimum": 0 },
+    "lines_removed":   { "type": "integer", "minimum": 0 },
+    "is_merge_commit": { "type": "integer", "enum": [0, 1] }
+  },
+  "required": [
+    "tenant_id", "source_id", "data_source", "unique_key", "person_id", "_version",
+    "project_key", "repo_slug", "commit_hash", "patch_id", "branch",
+    "author_name", "author_email", "date",
+    "files_changed", "lines_added", "lines_removed", "is_merge_commit"
+  ]
+}
+```
+
+### 3.2 Fields
+
+| Field | Description |
+|---|---|
+| `person_id` | Resolved commit **author** (not committer). |
+| `project_key` | Grouping above repo: GitHub org / GitLab namespace / Bitbucket workspace. |
+| `repo_slug` | Repository name within the project. |
+| `commit_hash` | VCS SHA. Unique per repo, but NOT stable across rebase / cherry-pick. |
+| `patch_id` | **New in this contract.** `git patch-id --stable` of the commit diff. Identical for all copies of the same change (across branches, rebases, cherry-picks). `NULL` for merge commits and empty-diff commits. |
+| `branch` | The branch on which this record was observed (single branch name; a commit reachable from N branches yields N records). |
+| `author_name` / `author_email` | Raw identity, traceability only. `author_email` is `""` for Bitbucket (no email in the API). |
+| `committer_name` / `committer_email` | Raw identity; `""` for Bitbucket and GitLab. |
+| `date` | Committer timestamp, UTC. Nullable (unparseable source timestamps become `NULL`). |
+| `files_changed`, `lines_added`, `lines_removed` | Diff size stats (0 default when the source did not provide them). |
+| `is_merge_commit` | `1` when the commit has more than one parent, else `0`. |
+
+### 3.3 Example
+
+Two records — the same change on a feature branch and, after squash-merge, on the default branch. `commit_hash` differs, `patch_id` is identical:
+
+```json
+{
+  "tenant_id": "acme", "source_id": "gh-main", "data_source": "insight_github",
+  "unique_key": "acme-gh-main-platform-api-8c1f2ab9",
+  "person_id": "0197c2a4-7b1e-7f3a-9c2d-4a5b6c7d8e9f",
+  "_version": 1752571200000,
+  "project_key": "platform", "repo_slug": "api",
+  "commit_hash": "8c1f2ab94e7d3c5f6a0b1d2e3f4a5b6c7d8e9f01",
+  "patch_id": "3d0f5a9b8c7e6d5f4a3b2c1d0e9f8a7b6c5d4e3f",
+  "branch": "feature/retry-backoff",
+  "author_name": "Jane Doe", "author_email": "jane.doe@acme.com",
+  "committer_name": "Jane Doe", "committer_email": "jane.doe@acme.com",
+  "message": "Add retry with exponential backoff to webhook sender",
+  "date": "2026-07-10T09:14:32Z",
+  "files_changed": 3, "lines_added": 120, "lines_removed": 14,
+  "is_merge_commit": 0
+}
+```
+
+```json
+{
+  "tenant_id": "acme", "source_id": "gh-main", "data_source": "insight_github",
+  "unique_key": "acme-gh-main-platform-api-f04d77c2",
+  "person_id": "0197c2a4-7b1e-7f3a-9c2d-4a5b6c7d8e9f",
+  "_version": 1752614400000,
+  "project_key": "platform", "repo_slug": "api",
+  "commit_hash": "f04d77c21b3a4c5d6e7f8a9b0c1d2e3f4a5b6c7d",
+  "patch_id": "3d0f5a9b8c7e6d5f4a3b2c1d0e9f8a7b6c5d4e3f",
+  "branch": "main",
+  "author_name": "Jane Doe", "author_email": "jane.doe@acme.com",
+  "committer_name": "GitHub", "committer_email": "noreply@github.com",
+  "message": "Add retry with exponential backoff to webhook sender (#412)",
+  "date": "2026-07-10T15:02:11Z",
+  "files_changed": 3, "lines_added": 120, "lines_removed": 14,
+  "is_merge_commit": 0
+}
+```
+
+### 3.4 Cases
+
+#### Case: using patch_id — statistics across all branches
+
+**Question:** how do I count commits / lines of code without double-counting the same change that exists on several branches?
+
+**Rule:** group commits by `patch_id` and keep only the **first** commit of each group (earliest `date`; tie-break by `commit_hash` ascending). The first commit is the original authorship event; later copies are mechanical (merges of the branch, cherry-picks, rebases). Records with `patch_id = NULL` form singleton groups — fall back to `commit_hash`.
+
+```sql
+SELECT * FROM commits FINAL
+QUALIFY ROW_NUMBER() OVER (
+    PARTITION BY tenant_id, coalesce(patch_id, commit_hash)
+    ORDER BY date ASC, commit_hash ASC
+) = 1
+```
+
+#### Case: using patch_id — statistics on the default branch only
+
+**Question:** how do I count only work that actually landed on the default branch?
+
+**Rule:** restrict to records observed on the repository's default branch (via the `repository_branches` dimension, [§3.5](#35-auxiliary-dimension-repository_branches)), then within each `patch_id` group keep only the **last** commit (latest `date`; tie-break by `commit_hash` ascending). The last copy is the one that finally landed (e.g. the squash or rebase result); earlier copies are superseded.
+
+```sql
+SELECT c.* FROM commits AS c FINAL
+INNER JOIN repository_branches AS b FINAL
+    ON  b.tenant_id  = c.tenant_id
+    AND b.source_id  = c.source_id
+    AND b.repo_slug  = c.repo_slug
+    AND b.branch_name = c.branch
+    AND b.is_default = 1
+QUALIFY ROW_NUMBER() OVER (
+    PARTITION BY c.tenant_id, coalesce(c.patch_id, c.commit_hash)
+    ORDER BY c.date DESC, c.commit_hash ASC
+) = 1
+```
+
+#### Case: merge commits
+
+**Question:** do merge commits count as authored work?
+
+**Rule:** no. Exclude `is_merge_commit = 1` from authored-work metrics (commit counts, LOC). Merge commits carry `patch_id = NULL` and their diff stats double the merged work. They may still be used to detect *integration events* (something landed on a branch).
+
+### 3.5 Auxiliary dimension: repository_branches
+
+One record per (repo × branch), from `silver.class_git_repository_branches`. Used to identify the default branch.
+
+| Field | Type | Description |
+|---|---|---|
+| `tenant_id`, `source_id`, `data_source`, `unique_key`, `_version` | envelope | See [§2.1](#21-envelope-fields). |
+| `project_key`, `repo_slug` | String | Repository. |
+| `branch_name` | String | Branch name. |
+| `is_default` | UInt8 (0/1) | `1` when this is the repository's default branch. |
+| `last_commit_hash` | String | Head commit SHA of the branch. |
+| `last_commit_date` | Nullable(DateTime) | Head commit timestamp. |
+
+## 4. Pull / Merge Requests
+
+One record per pull request (GitHub, Bitbucket) or merge request (GitLab), from `silver.class_git_pull_requests`. GitLab MRs are mapped to this shape by the producer.
+
+### 4.1 JSON Schema
+
+```json
+{
+  "$schema": "https://json-schema.org/draft/2020-12/schema",
+  "$id": "insight/contracts/silver-gold/pull_request.schema.json",
+  "title": "Pull / Merge Request",
+  "type": "object",
+  "properties": {
+    "tenant_id":          { "type": "string", "minLength": 1 },
+    "source_id":          { "type": "string", "minLength": 1 },
+    "data_source":        { "type": "string", "pattern": "^insight_[a-z0-9_]+$" },
+    "unique_key":         { "type": "string", "minLength": 1 },
+    "person_id":          { "type": ["string", "null"], "format": "uuid" },
+    "_version":           { "type": "integer", "minimum": 0 },
+    "project_key":        { "type": "string" },
+    "repo_slug":          { "type": "string" },
+    "pr_id":              { "type": "integer" },
+    "pr_number":          { "type": "integer" },
+    "title":              { "type": "string" },
+    "description":        { "type": "string" },
+    "state":              { "type": "string", "enum": ["OPEN", "MERGED", "CLOSED", "DECLINED"] },
+    "author_name":        { "type": "string" },
+    "author_email":       { "type": "string" },
+    "source_branch":      { "type": "string" },
+    "destination_branch": { "type": "string" },
+    "created_on":         { "type": ["string", "null"], "format": "date-time" },
+    "updated_on":         { "type": ["string", "null"], "format": "date-time" },
+    "closed_on":          { "type": ["string", "null"], "format": "date-time" },
+    "merge_commit_hash":  { "type": "string" },
+    "files_changed":      { "type": "integer", "minimum": 0 },
+    "lines_added":        { "type": "integer", "minimum": 0 },
+    "lines_removed":      { "type": "integer", "minimum": 0 }
+  },
+  "required": [
+    "tenant_id", "source_id", "data_source", "unique_key", "person_id", "_version",
+    "project_key", "repo_slug", "pr_id", "pr_number", "title", "state",
+    "author_name", "author_email", "source_branch", "destination_branch",
+    "created_on", "closed_on", "merge_commit_hash"
+  ]
+}
+```
+
+### 4.2 Fields
+
+| Field | Description |
+|---|---|
+| `person_id` | Resolved PR **author**. |
+| `pr_id` | Source-native id. GitHub: global `database_id`; GitLab: per-project `iid`; Bitbucket: per-repo `id`. Unique only within (`source_id`, `repo_slug`) — always join with repo context. |
+| `pr_number` | Human-facing number, unique within a repo. |
+| `state` | Lifecycle state, uppercase. `MERGED` = accepted; `CLOSED` / `DECLINED` = rejected without merge (which one depends on the source: Bitbucket emits `DECLINED`, GitHub/GitLab emit `CLOSED`); `OPEN` = in progress. Compare case-insensitively to be safe. |
+| `author_name` | Raw identity: GitHub login / Bitbucket display name / GitLab username. |
+| `author_email` | Raw identity; `""` for Bitbucket (no email in the API). |
+| `created_on`, `updated_on` | Lifecycle timestamps, UTC. |
+| `closed_on` | Terminal timestamp. For `MERGED` PRs this **is** the merge time — there is no separate `merged_at` (GitLab: `coalesce(closed_at, merged_at)`; Bitbucket: heuristic from the closer's participation timestamp). `NULL` while `OPEN`. |
+| `merge_commit_hash` | SHA of the merge/squash commit on the destination branch; `""` unless merged. Joins to `commits.commit_hash`. |
+| `files_changed`, `lines_added`, `lines_removed` | Diff size. **GitLab: always `0`** (not collected) — treat `0` as "unknown", not "empty diff". |
+
+### 4.3 Example
+
+```json
+{
+  "tenant_id": "acme", "source_id": "gh-main", "data_source": "insight_github",
+  "unique_key": "acme-gh-main-platform-api-pr-412",
+  "person_id": "0197c2a4-7b1e-7f3a-9c2d-4a5b6c7d8e9f",
+  "_version": 1752614400000,
+  "project_key": "platform", "repo_slug": "api",
+  "pr_id": 2210443, "pr_number": 412,
+  "title": "Add retry with exponential backoff to webhook sender",
+  "description": "Fixes #388. Retries 5xx with jittered backoff.",
+  "state": "MERGED",
+  "author_name": "jane-doe", "author_email": "jane.doe@acme.com",
+  "source_branch": "feature/retry-backoff", "destination_branch": "main",
+  "created_on": "2026-07-10T09:20:05Z",
+  "updated_on": "2026-07-10T15:02:11Z",
+  "closed_on": "2026-07-10T15:02:11Z",
+  "merge_commit_hash": "f04d77c21b3a4c5d6e7f8a9b0c1d2e3f4a5b6c7d",
+  "files_changed": 3, "lines_added": 120, "lines_removed": 14
+}
+```
+
+### 4.4 Cases
+
+#### Case: determining the PR outcome
+
+**Question:** how do I tell merged, rejected, and in-progress PRs apart?
+
+**Rule:** use `state` only. `MERGED` = accepted. `CLOSED` and `DECLINED` are both "rejected without merge" and MUST be treated identically. `OPEN` = in progress. Never infer the outcome from timestamps or `merge_commit_hash` alone.
+
+#### Case: measuring cycle time
+
+**Question:** how long did a PR take?
+
+**Rule:** cycle time = `closed_on - created_on`, computed **only** for `state = 'MERGED'`. Rejected PRs are not cycle time — they are a separate rejection metric. Guard against clock skew: if `closed_on < created_on`, emit `NULL`, not a negative value.
+
+#### Case: linking a PR to its landed commit
+
+**Question:** how do I connect a merged PR to the commit that landed on the destination branch?
+
+**Rule:** join `pull_requests.merge_commit_hash = commits.commit_hash` within the same `tenant_id` + `source_id`. For squash merges this is the single squashed commit; for merge-commit merges it is the merge node (`is_merge_commit = 1`).
+
+## 5. PR / MR Comments
+
+One record per human comment on a pull/merge request, from `silver.class_git_pull_requests_comments` — both general discussion comments and inline code-review comments. System/bot notes (GitLab "system notes") are excluded by the producer.
+
+### 5.1 JSON Schema
+
+```json
+{
+  "$schema": "https://json-schema.org/draft/2020-12/schema",
+  "$id": "insight/contracts/silver-gold/pull_request_comment.schema.json",
+  "title": "Pull / Merge Request Comment",
+  "type": "object",
+  "properties": {
+    "tenant_id":   { "type": "string", "minLength": 1 },
+    "source_id":   { "type": "string", "minLength": 1 },
+    "data_source": { "type": "string", "pattern": "^insight_[a-z0-9_]+$" },
+    "unique_key":  { "type": "string", "minLength": 1 },
+    "person_id":   { "type": ["string", "null"], "format": "uuid" },
+    "_version":    { "type": "integer", "minimum": 0 },
+    "project_key": { "type": "string" },
+    "repo_slug":   { "type": "string" },
+    "pr_id":       { "type": "integer" },
+    "comment_id":  { "type": "integer" },
+    "content":     { "type": "string" },
+    "author_name": { "type": "string" },
+    "author_uuid": { "type": "string" },
+    "created_at":  { "type": ["string", "null"], "format": "date-time" },
+    "updated_at":  { "type": ["string", "null"], "format": "date-time" },
+    "is_inline":   { "type": "integer", "enum": [0, 1] },
+    "file_path":   { "type": "string" },
+    "line_number": { "type": "integer", "minimum": 0 }
+  },
+  "required": [
+    "tenant_id", "source_id", "data_source", "unique_key", "person_id", "_version",
+    "project_key", "repo_slug", "pr_id", "comment_id", "content",
+    "author_name", "created_at", "is_inline", "file_path", "line_number"
+  ]
+}
+```
+
+### 5.2 Fields
+
+| Field | Description |
+|---|---|
+| `person_id` | Resolved comment **author**. |
+| `pr_id` | Joins to `pull_requests.pr_id` within the same (`source_id`, `repo_slug`). |
+| `content` | Comment body (markdown as delivered by the source). |
+| `author_name`, `author_uuid` | Raw source identity (login/display name + source account id), traceability only. |
+| `is_inline` | `1` = attached to a diff line (code review); `0` = general discussion. |
+| `file_path`, `line_number` | Meaningful only when `is_inline = 1`; `""` / `0` otherwise. |
+| `updated_at` | Last edit time; edits bump `_version` on the same `unique_key`, they do not create a new record. |
+
+### 5.3 Example
+
+```json
+{
+  "tenant_id": "acme", "source_id": "gh-main", "data_source": "insight_github",
+  "unique_key": "acme-gh-main-platform-api-pr-412-c-99801",
+  "person_id": "0197c8d1-2e4f-7a6b-8c9d-0e1f2a3b4c5d",
+  "_version": 1752582600000,
+  "project_key": "platform", "repo_slug": "api",
+  "pr_id": 2210443, "comment_id": 99801,
+  "content": "Backoff cap of 30s seems low for webhook consumers — make it configurable?",
+  "author_name": "bob-reviewer", "author_uuid": "5521907",
+  "created_at": "2026-07-10T11:30:00Z", "updated_at": null,
+  "is_inline": 1,
+  "file_path": "src/webhooks/sender.py", "line_number": 84
+}
+```
+
+### 5.4 Cases
+
+#### Case: code-review comments vs discussion
+
+**Question:** how do I measure code-review depth as opposed to general PR chatter?
+
+**Rule:** `is_inline = 1` is a code-review comment (anchored to a file and line); `is_inline = 0` is discussion. Review-depth metrics count inline comments; communication metrics may count both.
+
+#### Case: counting review participation
+
+**Question:** who reviewed a PR?
+
+**Rule:** a person *participated in reviewing* a PR when they authored at least one comment on it with `person_id != pull_requests.person_id` (self-comments are not review). Formal approvals (approve / request-changes verdicts) live in `silver.class_git_pull_requests_reviewers` — a candidate for a future contract dataset; do not infer approval from comments.
+
+## 6. Task Field-Change History
+
+The event-sourced history of task/issue fields, produced by the **enrich** step (Rust `jira-enrich`), from `silver.class_task_field_history`. One record per **(issue × field × event)**. This is the only task dataset gold needs for lifecycle metrics: current state, state at any time T, transitions, and durations are all derivable from it.
+
+`unique_key` formula: `{insight_source_id}-{data_source}-{id_readable}-{field_id}-{event_id}`.
+
+### 6.1 JSON Schema
+
+```json
+{
+  "$schema": "https://json-schema.org/draft/2020-12/schema",
+  "$id": "insight/contracts/silver-gold/task_field_history.schema.json",
+  "title": "Task Field-Change Event",
+  "type": "object",
+  "properties": {
+    "insight_source_id":   { "type": "string", "minLength": 1 },
+    "data_source":         { "type": "string", "pattern": "^insight_[a-z0-9_]+$" },
+    "unique_key":          { "type": "string", "minLength": 1 },
+    "person_id":           { "type": ["string", "null"], "format": "uuid" },
+    "_version":            { "type": "integer", "minimum": 0 },
+    "issue_id":            { "type": "string" },
+    "id_readable":         { "type": "string" },
+    "event_id":            { "type": "string" },
+    "event_at":            { "type": "string", "format": "date-time" },
+    "event_kind":          { "type": "string", "enum": ["changelog", "synthetic_initial"] },
+    "_seq":                { "type": "integer", "minimum": 0 },
+    "author_id":           { "type": ["string", "null"] },
+    "author_display":      { "type": ["string", "null"] },
+    "field_id":            { "type": "string" },
+    "field_name":          { "type": "string" },
+    "field_cardinality":   { "type": "string", "enum": ["single", "multi"] },
+    "delta_action":        { "type": "string", "enum": ["set", "add", "remove"] },
+    "delta_value_id":      { "type": ["string", "null"] },
+    "delta_value_display": { "type": ["string", "null"] },
+    "value_ids":           { "type": "array", "items": { "type": "string" } },
+    "value_displays":      { "type": "array", "items": { "type": "string" } },
+    "value_id_type":       { "type": "string", "enum": ["opaque_id", "account_id", "string_literal", "path", "none"] },
+    "collected_at":        { "type": "string", "format": "date-time" }
+  },
+  "required": [
+    "insight_source_id", "data_source", "unique_key", "person_id", "_version",
+    "issue_id", "id_readable", "event_id", "event_at", "event_kind", "_seq",
+    "field_id", "field_name", "field_cardinality", "delta_action",
+    "value_ids", "value_displays", "value_id_type", "collected_at"
+  ]
+}
+```
+
+### 6.2 Fields
+
+| Field | Description |
+|---|---|
+| `person_id` | Resolved **change author**: for `changelog` events, who made the change; for `synthetic_initial` events, the issue **reporter** (creation-time values are attributed to the reporter). |
+| `issue_id` / `id_readable` | Source-native id and human-readable key (`PROJ-123`). |
+| `event_id` | Source changelog entry id, or a synthetic id for `synthetic_initial` rows. |
+| `event_at` | When the change happened (for synthetic rows: issue creation time). |
+| `event_kind` | `changelog` = a real recorded change. `synthetic_initial` = a reconstructed creation-time baseline row — see the first case below. |
+| `_seq` | Deterministic tie-break within one `event_at`: `changelog` rows have `_seq = 0`; `synthetic_initial` rows are numbered `1..N` (field-id order). Order events by `(event_at, _seq, _version)`. |
+| `author_id`, `author_display` | Raw source identity (e.g. Atlassian accountId), traceability only. |
+| `field_id` / `field_name` | Which field changed (`status`, `assignee`, `labels`, …). The sentinel `field_id = "created"` marks the issue-creation event. |
+| `field_cardinality` | `single` (status, assignee) vs `multi` (labels, components). |
+| `delta_action` | What the change did: `set` (replace, single-value fields), `add` / `remove` (multi-value fields). |
+| `delta_value_id` / `delta_value_display` | The changed value itself (id and display name). |
+| `value_ids` / `value_displays` | **Full field value after this event** — the enriched running state. Empty arrays mean the field became empty. For `field_id = 'status'`, `value_ids[1]` joins to `task_statuses.status_id` ([§6.5](#65-auxiliary-dimension-task_statuses)). |
+| `value_id_type` | What kind of identifier `value_ids` holds: `opaque_id` (status/priority ids), `account_id` (assignee/reporter), `string_literal` (labels), `path`, `none`. |
+| `collected_at` | When the enrich run produced this row (`_version` = its ms epoch). |
+
+### 6.3 Example
+
+Three events for one issue: the creation marker, the initial status baseline, and a real status transition:
+
+```json
+{
+  "insight_source_id": "jira-main", "data_source": "insight_jira",
+  "unique_key": "jira-main-insight_jira-PROJ-123-created-synth-0",
+  "person_id": "0197c2a4-7b1e-7f3a-9c2d-4a5b6c7d8e9f",
+  "_version": 1752571200000,
+  "issue_id": "10500", "id_readable": "PROJ-123",
+  "event_id": "synth-0", "event_at": "2026-07-01T08:00:00Z",
+  "event_kind": "synthetic_initial", "_seq": 0,
+  "author_id": "557058:aa11bb22", "author_display": "Jane Doe",
+  "field_id": "created", "field_name": "created", "field_cardinality": "single",
+  "delta_action": "set", "delta_value_id": null, "delta_value_display": null,
+  "value_ids": [], "value_displays": [], "value_id_type": "none",
+  "collected_at": "2026-07-16T02:00:00Z"
+}
+```
+
+```json
+{
+  "insight_source_id": "jira-main", "data_source": "insight_jira",
+  "unique_key": "jira-main-insight_jira-PROJ-123-status-synth-1",
+  "person_id": "0197c2a4-7b1e-7f3a-9c2d-4a5b6c7d8e9f",
+  "_version": 1752571200000,
+  "issue_id": "10500", "id_readable": "PROJ-123",
+  "event_id": "synth-1", "event_at": "2026-07-01T08:00:00Z",
+  "event_kind": "synthetic_initial", "_seq": 1,
+  "author_id": "557058:aa11bb22", "author_display": "Jane Doe",
+  "field_id": "status", "field_name": "Status", "field_cardinality": "single",
+  "delta_action": "set", "delta_value_id": "10000", "delta_value_display": "To Do",
+  "value_ids": ["10000"], "value_displays": ["To Do"], "value_id_type": "opaque_id",
+  "collected_at": "2026-07-16T02:00:00Z"
+}
+```
+
+```json
+{
+  "insight_source_id": "jira-main", "data_source": "insight_jira",
+  "unique_key": "jira-main-insight_jira-PROJ-123-status-45021",
+  "person_id": "0197c8d1-2e4f-7a6b-8c9d-0e1f2a3b4c5d",
+  "_version": 1752671100000,
+  "issue_id": "10500", "id_readable": "PROJ-123",
+  "event_id": "45021", "event_at": "2026-07-12T16:45:00Z",
+  "event_kind": "changelog", "_seq": 0,
+  "author_id": "557058:cc33dd44", "author_display": "Bob Smith",
+  "field_id": "status", "field_name": "Status", "field_cardinality": "single",
+  "delta_action": "set", "delta_value_id": "10002", "delta_value_display": "Готово",
+  "value_ids": ["10002"], "value_displays": ["Готово"], "value_id_type": "opaque_id",
+  "collected_at": "2026-07-16T02:00:00Z"
+}
+```
+
+### 6.4 Cases
+
+#### Case: enriched events vs raw field-change deltas
+
+**Question:** how do the records in this dataset differ from the raw changelog deltas the source APIs return, and which part do I use?
+
+**Rule:** a raw delta only says *what changed* (from → to), exists only for fields somebody touched, and carries no state. The enrich step turns deltas into **events** that add three things:
+
+1. **Running state.** Every event carries the *complete field value after the event* in `value_ids` / `value_displays` — for multi-value fields the whole resulting set, not just the added/removed element. Gold never folds deltas to reconstruct state; enrich already did it.
+2. **Creation baseline.** `synthetic_initial` events reconstruct the value of *every* field at issue-creation time — including fields never touched by any changelog — plus the `field_id = "created"` marker event. So every field has a complete timeline from creation, and "state at time T" is always answerable.
+3. **Deterministic order.** `(event_at, _seq, _version)` totally orders the history.
+
+Use `value_ids` / `value_displays` when you need state (current status, assignee at time T). Use `delta_action` / `delta_value_*` when you need the change itself (what was added to labels, who flipped the status). Use `event_kind` to separate real activity (`changelog`) from reconstructed baseline (`synthetic_initial`) — e.g. "number of status changes" counts only `changelog` events.
+
+#### Case: determining that a task is closed
+
+**Question:** how do I tell a task is closed, and when it was closed?
+
+**Rule:** a task is closed iff the **category of its current status is `done`**. Never match status display names (`Closed`, `Done`, `Готово`, …) — names are workflow- and locale-specific. Join the status id (`value_ids[1]`) to the `task_statuses` dimension ([§6.5](#65-auxiliary-dimension-task_statuses)) to get the source-neutral `status_category` (Jira `statusCategory.key`, YouTrack `isResolved`). The close timestamp is the `event_at` of the latest event that put the task into a done-category status.
+
+```sql
+WITH status_events AS (
+    SELECT fh.insight_source_id, fh.id_readable,
+           fh.event_at, fh._seq, fh._version,
+           st.status_category
+    FROM task_field_history AS fh FINAL
+    LEFT JOIN task_statuses AS st FINAL
+        ON  st.insight_source_id = fh.insight_source_id
+        AND st.status_id = fh.value_ids[1]
+    WHERE fh.field_id = 'status'
+)
+SELECT id_readable,
+       argMax(status_category, (event_at, _seq, _version)) AS current_category,
+       maxIf(event_at, status_category = 'done')           AS closed_at
+FROM status_events
+GROUP BY insight_source_id, id_readable
+HAVING current_category = 'done'
+```
+
+Note the two-part semantics: a task that passed through a done status but was later reopened is **not closed** (fails `HAVING`); a task that was reopened and closed again is closed at the *latest* transition into done.
+
+#### Case: detecting reopens
+
+**Question:** how do I count reopened tasks?
+
+**Rule:** order the `field_id = 'status'` events of an issue by `(event_at, _seq, _version)`, resolve each event's `status_category` via the `task_statuses` join, and compare adjacent events: a **reopen** is a transition from `done` to any non-`done` category; a **close** is a transition from non-`done` to `done`. Use window `lagInFrame(status_category)` per issue.
+
+#### Case: field value at an arbitrary time T
+
+**Question:** what was the assignee (or any field) of a task on a given date?
+
+**Rule:** take the latest event for that `(issue, field)` with `event_at <= T`, ordered by `(event_at, _seq, _version)`; its `value_ids` / `value_displays` are the answer. Thanks to `synthetic_initial` baselines this always yields a value for any `T >=` issue creation; if no event qualifies, the issue did not exist yet at `T`.
+
+#### Case: task creation time
+
+**Question:** when was a task created?
+
+**Rule:** the `event_at` of its `field_id = 'created'` event (`event_kind = 'synthetic_initial'`, `_seq = 0`). Do not use `min(event_at)` across all fields — it is equal today but is not the contract.
+
+### 6.5 Auxiliary dimension: task_statuses
+
+One record per (source × status id), from `silver.class_task_statuses`. Maps workflow statuses to the locale-independent category used for close detection.
+
+| Field | Type | Description |
+|---|---|---|
+| `insight_source_id`, `data_source`, `unique_key`, `_version` | envelope | See [§2.1](#21-envelope-fields). |
+| `status_id` | String | Status id; joins `task_field_history.value_ids[1]` for `field_id = 'status'` events. |
+| `status_name` | String | Display name (locale/workflow-specific — never used for logic). |
+| `category_id`, `category_key` | Nullable | Raw vendor category (Jira `statusCategory.id` / `.key`). |
+| `status_category` | String | Unified category: `new`, `in_progress`, `done`, `undefined`. Jira: from `statusCategory.key`; YouTrack: from the State bundle's `isResolved` flag. |
+| `collected_at` | DateTime | Sync time. |
+
+Example:
+
+```json
+{
+  "insight_source_id": "jira-main", "data_source": "insight_jira",
+  "unique_key": "jira-main-insight_jira-status-10002",
+  "_version": 1752671100000,
+  "status_id": "10002", "status_name": "Готово",
+  "category_id": "3", "category_key": "done",
+  "status_category": "done",
+  "collected_at": "2026-07-16T02:00:00Z"
+}
+```
+
+## 7. Messenger Chat Activity
+
+What exists today for messengers is **not per-message data**. Vendor *analytics/report* endpoints provide only **daily per-user activity counters**; that is what `silver.class_collab_chat_activity` holds and what this contract fixes: one record per (user × date × source), covering Slack, MS Teams (M365), and Zulip. There is no channel/thread/message-level structure, no message content, and no edit/delete signal.
+
+Several columns are structurally `NULL` depending on the vendor (each vendor reports a different subset of counters) — see the honest-NULL case below. A per-message dataset is a **future extension** ([§7.5](#75-future-per-message-dataset)).
+
+### 7.1 JSON Schema
+
+```json
+{
+  "$schema": "https://json-schema.org/draft/2020-12/schema",
+  "$id": "insight/contracts/silver-gold/chat_activity.schema.json",
+  "title": "Daily Chat Activity",
+  "type": "object",
+  "properties": {
+    "tenant_id":                 { "type": "string", "minLength": 1 },
+    "insight_source_id":         { "type": "string", "minLength": 1 },
+    "data_source":               { "type": "string", "enum": ["insight_slack", "insight_m365", "insight_zulip_proxy"] },
+    "unique_key":                { "type": "string", "minLength": 1 },
+    "person_id":                 { "type": ["string", "null"], "format": "uuid" },
+    "_version":                  { "type": "integer", "minimum": 0 },
+    "user_id":                   { "type": "string" },
+    "user_name":                 { "type": "string" },
+    "email":                     { "type": "string" },
+    "person_key":                { "type": "string" },
+    "date":                      { "type": "string", "format": "date" },
+    "direct_messages":           { "type": ["integer", "null"], "minimum": 0 },
+    "group_chat_messages":       { "type": ["integer", "null"], "minimum": 0 },
+    "direct_and_group_messages": { "type": ["integer", "null"], "minimum": 0 },
+    "total_chat_messages":       { "type": "integer", "minimum": 0 },
+    "channel_posts":             { "type": ["integer", "null"], "minimum": 0 },
+    "channel_replies":           { "type": ["integer", "null"], "minimum": 0 },
+    "urgent_messages":           { "type": ["integer", "null"], "minimum": 0 },
+    "report_period":             { "type": ["string", "null"] },
+    "collected_at":              { "type": "string", "format": "date-time" }
+  },
+  "required": [
+    "tenant_id", "insight_source_id", "data_source", "unique_key", "person_id",
+    "_version", "user_id", "person_key", "date", "total_chat_messages"
+  ]
+}
+```
+
+### 7.2 Fields
+
+| Field | Description |
+|---|---|
+| `person_id` | Resolved person for this activity row. |
+| `user_id` | Raw source user id: Slack user id; M365 `userPrincipalName`; Zulip lowercased email. Traceability only. |
+| `user_name`, `email`, `person_key` | Raw identity. `person_key = lower(email)` (fallback `lower(user_id)`) — the legacy cross-source join key; superseded by `person_id`, kept for traceability. |
+| `date` | Activity date, UTC. |
+| `total_chat_messages` | Headline count: all messages the user sent that day, all surfaces combined. The only counter guaranteed non-NULL across vendors. |
+| `direct_messages` | 1:1 chat messages. M365 only; `NULL` for Slack/Zulip. |
+| `group_chat_messages` | Group-chat messages. Currently `NULL` for all vendors. |
+| `direct_and_group_messages` | DMs + group chats combined. Slack and M365; `NULL` for Zulip. |
+| `channel_posts` | New top-level channel posts. Slack and M365; `NULL` for Zulip. |
+| `channel_replies` | Channel thread replies. M365 only. |
+| `urgent_messages` | Messages marked urgent. M365 only. |
+| `report_period` | Vendor report window marker (M365 only). |
+| `collected_at` | Sync time. |
+
+### 7.3 Example
+
+```json
+{
+  "tenant_id": "acme", "insight_source_id": "m365-main", "data_source": "insight_m365",
+  "unique_key": "9c3f2e1d0b8a7654fedcba9876543210",
+  "person_id": "0197c8d1-2e4f-7a6b-8c9d-0e1f2a3b4c5d",
+  "_version": 1752658201000,
+  "user_id": "bob.smith@acme.com", "user_name": "bob.smith@acme.com",
+  "email": "bob.smith@acme.com", "person_key": "bob.smith@acme.com",
+  "date": "2026-07-15",
+  "direct_messages": 12, "group_chat_messages": null,
+  "direct_and_group_messages": 12, "total_chat_messages": 19,
+  "channel_posts": 4, "channel_replies": 3, "urgent_messages": 0,
+  "report_period": "7", "collected_at": "2026-07-16T02:00:00Z"
+}
+```
+
+### 7.4 Cases
+
+#### Case: counting messaging activity
+
+**Question:** how many messages did a person send in a period?
+
+**Rule:** sum `total_chat_messages` per `person_id` per `date`. It is the only counter comparable across vendors. Do not add `direct_* + channel_*` yourself — the vendor-specific counters overlap differently per vendor (`total ≈ dm + channel` holds only approximately).
+
+#### Case: honest NULLs in vendor counters
+
+**Question:** a counter is `NULL` — is that zero activity?
+
+**Rule:** no. `NULL` means **the vendor does not report this counter**, not "zero". When aggregating a nullable counter, keep the distinction: emit `NULL` when *all* contributing rows are `NULL`, and a sum otherwise (`if(countIf(x IS NOT NULL) > 0, sumIf(x, x IS NOT NULL), NULL)`). Never `coalesce(x, 0)` a vendor counter — it silently converts "not measured" into "inactive".
+
+#### Case: direct vs channel communication split
+
+**Question:** how do I split DM load from channel communication?
+
+**Rule:** the split is only available where the vendor reports it: use `direct_and_group_messages` (Slack, M365) vs `channel_posts + channel_replies` (with the honest-NULL rule). Zulip rows support only the total. Report the split per-vendor or restrict to sources that provide it — do not fabricate a split from the total.
+
+### 7.5 Future: per-message dataset
+
+When source permissions allow message-level ingestion (Slack `*:history` scopes, MS Graph `Chat.Read.All`), a `messages` dataset will be added: one record per message with `message_id`, `channel_id`/`channel_type` (`channel` / `dm` / `group_dm`), `thread_id` + `is_thread_reply`, `sent_at` / `edited_at` / `is_deleted` (tombstone), sender `person_id` — **without message content**. Until then, all messenger metrics must be expressible over the daily aggregates above.
+
+## 8. Extending this contract
+
+- **New case** (interpretation rule): append a `#### Case:` block to the relevant dataset section — format: *Question / Rule / optional SQL sketch*. This is a non-breaking, documentation-only change, but it is normative once merged: gold models must follow it.
+- **New field:** add to the JSON Schema (optional first), the field table, and the example. Non-breaking.
+- **New dataset:** new top-level section with the same structure (schema, fields, example, cases). Next candidates: PR reviewers/approvals (`class_git_pull_requests_reviewers`), per-message chat data ([§7.5](#75-future-per-message-dataset)).
+- **Breaking change** (rename, retype, semantics change): requires a contract version bump and a dual-publish migration window.
+
+## Appendix A. Gaps between this contract and the current implementation
+
+The schemas above match the current silver tables; the items below are where the contract demands more than silver delivers today (2026-07-16):
+
+| Gap | Detail |
+|---|---|
+| `person_id` not attached anywhere | No dataset carries a resolved `person_id` today. Git models expose interim `person_key` (lowercased email/login) on `fct_git_*` only; task gold joins `class_task_users` by email at read time; chat rows carry `person_key`. The Identity Resolution machinery (persons registry, UUIDv7, `identity_inputs`) exists but git connectors emit no `identity_inputs` at all; comment authors are never resolved. |
+| `patch_id` missing | No git connector computes `git patch-id`. Commit identity today is `commit_hash` only, so cross-branch dedup falls back to hash equality and misses rebases/cherry-picks. |
+| `task_field_history` has no tenant column | The table (and its `unique_key` formula) carries `insight_source_id` but no `insight_tenant_id`. Single-tenant-safe today; the DB split makes a tenant column mandatory. |
+| No `merged_by` / `closed_by` on PRs | Only the author identity is captured; Bitbucket's `closed_by` is used internally for the `closed_at` heuristic but not surfaced. |
+| GitLab PR diff stats | `files_changed` / `lines_added` / `lines_removed` are constant `0` for GitLab MRs (not collected). |
+| Per-message data absent | Only daily per-user aggregates exist (vendor analytics endpoints). Message-level ingestion requires new source scopes — [§7.5](#75-future-per-message-dataset). |