Skip to content

feat: add source sync observability#103

Merged
jonathanhaaswriter merged 7 commits into
mainfrom
feat/source-sync-observability
Jun 15, 2026
Merged

feat: add source sync observability#103
jonathanhaaswriter merged 7 commits into
mainfrom
feat/source-sync-observability

Conversation

@jonathanhaaswriter

Copy link
Copy Markdown
Contributor

Summary

  • add typed per-source sync status, sync-now, and backfill RPCs
  • aggregate Google Reports, BigQuery, Directory, OAuth, and ingestion queue state
  • surface source health and recovery controls in the connectors UI
  • add wake listeners for Directory and OAuth sync workers

Validation

  • npm run db:generate
  • npm run db:validate
  • npm run typecheck
  • npm run guardrails:migration
  • npm run test:api
  • npm run proto:check
  • go test ./...
  • make lint
  • npm run build:web

Note: make test-go-db could not run locally because Docker/Colima is not running (missing /Users/jonathan/.colima/default/docker.sock).

@github-actions

github-actions Bot commented Jun 15, 2026

Copy link
Copy Markdown

Droid encountered an error —— View job


Security Review

Droid is reviewing code and running a security check…

@jonathanhaaswriter jonathanhaaswriter force-pushed the feat/source-sync-observability branch from b5cde13 to 3afaf62 Compare June 15, 2026 01:46
@github-actions

github-actions Bot commented Jun 15, 2026

Copy link
Copy Markdown

Droid finished @jonathanhaaswriter's task —— View job


Security Review

Validated 4 candidates and approved all 4. The source-level observability work is valuable, but the PR still has real regressions around rate limiting, wake-triggered sync behavior and error reporting, and dialog availability on unhealthy Google Workspace connectors.

Comment thread internal/bootstrap/sync_status.go
Comment thread cmd/google-workspace-directory-sync/main.go Outdated
Comment thread internal/googleworkspacedirectorysync/sync.go Outdated
Comment thread apps/web/components/connectors/connectors-page.tsx
@jonathanhaaswriter jonathanhaaswriter force-pushed the feat/source-sync-observability branch from 3afaf62 to 894894f Compare June 15, 2026 02:10
@github-actions

github-actions Bot commented Jun 15, 2026

Copy link
Copy Markdown

Droid finished @jonathanhaaswriter's task —— View job


Security Review

Validated 3 candidates and approved all 3. The new source observability surface still has real correctness gaps: per-source Sync ignores the selected stream, queued backfills look successful before workers run, and failed Directory or OAuth sweeps can appear current even when the underlying data is stale.

Comment thread internal/bootstrap/sync_status.go
Comment thread internal/bootstrap/sync_status.go
Comment thread internal/bootstrap/sync_status.go Outdated
@jonathanhaaswriter jonathanhaaswriter force-pushed the feat/source-sync-observability branch from 894894f to 2584fe6 Compare June 15, 2026 02:41
@github-actions

github-actions Bot commented Jun 15, 2026

Copy link
Copy Markdown

Droid finished @jonathanhaaswriter's task —— View job


Security Review

Validated all 3 review candidates for PR #103 and wrote /home/runner/work/_temp/droid-prompts/review_validated.json.

Approved and posted 3 batched review comments:

  • connector freshness still advances after single-stream Reports or BigQuery runs
  • -once workers can still drop later wake notifications while draining
  • source sync still accepts stream names that Directory and OAuth workers cannot honor

Comment thread internal/googleworkspacepoller/poller.go Outdated
Comment thread cmd/google-workspace-directory-sync/main.go Outdated
Comment thread internal/bootstrap/sync_status.go
@jonathanhaaswriter jonathanhaaswriter force-pushed the feat/source-sync-observability branch from 2584fe6 to 655fe79 Compare June 15, 2026 03:03
@github-actions

github-actions Bot commented Jun 15, 2026

Copy link
Copy Markdown

Droid finished @jonathanhaaswriter's task —— View job


Security Review

Validated 4 candidates and approved all 4. The source-level observability work is valuable, but the PR still has actionable correctness gaps around dialog state races, durable backfill queuing, wake-listener error handling, and BigQuery queue attribution.

Comment thread apps/web/components/connectors/connectors-page.tsx
Comment thread internal/bootstrap/sync_status.go
Comment thread cmd/google-workspace-directory-sync/main.go
Comment thread internal/bootstrap/sync_status.go Outdated
@github-actions

github-actions Bot commented Jun 15, 2026

Copy link
Copy Markdown

Droid finished @jonathanhaaswriter's task —— View job


Security Review

Validated 1 candidate and approved 1.

Sync all still has a real ordering bug: it can wake OAuth before Directory refreshes identities, which makes Shadow IT look current while omitting grants until a later OAuth run.

Comment thread internal/bootstrap/sync_status.go Outdated
@github-actions

github-actions Bot commented Jun 15, 2026

Copy link
Copy Markdown

Droid finished @jonathanhaaswriter's task —— View job


Security Review

Validated all 2 review candidates for PR #103 and wrote /home/runner/work/_temp/droid-prompts/review_validated.json.

Approved and posted 2 batched review comments:

  • Reports per-stream setup failures can leave queued or stale-success state behind
  • BigQuery setup failures can leave queued backfills stuck forever

Overall assessment: the source-level observability work is useful, but the PR still hides real wake-triggered setup failures for Reports and BigQuery streams.

Comment thread internal/googleworkspacepoller/poller.go
Comment thread internal/googleworkspacepoller/bigquery.go
@github-actions

github-actions Bot commented Jun 15, 2026

Copy link
Copy Markdown

Droid finished @jonathanhaaswriter's task —— View job


Security Review

Validated all 3 review candidates for PR #103 and wrote /home/runner/work/_temp/droid-prompts/review_validated.json.

Approved and posted 3 batched review comments:

  • once-mode wake draining can abort active sync work after a listener error
  • Sync all stays clickable for the wrong connector while another sync is queueing
  • errored source rows can regress seen counts to 0

Overall assessment: the source-level observability work is useful, but the PR still has actionable correctness gaps in once-mode wake draining, cross-connector Sync all state, and errored source row-count reporting.

Comment thread cmd/google-workspace-directory-sync/main.go
Comment thread apps/web/components/connectors/connectors-page.tsx Outdated
Comment thread internal/bootstrap/sync_status.go
@github-actions

github-actions Bot commented Jun 15, 2026

Copy link
Copy Markdown

Droid finished @jonathanhaaswriter's task —— View job


Security Review

Validated all 2 review candidates for PR #103 and wrote /home/runner/work/_temp/droid-prompts/review_validated.json.

Approved and posted 2 batched review comments:

  • per-source busy state can be lost across concurrent row actions
  • once-mode OAuth wake draining can cancel long-running manual syncs after 60 seconds

Overall assessment: The source observability work is useful, but the PR still has two actionable correctness gaps in concurrent row actions and once-mode OAuth wake handling.

Comment thread apps/web/components/connectors/connectors-page.tsx Outdated
Comment thread cmd/google-workspace-oauth-sync/main.go
@github-actions

github-actions Bot commented Jun 15, 2026

Copy link
Copy Markdown

Droid finished @jonathanhaaswriter's task —— View job


Security Review

Validated 1 candidate and approved it. The new source-level sync UI still exposes a real correctness gap: OAuth sync can be manually queued before Directory has seeded identities, which makes Shadow IT look current without scanning any users.

Comment thread internal/bootstrap/sync_status.go Outdated
@github-actions

github-actions Bot commented Jun 15, 2026

Copy link
Copy Markdown

Droid finished @jonathanhaaswriter's task —— View job


Security Review

I validated the lone remaining candidate and rejected it, so no new review comments were posted. The claimed same-row double-click race is not a high-confidence bug because React flushes the busy-state update between discrete clicks, and existing review feedback already covered the real busy-state regression this patch fixed.

@jonathanhaaswriter jonathanhaaswriter merged commit cf8c107 into main Jun 15, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant