Skip to content

feat: establish external connector plugin architecture - #1

Draft
sabino wants to merge 1 commit into
mainfrom
feat/connector-plugins
Draft

feat: establish external connector plugin architecture#1
sabino wants to merge 1 commit into
mainfrom
feat/connector-plugins

Conversation

@sabino

@sabino sabino commented Aug 11, 2026

Copy link
Copy Markdown
Member

Why

Upstream PeerDB PR #3204 is useful context, but adding connector code through the current static Go registry still requires rebuilding Core and preserving in-process coupling. This PR establishes the staged external connector boundary instead.

It adopts Metabase's metadata-first discovery, lazy activation, capability reporting, and community packaging model, while deliberately using supervised subprocesses rather than a shared classloader or Go .so ABI.

Companion catalog PR: https://github.com/sabino-software/peerdb-plugins/pull/1

Included

  • a complete architecture RFC covering all 35 current connector interfaces, Flow/Temporal session scopes, UI/Nexus/Switchboard migration, security, deployment, compatibility, failure semantics, and per-connector extraction order
  • experimental peerdb.plugin.v1 protobuf services for control, schema, CDC, QRep, table lifecycle, diagnostics, query, and narrowly scoped host callbacks
  • canonical typed values, bounded streaming frames, snapshot/replication leases, structured errors, and durable CDC/QRep mutation-status recovery
  • strict manifest discovery with immutable parsed-byte digests, path/artifact checks, dependency DAGs, side-by-side versions, and connector-scoped capabilities
  • fail-closed lock reconciliation pinned to artifact + manifest digests, protocol, platform, connectors, and verified trust attestation
  • registry/lock → supervisor adapter with explicit Core-version compatibility checks
  • supervised direct process launch over private 0600 Unix sockets, launch-token HMAC proof, standard gRPC health, protocol/format/compression negotiation, generation-bound leases, crash invalidation, drain/shutdown, bounded jitter, and a restart circuit breaker
  • additive PeerV2/connector identity messages and a catalog migration that preserves the legacy enum/oneof path
  • pinned CI for Buf generation/lint plus Go tests for the four foundation packages

Correctness invariants

  • Core remains the source→destination broker and owns backpressure/checkpoints.
  • Destination durability precedes Core catalog checkpoint persistence; only then may the source receive an ACK.
  • Unknown commit outcomes are recovered by stable mutation keys and status lookup, never by inventing a new batch key.
  • Stateful snapshot/replication handles are process-generation leases and are never silently recreated after a crash.
  • Runtime roles/capabilities/formats may only narrow the signed manifest, including per-connector scoping.
  • No plugin receives the PeerDB catalog DSN, Temporal client, or an unrestricted host SQL API.

Deliberately not in this PR

This is the mergeable foundation, not a false “all connectors extracted” claim. Existing connectors remain on the legacy in-process path. No external artifact can be enabled until a real signature verifier supplies a matching attestation.

The following implementation slices remain:

  • Temporal activity/data-plane adapters and the reverse host-callback server
  • descriptor APIs and dynamic UI/Nexus/Switchboard routing
  • SDK/conformance/fault-injection suites
  • content-addressed packaging/sandbox policy
  • connector-by-connector extraction in the companion repository

Windows launch-token delivery fails closed until an inherited HANDLE/named-pipe contract is implemented.

Validation

Locally completed:

  • Buf lint, build, and all-language generation for peerdb.plugin.v1
  • JSON Schema/catalog/lock invariants for the companion repository
  • gofmt and Go syntax/static generated-symbol checks
  • fixture executable digest/mode, workflow YAML, documentation links, native-library rejection, and whitespace checks

A Go toolchain was unavailable in the construction environment, so this PR's new pinned GitHub Actions job is the authoritative go test compilation gate.

Current CI infrastructure status

GitHub did not execute test steps for these drafts. In the Core fork, every pull-request workflow—including unrelated existing Flow, UI, CodeQL, migration, and lint workflows—failed before its first step; the new protocol job likewise has zero steps and no downloadable log. In the plugin repository, the validation run ended as startup_failure with zero jobs. This is consistent with an Actions policy/runner/billing startup block, not a test assertion. Enable or repair Actions for the forks and rerun before treating either PR as compile-validated.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant