feat(bigquery): select only those columns that were configured via table mappings

[dtunikov]

Select columns configured for the table explicitly instead of relying on SELECT * EXCEPT a, b, c FROM ....
Edge case - If a column was removed on the source, but wasn't removed in PeerDB:

• query all columns configured on the TableMapping lvl
• get an error from BQ
• exclude removed column from the selection set
• store it in in-memory connector map with TTL
  This means that there could be gaps in the data for such column if it was removed and re-added later on the source (up to configured TTL)

[claude]

Code review — found 1 issue (checked for bugs and CLAUDE.md compliance):

bqUnrecognizedNameRe swallows BigQuery's ; Did you mean …? suffix, defeating the missing-column recovery path

peerdb/flow/connectors/bigquery/cdc.go

Lines 397 to 399 in 856f861

	// Matches BigQuery's error for a SELECT list column that doesn't exist on the source,
	// e.g. "Unrecognized name: foo at [1:8]" (verified against a live table).
	var bqUnrecognizedNameRe = regexp.MustCompile(`Unrecognized name: (\S+)`)

BigQuery's unrecognized-name errors frequently carry a suggestion when a similarly-named column is in scope, e.g. Unrecognized name: old_col; Did you mean new_col? at [1:8] — and a rename (drop + re-add under a close name) is one of the most common ways a mapped column goes missing. With (\S+), the captured token is old_col; (trailing semicolon included), which then fails the slices.Contains(candidates, col) check in missingSourceColumn, so runPullQuery propagates the error instead of dropping the column — every subsequent poll for the table fails identically, and the recovery this PR adds never triggers in exactly the scenario it targets.

Suggested fix — stop the capture at the suggestion delimiter:

var bqUnrecognizedNameRe = regexp.MustCompile(`Unrecognized name: ([^;\s]+)`)

and add a regression case to TestMissingSourceColumn with the ; Did you mean …? form, e.g. "Unrecognized name: secret_column; Did you mean secret_columns? at [1:8]".

---

Also verified but not flagged: the req.TableNameSchemaMapping compile error at the previous head (934edb4) is resolved by the latest push, which adds the field to model.PullTableRecordsRequest and populates it in flowable_isolated_cdc.go; and the narrow lock scope in effectiveColumns (inner map read after Unlock()) is unreachable as a race in supported configurations since pull loops are per table mapping, though Lock()/defer Unlock() would be cheap hygiene.

[claude]

Code review

Reviewed for bugs and CLAUDE.md compliance. Two issues found, both in the new missing-column handling:

1. Regex misses BigQuery's "Did you mean" error variant, defeating the missing-column recovery (bug)

flow/connectors/bigquery/cdc.go L397-L417

When a similarly-named column exists in scope, BigQuery emits the suggestion form of this error: Unrecognized name: val; Did you mean value? at [1:8]. With (\S+) being greedy over non-whitespace, match[1] becomes val; (including the semicolon), so the slices.Contains(candidates, col) check in missingSourceColumn fails and runPullQuery propagates the raw error instead of dropping the column and retrying — replication for that table halts, which is exactly the scenario this PR is meant to survive. Since a common cause of a missing column is a rename (e.g. email → email_address), the suggestion form is quite likely in practice. The old SELECT * EXCEPT regex was immune because its capture was bounded on both sides by literals.

Bounding the capture at ; fixes it:

var bqUnrecognizedNameRe = regexp.MustCompile(`Unrecognized name: ([^\s;]+)`)

Worth also adding a TestMissingSourceColumn case for the suggestion form ("Unrecognized name: secret_column; Did you mean secret_columns? at [1:8]") — the current tests only cover the plain variant.

2. New dynamic setting targeted at the wrong connector, hiding it in ClickHouse-only deployments (bug, low severity)

flow/internal/dynamicconf.go L302-L310

PEERDB_BIGQUERY_CDC_MISSING_COLUMN_RETRY_SECONDS is declared with TargetForSetting: protos.DynconfTarget_BIGQUERY, but that target denotes BigQuery-as-destination settings. Its sibling BigQuery-source CDC settings — PEERDB_BIGQUERY_CDC_SAFETY_LAG_SECONDS and PEERDB_BIGQUERY_CDC_MAX_QUERY_WINDOW_SECONDS — deliberately use DynconfTarget_CLICKHOUSE because BigQuery-source CDC replicates into ClickHouse. In ClickHouse-only deployments, GetDynamicSettings filters out every setting whose target isn't ALL/CLICKHOUSE/POSTGRES, so the new setting is invisible in the settings UI exactly where BigQuery-source CDC runs, while its two siblings remain visible. (The runtime lookup path ignores TargetForSetting, so only UI visibility/configurability is affected.)

Fix:

TargetForSetting: protos.DynconfTarget_CLICKHOUSE,

---

No CLAUDE.md/REVIEW.md violations found (no new logging of PII/secrets; no dependency bumps).

[github-actions]

🔄 Flaky Test Detected

Analysis: The lone failure, TestCockroachDBSuite/Test_CDC_Exactly_Once_Across_Batches, is a timing-sensitive CockroachDB changefeed test unrelated to this BigQuery-only PR — it passed on the other two matrix legs, and the connector's own code documents that its force-emit paths (grace window / batch deadline without a covering resolved timestamp) can emit past the checkpoint and replay records in the next batch.
Confidence: 0.92

✅ Automatically retrying the workflow

View workflow run

[github-actions]

🔄 Flaky Test Detected

Analysis: TestPeerFlowE2ETestSuiteMySQL_CH/Test_Partition_Key_Empty hit the hardcoded 60s "UNEXPECTED STATUS TIMEOUT STATUS_SNAPSHOT" poll limit at 62.58s on one matrix job while passing on the other two for the same commit — a slow-snapshot timing flake (already acknowledged in-repo via the "stuck in snapshot somehow" skip), unrelated to this BigQuery-scoped PR.
Confidence: 0.88

✅ Automatically retrying the workflow

View workflow run