Skip to content

feat: backups taken from a standby via pgBackRest multi-host TLS (draft PoC for #103)#104

Draft
melancholictheory wants to merge 1 commit into
operasoftware:mainfrom
melancholictheory:feat/backup-from-standby
Draft

feat: backups taken from a standby via pgBackRest multi-host TLS (draft PoC for #103)#104
melancholictheory wants to merge 1 commit into
operasoftware:mainfrom
melancholictheory:feat/backup-from-standby

Conversation

@melancholictheory

Copy link
Copy Markdown

This is a draft proof of concept for #103 (support backups taken from a standby). It is meant to make the design concrete and get direction on the operational pieces before investing further. It is not proposed as merge-ready.

Problem

CloudNativePG's default backup target is prefer-standby, so backups are dispatched to a replica to keep I/O off the primary. Today the plugin configures pgBackRest for a single local instance, so a backup that lands on a standby fails at stanza-create with pgBackRest exit 56 (unable to find primary cluster). Multi-instance clusters are forced back to target: primary.

As discussed on #103, simply skipping stanza-create on a standby is not enough: pgbackrest backup itself needs a primary among its configured hosts to run pg_backup_start/pg_backup_stop, so the real fix is to give pgBackRest a connection to the current primary (multi-host).

What this PR does

  • Adds an opt-in PgbackrestConfiguration.BackupStandby field (y/prefer/n, default disabled) that gates the feature.
  • When a backup runs on a standby and the feature is enabled, configures the current primary as a second pgBackRest host over TLS (--pg2-host ... --pg2-host-type tls ... --pg2-host-cert-file/key-file/ca-file ... --pg2-path/user/socket-path) and passes --backup-standby=<mode>.
  • Gives stanza-create the same primary peer (so it can initialise from a standby) without --backup-standby.
  • Discovers the primary from cluster.Status.CurrentPrimary (already used in the WAL path) and resolves it to the primary pod's IP. The pod is read uncached (added to the manager cache DisableFor, so it needs only get pods, not a cluster-wide Pod informer), and a pods get RBAC rule is declared. Falls back to a normal local backup when this instance is the primary or no primary is known.
  • Adds PgbackrestServerOptions (the pgbackrest server TLS-server invocation) as the client-visible half of the design.
  • Unit tests for the generated backup / stanza-create options (including the disabled no-op path and both y and prefer modes), the server options, the primary-peer decision, and resolveStandbyTopology (via a fake client).

Deliberately out of scope (needs maintainer direction)

The operational half from #103 is not wired here, because it depends on decisions I did not want to pre-empt:

  • Running the long-lived pgbackrest server process in each sidecar (lifecycle, port exposure).
  • Provisioning the pgBackRest TLS certificates (a dedicated chain vs reusing existing material) and the tls-server-auth identity.

Because of that, this cannot yet run a real standby backup end to end without those certificates and the server in place, and there is no e2e test flipping test/e2e/.../backup/fixtures.go off target: primary yet. The cert and server paths are constants and a command builder, documented as the contract the plugin expects.

Testing

go build ./..., go vet, gofmt, and the unit tests (go test ./internal/..., with envtest for the controller package) all pass. New unit tests assert the exact generated pgBackRest invocations, including that the disabled path is byte-for-byte unchanged.

Open questions

Same as #103. I would appreciate direction on the TLS server lifecycle, certificate provisioning, and primary discovery (pod IP vs a dedicated headless service) before extending this.

Backups that CloudNativePG schedules on a standby currently fail at
stanza-create with pgBackRest exit 56 (unable to find primary cluster),
because the plugin only ever configures a single local pgBackRest host. This
forces multi-instance clusters back to target: primary and puts all backup I/O
on the primary.

This adds an opt-in BackupStandby field (y/prefer/n, disabled by default) that,
when a backup runs on a standby, configures the current primary as a second
pgBackRest host over TLS (pg2-host-type=tls) and passes --backup-standby, so
pgBackRest coordinates pg_backup_start/stop on the primary while copying data
from the standby. The primary is discovered from the CloudNativePG cluster
status (currentPrimary) and resolved to the primary pod's IP via an uncached
read; a pods get permission is declared for it. stanza-create is also given the
primary peer so it can initialise from a standby, without --backup-standby.

This is a draft proof of concept. Config generation, primary discovery and unit
tests are included; the long-running pgbackrest TLS server process lifecycle and
its certificate provisioning are deferred pending maintainer direction.

Refs: operasoftware#103
Signed-off-by: melancholictheory <61789920+melancholictheory@users.noreply.github.com>
@melancholictheory
melancholictheory marked this pull request as ready for review July 22, 2026 19:46
@melancholictheory
melancholictheory marked this pull request as draft July 23, 2026 19:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant