Implement portable operator-managed HA seed artifacts for bootstrap and reseed

[bpopadiuk]

Codex (GPT-5): This is the Antfly foundation required to close the no-out-of-band-bootstrap and automatic-reseed acceptance criteria in https://github.com/antflydb/colony/issues/343. It follows from the gaps proven by https://github.com/antflydb/colony/pull/356 and assumes the runtime/controller race fixes in #347.

Problem

Antfly has the correctness primitives for HA base backup and standby bootstrap, but it does not move a complete seed artifact between isolated runtime pods.

Current behavior:

• POST /admin/v1/ha/base-backups reserves a slot and backup boundary.
• POST /admin/v1/ha/base-backups/finish consumes a manifest from a pod-local path.
• POST /admin/v1/ha/standby/bootstrap consumes a manifest plus files that must already exist at a pod-local path on the standby.
• HAStandbySpec.seedManifestPath and seedContentRoot let the operator plan begin/finish/bootstrap actions, but the operator does not create, transfer, download, or lifecycle-manage the referenced artifact.
• The current Colony HA E2E has to read /antflydb/metadata/local-metadata.json, construct a manifest, copy files into another pod, and restart that pod. That proves the runtime primitive, not a production bootstrap workflow.
• Reseed similarly records that a base backup is required; without a transferable artifact, it cannot rebuild an isolated former-primary PVC.

A pod-local path is not a production transport. Primary and standby commonly use different ReadWriteOnce PVCs and may be scheduled on different nodes.

Goal

Implement an operator-owned, crash-resumable seed-artifact workflow that creates a complete immutable base backup from the current primary and restores it into a fresh or explicitly approved empty standby data root without kubectl cp, kubectl exec, hostPath sharing, or test-only catalog copying.

At least one production-capable transport must be implemented. Object storage is the preferred default because it works across nodes, zones, and ReadWriteOnce PVCs. A pluggable transport contract is acceptable, but this issue cannot be closed with only a KinD shared-filesystem shortcut.

Required design

Artifact identity and integrity

• Key every seed generation by cluster/shard/table identity, source timeline and epoch, slot name, backup/checkpoint LSN, and an operator-chosen immutable generation ID.
• Include every file required to reopen the runtime, not only the metadata catalog.
• Verify manifest framing, paths, sizes, per-file checksums, aggregate identity, and backup boundary before modifying the target data root.
• Reject cross-cluster, cross-timeline, stale-generation, path-traversal, partial-upload, and checksum-mismatch artifacts.
• Publish an artifact only after upload is complete; consumers must never observe a partially written generation.

Controller workflow

Model durable phases in spec/status or a dedicated subresource/CR:

1. reserve slot and backup boundary;
2. capture and upload the complete content set;
3. finish/finalize the manifest;
4. make the immutable generation available;
5. restore into a fresh/empty target;
6. bootstrap and verify runtime identity;
7. start streaming from the declared boundary;
8. mark the seed complete only after durable receive/apply evidence;
9. garbage-collect artifacts after a bounded retention period.

Reconciliation must be idempotent across operator restart, Job restart, pod replacement, duplicate API calls, and status-update conflicts. Each phase needs typed receipts; terminal failures must be visible and must not be retried forever without backoff or operator-visible reason.

Target safety

• Never overwrite a running primary or an unfenced former primary.
• Require explicit proof that the target is a standby/fresh replacement or carries the matching durable former-primary fence.
• Restore through a staging directory or fresh PVC and atomically activate only after full validation.
• Preserve the old data root for diagnosis until the new artifact is verified, or document and test the exact rollback boundary.
• Do not mark a slot or standby healthy until the restored process is streaming and its applied/safe-read LSN satisfies the declared boundary.

Security and operations

• Use Secret references for object-store credentials; never copy credential values into CR status, Jobs, events, or logs.
• Support encryption in transit and the configured storage encryption-at-rest mechanism.
• Bound artifact size, upload/download time, retry backoff, and retained generations.
• Expose phase, generation, source identity, boundary LSN, bytes/files transferred, retry count, last error class, and timestamps through status and metrics.
• Cleanup must be safe under cancellation, instance deletion, and superseded generations.

Likely code boundaries

• zig/pkg/antfly/src/storage/ha/bootstrap.zig
• zig/pkg/antfly/src/storage/ha/backup_manifest.zig
• zig/pkg/antfly/src/storage/ha/admin.zig
• zig/pkg/antfly/src/storage/ha/admin_exec.zig
• specs/openapi/antfly/admin.yaml
• go/pkg/sdk/admin/ha.go
• go/pkg/operator/api/antfly/v1/antflycluster_types.go
• go/pkg/operator/controllers/antfly/ha_status.go
• go/pkg/operator/controllers/antfly/antflycluster_controller.go

The implementation may introduce a dedicated seed Job or CR/status model if that makes ownership and retryability clearer than extending planned-action strings.

Red/green test requirements

Tests must first fail against the current implementation because no cross-pod artifact transport exists, then pass with this work.

• Runtime unit tests: incomplete manifest, wrong identity/timeline/epoch, path traversal, checksum mismatch, duplicate generation, interrupted activation, and safe retry.
• Operator tests: phase transitions and typed receipts; restart/reconcile at every phase; duplicate Jobs; stale/superseded generation; target-fence validation; bounded failure state.
• Real-process test: primary and standby use isolated directories with no shared filesystem; seed all required runtime state; restart both processes; verify exact data and continued streaming.
• KinD test: separate primary/standby PVCs, delete/recreate the standby pod during upload and restore, then prove the same generation converges without manual file copying.
• Reseed test: diverged fenced former primary requires base backup, is rebuilt from a current-primary artifact, rejoins as a standby, and does not become healthy before apply catches up.

Acceptance criteria

• No user, fixture, or Cloud worker must copy files between pods.
• A complete seed can cross nodes and isolated ReadWriteOnce PVCs.
• The workflow survives controller, Job, and pod restarts without corruption or duplicate side effects.
• A bad or partial artifact never replaces a target data root.
• Status clearly distinguishes requested, capturing, transferring, restoring, bootstrapping, catching up, ready, failed, and superseded.
• Both initial bootstrap and former-primary reseed can consume the same production mechanism.
• Documentation includes failure recovery, artifact retention/cleanup, credential boundaries, and operational metrics.
• The no-out-of-band-bootstrap path is demonstrated in local KinD and is suitable for the Colony black-box HA E2E.

Out of scope

• Choosing which Cloud node should become the replacement standby.
• Persisting Cloud product topology in Colony.
• Changing customer billing or HA entitlement behavior.