Migrate production VM from GCP to Hetzner (April 2027)

[ulanpy]

Summary

Replace the GCE production VM (nuspace-instance) with a Hetzner VPS around April 2027, when the current GCP commitment ends. Keep GCP managed services where it still makes sense (GCS, Pub/Sub, Cloud Run schedule-sync, Secret Manager, WIF) unless we explicitly decide to leave GCP entirely later.

Goal: cheaper compute without rewriting the product — mainly host identity, deploy path, DNS/firewall, and backups.

Context

• Today: single GCE VM runs the full Docker stack (fastapi, postgres, meili, nginx, wg-easy, monitoring).
• Deploy: GitHub Actions → WIF → IAP SSH → Ansible.
• Runtime ADC: containers rely on GCE metadata (metadata.google.internal) for WAL-G/backups, host-side gcloud secrets, and Gemini enterprise.
• Cloudflare terminates public HTTP(S); VPN is WireGuard (vpn.nuspace.kz).

Migration checklist

1. Host & networking

• Provision Hetzner VPS (CPU/RAM sized for fastapi + meilisearch + postgres + monitoring; start from current e2-standard-2 baseline and leave headroom).
• Configure firewall: Cloudflare-only 80/443, WireGuard UDP 51820, SSH locked down (VPN-only or key + allowlist).
• Point Cloudflare DNS (nuspace.kz, vpn.nuspace.kz, related records) to the new origin IP.
• Re-issue / reinstall Cloudflare origin certs on the new host.
• Bootstrap WireGuard / wg-easy and publish new endpoint to peers.

2. Deploy path (IAP → normal SSH)

• Replace IAP/OS Login flow in .github/workflows/deploy.yml with SSH key (or Hetzner-compatible access).
• Update Ansible inventory generation (drop gcloud compute start-iap-tunnel ProxyCommand / OS Login certs).
• Update ansible/roles/prerequisites and secrets so the host does not depend on GCE metadata ADC.
• Document new SSH/VPN operator access (docs/ssh-access.md, infra/wg-easy/README.md).

3. Credentials: metadata → JSON via env/secrets

• Stop relying on metadata.google.internal in infra/prod.docker-compose.yml (extra_hosts on fastapi/postgres/backup/migrate).
• Provide GOOGLE_APPLICATION_CREDENTIALS / SA JSON for services that still talk to GCP (storage, backups, optional AI).
• Prefer injecting signing + runtime secrets from Secret Manager into .env on deploy (same pattern as today, but without VM SA metadata).
• Verify signed URL flow still works with GCP_SIGNING_SERVICE_ACCOUNT_KEY_JSON.
• Decide Gemini path: keep enterprise with SA JSON, or switch to API key mode.

4. Backups (highest silent-failure risk)

• Make WAL-G / pg_dump work with explicit SA credentials (not GCE ADC).
• Confirm uploads to nuspace-backups-prod (or chosen bucket) after cutover.
• Run restore drill once on Hetzner before decommissioning the GCP VM.

5. App integrations that stay on GCP (for now)

• Pub/Sub push to https://nuspace.kz/api/bucket/gcs-hook still reaches the new origin after DNS cutover.
• Confirm OIDC audience/SA (PUSH_AUTH_*) unchanged and verified.
• Cloud Run schedule-sync job still writes catalog JSON to GCS; fastapi ingest still works.
• Media bucket CORS / uploads / signed URLs smoke-tested.

6. Terraform / GCP cleanup

• Split or retire terraform/compute.tf + VM-only firewall/IAP IAM after cutover.
• Keep (initially): GCS media/backups, Pub/Sub, Cloud Run schedule-sync, Secret Manager, signing SA, WIF for CI→GCP APIs.
• Release/stop GCP VM and static IP once Hetzner is stable.
• Cancel/avoid renewing the GCP compute commitment after April 2027.

7. Cutover plan

• Staging rehearsal on a small Hetzner box (deploy + backups + pubsub hook + VPN).
• Maintenance window: drain traffic / brief Cloudflare cutover.
• Post-cutover checks: homepage, auth, planner/search, uploads, backup push, VPN monitoring.
• Keep GCP VM stopped (not deleted) for a short rollback window.

Out of scope (unless we decide later)

• Leaving GCS / Pub/Sub / Secret Manager entirely
• Rewriting google_bucket / signed URL stack for S3-compatible storage
• Multi-node / k8s — still one VM + Docker Compose

Success criteria

• Production serves on Hetzner behind Cloudflare with no IAP dependency.
• Backups land in GCS (or chosen store) for ≥7 consecutive days.
• Deploy from dev/main works via SSH+Ansible.
• GCP compute VM can be powered off without user-facing impact.