Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 18 additions & 0 deletions docs/superpowers/plans/2026-08-23-gcp-foundation-resume.md
Original file line number Diff line number Diff line change
Expand Up @@ -97,6 +97,24 @@ flux reconcile source git flux-system
failure mode, `depends_on` gaps, and the `clusters/gcp-mycluster-0/` Flux wiring.
Two review agents produced nothing across repeated asks.

### Private certificates need their own OpenBao on GCP

**Decided 2026-08-24. Rationale and options live in the design under *Private
certificates on GCP*** — not repeated here, because this file is the one edited
on resume and the two would drift.

What matters for sequencing:

- **Workstream 11** (private certs, OpenBao on GCP: MIG + internal LB + Cloud KMS
auto-unseal) depends only on workstream 1, so it is **not blocked by slice 5**
and can start whenever.
- **Workstream 10** (external-dns + public certs) is blocked on an open question
the design records: GCP has only a *private* Cloud DNS zone, so DNS-01 has
nothing publicly resolvable to solve against. Settle that before starting it.
- The decision is **two OpenBaos, two roots** — so workstream 11 needs its own
GCP Secret Manager entries for the root token and cert-manager AppRole, and
tailnet clients end up trusting both roots.

### GPU quota — blocks slice 4's last criterion

`GPUS_ALL_REGIONS` on project `ogenki-435905` is **0**, so no GPU node can be
Expand Down
110 changes: 102 additions & 8 deletions docs/superpowers/specs/2026-08-18-gcp-support-design.md
Original file line number Diff line number Diff line change
Expand Up @@ -213,8 +213,8 @@ known rather than predicted.
| 7 | Extract the Crossplane Configuration packages, OCI-released | 3 | **DONE 2026-08-19 (v0.1.0)** |
| 8 | `objectStore` API migration + `App`/`SQLInstance` branching | 5, 7 | unblocked by 7 |
| 9 | Object-storage call sites: Harbor (GCS driver), `openbao-snapshot` (GCS + Cloud KMS), CNPG barman (GCS) | 5, 8 | |
| 10 | DNS + PKI: `external-dns` google provider, cert-manager clouddns DNS-01 | 5 | |
| 11 | OpenBao on GCP: MIG + internal LB + Cloud KMS auto-unseal | 1 | |
| 10 | DNS + PKI: `external-dns` google provider, cert-manager clouddns DNS-01 (**public** certs) — see [Private certificates on GCP](#private-certificates-on-gcp) | 5 | |
| 11 | OpenBao on GCP: MIG + internal LB + Cloud KMS auto-unseal (**private** certs) — see [Private certificates on GCP](#private-certificates-on-gcp) | 1 | |
| 12 | Gateway/LB: GCP public-LB annotations, drop `aws-load-balancer-controller` | 3 | |
| 13 | Storage: `gp3` → `pd-balanced`/hyperdisk, EFS CSI → Filestore CSI | 3 | |
| 14 | GPU + LLM platform: GPU `ComputeClass`, GCS Fuse weights, no `runtimeclass-nvidia` | 4, 9 | |
Expand Down Expand Up @@ -299,14 +299,22 @@ kind: GCPWorkloadIdentity
metadata: { name: external-dns, namespace: infrastructure }
spec:
serviceAccount: { name: external-dns, namespace: infrastructure }
roles: [roles/dns.admin] # predefined roles, named as GCP names them
customRole: # optional — creates + binds a custom role
permissions: [dns.resourceRecordSets.create]
roles: [projects/ogenki-435905/roles/xplane_dns_editor]
```

Renders one **`ProjectIAMMember`** per role, member = the `principal://` KSA string, plus a
`ProjectIAMCustomRole` when `customRole.permissions` is set. No Google service account, no
annotation. `xplane-*` prefix owned by the composition.
Renders one **`ProjectIAMMember`** per role, member = the `principal://` KSA string. No Google
service account, no annotation. `xplane-*` prefix owned by the composition.

> **`roles:` may only name roles the OpenTofu allowlist permits**, and as of 2026-08-24 that is the
> single pre-created `xplane_dns_editor` — *not* `roles/dns.admin`, which was the first draft and was
> dropped for carrying zone deletion and response policies (see `opentofu/gcp/gke/init/iam.tf` for
> the full reasoning). A claim naming anything else is refused by the IAM condition, not by the
> composition, so the error surfaces at the provider and names the role.
>
> **`customRole.permissions` is unusable on this platform** and should not appear in claims. The
> condition matches exact role names via `hasOnly`, and a role the composition names at render time
> cannot be allowlisted in advance. Adding a capability means adding a
> `google_project_iam_custom_role` in OpenTofu and referencing it — the same two-step every time.

**Two hard constraints, both silent-failure classes:**

Expand Down Expand Up @@ -351,6 +359,92 @@ Falsifiable, verified against a live cluster.
11. A written monthly run-rate estimate exists (cluster fee, static pool, Cloud NAT, Cloud DNS,
Tailscale instance) stating the zonal-vs-regional choice and its price delta.

### Private certificates on GCP

*Added 2026-08-24, while scoping slice 5. Workstreams 10 and 11 both touch
certificates and are easy to conflate; they are separable and only one needs
OpenBao.*

**The split.** Workstream 11 gives **private** certificates from OpenBao's own
PKI, the GCP counterpart to what `bao.priv.aws.ogenki.io` serves today, for names
under `priv.gcp.ogenki.io`. Workstream 10 covers `external-dns` plus **public**
certificates, which are for `cloud.ogenki.io` — [ADR-0017](../../../website/content/docs/decisions/0017-multi-cloud-dns-naming.md)
keeps the public zone cloud-agnostic and `priv.<cloud>.ogenki.io` private.

Only 11 requires running OpenBao, and it depends only on workstream 1 (network),
so it is not blocked by slice 5.

> **Open question, surfaced 2026-08-24 while writing this section: workstream 10's
> DNS-01 has nothing to solve against on GCP.** `opentofu/gcp/network/dns.tf`
> creates a **private** Cloud DNS zone and nothing else, while Let's Encrypt must
> resolve the `_acme-challenge` TXT record **publicly**. `cloud.ogenki.io` is a
> Route53 zone that this repository does not even manage — it appears only as a
> `data` lookup. So "cert-manager clouddns DNS-01" is not yet a complete plan.
>
> Three ways out, none chosen: solve DNS-01 against **Route53** from the GCP
> cluster (works today, but reintroduces exactly the cross-cloud dependency
> option A was rejected for); **delegate** a public subdomain to a new public
> Cloud DNS zone (clean, needs a registrar change and a public zone this repo
> would then own); or serve **no public certificates from GCP at all** and keep
> public ingress on AWS — which is coherent while GCP has no public endpoints.
>
> Worth settling before workstream 10 starts, not during it. Nothing about it
> affects workstream 11, which is self-contained.

**Why GCP needs its own OpenBao at all.** See the 2026-08-23 correction above:
OpenBao *the product* is cloud-agnostic, our OpenBao *stacks* are not.

#### Options considered

**A — Shared OpenBao.** GCP's cert-manager reaches `bao.priv.aws.ogenki.io` over
the tailnet. No new infrastructure, one CA, one trust anchor. Rejected: it makes
GCP certificate issuance hard-depend on AWS *and* on the tailnet, in a platform
whose stated point is that each cloud stands alone. It is the same coupling this
design already flags as undesirable for the Flux GitHub App secret.

**B — Two OpenBaos, two roots. CHOSEN.** Each cloud runs its own OpenBao with its
own root CA. Fully independent; matches [ADR-0007](../../../website/content/docs/decisions/0007-cloud-abstraction-boundaries.md)'s
rule that platform-facing infrastructure stays cloud-shaped. Costs one extra
trust anchor for tailnet clients.

**C — Two OpenBaos, one shared root.** Each cloud holds its own intermediate,
both signed by a common root: one trust anchor, independent operation. Rejected
on operational grounds — it requires either copying the root PRIVATE KEY into GCP
Secret Manager, doubling exposure of the most sensitive material the platform
holds, or cross-signing GCP's intermediate at bootstrap, which is a manual
ceremony **on every rebuild** of a platform whose lifecycle is
build-validate-destroy.

#### Why B, concretely

The 2026-08-24 rebuild made the argument better than theory could. The private
domain rename forced a new server certificate, and re-issuing it under the
existing chain proved impossible: the intermediate that had signed it had **no
private key stored anywhere**, because OpenBao issued it and the key never left
OpenBao. A fresh CA was the only way forward.

That is exactly the failure mode option C institutionalises. Cross-cloud PKI
ceremony is the first thing to break on a platform rebuilt this often, and it
breaks at rebuild time, which is the worst moment. Two independent roots cost one
extra trust anchor and remove the entire class.

#### Consequences to carry into workstream 11

- Tailnet clients must trust **both** roots. That is the accepted cost; it is a
one-line addition wherever the AWS root is already distributed.
- GCP needs its own secret store for the root token and the cert-manager AppRole
— **GCP Secret Manager**, mirroring what AWS Secrets Manager does today. The
`flux-github-app` secret already establishes that pattern on GCP.
- **Cloud KMS auto-unseal** is the GCP analogue of the AWS KMS unseal, already
named in the workstream row. It is what makes an unattended rebuild possible.
- The GCP PKI issues for `*.priv.gcp.ogenki.io` only; the AWS one keeps
`*.priv.aws.ogenki.io`. The per-cloud private domains from
[ADR-0017](../../../website/content/docs/decisions/0017-multi-cloud-dns-naming.md)
make that split clean — there is no name a client could resolve to either CA.
- Nothing about this reaches application manifests: workloads request a
cert-manager `Certificate`, which is already cloud-neutral. The issuer differs
per cloud, the developer-facing API does not — ADR-0007's split by audience.

**Slice 4 (autoscaling)** — *results recorded 2026-08-24, measured on gcp-mycluster-0.
Four PASS, one partial, one blocked on a GCP quota. Each is annotated below.*

Expand Down
Loading
Loading