diff --git a/docs/superpowers/plans/2026-08-23-gcp-foundation-resume.md b/docs/superpowers/plans/2026-08-23-gcp-foundation-resume.md index 6f7d27a34..1c80ed6ff 100644 --- a/docs/superpowers/plans/2026-08-23-gcp-foundation-resume.md +++ b/docs/superpowers/plans/2026-08-23-gcp-foundation-resume.md @@ -97,6 +97,24 @@ flux reconcile source git flux-system failure mode, `depends_on` gaps, and the `clusters/gcp-mycluster-0/` Flux wiring. Two review agents produced nothing across repeated asks. +### Private certificates need their own OpenBao on GCP + +**Decided 2026-08-24. Rationale and options live in the design under *Private +certificates on GCP*** — not repeated here, because this file is the one edited +on resume and the two would drift. + +What matters for sequencing: + +- **Workstream 11** (private certs, OpenBao on GCP: MIG + internal LB + Cloud KMS + auto-unseal) depends only on workstream 1, so it is **not blocked by slice 5** + and can start whenever. +- **Workstream 10** (external-dns + public certs) is blocked on an open question + the design records: GCP has only a *private* Cloud DNS zone, so DNS-01 has + nothing publicly resolvable to solve against. Settle that before starting it. +- The decision is **two OpenBaos, two roots** — so workstream 11 needs its own + GCP Secret Manager entries for the root token and cert-manager AppRole, and + tailnet clients end up trusting both roots. + ### GPU quota — blocks slice 4's last criterion `GPUS_ALL_REGIONS` on project `ogenki-435905` is **0**, so no GPU node can be diff --git a/docs/superpowers/specs/2026-08-18-gcp-support-design.md b/docs/superpowers/specs/2026-08-18-gcp-support-design.md index 8d0dfe4a4..f8bd66e3e 100644 --- a/docs/superpowers/specs/2026-08-18-gcp-support-design.md +++ b/docs/superpowers/specs/2026-08-18-gcp-support-design.md @@ -213,8 +213,8 @@ known rather than predicted. | 7 | Extract the Crossplane Configuration packages, OCI-released | 3 | **DONE 2026-08-19 (v0.1.0)** | | 8 | `objectStore` API migration + `App`/`SQLInstance` branching | 5, 7 | unblocked by 7 | | 9 | Object-storage call sites: Harbor (GCS driver), `openbao-snapshot` (GCS + Cloud KMS), CNPG barman (GCS) | 5, 8 | | -| 10 | DNS + PKI: `external-dns` google provider, cert-manager clouddns DNS-01 | 5 | | -| 11 | OpenBao on GCP: MIG + internal LB + Cloud KMS auto-unseal | 1 | | +| 10 | DNS + PKI: `external-dns` google provider, cert-manager clouddns DNS-01 (**public** certs) — see [Private certificates on GCP](#private-certificates-on-gcp) | 5 | | +| 11 | OpenBao on GCP: MIG + internal LB + Cloud KMS auto-unseal (**private** certs) — see [Private certificates on GCP](#private-certificates-on-gcp) | 1 | | | 12 | Gateway/LB: GCP public-LB annotations, drop `aws-load-balancer-controller` | 3 | | | 13 | Storage: `gp3` → `pd-balanced`/hyperdisk, EFS CSI → Filestore CSI | 3 | | | 14 | GPU + LLM platform: GPU `ComputeClass`, GCS Fuse weights, no `runtimeclass-nvidia` | 4, 9 | | @@ -299,14 +299,22 @@ kind: GCPWorkloadIdentity metadata: { name: external-dns, namespace: infrastructure } spec: serviceAccount: { name: external-dns, namespace: infrastructure } - roles: [roles/dns.admin] # predefined roles, named as GCP names them - customRole: # optional — creates + binds a custom role - permissions: [dns.resourceRecordSets.create] + roles: [projects/ogenki-435905/roles/xplane_dns_editor] ``` -Renders one **`ProjectIAMMember`** per role, member = the `principal://` KSA string, plus a -`ProjectIAMCustomRole` when `customRole.permissions` is set. No Google service account, no -annotation. `xplane-*` prefix owned by the composition. +Renders one **`ProjectIAMMember`** per role, member = the `principal://` KSA string. No Google +service account, no annotation. `xplane-*` prefix owned by the composition. + +> **`roles:` may only name roles the OpenTofu allowlist permits**, and as of 2026-08-24 that is the +> single pre-created `xplane_dns_editor` — *not* `roles/dns.admin`, which was the first draft and was +> dropped for carrying zone deletion and response policies (see `opentofu/gcp/gke/init/iam.tf` for +> the full reasoning). A claim naming anything else is refused by the IAM condition, not by the +> composition, so the error surfaces at the provider and names the role. +> +> **`customRole.permissions` is unusable on this platform** and should not appear in claims. The +> condition matches exact role names via `hasOnly`, and a role the composition names at render time +> cannot be allowlisted in advance. Adding a capability means adding a +> `google_project_iam_custom_role` in OpenTofu and referencing it — the same two-step every time. **Two hard constraints, both silent-failure classes:** @@ -351,6 +359,92 @@ Falsifiable, verified against a live cluster. 11. A written monthly run-rate estimate exists (cluster fee, static pool, Cloud NAT, Cloud DNS, Tailscale instance) stating the zonal-vs-regional choice and its price delta. +### Private certificates on GCP + +*Added 2026-08-24, while scoping slice 5. Workstreams 10 and 11 both touch +certificates and are easy to conflate; they are separable and only one needs +OpenBao.* + +**The split.** Workstream 11 gives **private** certificates from OpenBao's own +PKI, the GCP counterpart to what `bao.priv.aws.ogenki.io` serves today, for names +under `priv.gcp.ogenki.io`. Workstream 10 covers `external-dns` plus **public** +certificates, which are for `cloud.ogenki.io` — [ADR-0017](../../../website/content/docs/decisions/0017-multi-cloud-dns-naming.md) +keeps the public zone cloud-agnostic and `priv..ogenki.io` private. + +Only 11 requires running OpenBao, and it depends only on workstream 1 (network), +so it is not blocked by slice 5. + +> **Open question, surfaced 2026-08-24 while writing this section: workstream 10's +> DNS-01 has nothing to solve against on GCP.** `opentofu/gcp/network/dns.tf` +> creates a **private** Cloud DNS zone and nothing else, while Let's Encrypt must +> resolve the `_acme-challenge` TXT record **publicly**. `cloud.ogenki.io` is a +> Route53 zone that this repository does not even manage — it appears only as a +> `data` lookup. So "cert-manager clouddns DNS-01" is not yet a complete plan. +> +> Three ways out, none chosen: solve DNS-01 against **Route53** from the GCP +> cluster (works today, but reintroduces exactly the cross-cloud dependency +> option A was rejected for); **delegate** a public subdomain to a new public +> Cloud DNS zone (clean, needs a registrar change and a public zone this repo +> would then own); or serve **no public certificates from GCP at all** and keep +> public ingress on AWS — which is coherent while GCP has no public endpoints. +> +> Worth settling before workstream 10 starts, not during it. Nothing about it +> affects workstream 11, which is self-contained. + +**Why GCP needs its own OpenBao at all.** See the 2026-08-23 correction above: +OpenBao *the product* is cloud-agnostic, our OpenBao *stacks* are not. + +#### Options considered + +**A — Shared OpenBao.** GCP's cert-manager reaches `bao.priv.aws.ogenki.io` over +the tailnet. No new infrastructure, one CA, one trust anchor. Rejected: it makes +GCP certificate issuance hard-depend on AWS *and* on the tailnet, in a platform +whose stated point is that each cloud stands alone. It is the same coupling this +design already flags as undesirable for the Flux GitHub App secret. + +**B — Two OpenBaos, two roots. CHOSEN.** Each cloud runs its own OpenBao with its +own root CA. Fully independent; matches [ADR-0007](../../../website/content/docs/decisions/0007-cloud-abstraction-boundaries.md)'s +rule that platform-facing infrastructure stays cloud-shaped. Costs one extra +trust anchor for tailnet clients. + +**C — Two OpenBaos, one shared root.** Each cloud holds its own intermediate, +both signed by a common root: one trust anchor, independent operation. Rejected +on operational grounds — it requires either copying the root PRIVATE KEY into GCP +Secret Manager, doubling exposure of the most sensitive material the platform +holds, or cross-signing GCP's intermediate at bootstrap, which is a manual +ceremony **on every rebuild** of a platform whose lifecycle is +build-validate-destroy. + +#### Why B, concretely + +The 2026-08-24 rebuild made the argument better than theory could. The private +domain rename forced a new server certificate, and re-issuing it under the +existing chain proved impossible: the intermediate that had signed it had **no +private key stored anywhere**, because OpenBao issued it and the key never left +OpenBao. A fresh CA was the only way forward. + +That is exactly the failure mode option C institutionalises. Cross-cloud PKI +ceremony is the first thing to break on a platform rebuilt this often, and it +breaks at rebuild time, which is the worst moment. Two independent roots cost one +extra trust anchor and remove the entire class. + +#### Consequences to carry into workstream 11 + +- Tailnet clients must trust **both** roots. That is the accepted cost; it is a + one-line addition wherever the AWS root is already distributed. +- GCP needs its own secret store for the root token and the cert-manager AppRole + — **GCP Secret Manager**, mirroring what AWS Secrets Manager does today. The + `flux-github-app` secret already establishes that pattern on GCP. +- **Cloud KMS auto-unseal** is the GCP analogue of the AWS KMS unseal, already + named in the workstream row. It is what makes an unattended rebuild possible. +- The GCP PKI issues for `*.priv.gcp.ogenki.io` only; the AWS one keeps + `*.priv.aws.ogenki.io`. The per-cloud private domains from + [ADR-0017](../../../website/content/docs/decisions/0017-multi-cloud-dns-naming.md) + make that split clean — there is no name a client could resolve to either CA. +- Nothing about this reaches application manifests: workloads request a + cert-manager `Certificate`, which is already cloud-neutral. The issuer differs + per cloud, the developer-facing API does not — ADR-0007's split by audience. + **Slice 4 (autoscaling)** — *results recorded 2026-08-24, measured on gcp-mycluster-0. Four PASS, one partial, one blocked on a GCP quota. Each is annotated below.* diff --git a/opentofu/gcp/gke/init/iam.tf b/opentofu/gcp/gke/init/iam.tf index a29a6edef..82b43bdcf 100644 --- a/opentofu/gcp/gke/init/iam.tf +++ b/opentofu/gcp/gke/init/iam.tf @@ -1,49 +1,252 @@ -# NO project IAM bindings here, deliberately. -# -# This file used to grant roles/editor project-wide to the Workload Identity -# principal -# -# principal://iam.googleapis.com/projects//locations/global -# /workloadIdentityPools/.svc.id.goog -# /subject/ns/crossplane-system/sa/crossplane -# -# as "bootstrap-only breadth", to be narrowed in slice 5 (GCPWorkloadIdentity). -# It was removed instead, for a reason that only became clear once the GCP -# cluster was actually built: CROSSPLANE IS NOT DEPLOYED ON GCP. The cluster's -# Flux tree is crds, flux and namespaces -- no Crossplane, no providers, no -# compositions. Nothing created that ServiceAccount and nothing used the grant. -# -# So it was not "a grant that is wider than what Crossplane needs". It was a -# project-wide editor grant with NO consumer, and it was reachable: namespaces/ -# base is shared between clouds and creates crossplane-system on GCP too, so -# anyone able to create a ServiceAccount named `crossplane` in that existing -# namespace inherited editor on the whole project. The binding is also scoped to -# the project-wide identity pool rather than to this cluster, so every future -# GKE cluster in the project -- including a throwaway one -- would have inherited -# it as well. -# -# Narrowing was rejected as the fix because any role set chosen now would be a -# guess about compositions that do not exist yet. Note also that GCP IAM -# conditions would NOT reproduce the AWS `xplane-*` scoping: resource-level -# conditions are unsupported for most services Crossplane would touch, so the -# AWS parity is not directly available here. -# -# SLICE 5 MUST create its own binding alongside the GCP compositions that define -# what is actually needed, and must NOT restore this one. Two traps that cost -# real time when it existed, worth keeping when it comes back: -# -# 1. `projects/` takes the project NUMBER, `workloadIdentityPools/` takes the -# project ID. Reversed, the API accepts the binding and it simply never -# matches -- a permission error that points nowhere. Derive the number from -# data.google_project.this.number, never hand-copy it. -# 2. The principal string is built from variables, so OpenTofu sees no -# reference to module.gke and schedules the binding in PARALLEL with the -# cluster -- but the pool `.svc.id.goog` does not exist until a -# cluster with workload_pool has been created. Without an explicit -# depends_on = [module.gke] a FRESH apply fails with -# Error 400: Identity Pool does not exist (ogenki-435905.svc.id.goog) -# and does not reproduce on re-apply, because by then the pool exists. -# Measured on the first real deploy, 2026-08-23. -# -# data.google_project.this stays in data.tf: output "project_number" feeds the -# Flux postBuild substitution ConfigMap and is unrelated to any of the above. +# Crossplane's GCP identity, for slice 5 (GCPWorkloadIdentity). +# +# This file previously granted roles/editor project-wide and was REMOVED in +# #1818, because Crossplane was not deployed on GCP at all and the grant had no +# consumer. It comes back for slice 5 — deliberately scoped this time, with +# everything that was measured rather than assumed recorded below. +# +# HONEST CAVEAT: the consumer still does not exist. Crossplane is not yet +# deployed on GCP, so this binding again lands ahead of the workload it is for, +# which is the situation #1818 removed the old one over. Two things make that +# acceptable where roles/editor was not: +# +# - The blast radius is one narrow role. Anyone able to create a ServiceAccount +# named `provider-gcp` in crossplane-system could assume this identity, but +# all it can then do is grant DNS record-set management — not grant itself +# owner, which is what editor allowed. +# - It is a prerequisite, not a leftover: the GCP provider cannot authenticate +# without it, so it has to precede the deployment rather than follow it. +# +# If slice 5 stalls, REMOVE THIS AGAIN rather than letting it sit. The reasoning +# that justified deleting the last one applies to a narrow grant too, just more +# slowly. +# +# ── WHAT IT REPLACES ──────────────────────────────────────────────────────── +# +# roles/editor is thousands of permissions across every service. What +# GCPWorkloadIdentity renders is ProjectIAMMember, one per requested role, so the +# capability actually needed is setIamPolicy on the project: +# +# roles/resourcemanager.projectIamAdmin +# +# ── WHY THAT ALONE IS NOT ENOUGH ──────────────────────────────────────────── +# +# projectIamAdmin is a PRIVILEGE-ESCALATION PATH: setIamPolicy can grant any role +# to any principal, including granting Crossplane itself roles/owner. Narrowing +# editor to it would be a large improvement and still leave that open. +# +# The mitigation is an IAM Condition on `modifiedGrantsByRole`, limiting WHICH +# roles this binding may modify. That is GCP's analogue of the AWS side's +# `xplane-*` scoping (platform constitution): AWS restricts Crossplane by +# resource NAME; GCP cannot for project IAM, because the resource IS the project, +# so it restricts by role instead. +# +# The attribute covers GRANTS AND REVOCATIONS both — Google documents it as "role +# names from the role bindings that the request modifies", and the reference +# table is headed "Granted/revoked roles". So the condition also stops Crossplane +# REMOVING bindings it did not create, including break-glass human access. +# Verified against the attribute reference 2026-08-24. +# +# Adding a role to the allowlist is therefore a deliberate act, which is the +# point. A workload needing something outside it fails with a permission error +# naming the role, rather than Crossplane quietly having had it all along. + +# The DNS role Crossplane is permitted to grant. +# +# Pre-created HERE rather than by the composition, and that is what makes it +# grantable at all: the condition below uses `hasOnly`, which matches exact role +# names, so a role whose name the composition invents at render time could never +# be allowlisted. A role created in OpenTofu has a DETERMINISTIC name, so it can. +# +# NOT roles/dns.admin, which was the first draft. That predefined role carries +# `dns.managedZones.delete` and the whole `dns.responsePolicies.*` / +# `dns.policies.*` family, both beyond anything slice 5 needs: +# +# - Zone deletion contradicts the platform constitution outright — "no deletion +# permissions for stateful services (S3, IAM, Route53)". The AWS side honours +# that; granting dns.admin here would not have. +# - Response policies are the sharper one. A compromised provider-gcp could +# bind a response policy to the cluster's VPC overriding +# `metadata.google.internal` or `*.googleapis.com`, redirecting in-cluster +# traffic and harvesting credentials — invisible to external-dns, which only +# ever looks at record sets. +# +# `role_id` may not contain dashes, so the platform's `xplane-` convention is +# spelled `xplane_` here. +resource "google_project_iam_custom_role" "crossplane_dns" { + project = var.project_id + role_id = "xplane_dns_editor" + title = "Crossplane DNS editor" + description = "Record-set management for external-dns and cert-manager DNS-01. Deliberately excludes zone deletion and response policies; see opentofu/gcp/gke/init/iam.tf." + + permissions = [ + # Record sets: the actual job. Delete IS included — external-dns removes + # records when a route goes away, and cert-manager cleans up its + # _acme-challenge TXT records after validation. + "dns.resourceRecordSets.create", + "dns.resourceRecordSets.delete", + "dns.resourceRecordSets.get", + "dns.resourceRecordSets.list", + "dns.resourceRecordSets.update", + + # Changes are how Cloud DNS applies record-set edits transactionally. + "dns.changes.create", + "dns.changes.get", + "dns.changes.list", + + # Read-only on zones. external-dns must discover which zone owns a name; it + # must never create or destroy one. + "dns.managedZones.get", + "dns.managedZones.list", + ] +} + +locals { + # Roles Crossplane may grant. Keep tight; grow on evidence. + crossplane_grantable_roles = [ + google_project_iam_custom_role.crossplane_dns.name, + ] + + # TRAP 1, and it is silent: `projects/` takes the project NUMBER while + # `workloadIdentityPools/` takes the project ID. Reversed, the API ACCEPTS the + # binding and it simply never matches — a permission error that points nowhere. + # Derived from data.google_project rather than hand-copied for that reason. + crossplane_principal = join("", [ + "principal://iam.googleapis.com/projects/${data.google_project.this.number}", + "/locations/global/workloadIdentityPools/${var.project_id}.svc.id.goog", + # sa/provider-gcp, NOT sa/crossplane. The PROVIDER pod makes the cloud API + # calls; Crossplane core never talks to GCP -- the AWS side binds + # crossplane-system/provider-aws for the same reason + # (opentofu/aws/eks/init/iam.tf:56-57). + # + # OBLIGATION ON SLICE 5, not a description of something that exists: when the + # GCP provider tree is written, its DeploymentRuntimeConfig MUST name this + # ServiceAccount `provider-gcp`. Nothing checks that the two agree, and a + # mismatch fails in the same silent way as TRAP 1. + "/subject/ns/crossplane-system/sa/provider-gcp", + ]) + + # Grants are limited to the allowlist above, and nothing else. + # + # `hasOnly` is the ONLY usable form here. GCP IAM conditions run a restricted + # CEL dialect: the `.all()` macro is rejected at apply time with + # undeclared reference to '@not_strictly_false' + # so an expression mixing exact matches with a `startsWith` prefix cannot be + # written. Measured 2026-08-24, not read in docs. + crossplane_grant_condition = join("", [ + "api.getAttribute('iam.googleapis.com/modifiedGrantsByRole', []).hasOnly([", + join(",", [for r in local.crossplane_grantable_roles : "'${r}'"]), + "])", + ]) +} + +# Additive binding, and it must stay that way. +# +# NEVER use google_project_iam_policy or google_project_iam_binding here: both +# are AUTHORITATIVE for the roles they manage and would delete every binding +# they do not know about — other workloads, and break-glass human access. The +# failure is silent until something unrelated loses permission. +resource "google_project_iam_member" "crossplane_iam_admin" { + project = var.project_id + role = "roles/resourcemanager.projectIamAdmin" + member = local.crossplane_principal + + condition { + title = "xplane-scoped-grants-only" + description = "Crossplane may modify only the allowlisted role bindings. Without this, projectIamAdmin can grant itself roles/owner." + expression = local.crossplane_grant_condition + } + + # TRAP 2, fresh-apply only. The principal string is built from variables, so + # OpenTofu sees no reference to module.gke and schedules this in PARALLEL with + # the cluster — but the pool `.svc.id.goog` does not exist until a + # cluster with workload_pool has been created. Without this a FRESH apply fails + # with `Error 400: Identity Pool does not exist`, and it does NOT reproduce on + # re-apply, because by then the pool exists. Measured 2026-08-23. + depends_on = [module.gke] +} + +# Read-only access to role definitions, so the grant above can actually be made. +# +# `gcloud iam roles describe roles/resourcemanager.projectIamAdmin` returns 9 +# permissions and `iam.roles.get` is NOT among them (checked 2026-08-24). +# Referencing a CUSTOM role in a setIamPolicy call can require the caller to read +# that role's definition — a constraint that does not exist for predefined roles, +# and therefore one the previous roles/dns.admin draft would never have hit. +# +# UNVERIFIED whether GCP enforces it on this path: confirming needs a live +# provider-gcp pod attempting the grant, and GCP is torn down. Granted +# pre-emptively because the downside is asymmetric — without it slice 5 fails at +# runtime with an error naming the wrong thing, and with it the identity gains +# only the ability to READ role definitions, which are not secret and confer +# nothing. Unconditioned deliberately: `modifiedGrantsByRole` is undefined for +# read requests, so a condition here would be vacuous anyway (see gap 1 below). +resource "google_project_iam_custom_role" "crossplane_role_reader" { + project = var.project_id + role_id = "xplane_role_reader" + title = "Crossplane role reader" + description = "Read-only on IAM role definitions, so ProjectIAMMember can reference the custom DNS role. Confers no grant capability." + + permissions = [ + "iam.roles.get", + "iam.roles.list", + ] +} + +resource "google_project_iam_member" "crossplane_role_reader" { + project = var.project_id + role = google_project_iam_custom_role.crossplane_role_reader.name + member = local.crossplane_principal + + # Same fresh-apply race as the binding above — see TRAP 2. + depends_on = [module.gke] +} + +# ── WHAT THIS BINDING STILL DOES NOT CLOSE ────────────────────────────────── +# +# Written down because both are inherent to the condition mechanism, neither has +# a fix available in the restricted CEL dialect, and an undocumented gap becomes +# an assumed-safe one. Reviewed 2026-08-24. +# +# 1. The condition gates 1 of the role's 9 permissions. +# `modifiedGrantsByRole` is populated only for setIamPolicy-shaped requests. +# Google, verbatim: "For other types of requests, the attribute is not +# defined." Undefined falls back to the mandated `[]` default, and +# `[].hasOnly(...)` is vacuously TRUE — so projectIamAdmin's other verbs +# (resourcemanager.projects.{create,update,delete,search}PolicyBinding and +# iam.policybindings.{get,list}) are granted UNCONDITIONED. +# Concretely: deletePolicyBinding could remove a Principal Access Boundary +# binding constraining this very workload. Latent today — the project uses no +# PAB policies, and creating one is separately blocked by the missing +# iam.principalaccessboundarypolicies.bind — but it goes live silently the day +# the org adopts PAB. Re-examine this binding then. +# +# 2. The condition constrains the ROLE, never the MEMBER. +# Nothing stops Crossplane granting the allowlisted role to a principal other +# than itself — e.g. any pod identity in the cluster. Two things bound it: +# the org policy `constraints/iam.allowedPolicyMemberDomains` is enforced +# (allowedValues: C01fvyerd), so allUsers/allAuthenticatedUsers and external +# Google accounts are already refused at the perimeter; and the allowlisted +# role is narrow by construction, so the worst in-org outcome is DNS +# record-set management, not project control. Shrinking the role shrinks this. +# +# 3. The binding is scoped to the PROJECT-WIDE workload identity pool +# `.svc.id.goog`, not to this cluster — GKE workload-identity +# subjects carry no cluster dimension, so there is no way to write it +# otherwise. Every future GKE cluster in this project, including a throwaway +# one, therefore inherits it: a `crossplane-system/provider-gcp` ServiceAccount +# in ANY cluster here matches. Unavoidable, so it is recorded rather than +# fixed; it is a further reason to keep the allowlist narrow. + +# NO roles/iam.roleAdmin BINDING, deliberately. +# +# GCPWorkloadIdentity's optional `customRole.permissions` renders a role whose +# name the composition chooses at render time, and `hasOnly` cannot allowlist a +# name that does not exist yet. Granting roleAdmin would mean either dropping the +# condition — restoring the escalation path this file exists to close — or +# accepting arbitrary role creation. +# +# The pre-created role above is the supported alternative and covers slice 5's +# actual need. Extend the SAME pattern for anything further: add a +# google_project_iam_custom_role here, reference it in the allowlist. Do NOT +# widen the condition.