feat(gcp): OpenBao on GCP — offline-root PKI, code-only half of workstream 11 - #1827
Merged
Conversation
Workstream 11. Scope is PKI only — a root of trust, an issuing CA, a cert-manager role and one AppRole. The app tenant namespace, kv-v2 mount, snapshot AppRole and operator userpass are deliberately not ported: none has a consumer on GCP. The substantive change is to the certificate chain, and it supersedes the decision recorded in the 2026-08-18 design. That design chose two independent roots, one per cloud, and rejected a shared root because cross-signing would be "a manual ceremony on every rebuild". The premise was false. The intermediate lives in Secret Manager independently of OpenBao's lifecycle — which is exactly why AWS's survives rebuilds today — so signing happens once per cloud, ever. The better option was rejected over a cost it does not have. The first draft of this design also carried over the AWS shape where config_ca imports a pem_bundle containing the ROOT key into the live pki mount. The repository's own pki-and-secrets.md warns against it verbatim: "do not carry it into a deployment where the root CA matters." Compromising OpenBao would yield the root, which inverts the property an offline root exists to provide. So: one OFFLINE root, per-cloud intermediates, and the intermediate imported directly as the issuer. The root key is never in Secret Manager, never in the mount, never in OpenTofu state. This also removes four resources — the vault_pki_secret_backend_key -> intermediate_cert_request -> root_sign_intermediate -> intermediate_set_signed sequence exists only to generate an OpenBao-internal intermediate under an imported root. GCP adopts it first, AWS follows later. The interim cost is recorded honestly: until AWS migrates, clients trust two anchors. That was permanent under the superseded decision and is now a state with a defined end. Two findings the design surfaced that were not in the workstream row: - scripts/openbao-config.sh is AWS-only and needs a --cloud gcp flag. Its coupling is three seams (secret read, secret write, CLI prefix), which is what makes a flag right rather than a sibling script. - GCP Secret Manager IDs permit only letters, digits, hyphen and underscore. AWS's names are paths (certificates/priv.aws.ogenki.io/root-ca), so every name must be rewritten and the two clouds cannot share one convention. It also needs an addition to slice 5's IAM allowlist: External Secrets reading Secret Manager requires a GCPWorkloadIdentity claim, and the condition in gke/init/iam.tf allowlists only xplane_dns_editor. A second pre-created custom role is required. That is the mechanism working as intended. Risks are recorded rather than resolved, including the one assumption the whole chain rests on: that config_ca alone, without the CSR/sign/set-signed sequence, leaves the mount able to issue. Standard import-an-existing-CA flow, but never exercised in this repository — verify it early. Evidence: validate-links.sh exit 0.
Eight tasks for docs/superpowers/specs/2026-08-24-gcp-openbao-design.md. The verification model is adapted rather than borrowed: this is infrastructure, so "write the failing test first" becomes "run the verification command and watch it fail for the right reason". The gates are the repo's real ones — tofu validate, trivy config, tofu plan, shellcheck, validate-manifests.sh. Forcing a unit-test framework onto Terraform would have produced ceremony, not evidence. Task ordering is deliberate: - Task 1 (the Secret Manager IAM role) is first because it is independent of everything else and safe to merge alone. - Task 3 (the offline PKI ceremony) comes before any stack, because cluster/ reads the server certificate at boot. It produces secrets and a runbook, not code. - Task 7 Step 1 verifies the design's one untested assumption BEFORE any HCL is written: that config_ca alone, without the CSR/sign/set-signed sequence, leaves the mount able to issue. If it fails, the four removed resources may be required and the design needs revisiting. Discovering that after writing the stack would be the expensive order. Two things the plan carries from hard experience in this repo rather than from the spec: read Terramate's output rather than its exit code, which has reported 0 over a failed run repeatedly; and the KMS key ring survives teardown by design via prevent_destroy, so it must not be mistaken for a leak. Self-review found two inconsistencies, both fixed inline: the startup script's templated-variable list omitted region, kms_key_ring and kms_crypto_key, which templatefile would have failed on at plan time; and dns.tf referenced google_compute_address.openbao while the load-balancer step never named that resource. The .secrets.baseline entry is a false positive: detect-secrets' Secret Keyword heuristic fires on a PROSE line naming manifest fields (caBundleSecretRef). A pragma would have been visible text in the rendered document, so the baseline is the right mechanism. The diff adds exactly one entry and suppresses nothing else. Evidence: validate-links.sh exit 0; no placeholders; the six Secret Manager names are consistent across all eight tasks.
Task 1's fallback check said grep -c 'google_project_iam_custom_role\.' should return at least 4, "two definitions, two references". Both halves were wrong: grep -c counts matching LINES, and the trailing dot matches only REFERENCES — resource declarations use a quote and never match. The real count is 3. Caught by Task 1's reviewer, which noticed the implementer's report cited a number that did not reproduce. The wiring was correct and independently verified; only the check was bad. Replaced with a grep for the specific reference, which names what it looks for instead of asserting a bar that must be recomputed whenever a role is added.
C1: GCPWorkloadIdentity renders project-scoped ProjectIAMMember, so the previous permission set (versions.access + versions.list + secrets.get + secrets.list) let any holder enumerate every Secret Manager secret in the project. Drop the three list/get permissions -- ESO's gcpsm provider only needs versions.access to read a NAMED secret, and Task 8's ExternalSecrets are written against two named secrets, not dataFrom.find. The role stays project-wide by name even after this change (ProjectIAMMember has no per-secret scope). Record that honestly in the resource comment and point Task 8 at the stronger fix -- google_secret_manager_secret_iam_member per secret, as opentofu/gcp/openbao/cluster/iam.tf already does for the server certificate. Not applied here: Task 8 is unwritten, and the decision belongs with its own OpenTofu. Same guidance recorded in the design (A dependency on slice 5) and the plan (Task 8) so neither document is silent about the residual blast radius.
I1: variables.tf declared openbao_version with no default and variables.tfvars didn't set it, so tofu apply would prompt on a TTY or fail outright in CI with "No value for required variable". Default to 2.6.2, matching the AWS pin, and carry the Renovate marker plus the openbao/openbao#3411 deadlock note. Also fix variables.tfvars's comment, which asserted every non-project_id variable was left on its default while one required variable had none.
I3: write_ca() unconditionally ran secret_read | jq -r '.ca // empty', which assumes the AWS JSON shape. GCP's openbao-priv-gcp-ca-chain secret is raw PEM by design (gcloud secrets create --data-file=ca-chain.pem, plan Task 3), so bao ca --cloud gcp would either read nothing back or, under pipefail, die on jq's own parse error before the script's friendly message ever printed. Branch on $CLOUD: use the GCP payload as-is, keep the AWS `.ca` unwrap, and guard the jq call explicitly so a parse failure reports through this script's error path instead of jq's raw stderr. Also documents the two secrets' shapes in usage() with a GCP example.
… install I5: setup-local-disks.sh and startup-script.sh are joined into ONE metadata_startup_script and share a single process. The missing-disk branch ran exit 0, which under set -o errexit terminates the WHOLE combined script -- not just the disk-setup half -- so a missing data disk means OpenBao is never installed, configured or started, while the instance still boots, joins the MIG and reports RUNNING. Change that branch to exit 1: unlike AWS's optional ephemeral NVMe, this stack's data disk is declared by compute.tf and always attached, so its absence is a provisioning bug, not a state to boot past. Separately, setup-local-disks.sh requires mkfs.xfs but xfsprogs was only installed later, in startup-script.sh's apt-get -- the half that runs AFTER disk setup. Move the xfsprogs install into setup-local-disks.sh, ahead of its check_command calls, and drop it from startup-script.sh's package list.
…nges I4: the MIG had no update_policy, so GCP's default OPPORTUNISTIC applies -- a new instance_template is adopted but the running instance is never replaced. Bumping openbao_version or editing either boot script would make tofu apply report success while the node keeps running the old template indefinitely, with no drift signal on the next plan. Add PROACTIVE/REPLACE with max_unavailable_fixed=1, max_surge_fixed=0 -- this is a single-node MIG, so there's no second instance to surge onto. Same failure mode AWS hit on its launch template's default version (autoscaling_group.tf, update_default_version = true).
Minor cleanup, no behavior change: - stack.tm.hcl: description said the stack creates the KMS key and predates compute; both are false now (kms.tf reads it via data source, compute.tf shipped in the previous commit). - versions.tf: 'every resource below' listed the KMS key ring/key as managed here; it's a data source. - iam.tf: replaced the false 'this stack only runs tofu validate, never apply' with the real constraint -- the server-certificate secret (Task 3) must exist before this stack applies, an ordering enforced by the boot script's runtime fetch, not by OpenTofu. - data.tf: 'Kept for Tasks 5/6' -- Task 5 (compute) shipped without using data.google_project. - kms.tf: the stated reason for keeping KMS out of this stack's state was wrong. tofu destroy does NOT fail on a key ring/crypto key -- the provider's delete is a no-op that removes the object from state and returns success. The real reasons are (a) a rebuild's create then hits ALREADY_EXISTS against a key ring that still exists server-side, and (b) destroying google_kms_crypto_key schedules its VERSIONS for destruction while still exiting 0 -- a teardown that silently schedules the unseal key's destruction. Rewritten so a future re-test of 'does destroy fail' doesn't conclude the bootstrap step is cargo cult. - variables.tf: data_disk_size_gb/openbao_data_path described 'raft storage'/'raft data'; this deployment runs storage "file".
Contributor
🔍 Rendered manifest diff — this PR vs
|
…ents
Simplify pass over the branch. Three findings applied, one declined.
Removed three declarations that nothing references: data.google_project.this,
local.private_dns_zone and local.private_domain_name. Each carried a
tflint-ignore pragma and a paragraph justifying itself as "kept for Task 6" —
roughly fifteen lines of comment defending dead code for a task that is not
written yet. Task 6 can add them back when something consumes them. The
stale-comment commit earlier in this branch had already noticed data.tf's
justification was false ("Task 5 shipped without needing it") and left the
declaration in place anyway, which is how scaffolding becomes permanent.
Deduplicated the S3-bucket-region explanation, which appeared near-verbatim in
backend.tf and data.tf with the same example region. data.tf now points at
backend.tf, matching how it already defers there for the S3-not-GCS rationale.
Dropped one of two identical "(single-node, storage file -- not raft)"
parentheticals in variables.tf; compute.tf's header already establishes it, and
the qualifier is only genuinely useful on the disk-size variable, where asking
how big invites the raft question.
DECLINED: factoring secret_write()'s per-cloud control flow into a shared
skeleton. The exists-check/create-or-update/error-handle shape is written once
per cloud and only four lines differ, so roughly fifteen lines are recoverable —
but it trades directness for indirection in a bash script that is read rarely
and usually while something is broken. The reviewer weighed it moderate-value
and left the call to me; directness wins here.
Also recorded in the design's Risks section that xplane_secret_reader's grant is
project-wide until Task 8 narrows it, and that it must be narrowed or removed
rather than inherited if Task 8 slips. That debt was previously visible only in
a code comment, unlike the PKI two-anchor interim state which the Risks section
already flags.
Evidence: tofu validate Success, fmt clean, tflint exit 0 (confirming no removed
pragma masked a real unused declaration), trivy 0 findings, validate-links.sh
exit 0.
…trap Running the documented bootstrap on a fresh project failed: ERROR: PERMISSION_DENIED: Google Cloud KMS API has not been used in project 323586397743 before or it is disabled. gcloud offers to enable it interactively, but the prompt defaults to no, so a non-interactive run just fails — and the message reads like a permissions problem rather than a missing API. Neither the design nor the plan mentioned it. Added to kms.tf's bootstrap comment, which is where someone hitting this will look.
The design said the instance service account needs roles/cloudkms.cryptoKeyEncrypterDecrypter "and nothing else". That is wrong, and no review caught it — only a live boot did: Error configuring seal "gcpckms": error checking key existence: PermissionDenied: Permission 'cloudkms.cryptoKeys.get' denied on resource .../cryptoKeys/openbao-unseal (or it may not exist). OpenBao's gcpckms seal verifies the key EXISTS before using it, and cryptoKeyEncrypterDecrypter grants encrypt/decrypt without cryptoKeys.get. Added roles/cloudkms.viewer, the least-privileged predefined role carrying that permission, bound at the crypto-key level so it sees only this one key. The failure mode is the quiet one this stack was written to avoid: the seal is configured after the process starts, so the instance reaches RUNNING and joins the MIG while openbao.service crashloops. Nothing short of the serial console says why.
Task 6 of the plan. Makes the OpenBao node reachable at bao.priv.gcp.ogenki.io:8200 from the tailnet. Passthrough LB, not a proxy: OpenBao terminates TLS with a certificate from the offline root, and a proxying LB would need that private CA in its trust store or would terminate TLS itself. Passthrough forwards the stream untouched so the client verifies OpenBao's own certificate end to end. The health check is TCP rather than HTTPS for the same reason — Google's probers cannot be given a private CA. TCP also deliberately does not answer 'is OpenBao sealed', because a sealed node still has to be reachable for an operator to unseal it. The DNS record is load-bearing, not cosmetic: the server certificate carries DNS:bao.priv.gcp.ogenki.io and no IP SAN, so connecting to the load balancer by address cannot verify TLS. local.fqdn derives both the record and the endpoint output so they cannot drift. Firewall is scoped by target service account and source range rather than opened to the VPC — the GKE nodes share this network and have no business reaching OpenBao's API directly. The health-check ranges are separate: without them the backend marks every instance UNHEALTHY and the forwarding rule blackholes traffic, with the only symptom buried in backend health status. IAP SSH stays off by default. Worth noting the crashlooping openbao.service debugged during this deploy was diagnosed entirely from the serial console, which needs no firewall rule at all.
The provider defaults a backend to UTILIZATION, which an INTERNAL backend service rejects outright: Error 400: Invalid value for field 'resource.backends[0].balancingMode': 'UTILIZATION'. Balancing mode must be CONNECTION for an INTERNAL backend service. Passthrough load balancers distribute connections rather than requests, so there is no utilization signal to balance on. Recorded the error verbatim in the file, because the fix is not guessable from the field name.
Deployed the OpenBao stack against real GCP, verified the PKI chain end to end, and tore it down to zero billable resources. Three design criteria proven: auto-unseal without operator input (criterion 2), a chain verifying to the offline root with a DNS-only SAN (criterion 3), and a teardown that removes everything billable (criterion 8). Plus the design's one untested premise — that config_ca alone leaves the mount able to issue, which justifies deleting four resources the AWS stack uses. It holds: the import produced an issuer and a key, a role issued a leaf, and the leaf verified to the offline root. Nine findings. Four fixed on this branch, five recorded. The one that matters most: the design said the instance service account needs cloudkms.cryptoKeyEncrypterDecrypter 'and nothing else'. False. The gcpckms seal checks the key EXISTS before using it, which needs cryptoKeys.get. Three reviews read that sentence and none could have caught it — the seal is configured after the process starts, so the instance reaches RUNNING and reports healthy while openbao.service crashloops. The worst still-open one is pre-existing: the network stack's Tailscale split-DNS is EMPTY after a first apply, because depends_on on a data source does not wait for the inbound DNS policy to finish allocating its address. The deploy reports success and no tailnet client can resolve the private zone at all. A second apply fixes it. Verified the VPC resolver was never at fault. Also open: systemd gives up permanently after three rapid failures, and the MIG still has no auto_healing_policies, so a transient boot problem leaves a dead service on a RUNNING instance forever. Records what remains unverified rather than implying completeness: no cert-manager Certificate was issued (needs Tasks 7-8 and a cluster), no 0-change-plan check, and the config_ca probe used a throwaway mount rather than the one the management stack will create.
Finding 7 from the live deploy. The network stack builds the Tailscale split nameserver from a data source filtered on purpose = DNS_RESOLVER, with depends_on the inbound DNS policy. That is not sufficient: creating the policy RETURNS before GCP finishes allocating the resolver address, and depends_on defers the data-source read until after the policy RESOURCE exists, not after the ADDRESS does. So a first apply matched nothing and wrote nameservers = []. Nothing failed. The deploy reported success while every tailnet client silently could not resolve *.priv.gcp.ogenki.io — the VPC resolver was fine, the private zone was fine, there was simply no nameserver configured. Measured 2026-08-25: first apply produced [], re-apply produced ["10.10.0.2"]. Confirmed the VPC side was never at fault by querying the resolver directly throughout. Two changes, and the second matters more than the first: - A 60s time_sleep between the policy and the data read, using the same hashicorp/time provider the AWS llm-platform stack already uses for EFS mount-target propagation. - A precondition asserting the address list is non-empty. A fixed wait alone would still degrade to the original bug on a slow allocation — a successful apply configuring no resolver. The precondition converts that silent failure into a loud one naming the consequence, which is recoverable by re-running. The bug was pre-existing and not introduced by the OpenBao work, but it blocks reaching OpenBao, so it belongs to this workstream.
Finding 7 moves from OPEN to FIXED (f9222ed), and the header count with it. Notes explicitly that the fix is not verified against a live first apply — proving it needs another full network build and teardown. What the precondition does buy is that the untested path now fails loudly rather than silently, which was the actual defect.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Workstream 11 of the GCP support design: OpenBao on GCP, giving the GKE cluster a private certificate authority.
This is the code-only half. Four of the plan's eight tasks are implemented. Tasks 3, 6, 7 and 8 need a live GCP deployment — the PKI ceremony, the actual
terramate script run deploy, the PKI configuration and the cert-manager wiring — and are deliberately not in this PR. What is here stands on its own: two OpenTofu stacks' worth of code, a script that now serves both clouds, and an IAM role, none of which has been applied to a cloud.Design:
2026-08-24-gcp-openbao-design.md· Plan:2026-08-24-gcp-openbao.mdThe design decision that changed during review
The 2026-08-18 design chose two independent roots, one per cloud, and rejected a shared root because cross-signing would be "a manual ceremony on every rebuild".
That premise was wrong, and this PR retracts it. The intermediate lives in Secret Manager independently of OpenBao's lifecycle — which is exactly why AWS's survives rebuilds today — so signing happens once per cloud, ever. The better option was rejected over a cost it does not have.
Worse, the first draft carried over the AWS shape where
config_caimports a pem_bundle containing the root key into the live PKI mount. The repo's ownpki-and-secrets.mdwarns against precisely this: "do not carry it into a deployment where the root CA matters." Compromising OpenBao would have yielded the root.Now: one offline root, per-cloud intermediates, intermediate imported directly as the issuer. The root key never reaches Secret Manager, the mount, or OpenTofu state. It also removes four resources — the
key → intermediate_cert_request → root_sign_intermediate → set_signedsequence exists only to generate an internal intermediate under an imported root.GCP adopts it first; AWS follows later. The interim cost is recorded honestly in the design: until AWS migrates, clients trust two anchors.
What's in the four tasks
xplane_secret_readercustom role, allowlisted so aGCPWorkloadIdentityclaim can grant External Secrets read access to Secret Manager--cloud gcponscripts/openbao-config.sh— three AWS-coupled seams replaced by dispatching helpersopentofu/gcp/openbao/cluster/scaffolding — backend, providers, variables, remote state, KMS data sources, service account, scoped IAMCloud KMS is a bootstrap prerequisite, not a managed resource
The plan originally had this stack create the key ring and crypto key with
prevent_destroy = true. An implementer caught that it was wrong; review later corrected why, which is worth reading below under the KMS rationale — the obvious explanation is not the true one.They are now data sources, with the creation commands documented in
kms.tfthe same way the S3 state bucket is:The key surviving teardown is now intentional rather than an error being swallowed, and the destroy script is genuinely strict — this stack owns billable compute, so a failed destroy must stop a
--reverseteardown rather than let it delete the VPC out from under surviving instances.Notable details carried deliberately
gcpckmstakes the key ring and key by NAME, not id. The wrong one fails at unseal, not boot.templatefile; TLS material is fetched at runtime by the instance's service account.disable_mlockis deliberately dropped. Review traced it to upstream: mlock was removed from OpenBao in PR chore(deps): update helm release kube-prometheus-stack to v61.8.0 #363 (GA 2.0.0), so the setting is a deprecated no-op in every version this repo could deploy.Evidence
Every task passed an independent review that re-ran the checks rather than trusting the implementer's report.
tofu validate/fmt -checktrivy config --exit-code=1tflint(incl.terraform_unused_declarations)shellcheck -e SC2154terramate fmt --checkdetect-secretsNot verified
None of this has run against a live GCP project. No cluster was deployed, no certificate issued, no binding created. The design's success criteria are all cluster-observable and remain unchecked, including the one assumption the whole chain rests on: that
config_caalone, without the CSR/sign/set-signed sequence, leaves the mount able to issue. The plan verifies that by hand as Task 7's first step, before any HCL is written.What the whole-branch review caught
Every mechanical gate passed —
tofu validate,fmt,trivy,tflint,shellcheck, links, doc-claims — and a review still found one Critical and five Important issues. None was reachable by tooling.Critical: the Secret Manager role granted project-wide read of every secret.
GCPWorkloadIdentityrendersProjectIAMMember, which is project-scoped, so External Secrets would have been able to readopenbao-priv-gcp-root-token, the recovery keys, and the intermediate CA private key — enough to mint any*.priv.gcp.ogenki.iocertificate offline, indefinitely. The branch contradicted itself:openbao/cluster/iam.tfexplicitly refuses project-widesecretAccessorfor the node, and the same boundary was opened one file over. Reduced to a singleversions.accesspermission, with the residual project-wide scope stated plainly and per-secret bindings recorded as the better approach for Task 8. This fails open — no deploy would ever have surfaced it.The stack validated but could not apply.
openbao_versionhad no default and was absent fromvariables.tfvars.tofu validatedoes not need variable values, so every gate stayed green while the first realterramate script run deploywould have prompted on a TTY or died in CI.A contract mismatch between the script and the secret it reads.
openbao-config.sh caparsed.cafrom JSON; the GCP CA chain is raw PEM. The exact invocation in the plan would have died on a jq parse error. Settled before Task 3 writes the secret rather than after.A boot path that fails silently. The two boot scripts are
join-ed into one process, sosetup-local-disks.sh'sexit 0on a missing data disk terminated the entire script — instance RUNNING, joined to the MIG, with no OpenBao installed. Now a hardexit 1, with the package install moved ahead of the disk setup that needsmkfs.xfs.A template change that never reached the node. The MIG had no
update_policy, so GCP's defaultOPPORTUNISTICwould adopt a new template without replacing the running instance — the GCP analogue of the AWS launch-template trap already recorded for this same service.A correction to the KMS rationale
The design's stated reason for moving Cloud KMS out of the stack was wrong, and the comment now says so. The provider's delete for key rings and crypto keys is a no-op that returns success —
tofu destroydoes not fail on them. The real reasons are that a rebuild's create hitsALREADY_EXISTSagainst a key ring that still exists server-side, and that destroying a managedgoogle_kms_crypto_keyschedules its key versions for destruction while exiting 0 — a silent trap, not a loud one.The comment explicitly pre-empts the wrong conclusion: re-testing "does destroy fail on these" and finding it doesn't is exactly what would tempt someone to move KMS back into the stack.