From 787ca90728cb37df5174a1e7bf844be7fb57a84d Mon Sep 17 00:00:00 2001 From: Smana Date: Mon, 24 Aug 2026 18:47:10 +0200 Subject: [PATCH 01/21] docs(gcp): design OpenBao on GCP, and retract the two-root PKI decision MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Workstream 11. Scope is PKI only — a root of trust, an issuing CA, a cert-manager role and one AppRole. The app tenant namespace, kv-v2 mount, snapshot AppRole and operator userpass are deliberately not ported: none has a consumer on GCP. The substantive change is to the certificate chain, and it supersedes the decision recorded in the 2026-08-18 design. That design chose two independent roots, one per cloud, and rejected a shared root because cross-signing would be "a manual ceremony on every rebuild". The premise was false. The intermediate lives in Secret Manager independently of OpenBao's lifecycle — which is exactly why AWS's survives rebuilds today — so signing happens once per cloud, ever. The better option was rejected over a cost it does not have. The first draft of this design also carried over the AWS shape where config_ca imports a pem_bundle containing the ROOT key into the live pki mount. The repository's own pki-and-secrets.md warns against it verbatim: "do not carry it into a deployment where the root CA matters." Compromising OpenBao would yield the root, which inverts the property an offline root exists to provide. So: one OFFLINE root, per-cloud intermediates, and the intermediate imported directly as the issuer. The root key is never in Secret Manager, never in the mount, never in OpenTofu state. This also removes four resources — the vault_pki_secret_backend_key -> intermediate_cert_request -> root_sign_intermediate -> intermediate_set_signed sequence exists only to generate an OpenBao-internal intermediate under an imported root. GCP adopts it first, AWS follows later. The interim cost is recorded honestly: until AWS migrates, clients trust two anchors. That was permanent under the superseded decision and is now a state with a defined end. Two findings the design surfaced that were not in the workstream row: - scripts/openbao-config.sh is AWS-only and needs a --cloud gcp flag. Its coupling is three seams (secret read, secret write, CLI prefix), which is what makes a flag right rather than a sibling script. - GCP Secret Manager IDs permit only letters, digits, hyphen and underscore. AWS's names are paths (certificates/priv.aws.ogenki.io/root-ca), so every name must be rewritten and the two clouds cannot share one convention. It also needs an addition to slice 5's IAM allowlist: External Secrets reading Secret Manager requires a GCPWorkloadIdentity claim, and the condition in gke/init/iam.tf allowlists only xplane_dns_editor. A second pre-created custom role is required. That is the mechanism working as intended. Risks are recorded rather than resolved, including the one assumption the whole chain rests on: that config_ca alone, without the CSR/sign/set-signed sequence, leaves the mount able to issue. Standard import-an-existing-CA flow, but never exercised in this repository — verify it early. Evidence: validate-links.sh exit 0. --- .../specs/2026-08-18-gcp-support-design.md | 30 +- .../specs/2026-08-24-gcp-openbao-design.md | 260 ++++++++++++++++++ 2 files changed, 284 insertions(+), 6 deletions(-) create mode 100644 docs/superpowers/specs/2026-08-24-gcp-openbao-design.md diff --git a/docs/superpowers/specs/2026-08-18-gcp-support-design.md b/docs/superpowers/specs/2026-08-18-gcp-support-design.md index f8bd66e3e..93d035502 100644 --- a/docs/superpowers/specs/2026-08-18-gcp-support-design.md +++ b/docs/superpowers/specs/2026-08-18-gcp-support-design.md @@ -402,10 +402,11 @@ GCP certificate issuance hard-depend on AWS *and* on the tailnet, in a platform whose stated point is that each cloud stands alone. It is the same coupling this design already flags as undesirable for the Flux GitHub App secret. -**B — Two OpenBaos, two roots. CHOSEN.** Each cloud runs its own OpenBao with its -own root CA. Fully independent; matches [ADR-0007](../../../website/content/docs/decisions/0007-cloud-abstraction-boundaries.md)'s +**B — Two OpenBaos, two roots. ~~CHOSEN~~ SUPERSEDED 2026-08-24.** Each cloud runs +its own OpenBao with its own root CA. Fully independent; matches [ADR-0007](../../../website/content/docs/decisions/0007-cloud-abstraction-boundaries.md)'s rule that platform-facing infrastructure stays cloud-shaped. Costs one extra -trust anchor for tailnet clients. +trust anchor for tailnet clients — and, as written here, leaves a live root key +in each cloud. **C — Two OpenBaos, one shared root.** Each cloud holds its own intermediate, both signed by a common root: one trust anchor, independent operation. Rejected @@ -415,7 +416,23 @@ holds, or cross-signing GCP's intermediate at bootstrap, which is a manual ceremony **on every rebuild** of a platform whose lifecycle is build-validate-destroy. -#### Why B, concretely +> **That rejection was wrong, and C is now the chosen shape.** The "ceremony on +> every rebuild" premise is false: the intermediate lives in Secret Manager +> independently of OpenBao's lifecycle, which is exactly why AWS's survives +> rebuilds today. Signing happens once per cloud, ever. Nor does a shared root +> require copying the root key anywhere — with a genuinely OFFLINE root it is +> never in any cloud's secret store. See +> [the OpenBao-on-GCP design](./2026-08-24-gcp-openbao-design.md). + +#### Why B was chosen, and why it no longer is + +> **Superseded 2026-08-24.** The reasoning below is preserved because the failure +> it describes is real and worth keeping; the conclusion drawn from it was too +> narrow. The lesson is that a signing key must be held outside OpenBao — not +> that each cloud needs its own root. An offline root shared across clouds +> satisfies the same lesson and additionally keeps the root key off every +> networked system. The current decision is in +> [the OpenBao-on-GCP design](./2026-08-24-gcp-openbao-design.md). The 2026-08-24 rebuild made the argument better than theory could. The private domain rename forced a new server certificate, and re-issuing it under the @@ -430,8 +447,9 @@ extra trust anchor and remove the entire class. #### Consequences to carry into workstream 11 -- Tailnet clients must trust **both** roots. That is the accepted cost; it is a - one-line addition wherever the AWS root is already distributed. +- Tailnet clients trust **two anchors only until AWS migrates** — its existing + live-key root, and the new offline root. Under the superseded two-root decision + this was permanent; it is now an interim state with a defined end. - GCP needs its own secret store for the root token and the cert-manager AppRole — **GCP Secret Manager**, mirroring what AWS Secrets Manager does today. The `flux-github-app` secret already establishes that pattern on GCP. diff --git a/docs/superpowers/specs/2026-08-24-gcp-openbao-design.md b/docs/superpowers/specs/2026-08-24-gcp-openbao-design.md new file mode 100644 index 000000000..70e7f9330 --- /dev/null +++ b/docs/superpowers/specs/2026-08-24-gcp-openbao-design.md @@ -0,0 +1,260 @@ +# OpenBao on GCP — private PKI for the GKE cluster + +**Status:** design approved 2026-08-24, not yet implemented +**Workstream:** 11 of the [GCP support design](./2026-08-18-gcp-support-design.md) +**Depends on:** workstream 1 (GCP network) — deployed. Also needs one addition to +workstream 5's IAM allowlist; see *A dependency on slice 5* below. + +## Why + +GCP has no private certificate authority. The GKE cluster can issue nothing for +`*.priv.gcp.ogenki.io`, which blocks every private-TLS consumer the AWS platform +takes for granted. AWS solves this with OpenBao at `bao.priv.aws.ogenki.io`; GCP +needs its own, because [`opentofu/aws/openbao/`](../../../opentofu/aws/openbao/) +is AWS-shaped throughout — ASG, ELB, KMS, Route53, and Secrets Manager for every +credential. + +## Scope + +**PKI only.** A root of trust, an issuing CA, a cert-manager role and one +AppRole. Deliberately NOT ported: the `app` tenant namespace, its kv-v2 mount, +the snapshot AppRole, and operator userpass login. None has a consumer on GCP, +and the GCP tree has been built minimal-until-needed throughout. + +## Decisions + +| Decision | Choice | Why | +|---|---|---| +| Scope | PKI only | No GCP consumer for the rest yet | +| Topology | Single node, `file` backend | Mirrors what AWS actually runs (`mode = "dev"` is committed); cheapest for a build-validate-destroy platform | +| Root CA custody | **Offline**. Signs GCP's intermediate now; becomes the shared anchor when AWS migrates | See *The PKI reconsideration* | +| cert-manager auth | AppRole via GCP Secret Manager + External Secrets | Parity with AWS; the existing `security/base/cert-manager` manifests work with only the SecretStore backend changed | +| Stack split | Two stacks, `cluster/` + `management/` | The `vault` provider needs a reachable, initialised server at plan time — the same constraint that splits `gke/init` from `gke/configure` | + +## The PKI reconsideration + +**This supersedes the two-root decision recorded in the GCP support design's +*Private certificates on GCP* section.** That section is amended to point here. + +### What was wrong + +The AWS design imports a pem_bundle containing the **root key** into the live +`pki_private_issuer` mount, then has OpenBao generate its own intermediate and +sign it internally. [`pki-and-secrets.md`](../../../website/content/docs/platform/security/pki-and-secrets.md) +documents this and warns against it verbatim: *"do not carry it into a deployment +where the root CA matters."* The first draft of this design carried it into GCP +anyway. + +It inverts the property an offline root exists to provide. A root signs +intermediates a handful of times in its life; keeping it off any +network-reachable system means compromising the issuing CA does not compromise +the trust anchor. As written, compromising OpenBao yields the root. + +The earlier two-root decision was also rejected-for-the-wrong-reason. It ruled +out a shared root because cross-signing would be *"a manual ceremony on every +rebuild"*. That premise is false: the intermediate lives in Secret Manager +independently of OpenBao's lifecycle, which is precisely why AWS's survives +rebuilds today. Signing happens **once per cloud, ever**. The better option was +rejected over a cost it does not have. + +### The chain + +``` +Offline Root CA (EC secp384r1) — never on a networked system + ├── GCP intermediate (EC secp384r1) → imported as pki_private_issuer → leaves + └── AWS intermediate → later; see Migration below +``` + +The root signs the GCP intermediate's CSR **once, offline**, then returns to +offline storage. Only the intermediate's certificate and key reach GCP Secret +Manager. `vault_pki_secret_backend_config_ca` imports that bundle and it *is* +the issuer. + +This also removes work. The AWS sequence — +`vault_pki_secret_backend_key` → `intermediate_cert_request` → +`root_sign_intermediate` → `intermediate_set_signed` — exists only to generate +an OpenBao-internal intermediate under an imported root. Importing the +openssl-made intermediate directly deletes all four resources. + +**The root key is never in Secret Manager, never in the mount, never in +OpenTofu state.** + +### Migration, and the interim cost + +GCP adopts this first; AWS follows later. Until it does, tailnet clients trust +**two** anchors — AWS's existing live-key root and the new offline root. That is +the same cost as the two-root design being abandoned, but temporary with a +defined end rather than permanent by construction. + +Migrating AWS means a new intermediate signed by the offline root and a +coordinated re-trust. Out of scope here. + +## Architecture + +Two stacks under `opentofu/gcp/openbao/`. + +| AWS | GCP | +|---|---| +| Auto Scaling Group + launch template | Zonal MIG + instance template (`europe-west4-a`) | +| Internal NLB, TCP 8200 | Internal passthrough Network LB — forwarding rule, backend service, health check | +| `aws_kms_key` | `google_kms_key_ring` + `google_kms_crypto_key` | +| Route53 private A record | `google_dns_record_set` in the existing `priv-gcp-ogenki-io` zone | +| Security group | `google_compute_firewall` | +| Instance profile + role policies | `google_service_account` + scoped Secret Manager / KMS bindings | +| Secrets Manager | GCP Secret Manager | +| cloud-init `user_data` | metadata `startup-script` | + +**Zonal is a real availability property, not an oversight.** The node dies with +the zone. That is consistent with the GKE cluster, which is also zonal, and with +a platform that is rebuilt rather than repaired. + +### Deliberately not ported + +- **Hardened-image plumbing.** AWS carries `ami_owner` / `ami_filter` variables + that are commented out and unused. Copying them would be copying scaffolding. +- **The `mode` variable.** AWS defines `dev`/`ha` and commits `dev`. A variable + with one reachable value is worse than a constant, and the HA path would ship + untested. Single node, stated plainly. +- **An always-on admin path.** AWS keeps SSM enabled because it is how you reach + a node whose boot script failed. GCP's counterpart is IAP TCP forwarding. The + firewall rule is written but disabled by default: the tailnet already reaches + the subnet, and an always-on admin path is a standing exposure. + +## Boot sequence + +Per instance, via metadata `startup-script`: + +1. Format and mount the data disk. +2. Install a pinned OpenBao version. +3. Fetch the server leaf certificate and key from Secret Manager using the + instance service account. +4. Write config: TLS listener on 8200, `file` storage, and a `gcpckms` seal + stanza. The instance service account needs + `cloudkms.cryptoKeyEncrypterDecrypter` on that key and nothing else. +5. Start the service. + +Initialisation stays a separate, deliberate step — +`scripts/openbao-config.sh init` — as on AWS. An auto-init path was considered +and rejected: a documented procedure already exists, and a second one is how the +two clouds drift. + +### Certificate details to carry forward + +From `pki-and-secrets.md`, all three still apply: + +- OpenBao's own leaf is **EC P-256** against the P-384 CAs. +- `openssl` writes key files world-readable — `chmod 600`. This key terminates + TLS for every client. +- The SAN list carries the **DNS name only, no IP**. On AWS that is why a client + reaching a Raft peer by private IP cannot verify TLS. Single-node here makes it + moot today; the constraint stays so it does not surprise a future HA change. + +## Management stack + +Mirrors AWS's `pki.tf` with the data source swapped and the internal-intermediate +sequence removed: + +- `vault_mount` — the `pki_private_issuer` mount +- `vault_pki_secret_backend_config_ca` — imports the intermediate bundle from + GCP Secret Manager +- `vault_pki_secret_backend_issuer` — to name the issuer `config_ca` creates and + set its usage. Whether this resource is needed at all depends on the same + assumption flagged in *Risks*: confirm what `config_ca` leaves behind before + writing it, rather than porting it from AWS on faith +- a cert-manager role scoped to `priv.gcp.ogenki.io` +- one AppRole, whose `role-id` / `secret-id` are written to Secret Manager + +## `openbao-config.sh` gains `--cloud gcp` + +The script is AWS-only today. Its coupling is contained to three seams, which is +what makes a flag the right shape rather than a sibling script: + +| Seam | AWS | GCP | +|---|---|---| +| write | `secretsmanager create-secret` / `update-secret` | `gcloud secrets create` / `versions add` | +| read | `secretsmanager get-secret-value` | `gcloud secrets versions access latest` | +| CLI prefix | `get_aws_cmd()` | project-scoped `gcloud` | + +`--region` / `--profile` stay AWS-only and `--project` is GCP-only; passing one +to the wrong cloud must fail at argument parsing, not at the API call. + +### Secret naming — a constraint that forces a rename + +AWS secret names are paths: `certificates/priv.aws.ogenki.io/root-ca`. +**GCP Secret Manager secret IDs permit only letters, digits, `-` and `_`** — no +slashes, no dots. Every name has to be rewritten; the two clouds cannot share one +convention. Following the `flux-github-app` precedent already in the GCP tree: + +| Secret | Contents | +|---|---| +| `openbao-priv-gcp-intermediate-ca` | intermediate certificate **+ key** | +| `openbao-priv-gcp-server-cert` | OpenBao's leaf, EC P-256 | +| `openbao-priv-gcp-ca-chain` | root + intermediate **certificates only** — public, what clients trust | +| `openbao-priv-gcp-root-token` | written by `openbao-config.sh init` | +| `openbao-priv-gcp-recovery-keys` | written by `openbao-config.sh init` | + +The domain's dots are dropped rather than encoded: `priv-gcp-ogenki-io` adds +length without disambiguating anything in a single-domain project. + +**No root-CA secret exists on GCP.** Its absence is the design. + +## How GKE consumes it + +External Secrets pulls the AppRole credentials and the CA chain from GCP Secret +Manager. cert-manager's `ClusterIssuer` points at `bao.priv.gcp.ogenki.io:8200`. +A `Certificate` resource proves the chain end to end. + +### A dependency on slice 5 + +External Secrets needs to read GCP Secret Manager, which means a Google identity +— a `GCPWorkloadIdentity` claim. But the IAM condition in +[`opentofu/gcp/gke/init/iam.tf`](../../../opentofu/gcp/gke/init/iam.tf) +allowlists **only** `xplane_dns_editor`. A claim requesting Secret Manager access +is refused at the provider. + +So this workstream requires a second pre-created custom role — `xplane_secret_reader`, +holding `secretmanager.versions.access` — added to `crossplane_grantable_roles`. + +That is the mechanism working as intended, not an obstacle: adding a capability +is a deliberate act in OpenTofu, and a claim cannot grant itself one. + +## Success criteria + +Falsifiable, verified against a live cluster. + +1. `bao status` against `https://bao.priv.gcp.ogenki.io:8200` reports + `Initialized: true`, `Sealed: false`, from a tailnet device. +2. The node is unsealed **without operator input** after a stop/start — proving + the `gcpckms` seal, not a manual unseal. +3. `openssl s_client` against the endpoint presents a chain that verifies to the + **offline root**, and the served leaf's SAN contains the DNS name and no IP. +4. `gcloud secrets list` shows **no secret containing the root CA private key**. +5. The `pki_private_issuer` mount's issuer is the openssl-made intermediate: + `bao read pki_private_issuer/issuer/default` returns a certificate whose + subject matches it, and no OpenBao-generated intermediate exists. +6. A cert-manager `Certificate` in the GKE cluster reaches `Ready=True`, issued + by the ClusterIssuer, with a chain terminating at the offline root. +7. `terramate script run deploy` for both stacks produces a 0-change plan on a + second run. +8. Teardown removes every billable resource; `gcloud compute instances list` and + `gcloud container clusters list` are empty afterwards. + +## Risks and open questions + +- **Offline root custody is a procedure, not a resource.** The design says the + root never touches a networked system, but where it *does* live — encrypted + media, a password manager — is an operator decision this document does not + make. It must be written down somewhere before the first signing ceremony, + or the property is aspirational. +- **The interim two-anchor state has no deadline.** AWS migrates "later". If + that slips indefinitely the platform keeps two trust anchors, which is the + outcome this design set out to avoid. +- **`file` storage on a single node has no backup story here.** The snapshot + AppRole was scoped out because GCP has no consumer, but that also means + nothing is backing up the PKI. Acceptable while the platform is rebuilt from + scratch; not acceptable the moment anything long-lived depends on it. +- **Untested at design time:** that `config_ca` alone, without the + CSR/sign/set-signed sequence, leaves the mount able to issue. This is the + standard import-an-existing-CA flow, but it has not been exercised in this + repository, and it is the single assumption the chain rests on. Verify it + early in implementation rather than at the end. From fd3bba74e2bae2654674179846d23787be16c256 Mon Sep 17 00:00:00 2001 From: Smana Date: Mon, 24 Aug 2026 18:54:12 +0200 Subject: [PATCH 02/21] docs(gcp): implementation plan for OpenBao on GCP MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Eight tasks for docs/superpowers/specs/2026-08-24-gcp-openbao-design.md. The verification model is adapted rather than borrowed: this is infrastructure, so "write the failing test first" becomes "run the verification command and watch it fail for the right reason". The gates are the repo's real ones — tofu validate, trivy config, tofu plan, shellcheck, validate-manifests.sh. Forcing a unit-test framework onto Terraform would have produced ceremony, not evidence. Task ordering is deliberate: - Task 1 (the Secret Manager IAM role) is first because it is independent of everything else and safe to merge alone. - Task 3 (the offline PKI ceremony) comes before any stack, because cluster/ reads the server certificate at boot. It produces secrets and a runbook, not code. - Task 7 Step 1 verifies the design's one untested assumption BEFORE any HCL is written: that config_ca alone, without the CSR/sign/set-signed sequence, leaves the mount able to issue. If it fails, the four removed resources may be required and the design needs revisiting. Discovering that after writing the stack would be the expensive order. Two things the plan carries from hard experience in this repo rather than from the spec: read Terramate's output rather than its exit code, which has reported 0 over a failed run repeatedly; and the KMS key ring survives teardown by design via prevent_destroy, so it must not be mistaken for a leak. Self-review found two inconsistencies, both fixed inline: the startup script's templated-variable list omitted region, kms_key_ring and kms_crypto_key, which templatefile would have failed on at plan time; and dns.tf referenced google_compute_address.openbao while the load-balancer step never named that resource. The .secrets.baseline entry is a false positive: detect-secrets' Secret Keyword heuristic fires on a PROSE line naming manifest fields (caBundleSecretRef). A pragma would have been visible text in the rendered document, so the baseline is the right mechanism. The diff adds exactly one entry and suppresses nothing else. Evidence: validate-links.sh exit 0; no placeholders; the six Secret Manager names are consistent across all eight tasks. --- .secrets.baseline | 11 +- .../plans/2026-08-24-gcp-openbao.md | 896 ++++++++++++++++++ 2 files changed, 906 insertions(+), 1 deletion(-) create mode 100644 docs/superpowers/plans/2026-08-24-gcp-openbao.md diff --git a/.secrets.baseline b/.secrets.baseline index 0761e0a7b..6bf1cfe56 100644 --- a/.secrets.baseline +++ b/.secrets.baseline @@ -145,6 +145,15 @@ "line_number": 46 } ], + "docs/superpowers/plans/2026-08-24-gcp-openbao.md": [ + { + "type": "Secret Keyword", + "filename": "docs/superpowers/plans/2026-08-24-gcp-openbao.md", + "hashed_secret": "13d26a89a916713f8f40cf05869c7b25c3e6ea54", + "is_verified": false, + "line_number": 853 + } + ], "infrastructure/base/gapi/platform-private-gateway-certificate.yaml": [ { "type": "Secret Keyword", @@ -441,5 +450,5 @@ } ] }, - "generated_at": "2026-08-24T08:08:41Z" + "generated_at": "2026-08-24T16:53:56Z" } diff --git a/docs/superpowers/plans/2026-08-24-gcp-openbao.md b/docs/superpowers/plans/2026-08-24-gcp-openbao.md new file mode 100644 index 000000000..0255b2421 --- /dev/null +++ b/docs/superpowers/plans/2026-08-24-gcp-openbao.md @@ -0,0 +1,896 @@ +# OpenBao on GCP Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Give the GKE cluster a private certificate authority — OpenBao at `bao.priv.gcp.ogenki.io:8200`, issuing for `*.priv.gcp.ogenki.io` from an intermediate signed by an offline root. + +**Architecture:** Two OpenTofu stacks under `opentofu/gcp/openbao/`, mirroring the AWS split. `cluster/` builds a single-node zonal MIG behind an internal passthrough Network LB, auto-unsealed by Cloud KMS. `management/` runs the `vault` provider against the live endpoint and configures the PKI only. The root CA never touches a networked system; only the intermediate reaches GCP Secret Manager, imported directly as the issuer. + +**Tech Stack:** OpenTofu, Terramate, `google` provider, `vault` provider, GCP Secret Manager, Cloud KMS, Cloud DNS, External Secrets, cert-manager. + +**Spec:** [`docs/superpowers/specs/2026-08-24-gcp-openbao-design.md`](../specs/2026-08-24-gcp-openbao-design.md) + +## Global Constraints + +- **Domain:** `priv.gcp.ogenki.io`. Endpoint `bao.priv.gcp.ogenki.io:8200`. +- **Region/zone:** `europe-west4` / `europe-west4-a` — zonal, matching the GKE cluster. +- **CA key types:** root and intermediate `EC secp384r1`; OpenBao's server leaf `EC prime256v1` (P-256). +- **Server leaf SAN:** DNS name only. **No IP SAN.** +- **Key file permissions:** `chmod 600` — `openssl` writes world-readable by default. +- **Secret Manager IDs** permit only letters, digits, `-`, `_`. No slashes, no dots. +- **State backend:** S3 bucket `demo-smana-remote-backend`, region `eu-west-3` (the *bucket's* region, unrelated to the GCP region). +- **Terramate gate:** every script guards on `TM_GCP_ENABLED=true` and no-ops with `[skip]` otherwise. +- **No secret may be templated into instance metadata.** TLS material is fetched at boot from Secret Manager. +- **Never commit CA private keys.** The root key never leaves offline storage; the intermediate key exists only in Secret Manager. + +### Verification model for this plan + +This is infrastructure, so "write a failing test first" means **run the verification command and watch it fail for the right reason**, then implement, then watch it pass. The gates are: + +| Layer | Command | +|---|---| +| OpenTofu | `tofu validate`, `trivy config --exit-code=1 --ignorefile=./.trivyignore.yaml .`, `tofu plan -var-file=variables.tfvars` | +| Shell | `shellcheck