From d378498ae202c369a3b39d21875d9d22995fcf31 Mon Sep 17 00:00:00 2001 From: Smana Date: Mon, 24 Aug 2026 11:43:05 +0200 Subject: [PATCH 1/7] =?UTF-8?q?feat(gcp):=20slice=204=20=E2=80=94=20node?= =?UTF-8?q?=20auto-provisioning=20and=20one=20ComputeClass?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Narrow on purpose. The design's slice 4 exists to settle ONE question -- criterion 12: does a freshly auto-created node carry `node.cilium.io/agent-not-ready` at registration, and does Cilium clear it? The `io` and GPU classes follow once that is answered; writing three now would mean debugging three variants of the same unknown. TWO HALVES, BOTH REQUIRED opentofu: `cluster_autoscaling` on the GKE module. Without it a ComputeClass with nodePoolAutoCreation still schedules, but only onto pools that already exist -- nothing is ever provisioned and the premise goes untested. The ceiling (criterion 16) is set low deliberately: a too-low limit is an Unschedulable pod and a one-line change, a too-high one is a bill. manifest: one ComputeClass, spot-only across e2 then n2, with `whenUnsatisfiable: DoNotScaleUp` -- criterion 14 requires zero on-demand fallback, so exhausted spot leaves pods Pending rather than quietly becoming expensive. THE PART THAT IS ACTUALLY THE SLICE `spec.nodePoolConfig.taints` carries node.cilium.io/agent-not-ready. The static pool sets that taint through `node_pools_taints` in OpenTofu; an auto-created pool has no OpenTofu to set it, because NAP creates the pool. Anything the pool must carry has to be declared in the ComputeClass or it does not exist. Without it a new node registers Ready before Cilium owns its networking and accepts pods it cannot network -- surfacing as FailedCreatePodSandBox referencing a missing CNI (criterion 13), which points at the CNI rather than at autoscaling. I nearly shipped the class without it; the schema check is what surfaced `nodePoolConfig.taints`. SCHEMA VALIDATION NEEDED A FOURTH CATALOG SOURCE `flux schema validate` runs with skipMissingSchemas: false by design -- an unknown Kind FAILS rather than passing unvalidated -- and ComputeClass is in no public catalog. Unlike the Envoy AI Gateway and Karpenter CRDs, which gen-catalog.sh renders from pinned Helm charts, GKE installs this one itself and publishes no chart, so it is vendored under scripts/flux-schema/vendored-crds/ with its GKE component version recorded in the annotations. It is NOT under crds/ -- Flux applies that tree, and this CRD is reconciled by GKE's addon manager. Applying it would fight GKE. COST (test clusters, spot and cheapest by default) `cluster_autoscaling` left disk on module defaults of 100 GB pd-standard -- TWICE the static pool's 50 GB, purely because the field was unset. Now 50 GB to match, and pd-standard as the cheapest type, with the pd-balanced asymmetry against the static pool noted in place so a later performance surprise has a visible cause. Verified: ./scripts/validate-manifests.sh -> Valid: 1189, Invalid: 0, Skipped: 0, all gates passed -- the ComputeClass is genuinely validated, not skipped; `kubectl apply --dry-run=server` accepts it against the live GKE API; tofu validate and tofu fmt clean; shellcheck -x -S warning clean on the generator. Criterion 12 itself is NOT yet verified -- that needs the cluster to track this branch and actually scale up. Next step, not a claim. --- clusters/gcp-mycluster-0/infrastructure.yaml | 30 + .../computeclass/general-purpose.yaml | 78 + .../computeclass/kustomization.yaml | 5 + .../gcp-mycluster-0/kustomization.yaml | 16 + opentofu/gcp/gke/init/main.tf | 53 + opentofu/gcp/gke/init/variables.tf | 22 + scripts/flux-schema/gen-catalog.sh | 6 + .../vendored-crds/gke-computeclass.yaml | 2937 +++++++++++++++++ 8 files changed, 3147 insertions(+) create mode 100644 clusters/gcp-mycluster-0/infrastructure.yaml create mode 100644 infrastructure/gcp-mycluster-0/computeclass/general-purpose.yaml create mode 100644 infrastructure/gcp-mycluster-0/computeclass/kustomization.yaml create mode 100644 infrastructure/gcp-mycluster-0/kustomization.yaml create mode 100644 scripts/flux-schema/vendored-crds/gke-computeclass.yaml diff --git a/clusters/gcp-mycluster-0/infrastructure.yaml b/clusters/gcp-mycluster-0/infrastructure.yaml new file mode 100644 index 000000000..004a1ac3f --- /dev/null +++ b/clusters/gcp-mycluster-0/infrastructure.yaml @@ -0,0 +1,30 @@ +# GCP-only infrastructure for this cluster. +# +# Points at infrastructure/gcp-mycluster-0, NOT at infrastructure/base — that +# tree is AWS-shaped (Karpenter, aws-load-balancer-controller, EKS Pod +# Identities) and applying it here would fail on resources whose CRDs do not +# exist. The GCP tree is deliberately minimal until the design's slices 6-7 +# settle which parts of the shared tree are genuinely cloud-neutral. +# +# It sources infra-artifact, which flux/artifact-generators/monorepo-split.yaml +# already builds from `@repo/infrastructure/**` — so the new directory needs no +# generator change, only this Kustomization. +# +# No dependsOn: crds. The only resource here is a ComputeClass, whose CRD is +# installed by GKE itself (`cloud.google.com/v1`, present on a fresh cluster) +# rather than by crds/base. Adding that edge would couple this to a Kustomization +# it does not need and delay it behind the whole upstream CRD set. +apiVersion: kustomize.toolkit.fluxcd.io/v1 +kind: Kustomization +metadata: + name: infrastructure + namespace: flux-system +spec: + prune: true + interval: 1m0s + path: ./infrastructure/gcp-mycluster-0 + sourceRef: + kind: ExternalArtifact + name: infra-artifact + dependsOn: + - name: namespaces diff --git a/infrastructure/gcp-mycluster-0/computeclass/general-purpose.yaml b/infrastructure/gcp-mycluster-0/computeclass/general-purpose.yaml new file mode 100644 index 000000000..b5343e0d2 --- /dev/null +++ b/infrastructure/gcp-mycluster-0/computeclass/general-purpose.yaml @@ -0,0 +1,78 @@ +# General-purpose ComputeClass — the FIRST and, for now, the ONLY one. +# +# ADR-0006 chose GKE node auto-provisioning over Karpenter on GCP. This object is +# what turns that decision into running nodes: it tells NAP which machine shapes +# to try, in what order, and lets it CREATE a node pool rather than only pick +# among existing ones. +# +# Deliberately one class, not three. The design's slice 4 exists to settle a +# single question -- does a freshly auto-created node carry +# `node.cilium.io/agent-not-ready` at registration, and does Cilium clear it? +# Writing three classes before that is answered would mean debugging three +# variants of the same unknown. The `io` and GPU classes follow once this one is +# proven. +# +# WHY THE TAINT QUESTION MATTERS: on GKE, Cilium replaces the CNI. A node that +# registers Ready before Cilium owns its networking will accept pods it cannot +# network, which surfaces as FailedCreatePodSandBox referencing a missing CNI +# rather than as anything pointing at autoscaling. Static pools set the taint in +# OpenTofu; an auto-created pool has no OpenTofu to set it, which is precisely +# why this is the criterion the slice is built around. +apiVersion: cloud.google.com/v1 +kind: ComputeClass +metadata: + name: general-purpose +spec: + # Ordered fallback. NAP walks these in sequence and takes the first that can be + # provisioned, so the list is a preference order, not a set. + # + # e2-standard-4 first because it is what the static pool runs -- keeping the + # auto-created shape identical to the hand-created one removes a variable from + # the taint experiment. n2-standard-4 second as a spot-availability fallback: + # a GKE node pool takes ONE machine type, so a single shape concentrates spot + # interruption risk, which is the limitation the static pool's own comment + # calls out as "breadth coming later from ComputeClass". + priorities: + - machineFamily: e2 + spot: true + - machineFamily: n2 + spot: true + + # No on-demand entry, deliberately. Design criterion 14 requires general-purpose + # nodes to be spot with ZERO on-demand fallback: this is a reference platform + # where an unnoticed fallback to on-demand is a silent cost regression, and the + # workloads are all restartable. + # + # The consequence is honest: when no spot capacity exists in any listed family, + # pods stay Pending rather than quietly becoming expensive. + whenUnsatisfiable: DoNotScaleUp + + nodePoolConfig: + # THE POINT OF THIS SLICE. + # + # The static pool sets this taint through `node_pools_taints` in + # opentofu/gcp/gke/init/main.tf. An auto-created pool has no OpenTofu to set + # it -- NAP creates the pool, so anything the pool must carry has to be + # declared here or it does not exist. + # + # Without it, a new node registers Ready before Cilium owns its networking + # and accepts pods it cannot network. That surfaces as + # FailedCreatePodSandBox referencing a missing CNI (design criterion 13), + # which points at the CNI rather than at autoscaling -- so the cause is a + # long way from the symptom. + # + # Cilium removes the taint once its agent is ready on the node. Nothing else + # clears it, which is what makes it a safe gate rather than a deadlock. + taints: + - key: node.cilium.io/agent-not-ready + value: "true" + effect: NoSchedule + + # Must match the static pool and the NAP-level image_type. Cilium's + # DaemonSet depends on a writable /home/kubernetes/bin, which is a + # Container-Optimized OS path -- the very first GKE deploy failed on exactly + # this, and on an auto-created node it would fail where nobody was looking. + imageType: cos_containerd + + nodePoolAutoCreation: + enabled: true diff --git a/infrastructure/gcp-mycluster-0/computeclass/kustomization.yaml b/infrastructure/gcp-mycluster-0/computeclass/kustomization.yaml new file mode 100644 index 000000000..ff94770f8 --- /dev/null +++ b/infrastructure/gcp-mycluster-0/computeclass/kustomization.yaml @@ -0,0 +1,5 @@ +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization + +resources: + - general-purpose.yaml diff --git a/infrastructure/gcp-mycluster-0/kustomization.yaml b/infrastructure/gcp-mycluster-0/kustomization.yaml new file mode 100644 index 000000000..58464989d --- /dev/null +++ b/infrastructure/gcp-mycluster-0/kustomization.yaml @@ -0,0 +1,16 @@ +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization + +# GCP-only infrastructure for gcp-mycluster-0. +# +# Separate from infrastructure/mycluster-0/ rather than an overlay on it: that +# tree is AWS-shaped (Karpenter NodePools, aws-load-balancer-controller, +# EKS Pod Identities) and shares no resources with this one. A ComputeClass has +# no AWS counterpart at all -- ADR-0006 chose NAP over Karpenter precisely +# because the two autoscalers do not share an API. +# +# This is the manifest-side counterpart to the opentofu/{aws,gcp,shared} split, +# and is deliberately minimal until the design's slices 6-7 settle which parts of +# the shared tree are genuinely cloud-neutral. +resources: + - computeclass diff --git a/opentofu/gcp/gke/init/main.tf b/opentofu/gcp/gke/init/main.tf index 5ce5542e1..bfa509d1c 100644 --- a/opentofu/gcp/gke/init/main.tf +++ b/opentofu/gcp/gke/init/main.tf @@ -136,6 +136,59 @@ module "gke" { # `terramate script run destroy` fail rather than protect anything of value. deletion_protection = false + # ── 6. NODE AUTO-PROVISIONING (ADR-0006) ────────────────────────────────── + # This is what makes a ComputeClass able to CREATE node pools rather than only + # select among existing ones. Without it a ComputeClass with + # nodePoolAutoCreation still schedules, but only onto pools that already exist, + # so nothing new is ever provisioned and the slice's whole premise is untested. + # + # The ceiling is design criterion 16: an oversized workload must stay + # Unschedulable rather than growing the cluster without bound. It is set low on + # purpose -- this is a reference platform, and the failure mode of a too-high + # limit is a bill rather than an error. + # + # image_type MUST match the static pool's (criterion 14). Cilium's DaemonSet is + # built around a writable /home/kubernetes/bin on Container-Optimized OS; an + # auto-created node on a different image would fail the same way the very first + # GKE deploy did, but only on nodes nobody created by hand. + # + # OPTIMIZE_UTILIZATION over BALANCED: criterion 17 wants empty auto-created + # pools removed on scale-down, and BALANCED is deliberately reluctant to do so. + # The trade is more pod churn, which is acceptable here and would not be on a + # latency-sensitive platform. + cluster_autoscaling = { + enabled = true + autoscaling_profile = "OPTIMIZE_UTILIZATION" + + min_cpu_cores = 0 + max_cpu_cores = var.autoscaling_max_cpu_cores + min_memory_gb = 0 + max_memory_gb = var.autoscaling_max_memory_gb + + # GPU limits stay empty until the GPU ComputeClass exists. An entry here + # would let NAP provision accelerators that nothing yet asks for. + gpu_resources = [] + + auto_repair = true + auto_upgrade = true + + image_type = var.node_image_type + + # COST. This is a test cluster that gets rebuilt, so the cheap option wins + # wherever it is not actively misleading. + # + # Both of these are otherwise left on module defaults of 100 GB pd-standard + # -- TWICE the static pool's disk, which is the sort of thing that costs + # money quietly because nobody set it. + # + # 50 GB matches the static pool rather than being an independent guess. + # pd-standard is the cheapest disk type; the static pool uses pd-balanced, + # so if auto-created nodes ever behave worse than hand-created ones under + # image pulls or log writes, this asymmetry is the first thing to look at. + disk_size = var.node_disk_size_gb + disk_type = "pd-standard" + } + node_pools = [ { name = "static" diff --git a/opentofu/gcp/gke/init/variables.tf b/opentofu/gcp/gke/init/variables.tf index f13ee245e..17defe990 100644 --- a/opentofu/gcp/gke/init/variables.tf +++ b/opentofu/gcp/gke/init/variables.tf @@ -87,3 +87,25 @@ variable "tags" { type = map(string) default = {} } + +# ── Node auto-provisioning ceiling ────────────────────────────────────────── +# Design criterion 16: cluster resourceLimits are set, and an oversized workload +# stays Unschedulable at the ceiling rather than growing the cluster to fit it. +# +# Deliberately small. This is a reference platform that gets rebuilt, so the cost +# of a limit that is too LOW is an Unschedulable pod and a one-line change; the +# cost of one that is too HIGH is a bill nobody notices until it arrives. +# +# The static pool is 2-3 x e2-standard-4 (4 vCPU / 16 GiB each), so this leaves +# room for roughly four more comparable nodes before the ceiling bites. +variable "autoscaling_max_cpu_cores" { + description = "Total vCPU ceiling across all auto-provisioned node pools" + type = number + default = 32 +} + +variable "autoscaling_max_memory_gb" { + description = "Total memory ceiling in GiB across all auto-provisioned node pools" + type = number + default = 128 +} diff --git a/scripts/flux-schema/gen-catalog.sh b/scripts/flux-schema/gen-catalog.sh index 61f47117c..c68a2fb9c 100755 --- a/scripts/flux-schema/gen-catalog.sh +++ b/scripts/flux-schema/gen-catalog.sh @@ -5,6 +5,8 @@ # 1. The repo's own Crossplane XRDs -> cloud.ogenki.io/* # 2. Envoy AI Gateway CRDs -> aigateway.envoyproxy.io/* # (absent from the hosted ecosystem catalog) +# 4. GKE ComputeClass CRD -> cloud.google.com/v1 ComputeClass +# (VENDORED, not rendered: GKE installs it and publishes no chart) # 3. Karpenter CRDs -> karpenter.k8s.aws/*, karpenter.sh/* # (PRESENT in the hosted ecosystem catalog but STALE: it pins an older # provider release that predates fields we use — e.g. EC2NodeClass @@ -188,6 +190,10 @@ echo "==> Extracting JSON Schemas into ${build_dir}/" "${FLUX_BIN}" schema extract crd "${tmp}/aigateway-crds.yaml" -d "${build_dir}" "${FLUX_BIN}" schema extract crd "${tmp}/karpenter-crds.yaml" -d "${build_dir}" "${FLUX_BIN}" schema extract crd "${tmp}/barman-crds.yaml" -d "${build_dir}" +# GKE ComputeClass. Vendored rather than rendered: unlike the three above, GKE +# installs this CRD itself and publishes no chart to render it from. See the +# header of the file for how it was captured and when to refresh it. +"${FLUX_BIN}" schema extract crd "${REPO_ROOT}/scripts/flux-schema/vendored-crds/gke-computeclass.yaml" -d "${build_dir}" echo "==> Verifying the catalog is complete" for kind in app sqlinstance inferenceservice epi; do diff --git a/scripts/flux-schema/vendored-crds/gke-computeclass.yaml b/scripts/flux-schema/vendored-crds/gke-computeclass.yaml new file mode 100644 index 000000000..bdfca1d3e --- /dev/null +++ b/scripts/flux-schema/vendored-crds/gke-computeclass.yaml @@ -0,0 +1,2937 @@ +# VENDORED, NOT APPLIED. Do not move this under crds/ -- Flux applies that tree, +# and this CRD is installed and reconciled by GKE's own addon manager +# (addonmanager.kubernetes.io/mode: Reconcile). Applying it would fight GKE. +# +# It exists solely so `flux schema validate` has a schema for +# cloud.google.com/v1 ComputeClass. The repo runs with skipMissingSchemas: false +# by design -- an unknown Kind FAILS the build rather than passing unvalidated -- +# and ComputeClass is in no public catalog: unlike the Envoy AI Gateway and +# Karpenter CRDs, which gen-catalog.sh renders from pinned Helm charts, GKE +# installs this one itself and publishes no chart to render. +# +# Captured with: +# kubectl get crd computeclasses.cloud.google.com -o yaml +# from gcp-mycluster-0, with cluster-specific metadata (uid, resourceVersion, +# managedFields, status) stripped. +# +# Provenance is in the annotations below: components.gke.io/component-version +# records the GKE component release this was taken from. Refresh it when the +# cluster's GKE version moves and a ComputeClass field you need is missing -- +# a stale copy shows up as a validation error naming the field, not as silence. +apiVersion: apiextensions.k8s.io/v1 +kind: CustomResourceDefinition +metadata: + annotations: + components.gke.io/component-name: clusterautoscaler + components.gke.io/component-version: 35.202.10-gke.1 + components.gke.io/layer: addon + controller-gen.kubebuilder.io/version: v0.19.0 + labels: + addonmanager.kubernetes.io/mode: Reconcile + name: computeclasses.cloud.google.com +spec: + conversion: + strategy: None + group: cloud.google.com + names: + kind: ComputeClass + listKind: ComputeClassList + plural: computeclasses + shortNames: + - cc + - ccs + singular: computeclass + scope: Cluster + versions: + - name: v1 + schema: + openAPIV3Schema: + description: 'ComputeClass is a way to impact Cluster Autoscaler scaling + + decisions based on user preferences. It gives control over preference of + + hardware to be selected by Cluster Autoscaler. + + Given ComputeClass affects only workloads using workload separation + + label equal to CCs name, except ComputeClass with name default + + which will be used for workloads not specifying any preferences.' + properties: + apiVersion: + description: 'APIVersion defines the versioned schema of this representation + of an object. + + Servers should convert recognized schemas to the latest internal value, + and + + may reject unrecognized values. + + More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources' + type: string + kind: + description: 'Kind is a string value representing the REST resource this + object represents. + + Servers may infer this from the endpoint the client submits requests + to. + + Cannot be updated. + + In CamelCase. + + More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds' + type: string + metadata: + type: object + spec: + description: 'Specification of the ComputeClass object. + + More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#spec-and-status.' + properties: + activeMigration: + description: 'ActiveMigration describes settings related to active + reconciliation of + + a given ComputeClass.' + properties: + ensureAllDaemonSetPodsRunning: + description: 'EnsureAllDaemonSetPodsRunning defines whether node + pools should be migrated + + to larger ones to ensure that all daemon sets are schedulable.' + type: boolean + optimizeRulePriority: + default: false + description: 'OptimizeRulePriority defines whether workloads affected + by given + + ComputeClass should be migrated to nodepool defined by higher + priority rule, if possible.' + type: boolean + required: + - optimizeRulePriority + type: object + autopilot: + description: Autopilot describes the autopilot settings for a given + ComputeClass. + properties: + enabled: + default: false + description: Enabled indicates whether nodes created for this + compute class should be Autopilot managed. + type: boolean + x-kubernetes-validations: + - message: Autopilot is immutable + rule: self == oldSelf + required: + - enabled + type: object + x-kubernetes-validations: + - message: Autopilot is immutable + rule: self == oldSelf + autoscalingPolicy: + description: 'AutoscalingPolicy describes settings related to active + reconciliation of + + a given ComputeClass.' + properties: + consolidationDelayMinutes: + description: 'ConsolidationDelayMinutes determines how long a + node should be unneeded before it is eligible for scale down. + + Minimum duration is 1 minute, maximum is 24 hours or 1440 minutes' + maximum: 1440 + minimum: 1 + type: integer + consolidationThreshold: + description: ConsolidationThreshold determines resource utilization + threshold below which a node can be considered for scale down. + maximum: 100 + minimum: 0 + type: integer + gpuConsolidationThreshold: + description: 'GPUConsolidationThreshold determines GPU resource + utilization threshold below which a node can be considered for + scale down. + + Utilization calculation only cares about GPU resource for accelerator + node, CPU and memory utilization will be ignored.' + maximum: 100 + minimum: 0 + type: integer + type: object + description: + description: 'Description is an arbitrary string that usually provides + guidelines on + + when this compute class should be used.' + type: string + nodePoolAutoCreation: + default: + enabled: false + description: 'NodePoolAutoCreation describes the auto provisioning + settings for a given + + ComputeClass.' + properties: + enabled: + default: false + description: Enabled indicates whether NodePoolAutoCreation is + enabled for a given ComputeClass. + type: boolean + required: + - enabled + type: object + nodePoolConfig: + description: 'NodePoolConfig defines required node pool configuration. + Existing node pools will be matched with the ComputeClass + + only if their configuration match this field. Auto-provisioned node + pools will be created with this configuration.' + properties: + autoRepair: + description: AutoRepair if set to true specifies that a node pool + should have auto repair enabled, disabled in case of being set + to false. + type: boolean + autoUpgrade: + description: AutoUpgrade if set to true specifies that a node + pool should have auto upgrade enabled, disabled in case of being + set to false. + type: boolean + confidentialNodeType: + description: "ConfidentialNodeType: Defines the type of technology\ + \ used by the\nconfidential node.\n\nPossible values:\n \"\ + CONFIDENTIAL_INSTANCE_TYPE_UNSPECIFIED\" - No type specified.\ + \ Do not use\nthis value.\n \"SEV\" - AMD Secure Encrypted\ + \ Virtualization.\n \"SEV_SNP\" - AMD Secure Encrypted Virtualization\ + \ - Secure Nested Paging.\n \"TDX\" - Intel Trust Domain eXtension." + enum: + - CONFIDENTIAL_INSTANCE_TYPE_UNSPECIFIED + - SEV + - SEV_SNP + - TDX + type: string + dra: + description: 'Dra describes settings related to dynamic resource + allocation + + and its integration with autoprovisioning' + properties: + networking: + properties: + enabled: + default: false + type: boolean + type: object + type: object + gvnic: + description: Gvnic contains Google Virtual NIC settings. + properties: + enabled: + default: false + description: Enabled indicates whether gVNIC is enabled on + the node pool. + type: boolean + required: + - enabled + type: object + imageStreaming: + description: ImageStreaming contains image streaming settings. + properties: + enabled: + default: false + description: Enabled enables container image` streaming. + type: boolean + required: + - enabled + type: object + imageType: + description: Image type used by nodes in the node pool. + enum: + - cos_containerd + - ubuntu_containerd + type: string + ipType: + description: 'IPType specifies whether the nodes in the node pool + use public or private IP addresses. + + Possible values are "public" or "private". + + An empty string indicates the default IP type. + + This setting corresponds to the presence and value of the cloud.google.com/private-node + node selector.' + enum: + - public + - private + type: string + loggingConfig: + description: Contains logging configuration. + properties: + loggingVariantConfig: + description: Logging variant configuration. + properties: + variant: + description: Logging variant deployed on nodes. + enum: + - DEFAULT + - MAX_THROUGHPUT + type: string + type: object + type: object + nodeLabels: + additionalProperties: + type: string + description: 'NodeLabels is used to add user defined Kubernetes + labels to all nodes in the new node pool. + + These labels are applied to the Kubernetes API node object and + can be used in nodeSelectors for pod scheduling. + + Note: Node labels are distinct from GKE labels. + + More info: https://cloud.google.com/sdk/gcloud/reference/container/node-pools/create#--node-labels' + maxProperties: 100 + type: object + resourceManagerTags: + description: 'ResourceManagerTags defines what existing GCE resource + manager tag key/value pairs + + with purpose GCE_FIREWALL to attach to all node pools. + + Referenced Tags must be created beforehand via Resource Manager + API.' + items: + description: 'Tags define the key/value of resource manager + tags. + + Tags must be in one of the following formats ([KEY]=[VALUE]) + + 1. tagKeys/{tag_key_id}=tagValues/{tag_value_id} + + 2. {org_id}/{tag_key_name}={tag_value_name} + + 3. {project_id}/{tag_key_name}={tag_value_name}' + properties: + key: + type: string + value: + type: string + required: + - key + - value + type: object + maxItems: 5 + type: array + serviceAccount: + description: ServiceAccount used by the node pool. + type: string + taints: + description: 'Taints is used to add user defined Kubernetes taints + to all nodes in the new node pool. + + These taints are applied to the Kubernetes API node object and + can be used in tolerations for pod scheduling.' + items: + description: 'TaintConfig applies the given kubernetes taints + on all nodes in the new node pool, which can be used with + tolerations for pod scheduling. + + Any workload that does not tolerate the taints specified in + this object will not be scheduled to the node pool. + + More info: https://cloud.google.com/sdk/gcloud/reference/container/node-pools/create#--node-taints' + properties: + effect: + description: 'It defines the taint''s effect on pods that + does not have the necessary toleration. + + The following values are supported: NoSchedule, PreferNoSchedule, + and NoExecute.' + enum: + - NoSchedule + - PreferNoSchedule + - NoExecute + type: string + key: + description: Node taint key. The key must conform to syntax + described in https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#syntax-and-character-set. + maxLength: 320 + type: string + value: + description: The value that matches the specified taint + key. + maxLength: 63 + pattern: ^([a-z0-9][-A-Za-z0-9_.]{1,61})?[A-Za-z0-9]$ + type: string + required: + - effect + - key + type: object + maxItems: 100 + type: array + workloadType: + description: 'WorkloadType defines Collection or Goodput SLO for + the workload. Currently + + supported values: + + * HIGH_AVAILABILITY - for Collection SLO + + * HIGH_THROUGHPUT - for Goodput SLO + + HIGH_AVAILABILITY is desired for running serving workloads which + require + + most of the infrastructure (slices) running all the time to + achieve high + + availability. + + HIGH_THROUGHPUT is desired for running batch/training jobs + + which require all underlying infrastructure (slices) running + for most of + + the time to make progress. HIGH_THROUGHPUT can be only set for + a multi-host + + scenario, that is, when NodePoolGroup is set.' + enum: + - HIGH_AVAILABILITY + - HIGH_THROUGHPUT + type: string + type: object + nodePoolGroup: + description: 'NodePoolGroup defines required node pool configurations + that are shared between a group of node pools. + + Existing node pools will be matched with the ComputeClass only if + their configuration matches this field. + + Auto-provisioned node pools will be created with this configuration.' + properties: + name: + description: Name defines the name of the node pool group, e.g. + MultiMIG + minLength: 1 + type: string + required: + - name + type: object + priorities: + default: [] + description: 'Priorities is a description of user preferences to be + + used by a given ComputeClass.' + items: + description: Priority is a specification of preferred machine characteristics. + minProperties: 1 + properties: + acceleratorNetworkProfile: + description: 'AcceleratorNetworkProfile defines the type of + automated accelerator network provisioning to use. + + Possible values: + + "auto": Enables automatic ANP configuration based on the machine + type. + + "auto-": Enables automatic ANP with a custom network + profile suffix.' + type: string + x-kubernetes-validations: + - message: acceleratorNetworkProfile must be 'auto' or start + with 'auto-' + rule: self == 'auto' || self.startsWith('auto-') + capacityCheckWaitTimeSeconds: + description: CapacityCheckWaitTimeSeconds defines for how long + will this priority be attempted to scale up before moving + on to the next priority. + maximum: 86400 + minimum: 1 + type: integer + flexStart: + description: FlexStart defines Flex Start provisioning model. + properties: + enabled: + default: false + description: Enabled indicates whether Flex Start provisioning + model is enabled. + type: boolean + nodeRecycling: + description: NodeRecycling defines node recycling config. + properties: + leadTimeSeconds: + description: LeadTimeSeconds defines how much time before + node termination timestamp CA should start looking + for a replacement node. + maximum: 604800 + minimum: 1 + type: integer + required: + - leadTimeSeconds + type: object + required: + - enabled + type: object + gpu: + description: Gpu defines preferred GPU config for a node. + properties: + count: + description: Count describes preferred count of GPUs for + a node. + format: int64 + minimum: 0 + type: integer + driverVersion: + default: default + description: DriverVersion describes version of GPU driver + for a node. + enum: + - default + - latest + type: string + gpuSharing: + description: GpuSharing defines the way the nodes would + share the GPU. + properties: + gpuPartitionSize: + description: 'GpuPartitionSize is size of partitions + to create on the GPU. Valid values are + + described in the NVIDIA mig user guide. Example: "1g.5gb" + + (https://docs.nvidia.com/datacenter/tesla/mig-user-guide/#partitioning).' + type: string + maxSharedClientsPerGPU: + description: 'MaxSharedClientsPerGPU describes the max + number of containers that can + + share a physical GPU.' + format: int64 + minimum: 0 + type: integer + sharingStrategy: + description: 'SharingStrategy The type of GPU sharing + strategy to enable on the GPU node. + + Possible values: + + * TIME_SHARING - GPUs are time-shared between containers. + + * MPS - GPUs are shared between containers with NVIDIA + MPS.' + enum: + - MPS + - TIME_SHARING + type: string + type: object + type: + description: Type describes preferred GPU accelerator type + for a node. + type: string + type: object + location: + description: Location describes CCC zonal preferences config. + properties: + locationPolicy: + description: 'LocationPolicy specifies the strategy for + selecting zones when scaling up a node + + pool managed by this Compute Class. This setting controls + the distribution of new + + nodes across zones in the node pool''s region and corresponds + to the node pool + + setting of the same name. + + More info: https://cloud.google.com/sdk/gcloud/reference/container/node-pools/create#--location-policy' + enum: + - ANY + - BALANCED + type: string + zoneTypes: + description: "ZoneTypes specifies sets of zones used for\ + \ provisioning.\nSTANDARD zone type designates the core\ + \ Google Cloud zones within a region.\nAI zone type designates\ + \ specialized zones optimized for AI.\nCLUSTER_DEFAULT\ + \ zone type designate zones specified in the cluster's\ + \ autoprovisioningLocations or cluster\u2019s locations\ + \ if autoprovisioningLocations is empty." + items: + description: ZoneType is an enumeration of supported zone + types. + enum: + - STANDARD + - AI + - CLUSTER_DEFAULT + type: string + maxItems: 3 + minItems: 1 + type: array + zones: + description: Zones lists zones considered for node autoprovisioning. + items: + type: string + minItems: 1 + type: array + type: object + machineFamily: + description: 'Machine family describes preferred instance family + for a node. If none is specified, + + the default autoprovisioning machine family is used.' + maxLength: 10 + type: string + machineType: + description: MachineType defines preferred machine type for + a node. + maxLength: 100 + type: string + maxPodsPerNode: + description: MaxPodsPerNode describes the maximum number of + pods a node can accommodate. + maximum: 256 + minimum: 8 + type: integer + maxRunDurationSeconds: + description: MaxRunDurationSeconds defines the maximum duration + for the nodes to exist. If unspecified, the nodes can exist + indefinitely. + type: integer + minCores: + description: MinCores describes a minimum number of CPU cores + of a node. + minimum: 0 + type: integer + minCpuPlatform: + description: MinCpuPlatform defines the minimum CPU platform + for a node. + enum: + - Intel Sandy Bridge + - Intel Ivy Bridge + - Intel Haswell + - Intel Broadwell + - Intel Skylake + - Intel Cascade Lake + - Intel Ice Lake + - Intel Sapphire Rapids + - Intel Emerald Rapids + - Intel Granite Rapids + - AMD Rome + - AMD Milan + - AMD Genoa + - AMD Turin + - Ampere Altra + - Google Axion + - Nvidia Grace + type: string + minMemoryGb: + description: MinMemoryGb describes a minimum GBs of memory of + a node. + minimum: 0 + type: integer + nodeLabels: + additionalProperties: + type: string + description: 'NodeLabels is used to add user defined Kubernetes + labels to all nodes in the new node pool. + + These labels are applied to the Kubernetes API node object + and can be used in nodeSelectors for pod scheduling. + + Note: Node labels are distinct from GKE labels. + + More info: https://cloud.google.com/sdk/gcloud/reference/container/node-pools/create#--node-labels' + maxProperties: 100 + type: object + nodeSystemConfig: + description: NodeSystemConfig defines node system config for + a node. + properties: + kubeletConfig: + description: KubeletConfig defines kubelet config for a + node. + properties: + allowedUnsafeSysctls: + description: 'This setting defines a comma-separated + allowlist of unsafe sysctls or sysctl patterns + + (ending in `*`). The unsafe namespaced sysctl groups + are `kernel.shm*`, `kernel.msg*`, + + `kernel.sem`, `fs.mqueue.*`, and `net.*`. Leaving + this allowlist empty means they cannot be set on Pods.' + items: + maxLength: 253 + minLength: 1 + pattern: ^([a-z0-9]([-_a-z0-9]*[a-z0-9])?[./])*([a-z0-9][-_a-z0-9]*)?[a-z0-9*]$ + type: string + maxItems: 100 + type: array + containerLogMaxFiles: + description: 'This setting sets the maximum number of + container log files that can be present for a + + container. Default is 5 in OSS if unspecified.' + format: int64 + maximum: 10 + minimum: 2 + type: integer + containerLogMaxSize: + description: 'This setting sets the maximum size of + the container log file before it is rotated. + + Format: positive number + unit, Eg. 100Ki, 10Mi, 5Gi. + Valid units are Ki, + + Mi, Gi. The value must be between 10Mi and 500Mi. + And the total + + container log size (container_log_max_size * container_log_max_files) + + cannot exceed 1% of the total storage of the node. + + Default is 10Mi in OSS if unspecified.' + pattern: ^([0-9]+([.][0-9]+)?(Ki|Mi|Gi))+$ + type: string + cpuCfsQuota: + description: 'This setting enforces the Pod''s CPU limit. + Setting this value to false means that the CPU limits + for Pods are ignored. + + Ignoring CPU limits might be desirable in certain + scenarios where Pods are sensitive to CPU limits. + + The risk of disabling cpuCFSQuota is that a rogue + Pod can consume more CPU resources than intended.' + type: boolean + cpuCfsQuotaPeriod: + description: 'This setting sets the CPU CFS quota period + value, cpu.cfs_period_us, which specifies the period + of how often a cgroup''s access to CPU resources should + be reallocated. + + This option lets you tune the CPU throttling behavior. + Value must be 1ms <= period <= 1s.' + pattern: ^([1-9][0-9]*)m?s$ + type: string + cpuManagerPolicy: + description: 'This setting controls the kubelet''s CPU + Manager Policy. The default value is none which is + the default CPU affinity scheme, providing no affinity + beyond what the OS scheduler does automatically. + + Setting this value to static allows Pods in the Guaranteed + QoS class with integer CPU requests to be assigned + exclusive use of CPUs.' + enum: + - none + - static + type: string + evictionMaxPodGracePeriodSeconds: + description: 'EvictionMaxPodGracePeriodSeconds is the + maximum allowed grace period + + (in seconds) to use when terminating pods in response + to a soft eviction + + threshold being met.' + format: int64 + maximum: 300 + minimum: 0 + type: integer + evictionMinimumReclaim: + description: EvictionMinimumReclaim defines minimum + reclaims. + properties: + imagefsAvailable: + description: 'ImagefsAvailable is the minimum reclaim + for imagefs.available. + + The value must be a percentage, e.g., "5%". + + The value must be positive and less than 10%.' + pattern: ^[0-9]+(\.[0-9]+)?%$ + type: string + imagefsInodesFree: + description: 'ImagefsInodesFree is the minimum reclaim + for imagefs.inodesFree. + + The value must be a percentage, e.g., "5%". + + The value must be positive and less than 10%.' + pattern: ^[0-9]+(\.[0-9]+)?%$ + type: string + memoryAvailable: + description: 'MemoryAvailable is the minimum reclaim + for memory.available. + + The value must be a percentage, e.g., "5%". + + The value must be positive and less than 10%.' + pattern: ^[0-9]+(\.[0-9]+)?%$ + type: string + nodefsAvailable: + description: 'NodefsAvailable is the minimum reclaim + for nodefs.available. + + The value must be a percentage, e.g., "5%". + + The value must be positive and less than 10%.' + pattern: ^[0-9]+(\.[0-9]+)?%$ + type: string + nodefsInodesFree: + description: 'NodefsInodesFree is the minimum reclaim + for nodefs.inodesFree. + + The value must be a percentage, e.g., "5%". + + The value must be positive and less than 10%.' + pattern: ^[0-9]+(\.[0-9]+)?%$ + type: string + pidAvailable: + description: 'PidAvailable is the minimum reclaim + for pid.available. + + The value must be a percentage, e.g., "5%". + + The value must be positive and less than 10%.' + pattern: ^[0-9]+(\.[0-9]+)?%$ + type: string + type: object + evictionSoft: + description: EvictionSoft defines soft eviction thresholds. + properties: + imagefsAvailable: + description: 'ImagefsAvailable is the soft eviction + threshold for imagefs.available. + + The value must be a percentage. Eg. "10%". + + The value must be between 15% and 50% inclusive.' + pattern: ^[0-9]+(\.[0-9]+)?%$ + type: string + imagefsInodesFree: + description: 'ImagefsInodesFree is the soft eviction + threshold for imagefs.inodesFree. + + The value must be a percentage. Eg. "5%". + + The value must be between 5% and 50% inclusive.' + pattern: ^[0-9]+(\.[0-9]+)?%$ + type: string + memoryAvailable: + description: 'MemoryAvailable is the soft eviction + threshold for memory.available. + + The value must be a quantity, e.g., "100Mi". + + The value must be greater than the GKE default + hard eviction threshold of 100Mi and less than + 50% of machine memory.' + pattern: ^[0-9]+(\.[0-9]+)?(Ki|Mi|Gi)$ + type: string + nodefsAvailable: + description: 'NodefsAvailable is the soft eviction + threshold for nodefs.available. + + The value must be a percentage, e.g., "20%". + + The value must be between 10% and 50% inclusive.' + pattern: ^[0-9]+(\.[0-9]+)?%$ + type: string + nodefsInodesFree: + description: 'NodefsInodesFree is the soft eviction + threshold for nodefs.inodesFree. + + The value must be a percentage. Eg. "5%". + + The value must be between 5% and 50% inclusive.' + pattern: ^[0-9]+(\.[0-9]+)?%$ + type: string + pidAvailable: + description: 'PidAvailable is the soft eviction + threshold for pid.available. + + The value must be a percentage. Eg. "10%". + + The value must be between 10% and 50% inclusive.' + pattern: ^[0-9]+(\.[0-9]+)?%$ + type: string + type: object + evictionSoftGracePeriod: + description: EvictionSoftGracePeriod defines grace periods + for soft eviction thresholds. + properties: + imagefsAvailable: + description: 'ImagefsAvailable is the grace period + for the imagefs.available soft eviction threshold. + + The value must be a duration string. Eg. "30s", + "1m30s". + + The value must be positive and less than ''5m''.' + pattern: "^([0-9]+([.][0-9]+)?(ns|us|\xB5s|ms|s|m|h))+$" + type: string + imagefsInodesFree: + description: 'ImagefsInodesFree is the grace period + for the imagefs.inodesFree soft eviction threshold. + + The value must be a duration string. Eg. "30s", + "1m30s". + + The value must be positive and less than ''5m''.' + pattern: "^([0-9]+([.][0-9]+)?(ns|us|\xB5s|ms|s|m|h))+$" + type: string + memoryAvailable: + description: 'MemoryAvailable is the grace period + for the memory.available soft eviction threshold. + + The value must be a duration string. Eg. "30s", + "1m30s". + + The value must be positive and less than ''5m''.' + pattern: "^([0-9]+([.][0-9]+)?(ns|us|\xB5s|ms|s|m|h))+$" + type: string + nodefsAvailable: + description: 'NodefsAvailable is the grace period + for the nodefs.available soft eviction threshold. + + The value must be a duration string. Eg. "30s", + "1m30s". + + The value must be positive and less than ''5m''.' + pattern: "^([0-9]+([.][0-9]+)?(ns|us|\xB5s|ms|s|m|h))+$" + type: string + nodefsInodesFree: + description: 'NodefsInodesFree is the grace period + for the nodefs.inodesFree soft eviction threshold. + + The value must be a duration string. Eg. "30s", + "1m30s". + + The value must be positive and less than ''5m''.' + pattern: "^([0-9]+([.][0-9]+)?(ns|us|\xB5s|ms|s|m|h))+$" + type: string + pidAvailable: + description: 'PidAvailable is the grace period for + the pid.available soft eviction threshold. + + The value must be a duration string. Eg. "30s", + "1m30s". + + The value must be positive and less than ''5m''.' + pattern: "^([0-9]+([.][0-9]+)?(ns|us|\xB5s|ms|s|m|h))+$" + type: string + type: object + imageGcHighThresholdPercent: + description: 'This setting sets the percent of disk + usage after which image garbage collection is always + + run. The percent is calculated as this field value + out of 100. Default is 85 if unspecified.' + format: int64 + maximum: 85 + minimum: 11 + type: integer + imageGcLowThresholdPercent: + description: 'This setting sets the percent of disk + usage before which image garbage collection is never + + run. Lowest disk usage to garbage collect to. The + percent is calculated as + + this field value out of 100. Default is 80 if unspecified.' + format: int64 + maximum: 84 + minimum: 10 + type: integer + imageMaximumGcAge: + description: "This setting sets the maximum age an image\ + \ can be unused before it is garbage collected.\n\ + The string must be a decimal number with a unit suffix,\ + \ such as \"300s\", \"1.5h\", and \"2h45m\".\nValid\ + \ time units are \"ns\", \"us\" (or \"\xB5s\"), \"\ + ms\", \"s\", \"m\", \"h\".\nThe value must be a positive\ + \ duration.\nDefault is \"0s\" if unspecified, which\ + \ disables the field." + pattern: "^([0-9]+([.][0-9]+)?(ns|us|\xB5s|ms|s|m|h))+$" + type: string + imageMinimumGcAge: + description: "This setting sets the minimum age for\ + \ an unused image before it is garbage collected.\n\ + The string must be a decimal number with a unit suffix,\ + \ such as \"300s\", \"1.5h\", and \"2h45m\".\nValid\ + \ time units are \"ns\", \"us\" (or \"\xB5s\"), \"\ + ms\", \"s\", \"m\", \"h\".\nThe value must be a positive\ + \ duration and less than or equal to 2 minutes.\n\ + Default is \"2m\" if unspecified." + pattern: "^([0-9]+([.][0-9]+)?(ns|us|\xB5s|ms|s|m|h))+$" + type: string + maxParallelImagePulls: + description: This setting sets the maximum number of + image pulls in parallel. Default is 2 or 3 depending + on boot disk type. + format: int64 + maximum: 5 + minimum: 2 + type: integer + podPidsLimit: + description: This setting sets the maximum number of + process IDs (PIDs) that each Pod can use. + format: int64 + maximum: 4194304 + minimum: 1024 + type: integer + singleProcessOOMKill: + description: 'This setting sets whether to enable single + process OOM killer. + + If set to true, the processes in a container will + be OOM killed individually instead of as a group.' + type: boolean + type: object + x-kubernetes-validations: + - message: ImageGcLowThresholdPercent must be lower than + imageGcHighThresholdPercent + rule: 'has(self.imageGcHighThresholdPercent)&&has(self.imageGcLowThresholdPercent) + ? self.imageGcHighThresholdPercent>self.imageGcLowThresholdPercent + : true' + - message: ImageGcHighThresholdPercent must be higher than + 80 which is default value of imageGcLowThresholdPercent + rule: 'has(self.imageGcHighThresholdPercent)&&!has(self.imageGcLowThresholdPercent) + ? self.imageGcHighThresholdPercent>80 : true' + linuxNodeConfig: + description: LinuxNodeConfig defines linux node config for + a node. + properties: + hugepageConfig: + description: HugepagesConfig defines hugepages config + for a node. + properties: + hugepage_size1g: + description: Number of 1-gigabyte-sized huge pages + to allocate. + format: int64 + minimum: 1 + type: integer + hugepage_size2m: + description: Number of 2-megabyte-sized huge pages + to allocate. + format: int64 + minimum: 1 + type: integer + type: object + swapConfig: + description: SwapConfig specifies the swap memory configuration + for a node pool. + properties: + bootDiskProfile: + description: Use the node's boot disk for swap. + properties: + swapSizeGib: + description: The size of the swap space in GiB. + format: int64 + minimum: 1 + type: integer + swapSizePercent: + description: The size of the swap space as a + percentage of the node's boot disk. + format: int32 + maximum: 50 + minimum: 1 + type: integer + type: object + x-kubernetes-validations: + - message: only one of swapSizeGib or swapSizePercent + may be set + rule: '(has(self.swapSizeGib) ? 1 : 0) + (has(self.swapSizePercent) + ? 1 : 0) <= 1' + dedicatedLocalSsdProfile: + description: Provision a new, separate local NVMe + SSD exclusively for swap. + properties: + diskCount: + description: The number of physical local NVMe + SSD disks to attach. + format: int64 + minimum: 1 + type: integer + type: object + enabled: + description: Enables or disables swap for the node + pool. Default to false. + type: boolean + encryptionConfig: + description: If omitted, swap space is encrypted + by default. + properties: + disabled: + description: 'If true, swap space will NOT be + encrypted. + + Defaults to false, swap space is encrypted + by default.' + type: boolean + type: object + ephemeralLocalSsdProfile: + description: Use the local SSD (shared with ephemeral + storage) for swap. + properties: + swapSizeGib: + description: The size of the swap space in GiB. + format: int64 + minimum: 1 + type: integer + swapSizePercent: + description: The size of the swap space as a + percentage of the node's ephemeral storage + local SSDs. + format: int32 + maximum: 80 + minimum: 1 + type: integer + type: object + x-kubernetes-validations: + - message: only one of swapSizeGib or swapSizePercent + may be set + rule: '(has(self.swapSizeGib) ? 1 : 0) + (has(self.swapSizePercent) + ? 1 : 0) <= 1' + type: object + x-kubernetes-validations: + - message: only one of bootDiskProfile, ephemeralLocalSsdProfile, + or dedicatedLocalSsdProfile may be set + rule: '(has(self.bootDiskProfile) ? 1 : 0) + (has(self.ephemeralLocalSsdProfile) + ? 1 : 0) + (has(self.dedicatedLocalSsdProfile) ? + 1 : 0) <= 1' + sysctls: + description: SysctlsConfig defines sysctls config for + a node. + properties: + fs.aio-max-nr: + description: The maximum system-wide number of asynchronous + io requests. + format: int64 + maximum: 4194304 + minimum: 65536 + type: integer + fs.file-max: + description: Maximum number of file-handles that + the Linux kernel will allocate. + format: int64 + maximum: 67108864 + minimum: 104857 + type: integer + fs.inotify.max_user_instances: + description: The maximum number of inotify instances + that a user can create. + format: int64 + maximum: 1048576 + minimum: 8192 + type: integer + fs.inotify.max_user_watches: + description: The maximum number of inotify watches + that a user can create. + format: int64 + maximum: 1048576 + minimum: 8192 + type: integer + fs.nr_open: + description: The maximum number of file descriptors + that can be opened by a process. + format: int64 + maximum: 2147483584 + minimum: 1048576 + type: integer + kernel.keys.maxbytes: + description: Represents the maximum number of bytes + that a nonroot user can hold in the payload section + of all their keys. + format: int64 + maximum: 2097152 + minimum: 20000 + type: integer + kernel.keys.maxkeys: + description: Controls the maximum number of keys + that a nonroot user may own. + format: int64 + maximum: 1048576 + minimum: 200 + type: integer + kernel.shmall: + description: 'The maximum size (in bytes) of a single + shared memory segment allowed by the kernel. + + Note that the actual range should be integer between + 0 and 18446744073692774399, while kubebuilder + would lose some precision on uint64 during the + internal representation and parsing.' + maxLength: 20 + minLength: 1 + pattern: ^([0-9]+)$ + type: string + kernel.shmmax: + description: 'The total amount of shared memory + pages that can be used on the system at one time. + + Note that the actual range should be integer between + 0 and 18446744073692774399, while kubebuilder + would lose some precision on uint64 during the + internal representation and parsing.' + maxLength: 20 + minLength: 1 + pattern: ^([0-9]+)$ + type: string + kernel.shmmni: + description: The system-wide maximum number of shared + memory segments. + format: int64 + maximum: 32768 + minimum: 4096 + type: integer + net.core.busy_poll: + description: Low latency busy poll timeout for poll + and select. (needs CONFIG_NET_RX_BUSY_POLL) Approximate + time in us to busy loop waiting for events. + format: int64 + maximum: 2147483647 + minimum: 0 + type: integer + net.core.busy_read: + description: Low latency busy poll timeout for socket + reads. (needs CONFIG_NET_RX_BUSY_POLL) Approximate + time in us to busy loop waiting for packets on + the device queue. + format: int64 + maximum: 2147483647 + minimum: 0 + type: integer + net.core.netdev_max_backlog: + description: Maximum number of packets, queued on + the INPUT side, when the interface receives packets + faster than kernel can process them. + format: int64 + maximum: 2147483647 + minimum: 1 + type: integer + net.core.optmem_max: + description: Maximum ancillary buffer size allowed + per socket. Ancillary data is a sequence of struct + cmsghdr structures with appended data. + format: int64 + maximum: 2147483647 + minimum: 1 + type: integer + net.core.rmem_default: + description: The default receive socket buffer size + in bytes. + format: int64 + maximum: 2147483647 + minimum: 2304 + type: integer + net.core.rmem_max: + description: The maximum receive socket buffer size + in bytes. + format: int64 + maximum: 2147483647 + minimum: 2304 + type: integer + net.core.somaxconn: + description: Limit of socket listen() backlog, known + in userspace as SOMAXCONN. Defaults to 128. See + also tcp_max_syn_backlog for additional tuning + for TCP sockets. + format: int64 + maximum: 2147483647 + minimum: 128 + type: integer + net.core.wmem_default: + description: The default setting (in bytes) of the + socket send buffer. + format: int64 + maximum: 2147483647 + minimum: 4608 + type: integer + net.core.wmem_max: + description: The maximum send socket buffer size + in bytes. + format: int64 + maximum: 2147483647 + minimum: 4608 + type: integer + net.ipv4.neigh.default.gc_thresh1: + description: Tells the garbage collector the minimum + number of network entries that can sit in cache + (floor). + format: int64 + maximum: 262144 + minimum: 0 + type: integer + net.ipv4.neigh.default.gc_thresh2: + description: Acts as a soft limit to the number + of network device entries stored in cache (soft + ceiling). + format: int64 + maximum: 524288 + minimum: 512 + type: integer + net.ipv4.neigh.default.gc_thresh3: + description: Sets a hard ceiling (absolute maximum) + for the network neighbor cache. + format: int64 + maximum: 1048576 + minimum: 1024 + type: integer + net.ipv4.tcp_max_orphans: + format: int64 + maximum: 262144 + minimum: 16384 + type: integer + net.ipv4.tcp_rmem: + description: 'Minimal size of receive buffer used + by UDP sockets in moderation. Each UDP socket + is able to use the size for receiving data, even + if total pages of UDP sockets exceed udp_mem pressure. + The unit is byte. Default: 1 page. The three values + are: min, default, max. Eg. ''4096 87380 6291456''.' + type: string + net.ipv4.tcp_tw_reuse: + description: Allow to reuse TIME-WAIT sockets for + new connections when it is safe from protocol + viewpoint. It should not be changed without advice/request + of technical experts. + format: int64 + maximum: 2 + minimum: 0 + type: integer + net.ipv4.tcp_wmem: + description: 'Minimal size of send buffer used by + UDP sockets in moderation. Each UDP socket is + able to use the size for sending data, even if + total pages of UDP sockets exceed udp_mem pressure. + The unit is byte. Default: 1 page. The three values + are: min, default, max. Eg. ''4096 87380 6291456''.' + type: string + net.ipv6.conf.all.disable_ipv6: + description: Changing this value is same as changing + conf/default/disable_ipv6 setting and also all + per-interface disable_ipv6 settings to the same + value. + type: boolean + net.ipv6.conf.default.disable_ipv6: + description: Disable IPv6 operation. + type: boolean + net.netfilter.nf_conntrack_acct: + description: Whether to enable connection tracking + flow accounting. + type: boolean + net.netfilter.nf_conntrack_buckets: + description: The size of hash table for connection + tracking. + format: int64 + maximum: 524288 + minimum: 65536 + type: integer + net.netfilter.nf_conntrack_max: + description: The size of connection tracking table. + format: int64 + maximum: 4194304 + minimum: 65536 + type: integer + net.netfilter.nf_conntrack_tcp_timeout_close_wait: + description: The period for which the TCP connections + can remain in the CLOSE_WAIT state, and stay in + the table. + format: int64 + maximum: 3600 + minimum: 60 + type: integer + net.netfilter.nf_conntrack_tcp_timeout_established: + description: The duration of dead connections before + deleted automatically from connection tracking + table. + format: int64 + maximum: 86400 + minimum: 600 + type: integer + net.netfilter.nf_conntrack_tcp_timeout_time_wait: + description: The period for which the TCP connections + can remain in the TIME_WAIT state, and stay in + the table. + format: int64 + maximum: 600 + minimum: 1 + type: integer + vm.dirty_background_ratio: + description: 'Percentage of system memory that can + be filled with dirty pages (modified but not yet + written to disk) before background kernel flusher + threads begin writeback. + + This value should be less than ''vm.dirty_ratio''.' + format: int64 + maximum: 100 + minimum: 1 + type: integer + vm.dirty_expire_centisecs: + description: 'Maximum age (in hundredths of a second) + that dirty data can remain in memory before kernel + flusher threads write it to disk. + + Lower values result in faster, more frequent writebacks.' + format: int64 + maximum: 6000 + minimum: 0 + type: integer + vm.dirty_ratio: + description: 'Percentage of system memory that can + be filled with dirty pages before processes performing + writes are forced to block and write out dirty + data synchronously. + + This value should be greater than ''vm.dirty_background_ratio''.' + format: int64 + maximum: 100 + minimum: 1 + type: integer + vm.dirty_writeback_centisecs: + description: Interval (in hundredths of a second) + at which kernel flusher threads wake up to write + 'old' dirty data to disk. + format: int64 + maximum: 1000 + minimum: 0 + type: integer + vm.max_map_count: + description: Maximum number of memory map areas + a process may have. + format: int64 + maximum: 2147483647 + minimum: 65536 + type: integer + vm.min_free_kbytes: + format: int64 + maximum: 1048576 + minimum: 67584 + type: integer + vm.overcommit_memory: + description: 'Determines the kernel''s memory overcommit + handling strategy. + + Supported values: + + 0: Rejects allocations that are obviously too + large. + + 1: Allows overcommit until memory is exhausted. + + 2 (strict): Prevents overcommit beyond swap space + plus a percentage of RAM defined by ''vm.overcommit_ratio''.' + enum: + - 0 + - 1 + - 2 + format: int64 + type: integer + vm.overcommit_ratio: + description: 'Specifies the percentage of physical + RAM allowed for overcommit when ''vm.overcommit_memory'' + is set to 2. + + The total committed address space cannot exceed + swap plus this RAM percentage.' + format: int64 + maximum: 100 + minimum: 0 + type: integer + vm.swappiness: + format: int64 + maximum: 200 + minimum: 0 + type: integer + vm.vfs_cache_pressure: + description: Adjusts the kernel's preference for + reclaiming memory used for dentry (directory) + and inode caches. + format: int64 + maximum: 100 + minimum: 0 + type: integer + vm.watermark_scale_factor: + format: int64 + maximum: 3000 + minimum: 10 + type: integer + type: object + transparentHugepageDefrag: + description: 'Defines the transparent hugepage defrag + configuration on the node. Currently supported values: + + * TRANSPARENT_HUGEPAGE_DEFRAG_ALWAYS: An application + requesting THP will stall on allocation failure and + directly reclaim pages and compact memory in an effort + to allocate a THP immediately. + + * TRANSPARENT_HUGEPAGE_DEFRAG_DEFER: An application + will wake kswapd in the background to reclaim pages + and wake kcompactd to compact memory so that THP is + available in the near future. It is the responsibility + of khugepaged to then install the THP pages later. + + * TRANSPARENT_HUGEPAGE_DEFRAG_DEFER_WITH_MADVISE: + An application will enter direct reclaim and compaction + like always, but only for regions that have used madvise(MADV_HUGEPAGE); + all other regions will wake kswapd in the background + to reclaim pages and wake kcompactd to compact memory + so that THP is available in the near future. + + * TRANSPARENT_HUGEPAGE_DEFRAG_MADVISE: An application + will enter direct reclaim and compaction like always, + but only for regions that have used madvise(MADV_HUGEPAGE); + all other regions will wake kswapd in the background + to reclaim pages and wake kcompactd to compact memory + so that THP is available in the near future. + + * TRANSPARENT_HUGEPAGE_DEFRAG_NEVER: An application + will never enter direct reclaim or compaction. + + * TRANSPARENT_HUGEPAGE_DEFRAG_UNSPECIFIED: Default + value. GKE will not modify the kernel configuration.' + enum: + - TRANSPARENT_HUGEPAGE_DEFRAG_ALWAYS + - TRANSPARENT_HUGEPAGE_DEFRAG_DEFER + - TRANSPARENT_HUGEPAGE_DEFRAG_DEFER_WITH_MADVISE + - TRANSPARENT_HUGEPAGE_DEFRAG_MADVISE + - TRANSPARENT_HUGEPAGE_DEFRAG_NEVER + - TRANSPARENT_HUGEPAGE_DEFRAG_UNSPECIFIED + type: string + transparentHugepageEnabled: + description: 'Controls transparent hugepage support + for anonymous memory. Currently supported values: + + * TRANSPARENT_HUGEPAGE_ENABLED_ALWAYS: Transparent + hugepage is enabled system wide. + + * TRANSPARENT_HUGEPAGE_ENABLED_MADVISE: Transparent + hugepage is enabled inside MADV_HUGEPAGE regions. + This is the default kernel configuration. + + * TRANSPARENT_HUGEPAGE_ENABLED_NEVER: Transparent + hugepage is disabled. + + * TRANSPARENT_HUGEPAGE_ENABLED_UNSPECIFIED: Default + value. GKE will not modify the kernel configuration.' + enum: + - TRANSPARENT_HUGEPAGE_ENABLED_ALWAYS + - TRANSPARENT_HUGEPAGE_ENABLED_MADVISE + - TRANSPARENT_HUGEPAGE_ENABLED_NEVER + - TRANSPARENT_HUGEPAGE_ENABLED_UNSPECIFIED + type: string + type: object + type: object + nodepools: + description: Nodepools describes preference of specific, preexisting + nodepools. + items: + type: string + type: array + placement: + description: Placement defines resource policy used for BYOPP + and BYOWP + properties: + policyName: + description: PolicyName defines the name of the resource + policy, e.g. my-resource-policy + minLength: 1 + type: string + required: + - policyName + type: object + podFamily: + description: PodFamily represents pod-based provisioning and + billing config. + enum: + - general-purpose + - general-purpose-arm + type: string + priorityScore: + description: 'A higher value is treated as a higher priority. + + Priorities with the same priorityScore value are treated equally. + + Not more than 3 priorities can have the same priorityScore.' + maximum: 1000 + minimum: 1 + type: integer + reservations: + description: Reservations defines reservations config for a + node. + properties: + affinity: + description: 'ReservationAffinity affects reservations considered + and the way how they are consumed. + + "Specific" means that only specific reservations are considered + with no fallback possible. + + "AnyBestEffort" affinity would consider any non-specific + reservation available + + to be claimed with a fallback to on-demand nodes in case + of none claimable. + + "None" affinity would prevent reservations from being + used' + enum: + - Specific + - AnyBestEffort + - None + type: string + specific: + description: Specific is a non prioritized list of specific + reservations to be considered by the priority rule. + items: + description: SpecificReservation defines a single specific + reservation to be consumed by the created node. + properties: + name: + description: Name of the reservation to be used. + type: string + project: + description: Project is the project where the specific + reservation lives. + type: string + reservationBlock: + description: ReservationBlock is the block of the + reservation. + properties: + name: + description: Name is the name of the block. + type: string + reservationSubBlock: + description: ReservationSubBlock is the subBlock + of the reservation block. + properties: + name: + description: Name is the name of the subBlock. + type: string + required: + - name + type: object + required: + - name + type: object + zones: + description: Zones is a list of GCE zones where reservations + are to be consumed. + items: + type: string + minItems: 1 + type: array + required: + - name + type: object + minItems: 0 + type: array + required: + - affinity + type: object + x-kubernetes-validations: + - message: Unable to set specific reservations for non specific + affinity + rule: 'has(self.specific) && self.specific.size() > 0 ? self.affinity + == "Specific" : true' + - message: At least 1 specific reservation required for specific + affinity + rule: 'self.affinity == "Specific" ? has(self.specific) && + self.specific.size() > 0 : true' + spot: + description: Spot if set to true specifies that a node should + be a spot instance, on-demand otherwise. + type: boolean + storage: + description: Storage describes storage config of a node. + properties: + bootDiskKMSKey: + description: BootDiskKMSKey defines a key used to encrypt + the boot disk attached. + pattern: projects/[^/]+/locations/[^/]+/keyRings/[^/]+/cryptoKeys/[^/]+ + type: string + bootDiskSize: + description: BootDiskSize defines the size of a disk attached + to node, specified in GB. + minimum: 10 + type: integer + bootDiskType: + description: 'BootDiskType defines type of the disk attached + to the node. + + Note that available boot disk types depend on the machine + family / machine type selected. + + Currently supported types: + + * pd-balanced + + * pd-standard + + * pd-ssd + + * hyperdisk-balanced' + enum: + - pd-balanced + - pd-standard + - pd-ssd + - hyperdisk-balanced + type: string + localSSDCount: + description: LocalSSDCount defines a number of local SSDs + attached to node. + minimum: 1 + type: integer + secondaryBootDisks: + description: SecondaryBootDisks represent persistent disks + attached to a node with special configurations based on + their modes. + items: + description: SecondaryBootDisk represents a persistent + disk attached to a node with special configurations + based on its mode. + properties: + diskImageName: + description: The name of the disk image. + type: string + mode: + description: 'Currently supported modes: + + * MODE_UNSPECIFIED - MODE_UNSPECIFIED is when mode + is not set. + + * CONTAINER_IMAGE_CACHE - it is for using the secondary + boot disk as a container image cache.' + enum: + - MODE_UNSPECIFIED + - CONTAINER_IMAGE_CACHE + type: string + project: + description: The name of the project that the disk + image belongs to. + type: string + required: + - diskImageName + type: object + type: array + type: object + taints: + description: 'Taints is used to add user defined Kubernetes + taints to all nodes in the new node pool. + + These taints are applied to the Kubernetes API node object + and can be used in tolerations for pod scheduling.' + items: + description: 'TaintConfig applies the given kubernetes taints + on all nodes in the new node pool, which can be used with + tolerations for pod scheduling. + + Any workload that does not tolerate the taints specified + in this object will not be scheduled to the node pool. + + More info: https://cloud.google.com/sdk/gcloud/reference/container/node-pools/create#--node-taints' + properties: + effect: + description: 'It defines the taint''s effect on pods that + does not have the necessary toleration. + + The following values are supported: NoSchedule, PreferNoSchedule, + and NoExecute.' + enum: + - NoSchedule + - PreferNoSchedule + - NoExecute + type: string + key: + description: Node taint key. The key must conform to syntax + described in https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#syntax-and-character-set. + maxLength: 320 + type: string + value: + description: The value that matches the specified taint + key. + maxLength: 63 + pattern: ^([a-z0-9][-A-Za-z0-9_.]{1,61})?[A-Za-z0-9]$ + type: string + required: + - effect + - key + type: object + maxItems: 100 + type: array + tpu: + description: Tpu defines preferred TPU config for a node. + properties: + count: + description: Count describes preferred count of TPU chips + for a node. + format: int64 + type: integer + topology: + description: Topology describes preferred TPU topology of + a node. + type: string + type: + description: Type describes preferred TPU type for a node. + type: string + type: object + type: object + x-kubernetes-validations: + - message: Nodepool field cannot be set along with other fields + rule: 'has(self.nodepools) ? (size(dyn(self)) == 1) : true' + - message: MachineFamily and MachineType cannot be set together + rule: '!(has(self.machineFamily) && has(self.machineType))' + - message: MachineType cannot be set together with MinCores/MinMemoryGb + rule: '!(has(self.machineType) && (has(self.minCores) || has(self.minMemoryGb)))' + - message: MachineFamily cannot be equal to 'ek' + rule: '!(has(self.machineFamily) && self.machineFamily == ''ek'')' + - message: MachineType cannot start with 'ek' prefix + rule: '!(has(self.machineType) && self.machineType.startsWith(''ek''))' + - message: MachineFamily cannot be equal to 'e4a' + rule: '!(has(self.machineFamily) && self.machineFamily == ''e4a'')' + - message: MachineType cannot start with 'e4a' prefix + rule: '!(has(self.machineType) && self.machineType.startsWith(''e4a''))' + - message: Flex Start provisioning model is incompatible with Spot + rule: '!(has(self.flexStart) && has(self.spot) && self.spot == + true && self.flexStart.enabled == true)' + - message: capacityCheckWaitTimeSeconds is only supported for Flex + Start and for multi-host TPUs + rule: '!has(self.capacityCheckWaitTimeSeconds) || has(self.tpu) + || (has(self.flexStart) && self.flexStart.enabled)' + maxItems: 1000 + minItems: 0 + type: array + priorityDefaults: + description: 'PriorityDefaults define the default rules for all priorities + if the rule doesn''t exist in some priority. + + Note: PriorityDefaults doesn''t apply to priorities with only Nodepools.' + properties: + location: + description: Location describes CCC zonal preferences config. + properties: + locationPolicy: + description: 'LocationPolicy specifies the strategy for selecting + zones when scaling up a node + + pool managed by this Compute Class. This setting controls + the distribution of new + + nodes across zones in the node pool''s region and corresponds + to the node pool + + setting of the same name. + + More info: https://cloud.google.com/sdk/gcloud/reference/container/node-pools/create#--location-policy' + enum: + - ANY + - BALANCED + type: string + zoneTypes: + description: "ZoneTypes specifies sets of zones used for provisioning.\n\ + STANDARD zone type designates the core Google Cloud zones\ + \ within a region.\nAI zone type designates specialized\ + \ zones optimized for AI.\nCLUSTER_DEFAULT zone type designate\ + \ zones specified in the cluster's autoprovisioningLocations\ + \ or cluster\u2019s locations if autoprovisioningLocations\ + \ is empty." + items: + description: ZoneType is an enumeration of supported zone + types. + enum: + - STANDARD + - AI + - CLUSTER_DEFAULT + type: string + maxItems: 3 + minItems: 1 + type: array + zones: + description: Zones lists zones considered for node autoprovisioning. + items: + type: string + minItems: 1 + type: array + type: object + nodeSystemConfig: + description: NodeSystemConfig defines node system config for a + node. + properties: + kubeletConfig: + description: KubeletConfig defines kubelet config for a node. + properties: + allowedUnsafeSysctls: + description: 'This setting defines a comma-separated allowlist + of unsafe sysctls or sysctl patterns + + (ending in `*`). The unsafe namespaced sysctl groups + are `kernel.shm*`, `kernel.msg*`, + + `kernel.sem`, `fs.mqueue.*`, and `net.*`. Leaving this + allowlist empty means they cannot be set on Pods.' + items: + maxLength: 253 + minLength: 1 + pattern: ^([a-z0-9]([-_a-z0-9]*[a-z0-9])?[./])*([a-z0-9][-_a-z0-9]*)?[a-z0-9*]$ + type: string + maxItems: 100 + type: array + containerLogMaxFiles: + description: 'This setting sets the maximum number of + container log files that can be present for a + + container. Default is 5 in OSS if unspecified.' + format: int64 + maximum: 10 + minimum: 2 + type: integer + containerLogMaxSize: + description: 'This setting sets the maximum size of the + container log file before it is rotated. + + Format: positive number + unit, Eg. 100Ki, 10Mi, 5Gi. + Valid units are Ki, + + Mi, Gi. The value must be between 10Mi and 500Mi. And + the total + + container log size (container_log_max_size * container_log_max_files) + + cannot exceed 1% of the total storage of the node. + + Default is 10Mi in OSS if unspecified.' + pattern: ^([0-9]+([.][0-9]+)?(Ki|Mi|Gi))+$ + type: string + cpuCfsQuota: + description: 'This setting enforces the Pod''s CPU limit. + Setting this value to false means that the CPU limits + for Pods are ignored. + + Ignoring CPU limits might be desirable in certain scenarios + where Pods are sensitive to CPU limits. + + The risk of disabling cpuCFSQuota is that a rogue Pod + can consume more CPU resources than intended.' + type: boolean + cpuCfsQuotaPeriod: + description: 'This setting sets the CPU CFS quota period + value, cpu.cfs_period_us, which specifies the period + of how often a cgroup''s access to CPU resources should + be reallocated. + + This option lets you tune the CPU throttling behavior. + Value must be 1ms <= period <= 1s.' + pattern: ^([1-9][0-9]*)m?s$ + type: string + cpuManagerPolicy: + description: 'This setting controls the kubelet''s CPU + Manager Policy. The default value is none which is the + default CPU affinity scheme, providing no affinity beyond + what the OS scheduler does automatically. + + Setting this value to static allows Pods in the Guaranteed + QoS class with integer CPU requests to be assigned exclusive + use of CPUs.' + enum: + - none + - static + type: string + evictionMaxPodGracePeriodSeconds: + description: 'EvictionMaxPodGracePeriodSeconds is the + maximum allowed grace period + + (in seconds) to use when terminating pods in response + to a soft eviction + + threshold being met.' + format: int64 + maximum: 300 + minimum: 0 + type: integer + evictionMinimumReclaim: + description: EvictionMinimumReclaim defines minimum reclaims. + properties: + imagefsAvailable: + description: 'ImagefsAvailable is the minimum reclaim + for imagefs.available. + + The value must be a percentage, e.g., "5%". + + The value must be positive and less than 10%.' + pattern: ^[0-9]+(\.[0-9]+)?%$ + type: string + imagefsInodesFree: + description: 'ImagefsInodesFree is the minimum reclaim + for imagefs.inodesFree. + + The value must be a percentage, e.g., "5%". + + The value must be positive and less than 10%.' + pattern: ^[0-9]+(\.[0-9]+)?%$ + type: string + memoryAvailable: + description: 'MemoryAvailable is the minimum reclaim + for memory.available. + + The value must be a percentage, e.g., "5%". + + The value must be positive and less than 10%.' + pattern: ^[0-9]+(\.[0-9]+)?%$ + type: string + nodefsAvailable: + description: 'NodefsAvailable is the minimum reclaim + for nodefs.available. + + The value must be a percentage, e.g., "5%". + + The value must be positive and less than 10%.' + pattern: ^[0-9]+(\.[0-9]+)?%$ + type: string + nodefsInodesFree: + description: 'NodefsInodesFree is the minimum reclaim + for nodefs.inodesFree. + + The value must be a percentage, e.g., "5%". + + The value must be positive and less than 10%.' + pattern: ^[0-9]+(\.[0-9]+)?%$ + type: string + pidAvailable: + description: 'PidAvailable is the minimum reclaim + for pid.available. + + The value must be a percentage, e.g., "5%". + + The value must be positive and less than 10%.' + pattern: ^[0-9]+(\.[0-9]+)?%$ + type: string + type: object + evictionSoft: + description: EvictionSoft defines soft eviction thresholds. + properties: + imagefsAvailable: + description: 'ImagefsAvailable is the soft eviction + threshold for imagefs.available. + + The value must be a percentage. Eg. "10%". + + The value must be between 15% and 50% inclusive.' + pattern: ^[0-9]+(\.[0-9]+)?%$ + type: string + imagefsInodesFree: + description: 'ImagefsInodesFree is the soft eviction + threshold for imagefs.inodesFree. + + The value must be a percentage. Eg. "5%". + + The value must be between 5% and 50% inclusive.' + pattern: ^[0-9]+(\.[0-9]+)?%$ + type: string + memoryAvailable: + description: 'MemoryAvailable is the soft eviction + threshold for memory.available. + + The value must be a quantity, e.g., "100Mi". + + The value must be greater than the GKE default hard + eviction threshold of 100Mi and less than 50% of + machine memory.' + pattern: ^[0-9]+(\.[0-9]+)?(Ki|Mi|Gi)$ + type: string + nodefsAvailable: + description: 'NodefsAvailable is the soft eviction + threshold for nodefs.available. + + The value must be a percentage, e.g., "20%". + + The value must be between 10% and 50% inclusive.' + pattern: ^[0-9]+(\.[0-9]+)?%$ + type: string + nodefsInodesFree: + description: 'NodefsInodesFree is the soft eviction + threshold for nodefs.inodesFree. + + The value must be a percentage. Eg. "5%". + + The value must be between 5% and 50% inclusive.' + pattern: ^[0-9]+(\.[0-9]+)?%$ + type: string + pidAvailable: + description: 'PidAvailable is the soft eviction threshold + for pid.available. + + The value must be a percentage. Eg. "10%". + + The value must be between 10% and 50% inclusive.' + pattern: ^[0-9]+(\.[0-9]+)?%$ + type: string + type: object + evictionSoftGracePeriod: + description: EvictionSoftGracePeriod defines grace periods + for soft eviction thresholds. + properties: + imagefsAvailable: + description: 'ImagefsAvailable is the grace period + for the imagefs.available soft eviction threshold. + + The value must be a duration string. Eg. "30s", + "1m30s". + + The value must be positive and less than ''5m''.' + pattern: "^([0-9]+([.][0-9]+)?(ns|us|\xB5s|ms|s|m|h))+$" + type: string + imagefsInodesFree: + description: 'ImagefsInodesFree is the grace period + for the imagefs.inodesFree soft eviction threshold. + + The value must be a duration string. Eg. "30s", + "1m30s". + + The value must be positive and less than ''5m''.' + pattern: "^([0-9]+([.][0-9]+)?(ns|us|\xB5s|ms|s|m|h))+$" + type: string + memoryAvailable: + description: 'MemoryAvailable is the grace period + for the memory.available soft eviction threshold. + + The value must be a duration string. Eg. "30s", + "1m30s". + + The value must be positive and less than ''5m''.' + pattern: "^([0-9]+([.][0-9]+)?(ns|us|\xB5s|ms|s|m|h))+$" + type: string + nodefsAvailable: + description: 'NodefsAvailable is the grace period + for the nodefs.available soft eviction threshold. + + The value must be a duration string. Eg. "30s", + "1m30s". + + The value must be positive and less than ''5m''.' + pattern: "^([0-9]+([.][0-9]+)?(ns|us|\xB5s|ms|s|m|h))+$" + type: string + nodefsInodesFree: + description: 'NodefsInodesFree is the grace period + for the nodefs.inodesFree soft eviction threshold. + + The value must be a duration string. Eg. "30s", + "1m30s". + + The value must be positive and less than ''5m''.' + pattern: "^([0-9]+([.][0-9]+)?(ns|us|\xB5s|ms|s|m|h))+$" + type: string + pidAvailable: + description: 'PidAvailable is the grace period for + the pid.available soft eviction threshold. + + The value must be a duration string. Eg. "30s", + "1m30s". + + The value must be positive and less than ''5m''.' + pattern: "^([0-9]+([.][0-9]+)?(ns|us|\xB5s|ms|s|m|h))+$" + type: string + type: object + imageGcHighThresholdPercent: + description: 'This setting sets the percent of disk usage + after which image garbage collection is always + + run. The percent is calculated as this field value out + of 100. Default is 85 if unspecified.' + format: int64 + maximum: 85 + minimum: 11 + type: integer + imageGcLowThresholdPercent: + description: 'This setting sets the percent of disk usage + before which image garbage collection is never + + run. Lowest disk usage to garbage collect to. The percent + is calculated as + + this field value out of 100. Default is 80 if unspecified.' + format: int64 + maximum: 84 + minimum: 10 + type: integer + imageMaximumGcAge: + description: "This setting sets the maximum age an image\ + \ can be unused before it is garbage collected.\nThe\ + \ string must be a decimal number with a unit suffix,\ + \ such as \"300s\", \"1.5h\", and \"2h45m\".\nValid\ + \ time units are \"ns\", \"us\" (or \"\xB5s\"), \"ms\"\ + , \"s\", \"m\", \"h\".\nThe value must be a positive\ + \ duration.\nDefault is \"0s\" if unspecified, which\ + \ disables the field." + pattern: "^([0-9]+([.][0-9]+)?(ns|us|\xB5s|ms|s|m|h))+$" + type: string + imageMinimumGcAge: + description: "This setting sets the minimum age for an\ + \ unused image before it is garbage collected.\nThe\ + \ string must be a decimal number with a unit suffix,\ + \ such as \"300s\", \"1.5h\", and \"2h45m\".\nValid\ + \ time units are \"ns\", \"us\" (or \"\xB5s\"), \"ms\"\ + , \"s\", \"m\", \"h\".\nThe value must be a positive\ + \ duration and less than or equal to 2 minutes.\nDefault\ + \ is \"2m\" if unspecified." + pattern: "^([0-9]+([.][0-9]+)?(ns|us|\xB5s|ms|s|m|h))+$" + type: string + maxParallelImagePulls: + description: This setting sets the maximum number of image + pulls in parallel. Default is 2 or 3 depending on boot + disk type. + format: int64 + maximum: 5 + minimum: 2 + type: integer + podPidsLimit: + description: This setting sets the maximum number of process + IDs (PIDs) that each Pod can use. + format: int64 + maximum: 4194304 + minimum: 1024 + type: integer + singleProcessOOMKill: + description: 'This setting sets whether to enable single + process OOM killer. + + If set to true, the processes in a container will be + OOM killed individually instead of as a group.' + type: boolean + type: object + x-kubernetes-validations: + - message: ImageGcLowThresholdPercent must be lower than imageGcHighThresholdPercent + rule: 'has(self.imageGcHighThresholdPercent)&&has(self.imageGcLowThresholdPercent) + ? self.imageGcHighThresholdPercent>self.imageGcLowThresholdPercent + : true' + - message: ImageGcHighThresholdPercent must be higher than + 80 which is default value of imageGcLowThresholdPercent + rule: 'has(self.imageGcHighThresholdPercent)&&!has(self.imageGcLowThresholdPercent) + ? self.imageGcHighThresholdPercent>80 : true' + linuxNodeConfig: + description: LinuxNodeConfig defines linux node config for + a node. + properties: + hugepageConfig: + description: HugepagesConfig defines hugepages config + for a node. + properties: + hugepage_size1g: + description: Number of 1-gigabyte-sized huge pages + to allocate. + format: int64 + minimum: 1 + type: integer + hugepage_size2m: + description: Number of 2-megabyte-sized huge pages + to allocate. + format: int64 + minimum: 1 + type: integer + type: object + swapConfig: + description: SwapConfig specifies the swap memory configuration + for a node pool. + properties: + bootDiskProfile: + description: Use the node's boot disk for swap. + properties: + swapSizeGib: + description: The size of the swap space in GiB. + format: int64 + minimum: 1 + type: integer + swapSizePercent: + description: The size of the swap space as a percentage + of the node's boot disk. + format: int32 + maximum: 50 + minimum: 1 + type: integer + type: object + x-kubernetes-validations: + - message: only one of swapSizeGib or swapSizePercent + may be set + rule: '(has(self.swapSizeGib) ? 1 : 0) + (has(self.swapSizePercent) + ? 1 : 0) <= 1' + dedicatedLocalSsdProfile: + description: Provision a new, separate local NVMe + SSD exclusively for swap. + properties: + diskCount: + description: The number of physical local NVMe + SSD disks to attach. + format: int64 + minimum: 1 + type: integer + type: object + enabled: + description: Enables or disables swap for the node + pool. Default to false. + type: boolean + encryptionConfig: + description: If omitted, swap space is encrypted by + default. + properties: + disabled: + description: 'If true, swap space will NOT be + encrypted. + + Defaults to false, swap space is encrypted by + default.' + type: boolean + type: object + ephemeralLocalSsdProfile: + description: Use the local SSD (shared with ephemeral + storage) for swap. + properties: + swapSizeGib: + description: The size of the swap space in GiB. + format: int64 + minimum: 1 + type: integer + swapSizePercent: + description: The size of the swap space as a percentage + of the node's ephemeral storage local SSDs. + format: int32 + maximum: 80 + minimum: 1 + type: integer + type: object + x-kubernetes-validations: + - message: only one of swapSizeGib or swapSizePercent + may be set + rule: '(has(self.swapSizeGib) ? 1 : 0) + (has(self.swapSizePercent) + ? 1 : 0) <= 1' + type: object + x-kubernetes-validations: + - message: only one of bootDiskProfile, ephemeralLocalSsdProfile, + or dedicatedLocalSsdProfile may be set + rule: '(has(self.bootDiskProfile) ? 1 : 0) + (has(self.ephemeralLocalSsdProfile) + ? 1 : 0) + (has(self.dedicatedLocalSsdProfile) ? 1 + : 0) <= 1' + sysctls: + description: SysctlsConfig defines sysctls config for + a node. + properties: + fs.aio-max-nr: + description: The maximum system-wide number of asynchronous + io requests. + format: int64 + maximum: 4194304 + minimum: 65536 + type: integer + fs.file-max: + description: Maximum number of file-handles that the + Linux kernel will allocate. + format: int64 + maximum: 67108864 + minimum: 104857 + type: integer + fs.inotify.max_user_instances: + description: The maximum number of inotify instances + that a user can create. + format: int64 + maximum: 1048576 + minimum: 8192 + type: integer + fs.inotify.max_user_watches: + description: The maximum number of inotify watches + that a user can create. + format: int64 + maximum: 1048576 + minimum: 8192 + type: integer + fs.nr_open: + description: The maximum number of file descriptors + that can be opened by a process. + format: int64 + maximum: 2147483584 + minimum: 1048576 + type: integer + kernel.keys.maxbytes: + description: Represents the maximum number of bytes + that a nonroot user can hold in the payload section + of all their keys. + format: int64 + maximum: 2097152 + minimum: 20000 + type: integer + kernel.keys.maxkeys: + description: Controls the maximum number of keys that + a nonroot user may own. + format: int64 + maximum: 1048576 + minimum: 200 + type: integer + kernel.shmall: + description: 'The maximum size (in bytes) of a single + shared memory segment allowed by the kernel. + + Note that the actual range should be integer between + 0 and 18446744073692774399, while kubebuilder would + lose some precision on uint64 during the internal + representation and parsing.' + maxLength: 20 + minLength: 1 + pattern: ^([0-9]+)$ + type: string + kernel.shmmax: + description: 'The total amount of shared memory pages + that can be used on the system at one time. + + Note that the actual range should be integer between + 0 and 18446744073692774399, while kubebuilder would + lose some precision on uint64 during the internal + representation and parsing.' + maxLength: 20 + minLength: 1 + pattern: ^([0-9]+)$ + type: string + kernel.shmmni: + description: The system-wide maximum number of shared + memory segments. + format: int64 + maximum: 32768 + minimum: 4096 + type: integer + net.core.busy_poll: + description: Low latency busy poll timeout for poll + and select. (needs CONFIG_NET_RX_BUSY_POLL) Approximate + time in us to busy loop waiting for events. + format: int64 + maximum: 2147483647 + minimum: 0 + type: integer + net.core.busy_read: + description: Low latency busy poll timeout for socket + reads. (needs CONFIG_NET_RX_BUSY_POLL) Approximate + time in us to busy loop waiting for packets on the + device queue. + format: int64 + maximum: 2147483647 + minimum: 0 + type: integer + net.core.netdev_max_backlog: + description: Maximum number of packets, queued on + the INPUT side, when the interface receives packets + faster than kernel can process them. + format: int64 + maximum: 2147483647 + minimum: 1 + type: integer + net.core.optmem_max: + description: Maximum ancillary buffer size allowed + per socket. Ancillary data is a sequence of struct + cmsghdr structures with appended data. + format: int64 + maximum: 2147483647 + minimum: 1 + type: integer + net.core.rmem_default: + description: The default receive socket buffer size + in bytes. + format: int64 + maximum: 2147483647 + minimum: 2304 + type: integer + net.core.rmem_max: + description: The maximum receive socket buffer size + in bytes. + format: int64 + maximum: 2147483647 + minimum: 2304 + type: integer + net.core.somaxconn: + description: Limit of socket listen() backlog, known + in userspace as SOMAXCONN. Defaults to 128. See + also tcp_max_syn_backlog for additional tuning for + TCP sockets. + format: int64 + maximum: 2147483647 + minimum: 128 + type: integer + net.core.wmem_default: + description: The default setting (in bytes) of the + socket send buffer. + format: int64 + maximum: 2147483647 + minimum: 4608 + type: integer + net.core.wmem_max: + description: The maximum send socket buffer size in + bytes. + format: int64 + maximum: 2147483647 + minimum: 4608 + type: integer + net.ipv4.neigh.default.gc_thresh1: + description: Tells the garbage collector the minimum + number of network entries that can sit in cache + (floor). + format: int64 + maximum: 262144 + minimum: 0 + type: integer + net.ipv4.neigh.default.gc_thresh2: + description: Acts as a soft limit to the number of + network device entries stored in cache (soft ceiling). + format: int64 + maximum: 524288 + minimum: 512 + type: integer + net.ipv4.neigh.default.gc_thresh3: + description: Sets a hard ceiling (absolute maximum) + for the network neighbor cache. + format: int64 + maximum: 1048576 + minimum: 1024 + type: integer + net.ipv4.tcp_max_orphans: + format: int64 + maximum: 262144 + minimum: 16384 + type: integer + net.ipv4.tcp_rmem: + description: 'Minimal size of receive buffer used + by UDP sockets in moderation. Each UDP socket is + able to use the size for receiving data, even if + total pages of UDP sockets exceed udp_mem pressure. + The unit is byte. Default: 1 page. The three values + are: min, default, max. Eg. ''4096 87380 6291456''.' + type: string + net.ipv4.tcp_tw_reuse: + description: Allow to reuse TIME-WAIT sockets for + new connections when it is safe from protocol viewpoint. + It should not be changed without advice/request + of technical experts. + format: int64 + maximum: 2 + minimum: 0 + type: integer + net.ipv4.tcp_wmem: + description: 'Minimal size of send buffer used by + UDP sockets in moderation. Each UDP socket is able + to use the size for sending data, even if total + pages of UDP sockets exceed udp_mem pressure. The + unit is byte. Default: 1 page. The three values + are: min, default, max. Eg. ''4096 87380 6291456''.' + type: string + net.ipv6.conf.all.disable_ipv6: + description: Changing this value is same as changing + conf/default/disable_ipv6 setting and also all per-interface + disable_ipv6 settings to the same value. + type: boolean + net.ipv6.conf.default.disable_ipv6: + description: Disable IPv6 operation. + type: boolean + net.netfilter.nf_conntrack_acct: + description: Whether to enable connection tracking + flow accounting. + type: boolean + net.netfilter.nf_conntrack_buckets: + description: The size of hash table for connection + tracking. + format: int64 + maximum: 524288 + minimum: 65536 + type: integer + net.netfilter.nf_conntrack_max: + description: The size of connection tracking table. + format: int64 + maximum: 4194304 + minimum: 65536 + type: integer + net.netfilter.nf_conntrack_tcp_timeout_close_wait: + description: The period for which the TCP connections + can remain in the CLOSE_WAIT state, and stay in + the table. + format: int64 + maximum: 3600 + minimum: 60 + type: integer + net.netfilter.nf_conntrack_tcp_timeout_established: + description: The duration of dead connections before + deleted automatically from connection tracking table. + format: int64 + maximum: 86400 + minimum: 600 + type: integer + net.netfilter.nf_conntrack_tcp_timeout_time_wait: + description: The period for which the TCP connections + can remain in the TIME_WAIT state, and stay in the + table. + format: int64 + maximum: 600 + minimum: 1 + type: integer + vm.dirty_background_ratio: + description: 'Percentage of system memory that can + be filled with dirty pages (modified but not yet + written to disk) before background kernel flusher + threads begin writeback. + + This value should be less than ''vm.dirty_ratio''.' + format: int64 + maximum: 100 + minimum: 1 + type: integer + vm.dirty_expire_centisecs: + description: 'Maximum age (in hundredths of a second) + that dirty data can remain in memory before kernel + flusher threads write it to disk. + + Lower values result in faster, more frequent writebacks.' + format: int64 + maximum: 6000 + minimum: 0 + type: integer + vm.dirty_ratio: + description: 'Percentage of system memory that can + be filled with dirty pages before processes performing + writes are forced to block and write out dirty data + synchronously. + + This value should be greater than ''vm.dirty_background_ratio''.' + format: int64 + maximum: 100 + minimum: 1 + type: integer + vm.dirty_writeback_centisecs: + description: Interval (in hundredths of a second) + at which kernel flusher threads wake up to write + 'old' dirty data to disk. + format: int64 + maximum: 1000 + minimum: 0 + type: integer + vm.max_map_count: + description: Maximum number of memory map areas a + process may have. + format: int64 + maximum: 2147483647 + minimum: 65536 + type: integer + vm.min_free_kbytes: + format: int64 + maximum: 1048576 + minimum: 67584 + type: integer + vm.overcommit_memory: + description: 'Determines the kernel''s memory overcommit + handling strategy. + + Supported values: + + 0: Rejects allocations that are obviously too large. + + 1: Allows overcommit until memory is exhausted. + + 2 (strict): Prevents overcommit beyond swap space + plus a percentage of RAM defined by ''vm.overcommit_ratio''.' + enum: + - 0 + - 1 + - 2 + format: int64 + type: integer + vm.overcommit_ratio: + description: 'Specifies the percentage of physical + RAM allowed for overcommit when ''vm.overcommit_memory'' + is set to 2. + + The total committed address space cannot exceed + swap plus this RAM percentage.' + format: int64 + maximum: 100 + minimum: 0 + type: integer + vm.swappiness: + format: int64 + maximum: 200 + minimum: 0 + type: integer + vm.vfs_cache_pressure: + description: Adjusts the kernel's preference for reclaiming + memory used for dentry (directory) and inode caches. + format: int64 + maximum: 100 + minimum: 0 + type: integer + vm.watermark_scale_factor: + format: int64 + maximum: 3000 + minimum: 10 + type: integer + type: object + transparentHugepageDefrag: + description: 'Defines the transparent hugepage defrag + configuration on the node. Currently supported values: + + * TRANSPARENT_HUGEPAGE_DEFRAG_ALWAYS: An application + requesting THP will stall on allocation failure and + directly reclaim pages and compact memory in an effort + to allocate a THP immediately. + + * TRANSPARENT_HUGEPAGE_DEFRAG_DEFER: An application + will wake kswapd in the background to reclaim pages + and wake kcompactd to compact memory so that THP is + available in the near future. It is the responsibility + of khugepaged to then install the THP pages later. + + * TRANSPARENT_HUGEPAGE_DEFRAG_DEFER_WITH_MADVISE: An + application will enter direct reclaim and compaction + like always, but only for regions that have used madvise(MADV_HUGEPAGE); + all other regions will wake kswapd in the background + to reclaim pages and wake kcompactd to compact memory + so that THP is available in the near future. + + * TRANSPARENT_HUGEPAGE_DEFRAG_MADVISE: An application + will enter direct reclaim and compaction like always, + but only for regions that have used madvise(MADV_HUGEPAGE); + all other regions will wake kswapd in the background + to reclaim pages and wake kcompactd to compact memory + so that THP is available in the near future. + + * TRANSPARENT_HUGEPAGE_DEFRAG_NEVER: An application + will never enter direct reclaim or compaction. + + * TRANSPARENT_HUGEPAGE_DEFRAG_UNSPECIFIED: Default value. + GKE will not modify the kernel configuration.' + enum: + - TRANSPARENT_HUGEPAGE_DEFRAG_ALWAYS + - TRANSPARENT_HUGEPAGE_DEFRAG_DEFER + - TRANSPARENT_HUGEPAGE_DEFRAG_DEFER_WITH_MADVISE + - TRANSPARENT_HUGEPAGE_DEFRAG_MADVISE + - TRANSPARENT_HUGEPAGE_DEFRAG_NEVER + - TRANSPARENT_HUGEPAGE_DEFRAG_UNSPECIFIED + type: string + transparentHugepageEnabled: + description: 'Controls transparent hugepage support for + anonymous memory. Currently supported values: + + * TRANSPARENT_HUGEPAGE_ENABLED_ALWAYS: Transparent hugepage + is enabled system wide. + + * TRANSPARENT_HUGEPAGE_ENABLED_MADVISE: Transparent + hugepage is enabled inside MADV_HUGEPAGE regions. This + is the default kernel configuration. + + * TRANSPARENT_HUGEPAGE_ENABLED_NEVER: Transparent hugepage + is disabled. + + * TRANSPARENT_HUGEPAGE_ENABLED_UNSPECIFIED: Default + value. GKE will not modify the kernel configuration.' + enum: + - TRANSPARENT_HUGEPAGE_ENABLED_ALWAYS + - TRANSPARENT_HUGEPAGE_ENABLED_MADVISE + - TRANSPARENT_HUGEPAGE_ENABLED_NEVER + - TRANSPARENT_HUGEPAGE_ENABLED_UNSPECIFIED + type: string + type: object + type: object + type: object + whenUnsatisfiable: + default: DoNotScaleUp + description: 'WhenUnsatisfiable describes autoscaler behaviour in + case none + + of the provided priorities is satisfiable. + + Currently supported values: + + * ScaleUpAnyway + + * DoNotScaleUp' + enum: + - ScaleUpAnyway + - DoNotScaleUp + type: string + required: + - whenUnsatisfiable + type: object + x-kubernetes-validations: + - message: Autopilot is required once set + rule: '!has(oldSelf.autopilot) || has(self.autopilot)' + - message: Nodepools priority cannot be used when Autopilot is enabled + rule: '(has(self.autopilot) && self.autopilot.enabled) ? !self.priorities.exists(priority, + has(priority.nodepools)) : true' + - message: NodePoolAutoCreation cannot be disabled when Autopilot is enabled + rule: '(has(self.autopilot) && self.autopilot.enabled) ? !(has(self.nodePoolAutoCreation) + && !self.nodePoolAutoCreation.enabled) : true' + - message: Only cos_containerd image type can be used when Autopilot is + enabled + rule: '(has(self.autopilot) && self.autopilot.enabled) ? (!has(self.nodePoolConfig) + || !has(self.nodePoolConfig.imageType) || self.nodePoolConfig.imageType + == "cos_containerd") : true' + - message: Only DEFAULT logging variant can be used when Autopilot is + enabled + rule: '(has(self.autopilot) && self.autopilot.enabled) ? (!has(self.nodePoolConfig) + || !has(self.nodePoolConfig.loggingConfig) || !has(self.nodePoolConfig.loggingConfig.loggingVariantConfig) + || !has(self.nodePoolConfig.loggingConfig.loggingVariantConfig.variant) + || self.nodePoolConfig.loggingConfig.loggingVariantConfig.variant + == "DEFAULT") : true' + - message: If NodePoolGroup is not specified NodePoolConfig.WorkloadType + can only be HIGH_AVAILABILITY if set + rule: '(has(self.nodePoolConfig) && has(self.nodePoolConfig.workloadType) + && !has(self.nodePoolGroup)) ? self.nodePoolConfig.workloadType == + "HIGH_AVAILABILITY" : true' + - message: In GKE Standard, pod family can be used only if Autopilot is + enabled + rule: 'self.priorities.exists(priority, has(priority.podFamily)) ? (has(self.autopilot) + && self.autopilot.enabled) : true' + - message: If using NodePoolConfig.ConfidentialNodeType, each priority + must specify either MachineFamily or MachineType. + rule: '(has(self.nodePoolConfig) && has(self.nodePoolConfig.confidentialNodeType)) + ? self.priorities.all(priority, has(priority.machineFamily) || has(priority.machineType)) + : true' + - message: ConfidentialNodeType SEV only supports N2D, C2D, C3D, C4D + rule: '(has(self.nodePoolConfig) && has(self.nodePoolConfig.confidentialNodeType) + && self.nodePoolConfig.confidentialNodeType == "SEV") ? self.priorities.all(priority, + ((has(priority.machineFamily) && priority.machineFamily in [''n2d'', + ''c2d'', ''c3d'', ''c4d'']) || (has(priority.machineType) && priority.machineType.split(''-'')[0] + in [''n2d'', ''c2d'', ''c3d'', ''c4d'']))) : true' + - message: ConfidentialNodeType SEV_SNP only supports N2D + rule: '(has(self.nodePoolConfig) && has(self.nodePoolConfig.confidentialNodeType) + && self.nodePoolConfig.confidentialNodeType == "SEV_SNP") ? self.priorities.all(priority, + ((has(priority.machineFamily) && priority.machineFamily in [''n2d'']) + || (has(priority.machineType) && priority.machineType.split(''-'')[0] + in [''n2d'']))) : true' + - message: ConfidentialNodeType TDX only supports C3 standard, C4 standard, + A3 and A4 + rule: '(has(self.nodePoolConfig) && has(self.nodePoolConfig.confidentialNodeType) + && self.nodePoolConfig.confidentialNodeType == "TDX") ? self.priorities.all(priority, + (has(priority.machineFamily) && priority.machineFamily in [''c3'', + ''c4'', ''a3'', ''a4'']) || (has(priority.machineType) && (priority.machineType.startsWith(''c3-standard-'') + || priority.machineType.startsWith(''c4-standard-'') || priority.machineType + == ''a3-highgpu-1g'' || priority.machineType == ''a4-highgpu-8g''))) + : true' + - message: ConfidentialNodeType TDX on C3 only supports c3-standard- machine + type and nvidia-h100-80gb GPU type + rule: '(has(self.nodePoolConfig) && has(self.nodePoolConfig.confidentialNodeType) + && self.nodePoolConfig.confidentialNodeType == "TDX") ? self.priorities.all(priority, + (has(priority.machineFamily) && priority.machineFamily == ''c3'' || + has(priority.machineType) && priority.machineType.startsWith(''c3-standard-'')) + ? (!has(priority.gpu) || has(priority.gpu) && (!has(priority.gpu.type) + || priority.gpu.type == ''nvidia-h100-80gb'')) : true) : true' + - message: ConfidentialNodeType TDX on C4 only supports c4-standard- machine + type and nvidia-h100-80gb GPU type + rule: '(has(self.nodePoolConfig) && has(self.nodePoolConfig.confidentialNodeType) + && self.nodePoolConfig.confidentialNodeType == "TDX") ? self.priorities.all(priority, + (has(priority.machineFamily) && priority.machineFamily == ''c4'' || + has(priority.machineType) && priority.machineType.startsWith(''c4-standard-'')) + ? (!has(priority.gpu) || has(priority.gpu) && (!has(priority.gpu.type) + || priority.gpu.type == ''nvidia-h100-80gb'')) : true) : true' + - message: ConfidentialNodeType TDX on A3 only supports a3-highgpu-1g + machine type and nvidia-h100-80gb GPU type + rule: '(has(self.nodePoolConfig) && has(self.nodePoolConfig.confidentialNodeType) + && self.nodePoolConfig.confidentialNodeType == "TDX") ? self.priorities.all(priority, + (has(priority.machineFamily) && priority.machineFamily == ''a3'' || + has(priority.machineType) && priority.machineType == ''a3-highgpu-1g'') + ? (!has(priority.gpu) || has(priority.gpu) && (!has(priority.gpu.type) + || priority.gpu.type == ''nvidia-h100-80gb'')) : true) : true' + - message: ConfidentialNodeType TDX on A4 only supports a4-highgpu-8g + machine type and nvidia-b200 GPU type + rule: '(has(self.nodePoolConfig) && has(self.nodePoolConfig.confidentialNodeType) + && self.nodePoolConfig.confidentialNodeType == "TDX") ? self.priorities.all(priority, + (has(priority.machineFamily) && priority.machineFamily == ''a4'' || + has(priority.machineType) && priority.machineType == ''a4-highgpu-8g'') + ? (!has(priority.gpu) || has(priority.gpu) && (!has(priority.gpu.type) + || priority.gpu.type == ''nvidia-b200'')) : true) : true' + - message: PriorityScore must be set for all priorities or for none of + them + rule: self.priorities.all(p, has(p.priorityScore)) || self.priorities.all(p, + !has(p.priorityScore)) + status: + description: Status of the ComputeClass. + properties: + conditions: + description: Conditions represent the observations of a ComputeClass's + current state. + items: + description: Condition contains details for one aspect of the current + state of this API Resource. + properties: + lastTransitionTime: + description: 'lastTransitionTime is the last time the condition + transitioned from one status to another. + + This should be when the underlying condition changed. If + that is not known, then using the time when the API field + changed is acceptable.' + format: date-time + type: string + message: + description: 'message is a human readable message indicating + details about the transition. + + This may be an empty string.' + maxLength: 32768 + type: string + observedGeneration: + description: 'observedGeneration represents the .metadata.generation + that the condition was set based upon. + + For instance, if .metadata.generation is currently 12, but + the .status.conditions[x].observedGeneration is 9, the condition + is out of date + + with respect to the current state of the instance.' + format: int64 + minimum: 0 + type: integer + reason: + description: 'reason contains a programmatic identifier indicating + the reason for the condition''s last transition. + + Producers of specific condition types may define expected + values and meanings for this field, + + and whether the values are considered a guaranteed API. + + The value should be a CamelCase string. + + This field may not be empty.' + maxLength: 1024 + minLength: 1 + pattern: ^[A-Za-z]([A-Za-z0-9_,:]*[A-Za-z0-9_])?$ + type: string + status: + description: status of the condition, one of True, False, Unknown. + enum: + - 'True' + - 'False' + - Unknown + type: string + type: + description: type of condition in CamelCase or in foo.example.com/CamelCase. + maxLength: 316 + pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$ + type: string + required: + - lastTransitionTime + - message + - reason + - status + - type + type: object + type: array + type: object + required: + - spec + type: object + served: true + storage: true + subresources: + status: {} From 9ecd5edbabba2a91184ae077e23092e9405a89bf Mon Sep 17 00:00:00 2001 From: Smana Date: Mon, 24 Aug 2026 12:13:14 +0200 Subject: [PATCH 2/7] docs(gcp): record the ComputeClass toleration requirement, measured Slice 4 was built to answer one question and answered a second one nobody had asked. Both are now written down where they will be hit. MEASURED, NOT ASSUMED (2026-08-24, on gcp-mycluster-0) Criterion 12 PASSES: an auto-created node carried node.cilium.io/agent-not-ready at registration, from spec.nodePoolConfig.taints, and Cilium cleared it. After scale-up the nodes retained only GKE's own cloud.google.com/compute-class taint. The unasked question: WITHOUT a toleration, nothing scales up at all. Three Pending pods, ten minutes, and no TriggeredScaleUp event of any kind -- only FailedScheduling. The autoscaler simulates scheduling against a node that will carry the class's taint, judges the pod unplaceable, and never provisions one. Adding ONLY the toleration produced four nodes and all pods Running. That is a real divergence from AWS, where the same taint is invisible to workloads: static pools carry it, and Karpenter provisions against NodePool requirements rather than simulating a tainted node. DECISION: keep the taint, require the toleration. It still gates every pod that does not opt in, and keeps auto-created nodes behaving like static ones. The accepted cost is that a tolerating pod can land before the Cilium agent is up and log plugin type="cilium-cni" failed (add): unable to create endpoint ... EOF which is the CNI PRESENT with its agent still starting, not a missing CNI. It self-heals: 0 restarts, pods reached Running unaided. kube-system/metrics-server hits the same transient on any fresh node, so it belongs to the platform rather than to this decision. Criterion 13 therefore holds as worded -- no pod recorded a MISSING CNI -- while its spirit is only partly met, and saying so is more useful than claiming a clean pass. WHERE IT IS RECORDED - The ComputeClass itself, in a box at the top of nodePoolConfig, with the toleration written out ready to copy. A consumer hits this file before they hit the ADR. - ADR-0006, as a Negative consequence. Its Positive list claimed Cilium's taint is "handled declaratively at the pool level", which is true but incomplete in a way that matters -- declaring the taint works; making it transparent to workloads does not. That bullet is now amended to point at the negative rather than left to mislead. Also observed, unprompted: criterion 14 (e2-highcpu-4, spot=true, zero on-demand -- NAP chose highcpu over standard, cheaper per vCPU) and criterion 17 (all four auto-created pools removed on scale-down; back to 2 nodes with no intervention). Verified: ./scripts/validate-manifests.sh -> Valid: 1189, Invalid: 0, Skipped: 0, all gates passed; ./scripts/validate-links.sh -> all relative links resolve. --- .../computeclass/general-purpose.yaml | 46 ++++++++++++++----- .../0006-nap-computeclass-over-karpenter.md | 23 ++++++++++ 2 files changed, 57 insertions(+), 12 deletions(-) diff --git a/infrastructure/gcp-mycluster-0/computeclass/general-purpose.yaml b/infrastructure/gcp-mycluster-0/computeclass/general-purpose.yaml index b5343e0d2..fd57fa51f 100644 --- a/infrastructure/gcp-mycluster-0/computeclass/general-purpose.yaml +++ b/infrastructure/gcp-mycluster-0/computeclass/general-purpose.yaml @@ -48,21 +48,43 @@ spec: whenUnsatisfiable: DoNotScaleUp nodePoolConfig: - # THE POINT OF THIS SLICE. + # ┌──────────────────────────────────────────────────────────────────────┐ + # │ EVERY WORKLOAD TARGETING THIS COMPUTE CLASS MUST TOLERATE THIS TAINT │ + # └──────────────────────────────────────────────────────────────────────┘ # - # The static pool sets this taint through `node_pools_taints` in - # opentofu/gcp/gke/init/main.tf. An auto-created pool has no OpenTofu to set - # it -- NAP creates the pool, so anything the pool must carry has to be - # declared here or it does not exist. + # tolerations: + # - key: node.cilium.io/agent-not-ready + # operator: Exists + # effect: NoSchedule # - # Without it, a new node registers Ready before Cilium owns its networking - # and accepts pods it cannot network. That surfaces as - # FailedCreatePodSandBox referencing a missing CNI (design criterion 13), - # which points at the CNI rather than at autoscaling -- so the cause is a - # long way from the symptom. + # NOT optional, and not merely good practice: WITHOUT the toleration nothing + # scales up at all. Measured on 2026-08-24 -- three Pending pods, ten + # minutes, and NO TriggeredScaleUp event of any kind. The autoscaler + # simulates scheduling against a node that will carry this taint, concludes + # the pod cannot be placed there, and never provisions one. Adding only the + # toleration produced four nodes and all pods Running. GKE's admission + # webhook warns about this on apply; the warning is load-bearing, not noise. # - # Cilium removes the taint once its agent is ready on the node. Nothing else - # clears it, which is what makes it a safe gate rather than a deadlock. + # WHY THE TAINT IS HERE AT ALL. The static pool sets it via + # `node_pools_taints` in opentofu/gcp/gke/init/main.tf. An auto-created pool + # has no OpenTofu to set it -- NAP creates the pool -- so anything it must + # carry has to be declared here or it simply does not exist. Without it a new + # node registers Ready before Cilium owns its networking and accepts pods it + # cannot network. + # + # THE TRADE, ACCEPTED DELIBERATELY. On a static pool this taint is invisible + # to workloads; here every consumer must tolerate it, which partly reopens + # the window it exists to close -- a tolerating pod can land before the + # Cilium agent is up and log + # plugin type="cilium-cni" failed (add): unable to create endpoint ... EOF + # That is the CNI present and its agent still starting, NOT a missing CNI, + # and it self-heals: measured 0 restarts, pods reached Running unaided. + # kube-system/metrics-server hits the same transient on any fresh node. + # + # Keeping the taint was chosen over dropping it: it still gates the pods that + # do NOT opt in, and it keeps auto-created nodes behaving like static ones. + # Cilium removes it once its agent is ready; nothing else clears it, so it is + # a gate rather than a deadlock. taints: - key: node.cilium.io/agent-not-ready value: "true" diff --git a/website/content/docs/decisions/0006-nap-computeclass-over-karpenter.md b/website/content/docs/decisions/0006-nap-computeclass-over-karpenter.md index aacff405e..bbf1f3ab3 100644 --- a/website/content/docs/decisions/0006-nap-computeclass-over-karpenter.md +++ b/website/content/docs/decisions/0006-nap-computeclass-over-karpenter.md @@ -125,6 +125,9 @@ arrives. ComputeClass is layered on afterwards, mirroring the AWS sequence. - Three `ComputeClass` manifests replace six Karpenter manifests. - Cilium's readiness taint is handled declaratively at the pool level rather than by a post-provisioning hook. + - *Amended 2026-08-24, after measuring it.* This is true but incomplete, and the omission + matters: see the toleration requirement under Negative. Declaring the taint works; making it + **transparent to workloads** does not. ### Negative @@ -133,6 +136,26 @@ arrives. ComputeClass is layered on afterwards, mirroring the AWS sequence. differ measurably between the two clouds. - *Mitigation*: the autoscaling slice ships an explicit written statement of the gap rather than leaving it to be discovered. This is a documented divergence, not an abstraction to be faked. +- **Every workload targeting a ComputeClass must tolerate `node.cilium.io/agent-not-ready`, + or nothing scales up at all.** Added 2026-08-24 from a live measurement, because this was not + anticipated when the ADR was written. + + The autoscaler simulates scheduling against a node that *will* carry the class's taint. A pod + that cannot tolerate it is judged unplaceable, so no node is provisioned — and the symptom is + silence: three Pending pods for ten minutes with **no `TriggeredScaleUp` event of any kind**, + only `FailedScheduling`. Adding the toleration alone produced four nodes and all pods Running. + GKE's admission webhook warns on apply; the warning is load-bearing. + + This is a real divergence from AWS, where the equivalent taint is invisible to workloads + because static pools carry it and Karpenter provisions against `NodePool` requirements rather + than simulating a tainted node. + - *Decision*: keep the taint and require the toleration, rather than dropping it. It still + gates every pod that does not opt in, and keeps auto-created nodes behaving like static ones. + - *Accepted cost*: a tolerating pod can land before the Cilium agent is up and log + `plugin type="cilium-cni" failed (add): unable to create endpoint ... EOF`. That is the CNI + present with its agent still starting, **not** a missing CNI, and it self-heals — measured 0 + restarts, pods reached `Running` unaided. `kube-system/metrics-server` hits the same transient + on any fresh node, so it is a property of the platform rather than of this decision. - No `min > 0` per auto-created pool, so "always keep N warm" must be expressed differently (for example a small static pool alongside the auto-created ones). - Karpenter knowledge does not transfer cleanly; operators need to learn a second model. From e761438a7cf44e0267ec5a2d0164f493052f7419 Mon Sep 17 00:00:00 2001 From: Smana Date: Mon, 24 Aug 2026 12:16:55 +0200 Subject: [PATCH 3/7] feat(gcp): add the io and gpu-l4 ComputeClasses Completes slice 4's three classes. general-purpose was proven on its own first because it carried the unknown the slice existed to settle; these two follow the pattern it established, and each inherits the mandatory Cilium toleration. MIRRORING AWS WHERE IT IS A CONVENTION, DIVERGING WHERE IT IS A FACT Taint keys are IDENTICAL to the AWS NodePools -- `ogenki/io` and `nvidia.com/gpu`. Those are platform conventions rather than cloud details, so a workload's tolerations are the one part of its manifest that does not change between clouds, even though everything underneath differs. gpu-l4 needs NO runtimeClassName, and that is a real divergence rather than an omission. On AWS the Bottlerocket NVIDIA variant requires `runtimeClassName: nvidia` plus a matching RuntimeClass (infrastructure/base/runtimeclass-nvidia/). GKE installs the driver via its own DaemonSet and wires the default runtime, so a pod just requests nvidia.com/gpu -- which is design criterion 15 exactly: "a GPU pod with NO runtimeClassName sees the device via nvidia-smi". TWO THINGS THE API TAUGHT US 1. `localSSDCount` is rejected with machineFamily. GKE's webhook: compute-class "io" doesn't support local ssd without machine type or GPU config Local SSD attachment is a per-machine-type property, so io must name explicit machineTypes. The cost is a narrower fallback list than general-purpose gets. Found by `kubectl apply --dry-run=server`, not by reading docs. 2. `cluster_autoscaling.gpu_resources` was an empty list, deferred when the GPU class did not yet exist. It is now REQUIRED: NAP will not create a node with an accelerator that has no resourceLimits entry, so leaving it empty caps GPU autoscaling at zero -- the class applies cleanly, pods stay Pending, and nothing says why. nvidia-l4, maximum 2. COST POSTURE UNCHANGED Both classes are spot-only with `whenUnsatisfiable: DoNotScaleUp`. That matters most for gpu-l4: spot GPUs are scarcer than spot CPU, so the class will sometimes provision nothing -- which is the intended behaviour, because an L4 silently falling back to on-demand is the single most expensive accident available in this repository. Verified: `kubectl apply --dry-run=server` accepts both against the live GKE API; ./scripts/validate-manifests.sh -> Valid: 1191, Invalid: 0, Skipped: 0, all gates passed; tofu validate and tofu fmt clean. NOT verified: neither class has actually provisioned a node. general-purpose was exercised end to end (criteria 12, 14, 17); io and gpu-l4 are schema-valid and follow a proven pattern, which is not the same as measured. Criterion 15 in particular needs a real GPU pod. --- .../gcp-mycluster-0/computeclass/gpu-l4.yaml | 71 ++++++++++++++++ .../gcp-mycluster-0/computeclass/io.yaml | 80 +++++++++++++++++++ .../computeclass/kustomization.yaml | 5 ++ opentofu/gcp/gke/init/main.tf | 20 ++++- 4 files changed, 173 insertions(+), 3 deletions(-) create mode 100644 infrastructure/gcp-mycluster-0/computeclass/gpu-l4.yaml create mode 100644 infrastructure/gcp-mycluster-0/computeclass/io.yaml diff --git a/infrastructure/gcp-mycluster-0/computeclass/gpu-l4.yaml b/infrastructure/gcp-mycluster-0/computeclass/gpu-l4.yaml new file mode 100644 index 000000000..67a8aaf07 --- /dev/null +++ b/infrastructure/gcp-mycluster-0/computeclass/gpu-l4.yaml @@ -0,0 +1,71 @@ +# GPU ComputeClass — the GCP counterpart to +# infrastructure/base/karpenter-nodepools-gpu/gpu-l4-nodepool.yaml. +# +# NVIDIA L4, matching the AWS pool's accelerator choice and the reason +# europe-west4 was picked in the first place: opentofu/config.tm.hcl records that +# nvidia-l4 exists in all three of its zones and NOT AT ALL in europe-west9, +# which would otherwise have been the geographic match for eu-west-3. +# +# NO runtimeClassName IS NEEDED HERE, and that is a real divergence from AWS +# rather than an omission. On AWS the Bottlerocket NVIDIA variant requires +# `runtimeClassName: nvidia` and a matching RuntimeClass +# (infrastructure/base/runtimeclass-nvidia/). GKE installs the driver through its +# own DaemonSet and wires the default runtime, so a GPU pod just requests +# `nvidia.com/gpu` and works -- which is exactly design criterion 15: "a GPU pod +# with NO runtimeClassName sees the device via nvidia-smi". +apiVersion: cloud.google.com/v1 +kind: ComputeClass +metadata: + name: gpu-l4 +spec: + # g2 is the only family that carries L4, so there is no fallback tier to list: + # either L4 spot capacity exists in this zone or nothing is provisioned. + # + # count: 1 mirrors the AWS pool's instance-gpu-count ["1"] -- one accelerator + # per node keeps the failure domain small and the bill legible, and nothing on + # this platform yet needs multi-GPU nodes. + priorities: + - machineFamily: g2 + spot: true + gpu: + type: nvidia-l4 + count: 1 + + # Spot GPUs are markedly scarcer than spot CPU, so this class will sometimes + # provision nothing at all. That is the intended behaviour on a test cluster: + # an L4 falling back to on-demand is the single most expensive thing this + # repository could do by accident. + whenUnsatisfiable: DoNotScaleUp + + nodePoolConfig: + # ┌──────────────────────────────────────────────────────────────────────┐ + # │ WORKLOADS MUST TOLERATE **BOTH** TAINTS BELOW │ + # └──────────────────────────────────────────────────────────────────────┘ + # + # tolerations: + # - key: node.cilium.io/agent-not-ready + # operator: Exists + # effect: NoSchedule + # - key: nvidia.com/gpu + # operator: Exists + # effect: NoSchedule + # + # The Cilium one is not optional: without it NOTHING SCALES UP -- the + # autoscaler judges an untolerating pod unplaceable on a node that will carry + # the taint and never provisions. Measured 2026-08-24; see ADR-0006. + # + # `nvidia.com/gpu` is the conventional accelerator taint and is identical to + # the AWS pool's, so a GPU workload's tolerations are portable between clouds + # even though everything under them differs. + taints: + - key: node.cilium.io/agent-not-ready + value: "true" + effect: NoSchedule + - key: nvidia.com/gpu + value: "true" + effect: NoSchedule + + imageType: cos_containerd + + nodePoolAutoCreation: + enabled: true diff --git a/infrastructure/gcp-mycluster-0/computeclass/io.yaml b/infrastructure/gcp-mycluster-0/computeclass/io.yaml new file mode 100644 index 000000000..1ac6455e7 --- /dev/null +++ b/infrastructure/gcp-mycluster-0/computeclass/io.yaml @@ -0,0 +1,80 @@ +# IO-optimised ComputeClass — the GCP counterpart to +# infrastructure/base/karpenter-nodepools/io-nodepool.yaml. +# +# For workloads that want local NVMe rather than network-attached disk. The +# `ogenki/io` taint key is kept IDENTICAL to the AWS NodePool's on purpose: it is +# a platform convention, not a cloud detail, so a workload's toleration is the +# one part of its manifest that does not have to change between clouds. +apiVersion: cloud.google.com/v1 +kind: ComputeClass +metadata: + name: io +spec: + # machineType, NOT machineFamily — and that is forced, not stylistic. GKE's + # admission webhook rejects the family form outright: + # + # compute-class "io" doesn't support local ssd without machine type or + # GPU config + # + # Local SSD attachment is a per-machine-type property, so the class has to name + # the type. The cost is a narrower fallback list than general-purpose gets: + # two explicit sizes rather than "any n2". + # + # localSSDCount is what makes this class different from general-purpose. GCP + # attaches Local SSD as fixed 375 GiB devices, so the count is a device count, + # not a size — one is ample for the ephemeral-storage workloads this exists for. + # + # Both entries are n2: the smaller first so a modest IO workload does not + # provision a large node, with the larger as the capacity fallback. Sticking to + # one family keeps this predictable; c3 would need its distinct `-lssd` machine + # types, which is a second thing to get right for no benefit on a test cluster. + priorities: + - machineType: n2-standard-4 + spot: true + storage: + localSSDCount: 1 + - machineType: n2-standard-8 + spot: true + storage: + localSSDCount: 1 + + # Same rationale as general-purpose: criterion 14 forbids on-demand fallback, + # so exhausted spot leaves pods Pending rather than quietly costing more. + whenUnsatisfiable: DoNotScaleUp + + nodePoolConfig: + # ┌──────────────────────────────────────────────────────────────────────┐ + # │ WORKLOADS MUST TOLERATE **BOTH** TAINTS BELOW │ + # └──────────────────────────────────────────────────────────────────────┘ + # + # tolerations: + # - key: node.cilium.io/agent-not-ready + # operator: Exists + # effect: NoSchedule + # - key: ogenki/io + # operator: Exists + # effect: NoSchedule + # + # The Cilium one is not optional: without it NOTHING SCALES UP. The + # autoscaler simulates against a node carrying the class's taints and judges + # an untolerating pod unplaceable, so it never provisions. Measured on + # 2026-08-24 with general-purpose — ten minutes Pending, no TriggeredScaleUp + # event at all. See ADR-0006 and general-purpose.yaml. + # + # `ogenki/io` is the deliberate one: it keeps general-purpose workloads off + # nodes provisioned for their local disk, which are more expensive and + # fewer. Identical key and value to the AWS io NodePool. + taints: + - key: node.cilium.io/agent-not-ready + value: "true" + effect: NoSchedule + - key: ogenki/io + value: "true" + effect: NoSchedule + + # Must match the static pool and the NAP default. Cilium's DaemonSet needs a + # writable /home/kubernetes/bin, which is a Container-Optimized OS path. + imageType: cos_containerd + + nodePoolAutoCreation: + enabled: true diff --git a/infrastructure/gcp-mycluster-0/computeclass/kustomization.yaml b/infrastructure/gcp-mycluster-0/computeclass/kustomization.yaml index ff94770f8..6b557dd94 100644 --- a/infrastructure/gcp-mycluster-0/computeclass/kustomization.yaml +++ b/infrastructure/gcp-mycluster-0/computeclass/kustomization.yaml @@ -1,5 +1,10 @@ apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization +# The three classes of design slice 4. general-purpose was proven first, on its +# own, because it carried the unknown the slice existed to settle (the Cilium +# taint); io and gpu-l4 follow the pattern it established. resources: - general-purpose.yaml + - io.yaml + - gpu-l4.yaml diff --git a/opentofu/gcp/gke/init/main.tf b/opentofu/gcp/gke/init/main.tf index bfa509d1c..1d35a6a57 100644 --- a/opentofu/gcp/gke/init/main.tf +++ b/opentofu/gcp/gke/init/main.tf @@ -165,9 +165,23 @@ module "gke" { min_memory_gb = 0 max_memory_gb = var.autoscaling_max_memory_gb - # GPU limits stay empty until the GPU ComputeClass exists. An entry here - # would let NAP provision accelerators that nothing yet asks for. - gpu_resources = [] + # REQUIRED for the gpu-l4 ComputeClass to provision anything. NAP will not + # create a node with an accelerator that has no resourceLimits entry, so an + # empty list here silently caps GPU autoscaling at zero -- the class applies + # cleanly, pods stay Pending, and nothing says why. + # + # nvidia-l4 only, matching the class and the reason europe-west4 was chosen + # (see opentofu/config.tm.hcl). Maximum 2 is a test-cluster ceiling: L4s are + # the most expensive thing this repository can provision, and criterion 16 + # wants an oversized workload to stay Unschedulable rather than scale into a + # bill. + gpu_resources = [ + { + resource_type = "nvidia-l4" + minimum = 0 + maximum = 2 + }, + ] auto_repair = true auto_upgrade = true From a6a93bd54aa81564002b9565cd709e8c2509f4e8 Mon Sep 17 00:00:00 2001 From: Smana Date: Mon, 24 Aug 2026 13:13:26 +0200 Subject: [PATCH 4/7] docs(gcp): record slice 4's measured results, including what is blocked Every criterion now carries its outcome and the evidence for it, rather than leaving the design as a list of intentions. Four PASS, measured on gcp-mycluster-0: 12 taint applied at registration and cleared by Cilium 14 e2-highcpu-4, spot=true, cos_containerd -- NAP picked highcpu over standard unprompted, cheaper per vCPU 16 a 64-vCPU pod against a 32-vCPU ceiling held Pending 4 minutes, node count never moved, autoscaler logged NotTriggerScaleUp 17 all four auto-created nodes reaped after the probe was deleted One PARTIAL (13), one BLOCKED (15). 13 holds as worded -- nothing recorded a MISSING CNI -- but the toleration that criterion 12 forces means tolerating pods can land before the Cilium agent is up and log a transient `cilium-cni failed (add) ... EOF`. It self-heals, and kube-system/metrics-server hits the same on any fresh node. Recording that is more useful than a green tick, because the two failure modes read almost identically in logs and only one is benign. 15 is BLOCKED, not failed, and the distinction is the point: GPUS_ALL_REGIONS on this project is 0, so no GPU node can exist at any price. Worth writing down loudly because the obvious diagnostic misleads -- the REGIONAL NVIDIA_L4_GPUS quota reads 1, which looks fine and would send the next person away to "retry when spot frees up". The spot attempt fails with `GCE out of resources` and only the on-demand attempt says `GCE quota exceeded`. Running both is what distinguished capacity from quota. Everything up to the GPU is proven: the class is selected and NAP resolves it to a g2-standard-4-gpu1 pool and attempts creation. That also retroactively confirms cluster_autoscaling.gpu_resources is load-bearing -- left empty, no scale-up is attempted at all and the symptom is silence. The on-demand probe cost nothing: the node was never created, and the throwaway ComputeClass and pod were deleted immediately. Also adds the quota blocker to the resume plan, where someone rebuilding will look, with the explicit warning not to diagnose it from the regional quota. Verified: ./scripts/validate-links.sh -> all relative links resolve. --- .../plans/2026-08-23-gcp-foundation-resume.md | 14 ++++++++++ .../specs/2026-08-18-gcp-support-design.md | 28 ++++++++++++++++++- 2 files changed, 41 insertions(+), 1 deletion(-) diff --git a/docs/superpowers/plans/2026-08-23-gcp-foundation-resume.md b/docs/superpowers/plans/2026-08-23-gcp-foundation-resume.md index 6fa75a71a..6f7d27a34 100644 --- a/docs/superpowers/plans/2026-08-23-gcp-foundation-resume.md +++ b/docs/superpowers/plans/2026-08-23-gcp-foundation-resume.md @@ -97,6 +97,20 @@ flux reconcile source git flux-system failure mode, `depends_on` gaps, and the `clusters/gcp-mycluster-0/` Flux wiring. Two review agents produced nothing across repeated asks. +### GPU quota — blocks slice 4's last criterion + +`GPUS_ALL_REGIONS` on project `ogenki-435905` is **0**, so no GPU node can be +created at any price. Criterion 15 ("a GPU pod with no `runtimeClassName` sees +the device via `nvidia-smi`") is therefore blocked rather than failed — the +`gpu-l4` ComputeClass is proven correct up to the point of provisioning. + +Do **not** diagnose this from the regional quota: `NVIDIA_L4_GPUS` in +`europe-west4` shows a limit of 1, which looks fine and is meaningless while the +global cap is zero. Spot attempts fail with `GCE out of resources` and on-demand +with `GCE quota exceeded`; only the second points at the real cause. + +Closing it needs a quota increase request to Google, not a retry. + ### ClusterMesh prerequisites (recorded in ADR-0017, not implemented) - `cluster.id` is unset on both clusters; ClusterMesh needs a unique 1–255 ID. diff --git a/docs/superpowers/specs/2026-08-18-gcp-support-design.md b/docs/superpowers/specs/2026-08-18-gcp-support-design.md index 39a83bc60..ad7014af3 100644 --- a/docs/superpowers/specs/2026-08-18-gcp-support-design.md +++ b/docs/superpowers/specs/2026-08-18-gcp-support-design.md @@ -351,16 +351,42 @@ Falsifiable, verified against a live cluster. 11. A written monthly run-rate estimate exists (cluster fee, static pool, Cloud NAT, Cloud DNS, Tailscale instance) stating the zonal-vs-regional choice and its price delta. -**Slice 4 (autoscaling)** +**Slice 4 (autoscaling)** — *results recorded 2026-08-24, measured on gcp-mycluster-0. +Four PASS, one partial, one blocked on a GCP quota. Each is annotated below.* + 12. A **freshly auto-created** node carries `node.cilium.io/agent-not-ready` at registration, and Cilium clears it. *This is the criterion the slice exists to test.* + - **PASS.** Set via `spec.nodePoolConfig.taints`; after scale-up the nodes retained only GKE's + own `cloud.google.com/compute-class` taint. **Unanticipated finding:** every workload + targeting a ComputeClass must *tolerate* that taint or NOTHING SCALES UP — the autoscaler + judges an untolerating pod unplaceable on a node that will carry it. Ten minutes Pending, + no `TriggeredScaleUp` at all. See ADR-0006. 13. Across 5 scale-up cycles, 0 pods record `FailedCreatePodSandBox` referencing a missing CNI. + - **PARTIAL.** As worded it holds — nothing recorded a *missing* CNI. But tolerating pods can + land before the Cilium agent is up and log `plugin type="cilium-cni" failed (add) ... EOF`, + which is the CNI present with its agent starting. Self-heals (0 restarts); + `kube-system/metrics-server` hits the same on any fresh node. The toleration required by + criterion 12 is what reopens this window. 14. `imageType` on every auto-created pool matches the pinned value; general-purpose nodes are spot with **zero** on-demand fallback. + - **PASS.** Auto-created nodes came up `e2-highcpu-4` with `spot=true` on `cos_containerd`. + NAP chose `highcpu` over `standard` unprompted — cheaper per vCPU. 15. A GPU pod with **no** `runtimeClassName` sees the device via `nvidia-smi`. + - **BLOCKED — not failed.** Untestable in this project: `GPUS_ALL_REGIONS` is **0**, so no GPU + node can be created at any price. The per-region `NVIDIA_L4_GPUS: 1` is meaningless beneath + it — checking only the regional quota would wrongly suggest retrying later. + Spot attempt gave `GCE out of resources`; on-demand gave `GCE quota exceeded`. + Everything up to the GPU is proven: the class is selected, NAP resolves it to a + `g2-standard-4-gpu1` pool and attempts creation — which also confirms + `cluster_autoscaling.gpu_resources` is load-bearing (left empty, no scale-up is even + attempted). **Needs a GPUS_ALL_REGIONS quota grant from Google to close.** 16. Cluster `resourceLimits` set; an oversized workload stays `Unschedulable` at the ceiling. + - **PASS.** A 64-vCPU pod against a 32-vCPU ceiling held `Pending` for 4 minutes, node count + never moved, autoscaler logged `NotTriggerScaleUp`. It refuses rather than grinding. 17. Empty auto-created pools are removed on scale-down. + - **PASS.** All four auto-created nodes were reaped after the probe was deleted; back to the + 2 static nodes with no intervention. `OPTIMIZE_UTILIZATION` was chosen for this. **Slice 5 (identity)** From 31caf37163c0ad5548b3b554208d98e2b641ef45 Mon Sep 17 00:00:00 2001 From: Smana Date: Mon, 24 Aug 2026 13:21:46 +0200 Subject: [PATCH 5/7] refactor(gcp): apply the simplify pass to slice 4 Six findings from the review, all applied. The first is a flat contradiction and the second is a failure this session already fixed once elsewhere. 1. general-purpose.yaml opened with "the FIRST and, for now, the ONLY one" and "Deliberately one class, not three". Both false in this branch -- io and gpu-l4 sit beside it, and kustomization.yaml lists all three. That was true when written and never updated when the other two landed. The first thing a reader saw was wrong, so the paragraph is gone; kustomization.yaml already records the general-purpose-proven-first history, once and better. 2. The taint story was told FOUR times in general-purpose.yaml -- why it matters, the measurement, why it is there, the trade -- and a fifth time in ADR-0006, near-verbatim down to the `cilium-cni ... EOF` line. That is exactly the cilium.yaml failure corrected earlier in this branch's history: overlapping accounts of one decision, where the next amendment updates some and silently strands the rest. Now: the file states the REQUIREMENT (the toleration snippet, ready to copy, plus one sentence that without it nothing scales up), and the ADR explains it. One account, one place to amend. io and gpu-l4 keep only their own class-specific taint sentence -- ogenki/io and nvidia.com/gpu are genuinely local facts -- and cross-reference the rest. 3. imageType had three different treatments across three files: five comment lines, three, and none. Now one identical line in each, pointing at the image_type note in main.tf where the reasoning lives. 4. The criterion-16 ceiling rationale was stated in main.tf and again on the variables in variables.tf. Kept on the variables, where the default sits. 5. gen-catalog.sh listed "Three sources" with the new entry numbered 4 and inserted between 2 and 3. Now four sources, in order. 6. infrastructure.yaml argued carefully about the dependsOn it OMITS (crds) and left the one it keeps unexplained. Now says why: currently inert, since a ComputeClass is cluster-scoped and this path holds nothing namespaced, but kept because the first namespaced resource added here would otherwise race its namespace -- a failure that presents as a dependency cascade rather than as a missing namespace. Deliberately NOT changed: the taint declarations are duplicated across all three classes and stay that way. The review's own recommendation, and it is right -- the taints are what a reader opens the file to learn, and a kustomize patch injecting them would make io.yaml look like it needs one toleration when it needs two. Roughly ten shared YAML lines is below where indirection pays. Verified: ./scripts/validate-manifests.sh -> Valid: 1191, Invalid: 0, Skipped: 0, all gates passed; tofu validate and tofu fmt clean; bash -n clean on gen-catalog.sh; the API still accepts all three classes. --- clusters/gcp-mycluster-0/infrastructure.yaml | 5 ++ .../computeclass/general-purpose.yaml | 90 +++++-------------- .../gcp-mycluster-0/computeclass/gpu-l4.yaml | 11 +-- .../gcp-mycluster-0/computeclass/io.yaml | 18 ++-- opentofu/gcp/gke/init/main.tf | 5 +- scripts/flux-schema/gen-catalog.sh | 7 +- 6 files changed, 48 insertions(+), 88 deletions(-) diff --git a/clusters/gcp-mycluster-0/infrastructure.yaml b/clusters/gcp-mycluster-0/infrastructure.yaml index 004a1ac3f..d9c0d2a36 100644 --- a/clusters/gcp-mycluster-0/infrastructure.yaml +++ b/clusters/gcp-mycluster-0/infrastructure.yaml @@ -27,4 +27,9 @@ spec: kind: ExternalArtifact name: infra-artifact dependsOn: + # Currently inert -- a ComputeClass is cluster-scoped and this path holds + # nothing namespaced. Kept as future-proofing: this is the GCP cluster's only + # infrastructure Kustomization, so the first namespaced resource added here + # would otherwise race its namespace, which is a failure that presents as a + # dependency cascade rather than as a missing namespace. - name: namespaces diff --git a/infrastructure/gcp-mycluster-0/computeclass/general-purpose.yaml b/infrastructure/gcp-mycluster-0/computeclass/general-purpose.yaml index fd57fa51f..0b59af398 100644 --- a/infrastructure/gcp-mycluster-0/computeclass/general-purpose.yaml +++ b/infrastructure/gcp-mycluster-0/computeclass/general-purpose.yaml @@ -1,55 +1,36 @@ -# General-purpose ComputeClass — the FIRST and, for now, the ONLY one. +# General-purpose ComputeClass — the default shape for auto-provisioned nodes. # -# ADR-0006 chose GKE node auto-provisioning over Karpenter on GCP. This object is -# what turns that decision into running nodes: it tells NAP which machine shapes -# to try, in what order, and lets it CREATE a node pool rather than only pick -# among existing ones. -# -# Deliberately one class, not three. The design's slice 4 exists to settle a -# single question -- does a freshly auto-created node carry -# `node.cilium.io/agent-not-ready` at registration, and does Cilium clear it? -# Writing three classes before that is answered would mean debugging three -# variants of the same unknown. The `io` and GPU classes follow once this one is -# proven. -# -# WHY THE TAINT QUESTION MATTERS: on GKE, Cilium replaces the CNI. A node that -# registers Ready before Cilium owns its networking will accept pods it cannot -# network, which surfaces as FailedCreatePodSandBox referencing a missing CNI -# rather than as anything pointing at autoscaling. Static pools set the taint in -# OpenTofu; an auto-created pool has no OpenTofu to set it, which is precisely -# why this is the criterion the slice is built around. +# The GCP counterpart to infrastructure/base/karpenter-nodepools/default-nodepool.yaml. +# ADR-0006 chose GKE node auto-provisioning over Karpenter here; this object is +# what turns that decision into running nodes, telling NAP which machine shapes +# to try and in what order. apiVersion: cloud.google.com/v1 kind: ComputeClass metadata: name: general-purpose spec: - # Ordered fallback. NAP walks these in sequence and takes the first that can be - # provisioned, so the list is a preference order, not a set. + # Ordered fallback: NAP walks these in sequence and takes the first it can + # provision, so this is a preference order rather than a set. # - # e2-standard-4 first because it is what the static pool runs -- keeping the - # auto-created shape identical to the hand-created one removes a variable from - # the taint experiment. n2-standard-4 second as a spot-availability fallback: - # a GKE node pool takes ONE machine type, so a single shape concentrates spot - # interruption risk, which is the limitation the static pool's own comment - # calls out as "breadth coming later from ComputeClass". + # e2 first because it is what the static pool runs and it is the cheapest + # general-purpose family. n2 second as a spot-availability fallback: a GKE node + # pool takes ONE machine type, so a single family concentrates interruption + # risk — the breadth the static pool's own comment defers to ComputeClass. priorities: - machineFamily: e2 spot: true - machineFamily: n2 spot: true - # No on-demand entry, deliberately. Design criterion 14 requires general-purpose - # nodes to be spot with ZERO on-demand fallback: this is a reference platform - # where an unnoticed fallback to on-demand is a silent cost regression, and the - # workloads are all restartable. - # - # The consequence is honest: when no spot capacity exists in any listed family, - # pods stay Pending rather than quietly becoming expensive. + # No on-demand entry, deliberately (design criterion 14). On a test cluster an + # unnoticed fallback to on-demand is a silent cost regression, and these + # workloads are restartable. The honest consequence: when no spot capacity + # exists in either family, pods stay Pending rather than becoming expensive. whenUnsatisfiable: DoNotScaleUp nodePoolConfig: # ┌──────────────────────────────────────────────────────────────────────┐ - # │ EVERY WORKLOAD TARGETING THIS COMPUTE CLASS MUST TOLERATE THIS TAINT │ + # │ WORKLOADS TARGETING THIS CLASS MUST TOLERATE THIS TAINT │ # └──────────────────────────────────────────────────────────────────────┘ # # tolerations: @@ -57,43 +38,18 @@ spec: # operator: Exists # effect: NoSchedule # - # NOT optional, and not merely good practice: WITHOUT the toleration nothing - # scales up at all. Measured on 2026-08-24 -- three Pending pods, ten - # minutes, and NO TriggeredScaleUp event of any kind. The autoscaler - # simulates scheduling against a node that will carry this taint, concludes - # the pod cannot be placed there, and never provisions one. Adding only the - # toleration produced four nodes and all pods Running. GKE's admission - # webhook warns about this on apply; the warning is load-bearing, not noise. - # - # WHY THE TAINT IS HERE AT ALL. The static pool sets it via - # `node_pools_taints` in opentofu/gcp/gke/init/main.tf. An auto-created pool - # has no OpenTofu to set it -- NAP creates the pool -- so anything it must - # carry has to be declared here or it simply does not exist. Without it a new - # node registers Ready before Cilium owns its networking and accepts pods it - # cannot network. - # - # THE TRADE, ACCEPTED DELIBERATELY. On a static pool this taint is invisible - # to workloads; here every consumer must tolerate it, which partly reopens - # the window it exists to close -- a tolerating pod can land before the - # Cilium agent is up and log - # plugin type="cilium-cni" failed (add): unable to create endpoint ... EOF - # That is the CNI present and its agent still starting, NOT a missing CNI, - # and it self-heals: measured 0 restarts, pods reached Running unaided. - # kube-system/metrics-server hits the same transient on any fresh node. - # - # Keeping the taint was chosen over dropping it: it still gates the pods that - # do NOT opt in, and it keeps auto-created nodes behaving like static ones. - # Cilium removes it once its agent is ready; nothing else clears it, so it is - # a gate rather than a deadlock. + # Not optional: WITHOUT it nothing scales up at all — the autoscaler judges + # an untolerating pod unplaceable on a node that will carry the taint and + # never provisions one. Measured 2026-08-24. Full rationale, the trade it + # accepts, and the AWS divergence are in ADR-0006 (Negative consequences); + # this file states the requirement, the ADR explains it. taints: - key: node.cilium.io/agent-not-ready value: "true" effect: NoSchedule - # Must match the static pool and the NAP-level image_type. Cilium's - # DaemonSet depends on a writable /home/kubernetes/bin, which is a - # Container-Optimized OS path -- the very first GKE deploy failed on exactly - # this, and on an auto-created node it would fail where nobody was looking. + # Must match the static pool and the NAP default — see the image_type note in + # opentofu/gcp/gke/init/main.tf for why COS specifically. imageType: cos_containerd nodePoolAutoCreation: diff --git a/infrastructure/gcp-mycluster-0/computeclass/gpu-l4.yaml b/infrastructure/gcp-mycluster-0/computeclass/gpu-l4.yaml index 67a8aaf07..b90b67af1 100644 --- a/infrastructure/gcp-mycluster-0/computeclass/gpu-l4.yaml +++ b/infrastructure/gcp-mycluster-0/computeclass/gpu-l4.yaml @@ -50,12 +50,11 @@ spec: # operator: Exists # effect: NoSchedule # - # The Cilium one is not optional: without it NOTHING SCALES UP -- the - # autoscaler judges an untolerating pod unplaceable on a node that will carry - # the taint and never provisions. Measured 2026-08-24; see ADR-0006. + # The Cilium taint is not optional — without it nothing scales up. See + # general-purpose.yaml and ADR-0006. # - # `nvidia.com/gpu` is the conventional accelerator taint and is identical to - # the AWS pool's, so a GPU workload's tolerations are portable between clouds + # `nvidia.com/gpu` is the conventional accelerator taint, identical to the + # AWS pool's, so a GPU workload's tolerations are portable between clouds # even though everything under them differs. taints: - key: node.cilium.io/agent-not-ready @@ -65,6 +64,8 @@ spec: value: "true" effect: NoSchedule + # Must match the static pool and the NAP default — see the image_type note in + # opentofu/gcp/gke/init/main.tf for why COS specifically. imageType: cos_containerd nodePoolAutoCreation: diff --git a/infrastructure/gcp-mycluster-0/computeclass/io.yaml b/infrastructure/gcp-mycluster-0/computeclass/io.yaml index 1ac6455e7..988b932f2 100644 --- a/infrastructure/gcp-mycluster-0/computeclass/io.yaml +++ b/infrastructure/gcp-mycluster-0/computeclass/io.yaml @@ -55,15 +55,13 @@ spec: # operator: Exists # effect: NoSchedule # - # The Cilium one is not optional: without it NOTHING SCALES UP. The - # autoscaler simulates against a node carrying the class's taints and judges - # an untolerating pod unplaceable, so it never provisions. Measured on - # 2026-08-24 with general-purpose — ten minutes Pending, no TriggeredScaleUp - # event at all. See ADR-0006 and general-purpose.yaml. + # The Cilium taint is not optional — without it nothing scales up. See + # general-purpose.yaml and ADR-0006. # - # `ogenki/io` is the deliberate one: it keeps general-purpose workloads off - # nodes provisioned for their local disk, which are more expensive and - # fewer. Identical key and value to the AWS io NodePool. + # `ogenki/io` is this class's own: it keeps general-purpose workloads off + # nodes provisioned for their local disk, which are fewer and dearer. + # Identical key and value to the AWS io NodePool, so a workload's + # tolerations are portable between clouds. taints: - key: node.cilium.io/agent-not-ready value: "true" @@ -72,8 +70,8 @@ spec: value: "true" effect: NoSchedule - # Must match the static pool and the NAP default. Cilium's DaemonSet needs a - # writable /home/kubernetes/bin, which is a Container-Optimized OS path. + # Must match the static pool and the NAP default — see the image_type note in + # opentofu/gcp/gke/init/main.tf for why COS specifically. imageType: cos_containerd nodePoolAutoCreation: diff --git a/opentofu/gcp/gke/init/main.tf b/opentofu/gcp/gke/init/main.tf index 1d35a6a57..36af81c86 100644 --- a/opentofu/gcp/gke/init/main.tf +++ b/opentofu/gcp/gke/init/main.tf @@ -143,9 +143,8 @@ module "gke" { # so nothing new is ever provisioned and the slice's whole premise is untested. # # The ceiling is design criterion 16: an oversized workload must stay - # Unschedulable rather than growing the cluster without bound. It is set low on - # purpose -- this is a reference platform, and the failure mode of a too-high - # limit is a bill rather than an error. + # Unschedulable rather than growing the cluster without bound. Why the specific + # numbers are what they are lives on the variables in variables.tf. # # image_type MUST match the static pool's (criterion 14). Cilium's DaemonSet is # built around a writable /home/kubernetes/bin on Container-Optimized OS; an diff --git a/scripts/flux-schema/gen-catalog.sh b/scripts/flux-schema/gen-catalog.sh index c68a2fb9c..1924edf5c 100755 --- a/scripts/flux-schema/gen-catalog.sh +++ b/scripts/flux-schema/gen-catalog.sh @@ -1,12 +1,10 @@ #!/usr/bin/env bash # Build the local JSON-Schema catalog consumed by `flux schema validate`. # -# Three sources (SPEC-007 FR-002): +# Four sources (SPEC-007 FR-002): # 1. The repo's own Crossplane XRDs -> cloud.ogenki.io/* # 2. Envoy AI Gateway CRDs -> aigateway.envoyproxy.io/* # (absent from the hosted ecosystem catalog) -# 4. GKE ComputeClass CRD -> cloud.google.com/v1 ComputeClass -# (VENDORED, not rendered: GKE installs it and publishes no chart) # 3. Karpenter CRDs -> karpenter.k8s.aws/*, karpenter.sh/* # (PRESENT in the hosted ecosystem catalog but STALE: it pins an older # provider release that predates fields we use — e.g. EC2NodeClass @@ -14,6 +12,9 @@ # variant. Generating them here from the same OCI pin Flux installs makes # the local catalog win over the stale ecosystem entry, matching the # deployed CRD exactly.) +# 4. GKE ComputeClass CRD -> cloud.google.com/v1 ComputeClass +# (VENDORED rather than rendered: unlike the three above, GKE installs this +# CRD itself and publishes no chart to render it from.) # # The catalog is generated, never committed, so it cannot drift from the XRDs. # From 7b19951d7b91b03c494600c38b9e3128ea959621 Mon Sep 17 00:00:00 2001 From: Smana Date: Mon, 24 Aug 2026 13:26:39 +0200 Subject: [PATCH 6/7] docs(gcp): io ComputeClass verified by live scale-up Upgrades io from 'schema-valid, unexercised' to verified. It provisioned an n2-standard-4 spot node with a 375 GiB Local SSD attached -- 368 GiB ephemeral storage on a 50 GB boot disk is the giveaway -- and the probe pod reached Running. Tested because the GPU result made a second quota blocker look likely: PREEMPTIBLE_LOCAL_SSD_GB reads 0 in europe-west4, and io is spot + localSSDCount 1, which appeared to be exactly the gpu-l4 situation again. It is not. GKE-managed Local SSD on spot nodes provisions regardless of that metric. The prediction was wrong, and the false alarm is recorded in the design so the next person does not re-derive it and 'fix' a class that works -- the regional LOCAL_SSD_TOTAL_GB reads effectively unlimited, so the two quotas together are genuinely confusing. Per-class status is now explicit in the design: general-purpose and io VERIFIED, gpu-l4 proven up to provisioning and blocked on GPUS_ALL_REGIONS: 0. Verified: ./scripts/validate-links.sh -> all relative links resolve. --- .../superpowers/specs/2026-08-18-gcp-support-design.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/docs/superpowers/specs/2026-08-18-gcp-support-design.md b/docs/superpowers/specs/2026-08-18-gcp-support-design.md index ad7014af3..8d0dfe4a4 100644 --- a/docs/superpowers/specs/2026-08-18-gcp-support-design.md +++ b/docs/superpowers/specs/2026-08-18-gcp-support-design.md @@ -354,6 +354,16 @@ Falsifiable, verified against a live cluster. **Slice 4 (autoscaling)** — *results recorded 2026-08-24, measured on gcp-mycluster-0. Four PASS, one partial, one blocked on a GCP quota. Each is annotated below.* +*Per-class status: `general-purpose` and `io` are both VERIFIED by live scale-up — +`io` provisioned an `n2-standard-4` spot node with a 375 GiB Local SSD attached +(368 GiB ephemeral on a 50 GB boot disk). `gpu-l4` is proven up to the point of +provisioning and then blocked by `GPUS_ALL_REGIONS: 0` (see criterion 15).* + +*A predicted second quota blocker did NOT materialise: `PREEMPTIBLE_LOCAL_SSD_GB` +reads 0 in `europe-west4`, which looked like it would stop `io` for the same +reason GPU is stopped. It does not — GKE-managed Local SSD on spot nodes +provisions regardless. Recorded so nobody re-derives the false alarm.* + 12. A **freshly auto-created** node carries `node.cilium.io/agent-not-ready` at registration, and Cilium clears it. *This is the criterion the slice exists to test.* From f16ce56341236cc86ee581fe3fcb7602629dca18 Mon Sep 17 00:00:00 2001 From: Smana Date: Mon, 24 Aug 2026 13:29:21 +0200 Subject: [PATCH 7/7] =?UTF-8?q?fix(gcp):=20gpu-l4=20could=20never=20have?= =?UTF-8?q?=20provisioned=20=E2=80=94=20pd-standard=20is=20unsupported=20o?= =?UTF-8?q?n=20G2?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Correctness review found a blocker that no gate here could have caught, plus two smaller real issues. One of its suspicions is refuted by measurement. BLOCKER: gpu-l4 was unprovisionable. cluster_autoscaling sets disk_type = "pd-standard" as the boot disk for EVERY auto-created pool, and gpu-l4 overrode nothing. The G2 series does not support pd-standard -- Google's PD matrix allows only pd-ssd, pd-balanced and hyperdisk there, and the ComputeClass CRD says so itself: "available boot disk types depend on the machine family". The failure mode is the bad one: NAP either filters g2 out as unsatisfiable or the pool create fails on disk type, and with whenUnsatisfiable: DoNotScaleUp no event names the disk. A GPU pod just stays Pending -- exactly the silent mode the gpu_resources comment warns about. `kubectl apply --dry-run=server` provably cannot catch it: it never reaches VM creation. Nor could tofu validate, or the manifest gate, or the live test, because GPUS_ALL_REGIONS: 0 stopped us one step earlier. Two independent blockers were stacked and only the outer one was visible. Fixed with `storage.bootDiskType: pd-balanced` on the g2 priority rather than by changing the cluster-wide default: e2 and n2 both support pd-standard and are the common case, so only GPU nodes pay for the dearer disk. CONFIRMED: the autoscaling ceiling is cluster-wide, not per-class. GKE counts manually created pools toward max_cpu_cores. With the static pool at its max of 3 x e2-standard-4 (12 vCPU), only 20 of the 32 vCPU remain for all three classes combined -- so gpu_resources maximum 2 (8 vCPU) is reachable only while general-purpose and io stay under 12. A GPU scale-up can be starved by CPU scale-up, and the events do not distinguish the two. Documented on the variable. SUSPECTED, recorded not fixed: GKE applies its own nvidia.com/gpu=present:NoSchedule to GPU nodes, and Kubernetes forbids two taints sharing key+effect. Either GKE's value wins -- making our declared "true" inaccurate -- or the pool create errors. The toleration this file recommends uses operator: Exists, which survives both, so the block now says so explicitly and asks whoever sees the first GPU node to confirm and correct it. Guessing would be worse than flagging. REFUTED by measurement: the review suspected localSSDCount attaches raw NVMe block rather than pod ephemeral storage, which would have broken the AWS parity claim. The io node provisioned earlier reported 385926528Ki (~368 GiB) of ephemeral-storage on a 50 GB boot disk -- that is the Local SSD and nothing else. Recorded in io.yaml so the question is not reopened. Verified: ./scripts/validate-manifests.sh -> Valid: 1191, Invalid: 0, Skipped: 0; the API accepts the modified gpu-l4; tofu validate and fmt clean. --- .../gcp-mycluster-0/computeclass/gpu-l4.yaml | 27 +++++++++++++++++++ .../gcp-mycluster-0/computeclass/io.yaml | 9 ++++++- opentofu/gcp/gke/init/variables.tf | 11 ++++++++ 3 files changed, 46 insertions(+), 1 deletion(-) diff --git a/infrastructure/gcp-mycluster-0/computeclass/gpu-l4.yaml b/infrastructure/gcp-mycluster-0/computeclass/gpu-l4.yaml index b90b67af1..613157e06 100644 --- a/infrastructure/gcp-mycluster-0/computeclass/gpu-l4.yaml +++ b/infrastructure/gcp-mycluster-0/computeclass/gpu-l4.yaml @@ -30,6 +30,24 @@ spec: gpu: type: nvidia-l4 count: 1 + storage: + # REQUIRED, and the only reason this block exists. NAP's cluster-wide + # boot-disk default is `pd-standard` (opentofu/gcp/gke/init/main.tf), and + # the G2 series does NOT support it — Google's PD matrix allows only + # pd-ssd, pd-balanced and hyperdisk on G2. The CRD says as much: + # "available boot disk types depend on the machine family". + # + # Left unset, this class can never provision: NAP either filters g2 out + # as unsatisfiable or the pool create fails on disk type, and with + # `whenUnsatisfiable: DoNotScaleUp` no event names the disk. A GPU pod + # simply stays Pending — the silent mode the gpu_resources note warns + # about. `--dry-run=server` cannot catch it either: it never reaches VM + # creation. + # + # Overridden HERE rather than by changing the cluster-wide default, + # because e2 and n2 both support pd-standard and are the common case. + # Only GPU nodes pay for the dearer disk. + bootDiskType: pd-balanced # Spot GPUs are markedly scarcer than spot CPU, so this class will sometimes # provision nothing at all. That is the intended behaviour on a test cluster: @@ -56,6 +74,15 @@ spec: # `nvidia.com/gpu` is the conventional accelerator taint, identical to the # AWS pool's, so a GPU workload's tolerations are portable between clouds # even though everything under them differs. + # + # UNVERIFIED (no GPU node has existed — see the quota note in the design's + # criterion 15): GKE applies its own `nvidia.com/gpu=present:NoSchedule` to + # GPU nodes, and Kubernetes forbids two taints sharing key+effect. So either + # GKE's value wins and the `"true"` below is inaccurate, or the pool create + # errors. The toleration shown above uses `operator: Exists`, which survives + # both outcomes — use that form, NOT `operator: Equal` with a value. Confirm + # the node's real taints on the first successful GPU scale-up and correct + # this block. taints: - key: node.cilium.io/agent-not-ready value: "true" diff --git a/infrastructure/gcp-mycluster-0/computeclass/io.yaml b/infrastructure/gcp-mycluster-0/computeclass/io.yaml index 988b932f2..7ecf134ce 100644 --- a/infrastructure/gcp-mycluster-0/computeclass/io.yaml +++ b/infrastructure/gcp-mycluster-0/computeclass/io.yaml @@ -22,7 +22,14 @@ spec: # # localSSDCount is what makes this class different from general-purpose. GCP # attaches Local SSD as fixed 375 GiB devices, so the count is a device count, - # not a size — one is ample for the ephemeral-storage workloads this exists for. + # not a size. + # + # MEASURED 2026-08-24, because it was not obvious: the Local SSD does back POD + # EPHEMERAL STORAGE, it is not merely a raw block device. A provisioned node + # reported 385926528Ki (~368 GiB) of ephemeral-storage capacity on a 50 GB boot + # disk, which is the Local SSD and nothing else. So this class does give + # ephemeral-hungry pods what they need, and the parity with the AWS io pool + # (which uses `instanceStorePolicy: RAID0` to the same end) holds. # # Both entries are n2: the smaller first so a modest IO workload does not # provision a large node, with the larger as the capacity fallback. Sticking to diff --git a/opentofu/gcp/gke/init/variables.tf b/opentofu/gcp/gke/init/variables.tf index 17defe990..5ddf77973 100644 --- a/opentofu/gcp/gke/init/variables.tf +++ b/opentofu/gcp/gke/init/variables.tf @@ -98,6 +98,17 @@ variable "tags" { # # The static pool is 2-3 x e2-standard-4 (4 vCPU / 16 GiB each), so this leaves # room for roughly four more comparable nodes before the ceiling bites. +# +# IMPORTANT: this ceiling is CLUSTER-WIDE. GKE counts manually created pools +# toward it, not just auto-provisioned ones, and it is shared across every +# ComputeClass. With the static pool at its max of 3 nodes (12 vCPU / 48 GiB), +# only 20 vCPU remain for general-purpose, io and gpu-l4 combined. +# +# The consequence worth knowing: `gpu_resources` maximum 2 (2 x g2-standard-4 = +# 8 vCPU) is reachable only while the other classes stay under 12 vCPU. A GPU +# scale-up can therefore be starved by CPU scale-up, and the two are +# indistinguishable from the events -- both simply fail to provision. Raise this +# ceiling before relying on GPU capacity under load. variable "autoscaling_max_cpu_cores" { description = "Total vCPU ceiling across all auto-provisioned node pools" type = number