diff --git a/clusters/gcp-mycluster-0/infrastructure.yaml b/clusters/gcp-mycluster-0/infrastructure.yaml new file mode 100644 index 000000000..d9c0d2a36 --- /dev/null +++ b/clusters/gcp-mycluster-0/infrastructure.yaml @@ -0,0 +1,35 @@ +# GCP-only infrastructure for this cluster. +# +# Points at infrastructure/gcp-mycluster-0, NOT at infrastructure/base — that +# tree is AWS-shaped (Karpenter, aws-load-balancer-controller, EKS Pod +# Identities) and applying it here would fail on resources whose CRDs do not +# exist. The GCP tree is deliberately minimal until the design's slices 6-7 +# settle which parts of the shared tree are genuinely cloud-neutral. +# +# It sources infra-artifact, which flux/artifact-generators/monorepo-split.yaml +# already builds from `@repo/infrastructure/**` — so the new directory needs no +# generator change, only this Kustomization. +# +# No dependsOn: crds. The only resource here is a ComputeClass, whose CRD is +# installed by GKE itself (`cloud.google.com/v1`, present on a fresh cluster) +# rather than by crds/base. Adding that edge would couple this to a Kustomization +# it does not need and delay it behind the whole upstream CRD set. +apiVersion: kustomize.toolkit.fluxcd.io/v1 +kind: Kustomization +metadata: + name: infrastructure + namespace: flux-system +spec: + prune: true + interval: 1m0s + path: ./infrastructure/gcp-mycluster-0 + sourceRef: + kind: ExternalArtifact + name: infra-artifact + dependsOn: + # Currently inert -- a ComputeClass is cluster-scoped and this path holds + # nothing namespaced. Kept as future-proofing: this is the GCP cluster's only + # infrastructure Kustomization, so the first namespaced resource added here + # would otherwise race its namespace, which is a failure that presents as a + # dependency cascade rather than as a missing namespace. + - name: namespaces diff --git a/docs/superpowers/plans/2026-08-23-gcp-foundation-resume.md b/docs/superpowers/plans/2026-08-23-gcp-foundation-resume.md index 6fa75a71a..6f7d27a34 100644 --- a/docs/superpowers/plans/2026-08-23-gcp-foundation-resume.md +++ b/docs/superpowers/plans/2026-08-23-gcp-foundation-resume.md @@ -97,6 +97,20 @@ flux reconcile source git flux-system failure mode, `depends_on` gaps, and the `clusters/gcp-mycluster-0/` Flux wiring. Two review agents produced nothing across repeated asks. +### GPU quota — blocks slice 4's last criterion + +`GPUS_ALL_REGIONS` on project `ogenki-435905` is **0**, so no GPU node can be +created at any price. Criterion 15 ("a GPU pod with no `runtimeClassName` sees +the device via `nvidia-smi`") is therefore blocked rather than failed — the +`gpu-l4` ComputeClass is proven correct up to the point of provisioning. + +Do **not** diagnose this from the regional quota: `NVIDIA_L4_GPUS` in +`europe-west4` shows a limit of 1, which looks fine and is meaningless while the +global cap is zero. Spot attempts fail with `GCE out of resources` and on-demand +with `GCE quota exceeded`; only the second points at the real cause. + +Closing it needs a quota increase request to Google, not a retry. + ### ClusterMesh prerequisites (recorded in ADR-0017, not implemented) - `cluster.id` is unset on both clusters; ClusterMesh needs a unique 1–255 ID. diff --git a/docs/superpowers/specs/2026-08-18-gcp-support-design.md b/docs/superpowers/specs/2026-08-18-gcp-support-design.md index 39a83bc60..8d0dfe4a4 100644 --- a/docs/superpowers/specs/2026-08-18-gcp-support-design.md +++ b/docs/superpowers/specs/2026-08-18-gcp-support-design.md @@ -351,16 +351,52 @@ Falsifiable, verified against a live cluster. 11. A written monthly run-rate estimate exists (cluster fee, static pool, Cloud NAT, Cloud DNS, Tailscale instance) stating the zonal-vs-regional choice and its price delta. -**Slice 4 (autoscaling)** +**Slice 4 (autoscaling)** — *results recorded 2026-08-24, measured on gcp-mycluster-0. +Four PASS, one partial, one blocked on a GCP quota. Each is annotated below.* + +*Per-class status: `general-purpose` and `io` are both VERIFIED by live scale-up — +`io` provisioned an `n2-standard-4` spot node with a 375 GiB Local SSD attached +(368 GiB ephemeral on a 50 GB boot disk). `gpu-l4` is proven up to the point of +provisioning and then blocked by `GPUS_ALL_REGIONS: 0` (see criterion 15).* + +*A predicted second quota blocker did NOT materialise: `PREEMPTIBLE_LOCAL_SSD_GB` +reads 0 in `europe-west4`, which looked like it would stop `io` for the same +reason GPU is stopped. It does not — GKE-managed Local SSD on spot nodes +provisions regardless. Recorded so nobody re-derives the false alarm.* + 12. A **freshly auto-created** node carries `node.cilium.io/agent-not-ready` at registration, and Cilium clears it. *This is the criterion the slice exists to test.* + - **PASS.** Set via `spec.nodePoolConfig.taints`; after scale-up the nodes retained only GKE's + own `cloud.google.com/compute-class` taint. **Unanticipated finding:** every workload + targeting a ComputeClass must *tolerate* that taint or NOTHING SCALES UP — the autoscaler + judges an untolerating pod unplaceable on a node that will carry it. Ten minutes Pending, + no `TriggeredScaleUp` at all. See ADR-0006. 13. Across 5 scale-up cycles, 0 pods record `FailedCreatePodSandBox` referencing a missing CNI. + - **PARTIAL.** As worded it holds — nothing recorded a *missing* CNI. But tolerating pods can + land before the Cilium agent is up and log `plugin type="cilium-cni" failed (add) ... EOF`, + which is the CNI present with its agent starting. Self-heals (0 restarts); + `kube-system/metrics-server` hits the same on any fresh node. The toleration required by + criterion 12 is what reopens this window. 14. `imageType` on every auto-created pool matches the pinned value; general-purpose nodes are spot with **zero** on-demand fallback. + - **PASS.** Auto-created nodes came up `e2-highcpu-4` with `spot=true` on `cos_containerd`. + NAP chose `highcpu` over `standard` unprompted — cheaper per vCPU. 15. A GPU pod with **no** `runtimeClassName` sees the device via `nvidia-smi`. + - **BLOCKED — not failed.** Untestable in this project: `GPUS_ALL_REGIONS` is **0**, so no GPU + node can be created at any price. The per-region `NVIDIA_L4_GPUS: 1` is meaningless beneath + it — checking only the regional quota would wrongly suggest retrying later. + Spot attempt gave `GCE out of resources`; on-demand gave `GCE quota exceeded`. + Everything up to the GPU is proven: the class is selected, NAP resolves it to a + `g2-standard-4-gpu1` pool and attempts creation — which also confirms + `cluster_autoscaling.gpu_resources` is load-bearing (left empty, no scale-up is even + attempted). **Needs a GPUS_ALL_REGIONS quota grant from Google to close.** 16. Cluster `resourceLimits` set; an oversized workload stays `Unschedulable` at the ceiling. + - **PASS.** A 64-vCPU pod against a 32-vCPU ceiling held `Pending` for 4 minutes, node count + never moved, autoscaler logged `NotTriggerScaleUp`. It refuses rather than grinding. 17. Empty auto-created pools are removed on scale-down. + - **PASS.** All four auto-created nodes were reaped after the probe was deleted; back to the + 2 static nodes with no intervention. `OPTIMIZE_UTILIZATION` was chosen for this. **Slice 5 (identity)** diff --git a/infrastructure/gcp-mycluster-0/computeclass/general-purpose.yaml b/infrastructure/gcp-mycluster-0/computeclass/general-purpose.yaml new file mode 100644 index 000000000..0b59af398 --- /dev/null +++ b/infrastructure/gcp-mycluster-0/computeclass/general-purpose.yaml @@ -0,0 +1,56 @@ +# General-purpose ComputeClass — the default shape for auto-provisioned nodes. +# +# The GCP counterpart to infrastructure/base/karpenter-nodepools/default-nodepool.yaml. +# ADR-0006 chose GKE node auto-provisioning over Karpenter here; this object is +# what turns that decision into running nodes, telling NAP which machine shapes +# to try and in what order. +apiVersion: cloud.google.com/v1 +kind: ComputeClass +metadata: + name: general-purpose +spec: + # Ordered fallback: NAP walks these in sequence and takes the first it can + # provision, so this is a preference order rather than a set. + # + # e2 first because it is what the static pool runs and it is the cheapest + # general-purpose family. n2 second as a spot-availability fallback: a GKE node + # pool takes ONE machine type, so a single family concentrates interruption + # risk — the breadth the static pool's own comment defers to ComputeClass. + priorities: + - machineFamily: e2 + spot: true + - machineFamily: n2 + spot: true + + # No on-demand entry, deliberately (design criterion 14). On a test cluster an + # unnoticed fallback to on-demand is a silent cost regression, and these + # workloads are restartable. The honest consequence: when no spot capacity + # exists in either family, pods stay Pending rather than becoming expensive. + whenUnsatisfiable: DoNotScaleUp + + nodePoolConfig: + # ┌──────────────────────────────────────────────────────────────────────┐ + # │ WORKLOADS TARGETING THIS CLASS MUST TOLERATE THIS TAINT │ + # └──────────────────────────────────────────────────────────────────────┘ + # + # tolerations: + # - key: node.cilium.io/agent-not-ready + # operator: Exists + # effect: NoSchedule + # + # Not optional: WITHOUT it nothing scales up at all — the autoscaler judges + # an untolerating pod unplaceable on a node that will carry the taint and + # never provisions one. Measured 2026-08-24. Full rationale, the trade it + # accepts, and the AWS divergence are in ADR-0006 (Negative consequences); + # this file states the requirement, the ADR explains it. + taints: + - key: node.cilium.io/agent-not-ready + value: "true" + effect: NoSchedule + + # Must match the static pool and the NAP default — see the image_type note in + # opentofu/gcp/gke/init/main.tf for why COS specifically. + imageType: cos_containerd + + nodePoolAutoCreation: + enabled: true diff --git a/infrastructure/gcp-mycluster-0/computeclass/gpu-l4.yaml b/infrastructure/gcp-mycluster-0/computeclass/gpu-l4.yaml new file mode 100644 index 000000000..613157e06 --- /dev/null +++ b/infrastructure/gcp-mycluster-0/computeclass/gpu-l4.yaml @@ -0,0 +1,99 @@ +# GPU ComputeClass — the GCP counterpart to +# infrastructure/base/karpenter-nodepools-gpu/gpu-l4-nodepool.yaml. +# +# NVIDIA L4, matching the AWS pool's accelerator choice and the reason +# europe-west4 was picked in the first place: opentofu/config.tm.hcl records that +# nvidia-l4 exists in all three of its zones and NOT AT ALL in europe-west9, +# which would otherwise have been the geographic match for eu-west-3. +# +# NO runtimeClassName IS NEEDED HERE, and that is a real divergence from AWS +# rather than an omission. On AWS the Bottlerocket NVIDIA variant requires +# `runtimeClassName: nvidia` and a matching RuntimeClass +# (infrastructure/base/runtimeclass-nvidia/). GKE installs the driver through its +# own DaemonSet and wires the default runtime, so a GPU pod just requests +# `nvidia.com/gpu` and works -- which is exactly design criterion 15: "a GPU pod +# with NO runtimeClassName sees the device via nvidia-smi". +apiVersion: cloud.google.com/v1 +kind: ComputeClass +metadata: + name: gpu-l4 +spec: + # g2 is the only family that carries L4, so there is no fallback tier to list: + # either L4 spot capacity exists in this zone or nothing is provisioned. + # + # count: 1 mirrors the AWS pool's instance-gpu-count ["1"] -- one accelerator + # per node keeps the failure domain small and the bill legible, and nothing on + # this platform yet needs multi-GPU nodes. + priorities: + - machineFamily: g2 + spot: true + gpu: + type: nvidia-l4 + count: 1 + storage: + # REQUIRED, and the only reason this block exists. NAP's cluster-wide + # boot-disk default is `pd-standard` (opentofu/gcp/gke/init/main.tf), and + # the G2 series does NOT support it — Google's PD matrix allows only + # pd-ssd, pd-balanced and hyperdisk on G2. The CRD says as much: + # "available boot disk types depend on the machine family". + # + # Left unset, this class can never provision: NAP either filters g2 out + # as unsatisfiable or the pool create fails on disk type, and with + # `whenUnsatisfiable: DoNotScaleUp` no event names the disk. A GPU pod + # simply stays Pending — the silent mode the gpu_resources note warns + # about. `--dry-run=server` cannot catch it either: it never reaches VM + # creation. + # + # Overridden HERE rather than by changing the cluster-wide default, + # because e2 and n2 both support pd-standard and are the common case. + # Only GPU nodes pay for the dearer disk. + bootDiskType: pd-balanced + + # Spot GPUs are markedly scarcer than spot CPU, so this class will sometimes + # provision nothing at all. That is the intended behaviour on a test cluster: + # an L4 falling back to on-demand is the single most expensive thing this + # repository could do by accident. + whenUnsatisfiable: DoNotScaleUp + + nodePoolConfig: + # ┌──────────────────────────────────────────────────────────────────────┐ + # │ WORKLOADS MUST TOLERATE **BOTH** TAINTS BELOW │ + # └──────────────────────────────────────────────────────────────────────┘ + # + # tolerations: + # - key: node.cilium.io/agent-not-ready + # operator: Exists + # effect: NoSchedule + # - key: nvidia.com/gpu + # operator: Exists + # effect: NoSchedule + # + # The Cilium taint is not optional — without it nothing scales up. See + # general-purpose.yaml and ADR-0006. + # + # `nvidia.com/gpu` is the conventional accelerator taint, identical to the + # AWS pool's, so a GPU workload's tolerations are portable between clouds + # even though everything under them differs. + # + # UNVERIFIED (no GPU node has existed — see the quota note in the design's + # criterion 15): GKE applies its own `nvidia.com/gpu=present:NoSchedule` to + # GPU nodes, and Kubernetes forbids two taints sharing key+effect. So either + # GKE's value wins and the `"true"` below is inaccurate, or the pool create + # errors. The toleration shown above uses `operator: Exists`, which survives + # both outcomes — use that form, NOT `operator: Equal` with a value. Confirm + # the node's real taints on the first successful GPU scale-up and correct + # this block. + taints: + - key: node.cilium.io/agent-not-ready + value: "true" + effect: NoSchedule + - key: nvidia.com/gpu + value: "true" + effect: NoSchedule + + # Must match the static pool and the NAP default — see the image_type note in + # opentofu/gcp/gke/init/main.tf for why COS specifically. + imageType: cos_containerd + + nodePoolAutoCreation: + enabled: true diff --git a/infrastructure/gcp-mycluster-0/computeclass/io.yaml b/infrastructure/gcp-mycluster-0/computeclass/io.yaml new file mode 100644 index 000000000..7ecf134ce --- /dev/null +++ b/infrastructure/gcp-mycluster-0/computeclass/io.yaml @@ -0,0 +1,85 @@ +# IO-optimised ComputeClass — the GCP counterpart to +# infrastructure/base/karpenter-nodepools/io-nodepool.yaml. +# +# For workloads that want local NVMe rather than network-attached disk. The +# `ogenki/io` taint key is kept IDENTICAL to the AWS NodePool's on purpose: it is +# a platform convention, not a cloud detail, so a workload's toleration is the +# one part of its manifest that does not have to change between clouds. +apiVersion: cloud.google.com/v1 +kind: ComputeClass +metadata: + name: io +spec: + # machineType, NOT machineFamily — and that is forced, not stylistic. GKE's + # admission webhook rejects the family form outright: + # + # compute-class "io" doesn't support local ssd without machine type or + # GPU config + # + # Local SSD attachment is a per-machine-type property, so the class has to name + # the type. The cost is a narrower fallback list than general-purpose gets: + # two explicit sizes rather than "any n2". + # + # localSSDCount is what makes this class different from general-purpose. GCP + # attaches Local SSD as fixed 375 GiB devices, so the count is a device count, + # not a size. + # + # MEASURED 2026-08-24, because it was not obvious: the Local SSD does back POD + # EPHEMERAL STORAGE, it is not merely a raw block device. A provisioned node + # reported 385926528Ki (~368 GiB) of ephemeral-storage capacity on a 50 GB boot + # disk, which is the Local SSD and nothing else. So this class does give + # ephemeral-hungry pods what they need, and the parity with the AWS io pool + # (which uses `instanceStorePolicy: RAID0` to the same end) holds. + # + # Both entries are n2: the smaller first so a modest IO workload does not + # provision a large node, with the larger as the capacity fallback. Sticking to + # one family keeps this predictable; c3 would need its distinct `-lssd` machine + # types, which is a second thing to get right for no benefit on a test cluster. + priorities: + - machineType: n2-standard-4 + spot: true + storage: + localSSDCount: 1 + - machineType: n2-standard-8 + spot: true + storage: + localSSDCount: 1 + + # Same rationale as general-purpose: criterion 14 forbids on-demand fallback, + # so exhausted spot leaves pods Pending rather than quietly costing more. + whenUnsatisfiable: DoNotScaleUp + + nodePoolConfig: + # ┌──────────────────────────────────────────────────────────────────────┐ + # │ WORKLOADS MUST TOLERATE **BOTH** TAINTS BELOW │ + # └──────────────────────────────────────────────────────────────────────┘ + # + # tolerations: + # - key: node.cilium.io/agent-not-ready + # operator: Exists + # effect: NoSchedule + # - key: ogenki/io + # operator: Exists + # effect: NoSchedule + # + # The Cilium taint is not optional — without it nothing scales up. See + # general-purpose.yaml and ADR-0006. + # + # `ogenki/io` is this class's own: it keeps general-purpose workloads off + # nodes provisioned for their local disk, which are fewer and dearer. + # Identical key and value to the AWS io NodePool, so a workload's + # tolerations are portable between clouds. + taints: + - key: node.cilium.io/agent-not-ready + value: "true" + effect: NoSchedule + - key: ogenki/io + value: "true" + effect: NoSchedule + + # Must match the static pool and the NAP default — see the image_type note in + # opentofu/gcp/gke/init/main.tf for why COS specifically. + imageType: cos_containerd + + nodePoolAutoCreation: + enabled: true diff --git a/infrastructure/gcp-mycluster-0/computeclass/kustomization.yaml b/infrastructure/gcp-mycluster-0/computeclass/kustomization.yaml new file mode 100644 index 000000000..6b557dd94 --- /dev/null +++ b/infrastructure/gcp-mycluster-0/computeclass/kustomization.yaml @@ -0,0 +1,10 @@ +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization + +# The three classes of design slice 4. general-purpose was proven first, on its +# own, because it carried the unknown the slice existed to settle (the Cilium +# taint); io and gpu-l4 follow the pattern it established. +resources: + - general-purpose.yaml + - io.yaml + - gpu-l4.yaml diff --git a/infrastructure/gcp-mycluster-0/kustomization.yaml b/infrastructure/gcp-mycluster-0/kustomization.yaml new file mode 100644 index 000000000..58464989d --- /dev/null +++ b/infrastructure/gcp-mycluster-0/kustomization.yaml @@ -0,0 +1,16 @@ +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization + +# GCP-only infrastructure for gcp-mycluster-0. +# +# Separate from infrastructure/mycluster-0/ rather than an overlay on it: that +# tree is AWS-shaped (Karpenter NodePools, aws-load-balancer-controller, +# EKS Pod Identities) and shares no resources with this one. A ComputeClass has +# no AWS counterpart at all -- ADR-0006 chose NAP over Karpenter precisely +# because the two autoscalers do not share an API. +# +# This is the manifest-side counterpart to the opentofu/{aws,gcp,shared} split, +# and is deliberately minimal until the design's slices 6-7 settle which parts of +# the shared tree are genuinely cloud-neutral. +resources: + - computeclass diff --git a/opentofu/gcp/gke/init/main.tf b/opentofu/gcp/gke/init/main.tf index 5ce5542e1..36af81c86 100644 --- a/opentofu/gcp/gke/init/main.tf +++ b/opentofu/gcp/gke/init/main.tf @@ -136,6 +136,72 @@ module "gke" { # `terramate script run destroy` fail rather than protect anything of value. deletion_protection = false + # ── 6. NODE AUTO-PROVISIONING (ADR-0006) ────────────────────────────────── + # This is what makes a ComputeClass able to CREATE node pools rather than only + # select among existing ones. Without it a ComputeClass with + # nodePoolAutoCreation still schedules, but only onto pools that already exist, + # so nothing new is ever provisioned and the slice's whole premise is untested. + # + # The ceiling is design criterion 16: an oversized workload must stay + # Unschedulable rather than growing the cluster without bound. Why the specific + # numbers are what they are lives on the variables in variables.tf. + # + # image_type MUST match the static pool's (criterion 14). Cilium's DaemonSet is + # built around a writable /home/kubernetes/bin on Container-Optimized OS; an + # auto-created node on a different image would fail the same way the very first + # GKE deploy did, but only on nodes nobody created by hand. + # + # OPTIMIZE_UTILIZATION over BALANCED: criterion 17 wants empty auto-created + # pools removed on scale-down, and BALANCED is deliberately reluctant to do so. + # The trade is more pod churn, which is acceptable here and would not be on a + # latency-sensitive platform. + cluster_autoscaling = { + enabled = true + autoscaling_profile = "OPTIMIZE_UTILIZATION" + + min_cpu_cores = 0 + max_cpu_cores = var.autoscaling_max_cpu_cores + min_memory_gb = 0 + max_memory_gb = var.autoscaling_max_memory_gb + + # REQUIRED for the gpu-l4 ComputeClass to provision anything. NAP will not + # create a node with an accelerator that has no resourceLimits entry, so an + # empty list here silently caps GPU autoscaling at zero -- the class applies + # cleanly, pods stay Pending, and nothing says why. + # + # nvidia-l4 only, matching the class and the reason europe-west4 was chosen + # (see opentofu/config.tm.hcl). Maximum 2 is a test-cluster ceiling: L4s are + # the most expensive thing this repository can provision, and criterion 16 + # wants an oversized workload to stay Unschedulable rather than scale into a + # bill. + gpu_resources = [ + { + resource_type = "nvidia-l4" + minimum = 0 + maximum = 2 + }, + ] + + auto_repair = true + auto_upgrade = true + + image_type = var.node_image_type + + # COST. This is a test cluster that gets rebuilt, so the cheap option wins + # wherever it is not actively misleading. + # + # Both of these are otherwise left on module defaults of 100 GB pd-standard + # -- TWICE the static pool's disk, which is the sort of thing that costs + # money quietly because nobody set it. + # + # 50 GB matches the static pool rather than being an independent guess. + # pd-standard is the cheapest disk type; the static pool uses pd-balanced, + # so if auto-created nodes ever behave worse than hand-created ones under + # image pulls or log writes, this asymmetry is the first thing to look at. + disk_size = var.node_disk_size_gb + disk_type = "pd-standard" + } + node_pools = [ { name = "static" diff --git a/opentofu/gcp/gke/init/variables.tf b/opentofu/gcp/gke/init/variables.tf index f13ee245e..5ddf77973 100644 --- a/opentofu/gcp/gke/init/variables.tf +++ b/opentofu/gcp/gke/init/variables.tf @@ -87,3 +87,36 @@ variable "tags" { type = map(string) default = {} } + +# ── Node auto-provisioning ceiling ────────────────────────────────────────── +# Design criterion 16: cluster resourceLimits are set, and an oversized workload +# stays Unschedulable at the ceiling rather than growing the cluster to fit it. +# +# Deliberately small. This is a reference platform that gets rebuilt, so the cost +# of a limit that is too LOW is an Unschedulable pod and a one-line change; the +# cost of one that is too HIGH is a bill nobody notices until it arrives. +# +# The static pool is 2-3 x e2-standard-4 (4 vCPU / 16 GiB each), so this leaves +# room for roughly four more comparable nodes before the ceiling bites. +# +# IMPORTANT: this ceiling is CLUSTER-WIDE. GKE counts manually created pools +# toward it, not just auto-provisioned ones, and it is shared across every +# ComputeClass. With the static pool at its max of 3 nodes (12 vCPU / 48 GiB), +# only 20 vCPU remain for general-purpose, io and gpu-l4 combined. +# +# The consequence worth knowing: `gpu_resources` maximum 2 (2 x g2-standard-4 = +# 8 vCPU) is reachable only while the other classes stay under 12 vCPU. A GPU +# scale-up can therefore be starved by CPU scale-up, and the two are +# indistinguishable from the events -- both simply fail to provision. Raise this +# ceiling before relying on GPU capacity under load. +variable "autoscaling_max_cpu_cores" { + description = "Total vCPU ceiling across all auto-provisioned node pools" + type = number + default = 32 +} + +variable "autoscaling_max_memory_gb" { + description = "Total memory ceiling in GiB across all auto-provisioned node pools" + type = number + default = 128 +} diff --git a/scripts/flux-schema/gen-catalog.sh b/scripts/flux-schema/gen-catalog.sh index 61f47117c..1924edf5c 100755 --- a/scripts/flux-schema/gen-catalog.sh +++ b/scripts/flux-schema/gen-catalog.sh @@ -1,7 +1,7 @@ #!/usr/bin/env bash # Build the local JSON-Schema catalog consumed by `flux schema validate`. # -# Three sources (SPEC-007 FR-002): +# Four sources (SPEC-007 FR-002): # 1. The repo's own Crossplane XRDs -> cloud.ogenki.io/* # 2. Envoy AI Gateway CRDs -> aigateway.envoyproxy.io/* # (absent from the hosted ecosystem catalog) @@ -12,6 +12,9 @@ # variant. Generating them here from the same OCI pin Flux installs makes # the local catalog win over the stale ecosystem entry, matching the # deployed CRD exactly.) +# 4. GKE ComputeClass CRD -> cloud.google.com/v1 ComputeClass +# (VENDORED rather than rendered: unlike the three above, GKE installs this +# CRD itself and publishes no chart to render it from.) # # The catalog is generated, never committed, so it cannot drift from the XRDs. # @@ -188,6 +191,10 @@ echo "==> Extracting JSON Schemas into ${build_dir}/" "${FLUX_BIN}" schema extract crd "${tmp}/aigateway-crds.yaml" -d "${build_dir}" "${FLUX_BIN}" schema extract crd "${tmp}/karpenter-crds.yaml" -d "${build_dir}" "${FLUX_BIN}" schema extract crd "${tmp}/barman-crds.yaml" -d "${build_dir}" +# GKE ComputeClass. Vendored rather than rendered: unlike the three above, GKE +# installs this CRD itself and publishes no chart to render it from. See the +# header of the file for how it was captured and when to refresh it. +"${FLUX_BIN}" schema extract crd "${REPO_ROOT}/scripts/flux-schema/vendored-crds/gke-computeclass.yaml" -d "${build_dir}" echo "==> Verifying the catalog is complete" for kind in app sqlinstance inferenceservice epi; do diff --git a/scripts/flux-schema/vendored-crds/gke-computeclass.yaml b/scripts/flux-schema/vendored-crds/gke-computeclass.yaml new file mode 100644 index 000000000..bdfca1d3e --- /dev/null +++ b/scripts/flux-schema/vendored-crds/gke-computeclass.yaml @@ -0,0 +1,2937 @@ +# VENDORED, NOT APPLIED. Do not move this under crds/ -- Flux applies that tree, +# and this CRD is installed and reconciled by GKE's own addon manager +# (addonmanager.kubernetes.io/mode: Reconcile). Applying it would fight GKE. +# +# It exists solely so `flux schema validate` has a schema for +# cloud.google.com/v1 ComputeClass. The repo runs with skipMissingSchemas: false +# by design -- an unknown Kind FAILS the build rather than passing unvalidated -- +# and ComputeClass is in no public catalog: unlike the Envoy AI Gateway and +# Karpenter CRDs, which gen-catalog.sh renders from pinned Helm charts, GKE +# installs this one itself and publishes no chart to render. +# +# Captured with: +# kubectl get crd computeclasses.cloud.google.com -o yaml +# from gcp-mycluster-0, with cluster-specific metadata (uid, resourceVersion, +# managedFields, status) stripped. +# +# Provenance is in the annotations below: components.gke.io/component-version +# records the GKE component release this was taken from. Refresh it when the +# cluster's GKE version moves and a ComputeClass field you need is missing -- +# a stale copy shows up as a validation error naming the field, not as silence. +apiVersion: apiextensions.k8s.io/v1 +kind: CustomResourceDefinition +metadata: + annotations: + components.gke.io/component-name: clusterautoscaler + components.gke.io/component-version: 35.202.10-gke.1 + components.gke.io/layer: addon + controller-gen.kubebuilder.io/version: v0.19.0 + labels: + addonmanager.kubernetes.io/mode: Reconcile + name: computeclasses.cloud.google.com +spec: + conversion: + strategy: None + group: cloud.google.com + names: + kind: ComputeClass + listKind: ComputeClassList + plural: computeclasses + shortNames: + - cc + - ccs + singular: computeclass + scope: Cluster + versions: + - name: v1 + schema: + openAPIV3Schema: + description: 'ComputeClass is a way to impact Cluster Autoscaler scaling + + decisions based on user preferences. It gives control over preference of + + hardware to be selected by Cluster Autoscaler. + + Given ComputeClass affects only workloads using workload separation + + label equal to CCs name, except ComputeClass with name default + + which will be used for workloads not specifying any preferences.' + properties: + apiVersion: + description: 'APIVersion defines the versioned schema of this representation + of an object. + + Servers should convert recognized schemas to the latest internal value, + and + + may reject unrecognized values. + + More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources' + type: string + kind: + description: 'Kind is a string value representing the REST resource this + object represents. + + Servers may infer this from the endpoint the client submits requests + to. + + Cannot be updated. + + In CamelCase. + + More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds' + type: string + metadata: + type: object + spec: + description: 'Specification of the ComputeClass object. + + More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#spec-and-status.' + properties: + activeMigration: + description: 'ActiveMigration describes settings related to active + reconciliation of + + a given ComputeClass.' + properties: + ensureAllDaemonSetPodsRunning: + description: 'EnsureAllDaemonSetPodsRunning defines whether node + pools should be migrated + + to larger ones to ensure that all daemon sets are schedulable.' + type: boolean + optimizeRulePriority: + default: false + description: 'OptimizeRulePriority defines whether workloads affected + by given + + ComputeClass should be migrated to nodepool defined by higher + priority rule, if possible.' + type: boolean + required: + - optimizeRulePriority + type: object + autopilot: + description: Autopilot describes the autopilot settings for a given + ComputeClass. + properties: + enabled: + default: false + description: Enabled indicates whether nodes created for this + compute class should be Autopilot managed. + type: boolean + x-kubernetes-validations: + - message: Autopilot is immutable + rule: self == oldSelf + required: + - enabled + type: object + x-kubernetes-validations: + - message: Autopilot is immutable + rule: self == oldSelf + autoscalingPolicy: + description: 'AutoscalingPolicy describes settings related to active + reconciliation of + + a given ComputeClass.' + properties: + consolidationDelayMinutes: + description: 'ConsolidationDelayMinutes determines how long a + node should be unneeded before it is eligible for scale down. + + Minimum duration is 1 minute, maximum is 24 hours or 1440 minutes' + maximum: 1440 + minimum: 1 + type: integer + consolidationThreshold: + description: ConsolidationThreshold determines resource utilization + threshold below which a node can be considered for scale down. + maximum: 100 + minimum: 0 + type: integer + gpuConsolidationThreshold: + description: 'GPUConsolidationThreshold determines GPU resource + utilization threshold below which a node can be considered for + scale down. + + Utilization calculation only cares about GPU resource for accelerator + node, CPU and memory utilization will be ignored.' + maximum: 100 + minimum: 0 + type: integer + type: object + description: + description: 'Description is an arbitrary string that usually provides + guidelines on + + when this compute class should be used.' + type: string + nodePoolAutoCreation: + default: + enabled: false + description: 'NodePoolAutoCreation describes the auto provisioning + settings for a given + + ComputeClass.' + properties: + enabled: + default: false + description: Enabled indicates whether NodePoolAutoCreation is + enabled for a given ComputeClass. + type: boolean + required: + - enabled + type: object + nodePoolConfig: + description: 'NodePoolConfig defines required node pool configuration. + Existing node pools will be matched with the ComputeClass + + only if their configuration match this field. Auto-provisioned node + pools will be created with this configuration.' + properties: + autoRepair: + description: AutoRepair if set to true specifies that a node pool + should have auto repair enabled, disabled in case of being set + to false. + type: boolean + autoUpgrade: + description: AutoUpgrade if set to true specifies that a node + pool should have auto upgrade enabled, disabled in case of being + set to false. + type: boolean + confidentialNodeType: + description: "ConfidentialNodeType: Defines the type of technology\ + \ used by the\nconfidential node.\n\nPossible values:\n \"\ + CONFIDENTIAL_INSTANCE_TYPE_UNSPECIFIED\" - No type specified.\ + \ Do not use\nthis value.\n \"SEV\" - AMD Secure Encrypted\ + \ Virtualization.\n \"SEV_SNP\" - AMD Secure Encrypted Virtualization\ + \ - Secure Nested Paging.\n \"TDX\" - Intel Trust Domain eXtension." + enum: + - CONFIDENTIAL_INSTANCE_TYPE_UNSPECIFIED + - SEV + - SEV_SNP + - TDX + type: string + dra: + description: 'Dra describes settings related to dynamic resource + allocation + + and its integration with autoprovisioning' + properties: + networking: + properties: + enabled: + default: false + type: boolean + type: object + type: object + gvnic: + description: Gvnic contains Google Virtual NIC settings. + properties: + enabled: + default: false + description: Enabled indicates whether gVNIC is enabled on + the node pool. + type: boolean + required: + - enabled + type: object + imageStreaming: + description: ImageStreaming contains image streaming settings. + properties: + enabled: + default: false + description: Enabled enables container image` streaming. + type: boolean + required: + - enabled + type: object + imageType: + description: Image type used by nodes in the node pool. + enum: + - cos_containerd + - ubuntu_containerd + type: string + ipType: + description: 'IPType specifies whether the nodes in the node pool + use public or private IP addresses. + + Possible values are "public" or "private". + + An empty string indicates the default IP type. + + This setting corresponds to the presence and value of the cloud.google.com/private-node + node selector.' + enum: + - public + - private + type: string + loggingConfig: + description: Contains logging configuration. + properties: + loggingVariantConfig: + description: Logging variant configuration. + properties: + variant: + description: Logging variant deployed on nodes. + enum: + - DEFAULT + - MAX_THROUGHPUT + type: string + type: object + type: object + nodeLabels: + additionalProperties: + type: string + description: 'NodeLabels is used to add user defined Kubernetes + labels to all nodes in the new node pool. + + These labels are applied to the Kubernetes API node object and + can be used in nodeSelectors for pod scheduling. + + Note: Node labels are distinct from GKE labels. + + More info: https://cloud.google.com/sdk/gcloud/reference/container/node-pools/create#--node-labels' + maxProperties: 100 + type: object + resourceManagerTags: + description: 'ResourceManagerTags defines what existing GCE resource + manager tag key/value pairs + + with purpose GCE_FIREWALL to attach to all node pools. + + Referenced Tags must be created beforehand via Resource Manager + API.' + items: + description: 'Tags define the key/value of resource manager + tags. + + Tags must be in one of the following formats ([KEY]=[VALUE]) + + 1. tagKeys/{tag_key_id}=tagValues/{tag_value_id} + + 2. {org_id}/{tag_key_name}={tag_value_name} + + 3. {project_id}/{tag_key_name}={tag_value_name}' + properties: + key: + type: string + value: + type: string + required: + - key + - value + type: object + maxItems: 5 + type: array + serviceAccount: + description: ServiceAccount used by the node pool. + type: string + taints: + description: 'Taints is used to add user defined Kubernetes taints + to all nodes in the new node pool. + + These taints are applied to the Kubernetes API node object and + can be used in tolerations for pod scheduling.' + items: + description: 'TaintConfig applies the given kubernetes taints + on all nodes in the new node pool, which can be used with + tolerations for pod scheduling. + + Any workload that does not tolerate the taints specified in + this object will not be scheduled to the node pool. + + More info: https://cloud.google.com/sdk/gcloud/reference/container/node-pools/create#--node-taints' + properties: + effect: + description: 'It defines the taint''s effect on pods that + does not have the necessary toleration. + + The following values are supported: NoSchedule, PreferNoSchedule, + and NoExecute.' + enum: + - NoSchedule + - PreferNoSchedule + - NoExecute + type: string + key: + description: Node taint key. The key must conform to syntax + described in https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#syntax-and-character-set. + maxLength: 320 + type: string + value: + description: The value that matches the specified taint + key. + maxLength: 63 + pattern: ^([a-z0-9][-A-Za-z0-9_.]{1,61})?[A-Za-z0-9]$ + type: string + required: + - effect + - key + type: object + maxItems: 100 + type: array + workloadType: + description: 'WorkloadType defines Collection or Goodput SLO for + the workload. Currently + + supported values: + + * HIGH_AVAILABILITY - for Collection SLO + + * HIGH_THROUGHPUT - for Goodput SLO + + HIGH_AVAILABILITY is desired for running serving workloads which + require + + most of the infrastructure (slices) running all the time to + achieve high + + availability. + + HIGH_THROUGHPUT is desired for running batch/training jobs + + which require all underlying infrastructure (slices) running + for most of + + the time to make progress. HIGH_THROUGHPUT can be only set for + a multi-host + + scenario, that is, when NodePoolGroup is set.' + enum: + - HIGH_AVAILABILITY + - HIGH_THROUGHPUT + type: string + type: object + nodePoolGroup: + description: 'NodePoolGroup defines required node pool configurations + that are shared between a group of node pools. + + Existing node pools will be matched with the ComputeClass only if + their configuration matches this field. + + Auto-provisioned node pools will be created with this configuration.' + properties: + name: + description: Name defines the name of the node pool group, e.g. + MultiMIG + minLength: 1 + type: string + required: + - name + type: object + priorities: + default: [] + description: 'Priorities is a description of user preferences to be + + used by a given ComputeClass.' + items: + description: Priority is a specification of preferred machine characteristics. + minProperties: 1 + properties: + acceleratorNetworkProfile: + description: 'AcceleratorNetworkProfile defines the type of + automated accelerator network provisioning to use. + + Possible values: + + "auto": Enables automatic ANP configuration based on the machine + type. + + "auto-": Enables automatic ANP with a custom network + profile suffix.' + type: string + x-kubernetes-validations: + - message: acceleratorNetworkProfile must be 'auto' or start + with 'auto-' + rule: self == 'auto' || self.startsWith('auto-') + capacityCheckWaitTimeSeconds: + description: CapacityCheckWaitTimeSeconds defines for how long + will this priority be attempted to scale up before moving + on to the next priority. + maximum: 86400 + minimum: 1 + type: integer + flexStart: + description: FlexStart defines Flex Start provisioning model. + properties: + enabled: + default: false + description: Enabled indicates whether Flex Start provisioning + model is enabled. + type: boolean + nodeRecycling: + description: NodeRecycling defines node recycling config. + properties: + leadTimeSeconds: + description: LeadTimeSeconds defines how much time before + node termination timestamp CA should start looking + for a replacement node. + maximum: 604800 + minimum: 1 + type: integer + required: + - leadTimeSeconds + type: object + required: + - enabled + type: object + gpu: + description: Gpu defines preferred GPU config for a node. + properties: + count: + description: Count describes preferred count of GPUs for + a node. + format: int64 + minimum: 0 + type: integer + driverVersion: + default: default + description: DriverVersion describes version of GPU driver + for a node. + enum: + - default + - latest + type: string + gpuSharing: + description: GpuSharing defines the way the nodes would + share the GPU. + properties: + gpuPartitionSize: + description: 'GpuPartitionSize is size of partitions + to create on the GPU. Valid values are + + described in the NVIDIA mig user guide. Example: "1g.5gb" + + (https://docs.nvidia.com/datacenter/tesla/mig-user-guide/#partitioning).' + type: string + maxSharedClientsPerGPU: + description: 'MaxSharedClientsPerGPU describes the max + number of containers that can + + share a physical GPU.' + format: int64 + minimum: 0 + type: integer + sharingStrategy: + description: 'SharingStrategy The type of GPU sharing + strategy to enable on the GPU node. + + Possible values: + + * TIME_SHARING - GPUs are time-shared between containers. + + * MPS - GPUs are shared between containers with NVIDIA + MPS.' + enum: + - MPS + - TIME_SHARING + type: string + type: object + type: + description: Type describes preferred GPU accelerator type + for a node. + type: string + type: object + location: + description: Location describes CCC zonal preferences config. + properties: + locationPolicy: + description: 'LocationPolicy specifies the strategy for + selecting zones when scaling up a node + + pool managed by this Compute Class. This setting controls + the distribution of new + + nodes across zones in the node pool''s region and corresponds + to the node pool + + setting of the same name. + + More info: https://cloud.google.com/sdk/gcloud/reference/container/node-pools/create#--location-policy' + enum: + - ANY + - BALANCED + type: string + zoneTypes: + description: "ZoneTypes specifies sets of zones used for\ + \ provisioning.\nSTANDARD zone type designates the core\ + \ Google Cloud zones within a region.\nAI zone type designates\ + \ specialized zones optimized for AI.\nCLUSTER_DEFAULT\ + \ zone type designate zones specified in the cluster's\ + \ autoprovisioningLocations or cluster\u2019s locations\ + \ if autoprovisioningLocations is empty." + items: + description: ZoneType is an enumeration of supported zone + types. + enum: + - STANDARD + - AI + - CLUSTER_DEFAULT + type: string + maxItems: 3 + minItems: 1 + type: array + zones: + description: Zones lists zones considered for node autoprovisioning. + items: + type: string + minItems: 1 + type: array + type: object + machineFamily: + description: 'Machine family describes preferred instance family + for a node. If none is specified, + + the default autoprovisioning machine family is used.' + maxLength: 10 + type: string + machineType: + description: MachineType defines preferred machine type for + a node. + maxLength: 100 + type: string + maxPodsPerNode: + description: MaxPodsPerNode describes the maximum number of + pods a node can accommodate. + maximum: 256 + minimum: 8 + type: integer + maxRunDurationSeconds: + description: MaxRunDurationSeconds defines the maximum duration + for the nodes to exist. If unspecified, the nodes can exist + indefinitely. + type: integer + minCores: + description: MinCores describes a minimum number of CPU cores + of a node. + minimum: 0 + type: integer + minCpuPlatform: + description: MinCpuPlatform defines the minimum CPU platform + for a node. + enum: + - Intel Sandy Bridge + - Intel Ivy Bridge + - Intel Haswell + - Intel Broadwell + - Intel Skylake + - Intel Cascade Lake + - Intel Ice Lake + - Intel Sapphire Rapids + - Intel Emerald Rapids + - Intel Granite Rapids + - AMD Rome + - AMD Milan + - AMD Genoa + - AMD Turin + - Ampere Altra + - Google Axion + - Nvidia Grace + type: string + minMemoryGb: + description: MinMemoryGb describes a minimum GBs of memory of + a node. + minimum: 0 + type: integer + nodeLabels: + additionalProperties: + type: string + description: 'NodeLabels is used to add user defined Kubernetes + labels to all nodes in the new node pool. + + These labels are applied to the Kubernetes API node object + and can be used in nodeSelectors for pod scheduling. + + Note: Node labels are distinct from GKE labels. + + More info: https://cloud.google.com/sdk/gcloud/reference/container/node-pools/create#--node-labels' + maxProperties: 100 + type: object + nodeSystemConfig: + description: NodeSystemConfig defines node system config for + a node. + properties: + kubeletConfig: + description: KubeletConfig defines kubelet config for a + node. + properties: + allowedUnsafeSysctls: + description: 'This setting defines a comma-separated + allowlist of unsafe sysctls or sysctl patterns + + (ending in `*`). The unsafe namespaced sysctl groups + are `kernel.shm*`, `kernel.msg*`, + + `kernel.sem`, `fs.mqueue.*`, and `net.*`. Leaving + this allowlist empty means they cannot be set on Pods.' + items: + maxLength: 253 + minLength: 1 + pattern: ^([a-z0-9]([-_a-z0-9]*[a-z0-9])?[./])*([a-z0-9][-_a-z0-9]*)?[a-z0-9*]$ + type: string + maxItems: 100 + type: array + containerLogMaxFiles: + description: 'This setting sets the maximum number of + container log files that can be present for a + + container. Default is 5 in OSS if unspecified.' + format: int64 + maximum: 10 + minimum: 2 + type: integer + containerLogMaxSize: + description: 'This setting sets the maximum size of + the container log file before it is rotated. + + Format: positive number + unit, Eg. 100Ki, 10Mi, 5Gi. + Valid units are Ki, + + Mi, Gi. The value must be between 10Mi and 500Mi. + And the total + + container log size (container_log_max_size * container_log_max_files) + + cannot exceed 1% of the total storage of the node. + + Default is 10Mi in OSS if unspecified.' + pattern: ^([0-9]+([.][0-9]+)?(Ki|Mi|Gi))+$ + type: string + cpuCfsQuota: + description: 'This setting enforces the Pod''s CPU limit. + Setting this value to false means that the CPU limits + for Pods are ignored. + + Ignoring CPU limits might be desirable in certain + scenarios where Pods are sensitive to CPU limits. + + The risk of disabling cpuCFSQuota is that a rogue + Pod can consume more CPU resources than intended.' + type: boolean + cpuCfsQuotaPeriod: + description: 'This setting sets the CPU CFS quota period + value, cpu.cfs_period_us, which specifies the period + of how often a cgroup''s access to CPU resources should + be reallocated. + + This option lets you tune the CPU throttling behavior. + Value must be 1ms <= period <= 1s.' + pattern: ^([1-9][0-9]*)m?s$ + type: string + cpuManagerPolicy: + description: 'This setting controls the kubelet''s CPU + Manager Policy. The default value is none which is + the default CPU affinity scheme, providing no affinity + beyond what the OS scheduler does automatically. + + Setting this value to static allows Pods in the Guaranteed + QoS class with integer CPU requests to be assigned + exclusive use of CPUs.' + enum: + - none + - static + type: string + evictionMaxPodGracePeriodSeconds: + description: 'EvictionMaxPodGracePeriodSeconds is the + maximum allowed grace period + + (in seconds) to use when terminating pods in response + to a soft eviction + + threshold being met.' + format: int64 + maximum: 300 + minimum: 0 + type: integer + evictionMinimumReclaim: + description: EvictionMinimumReclaim defines minimum + reclaims. + properties: + imagefsAvailable: + description: 'ImagefsAvailable is the minimum reclaim + for imagefs.available. + + The value must be a percentage, e.g., "5%". + + The value must be positive and less than 10%.' + pattern: ^[0-9]+(\.[0-9]+)?%$ + type: string + imagefsInodesFree: + description: 'ImagefsInodesFree is the minimum reclaim + for imagefs.inodesFree. + + The value must be a percentage, e.g., "5%". + + The value must be positive and less than 10%.' + pattern: ^[0-9]+(\.[0-9]+)?%$ + type: string + memoryAvailable: + description: 'MemoryAvailable is the minimum reclaim + for memory.available. + + The value must be a percentage, e.g., "5%". + + The value must be positive and less than 10%.' + pattern: ^[0-9]+(\.[0-9]+)?%$ + type: string + nodefsAvailable: + description: 'NodefsAvailable is the minimum reclaim + for nodefs.available. + + The value must be a percentage, e.g., "5%". + + The value must be positive and less than 10%.' + pattern: ^[0-9]+(\.[0-9]+)?%$ + type: string + nodefsInodesFree: + description: 'NodefsInodesFree is the minimum reclaim + for nodefs.inodesFree. + + The value must be a percentage, e.g., "5%". + + The value must be positive and less than 10%.' + pattern: ^[0-9]+(\.[0-9]+)?%$ + type: string + pidAvailable: + description: 'PidAvailable is the minimum reclaim + for pid.available. + + The value must be a percentage, e.g., "5%". + + The value must be positive and less than 10%.' + pattern: ^[0-9]+(\.[0-9]+)?%$ + type: string + type: object + evictionSoft: + description: EvictionSoft defines soft eviction thresholds. + properties: + imagefsAvailable: + description: 'ImagefsAvailable is the soft eviction + threshold for imagefs.available. + + The value must be a percentage. Eg. "10%". + + The value must be between 15% and 50% inclusive.' + pattern: ^[0-9]+(\.[0-9]+)?%$ + type: string + imagefsInodesFree: + description: 'ImagefsInodesFree is the soft eviction + threshold for imagefs.inodesFree. + + The value must be a percentage. Eg. "5%". + + The value must be between 5% and 50% inclusive.' + pattern: ^[0-9]+(\.[0-9]+)?%$ + type: string + memoryAvailable: + description: 'MemoryAvailable is the soft eviction + threshold for memory.available. + + The value must be a quantity, e.g., "100Mi". + + The value must be greater than the GKE default + hard eviction threshold of 100Mi and less than + 50% of machine memory.' + pattern: ^[0-9]+(\.[0-9]+)?(Ki|Mi|Gi)$ + type: string + nodefsAvailable: + description: 'NodefsAvailable is the soft eviction + threshold for nodefs.available. + + The value must be a percentage, e.g., "20%". + + The value must be between 10% and 50% inclusive.' + pattern: ^[0-9]+(\.[0-9]+)?%$ + type: string + nodefsInodesFree: + description: 'NodefsInodesFree is the soft eviction + threshold for nodefs.inodesFree. + + The value must be a percentage. Eg. "5%". + + The value must be between 5% and 50% inclusive.' + pattern: ^[0-9]+(\.[0-9]+)?%$ + type: string + pidAvailable: + description: 'PidAvailable is the soft eviction + threshold for pid.available. + + The value must be a percentage. Eg. "10%". + + The value must be between 10% and 50% inclusive.' + pattern: ^[0-9]+(\.[0-9]+)?%$ + type: string + type: object + evictionSoftGracePeriod: + description: EvictionSoftGracePeriod defines grace periods + for soft eviction thresholds. + properties: + imagefsAvailable: + description: 'ImagefsAvailable is the grace period + for the imagefs.available soft eviction threshold. + + The value must be a duration string. Eg. "30s", + "1m30s". + + The value must be positive and less than ''5m''.' + pattern: "^([0-9]+([.][0-9]+)?(ns|us|\xB5s|ms|s|m|h))+$" + type: string + imagefsInodesFree: + description: 'ImagefsInodesFree is the grace period + for the imagefs.inodesFree soft eviction threshold. + + The value must be a duration string. Eg. "30s", + "1m30s". + + The value must be positive and less than ''5m''.' + pattern: "^([0-9]+([.][0-9]+)?(ns|us|\xB5s|ms|s|m|h))+$" + type: string + memoryAvailable: + description: 'MemoryAvailable is the grace period + for the memory.available soft eviction threshold. + + The value must be a duration string. Eg. "30s", + "1m30s". + + The value must be positive and less than ''5m''.' + pattern: "^([0-9]+([.][0-9]+)?(ns|us|\xB5s|ms|s|m|h))+$" + type: string + nodefsAvailable: + description: 'NodefsAvailable is the grace period + for the nodefs.available soft eviction threshold. + + The value must be a duration string. Eg. "30s", + "1m30s". + + The value must be positive and less than ''5m''.' + pattern: "^([0-9]+([.][0-9]+)?(ns|us|\xB5s|ms|s|m|h))+$" + type: string + nodefsInodesFree: + description: 'NodefsInodesFree is the grace period + for the nodefs.inodesFree soft eviction threshold. + + The value must be a duration string. Eg. "30s", + "1m30s". + + The value must be positive and less than ''5m''.' + pattern: "^([0-9]+([.][0-9]+)?(ns|us|\xB5s|ms|s|m|h))+$" + type: string + pidAvailable: + description: 'PidAvailable is the grace period for + the pid.available soft eviction threshold. + + The value must be a duration string. Eg. "30s", + "1m30s". + + The value must be positive and less than ''5m''.' + pattern: "^([0-9]+([.][0-9]+)?(ns|us|\xB5s|ms|s|m|h))+$" + type: string + type: object + imageGcHighThresholdPercent: + description: 'This setting sets the percent of disk + usage after which image garbage collection is always + + run. The percent is calculated as this field value + out of 100. Default is 85 if unspecified.' + format: int64 + maximum: 85 + minimum: 11 + type: integer + imageGcLowThresholdPercent: + description: 'This setting sets the percent of disk + usage before which image garbage collection is never + + run. Lowest disk usage to garbage collect to. The + percent is calculated as + + this field value out of 100. Default is 80 if unspecified.' + format: int64 + maximum: 84 + minimum: 10 + type: integer + imageMaximumGcAge: + description: "This setting sets the maximum age an image\ + \ can be unused before it is garbage collected.\n\ + The string must be a decimal number with a unit suffix,\ + \ such as \"300s\", \"1.5h\", and \"2h45m\".\nValid\ + \ time units are \"ns\", \"us\" (or \"\xB5s\"), \"\ + ms\", \"s\", \"m\", \"h\".\nThe value must be a positive\ + \ duration.\nDefault is \"0s\" if unspecified, which\ + \ disables the field." + pattern: "^([0-9]+([.][0-9]+)?(ns|us|\xB5s|ms|s|m|h))+$" + type: string + imageMinimumGcAge: + description: "This setting sets the minimum age for\ + \ an unused image before it is garbage collected.\n\ + The string must be a decimal number with a unit suffix,\ + \ such as \"300s\", \"1.5h\", and \"2h45m\".\nValid\ + \ time units are \"ns\", \"us\" (or \"\xB5s\"), \"\ + ms\", \"s\", \"m\", \"h\".\nThe value must be a positive\ + \ duration and less than or equal to 2 minutes.\n\ + Default is \"2m\" if unspecified." + pattern: "^([0-9]+([.][0-9]+)?(ns|us|\xB5s|ms|s|m|h))+$" + type: string + maxParallelImagePulls: + description: This setting sets the maximum number of + image pulls in parallel. Default is 2 or 3 depending + on boot disk type. + format: int64 + maximum: 5 + minimum: 2 + type: integer + podPidsLimit: + description: This setting sets the maximum number of + process IDs (PIDs) that each Pod can use. + format: int64 + maximum: 4194304 + minimum: 1024 + type: integer + singleProcessOOMKill: + description: 'This setting sets whether to enable single + process OOM killer. + + If set to true, the processes in a container will + be OOM killed individually instead of as a group.' + type: boolean + type: object + x-kubernetes-validations: + - message: ImageGcLowThresholdPercent must be lower than + imageGcHighThresholdPercent + rule: 'has(self.imageGcHighThresholdPercent)&&has(self.imageGcLowThresholdPercent) + ? self.imageGcHighThresholdPercent>self.imageGcLowThresholdPercent + : true' + - message: ImageGcHighThresholdPercent must be higher than + 80 which is default value of imageGcLowThresholdPercent + rule: 'has(self.imageGcHighThresholdPercent)&&!has(self.imageGcLowThresholdPercent) + ? self.imageGcHighThresholdPercent>80 : true' + linuxNodeConfig: + description: LinuxNodeConfig defines linux node config for + a node. + properties: + hugepageConfig: + description: HugepagesConfig defines hugepages config + for a node. + properties: + hugepage_size1g: + description: Number of 1-gigabyte-sized huge pages + to allocate. + format: int64 + minimum: 1 + type: integer + hugepage_size2m: + description: Number of 2-megabyte-sized huge pages + to allocate. + format: int64 + minimum: 1 + type: integer + type: object + swapConfig: + description: SwapConfig specifies the swap memory configuration + for a node pool. + properties: + bootDiskProfile: + description: Use the node's boot disk for swap. + properties: + swapSizeGib: + description: The size of the swap space in GiB. + format: int64 + minimum: 1 + type: integer + swapSizePercent: + description: The size of the swap space as a + percentage of the node's boot disk. + format: int32 + maximum: 50 + minimum: 1 + type: integer + type: object + x-kubernetes-validations: + - message: only one of swapSizeGib or swapSizePercent + may be set + rule: '(has(self.swapSizeGib) ? 1 : 0) + (has(self.swapSizePercent) + ? 1 : 0) <= 1' + dedicatedLocalSsdProfile: + description: Provision a new, separate local NVMe + SSD exclusively for swap. + properties: + diskCount: + description: The number of physical local NVMe + SSD disks to attach. + format: int64 + minimum: 1 + type: integer + type: object + enabled: + description: Enables or disables swap for the node + pool. Default to false. + type: boolean + encryptionConfig: + description: If omitted, swap space is encrypted + by default. + properties: + disabled: + description: 'If true, swap space will NOT be + encrypted. + + Defaults to false, swap space is encrypted + by default.' + type: boolean + type: object + ephemeralLocalSsdProfile: + description: Use the local SSD (shared with ephemeral + storage) for swap. + properties: + swapSizeGib: + description: The size of the swap space in GiB. + format: int64 + minimum: 1 + type: integer + swapSizePercent: + description: The size of the swap space as a + percentage of the node's ephemeral storage + local SSDs. + format: int32 + maximum: 80 + minimum: 1 + type: integer + type: object + x-kubernetes-validations: + - message: only one of swapSizeGib or swapSizePercent + may be set + rule: '(has(self.swapSizeGib) ? 1 : 0) + (has(self.swapSizePercent) + ? 1 : 0) <= 1' + type: object + x-kubernetes-validations: + - message: only one of bootDiskProfile, ephemeralLocalSsdProfile, + or dedicatedLocalSsdProfile may be set + rule: '(has(self.bootDiskProfile) ? 1 : 0) + (has(self.ephemeralLocalSsdProfile) + ? 1 : 0) + (has(self.dedicatedLocalSsdProfile) ? + 1 : 0) <= 1' + sysctls: + description: SysctlsConfig defines sysctls config for + a node. + properties: + fs.aio-max-nr: + description: The maximum system-wide number of asynchronous + io requests. + format: int64 + maximum: 4194304 + minimum: 65536 + type: integer + fs.file-max: + description: Maximum number of file-handles that + the Linux kernel will allocate. + format: int64 + maximum: 67108864 + minimum: 104857 + type: integer + fs.inotify.max_user_instances: + description: The maximum number of inotify instances + that a user can create. + format: int64 + maximum: 1048576 + minimum: 8192 + type: integer + fs.inotify.max_user_watches: + description: The maximum number of inotify watches + that a user can create. + format: int64 + maximum: 1048576 + minimum: 8192 + type: integer + fs.nr_open: + description: The maximum number of file descriptors + that can be opened by a process. + format: int64 + maximum: 2147483584 + minimum: 1048576 + type: integer + kernel.keys.maxbytes: + description: Represents the maximum number of bytes + that a nonroot user can hold in the payload section + of all their keys. + format: int64 + maximum: 2097152 + minimum: 20000 + type: integer + kernel.keys.maxkeys: + description: Controls the maximum number of keys + that a nonroot user may own. + format: int64 + maximum: 1048576 + minimum: 200 + type: integer + kernel.shmall: + description: 'The maximum size (in bytes) of a single + shared memory segment allowed by the kernel. + + Note that the actual range should be integer between + 0 and 18446744073692774399, while kubebuilder + would lose some precision on uint64 during the + internal representation and parsing.' + maxLength: 20 + minLength: 1 + pattern: ^([0-9]+)$ + type: string + kernel.shmmax: + description: 'The total amount of shared memory + pages that can be used on the system at one time. + + Note that the actual range should be integer between + 0 and 18446744073692774399, while kubebuilder + would lose some precision on uint64 during the + internal representation and parsing.' + maxLength: 20 + minLength: 1 + pattern: ^([0-9]+)$ + type: string + kernel.shmmni: + description: The system-wide maximum number of shared + memory segments. + format: int64 + maximum: 32768 + minimum: 4096 + type: integer + net.core.busy_poll: + description: Low latency busy poll timeout for poll + and select. (needs CONFIG_NET_RX_BUSY_POLL) Approximate + time in us to busy loop waiting for events. + format: int64 + maximum: 2147483647 + minimum: 0 + type: integer + net.core.busy_read: + description: Low latency busy poll timeout for socket + reads. (needs CONFIG_NET_RX_BUSY_POLL) Approximate + time in us to busy loop waiting for packets on + the device queue. + format: int64 + maximum: 2147483647 + minimum: 0 + type: integer + net.core.netdev_max_backlog: + description: Maximum number of packets, queued on + the INPUT side, when the interface receives packets + faster than kernel can process them. + format: int64 + maximum: 2147483647 + minimum: 1 + type: integer + net.core.optmem_max: + description: Maximum ancillary buffer size allowed + per socket. Ancillary data is a sequence of struct + cmsghdr structures with appended data. + format: int64 + maximum: 2147483647 + minimum: 1 + type: integer + net.core.rmem_default: + description: The default receive socket buffer size + in bytes. + format: int64 + maximum: 2147483647 + minimum: 2304 + type: integer + net.core.rmem_max: + description: The maximum receive socket buffer size + in bytes. + format: int64 + maximum: 2147483647 + minimum: 2304 + type: integer + net.core.somaxconn: + description: Limit of socket listen() backlog, known + in userspace as SOMAXCONN. Defaults to 128. See + also tcp_max_syn_backlog for additional tuning + for TCP sockets. + format: int64 + maximum: 2147483647 + minimum: 128 + type: integer + net.core.wmem_default: + description: The default setting (in bytes) of the + socket send buffer. + format: int64 + maximum: 2147483647 + minimum: 4608 + type: integer + net.core.wmem_max: + description: The maximum send socket buffer size + in bytes. + format: int64 + maximum: 2147483647 + minimum: 4608 + type: integer + net.ipv4.neigh.default.gc_thresh1: + description: Tells the garbage collector the minimum + number of network entries that can sit in cache + (floor). + format: int64 + maximum: 262144 + minimum: 0 + type: integer + net.ipv4.neigh.default.gc_thresh2: + description: Acts as a soft limit to the number + of network device entries stored in cache (soft + ceiling). + format: int64 + maximum: 524288 + minimum: 512 + type: integer + net.ipv4.neigh.default.gc_thresh3: + description: Sets a hard ceiling (absolute maximum) + for the network neighbor cache. + format: int64 + maximum: 1048576 + minimum: 1024 + type: integer + net.ipv4.tcp_max_orphans: + format: int64 + maximum: 262144 + minimum: 16384 + type: integer + net.ipv4.tcp_rmem: + description: 'Minimal size of receive buffer used + by UDP sockets in moderation. Each UDP socket + is able to use the size for receiving data, even + if total pages of UDP sockets exceed udp_mem pressure. + The unit is byte. Default: 1 page. The three values + are: min, default, max. Eg. ''4096 87380 6291456''.' + type: string + net.ipv4.tcp_tw_reuse: + description: Allow to reuse TIME-WAIT sockets for + new connections when it is safe from protocol + viewpoint. It should not be changed without advice/request + of technical experts. + format: int64 + maximum: 2 + minimum: 0 + type: integer + net.ipv4.tcp_wmem: + description: 'Minimal size of send buffer used by + UDP sockets in moderation. Each UDP socket is + able to use the size for sending data, even if + total pages of UDP sockets exceed udp_mem pressure. + The unit is byte. Default: 1 page. The three values + are: min, default, max. Eg. ''4096 87380 6291456''.' + type: string + net.ipv6.conf.all.disable_ipv6: + description: Changing this value is same as changing + conf/default/disable_ipv6 setting and also all + per-interface disable_ipv6 settings to the same + value. + type: boolean + net.ipv6.conf.default.disable_ipv6: + description: Disable IPv6 operation. + type: boolean + net.netfilter.nf_conntrack_acct: + description: Whether to enable connection tracking + flow accounting. + type: boolean + net.netfilter.nf_conntrack_buckets: + description: The size of hash table for connection + tracking. + format: int64 + maximum: 524288 + minimum: 65536 + type: integer + net.netfilter.nf_conntrack_max: + description: The size of connection tracking table. + format: int64 + maximum: 4194304 + minimum: 65536 + type: integer + net.netfilter.nf_conntrack_tcp_timeout_close_wait: + description: The period for which the TCP connections + can remain in the CLOSE_WAIT state, and stay in + the table. + format: int64 + maximum: 3600 + minimum: 60 + type: integer + net.netfilter.nf_conntrack_tcp_timeout_established: + description: The duration of dead connections before + deleted automatically from connection tracking + table. + format: int64 + maximum: 86400 + minimum: 600 + type: integer + net.netfilter.nf_conntrack_tcp_timeout_time_wait: + description: The period for which the TCP connections + can remain in the TIME_WAIT state, and stay in + the table. + format: int64 + maximum: 600 + minimum: 1 + type: integer + vm.dirty_background_ratio: + description: 'Percentage of system memory that can + be filled with dirty pages (modified but not yet + written to disk) before background kernel flusher + threads begin writeback. + + This value should be less than ''vm.dirty_ratio''.' + format: int64 + maximum: 100 + minimum: 1 + type: integer + vm.dirty_expire_centisecs: + description: 'Maximum age (in hundredths of a second) + that dirty data can remain in memory before kernel + flusher threads write it to disk. + + Lower values result in faster, more frequent writebacks.' + format: int64 + maximum: 6000 + minimum: 0 + type: integer + vm.dirty_ratio: + description: 'Percentage of system memory that can + be filled with dirty pages before processes performing + writes are forced to block and write out dirty + data synchronously. + + This value should be greater than ''vm.dirty_background_ratio''.' + format: int64 + maximum: 100 + minimum: 1 + type: integer + vm.dirty_writeback_centisecs: + description: Interval (in hundredths of a second) + at which kernel flusher threads wake up to write + 'old' dirty data to disk. + format: int64 + maximum: 1000 + minimum: 0 + type: integer + vm.max_map_count: + description: Maximum number of memory map areas + a process may have. + format: int64 + maximum: 2147483647 + minimum: 65536 + type: integer + vm.min_free_kbytes: + format: int64 + maximum: 1048576 + minimum: 67584 + type: integer + vm.overcommit_memory: + description: 'Determines the kernel''s memory overcommit + handling strategy. + + Supported values: + + 0: Rejects allocations that are obviously too + large. + + 1: Allows overcommit until memory is exhausted. + + 2 (strict): Prevents overcommit beyond swap space + plus a percentage of RAM defined by ''vm.overcommit_ratio''.' + enum: + - 0 + - 1 + - 2 + format: int64 + type: integer + vm.overcommit_ratio: + description: 'Specifies the percentage of physical + RAM allowed for overcommit when ''vm.overcommit_memory'' + is set to 2. + + The total committed address space cannot exceed + swap plus this RAM percentage.' + format: int64 + maximum: 100 + minimum: 0 + type: integer + vm.swappiness: + format: int64 + maximum: 200 + minimum: 0 + type: integer + vm.vfs_cache_pressure: + description: Adjusts the kernel's preference for + reclaiming memory used for dentry (directory) + and inode caches. + format: int64 + maximum: 100 + minimum: 0 + type: integer + vm.watermark_scale_factor: + format: int64 + maximum: 3000 + minimum: 10 + type: integer + type: object + transparentHugepageDefrag: + description: 'Defines the transparent hugepage defrag + configuration on the node. Currently supported values: + + * TRANSPARENT_HUGEPAGE_DEFRAG_ALWAYS: An application + requesting THP will stall on allocation failure and + directly reclaim pages and compact memory in an effort + to allocate a THP immediately. + + * TRANSPARENT_HUGEPAGE_DEFRAG_DEFER: An application + will wake kswapd in the background to reclaim pages + and wake kcompactd to compact memory so that THP is + available in the near future. It is the responsibility + of khugepaged to then install the THP pages later. + + * TRANSPARENT_HUGEPAGE_DEFRAG_DEFER_WITH_MADVISE: + An application will enter direct reclaim and compaction + like always, but only for regions that have used madvise(MADV_HUGEPAGE); + all other regions will wake kswapd in the background + to reclaim pages and wake kcompactd to compact memory + so that THP is available in the near future. + + * TRANSPARENT_HUGEPAGE_DEFRAG_MADVISE: An application + will enter direct reclaim and compaction like always, + but only for regions that have used madvise(MADV_HUGEPAGE); + all other regions will wake kswapd in the background + to reclaim pages and wake kcompactd to compact memory + so that THP is available in the near future. + + * TRANSPARENT_HUGEPAGE_DEFRAG_NEVER: An application + will never enter direct reclaim or compaction. + + * TRANSPARENT_HUGEPAGE_DEFRAG_UNSPECIFIED: Default + value. GKE will not modify the kernel configuration.' + enum: + - TRANSPARENT_HUGEPAGE_DEFRAG_ALWAYS + - TRANSPARENT_HUGEPAGE_DEFRAG_DEFER + - TRANSPARENT_HUGEPAGE_DEFRAG_DEFER_WITH_MADVISE + - TRANSPARENT_HUGEPAGE_DEFRAG_MADVISE + - TRANSPARENT_HUGEPAGE_DEFRAG_NEVER + - TRANSPARENT_HUGEPAGE_DEFRAG_UNSPECIFIED + type: string + transparentHugepageEnabled: + description: 'Controls transparent hugepage support + for anonymous memory. Currently supported values: + + * TRANSPARENT_HUGEPAGE_ENABLED_ALWAYS: Transparent + hugepage is enabled system wide. + + * TRANSPARENT_HUGEPAGE_ENABLED_MADVISE: Transparent + hugepage is enabled inside MADV_HUGEPAGE regions. + This is the default kernel configuration. + + * TRANSPARENT_HUGEPAGE_ENABLED_NEVER: Transparent + hugepage is disabled. + + * TRANSPARENT_HUGEPAGE_ENABLED_UNSPECIFIED: Default + value. GKE will not modify the kernel configuration.' + enum: + - TRANSPARENT_HUGEPAGE_ENABLED_ALWAYS + - TRANSPARENT_HUGEPAGE_ENABLED_MADVISE + - TRANSPARENT_HUGEPAGE_ENABLED_NEVER + - TRANSPARENT_HUGEPAGE_ENABLED_UNSPECIFIED + type: string + type: object + type: object + nodepools: + description: Nodepools describes preference of specific, preexisting + nodepools. + items: + type: string + type: array + placement: + description: Placement defines resource policy used for BYOPP + and BYOWP + properties: + policyName: + description: PolicyName defines the name of the resource + policy, e.g. my-resource-policy + minLength: 1 + type: string + required: + - policyName + type: object + podFamily: + description: PodFamily represents pod-based provisioning and + billing config. + enum: + - general-purpose + - general-purpose-arm + type: string + priorityScore: + description: 'A higher value is treated as a higher priority. + + Priorities with the same priorityScore value are treated equally. + + Not more than 3 priorities can have the same priorityScore.' + maximum: 1000 + minimum: 1 + type: integer + reservations: + description: Reservations defines reservations config for a + node. + properties: + affinity: + description: 'ReservationAffinity affects reservations considered + and the way how they are consumed. + + "Specific" means that only specific reservations are considered + with no fallback possible. + + "AnyBestEffort" affinity would consider any non-specific + reservation available + + to be claimed with a fallback to on-demand nodes in case + of none claimable. + + "None" affinity would prevent reservations from being + used' + enum: + - Specific + - AnyBestEffort + - None + type: string + specific: + description: Specific is a non prioritized list of specific + reservations to be considered by the priority rule. + items: + description: SpecificReservation defines a single specific + reservation to be consumed by the created node. + properties: + name: + description: Name of the reservation to be used. + type: string + project: + description: Project is the project where the specific + reservation lives. + type: string + reservationBlock: + description: ReservationBlock is the block of the + reservation. + properties: + name: + description: Name is the name of the block. + type: string + reservationSubBlock: + description: ReservationSubBlock is the subBlock + of the reservation block. + properties: + name: + description: Name is the name of the subBlock. + type: string + required: + - name + type: object + required: + - name + type: object + zones: + description: Zones is a list of GCE zones where reservations + are to be consumed. + items: + type: string + minItems: 1 + type: array + required: + - name + type: object + minItems: 0 + type: array + required: + - affinity + type: object + x-kubernetes-validations: + - message: Unable to set specific reservations for non specific + affinity + rule: 'has(self.specific) && self.specific.size() > 0 ? self.affinity + == "Specific" : true' + - message: At least 1 specific reservation required for specific + affinity + rule: 'self.affinity == "Specific" ? has(self.specific) && + self.specific.size() > 0 : true' + spot: + description: Spot if set to true specifies that a node should + be a spot instance, on-demand otherwise. + type: boolean + storage: + description: Storage describes storage config of a node. + properties: + bootDiskKMSKey: + description: BootDiskKMSKey defines a key used to encrypt + the boot disk attached. + pattern: projects/[^/]+/locations/[^/]+/keyRings/[^/]+/cryptoKeys/[^/]+ + type: string + bootDiskSize: + description: BootDiskSize defines the size of a disk attached + to node, specified in GB. + minimum: 10 + type: integer + bootDiskType: + description: 'BootDiskType defines type of the disk attached + to the node. + + Note that available boot disk types depend on the machine + family / machine type selected. + + Currently supported types: + + * pd-balanced + + * pd-standard + + * pd-ssd + + * hyperdisk-balanced' + enum: + - pd-balanced + - pd-standard + - pd-ssd + - hyperdisk-balanced + type: string + localSSDCount: + description: LocalSSDCount defines a number of local SSDs + attached to node. + minimum: 1 + type: integer + secondaryBootDisks: + description: SecondaryBootDisks represent persistent disks + attached to a node with special configurations based on + their modes. + items: + description: SecondaryBootDisk represents a persistent + disk attached to a node with special configurations + based on its mode. + properties: + diskImageName: + description: The name of the disk image. + type: string + mode: + description: 'Currently supported modes: + + * MODE_UNSPECIFIED - MODE_UNSPECIFIED is when mode + is not set. + + * CONTAINER_IMAGE_CACHE - it is for using the secondary + boot disk as a container image cache.' + enum: + - MODE_UNSPECIFIED + - CONTAINER_IMAGE_CACHE + type: string + project: + description: The name of the project that the disk + image belongs to. + type: string + required: + - diskImageName + type: object + type: array + type: object + taints: + description: 'Taints is used to add user defined Kubernetes + taints to all nodes in the new node pool. + + These taints are applied to the Kubernetes API node object + and can be used in tolerations for pod scheduling.' + items: + description: 'TaintConfig applies the given kubernetes taints + on all nodes in the new node pool, which can be used with + tolerations for pod scheduling. + + Any workload that does not tolerate the taints specified + in this object will not be scheduled to the node pool. + + More info: https://cloud.google.com/sdk/gcloud/reference/container/node-pools/create#--node-taints' + properties: + effect: + description: 'It defines the taint''s effect on pods that + does not have the necessary toleration. + + The following values are supported: NoSchedule, PreferNoSchedule, + and NoExecute.' + enum: + - NoSchedule + - PreferNoSchedule + - NoExecute + type: string + key: + description: Node taint key. The key must conform to syntax + described in https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#syntax-and-character-set. + maxLength: 320 + type: string + value: + description: The value that matches the specified taint + key. + maxLength: 63 + pattern: ^([a-z0-9][-A-Za-z0-9_.]{1,61})?[A-Za-z0-9]$ + type: string + required: + - effect + - key + type: object + maxItems: 100 + type: array + tpu: + description: Tpu defines preferred TPU config for a node. + properties: + count: + description: Count describes preferred count of TPU chips + for a node. + format: int64 + type: integer + topology: + description: Topology describes preferred TPU topology of + a node. + type: string + type: + description: Type describes preferred TPU type for a node. + type: string + type: object + type: object + x-kubernetes-validations: + - message: Nodepool field cannot be set along with other fields + rule: 'has(self.nodepools) ? (size(dyn(self)) == 1) : true' + - message: MachineFamily and MachineType cannot be set together + rule: '!(has(self.machineFamily) && has(self.machineType))' + - message: MachineType cannot be set together with MinCores/MinMemoryGb + rule: '!(has(self.machineType) && (has(self.minCores) || has(self.minMemoryGb)))' + - message: MachineFamily cannot be equal to 'ek' + rule: '!(has(self.machineFamily) && self.machineFamily == ''ek'')' + - message: MachineType cannot start with 'ek' prefix + rule: '!(has(self.machineType) && self.machineType.startsWith(''ek''))' + - message: MachineFamily cannot be equal to 'e4a' + rule: '!(has(self.machineFamily) && self.machineFamily == ''e4a'')' + - message: MachineType cannot start with 'e4a' prefix + rule: '!(has(self.machineType) && self.machineType.startsWith(''e4a''))' + - message: Flex Start provisioning model is incompatible with Spot + rule: '!(has(self.flexStart) && has(self.spot) && self.spot == + true && self.flexStart.enabled == true)' + - message: capacityCheckWaitTimeSeconds is only supported for Flex + Start and for multi-host TPUs + rule: '!has(self.capacityCheckWaitTimeSeconds) || has(self.tpu) + || (has(self.flexStart) && self.flexStart.enabled)' + maxItems: 1000 + minItems: 0 + type: array + priorityDefaults: + description: 'PriorityDefaults define the default rules for all priorities + if the rule doesn''t exist in some priority. + + Note: PriorityDefaults doesn''t apply to priorities with only Nodepools.' + properties: + location: + description: Location describes CCC zonal preferences config. + properties: + locationPolicy: + description: 'LocationPolicy specifies the strategy for selecting + zones when scaling up a node + + pool managed by this Compute Class. This setting controls + the distribution of new + + nodes across zones in the node pool''s region and corresponds + to the node pool + + setting of the same name. + + More info: https://cloud.google.com/sdk/gcloud/reference/container/node-pools/create#--location-policy' + enum: + - ANY + - BALANCED + type: string + zoneTypes: + description: "ZoneTypes specifies sets of zones used for provisioning.\n\ + STANDARD zone type designates the core Google Cloud zones\ + \ within a region.\nAI zone type designates specialized\ + \ zones optimized for AI.\nCLUSTER_DEFAULT zone type designate\ + \ zones specified in the cluster's autoprovisioningLocations\ + \ or cluster\u2019s locations if autoprovisioningLocations\ + \ is empty." + items: + description: ZoneType is an enumeration of supported zone + types. + enum: + - STANDARD + - AI + - CLUSTER_DEFAULT + type: string + maxItems: 3 + minItems: 1 + type: array + zones: + description: Zones lists zones considered for node autoprovisioning. + items: + type: string + minItems: 1 + type: array + type: object + nodeSystemConfig: + description: NodeSystemConfig defines node system config for a + node. + properties: + kubeletConfig: + description: KubeletConfig defines kubelet config for a node. + properties: + allowedUnsafeSysctls: + description: 'This setting defines a comma-separated allowlist + of unsafe sysctls or sysctl patterns + + (ending in `*`). The unsafe namespaced sysctl groups + are `kernel.shm*`, `kernel.msg*`, + + `kernel.sem`, `fs.mqueue.*`, and `net.*`. Leaving this + allowlist empty means they cannot be set on Pods.' + items: + maxLength: 253 + minLength: 1 + pattern: ^([a-z0-9]([-_a-z0-9]*[a-z0-9])?[./])*([a-z0-9][-_a-z0-9]*)?[a-z0-9*]$ + type: string + maxItems: 100 + type: array + containerLogMaxFiles: + description: 'This setting sets the maximum number of + container log files that can be present for a + + container. Default is 5 in OSS if unspecified.' + format: int64 + maximum: 10 + minimum: 2 + type: integer + containerLogMaxSize: + description: 'This setting sets the maximum size of the + container log file before it is rotated. + + Format: positive number + unit, Eg. 100Ki, 10Mi, 5Gi. + Valid units are Ki, + + Mi, Gi. The value must be between 10Mi and 500Mi. And + the total + + container log size (container_log_max_size * container_log_max_files) + + cannot exceed 1% of the total storage of the node. + + Default is 10Mi in OSS if unspecified.' + pattern: ^([0-9]+([.][0-9]+)?(Ki|Mi|Gi))+$ + type: string + cpuCfsQuota: + description: 'This setting enforces the Pod''s CPU limit. + Setting this value to false means that the CPU limits + for Pods are ignored. + + Ignoring CPU limits might be desirable in certain scenarios + where Pods are sensitive to CPU limits. + + The risk of disabling cpuCFSQuota is that a rogue Pod + can consume more CPU resources than intended.' + type: boolean + cpuCfsQuotaPeriod: + description: 'This setting sets the CPU CFS quota period + value, cpu.cfs_period_us, which specifies the period + of how often a cgroup''s access to CPU resources should + be reallocated. + + This option lets you tune the CPU throttling behavior. + Value must be 1ms <= period <= 1s.' + pattern: ^([1-9][0-9]*)m?s$ + type: string + cpuManagerPolicy: + description: 'This setting controls the kubelet''s CPU + Manager Policy. The default value is none which is the + default CPU affinity scheme, providing no affinity beyond + what the OS scheduler does automatically. + + Setting this value to static allows Pods in the Guaranteed + QoS class with integer CPU requests to be assigned exclusive + use of CPUs.' + enum: + - none + - static + type: string + evictionMaxPodGracePeriodSeconds: + description: 'EvictionMaxPodGracePeriodSeconds is the + maximum allowed grace period + + (in seconds) to use when terminating pods in response + to a soft eviction + + threshold being met.' + format: int64 + maximum: 300 + minimum: 0 + type: integer + evictionMinimumReclaim: + description: EvictionMinimumReclaim defines minimum reclaims. + properties: + imagefsAvailable: + description: 'ImagefsAvailable is the minimum reclaim + for imagefs.available. + + The value must be a percentage, e.g., "5%". + + The value must be positive and less than 10%.' + pattern: ^[0-9]+(\.[0-9]+)?%$ + type: string + imagefsInodesFree: + description: 'ImagefsInodesFree is the minimum reclaim + for imagefs.inodesFree. + + The value must be a percentage, e.g., "5%". + + The value must be positive and less than 10%.' + pattern: ^[0-9]+(\.[0-9]+)?%$ + type: string + memoryAvailable: + description: 'MemoryAvailable is the minimum reclaim + for memory.available. + + The value must be a percentage, e.g., "5%". + + The value must be positive and less than 10%.' + pattern: ^[0-9]+(\.[0-9]+)?%$ + type: string + nodefsAvailable: + description: 'NodefsAvailable is the minimum reclaim + for nodefs.available. + + The value must be a percentage, e.g., "5%". + + The value must be positive and less than 10%.' + pattern: ^[0-9]+(\.[0-9]+)?%$ + type: string + nodefsInodesFree: + description: 'NodefsInodesFree is the minimum reclaim + for nodefs.inodesFree. + + The value must be a percentage, e.g., "5%". + + The value must be positive and less than 10%.' + pattern: ^[0-9]+(\.[0-9]+)?%$ + type: string + pidAvailable: + description: 'PidAvailable is the minimum reclaim + for pid.available. + + The value must be a percentage, e.g., "5%". + + The value must be positive and less than 10%.' + pattern: ^[0-9]+(\.[0-9]+)?%$ + type: string + type: object + evictionSoft: + description: EvictionSoft defines soft eviction thresholds. + properties: + imagefsAvailable: + description: 'ImagefsAvailable is the soft eviction + threshold for imagefs.available. + + The value must be a percentage. Eg. "10%". + + The value must be between 15% and 50% inclusive.' + pattern: ^[0-9]+(\.[0-9]+)?%$ + type: string + imagefsInodesFree: + description: 'ImagefsInodesFree is the soft eviction + threshold for imagefs.inodesFree. + + The value must be a percentage. Eg. "5%". + + The value must be between 5% and 50% inclusive.' + pattern: ^[0-9]+(\.[0-9]+)?%$ + type: string + memoryAvailable: + description: 'MemoryAvailable is the soft eviction + threshold for memory.available. + + The value must be a quantity, e.g., "100Mi". + + The value must be greater than the GKE default hard + eviction threshold of 100Mi and less than 50% of + machine memory.' + pattern: ^[0-9]+(\.[0-9]+)?(Ki|Mi|Gi)$ + type: string + nodefsAvailable: + description: 'NodefsAvailable is the soft eviction + threshold for nodefs.available. + + The value must be a percentage, e.g., "20%". + + The value must be between 10% and 50% inclusive.' + pattern: ^[0-9]+(\.[0-9]+)?%$ + type: string + nodefsInodesFree: + description: 'NodefsInodesFree is the soft eviction + threshold for nodefs.inodesFree. + + The value must be a percentage. Eg. "5%". + + The value must be between 5% and 50% inclusive.' + pattern: ^[0-9]+(\.[0-9]+)?%$ + type: string + pidAvailable: + description: 'PidAvailable is the soft eviction threshold + for pid.available. + + The value must be a percentage. Eg. "10%". + + The value must be between 10% and 50% inclusive.' + pattern: ^[0-9]+(\.[0-9]+)?%$ + type: string + type: object + evictionSoftGracePeriod: + description: EvictionSoftGracePeriod defines grace periods + for soft eviction thresholds. + properties: + imagefsAvailable: + description: 'ImagefsAvailable is the grace period + for the imagefs.available soft eviction threshold. + + The value must be a duration string. Eg. "30s", + "1m30s". + + The value must be positive and less than ''5m''.' + pattern: "^([0-9]+([.][0-9]+)?(ns|us|\xB5s|ms|s|m|h))+$" + type: string + imagefsInodesFree: + description: 'ImagefsInodesFree is the grace period + for the imagefs.inodesFree soft eviction threshold. + + The value must be a duration string. Eg. "30s", + "1m30s". + + The value must be positive and less than ''5m''.' + pattern: "^([0-9]+([.][0-9]+)?(ns|us|\xB5s|ms|s|m|h))+$" + type: string + memoryAvailable: + description: 'MemoryAvailable is the grace period + for the memory.available soft eviction threshold. + + The value must be a duration string. Eg. "30s", + "1m30s". + + The value must be positive and less than ''5m''.' + pattern: "^([0-9]+([.][0-9]+)?(ns|us|\xB5s|ms|s|m|h))+$" + type: string + nodefsAvailable: + description: 'NodefsAvailable is the grace period + for the nodefs.available soft eviction threshold. + + The value must be a duration string. Eg. "30s", + "1m30s". + + The value must be positive and less than ''5m''.' + pattern: "^([0-9]+([.][0-9]+)?(ns|us|\xB5s|ms|s|m|h))+$" + type: string + nodefsInodesFree: + description: 'NodefsInodesFree is the grace period + for the nodefs.inodesFree soft eviction threshold. + + The value must be a duration string. Eg. "30s", + "1m30s". + + The value must be positive and less than ''5m''.' + pattern: "^([0-9]+([.][0-9]+)?(ns|us|\xB5s|ms|s|m|h))+$" + type: string + pidAvailable: + description: 'PidAvailable is the grace period for + the pid.available soft eviction threshold. + + The value must be a duration string. Eg. "30s", + "1m30s". + + The value must be positive and less than ''5m''.' + pattern: "^([0-9]+([.][0-9]+)?(ns|us|\xB5s|ms|s|m|h))+$" + type: string + type: object + imageGcHighThresholdPercent: + description: 'This setting sets the percent of disk usage + after which image garbage collection is always + + run. The percent is calculated as this field value out + of 100. Default is 85 if unspecified.' + format: int64 + maximum: 85 + minimum: 11 + type: integer + imageGcLowThresholdPercent: + description: 'This setting sets the percent of disk usage + before which image garbage collection is never + + run. Lowest disk usage to garbage collect to. The percent + is calculated as + + this field value out of 100. Default is 80 if unspecified.' + format: int64 + maximum: 84 + minimum: 10 + type: integer + imageMaximumGcAge: + description: "This setting sets the maximum age an image\ + \ can be unused before it is garbage collected.\nThe\ + \ string must be a decimal number with a unit suffix,\ + \ such as \"300s\", \"1.5h\", and \"2h45m\".\nValid\ + \ time units are \"ns\", \"us\" (or \"\xB5s\"), \"ms\"\ + , \"s\", \"m\", \"h\".\nThe value must be a positive\ + \ duration.\nDefault is \"0s\" if unspecified, which\ + \ disables the field." + pattern: "^([0-9]+([.][0-9]+)?(ns|us|\xB5s|ms|s|m|h))+$" + type: string + imageMinimumGcAge: + description: "This setting sets the minimum age for an\ + \ unused image before it is garbage collected.\nThe\ + \ string must be a decimal number with a unit suffix,\ + \ such as \"300s\", \"1.5h\", and \"2h45m\".\nValid\ + \ time units are \"ns\", \"us\" (or \"\xB5s\"), \"ms\"\ + , \"s\", \"m\", \"h\".\nThe value must be a positive\ + \ duration and less than or equal to 2 minutes.\nDefault\ + \ is \"2m\" if unspecified." + pattern: "^([0-9]+([.][0-9]+)?(ns|us|\xB5s|ms|s|m|h))+$" + type: string + maxParallelImagePulls: + description: This setting sets the maximum number of image + pulls in parallel. Default is 2 or 3 depending on boot + disk type. + format: int64 + maximum: 5 + minimum: 2 + type: integer + podPidsLimit: + description: This setting sets the maximum number of process + IDs (PIDs) that each Pod can use. + format: int64 + maximum: 4194304 + minimum: 1024 + type: integer + singleProcessOOMKill: + description: 'This setting sets whether to enable single + process OOM killer. + + If set to true, the processes in a container will be + OOM killed individually instead of as a group.' + type: boolean + type: object + x-kubernetes-validations: + - message: ImageGcLowThresholdPercent must be lower than imageGcHighThresholdPercent + rule: 'has(self.imageGcHighThresholdPercent)&&has(self.imageGcLowThresholdPercent) + ? self.imageGcHighThresholdPercent>self.imageGcLowThresholdPercent + : true' + - message: ImageGcHighThresholdPercent must be higher than + 80 which is default value of imageGcLowThresholdPercent + rule: 'has(self.imageGcHighThresholdPercent)&&!has(self.imageGcLowThresholdPercent) + ? self.imageGcHighThresholdPercent>80 : true' + linuxNodeConfig: + description: LinuxNodeConfig defines linux node config for + a node. + properties: + hugepageConfig: + description: HugepagesConfig defines hugepages config + for a node. + properties: + hugepage_size1g: + description: Number of 1-gigabyte-sized huge pages + to allocate. + format: int64 + minimum: 1 + type: integer + hugepage_size2m: + description: Number of 2-megabyte-sized huge pages + to allocate. + format: int64 + minimum: 1 + type: integer + type: object + swapConfig: + description: SwapConfig specifies the swap memory configuration + for a node pool. + properties: + bootDiskProfile: + description: Use the node's boot disk for swap. + properties: + swapSizeGib: + description: The size of the swap space in GiB. + format: int64 + minimum: 1 + type: integer + swapSizePercent: + description: The size of the swap space as a percentage + of the node's boot disk. + format: int32 + maximum: 50 + minimum: 1 + type: integer + type: object + x-kubernetes-validations: + - message: only one of swapSizeGib or swapSizePercent + may be set + rule: '(has(self.swapSizeGib) ? 1 : 0) + (has(self.swapSizePercent) + ? 1 : 0) <= 1' + dedicatedLocalSsdProfile: + description: Provision a new, separate local NVMe + SSD exclusively for swap. + properties: + diskCount: + description: The number of physical local NVMe + SSD disks to attach. + format: int64 + minimum: 1 + type: integer + type: object + enabled: + description: Enables or disables swap for the node + pool. Default to false. + type: boolean + encryptionConfig: + description: If omitted, swap space is encrypted by + default. + properties: + disabled: + description: 'If true, swap space will NOT be + encrypted. + + Defaults to false, swap space is encrypted by + default.' + type: boolean + type: object + ephemeralLocalSsdProfile: + description: Use the local SSD (shared with ephemeral + storage) for swap. + properties: + swapSizeGib: + description: The size of the swap space in GiB. + format: int64 + minimum: 1 + type: integer + swapSizePercent: + description: The size of the swap space as a percentage + of the node's ephemeral storage local SSDs. + format: int32 + maximum: 80 + minimum: 1 + type: integer + type: object + x-kubernetes-validations: + - message: only one of swapSizeGib or swapSizePercent + may be set + rule: '(has(self.swapSizeGib) ? 1 : 0) + (has(self.swapSizePercent) + ? 1 : 0) <= 1' + type: object + x-kubernetes-validations: + - message: only one of bootDiskProfile, ephemeralLocalSsdProfile, + or dedicatedLocalSsdProfile may be set + rule: '(has(self.bootDiskProfile) ? 1 : 0) + (has(self.ephemeralLocalSsdProfile) + ? 1 : 0) + (has(self.dedicatedLocalSsdProfile) ? 1 + : 0) <= 1' + sysctls: + description: SysctlsConfig defines sysctls config for + a node. + properties: + fs.aio-max-nr: + description: The maximum system-wide number of asynchronous + io requests. + format: int64 + maximum: 4194304 + minimum: 65536 + type: integer + fs.file-max: + description: Maximum number of file-handles that the + Linux kernel will allocate. + format: int64 + maximum: 67108864 + minimum: 104857 + type: integer + fs.inotify.max_user_instances: + description: The maximum number of inotify instances + that a user can create. + format: int64 + maximum: 1048576 + minimum: 8192 + type: integer + fs.inotify.max_user_watches: + description: The maximum number of inotify watches + that a user can create. + format: int64 + maximum: 1048576 + minimum: 8192 + type: integer + fs.nr_open: + description: The maximum number of file descriptors + that can be opened by a process. + format: int64 + maximum: 2147483584 + minimum: 1048576 + type: integer + kernel.keys.maxbytes: + description: Represents the maximum number of bytes + that a nonroot user can hold in the payload section + of all their keys. + format: int64 + maximum: 2097152 + minimum: 20000 + type: integer + kernel.keys.maxkeys: + description: Controls the maximum number of keys that + a nonroot user may own. + format: int64 + maximum: 1048576 + minimum: 200 + type: integer + kernel.shmall: + description: 'The maximum size (in bytes) of a single + shared memory segment allowed by the kernel. + + Note that the actual range should be integer between + 0 and 18446744073692774399, while kubebuilder would + lose some precision on uint64 during the internal + representation and parsing.' + maxLength: 20 + minLength: 1 + pattern: ^([0-9]+)$ + type: string + kernel.shmmax: + description: 'The total amount of shared memory pages + that can be used on the system at one time. + + Note that the actual range should be integer between + 0 and 18446744073692774399, while kubebuilder would + lose some precision on uint64 during the internal + representation and parsing.' + maxLength: 20 + minLength: 1 + pattern: ^([0-9]+)$ + type: string + kernel.shmmni: + description: The system-wide maximum number of shared + memory segments. + format: int64 + maximum: 32768 + minimum: 4096 + type: integer + net.core.busy_poll: + description: Low latency busy poll timeout for poll + and select. (needs CONFIG_NET_RX_BUSY_POLL) Approximate + time in us to busy loop waiting for events. + format: int64 + maximum: 2147483647 + minimum: 0 + type: integer + net.core.busy_read: + description: Low latency busy poll timeout for socket + reads. (needs CONFIG_NET_RX_BUSY_POLL) Approximate + time in us to busy loop waiting for packets on the + device queue. + format: int64 + maximum: 2147483647 + minimum: 0 + type: integer + net.core.netdev_max_backlog: + description: Maximum number of packets, queued on + the INPUT side, when the interface receives packets + faster than kernel can process them. + format: int64 + maximum: 2147483647 + minimum: 1 + type: integer + net.core.optmem_max: + description: Maximum ancillary buffer size allowed + per socket. Ancillary data is a sequence of struct + cmsghdr structures with appended data. + format: int64 + maximum: 2147483647 + minimum: 1 + type: integer + net.core.rmem_default: + description: The default receive socket buffer size + in bytes. + format: int64 + maximum: 2147483647 + minimum: 2304 + type: integer + net.core.rmem_max: + description: The maximum receive socket buffer size + in bytes. + format: int64 + maximum: 2147483647 + minimum: 2304 + type: integer + net.core.somaxconn: + description: Limit of socket listen() backlog, known + in userspace as SOMAXCONN. Defaults to 128. See + also tcp_max_syn_backlog for additional tuning for + TCP sockets. + format: int64 + maximum: 2147483647 + minimum: 128 + type: integer + net.core.wmem_default: + description: The default setting (in bytes) of the + socket send buffer. + format: int64 + maximum: 2147483647 + minimum: 4608 + type: integer + net.core.wmem_max: + description: The maximum send socket buffer size in + bytes. + format: int64 + maximum: 2147483647 + minimum: 4608 + type: integer + net.ipv4.neigh.default.gc_thresh1: + description: Tells the garbage collector the minimum + number of network entries that can sit in cache + (floor). + format: int64 + maximum: 262144 + minimum: 0 + type: integer + net.ipv4.neigh.default.gc_thresh2: + description: Acts as a soft limit to the number of + network device entries stored in cache (soft ceiling). + format: int64 + maximum: 524288 + minimum: 512 + type: integer + net.ipv4.neigh.default.gc_thresh3: + description: Sets a hard ceiling (absolute maximum) + for the network neighbor cache. + format: int64 + maximum: 1048576 + minimum: 1024 + type: integer + net.ipv4.tcp_max_orphans: + format: int64 + maximum: 262144 + minimum: 16384 + type: integer + net.ipv4.tcp_rmem: + description: 'Minimal size of receive buffer used + by UDP sockets in moderation. Each UDP socket is + able to use the size for receiving data, even if + total pages of UDP sockets exceed udp_mem pressure. + The unit is byte. Default: 1 page. The three values + are: min, default, max. Eg. ''4096 87380 6291456''.' + type: string + net.ipv4.tcp_tw_reuse: + description: Allow to reuse TIME-WAIT sockets for + new connections when it is safe from protocol viewpoint. + It should not be changed without advice/request + of technical experts. + format: int64 + maximum: 2 + minimum: 0 + type: integer + net.ipv4.tcp_wmem: + description: 'Minimal size of send buffer used by + UDP sockets in moderation. Each UDP socket is able + to use the size for sending data, even if total + pages of UDP sockets exceed udp_mem pressure. The + unit is byte. Default: 1 page. The three values + are: min, default, max. Eg. ''4096 87380 6291456''.' + type: string + net.ipv6.conf.all.disable_ipv6: + description: Changing this value is same as changing + conf/default/disable_ipv6 setting and also all per-interface + disable_ipv6 settings to the same value. + type: boolean + net.ipv6.conf.default.disable_ipv6: + description: Disable IPv6 operation. + type: boolean + net.netfilter.nf_conntrack_acct: + description: Whether to enable connection tracking + flow accounting. + type: boolean + net.netfilter.nf_conntrack_buckets: + description: The size of hash table for connection + tracking. + format: int64 + maximum: 524288 + minimum: 65536 + type: integer + net.netfilter.nf_conntrack_max: + description: The size of connection tracking table. + format: int64 + maximum: 4194304 + minimum: 65536 + type: integer + net.netfilter.nf_conntrack_tcp_timeout_close_wait: + description: The period for which the TCP connections + can remain in the CLOSE_WAIT state, and stay in + the table. + format: int64 + maximum: 3600 + minimum: 60 + type: integer + net.netfilter.nf_conntrack_tcp_timeout_established: + description: The duration of dead connections before + deleted automatically from connection tracking table. + format: int64 + maximum: 86400 + minimum: 600 + type: integer + net.netfilter.nf_conntrack_tcp_timeout_time_wait: + description: The period for which the TCP connections + can remain in the TIME_WAIT state, and stay in the + table. + format: int64 + maximum: 600 + minimum: 1 + type: integer + vm.dirty_background_ratio: + description: 'Percentage of system memory that can + be filled with dirty pages (modified but not yet + written to disk) before background kernel flusher + threads begin writeback. + + This value should be less than ''vm.dirty_ratio''.' + format: int64 + maximum: 100 + minimum: 1 + type: integer + vm.dirty_expire_centisecs: + description: 'Maximum age (in hundredths of a second) + that dirty data can remain in memory before kernel + flusher threads write it to disk. + + Lower values result in faster, more frequent writebacks.' + format: int64 + maximum: 6000 + minimum: 0 + type: integer + vm.dirty_ratio: + description: 'Percentage of system memory that can + be filled with dirty pages before processes performing + writes are forced to block and write out dirty data + synchronously. + + This value should be greater than ''vm.dirty_background_ratio''.' + format: int64 + maximum: 100 + minimum: 1 + type: integer + vm.dirty_writeback_centisecs: + description: Interval (in hundredths of a second) + at which kernel flusher threads wake up to write + 'old' dirty data to disk. + format: int64 + maximum: 1000 + minimum: 0 + type: integer + vm.max_map_count: + description: Maximum number of memory map areas a + process may have. + format: int64 + maximum: 2147483647 + minimum: 65536 + type: integer + vm.min_free_kbytes: + format: int64 + maximum: 1048576 + minimum: 67584 + type: integer + vm.overcommit_memory: + description: 'Determines the kernel''s memory overcommit + handling strategy. + + Supported values: + + 0: Rejects allocations that are obviously too large. + + 1: Allows overcommit until memory is exhausted. + + 2 (strict): Prevents overcommit beyond swap space + plus a percentage of RAM defined by ''vm.overcommit_ratio''.' + enum: + - 0 + - 1 + - 2 + format: int64 + type: integer + vm.overcommit_ratio: + description: 'Specifies the percentage of physical + RAM allowed for overcommit when ''vm.overcommit_memory'' + is set to 2. + + The total committed address space cannot exceed + swap plus this RAM percentage.' + format: int64 + maximum: 100 + minimum: 0 + type: integer + vm.swappiness: + format: int64 + maximum: 200 + minimum: 0 + type: integer + vm.vfs_cache_pressure: + description: Adjusts the kernel's preference for reclaiming + memory used for dentry (directory) and inode caches. + format: int64 + maximum: 100 + minimum: 0 + type: integer + vm.watermark_scale_factor: + format: int64 + maximum: 3000 + minimum: 10 + type: integer + type: object + transparentHugepageDefrag: + description: 'Defines the transparent hugepage defrag + configuration on the node. Currently supported values: + + * TRANSPARENT_HUGEPAGE_DEFRAG_ALWAYS: An application + requesting THP will stall on allocation failure and + directly reclaim pages and compact memory in an effort + to allocate a THP immediately. + + * TRANSPARENT_HUGEPAGE_DEFRAG_DEFER: An application + will wake kswapd in the background to reclaim pages + and wake kcompactd to compact memory so that THP is + available in the near future. It is the responsibility + of khugepaged to then install the THP pages later. + + * TRANSPARENT_HUGEPAGE_DEFRAG_DEFER_WITH_MADVISE: An + application will enter direct reclaim and compaction + like always, but only for regions that have used madvise(MADV_HUGEPAGE); + all other regions will wake kswapd in the background + to reclaim pages and wake kcompactd to compact memory + so that THP is available in the near future. + + * TRANSPARENT_HUGEPAGE_DEFRAG_MADVISE: An application + will enter direct reclaim and compaction like always, + but only for regions that have used madvise(MADV_HUGEPAGE); + all other regions will wake kswapd in the background + to reclaim pages and wake kcompactd to compact memory + so that THP is available in the near future. + + * TRANSPARENT_HUGEPAGE_DEFRAG_NEVER: An application + will never enter direct reclaim or compaction. + + * TRANSPARENT_HUGEPAGE_DEFRAG_UNSPECIFIED: Default value. + GKE will not modify the kernel configuration.' + enum: + - TRANSPARENT_HUGEPAGE_DEFRAG_ALWAYS + - TRANSPARENT_HUGEPAGE_DEFRAG_DEFER + - TRANSPARENT_HUGEPAGE_DEFRAG_DEFER_WITH_MADVISE + - TRANSPARENT_HUGEPAGE_DEFRAG_MADVISE + - TRANSPARENT_HUGEPAGE_DEFRAG_NEVER + - TRANSPARENT_HUGEPAGE_DEFRAG_UNSPECIFIED + type: string + transparentHugepageEnabled: + description: 'Controls transparent hugepage support for + anonymous memory. Currently supported values: + + * TRANSPARENT_HUGEPAGE_ENABLED_ALWAYS: Transparent hugepage + is enabled system wide. + + * TRANSPARENT_HUGEPAGE_ENABLED_MADVISE: Transparent + hugepage is enabled inside MADV_HUGEPAGE regions. This + is the default kernel configuration. + + * TRANSPARENT_HUGEPAGE_ENABLED_NEVER: Transparent hugepage + is disabled. + + * TRANSPARENT_HUGEPAGE_ENABLED_UNSPECIFIED: Default + value. GKE will not modify the kernel configuration.' + enum: + - TRANSPARENT_HUGEPAGE_ENABLED_ALWAYS + - TRANSPARENT_HUGEPAGE_ENABLED_MADVISE + - TRANSPARENT_HUGEPAGE_ENABLED_NEVER + - TRANSPARENT_HUGEPAGE_ENABLED_UNSPECIFIED + type: string + type: object + type: object + type: object + whenUnsatisfiable: + default: DoNotScaleUp + description: 'WhenUnsatisfiable describes autoscaler behaviour in + case none + + of the provided priorities is satisfiable. + + Currently supported values: + + * ScaleUpAnyway + + * DoNotScaleUp' + enum: + - ScaleUpAnyway + - DoNotScaleUp + type: string + required: + - whenUnsatisfiable + type: object + x-kubernetes-validations: + - message: Autopilot is required once set + rule: '!has(oldSelf.autopilot) || has(self.autopilot)' + - message: Nodepools priority cannot be used when Autopilot is enabled + rule: '(has(self.autopilot) && self.autopilot.enabled) ? !self.priorities.exists(priority, + has(priority.nodepools)) : true' + - message: NodePoolAutoCreation cannot be disabled when Autopilot is enabled + rule: '(has(self.autopilot) && self.autopilot.enabled) ? !(has(self.nodePoolAutoCreation) + && !self.nodePoolAutoCreation.enabled) : true' + - message: Only cos_containerd image type can be used when Autopilot is + enabled + rule: '(has(self.autopilot) && self.autopilot.enabled) ? (!has(self.nodePoolConfig) + || !has(self.nodePoolConfig.imageType) || self.nodePoolConfig.imageType + == "cos_containerd") : true' + - message: Only DEFAULT logging variant can be used when Autopilot is + enabled + rule: '(has(self.autopilot) && self.autopilot.enabled) ? (!has(self.nodePoolConfig) + || !has(self.nodePoolConfig.loggingConfig) || !has(self.nodePoolConfig.loggingConfig.loggingVariantConfig) + || !has(self.nodePoolConfig.loggingConfig.loggingVariantConfig.variant) + || self.nodePoolConfig.loggingConfig.loggingVariantConfig.variant + == "DEFAULT") : true' + - message: If NodePoolGroup is not specified NodePoolConfig.WorkloadType + can only be HIGH_AVAILABILITY if set + rule: '(has(self.nodePoolConfig) && has(self.nodePoolConfig.workloadType) + && !has(self.nodePoolGroup)) ? self.nodePoolConfig.workloadType == + "HIGH_AVAILABILITY" : true' + - message: In GKE Standard, pod family can be used only if Autopilot is + enabled + rule: 'self.priorities.exists(priority, has(priority.podFamily)) ? (has(self.autopilot) + && self.autopilot.enabled) : true' + - message: If using NodePoolConfig.ConfidentialNodeType, each priority + must specify either MachineFamily or MachineType. + rule: '(has(self.nodePoolConfig) && has(self.nodePoolConfig.confidentialNodeType)) + ? self.priorities.all(priority, has(priority.machineFamily) || has(priority.machineType)) + : true' + - message: ConfidentialNodeType SEV only supports N2D, C2D, C3D, C4D + rule: '(has(self.nodePoolConfig) && has(self.nodePoolConfig.confidentialNodeType) + && self.nodePoolConfig.confidentialNodeType == "SEV") ? self.priorities.all(priority, + ((has(priority.machineFamily) && priority.machineFamily in [''n2d'', + ''c2d'', ''c3d'', ''c4d'']) || (has(priority.machineType) && priority.machineType.split(''-'')[0] + in [''n2d'', ''c2d'', ''c3d'', ''c4d'']))) : true' + - message: ConfidentialNodeType SEV_SNP only supports N2D + rule: '(has(self.nodePoolConfig) && has(self.nodePoolConfig.confidentialNodeType) + && self.nodePoolConfig.confidentialNodeType == "SEV_SNP") ? self.priorities.all(priority, + ((has(priority.machineFamily) && priority.machineFamily in [''n2d'']) + || (has(priority.machineType) && priority.machineType.split(''-'')[0] + in [''n2d'']))) : true' + - message: ConfidentialNodeType TDX only supports C3 standard, C4 standard, + A3 and A4 + rule: '(has(self.nodePoolConfig) && has(self.nodePoolConfig.confidentialNodeType) + && self.nodePoolConfig.confidentialNodeType == "TDX") ? self.priorities.all(priority, + (has(priority.machineFamily) && priority.machineFamily in [''c3'', + ''c4'', ''a3'', ''a4'']) || (has(priority.machineType) && (priority.machineType.startsWith(''c3-standard-'') + || priority.machineType.startsWith(''c4-standard-'') || priority.machineType + == ''a3-highgpu-1g'' || priority.machineType == ''a4-highgpu-8g''))) + : true' + - message: ConfidentialNodeType TDX on C3 only supports c3-standard- machine + type and nvidia-h100-80gb GPU type + rule: '(has(self.nodePoolConfig) && has(self.nodePoolConfig.confidentialNodeType) + && self.nodePoolConfig.confidentialNodeType == "TDX") ? self.priorities.all(priority, + (has(priority.machineFamily) && priority.machineFamily == ''c3'' || + has(priority.machineType) && priority.machineType.startsWith(''c3-standard-'')) + ? (!has(priority.gpu) || has(priority.gpu) && (!has(priority.gpu.type) + || priority.gpu.type == ''nvidia-h100-80gb'')) : true) : true' + - message: ConfidentialNodeType TDX on C4 only supports c4-standard- machine + type and nvidia-h100-80gb GPU type + rule: '(has(self.nodePoolConfig) && has(self.nodePoolConfig.confidentialNodeType) + && self.nodePoolConfig.confidentialNodeType == "TDX") ? self.priorities.all(priority, + (has(priority.machineFamily) && priority.machineFamily == ''c4'' || + has(priority.machineType) && priority.machineType.startsWith(''c4-standard-'')) + ? (!has(priority.gpu) || has(priority.gpu) && (!has(priority.gpu.type) + || priority.gpu.type == ''nvidia-h100-80gb'')) : true) : true' + - message: ConfidentialNodeType TDX on A3 only supports a3-highgpu-1g + machine type and nvidia-h100-80gb GPU type + rule: '(has(self.nodePoolConfig) && has(self.nodePoolConfig.confidentialNodeType) + && self.nodePoolConfig.confidentialNodeType == "TDX") ? self.priorities.all(priority, + (has(priority.machineFamily) && priority.machineFamily == ''a3'' || + has(priority.machineType) && priority.machineType == ''a3-highgpu-1g'') + ? (!has(priority.gpu) || has(priority.gpu) && (!has(priority.gpu.type) + || priority.gpu.type == ''nvidia-h100-80gb'')) : true) : true' + - message: ConfidentialNodeType TDX on A4 only supports a4-highgpu-8g + machine type and nvidia-b200 GPU type + rule: '(has(self.nodePoolConfig) && has(self.nodePoolConfig.confidentialNodeType) + && self.nodePoolConfig.confidentialNodeType == "TDX") ? self.priorities.all(priority, + (has(priority.machineFamily) && priority.machineFamily == ''a4'' || + has(priority.machineType) && priority.machineType == ''a4-highgpu-8g'') + ? (!has(priority.gpu) || has(priority.gpu) && (!has(priority.gpu.type) + || priority.gpu.type == ''nvidia-b200'')) : true) : true' + - message: PriorityScore must be set for all priorities or for none of + them + rule: self.priorities.all(p, has(p.priorityScore)) || self.priorities.all(p, + !has(p.priorityScore)) + status: + description: Status of the ComputeClass. + properties: + conditions: + description: Conditions represent the observations of a ComputeClass's + current state. + items: + description: Condition contains details for one aspect of the current + state of this API Resource. + properties: + lastTransitionTime: + description: 'lastTransitionTime is the last time the condition + transitioned from one status to another. + + This should be when the underlying condition changed. If + that is not known, then using the time when the API field + changed is acceptable.' + format: date-time + type: string + message: + description: 'message is a human readable message indicating + details about the transition. + + This may be an empty string.' + maxLength: 32768 + type: string + observedGeneration: + description: 'observedGeneration represents the .metadata.generation + that the condition was set based upon. + + For instance, if .metadata.generation is currently 12, but + the .status.conditions[x].observedGeneration is 9, the condition + is out of date + + with respect to the current state of the instance.' + format: int64 + minimum: 0 + type: integer + reason: + description: 'reason contains a programmatic identifier indicating + the reason for the condition''s last transition. + + Producers of specific condition types may define expected + values and meanings for this field, + + and whether the values are considered a guaranteed API. + + The value should be a CamelCase string. + + This field may not be empty.' + maxLength: 1024 + minLength: 1 + pattern: ^[A-Za-z]([A-Za-z0-9_,:]*[A-Za-z0-9_])?$ + type: string + status: + description: status of the condition, one of True, False, Unknown. + enum: + - 'True' + - 'False' + - Unknown + type: string + type: + description: type of condition in CamelCase or in foo.example.com/CamelCase. + maxLength: 316 + pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$ + type: string + required: + - lastTransitionTime + - message + - reason + - status + - type + type: object + type: array + type: object + required: + - spec + type: object + served: true + storage: true + subresources: + status: {} diff --git a/website/content/docs/decisions/0006-nap-computeclass-over-karpenter.md b/website/content/docs/decisions/0006-nap-computeclass-over-karpenter.md index aacff405e..bbf1f3ab3 100644 --- a/website/content/docs/decisions/0006-nap-computeclass-over-karpenter.md +++ b/website/content/docs/decisions/0006-nap-computeclass-over-karpenter.md @@ -125,6 +125,9 @@ arrives. ComputeClass is layered on afterwards, mirroring the AWS sequence. - Three `ComputeClass` manifests replace six Karpenter manifests. - Cilium's readiness taint is handled declaratively at the pool level rather than by a post-provisioning hook. + - *Amended 2026-08-24, after measuring it.* This is true but incomplete, and the omission + matters: see the toleration requirement under Negative. Declaring the taint works; making it + **transparent to workloads** does not. ### Negative @@ -133,6 +136,26 @@ arrives. ComputeClass is layered on afterwards, mirroring the AWS sequence. differ measurably between the two clouds. - *Mitigation*: the autoscaling slice ships an explicit written statement of the gap rather than leaving it to be discovered. This is a documented divergence, not an abstraction to be faked. +- **Every workload targeting a ComputeClass must tolerate `node.cilium.io/agent-not-ready`, + or nothing scales up at all.** Added 2026-08-24 from a live measurement, because this was not + anticipated when the ADR was written. + + The autoscaler simulates scheduling against a node that *will* carry the class's taint. A pod + that cannot tolerate it is judged unplaceable, so no node is provisioned — and the symptom is + silence: three Pending pods for ten minutes with **no `TriggeredScaleUp` event of any kind**, + only `FailedScheduling`. Adding the toleration alone produced four nodes and all pods Running. + GKE's admission webhook warns on apply; the warning is load-bearing. + + This is a real divergence from AWS, where the equivalent taint is invisible to workloads + because static pools carry it and Karpenter provisions against `NodePool` requirements rather + than simulating a tainted node. + - *Decision*: keep the taint and require the toleration, rather than dropping it. It still + gates every pod that does not opt in, and keeps auto-created nodes behaving like static ones. + - *Accepted cost*: a tolerating pod can land before the Cilium agent is up and log + `plugin type="cilium-cni" failed (add): unable to create endpoint ... EOF`. That is the CNI + present with its agent still starting, **not** a missing CNI, and it self-heals — measured 0 + restarts, pods reached `Running` unaided. `kube-system/metrics-server` hits the same transient + on any fresh node, so it is a property of the platform rather than of this decision. - No `min > 0` per auto-created pool, so "always keep N warm" must be expressed differently (for example a small static pool alongside the auto-created ones). - Karpenter knowledge does not transfer cleanly; operators need to learn a second model.