EAI-7854 Spread CNPG instances across distinct nodes - #802
Draft
pre wants to merge 1 commit into
Draft
Conversation
The three CNPG clusters hardcoded anti-affinity on topology.kubernetes.io/zone. Single-zone clusters have one value for that key, and many nodes carry no zone label at all, so the spread was a no-op and instances co-scheduled onto one node. Spread on kubernetes.io/hostname instead, which every node has, and expose podAntiAffinityType and topologyKey as values. The default "preferred" keeps small and single-node clusters working, since instances co-schedule rather than sit Pending when nodes run out. keycloak-config runs a single instance, so this is a no-op there today and matters only if its instance count is ever raised.
pre
marked this pull request as ready for review
August 7, 2026 08:28
pre
marked this pull request as draft
August 7, 2026 08:29
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related:
Draft, and deliberately so. The
sources/eai-infra/{aiwb,airm}-cnpgcopies here are mirrors of charts that live in
silogen/core. What ArgoCDactually deploys is the published OCI chart pinned at
repoVersion: 2.0.0inroot/values.yaml, not these directories, so the deployed clusters only pickthis fix up after core/#4334 merges, a core release is cut and promoted to
amdenterpriseai, andrepoVersionis bumped here.This PR is the placeholder holding the cluster-forge side of the change until
then.
TODO: Before merging
amdenterpriseairepoVersionbumped for the core-published entries in root/values.yaml ,with the matching
sbom/components.yamlupdatepre-existing
managed.rolesdivergence these copies carryProblem
Three CNPG clusters in this repo hardcode anti-affinity on
topology.kubernetes.io/zone:Most of our clusters are single-zone, and many nodes carry no zone label at
all. A topology key with one value across the cluster spreads nothing, so
instances co-schedule onto a single node. Losing that node takes down every
replica at once, and draining it evicts them together.
This is the caveat left open by the
enablePDBwork in #798: a PDB only meanssomething if the replicas it protects actually sit on different nodes.
Change
Spread on
kubernetes.io/hostname, which every node has, and expose both knobsas values instead of hardcoding them:
podAntiAffinityTypepreferredtopologyKeykubernetes.io/hostnamepreferredkeeps single-node and small clusters working: instances land onseparate nodes where nodes allow it and co-schedule where they do not, so
nothing sits
Pending. Setrequiredto make the spread binding.Files touched:
sources/eai-infra/aiwb-cnpg/0.1.0andsources/eai-infra/airm-cnpg/0.1.0,mirroring core/#4334. Consumed only by
docs/manual_helm_install.sources/keycloak-config, which lives in this repo and is not mirrored.It runs
instances: 1, so this is a no-op today and matters only if thatcount is ever raised. Included so all three CNPG clusters stay consistent
rather than leaving one on a key that does not spread.
Testing
helm lintclean on botheai-infrachartseai-infrapair rendercorrectly with
--set podAntiAffinityType=requiredhelm template ./rootpasses on all four profiles (values,values_small,values_medium,values_large), matching thehelm-chart-checksmatrixNote:
helm template sources/keycloak-configfails on a pre-existing YAMLparse error in
keycloak-realm-templates-cm.yaml, unrelated to this change andreproducible on
main. The cluster template was verified by rendering withthat file excluded.