Skip to content

EAI-7854 Spread CNPG instances across distinct nodes - #802

Draft
pre wants to merge 1 commit into
mainfrom
EAI-7854-cnpg-node-antiaffinity
Draft

EAI-7854 Spread CNPG instances across distinct nodes#802
pre wants to merge 1 commit into
mainfrom
EAI-7854-cnpg-node-antiaffinity

Conversation

@pre

@pre pre commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

Related:

Draft, and deliberately so. The sources/eai-infra/{aiwb,airm}-cnpg
copies here are mirrors of charts that live in silogen/core. What ArgoCD
actually deploys is the published OCI chart pinned at repoVersion: 2.0.0 in
root/values.yaml, not these directories, so the deployed clusters only pick
this fix up after core/#4334 merges, a core release is cut and promoted to
amdenterpriseai, and repoVersion is bumped here.

This PR is the placeholder holding the cluster-forge side of the change until
then.

TODO: Before merging

  • core/#4334 merged
  • Core release cut and charts promoted to amdenterpriseai
  • repoVersion bumped for the core-published entries in root/values.yaml ,
    with the matching sbom/components.yaml update
  • Confirm the mirrors still match core's charts at that tag, including the
    pre-existing managed.roles divergence these copies carry
  • Mark ready for review

Problem

Three CNPG clusters in this repo hardcode anti-affinity on
topology.kubernetes.io/zone:

affinity:
  enablePodAntiAffinity: true
  topologyKey: topology.kubernetes.io/zone

Most of our clusters are single-zone, and many nodes carry no zone label at
all. A topology key with one value across the cluster spreads nothing, so
instances co-schedule onto a single node. Losing that node takes down every
replica at once, and draining it evicts them together.

This is the caveat left open by the enablePDB work in #798: a PDB only means
something if the replicas it protects actually sit on different nodes.

Change

Spread on kubernetes.io/hostname, which every node has, and expose both knobs
as values instead of hardcoding them:

Value Default
podAntiAffinityType preferred
topologyKey kubernetes.io/hostname

preferred keeps single-node and small clusters working: instances land on
separate nodes where nodes allow it and co-schedule where they do not, so
nothing sits Pending. Set required to make the spread binding.

Files touched:

  • sources/eai-infra/aiwb-cnpg/0.1.0 and sources/eai-infra/airm-cnpg/0.1.0,
    mirroring core/#4334. Consumed only by docs/manual_helm_install.
  • sources/keycloak-config, which lives in this repo and is not mirrored.
    It runs instances: 1, so this is a no-op today and matters only if that
    count is ever raised. Included so all three CNPG clusters stay consistent
    rather than leaving one on a key that does not spread.

Testing

  • helm lint clean on both eai-infra charts
  • All three render the expected affinity block, and the eai-infra pair render
    correctly with --set podAntiAffinityType=required
  • helm template ./root passes on all four profiles (values, values_small,
    values_medium, values_large), matching the helm-chart-checks matrix

Note: helm template sources/keycloak-config fails on a pre-existing YAML
parse error in keycloak-realm-templates-cm.yaml, unrelated to this change and
reproducible on main. The cluster template was verified by rendering with
that file excluded.

The three CNPG clusters hardcoded anti-affinity on
topology.kubernetes.io/zone. Single-zone clusters have one value for
that key, and many nodes carry no zone label at all, so the spread was
a no-op and instances co-scheduled onto one node.

Spread on kubernetes.io/hostname instead, which every node has, and
expose podAntiAffinityType and topologyKey as values. The default
"preferred" keeps small and single-node clusters working, since
instances co-schedule rather than sit Pending when nodes run out.

keycloak-config runs a single instance, so this is a no-op there today
and matters only if its instance count is ever raised.
@pre
pre marked this pull request as ready for review August 7, 2026 08:28
@pre
pre requested a review from a team as a code owner August 7, 2026 08:28
@pre
pre marked this pull request as draft August 7, 2026 08:29

@silokimmo silokimmo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants