feat(engines): make accelerator domain source labels configurable

CHANGELOG.md

@@ -11,7 +11,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
 - Pluggable accelerator-domain discovery for InfiniBand providers, independently selectable from fabric discovery with `nvidia-smi`, an explicitly configured Kubernetes Node label, or no accelerator source. Discovery is disabled when `accelerator` is omitted or empty; a non-empty section must set `source` explicitly. Helm defaults the `nvidia-smi` workload location to the `gpu-operator` namespace and `nvidia-device-plugin-daemonset` DaemonSet when those values are omitted.
 - Helm `kubeClient.qps` and `kubeClient.burst` values for tuning the DRA provider and the Kubernetes, NFD, and Slinky engine clients through deployment-level `KUBE_QPS` and `KUBE_BURST` settings.
 - The Kubernetes engine now publishes `accelerator.topograph.run/sub-domain` when a provider supplies `InstanceTopology.XclrSubDomainID`.
-- The NFD engine now publishes separate `xclr-domain` and `xclr-sub-domain` attributes and groups.
+- The NFD engine now publishes separate `accelerator-domain` and `accelerator-sub-domain` attributes and groups.
 - The graph engine now includes `accelerator.topograph.run/sub-domain` in instance labels when supplied alongside an XCLR domain.
 - Exported Go constant `topology.KeyTopologyXclrSubDomain` for the `accelerator.topograph.run/sub-domain` label key.
 - Slurm and Slinky block topology configurations can set `blockName.nodeNameRegexp` and `blockName.format` to derive unique block names from site-specific node naming conventions.
@@ -27,6 +27,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
 
 ### Changed
 
+- **BREAKING:** The Kubernetes, NFD, and Slinky engines now use the optional `acceleratorDomainSourceLabel` parameter to select an existing Kubernetes Node label as the authoritative accelerator-domain source. There is no default source label, so `nvidia.com/gpu.clique` is no longer implicitly authoritative. The Slinky `useGpuCliqueLabel` parameter has been removed. The k8s engine does not allow `acceleratorLabel` and `acceleratorDomainSourceLabel` to be configured together.
 - InfiniBand providers now query NVL partition IDs with the `nvidia-smi` CSV query interface, merge identical per-GPU rows, reject unavailable (`N/A`) fields, and normalize the result to `ClusterUUID.CliqueId`.
 - **BREAKING:** The default Kubernetes topology labels now use the vendor-neutral Topograph domains `fabric.topograph.run/tier-N`, `accelerator.topograph.run/domain`, and `accelerator.topograph.run/sub-domain`. Consumers must update topology keys, selectors, allowlists, and scheduling policies to use the new labels.
 - The node-observer now processes its existing topology-generation triggers through a client-go rate-limiting work queue, coalescing event bursts into a single cluster-wide reconciliation while preserving existing trigger and retry behavior.
@@ -42,7 +43,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
 - **BREAKING (Helm chart `0.5.0` → `0.6.0`):** the chart now ships a hardened default security context across the API server, node-observer, and node-data-broker: non-root (`runAsNonRoot`, UID/GID `65532`), `seccompProfile: RuntimeDefault`, `allowPrivilegeEscalation: false`, `readOnlyRootFilesystem: true`, and all capabilities dropped — satisfying the Kubernetes `restricted` Pod Security Standard out of the box. This changes the default runtime posture of every workload; operators who relied on root, a writable rootfs, or added capabilities must override the relevant keys (see the migration note below). `appVersion` is unchanged (`v0.5.0`; no binary change).
 - Go toolchain bumped to **1.26.5** (`go.mod`, `Dockerfile`, CI) to address reachable stdlib vulnerabilities reported by `govulncheck`.
 - Slinky partition discovery now prefers the Slinky controller pod and falls back to a login pod, so clusters without optional login pods can still discover partitions ([#362](https://github.com/NVIDIA/topograph/pull/362)).
-- Slinky engine `useGpuCliqueLabel` now emits an actionable diagnostic when no block domains can be built: the error reports how many nodes were scanned and why each was skipped (no Slurm mapping, missing `nvidia.com/gpu.clique` label, or missing the node-data-broker-written `topograph.nvidia.com/instance` annotation), and lists the offending node names. When no Kubernetes nodes are selected at all, it reports a distinct error pointing at the engine `nodeSelector`.
+- Slinky engine label-backed block-domain generation emits an actionable diagnostic when no domains can be built: the error reports the configured source label, how many nodes were scanned, and why each was skipped (no Slurm mapping, missing source label, or a missing/empty node-data-broker-written `topograph.nvidia.com/instance` annotation), and lists the offending node names. When no Kubernetes nodes are selected at all, it reports a distinct error pointing at the engine `nodeSelector`.
 - Simulation model YAML schema simplified: compute nodes are now declared through `blocks[].nodes`, with optional `blocks[].switch` attachment and `blocks[].labels`; the older separate `nodes` and `capacity_blocks` sections were removed from fixtures and docs ([#394](https://github.com/NVIDIA/topograph/pull/394)).
 
 ### Fixed
@@ -58,7 +59,6 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
 - Slinky dynamic-node reconciliation now reuses listed Node annotations, skips unchanged nodes without a per-node GET, and patches only changed topology annotations, substantially reducing Kubernetes client-side throttling on large clusters.
 - Corrected DRA provider guidance to document its Slinky-only block-topology scope, dependency on pre-existing `nvidia.com/gpu.clique` labels, and inability to guide placement across NVLink partitions without backend-fabric topology.
 - The NFD engine now rejects an empty generated object set when cleanup is enabled, preserving the last published topology instead of deleting every Topograph-managed NFD object after an empty provider result or over-narrow node selection.
-- The NFD engine now groups nodes with `nvidia.com/gpu.clique` by that authoritative accelerator value instead of omitting their accelerator attribute.
 - The NFD engine now publishes the `system.name/nodename` attribute required by NFD to populate `NodeFeatureGroup.status.nodes`, including for simulated KWOK nodes where no NFD worker executes.
 - The DRA provider now matches nodes using the `topograph.nvidia.com/instance` annotation instead of assuming the instance ID equals the Kubernetes node name.
 - `kwok-nodes` now maps generated instance IDs back to model hostnames when naming Kubernetes nodes and writing the Topograph instance annotation.

charts/topograph/templates/_validation.tpl

@@ -12,6 +12,15 @@
   {{- fail "env.NFD_NAMESPACE is managed by the chart for the nfd engine; configure nfdNamespace instead" }}
 {{- end }}
 
+{{- if eq .Values.engine.name "k8s" }}
+{{- $engineParams := default dict .Values.engine.params }}
+{{- $acceleratorLabel := trim (toString (get $engineParams "acceleratorLabel")) }}
+{{- $acceleratorDomainSourceLabel := trim (toString (get $engineParams "acceleratorDomainSourceLabel")) }}
+{{- if and (ne $acceleratorLabel "") (ne $acceleratorDomainSourceLabel "") }}
+  {{- fail "engine.params.acceleratorLabel and engine.params.acceleratorDomainSourceLabel cannot be set together for the k8s engine" }}
+{{- end }}
+{{- end }}
+
 {{- if hasKey (default dict .Values.env) "KUBE_QPS" }}
   {{- fail "env.KUBE_QPS is managed by the chart; configure kubeClient.qps instead" }}
 {{- end }}

charts/topograph/tests/validation_test.yaml

@@ -61,6 +61,17 @@ tests:
       - failedTemplate:
           errorMessage: "env.NFD_NAMESPACE is managed by the chart for the nfd engine; configure nfdNamespace instead"
 
+  - it: rejects accelerator output customization with an accelerator domain source label
+    set:
+      engine:
+        name: k8s
+        params:
+          acceleratorLabel: example.com/output-domain
+          acceleratorDomainSourceLabel: example.com/source-domain
+    asserts:
+      - failedTemplate:
+          errorMessage: "engine.params.acceleratorLabel and engine.params.acceleratorDomainSourceLabel cannot be set together for the k8s engine"
+
   - it: rejects overriding the chart-managed Kubernetes QPS environment variable
     set:
       env:

charts/topograph/values.schema.json

@@ -49,7 +49,14 @@
         },
         "params": {
           "type": "object",
-          "description": "Engine-specific parameters. The k8s engine accepts a closest-first fabricLabels array and a singular acceleratorLabel for the accelerator domain; accelerator sub-domains use the fixed accelerator.topograph.run/sub-domain key. Omitted values use the default label keys; when fabricLabels is provided, additional fabric tiers are omitted. For slinky, useGpuCliqueLabel=true reads nvidia.com/gpu.clique as the topology/block domain source."
+          "description": "Engine-specific parameters. The k8s engine accepts a closest-first fabricLabels array and a singular acceleratorLabel for the accelerator domain; accelerator sub-domains use the fixed accelerator.topograph.run/sub-domain key. Omitted values use the default label keys; when fabricLabels is provided, additional fabric tiers are omitted. The k8s, nfd, and slinky engines accept acceleratorDomainSourceLabel to select an existing Kubernetes Node label as the authoritative accelerator-domain source. There is no default. The k8s engine does not allow acceleratorLabel and acceleratorDomainSourceLabel to be set together.",
+          "properties": {
+            "acceleratorDomainSourceLabel": {
+              "type": "string",
+              "minLength": 1,
+              "description": "Optional Kubernetes Node label key used as the authoritative accelerator-domain source by the k8s, nfd, and slinky engines. Nodes without a non-empty value retain provider domains with k8s and nfd. Slinky excludes them from label-backed block domains and fails if no usable domains remain."
+            }
+          }
         }
       },
       "required": ["name"]

charts/topograph/values.slinky.block-example.yaml

@@ -19,7 +19,7 @@ engine:
         app.kubernetes.io/component: compute
     plugin: topology/block
     blockSizes: [4]
-    # useGpuCliqueLabel: true
+    # acceleratorDomainSourceLabel: example.com/accelerator-domain
     topologyConfigPath: topology.conf
     topologyConfigmapName: slurm-config
     useDynamicNodes: false

charts/topograph/values.slinky.ib.block-example.yaml

@@ -39,9 +39,10 @@ engine:
     # Base block size for topology/block. Additional entries, if used, must
     # be larger power-of-two multiples of the previous size.
     blockSizes: [4]
-    # For block topology, use nvidia.com/gpu.clique as the block-domain
-    # source. Keep this aligned with provider.params.accelerator.
-    useGpuCliqueLabel: true
+    # For block topology, use this existing Node label as the authoritative
+    # block-domain source. This engine setting is independent of provider
+    # discovery, even when both happen to select the same label.
+    acceleratorDomainSourceLabel: nvidia.com/gpu.clique
     # Key inside the target ConfigMap that Slinky mounts as topology.conf.
     topologyConfigPath: topology.conf
     # Target Slinky ConfigMap updated by Topograph with generated topology

charts/topograph/values.slinky.partition-example.yaml

@@ -44,7 +44,7 @@ engine:
         clusterDefault: true
     topologyConfigPath: topology.conf
     topologyConfigmapName: slurm-config
-    # useGpuCliqueLabel: true
+    # acceleratorDomainSourceLabel: example.com/accelerator-domain
     useDynamicNodes: true
     configUpdateMode: skeleton-only
 

charts/topograph/values.yaml

@@ -26,9 +26,13 @@ engine:
   # name: "k8s", "nfd", "slinky", "slurm" or "graph"
   name: k8s
   # params:
-  #   # For slinky topology/block output, use the GPU Operator's existing
-  #   # Kubernetes node label as the block-domain source.
-  #   useGpuCliqueLabel: true
+  #   # For the k8s, nfd, and slinky engines, optionally use an existing
+  #   # Kubernetes Node label as the authoritative accelerator-domain source.
+  #   # There is no default. The k8s engine does not allow acceleratorLabel
+  #   # to be customized when this parameter is set. Nodes missing the label
+  #   # retain provider domains with k8s and nfd; Slinky excludes them from
+  #   # label-backed block domains and fails if no usable domains remain.
+  #   acceleratorDomainSourceLabel: example.com/accelerator-domain
 
 # Kubernetes client rate limits for the DRA provider and the k8s, NFD, and
 # Slinky engines. The chart passes configured values to the Topograph deployment

demos/dra-slinky/values.dra-slinky.kwok.yaml

@@ -17,7 +17,6 @@ engine:
     blockName:
       nodeNameRegexp: '^srv([0-9]{2})'
       format: 'rack${1}'
-    useGpuCliqueLabel: true
     topologyConfigPath: topology.conf
     topologyConfigmapName: slurm-config-extra
     useDynamicNodes: true

docs/api.md

@@ -99,9 +99,9 @@ Topograph exposes three endpoints for interacting with the service. Below are th
       - **nodeSelector**: (optional) Used in: [`k8s`, `nfd`, `slinky`]. A Kubernetes node label map that filters which nodes participate in topology generation.
       - **fabricLabels**: (optional) Used in: [`k8s`]. Closest-first array of Kubernetes label keys for fabric tiers. If omitted, every discovered fabric tier uses its default numbered key; if provided, tiers beyond the array are omitted.
       - **acceleratorLabel**: (optional) Used in: [`k8s`]. Kubernetes label key for the accelerator domain. Defaults to `accelerator.topograph.run/domain`. The optional accelerator sub-domain always uses `accelerator.topograph.run/sub-domain`.
+      - **acceleratorDomainSourceLabel**: (optional) Used in: [`k8s`, `nfd`, `slinky`]. Existing Kubernetes Node label used as the authoritative accelerator-domain source. There is no default. Source-label overrides suppress provider accelerator sub-domains. For `k8s` and `nfd`, nodes without a non-empty source value retain their provider-derived accelerator domains. For Slinky block topology, those nodes are excluded from label-backed domains; generation fails if no usable label-backed domains can be constructed. For `k8s`, this parameter cannot be set together with `acceleratorLabel`, and the source label is never overwritten.
       - **topologyConfigmapName**: Used in: [`slinky`]. The required name of the ConfigMap containing the topology config.
       - **useDynamicNodes**: (optional) Used in: [`slinky`]. If `true`, Kubernetes nodes matched by the Node Selector will be annotated with the topology spec.
-      - **useGpuCliqueLabel**: (optional) Used in: [`slinky`]. If `true`, `topology/block` domains are built from the GPU Operator's `nvidia.com/gpu.clique` node label instead of provider accelerator-domain data.
       - **configUpdateMode**: (optional) Used in: [`slinky`]. By default, the full topology YAML is written in the Slurm ConfigMap. `skeleton-only` overrides to include switches or blocks only (no node lines); `none` skips updating the topology key in the ConfigMap.
   - **nodes**: (optional) Supplies the cluster nodes used for topology generation as an array of regions mapping instance IDs to node names.
 

docs/design/nfd-engine-sdd.md

@@ -9,7 +9,7 @@ Implemented.
 Add an experimental `nfd` engine that converts Topograph's canonical
 `topology.Graph` into Node Feature Discovery (NFD) `NodeFeatureGroup` objects.
 The engine creates one group for each distinct topology label value, such as one
-group for each distinct fabric tier, XCLR domain, or XCLR sub-domain value.
+group for each distinct fabric tier, accelerator domain, or accelerator sub-domain value.
 
 This should not replace the current `k8s` engine. The `k8s` engine writes node
 labels that can be consumed by native Kubernetes affinity and topology-aware
@@ -18,7 +18,7 @@ NFD CRs for consumers that already watch NFD.
 
 ## Background
 
-Topograph maps topology into optional XCLR dimensions and a variable-depth
+Topograph maps topology into optional accelerator dimensions and a variable-depth
 fabric label family:
 
 - `accelerator.topograph.run/domain`
@@ -74,8 +74,8 @@ spec:
           nodename: node-a
       topograph.network:
         elements:
-          xclr-domain: nvl3
-          xclr-sub-domain: nvl3.rack01
+          accelerator-domain: nvl3
+          accelerator-sub-domain: nvl3.rack01
           fabric-tier-0: leaf-12
           fabric-tier-1: spine-2
           fabric-tier-2: core-1
@@ -119,6 +119,11 @@ Initial parameters:
   graph. An empty generated object set is rejected while cleanup is enabled so
   a transient empty provider result cannot delete the entire published
   topology.
+- `acceleratorDomainSourceLabel`: optional Kubernetes Node label key. A
+  non-empty value replaces the provider accelerator domain for that node,
+  suppresses the provider accelerator sub-domain, and is recorded as the source
+  key on the corresponding `NodeFeatureGroup`. Nodes without a usable source
+  value retain provider-derived accelerator attributes.
 
 The NFD master namespace is deployment-scoped rather than request-scoped. Helm
 configures it through the top-level `nfdNamespace` value and passes it to the
@@ -131,8 +136,8 @@ returns an error if `NFD_NAMESPACE` is unset or blank.
 - Add `pkg/engines/nfd` with the standard `NamedLoader`.
 - Register it in `pkg/registry/registry.go`.
 - Factor the current `k8s` label projection into a shared helper so both engines
-  produce identical values at every discovered fabric tier, XCLR domain, and
-  XCLR sub-domain.
+  produce identical values at every discovered fabric tier, accelerator domain,
+  and accelerator sub-domain.
 - Use the dynamic Kubernetes client or generated NFD client types, depending on
   whether the project wants to pin an NFD API dependency.
 - Update Helm RBAC to allow create, update, patch, list, watch, and delete for
@@ -191,13 +196,13 @@ Two NFD-side extensions could make this model more practical:
 For a 10,000-node cluster, the live custom-resource payload depends on the
 number of topology dimensions and the number of distinct values within each
 dimension. Let `D` be the average number of published dimensions per node: all
-discovered fabric tiers plus any XCLR domain and sub-domain.
+discovered fabric tiers plus any accelerator domain and sub-domain.
 
 Assumptions:
 
 - 10,000 `NodeFeature` objects, one per node.
-- One topology attribute for every discovered fabric tier, XCLR domain, and
-  XCLR sub-domain.
+- One topology attribute for every discovered fabric tier, accelerator domain,
+  and accelerator sub-domain.
 - Each node appears in one `NodeFeatureGroup.status.nodes` list per topology
   dimension, so status contains about `10,000 × D` node references total.
 - Average node names and topology values are short, roughly 10-30 characters.
@@ -228,14 +233,14 @@ patches to reduce write amplification.
 
 ## Test Plan
 
-- Unit-test graph-to-group generation across variable fabric tiers, XCLR
-  domains, and XCLR sub-domains.
+- Unit-test graph-to-group generation across variable fabric tiers, accelerator
+  domains, and accelerator sub-domains.
 - Verify long and invalid topology values produce stable CR names.
 - Verify stale Topograph-managed groups are removed when `cleanup` is enabled.
 - Verify an empty generated object set returns an error and preserves existing
   objects when `cleanup` is enabled.
-- Verify nodes with `nvidia.com/gpu.clique` replace the provider XCLR domain
-  with the clique value and suppress the provider XCLR sub-domain, matching the
-  `k8s` engine's `skipXclrLabelsWhenGPUCliqueExists` behavior.
+- Verify nodes with a configured custom accelerator-domain source label replace
+  the provider accelerator domain and suppress the provider accelerator
+  sub-domain, while nodes without the label retain provider values.
 - Add a fake dynamic-client test that applies generated `NodeFeature` and
   `NodeFeatureGroup` objects without requiring a live NFD deployment.