Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).

### Changed

- **BREAKING:** Topograph-owned Kubernetes labels and annotations now use the `topograph.run` domain. Consumers must update selectors, allowlists, and metadata lookups to use the new keys.
- **BREAKING:** The Kubernetes, NFD, and Slinky engines now use the optional `acceleratorDomainSourceLabel` parameter to select an existing Kubernetes Node label as the authoritative accelerator-domain source. There is no default source label, so `nvidia.com/gpu.clique` is no longer implicitly authoritative. The Slinky `useGpuCliqueLabel` parameter has been removed. The k8s engine does not allow `acceleratorLabel` and `acceleratorDomainSourceLabel` to be configured together.
- InfiniBand providers now query NVL partition IDs with the `nvidia-smi` CSV query interface, merge identical per-GPU rows, reject unavailable (`N/A`) fields, and normalize the result to `ClusterUUID.CliqueId`.
- **BREAKING:** The default Kubernetes topology labels now use the vendor-neutral Topograph domains `fabric.topograph.run/tier-N`, `accelerator.topograph.run/domain`, and `accelerator.topograph.run/sub-domain`. Consumers must update topology keys, selectors, allowlists, and scheduling policies to use the new labels.
Expand All @@ -44,7 +45,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
- **BREAKING (Helm chart `0.5.0` → `0.6.0`):** the chart now ships a hardened default security context across the API server, node-observer, and node-data-broker: non-root (`runAsNonRoot`, UID/GID `65532`), `seccompProfile: RuntimeDefault`, `allowPrivilegeEscalation: false`, `readOnlyRootFilesystem: true`, and all capabilities dropped — satisfying the Kubernetes `restricted` Pod Security Standard out of the box. This changes the default runtime posture of every workload; operators who relied on root, a writable rootfs, or added capabilities must override the relevant keys (see the migration note below). `appVersion` is unchanged (`v0.5.0`; no binary change).
- Go toolchain bumped to **1.26.5** (`go.mod`, `Dockerfile`, CI) to address reachable stdlib vulnerabilities reported by `govulncheck`.
- Slinky partition discovery now prefers the Slinky controller pod and falls back to a login pod, so clusters without optional login pods can still discover partitions ([#362](https://github.com/NVIDIA/topograph/pull/362)).
- Slinky engine label-backed block-domain generation emits an actionable diagnostic when no domains can be built: the error reports the configured source label, how many nodes were scanned, and why each was skipped (no Slurm mapping, missing source label, or a missing/empty node-data-broker-written `topograph.nvidia.com/instance` annotation), and lists the offending node names. When no Kubernetes nodes are selected at all, it reports a distinct error pointing at the engine `nodeSelector`.
- Slinky engine label-backed block-domain generation emits an actionable diagnostic when no domains can be built: the error reports the configured source label, how many nodes were scanned, and why each was skipped (no Slurm mapping, missing source label, or a missing/empty node-data-broker-written `topograph.run/instance` annotation), and lists the offending node names. When no Kubernetes nodes are selected at all, it reports a distinct error pointing at the engine `nodeSelector`.
- Simulation model YAML schema simplified: compute nodes are now declared through `blocks[].nodes`, with optional `blocks[].switch` attachment and `blocks[].labels`; the older separate `nodes` and `capacity_blocks` sections were removed from fixtures and docs ([#394](https://github.com/NVIDIA/topograph/pull/394)).

### Fixed
Expand All @@ -55,13 +56,13 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
- Slinky node-resolution logs now explain that an unresolved Kubernetes node lacks a mapping from a Ready Slurm pod and identify the relevant namespace, pod selector, readiness, and node-name metadata checks.
- The DRA/Slinky KWOK demo now waits for all simulated worker pods before installing Topograph and retriggers topology generation when those pods become Ready, avoiding startup races during slow image pulls.
- The node-observer now remains active after informer startup and retriggers topology generation when an API-server pod is deleted, preventing rolling deployments from losing requests accepted by a retiring pod.
- `kwok-nodes` now prevents model-provided metadata from overriding its required KWOK selector and model-derived `topograph.nvidia.com/instance` and `topograph.nvidia.com/region` annotations.
- `kwok-nodes` now prevents model-provided metadata from overriding its required KWOK selector and model-derived `topograph.run/instance` and `topograph.run/region` annotations.
- Kubernetes engine label reconciliation now reuses the listed Nodes, skips unchanged labels without a per-node GET, and patches only changed topology labels, substantially reducing Kubernetes client-side throttling on large clusters.
- Slinky dynamic-node reconciliation now reuses listed Node annotations, skips unchanged nodes without a per-node GET, and patches only changed topology annotations, substantially reducing Kubernetes client-side throttling on large clusters.
- Corrected DRA provider guidance to document its Slinky-only block-topology scope, dependency on pre-existing `nvidia.com/gpu.clique` labels, and inability to guide placement across NVLink partitions without backend-fabric topology.
- The NFD engine now rejects an empty generated object set when cleanup is enabled, preserving the last published topology instead of deleting every Topograph-managed NFD object after an empty provider result or over-narrow node selection.
- The NFD engine now publishes the `system.name/nodename` attribute required by NFD to populate `NodeFeatureGroup.status.nodes`, including for simulated KWOK nodes where no NFD worker executes.
- The DRA provider now matches nodes using the `topograph.nvidia.com/instance` annotation instead of assuming the instance ID equals the Kubernetes node name.
- The DRA provider now matches nodes using the `topograph.run/instance` annotation instead of assuming the instance ID equals the Kubernetes node name.
- `kwok-nodes` now maps generated instance IDs back to model hostnames when naming Kubernetes nodes and writing the Topograph instance annotation.
- The node-observer now discovers the optional node-data-broker through `NODE_DATA_BROKER_NAME` and `NODE_DATA_BROKER_NAMESPACE` and gates topology generation on the broker DaemonSet's desired and ready replica counts. Helm injects the variables only when `nodeDataBroker.enabled=true`, so disabling the broker cannot leave the observer waiting for nonexistent pods.
- The node-observer reports an actionable error and defers topology generation when an enabled node-data-broker DaemonSet has zero desired replicas.
Expand All @@ -71,7 +72,6 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).

### Removed

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Clique removal guidance is dropped

This deletes the unreleased breaking-change notice for topology.KeyNvidiaGPUClique even though the constant remains removed. Consumers reviewing the release notes consequently lose the applicable migration guidance to configure accelerator discovery through provider.params.accelerator.kubernetesLabel.key.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

- **BREAKING:** Removed the exported Go constant `topology.KeyNvidiaGPUClique`. The GPU Operator's `nvidia.com/gpu.clique` label is now a DRA provider default rather than part of the canonical topology API; configure label-backed accelerator discovery through `provider.params.accelerator.kubernetesLabel.key`.
- Periodic node annotation refreshes, including the `--refresh-interval` broker flag and `nodeDataBroker.refreshInterval` Helm value. The broker now applies annotations once at startup.

### Security
Expand Down
2 changes: 1 addition & 1 deletion charts/topograph/Chart.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ sources:
- https://github.com/NVIDIA/topograph
maintainers:
- name: NVIDIA CORPORATION
url: https://www.nvidia.com
url: https://topograph.run
keywords:
- topograph
- topology
Expand Down
2 changes: 1 addition & 1 deletion demos/test-nfd/demo.sh
Original file line number Diff line number Diff line change
Expand Up @@ -27,4 +27,4 @@ step "KUBE_CONTEXT=\"$KUBE_CONTEXT\" ./scripts/install-topograph.sh demos/test-n

step "kubectl --context \"$KUBE_CONTEXT\" -n node-feature-discovery get nodefeaturegroups"

step "kubectl --context \"$KUBE_CONTEXT\" -n node-feature-discovery get nodefeaturegroups -l topograph.nvidia.com/group-type=fabric-tier-1 -o yaml"
step "kubectl --context \"$KUBE_CONTEXT\" -n node-feature-discovery get nodefeaturegroups -l topograph.run/group-type=fabric-tier-1 -o yaml"
6 changes: 3 additions & 3 deletions docs/design/nfd-engine-sdd.md
Original file line number Diff line number Diff line change
Expand Up @@ -90,10 +90,10 @@ metadata:
name: topograph-fabric-tier-0-leaf-12-...
labels:
app.kubernetes.io/managed-by: topograph
topograph.nvidia.com/group-type: fabric-tier-0
topograph.run/group-type: fabric-tier-0
annotations:
topograph.nvidia.com/label-key: fabric.topograph.run/tier-0
topograph.nvidia.com/label-value: leaf-12
topograph.run/label-key: fabric.topograph.run/tier-0
topograph.run/label-value: leaf-12
spec:
featureGroupRules:
- name: fabric-tier-0 equals leaf-12
Expand Down
2 changes: 1 addition & 1 deletion docs/engines/graph.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ Each instance entry contains:
"id": "I21",
"network_layers": ["leaf-a", "spine-a"],
"labels": {
"nvidia.com/gpu.product": "H100",
"example.com/gpu-product": "H100",
"accelerator.topograph.run/domain": "nvl-1",
"accelerator.topograph.run/sub-domain": "nvl-1.rack-1"
}
Expand Down
10 changes: 5 additions & 5 deletions docs/engines/nfd.md
Original file line number Diff line number Diff line change
Expand Up @@ -121,7 +121,7 @@ metadata:
labels:
nfd.node.kubernetes.io/node-name: node-a
app.kubernetes.io/managed-by: topograph
topograph.nvidia.com/engine: nfd
topograph.run/engine: nfd
spec:
features:
attributes:
Expand All @@ -147,11 +147,11 @@ metadata:
namespace: node-feature-discovery
labels:
app.kubernetes.io/managed-by: topograph
topograph.nvidia.com/engine: nfd
topograph.nvidia.com/group-type: fabric-tier-0
topograph.run/engine: nfd
topograph.run/group-type: fabric-tier-0
annotations:
topograph.nvidia.com/label-key: fabric.topograph.run/tier-0
topograph.nvidia.com/label-value: leaf-12
topograph.run/label-key: fabric.topograph.run/tier-0
topograph.run/label-value: leaf-12
spec:
featureGroupRules:
- name: fabric-tier-0 equals leaf-12
Expand Down
24 changes: 12 additions & 12 deletions docs/engines/slinky.md
Original file line number Diff line number Diff line change
Expand Up @@ -163,12 +163,12 @@ metadata:
name: slurm-config
annotations:
# Topograph metadata
topograph.nvidia.com/engine: "slinky"
topograph.nvidia.com/topology-managed-by: "topograph"
topograph.nvidia.com/last-updated: "2024-01-01T10:11:00Z"
topograph.nvidia.com/slurm-namespace: "slurm"
topograph.nvidia.com/plugin: "topology/tree"
topograph.nvidia.com/block-sizes: "8,16,32"
topograph.run/engine: "slinky"
topograph.run/topology-managed-by: "topograph"
topograph.run/last-updated: "2024-01-01T10:11:00Z"
topograph.run/slurm-namespace: "slurm"
topograph.run/plugin: "topology/tree"
topograph.run/block-sizes: "8,16,32"

# Original annotations preserved
meta.helm.sh/release-name: slurm
Expand All @@ -184,12 +184,12 @@ data:

| Annotation | Description |
| ------------------------------------------ | ----------------------------------------- |
| `topograph.nvidia.com/engine` | Engine that manages this ConfigMap |
| `topograph.nvidia.com/topology-managed-by` | Indicates topograph manages topology data |
| `topograph.nvidia.com/last-updated` | RFC3339 timestamp of last update |
| `topograph.nvidia.com/slurm-namespace` | SLURM cluster namespace |
| `topograph.nvidia.com/plugin` | Topology plugin used (tree/block) |
| `topograph.nvidia.com/block-sizes` | Block sizes for block topology |
| `topograph.run/engine` | Engine that manages this ConfigMap |
| `topograph.run/topology-managed-by` | Indicates topograph manages topology data |
| `topograph.run/last-updated` | RFC3339 timestamp of last update |
| `topograph.run/slurm-namespace` | SLURM cluster namespace |
| `topograph.run/plugin` | Topology plugin used (tree/block) |
| `topograph.run/block-sizes` | Block sizes for block topology |

## Usage Examples

Expand Down
6 changes: 3 additions & 3 deletions docs/modeling.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,13 +57,13 @@ To pass a kind cluster configuration file, add `--kind-config path/to/kind.yaml`

The utility uses the model-derived instance-to-hostname mapping, so model hostname `1101` becomes Kubernetes node `1101` with:

- `topograph.nvidia.com/instance: i-1101`
- `topograph.nvidia.com/region: <derived-region-or-none>`
- `topograph.run/instance: i-1101`
- `topograph.run/region: <derived-region-or-none>`
- `kwok.x-k8s.io/node=fake` as both a label and annotation
- Model-derived labels such as `topology.kubernetes.io/region` and `topology.kubernetes.io/zone`
- Model-derived annotations such as `accelerator.topology.test/domain`

Generated Kubernetes node names come from model hostnames and are normalized to valid lowercase DNS names. For example, model hostname `I21` becomes Kubernetes node `i21`, while its generated instance ID `i-I21` is stored in `topograph.nvidia.com/instance`.
Generated Kubernetes node names come from model hostnames and are normalized to valid lowercase DNS names. For example, model hostname `I21` becomes Kubernetes node `i21`, while its generated instance ID `i-I21` is stored in `topograph.run/instance`.

The script applies the manifest with kubeconfig context `kind-<cluster>`, matching the context name created by `kind create cluster --name=<cluster>`.

Expand Down
4 changes: 2 additions & 2 deletions docs/providers/dra.md
Original file line number Diff line number Diff line change
Expand Up @@ -111,10 +111,10 @@ Before triggering topology generation, verify that the configured labels exist o
kubectl get nodes -o json | jq '.items[] | {name: .metadata.name, clique: .metadata.labels["nvidia.com/gpu.clique"]}'
```

If topology generation returns a `502` error, check that the expected nodes have the configured source label and the `topograph.nvidia.com/region` / `topograph.nvidia.com/instance` annotations (the latter two are set by Topograph itself during topology discovery). For the default label:
If topology generation returns a `502` error, check that the expected nodes have the configured source label and the `topograph.run/region` / `topograph.run/instance` annotations (the latter two are set by Topograph itself during topology discovery). For the default label:

```bash
kubectl get nodes -o json | jq '.items[] | {name: .metadata.name, clique: .metadata.labels["nvidia.com/gpu.clique"], region: .metadata.annotations["topograph.nvidia.com/region"], instance: .metadata.annotations["topograph.nvidia.com/instance"]}'
kubectl get nodes -o json | jq '.items[] | {name: .metadata.name, clique: .metadata.labels["nvidia.com/gpu.clique"], region: .metadata.annotations["topograph.run/region"], instance: .metadata.annotations["topograph.run/instance"]}'
```

See the [Slinky engine documentation](../engines/slinky.md) for details on `topology/block` output.
6 changes: 3 additions & 3 deletions docs/providers/infiniband.md
Original file line number Diff line number Diff line change
Expand Up @@ -91,14 +91,14 @@ For the Slurm engine, verify the generated `topology.conf` reflects the expected

### Prerequisites

- Topograph deployed via Helm — when `accelerator.source` is `nvidia-smi`, the node-data-broker DaemonSet collects NVLink partition IDs from each node and stores them as Kubernetes node annotations (`topograph.nvidia.com/cluster-id`). With `kubernetes-label` or `none`, the broker skips that collection.
- Topograph deployed via Helm — when `accelerator.source` is `nvidia-smi`, the node-data-broker DaemonSet collects NVLink partition IDs from each node and stores them as Kubernetes node annotations (`topograph.run/cluster-id`). With `kubernetes-label` or `none`, the broker skips that collection.
- The default **`ghcr.io/nvidia/topograph`** image includes **`ibnetdiscover`** (Alpine `rdma-core`). No separate InfiniBand image is required. IB deployments typically run the broker **privileged** and mount host **`/sys/class`** so `ibnetdiscover` can reach IB devices — see [`values.k8s.ib-example.yaml`](../../charts/topograph/values.k8s.ib-example.yaml).
- NVIDIA GPU Operator — standard on NVIDIA GPU Kubernetes clusters; manages the device plugin DaemonSet used to read NVLink clique IDs. Required only for NVLink domain discovery; on clusters without NVLink-connected GPUs this does not apply and the provider will still discover the IB switch tree.

### How It Works

1. Runs `ibnetdiscover` by exec-ing into a node-data-broker pod on each node to map the switch tree
2. When `provider.params.accelerator` is non-empty, resolves accelerator domains independently using its `source`: `nvidia-smi` reads the broker-written `topograph.nvidia.com/cluster-id` annotation, `kubernetes-label` reads a configured Node label, and `none` explicitly disables accelerator discovery. Omitting the section or using an empty object also disables it. NVL partition IDs use `ClusterUUID.CliqueId`, the same format as `nvidia.com/gpu.clique`.
2. When `provider.params.accelerator` is non-empty, resolves accelerator domains independently using its `source`: `nvidia-smi` reads the broker-written `topograph.run/cluster-id` annotation, `kubernetes-label` reads a configured Node label, and `none` explicitly disables accelerator discovery. Omitting the section or using an empty object also disables it. NVL partition IDs use `ClusterUUID.CliqueId`, the same format as `nvidia.com/gpu.clique`.
3. Combines the switch tree and any NVLink clique data into the topology graph

### Configuration
Expand Down Expand Up @@ -129,7 +129,7 @@ The following optional parameters can be passed in the topology request payload:
| `accelerator.source` | `string` | — | Required when the `accelerator` section is non-empty. Accelerator-domain source: `nvidia-smi`, `kubernetes-label`, or `none`. |
| `accelerator.kubernetesLabel.key` | `string` | — | Required for the `kubernetes-label` source. Kubernetes Node label read as the accelerator-domain ID; no default is assumed. |

For a manual request, keep `accelerator.source` consistent with the source configured for the deployed node-data-broker. In particular, `source: nvidia-smi` reads the broker-written `topograph.nvidia.com/cluster-id` annotation; it does not execute `nvidia-smi` or change the broker's GPU Operator workload target.
For a manual request, keep `accelerator.source` consistent with the source configured for the deployed node-data-broker. In particular, `source: nvidia-smi` reads the broker-written `topograph.run/cluster-id` annotation; it does not execute `nvidia-smi` or change the broker's GPU Operator workload target.

#### Helm node-data-broker settings

Expand Down
4 changes: 2 additions & 2 deletions docs/providers/lambdai.md
Original file line number Diff line number Diff line change
Expand Up @@ -236,8 +236,8 @@ With the `k8s` engine you do not pass `nodes` explicitly. Instead, the node-data

| Node field (set by `lambda-cloud-controller`) | Topograph annotation (set by node-data-broker) |
|---|---|
| `.spec.providerID` — `lambda://<instance-id>` | `topograph.nvidia.com/instance` — `<instance-id>` (matches the API `id` 1:1) |
| `topology.kubernetes.io/region` label — e.g. `stg-sjc01-cl03` | `topograph.nvidia.com/region` |
| `.spec.providerID` — `lambda://<instance-id>` | `topograph.run/instance` — `<instance-id>` (matches the API `id` 1:1) |
| `topology.kubernetes.io/region` label — e.g. `stg-sjc01-cl03` | `topograph.run/region` |

The Kubernetes engine then discovers nodes from these annotations, the provider queries the Lambda API once per region, and the engine writes `fabric.topograph.run/*` labels. The Node Observer re-triggers generation when nodes change.

Expand Down
Loading
Loading