[FEA] Provider-independent GPU-to-NIC rail mapping for Kubernetes

[dmitsh]

Summary

Add an optional Kubernetes node-metadata plugin that maps each physical GPU to
the closest host-visible NICs on every network rail.

The feature is independent of provider.name and does not change
topology.Graph. The node-data-broker discovers and publishes the record for
its Node; the Node Observer creates and owns the shared ConfigMap and removes
stale records.

The initial rail source uses host LLDP. A future DOCA XPlane source is an
extension point, not part of the initial implementation, because a supported
external topology contract providing host PCI identity has not been confirmed.

Motivation

GPU-aware scheduling and device allocation may need more than node-to-switch
topology: for every selected GPU, a consumer needs the closest NIC on each
rail. This is node-local PCIe metadata and should be composable with any
Kubernetes topology provider rather than implemented separately by each
provider or added to the canonical topology graph.

Feature implementation

Responsibilities

Component	Responsibility
Node-data-broker plugin	Discover NIC rails and GPU locality; publish its Node record
Node Observer	Create and own the shared ConfigMap; remove deleted or replaced Node records
Topology provider	Continue producing canonical fabric and accelerator topology independently

The current broker path that returns LLDP provider annotations and NIC rails
together must be split. Provider collection should return only provider
annotations; the plugin should invoke reusable LLDP and device-resolution
helpers independently.

Discovery and selection

A rail source implements:

DiscoverNICRails(context) -> map[host PCI address][]rail ID

The initial host-lldp source:

1. Reads lldpctl -f json through the host lldpd socket.
2. Selects configured rail interfaces and derives rail IDs from an interface
   regular expression and expansion template.
3. Resolves /sys/class/net/<interface>/device to normalized host PCI
   addresses.

On GPU Nodes, the broker executes the following in exactly one configured,
same-node GPU Operator pod:

nvidia-smi --query-gpu=index,uuid,pci.bus_id --format=csv,noheader
nvidia-smi topo -m

NIC aliases are resolved through /sys/class/infiniband/<device>/device. The
rail and GPU views are joined by normalized PCI address. For every GPU and
rail, retain all NICs tied at the closest supported path rank:

PIX < PXB < PHB < NODE < SYS

Unknown path classes or a discovered rail without a qualified NIC fail the
Node update. NICs present in nvidia-smi but absent from rail discovery are
ignored.

Shared record and lifecycle

The shared ConfigMap uses each Kubernetes Node name as a data key. Each value
is one versioned JSON record containing:

• nodeUID and railSource;
• host NIC PCI address to rail arrays; and
• physical GPU UUID and PCI address to the preferred NIC candidates per rail,
  including the selected path class.

Each broker atomically patches only its Node key. Discovery or publication
errors preserve the previous value and keep the broker unhealthy. A successful
empty rail result removes that Node's record.

The Node Observer creates the ConfigMap with a controller owner reference to
its Deployment and refuses to adopt an unrelated same-named object. It removes
a record when the Node no longer exists or its UID no longer matches. Cleanup
uses an atomic test-and-remove JSON Patch so rapid Node-name reuse cannot
delete a replacement record. A full reconciliation after informer cache sync
cleans deletion events missed during observer downtime.

Deployment mechanism

The proposed Helm integration configures the plugin once, outside provider and
component-specific sections:

nodeMetadata:
  plugins:
    gpuNicRailMapping:
      enabled: true
      configMapName: topograph-nic-rails
      railSource:
        name: host-lldp
        hostLLDP:
          interfaceRegex: '^eth_r([0-9]+)_p[0-9]+$'
          railID: 'rail$1'
      gpuSource:
        gpuOperatorNamespace: gpu-operator
        daemonSet: nvidia-device-plugin-daemonset

The chart wires configuration to both components, adds source-specific mounts,
and grants conditional RBAC. The broker receives pod-exec and named-ConfigMap
patch access. The observer receives Node watch access, access to its Deployment,
and namespace-scoped ConfigMap lifecycle permissions. The chart does not render
the shared ConfigMap; the observer creates it at runtime.

xplane must remain rejected by deployment validation until its supported API,
transport, authentication, and host-visible PCI mapping are defined.

Non-goals and limitations

• Changing the canonical graph or scheduler topology labels
• Discovering the complete physical switch fabric
• Implementing lldp-bm or another bare-metal provider
• Allocating devices or advertising kubelet resources
• Enabling or validating GPUDirect RDMA
• Publishing separate MIG-device mappings
• Loading external runtime plugins

The initial broker reconciliation is startup-only. The shared ConfigMap has an
approximate 1 MiB limit; larger clusters may need one object per Node or a
custom resource.

Acceptance criteria

• Add a provider-independent, compiled-in GPU-NIC rail-mapping plugin
• Separate LLDP provider annotation collection from plugin rail discovery
• Implement and test the host-lldp rail source and PCI normalization
• Parse stable GPU identity and GPU-to-NIC locality from nvidia-smi
• Select the closest NIC candidates independently for every GPU and rail
• Publish deterministic, versioned per-Node records with atomic key updates
• Make Node Observer create and own the shared ConfigMap
• Garbage-collect deleted and replaced Node records using nodeUID
• Preserve previous records on discovery and publication failures
• Add provider-independent top-level configuration
• Add conditional broker/observer RBAC, mounts, and Helm validation
• Cover concurrent publication, observer restart, ownership conflicts, and
  Node-name reuse
• Validate on multi-GPU/multi-rail hardware, including ties and a pre-R610
  driver
• Document the record contract, prerequisites, limitations, and verification

Open questions

1. Which identifiers will the consumer allocate: GPU UUIDs, MIG UUIDs, PCI
   addresses, or CDI names?
2. What maximum cluster size must the shared ConfigMap support?
3. Which supported XPlane API supplies both rail identity and host-visible PCI
   identity?
4. Does a consumer also require physical neighbor-switch and port identity?