Suggestion Description
What we do manually today
To give a container access to the RDMA NIC on an Instinct node, we hand-write the
following into every docker run command:
--device /dev/infiniband \
-v /usr/lib/x86_64-linux-gnu/libibverbs.so.1:/lib/x86_64-linux-gnu/libibverbs.so.1 \
-v /usr/lib/x86_64-linux-gnu/libionic.so.1:/usr/lib/x86_64-linux-gnu/libionic.so.1 \
-v /usr/lib/x86_64-linux-gnu/libibverbs/libionic-rdmav34.so:/usr/lib/x86_64-linux-gnu/libibverbs/libionic-rdmav34.so \
-v /etc/libibverbs.d:/etc/libibverbs.d:ro
The node's kernel already knows about the NIC. What is missing is on the container side:
RDMA is used through a userspace library, and unless that library is present inside the
container, ibv_devinfo cannot use the ionic RDMA device. The four mounts supply it, and
--device /dev/infiniband supplies the device file it opens.
What we would like instead
amd-ctk cdi generate already scans the host and writes /etc/cdi/amd.json describing
the device nodes and mount points a container needs for each GPU, so that
--device amd.com/gpu=all sets everything up. We would like the same treatment for the
NIC: have amd-ctk cdi generate also emit entries for the RDMA devices and the matching
host provider libraries, so the five flags above reduce to
docker run --device amd.com/gpu=all --device amd.com/rdma=all ...
Because the spec is generated on the host, the injected library would always match the
installed driver, which is the property we cannot get by baking it into an image.
Handling bnxt_re, ionic and mlx5 would cover the network stacks shipping with Instinct
systems today.
Why it matters
Every project ends up reimplementing this. ROCm/ATOM, for instance, ships
atom/mesh/scripts/docker_start.sh includes logic to detect the NIC type and assemble these mount flags. That logic belongs in the toolkit, where it can be generated from the host once, rather than being copied into each project's launch
scripts.
Operating System
Ubuntu 24.04
GPU
MI355X
ROCm Component
No response
Suggestion Description
What we do manually today
To give a container access to the RDMA NIC on an Instinct node, we hand-write the
following into every
docker runcommand:The node's kernel already knows about the NIC. What is missing is on the container side:
RDMA is used through a userspace library, and unless that library is present inside the
container,
ibv_devinfocannot use the ionic RDMA device. The four mounts supply it, and--device /dev/infinibandsupplies the device file it opens.What we would like instead
amd-ctk cdi generatealready scans the host and writes/etc/cdi/amd.jsondescribingthe device nodes and mount points a container needs for each GPU, so that
--device amd.com/gpu=allsets everything up. We would like the same treatment for theNIC: have
amd-ctk cdi generatealso emit entries for the RDMA devices and the matchinghost provider libraries, so the five flags above reduce to
Because the spec is generated on the host, the injected library would always match the
installed driver, which is the property we cannot get by baking it into an image.
Handling bnxt_re, ionic and mlx5 would cover the network stacks shipping with Instinct
systems today.
Why it matters
Every project ends up reimplementing this. ROCm/ATOM, for instance, ships
atom/mesh/scripts/docker_start.shincludes logic to detect the NIC type and assemble these mount flags. That logic belongs in the toolkit, where it can be generated from the host once, rather than being copied into each project's launchscripts.
Operating System
Ubuntu 24.04
GPU
MI355X
ROCm Component
No response