Add CUDA 13.3 / DOCA multi-arch NCCL test image for GB300 fleets

[sirajrauff]

What

Adds docker/Dockerfile.cuda13, the image we use for NCCL testing on Grace-Blackwell (GB300, aarch64) hardware, plus a docker/README.md section describing it.

It differs from docker/Dockerfile in four ways:

• Multi-arch. Builds linux/amd64 and linux/arm64; every arch-specific path derives from TARGETARCH or dpkg-architecture, so there is no hardcoded x86_64.
• DOCA-OFED userspace instead of the HPC-X tarball plus Ubuntu rdma-core, matching hosts that run a DOCA stack on CX7 and CX8. OpenMPI is installed after the DOCA repo so apt resolves it against DOCA's libibverbs.
• CUDA 13. GB300 is sm_103, and ptxas in 12.9 and earlier rejects it.
• sshd included, so one image serves as both MPIJob launcher and worker (mpi-operator v2 execs workers over ssh).

Build inputs are pinned by ARG — DOCA_VERSION, LIBNCCL_VERSION, NCCL_TESTS_REF — so a rebuild reproduces the same image; each is overridable at build time.

The open-files limit

The line worth reviewing closely:

RUN printf '* soft nofile 1048576\nroot soft nofile 1048576\n' > /etc/security/limits.d/99-nofile.conf

pam_limits clamps sshd sessions to Ubuntu's default soft RLIMIT_NOFILE of 1024. NCCL's proxy service polls an fd array sized roughly one entry per rank, so above 1024 ranks the poll() fails with EINVAL, the proxy thread exits, lazy transport connect never completes, and the job hangs silently — TCP bootstrap chatter continues, so it still looks alive. Since mpirun reaches workers as mpirun -> sshd -> orted, every rank inherits the clamped limit.

It reproduces purely by scale: jobs at or below 1024 ranks pass, jobs above 1024 ranks hang indefinitely. With this override in place, the same job completes.

The entries are soft only, deliberately. Raising the hard limit requires CAP_SYS_RESOURCE, which pods do not get, and sshd's PAM stack runs pam_limits as required — so on any runtime whose hard ceiling sits below the requested value, setrlimit fails and the module returns PAM_PERM_DENIED, refusing every worker login. A soft-only entry is clamped to the runtime's own ceiling instead, so it degrades to "as high as this host allows" rather than failing closed. Raise the hard ceiling in the CRI (default_ulimits) where a host needs more.

Testing

Built with docker buildx for linux/arm64 and published as ghcr.io/voltagepark/nccl-tests:cuda13.3.0-ubuntu24.04-doca-ssh-server.

Exercised as a Kubeflow MPIJob (all_reduce_perf, 512M–8G) across a scale ramp — single node, single NVL72 rack, multiple racks, then a full cluster of 1k+ GPUs. Every stage completed with zero out-of-bounds errors, and bus bandwidth at the largest scale stayed in the expected band relative to the single-rack baseline. The >1024-rank hang described above was the only failure mode encountered, and it disappeared once the limit override was added.

Note that the published image was built from an earlier revision of this file; the pins, the soft-only limits change, and the sshd CMD have not themselves been rebuilt yet.

[aikido-pr-checks]

Docker container runs as default root user - medium severity
By default, containers are run with root privileges and also run as the root user inside the container. Running the app as root gives a hacker who was able to hack the application instant root access to the Docker host, which could help them to escalate a hack.

Show fix

Remediation: Add 'USER username' to the end of your file.

_{Reply @AikidoSec ignore: [REASON] to ignore this issue.
More info}