multinode benchmarks fail to start with error - [CUDA_ERROR_INVALID_VALUE] invalid argument in expression cuMemSetAccess

[chtcvl]

Running multinode benchmarks works on single node, but fails with the following error when running on multiple nodes:

mpirun --map-by ppr:8:node --bind-to core -np 16 --report-bindings --hostfile hostfile  /usr/local/bin/nvbandwidth -p multinode
...
Running multinode_device_to_device_memcpy_read_ce.
[CUDA_ERROR_INVALID_VALUE] invalid argument in expression cuMemSetAccess((CUdeviceptr) buffer, roundedUpAllocationSize, &desc, 1 ) on gpub200-nom6ae0518, rank = 8 in MultinodeMemoryAllocationUnicast::MultinodeMemoryAllocationUnicast(size_t, int)() : /home/code/external/nvbandwidth/multinode_memcpy.cpp:76

Binary was build with Cuda 12.8

[esitaridi]

@chtcvl is imex setup and running? Could you share the output of :
nvidia-smi nvlink -s

[chtcvl]

Yes, nvidia-imex.service is running, both nodes are in Connected state.

$ nvidia-smi nvlink -s
GPU 0: NVIDIA B200 (UUID: GPU-19a556ea-38df-750d-fcf7-3da28757ba5b)
         Link 0: 50 GB/s
         Link 1: 50 GB/s
         Link 2: 50 GB/s
         Link 3: 50 GB/s
         Link 4: 50 GB/s
         Link 5: 50 GB/s
         Link 6: 50 GB/s
         Link 7: 50 GB/s
         Link 8: 50 GB/s
         Link 9: 50 GB/s
         Link 10: 50 GB/s
         Link 11: 50 GB/s
         Link 12: 50 GB/s
         Link 13: 50 GB/s
         Link 14: 50 GB/s
         Link 15: 50 GB/s
         Link 16: 50 GB/s
         Link 17: 50 GB/s
...

(redacted same output for remaining 7 GPUs - all links are up)

[chtcvl]

@esitaridi is the above output sufficient or can anything else be of help to figure out what's the issue?

[developerisanonymous]

@chtcvl wait for IMEX service to finish restarting on all nodes before trying to run the multi-node nvbandwidth test. This worked for me.

[esitaridi]

@chtcvl do you still experience the issue? If it happens again could you also share the output of: $ systemctl status nvidia-imex.service ?

[pzcyyj]

@esitaridi I have the same issue, my output of: $ systemctl status nvidia-imex.service

ubuntu@192-168-0-173:~/nvbandwidth-main$ systemctl status nvidia-imex.service
● nvidia-imex.service - NVIDIA IMEX service
     Loaded: loaded (/lib/systemd/system/nvidia-imex.service; disabled; vendor preset: enabled)
     Active: active (running) since Tue 2025-09-02 15:30:55 +08; 24min ago
    Process: 42000 ExecStart=/usr/bin/nvidia-imex -c /etc/nvidia-imex/config.cfg (code=exited, status=0/SUCCESS)
   Main PID: 42002 (nvidia-imex)
      Tasks: 32 (limit: 629145)
     Memory: 7.0M
        CPU: 5.391s
     CGroup: /system.slice/nvidia-imex.service
             └─42002 /usr/bin/nvidia-imex -c /etc/nvidia-imex/config.cfg

Sep 02 15:30:55 192-168-0-173 systemd[1]: Starting NVIDIA IMEX service...
Sep 02 15:30:55 192-168-0-173 systemd[1]: Started NVIDIA IMEX service.

and for single node run mpirun -n 4 ./nvbandwidth -p multinode well

and for imex domain

ubuntu@192-168-0-173:~/nvbandwidth-main$ sudo nvidia-imex-ctl -N
Connectivity Table Legend:
I - Invalid - Node wasn't reachable, no connection status available
N - Never Connected
R - Recovering - Connection was lost, but clean up has not yet been triggered.
D - Disconnected - Connection was lost, and clean up has been triggreed.
A - Authenticating - If GSSAPI enabled, client has initiated mutual authentication.
!V! - Version mismatch, communication disabled.
!M! - Node map mismatch, communication disabled.
C - Connected - Ready for operation

9/2/2025 16:42:42.493
Nodes:
Node #0   - 192.168.0.173   - READY                - Version: 570.158.01
Node #1   - 192.168.0.214   - READY                - Version: 570.158.01

 Nodes From\To  0   1  
       0        C   C  
       1        C   C  
Domain State: UP

[esitaridi]

@pzcyyj are you getting the exact same issue? Do you see any error in dmesh?
@chtcvl do you still experience this issue? Could you share the output of systemctl status nvidia-imex.service if yes?

[pzcyyj]

@pzcyyj are you getting the exact same issue? Do you see any error in dmesh? @chtcvl do you still experience this issue? Could you share the output of systemctl status nvidia-imex.service if yes?

yes, the same issue, on B200 two nodes.
the same error also on H800 machine, I guess it's about nvidia-imex, but i follow the official document to set something about channels, it doesn't work

ubuntu@192-168-110-4:~$ mpirun --allow-run-as-root \
--hostfile /etc/nvidia-imex/nodes_config.cfg \
--map-by ppr:8:node \
--bind-to core -np 16 \
--report-bindings -q \
-mca btl_tcp_if_include eth0 \
-x LD_LIBRARY_PATH \
-x PATH /home/ubuntu/nvbandwidth/nvbandwidth -p multinode
[192-168-110-4:597866] MCW rank 0 bound to socket 0[core 0[hwt 0]]: [BB/../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../..][../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../..]
[192-168-110-4:597866] MCW rank 1 bound to socket 0[core 1[hwt 0]]: [../BB/../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../..][../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../..]
[192-168-110-4:597866] MCW rank 2 bound to socket 0[core 2[hwt 0]]: [../../BB/../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../..][../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../..]
[192-168-110-4:597866] MCW rank 3 bound to socket 0[core 3[hwt 0]]: [../../../BB/../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../..][../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../..]
[192-168-110-4:597866] MCW rank 4 bound to socket 0[core 4[hwt 0]]: [../../../../BB/../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../..][../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../..]
[192-168-110-4:597866] MCW rank 5 bound to socket 0[core 5[hwt 0]]: [../../../../../BB/../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../..][../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../..]
[192-168-110-4:597866] MCW rank 6 bound to socket 0[core 6[hwt 0]]: [../../../../../../BB/../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../..][../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../..]
[192-168-110-4:597866] MCW rank 7 bound to socket 0[core 7[hwt 0]]: [../../../../../../../BB/../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../..][../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../..]
[192-168-80-239:262340] MCW rank 8 bound to socket 0[core 0[hwt 0]]: [BB/../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../..][../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../..]
[192-168-80-239:262340] MCW rank 9 bound to socket 0[core 1[hwt 0]]: [../BB/../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../..][../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../..]
[192-168-80-239:262340] MCW rank 10 bound to socket 0[core 2[hwt 0]]: [../../BB/../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../..][../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../..]
[192-168-80-239:262340] MCW rank 11 bound to socket 0[core 3[hwt 0]]: [../../../BB/../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../..][../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../..]
[192-168-80-239:262340] MCW rank 12 bound to socket 0[core 4[hwt 0]]: [../../../../BB/../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../..][../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../..]
[192-168-80-239:262340] MCW rank 13 bound to socket 0[core 5[hwt 0]]: [../../../../../BB/../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../..][../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../..]
[192-168-80-239:262340] MCW rank 14 bound to socket 0[core 6[hwt 0]]: [../../../../../../BB/../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../..][../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../..]
[192-168-80-239:262340] MCW rank 15 bound to socket 0[core 7[hwt 0]]: [../../../../../../../BB/../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../..][../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../..]
nvbandwidth Version: v0.8
Built from Git version: v0.8

MPI version: Open MPI v4.1.7a1, package: Open MPI root@c-237-149-80-085 Distribution, ident: 4.1.7a1, repo rev: v4.1.5-105-g821f7a1, Nov 04, 2023
CUDA Runtime Version: 12060
CUDA Driver Version: 12060
Driver Version: 560.35.03

Process  0 (192-168-110-4): device 0: NVIDIA H800 (00000000:01:0a)
Process  1 (192-168-110-4): device 1: NVIDIA H800 (00000000:01:0b)
Process  2 (192-168-110-4): device 2: NVIDIA H800 (00000000:02:0a)
Process  3 (192-168-110-4): device 3: NVIDIA H800 (00000000:02:0b)
Process  4 (192-168-110-4): device 4: NVIDIA H800 (00000000:03:0a)
Process  5 (192-168-110-4): device 5: NVIDIA H800 (00000000:03:0b)
Process  6 (192-168-110-4): device 6: NVIDIA H800 (00000000:04:0a)
Process  7 (192-168-110-4): device 7: NVIDIA H800 (00000000:04:0b)
Process  8 (192-168-80-239): device 0: NVIDIA H800 (00000000:01:0a)
Process  9 (192-168-80-239): device 1: NVIDIA H800 (00000000:01:0b)
Process 10 (192-168-80-239): device 2: NVIDIA H800 (00000000:02:0a)
Process 11 (192-168-80-239): device 3: NVIDIA H800 (00000000:02:0b)
Process 12 (192-168-80-239): device 4: NVIDIA H800 (00000000:03:0a)
Process 13 (192-168-80-239): device 5: NVIDIA H800 (00000000:03:0b)
Process 14 (192-168-80-239): device 6: NVIDIA H800 (00000000:04:0a)
Process 15 (192-168-80-239): device 7: NVIDIA H800 (00000000:04:0b)

Running multinode_device_to_device_memcpy_read_ce.
[CUDA_ERROR_INVALID_VALUE] invalid argument in expression cuMemSetAccess((CUdeviceptr) buffer, roundedUpAllocationSize, &desc, 1 ) on 192-168-80-239, rank = 8 in MultinodeMemoryAllocationUnicast::MultinodeMemoryAllocationUnicast(size_t, int)() : /home/ubuntu/nvbandwidth/multinode_memcpy.cpp:76

And all of them can pass nccl-tests two nodes without any problem

[pzcyyj]

@esitaridi Is there any additional information needed? Is it true that the multi-node version can only run on the GB200 NVL72?

[esitaridi]

"Is it true that the multi-node version can only run on the GB200 NVL72?"
@pzcyyj that is not accurate, it should be able to run on your system, B200 and H800, your toolkit is 12.6 (>12.3).

[esitaridi]

@pzcyyj have you also created the imex channels after starting the imex daemon: https://docs.nvidia.com/multi-node-nvlink-systems/imex-guide/imexchannels.html ?

[pzcyyj]

@esitaridi yes, two nodes have the same config

ubuntu@192-168-80-239:~$ cat /proc/devices | grep nvidia-caps-imex-channels
235 nvidia-caps-imex-channels

ubuntu@192-168-80-239:~/nvbandwidth$ ll /dev/nvidia-caps-imex-channels/
total 0
drwxr-xr-x  2 root   root       60 Sep 12 18:09 ./
drwxr-xr-x 19 root   root     4920 Sep 12 18:09 ../
crw-------  1 ubuntu ubuntu 235, 0 Sep 12 18:09 channel0

[esitaridi]

@pzcyyj looking at your setup, everything appears correct, could you provide the output for:

journalctl -u nvidia-imex.service -f

How many GPUs are there on the B200 system? On H800 it's 8 per node?

[pzcyyj]

@esitaridi B200 system is two nodes, 8 per node. H800 else.
from the imex restart to run nvbandwidth, only two lines

Sep 17 09:24:40 192-168-110-4 systemd[1]: Starting NVIDIA IMEX service...
Sep 17 09:24:40 192-168-110-4 systemd[1]: Started NVIDIA IMEX service.

Which GPU is more likely to succeed? Do you have any recommended environment I could try again?

[esitaridi]

@pzcyyj any errors in dmesg around the test execution? I am not aware of any limitations for any of the systems